Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Kernel requirements","local":"kernel-requirements","sections":[{"title":"Directory layout","local":"directory-layout","sections":[],"depth":2},{"title":"Build variants","local":"build-variants","sections":[],"depth":2},{"title":"Versioning","local":"versioning","sections":[],"depth":2},{"title":"Native Python module","local":"native-python-module","sections":[{"title":"Linux","local":"linux","sections":[],"depth":3},{"title":"macOS","local":"macos","sections":[],"depth":3},{"title":"ABI checker","local":"abi-checker","sections":[],"depth":3}],"depth":2},{"title":"Torch extension","local":"torch-extension","sections":[],"depth":2},{"title":"Layers","local":"layers","sections":[{"title":"Writing layers","local":"writing-layers","sections":[],"depth":3},{"title":"Exporting layers","local":"exporting-layers","sections":[],"depth":3}],"depth":2},{"title":"Python requirements","local":"python-requirements","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/kernels/pr_150/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/kernels/pr_150/en/_app/immutable/entry/start.15f67972.js"> | |
| <link rel="modulepreload" href="/docs/kernels/pr_150/en/_app/immutable/chunks/scheduler.7da89386.js"> | |
| <link rel="modulepreload" href="/docs/kernels/pr_150/en/_app/immutable/chunks/singletons.e3529dd1.js"> | |
| <link rel="modulepreload" href="/docs/kernels/pr_150/en/_app/immutable/chunks/paths.dcb90d49.js"> | |
| <link rel="modulepreload" href="/docs/kernels/pr_150/en/_app/immutable/entry/app.13bdcfa3.js"> | |
| <link rel="modulepreload" href="/docs/kernels/pr_150/en/_app/immutable/chunks/index.0b7befd3.js"> | |
| <link rel="modulepreload" href="/docs/kernels/pr_150/en/_app/immutable/nodes/0.fadb2962.js"> | |
| <link rel="modulepreload" href="/docs/kernels/pr_150/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/kernels/pr_150/en/_app/immutable/nodes/10.4b6d58bd.js"> | |
| <link rel="modulepreload" href="/docs/kernels/pr_150/en/_app/immutable/chunks/CodeBlock.c5b6371f.js"> | |
| <link rel="modulepreload" href="/docs/kernels/pr_150/en/_app/immutable/chunks/getInferenceSnippets.dcce2733.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Kernel requirements","local":"kernel-requirements","sections":[{"title":"Directory layout","local":"directory-layout","sections":[],"depth":2},{"title":"Build variants","local":"build-variants","sections":[],"depth":2},{"title":"Versioning","local":"versioning","sections":[],"depth":2},{"title":"Native Python module","local":"native-python-module","sections":[{"title":"Linux","local":"linux","sections":[],"depth":3},{"title":"macOS","local":"macos","sections":[],"depth":3},{"title":"ABI checker","local":"abi-checker","sections":[],"depth":3}],"depth":2},{"title":"Torch extension","local":"torch-extension","sections":[],"depth":2},{"title":"Layers","local":"layers","sections":[{"title":"Writing layers","local":"writing-layers","sections":[],"depth":3},{"title":"Exporting layers","local":"exporting-layers","sections":[],"depth":3}],"depth":2},{"title":"Python requirements","local":"python-requirements","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="kernel-requirements" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#kernel-requirements"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Kernel requirements</span></h1> <p data-svelte-h="svelte-omjx4j">Kernels on the Hub must fulfill the requirements outlined on this page. By | |
| ensuring kernels are compliant, they can be used on a wide range of Linux | |
| systems and Torch builds.</p> <p data-svelte-h="svelte-mwob7l">You can use <a href="https://github.com/huggingface/kernel-builder/" rel="nofollow">kernel-builder</a> | |
| to build compliant kernels.</p> <h2 class="relative group"><a id="directory-layout" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#directory-layout"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Directory layout</span></h2> <p data-svelte-h="svelte-1j93b1m">A kernel repository on the Hub must contain a <code>build</code> directory. This | |
| directory contains build variants of a kernel in the form of directories | |
| following the template | |
| <code><framework><version>-cxx<abiver>-<cu><cudaver>-<arch>-<os></code>. | |
| For example <code>build/torch26-cxx98-cu118-x86_64-linux</code>.</p> <p data-svelte-h="svelte-1cuxqpl">Each variant directory must contain a single directory with the same name | |
| as the repository (replacing <code>-</code> by <code>_</code>). For instance, kernels in the | |
| <code>kernels-community/activation</code> repository have a directories like | |
| <code>build/<variant>/activation</code>. This directory | |
| must be a Python package with an <code>__init__.py</code> file.</p> <h2 class="relative group"><a id="build-variants" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#build-variants"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Build variants</span></h2> <p data-svelte-h="svelte-11ghnwy">A kernel can be compliant for a specific compute framework (e.g. CUDA) or | |
| architecture (e.g. x86_64). For compliance with a compute framework and | |
| architecture combination, all the variants from the <a href="https://github.com/huggingface/kernel-builder/blob/main/docs/build-variants.md" rel="nofollow">build variant list</a> | |
| must be available for that combination.</p> <h2 class="relative group"><a id="versioning" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#versioning"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Versioning</span></h2> <p data-svelte-h="svelte-1869t00">Kernels are versioned on the Hub using Git tags. Version tags must be of | |
| the form <code>v<major>.<minor>.<patch></code>. Versions are used by <a href="./locking.md">locking</a> | |
| to resolve the version constraints.</p> <p data-svelte-h="svelte-1lhbdsc">We recommend using <a href="https://semver.org/" rel="nofollow">semver</a> to version kernels.</p> <h2 class="relative group"><a id="native-python-module" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#native-python-module"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Native Python module</span></h2> <p data-svelte-h="svelte-17te4hc">Kernels will typically contain a native Python module with precompiled | |
| compute kernels and bindings. This module must fulfill the requirements | |
| outlined in this section. For all operating systems, a kernel must not | |
| have dynamic library dependencies outside:</p> <ul data-svelte-h="svelte-17x1sgf"><li>Torch;</li> <li>CUDA/ROCm libraries installed as dependencies of Torch.</li></ul> <h3 class="relative group"><a id="linux" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#linux"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Linux</span></h3> <ul data-svelte-h="svelte-gs8bip"><li>Use <a href="https://docs.python.org/3/c-api/stable.html#stable-application-binary-interface" rel="nofollow">ABI3/Limited API</a> | |
| for compatibility with Python 3.9 and later.</li> <li>Compatible with <a href="https://github.com/pypa/manylinux?tab=readme-ov-file#manylinux_2_28-almalinux-8-based" rel="nofollow"><code>manylinux_2_28</code></a>. | |
| This means that the extension <strong>must not</strong> use symbols versions higher than:<ul><li>GLIBC 2.28</li> <li>GLIBCXX 3.4.24</li> <li>CXXABI 1.3.11</li> <li>GCC 7.0.0</li></ul></li></ul> <p data-svelte-h="svelte-vq3k4i">These requirements can be checked with the ABI checker (see below).</p> <h3 class="relative group"><a id="macos" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#macos"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>macOS</span></h3> <ul data-svelte-h="svelte-nrm36c"><li>Use <a href="https://docs.python.org/3/c-api/stable.html#stable-application-binary-interface" rel="nofollow">ABI3/Limited API</a> | |
| for compatibility with Python 3.9 and later.</li> <li>macOS deployment target 15.0.</li> <li>Metal 3.0 (<code>-std=metal3.0</code>).</li></ul> <p data-svelte-h="svelte-1xbbttu">The ABI3 requirement can be checked with the ABI checker (see below).</p> <h3 class="relative group"><a id="abi-checker" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#abi-checker"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>ABI checker</span></h3> <p data-svelte-h="svelte-16l01xm">The manylinux_2_28 and Python ABI 3.9 version requirements can be checked with | |
| <a href="https://crates.io/crates/kernel-abi-check" rel="nofollow"><code>kernel-abi-check</code></a>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --> | |
| $ cargo install kernel-abi-check | |
| $ kernel-abi-check result/relu/_relu_e87e0ca_dirty.abi3.so | |
| 🐍 Checking <span class="hljs-keyword">for</span> compatibility with manylinux_2_28 and Python ABI version 3.9 | |
| ✅ No compatibility issues found<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="torch-extension" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#torch-extension"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Torch extension</span></h2> <p data-svelte-h="svelte-1z00i8l">Torch native extension functions must be <a href="https://pytorch.org/tutorials/advanced/cpp_custom_ops.html#cpp-custom-ops-tutorial" rel="nofollow">registered</a> | |
| in <code>torch.ops.<namespace></code>. Since we allow loading of multiple versions of | |
| a module in the same Python process, <code>namespace</code> must be unique for each | |
| version of a kernel. Failing to do so will create clashes when different | |
| versions of the same kernel are loaded. Two suggested ways of doing this | |
| are:</p> <ul data-svelte-h="svelte-1n44n6m"><li>Appending a truncated SHA-1 hash of the git commit that the kernel was | |
| built from to the name of the extension.</li> <li>Appending random material to the name of the extension.</li></ul> <p data-svelte-h="svelte-vo8uhp"><strong>Note:</strong> we recommend against appending a version number or git tag. | |
| Version numbers are typically not bumped on each commit, so users | |
| might use two different commits that happen to have the same version | |
| number. Git tags are not stable, so they do not provide a good way | |
| of guaranteeing uniqueness of the namespace.</p> <h2 class="relative group"><a id="layers" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#layers"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Layers</span></h2> <p data-svelte-h="svelte-17jmyq">A kernel can provide layers in addition to kernel functions. A layer from | |
| the Hub can replace the <code>forward</code> method of an existing layer for a certain | |
| device type. This makes it possible to provide more performant kernels for | |
| existing layers. See the <a href="layers.md">layers documentation</a> for more information | |
| on how to use layers.</p> <h3 class="relative group"><a id="writing-layers" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#writing-layers"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Writing layers</span></h3> <p data-svelte-h="svelte-ew142v">To make the extension of layers safe, the layers must fulfill the following | |
| requirements:</p> <ul data-svelte-h="svelte-pm5vd9"><li>The layers are subclasses of <code>torch.nn.Module</code>.</li> <li>The layers are pure, meaning that they do not have their own state. This | |
| means that:<ul><li>The layer must not define its own constructor.</li> <li>The layer must not use class variables.</li></ul></li> <li>No other methods must be defined than <code>forward</code>.</li> <li>The <code>forward</code> method has a signature that is compatible with the | |
| <code>forward</code> method that it is extending.</li></ul> <p data-svelte-h="svelte-17brked">There are two exceptions to the <em>no class variables rule</em>:</p> <ol data-svelte-h="svelte-1a9mvrx"><li>The <code>has_backward</code> variable can be used to indicate whether the layer has | |
| a backward pass implemented (<code>True</code> when absent).</li> <li>The <code>can_torch_compile</code> variable can be used to indicate whether the layer | |
| supports <code>torch.compile</code> (<code>False</code> when absent).</li></ol> <p data-svelte-h="svelte-mthj86">This is an example of a pure layer:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">class</span> <span class="hljs-title class_">SiluAndMul</span>(nn.Module): | |
| <span class="hljs-comment"># This layer does not implement backward.</span> | |
| has_backward: <span class="hljs-built_in">bool</span> = <span class="hljs-literal">False</span> | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">forward</span>(<span class="hljs-params">self, x: torch.Tensor</span>): | |
| d = x.shape[-<span class="hljs-number">1</span>] // <span class="hljs-number">2</span> | |
| output_shape = x.shape[:-<span class="hljs-number">1</span>] + (d,) | |
| out = torch.empty(output_shape, dtype=x.dtype, device=x.device) | |
| ops.silu_and_mul(out, x) | |
| <span class="hljs-keyword">return</span> out<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1xsl1kb">For some layers, the <code>forward</code> method has to use state from the adopting class. | |
| In these cases, we recommend to use type annotations to indicate what member | |
| variables are expected. For instance:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">class</span> <span class="hljs-title class_">LlamaRMSNorm</span>(nn.Module): | |
| weight: torch.Tensor | |
| variance_epsilon: <span class="hljs-built_in">float</span> | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">forward</span>(<span class="hljs-params">self, hidden_states: torch.Tensor</span>) -> torch.Tensor: | |
| <span class="hljs-keyword">return</span> rms_norm_fn( | |
| hidden_states, | |
| self.weight, | |
| bias=<span class="hljs-literal">None</span>, | |
| residual=<span class="hljs-literal">None</span>, | |
| eps=self.variance_epsilon, | |
| dropout_p=<span class="hljs-number">0.0</span>, | |
| prenorm=<span class="hljs-literal">False</span>, | |
| residual_in_fp32=<span class="hljs-literal">False</span>, | |
| )<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-9hcs08">This layer expects the adopting layer to have <code>weight</code> and <code>variance_epsilon</code> | |
| member variables and uses them in the <code>forward</code> method.</p> <h3 class="relative group"><a id="exporting-layers" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#exporting-layers"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Exporting layers</span></h3> <p data-svelte-h="svelte-1jwjnkj">To accommodate portable loading, <code>layers</code> must be defined in the main | |
| <code>__init__.py</code> file. For example:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> . <span class="hljs-keyword">import</span> layers | |
| __all__ = [ | |
| <span class="hljs-comment"># ...</span> | |
| <span class="hljs-string">"layers"</span> | |
| <span class="hljs-comment"># ...</span> | |
| ]<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="python-requirements" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#python-requirements"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Python requirements</span></h2> <ul><li data-svelte-h="svelte-1yjmw8t"><p>Python code must be compatible with Python 3.9 and later.</p></li> <li><p data-svelte-h="svelte-1xx048c">All Python code imports from the kernel itself must be relative. So, | |
| for instance if in the example kernel <code>example</code>, | |
| <code>module_b</code> needs a function from <code>module_a</code>, import as:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> .module_a <span class="hljs-keyword">import</span> foo<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-ca1g42"><strong>Never use:</strong></p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-comment"># DO NOT DO THIS!</span> | |
| <span class="hljs-keyword">from</span> example.module_a <span class="hljs-keyword">import</span> foo<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1v83762">The latter would import from the module <code>example</code> that is in Python’s | |
| global module dict. However, since we allow loading multiple versions | |
| of a module, we uniquely name the module.</p></li> <li data-svelte-h="svelte-wkdo8"><p>Only modules from the Python standard library, Torch, or the kernel itself | |
| can be imported.</p></li></ul> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/kernels/blob/main/docs/source/kernel-requirements.md" target="_blank"><span data-svelte-h="svelte-1kd6by1"><</span> <span data-svelte-h="svelte-x0xyl0">></span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_cbfb4u = { | |
| assets: "/docs/kernels/pr_150/en", | |
| base: "/docs/kernels/pr_150/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/kernels/pr_150/en/_app/immutable/entry/start.15f67972.js"), | |
| import("/docs/kernels/pr_150/en/_app/immutable/entry/app.13bdcfa3.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 10], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 37.1 kB
- Xet hash:
- b04c69e18893079a11de940b56e17a58d3b4e43d1f3d8652bc9087959adecf98
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.