transformers/pr_33892/en/main_classes/kernels.md

## Kernels

This page documents the kernels configuration utilities.

### KernelConfig[[transformers.KernelConfig]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class transformers.KernelConfig</name><anchor>transformers.KernelConfig</anchor><source>https://github.com/huggingface/transformers/blob/vr_33892/src/transformers/utils/kernel_config.py#L75</source><parameters>[{"name": "kernel_mapping", "val": " = {}"}]</parameters></docstring>

Kernel configuration class. This class is used to configure the kernel mapping for a model.



<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>create_compatible_mapping</name><anchor>transformers.KernelConfig.create_compatible_mapping</anchor><source>https://github.com/huggingface/transformers/blob/vr_33892/src/transformers/utils/kernel_config.py#L167</source><parameters>[{"name": "model", "val": ""}, {"name": "compile", "val": " = False"}]</parameters></docstring>

Transforms a simple kernel_mapping of the form:
{
"RMSNorm":
"kernels-community/layer_norm:LlamaRMSNorm",
...
},

or

{
"RMSNorm": {
"cuda":
"kernels-community/layer_norm:LlamaRMSNorm",
"rocm":
"kernels-community/layer_norm:LlamaRMSNorm",
...
},
...
}

into a nested mapping:

{
"RMSNorm": {
"cuda": {
Mode.INFERENCE: LayerRepository(
repo_id="kernels-community/layer_norm",
layer_name="LlamaRMSNorm",
)
}
}
}

that's compatible with the kernels library.

The device is inferred from the model's parameters if not provided.
The Mode is inferred from the model's training state.


</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>sanitize_kernel_mapping</name><anchor>transformers.KernelConfig.sanitize_kernel_mapping</anchor><source>https://github.com/huggingface/transformers/blob/vr_33892/src/transformers/utils/kernel_config.py#L100</source><parameters>[{"name": "model", "val": ""}]</parameters><paramsdesc>- **model** -- The model instance whose modules are checked for registered kernel_layer_name attributes.</paramsdesc><paramgroups>0</paramgroups><raises>- ``ValueError`` -- If a layer_name is not registered in the model, if a device is not supported,
  or if a repo_name is not a valid 'org/repo:layer_name' string.</raises><raisederrors>``ValueError``</raisederrors></docstring>

Validates the kernel_mapping to ensure that:
1. Each layer_name in the mapping is registered in the model (i.e., the model contains a module with a matching kernel_layer_name).
2. Each kernel value is either a string of the form 'org/repo:layer_name' or a dict mapping device types ("cuda", "rocm", "xpu") to such strings.
3. Each device key in a dict is one of "cuda", "rocm", or "xpu".
4. Each repo_name is a valid repository and layer name in the format 'org/repo:layer_name' (i.e., a string containing both a slash and a colon).








</div></div>

<EditOnGithub source="https://github.com/huggingface/transformers/blob/main/docs/source/en/main_classes/kernels.md" />