Buckets:

hf-doc-build
/

doc-dev

Files

xet

hf-doc-build/doc-dev / transformers /pr_46007 /en /_app /immutable /nodes /13.uO2F7zBs.js

HuggingFaceDocBuilder

12 days ago

download

raw

18.8 kB

import"../chunks/DsnmJJEf.js";import{i as T,h as w,C as b,H as l,a,E as g,s as _}from"../chunks/qxQfBDOH.js";import{p as C,o as I,s as e,f as Z,a as U,b as X,c as m,n as B}from"../chunks/Dpn4EqEf.js";const W='{"title":"Per-layer configurations","local":"per-layer-configurations","sections":[{"title":"Define per-layer overrides","local":"define-per-layer-overrides","sections":[],"depth":2},{"title":"Global attribute access","local":"global-attribute-access","sections":[],"depth":2},{"title":"Serialization","local":"serialization","sections":[],"depth":2}],"depth":1}';var V=m('<meta name="hf:doc:metadata"/>'),v=m(`<p></p> <!> <!> <p>Some checkpoints are heterogeneous (not layer-uniform). A smaller MLP in one layer, fewer key-value heads in another, or a different
layout on selected layers means a single global config doesn’t accurately describe the stack.</p> <p>Use <code>per_layer_config</code> on <a href="/docs/transformers/pr_46007/en/main_classes/configuration#transformers.PreTrainedConfig">PreTrainedConfig</a> to record those diffs when you’re authoring or
inspecting a config, and when model code will consume them. Each entry stores only what differs from the
global configuration, the rest inherits.</p> <blockquote class="note"><p>Heterogeneous configurations are a power feature. If a heterogeneous layout becomes a common or prominent
architecture, we will strive to model it explicitly in the architecture implementation rather than rely on <code>per_layer_config</code>. Prefer the explicit architecture when one exists.</p></blockquote> <p>The models below are heterogeneous checkpoints. Layers are not uniform across the stack. They use a dedicated architecture with <code>block_configs</code> and their own <code>model_type</code>, rather than <code>per_layer_config</code> on a standard architecture.</p> <table><thead><tr><th>Model</th><th>Derived from</th></tr></thead><tbody><tr><td><a href="https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1_5" rel="nofollow">nvidia/Llama-3_3-Nemotron-Super-49B-v1_5</a></td><td><a href="https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct" rel="nofollow">meta-llama/Llama-3.3-70B-Instruct</a></td></tr><tr><td><a href="https://huggingface.co/nvidia/Llama-3_1-Nemotron-Ultra-253B-v1" rel="nofollow">nvidia/Llama-3_1-Nemotron-Ultra-253B-v1</a></td><td><a href="https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct" rel="nofollow">meta-llama/Llama-3.1-405B-Instruct</a></td></tr><tr><td><a href="https://huggingface.co/nvidia/gpt-oss-puzzle-88B" rel="nofollow">nvidia/gpt-oss-puzzle-88B</a></td><td><a href="https://huggingface.co/openai/gpt-oss-120b" rel="nofollow">openai/gpt-oss-120b</a></td></tr><tr><td><a href="https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16" rel="nofollow">nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16</a></td><td><a href="https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16" rel="nofollow">nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16</a></td></tr></tbody></table> <!> <p>Pass <code>per_layer_config</code> to <a href="/docs/transformers/pr_46007/en/model_doc/llama2#transformers.LlamaConfig">LlamaConfig</a> as a mapping from layer indices to attribute overrides. Layer
indices are zero-based. Only attributes that differ from the global configuration need to be specified.</p> <p><code>per_layer_config</code> records and resolves configuration values. It does not by itself change the modules a model creates
or how those modules run. Applying a size override or a <code>skip</code> requires model code that reads the resolved per-layer
configuration when constructing or running each layer. The current <code>LlamaModel</code> constructs every <code>LlamaDecoderLayer</code> with the global configuration, so the <code>skip</code> entries below remain configuration values rather than removing modules
after <code>from_pretrained</code>.</p> <p>The following example records overrides for four layers: layer 5 uses a smaller MLP, layer 11 uses
fewer key-value heads, and layers 23 and 27 record <code>skip</code> values for architectures that support them.</p> <!> <p>The submodules that an architecture can skip (for example, <code>"mlp"</code> and <code>"attention"</code>) are defined per architecture. <code>skip</code> accepts a list, so a layer can record more than one submodule override.</p> <p>Accessing <code>config.per_layer_config[layer_idx]</code> returns a resolved layer configuration. The resolved configuration
combines the global configuration with the overrides for that layer.</p> <!> <p>Configurations that use <code>per_layer_config</code> support the same <a href="/docs/transformers/pr_46007/en/main_classes/configuration#transformers.PreTrainedConfig.save_pretrained">save_pretrained()</a> and <a href="/docs/transformers/pr_46007/en/main_classes/configuration#transformers.PreTrainedConfig.from_pretrained">from_pretrained()</a> round trip as other configurations.</p> <p>Each architecture defines in its code which attributes it consumes at the layer level. <code>per_layer_config</code> provides the
mechanism for recording those layer-level differences and resolving them against the global config.</p> <!> <p>An attribute with per-layer overrides does not have a single model-wide value. <code>num_key_value_heads</code> may be <code>8</code> on
most layers and <code>4</code> on selected layers. Reading <code>config.num_key_value_heads</code> outside a layer context is ambiguous.</p> <p>By default that access raises <code>AmbiguousGlobalPerLayerAttributeError</code> and points you to <code>config.per_layer_config[layer_idx]</code>. The attribute still exists on the global config, so this is not an <code>AttributeError</code>. Reading that global value without a layer index is still wrong. Code that builds a key-value cache from a global <code>num_key_value_heads</code> would size the wrong layers incorrectly.</p> <p>Set <code>allow_global_per_layer_attribute_access=True</code> only when you intentionally need the global fallback and can handle
heterogeneous configs. Global access is then allowed. A warning is emitted once.</p> <!> <!> <p><code>per_layer_config</code> serializes sparsely by default, and layers without overrides are omitted. Overridden attributes that
match the global value are omitted too.</p> <!> <p>Set <code>serialize_explicit_per_layer_config=True</code> to include every layer for the attributes represented in <code>per_layer_config</code>. That makes the layer layout easier to inspect when some values still match the global
configuration.</p> <!> <p>Use sparse serialization for compact configs, and explicit serialization when you need the full per-layer layout for
readability or tooling.</p> <!> <p></p>`,1);function G(J,f){C(f,!1),I(()=>{new URLSearchParams(window.location.search).get("fw")}),T();var s=v();w("pkalzc",d=>{var u=V();_(u,"content",W),U(d,u)});var n=e(Z(s),2);b(n,{containerStyle:"float: right; margin-left: 10px; display: inline-flex; position: relative; z-index: 10;"});var o=e(n,2);l(o,{title:"Per-layer configurations",local:"per-layer-configurations",headingTag:"h1"});var t=e(o,12);l(t,{title:"Define per-layer overrides",local:"define-per-layer-overrides",headingTag:"h2"});var r=e(t,8);a(r,{code:"ZnJvbSUyMHRyYW5zZm9ybWVycyUyMGltcG9ydCUyMExsYW1hQ29uZmlnJTBBJTBBJTBBY29uZmlnJTIwJTNEJTIwTGxhbWFDb25maWcoJTBBJTIwJTIwJTIwJTIwaGlkZGVuX3NpemUlM0Q0MDk2JTJDJTBBJTIwJTIwJTIwJTIwaW50ZXJtZWRpYXRlX3NpemUlM0QxNDMzNiUyQyUwQSUyMCUyMCUyMCUyMG51bV9oaWRkZW5fbGF5ZXJzJTNEMzIlMkMlMEElMjAlMjAlMjAlMjBudW1fYXR0ZW50aW9uX2hlYWRzJTNEMzIlMkMlMEElMjAlMjAlMjAlMjBudW1fa2V5X3ZhbHVlX2hlYWRzJTNEOCUyQyUwQSUyMCUyMCUyMCUyMHBlcl9sYXllcl9jb25maWclM0QlN0IlMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjMlMjBVc2UlMjBhJTIwc21hbGxlciUyME1MUCUyMGluJTIwb25lJTIwbGF5ZXIuJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwNSUzQSUyMCU3QiUyMmludGVybWVkaWF0ZV9zaXplJTIyJTNBJTIwODE5MiU3RCUyQyUwQSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMyUyMFVzZSUyMGZld2VyJTIwa2V5LXZhbHVlJTIwaGVhZHMlMjBpbiUyMGFub3RoZXIlMjBsYXllci4lMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAxMSUzQSUyMCU3QiUyMm51bV9rZXlfdmFsdWVfaGVhZHMlMjIlM0ElMjA0JTdEJTJDJTBBJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIzJTIwUmVjb3JkJTIwYSUyMHJlcXVlc3QlMjB0byUyMHNraXAlMjB0aGUlMjBNTFAlMjBpbiUyMGFyY2hpdGVjdHVyZXMlMjB0aGF0JTIwc3VwcG9ydCUyMGl0LiUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMDIzJTNBJTIwJTdCJTIyc2tpcCUyMiUzQSUyMCU1QiUyMm1scCUyMiU1RCU3RCUyQyUwQSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMyUyMFJlY29yZCUyMGElMjByZXF1ZXN0JTIwdG8lMjBza2lwJTIwYXR0ZW50aW9uJTIwaW4lMjBhcmNoaXRlY3R1cmVzJTIwdGhhdCUyMHN1cHBvcnQlMjBpdC4lMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAyNyUzQSUyMCU3QiUyMnNraXAlMjIlM0ElMjAlNUIlMjJhdHRlbnRpb24lMjIlNUQlN0QlMkMlMEElMjAlMjAlMjAlMjAlN0QlMkMlMEEp",highlighted:`<span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> LlamaConfig
config = LlamaConfig(
    hidden_size=<span class="hljs-number">4096</span>,
    intermediate_size=<span class="hljs-number">14336</span>,
    num_hidden_layers=<span class="hljs-number">32</span>,
    num_attention_heads=<span class="hljs-number">32</span>,
    num_key_value_heads=<span class="hljs-number">8</span>,
    per_layer_config={
        <span class="hljs-comment"># Use a smaller MLP in one layer.</span>
        <span class="hljs-number">5</span>: {<span class="hljs-string">&quot;intermediate_size&quot;</span>: <span class="hljs-number">8192</span>},
        <span class="hljs-comment"># Use fewer key-value heads in another layer.</span>
        <span class="hljs-number">11</span>: {<span class="hljs-string">&quot;num_key_value_heads&quot;</span>: <span class="hljs-number">4</span>},
        <span class="hljs-comment"># Record a request to skip the MLP in architectures that support it.</span>
        <span class="hljs-number">23</span>: {<span class="hljs-string">&quot;skip&quot;</span>: [<span class="hljs-string">&quot;mlp&quot;</span>]},
        <span class="hljs-comment"># Record a request to skip attention in architectures that support it.</span>
        <span class="hljs-number">27</span>: {<span class="hljs-string">&quot;skip&quot;</span>: [<span class="hljs-string">&quot;attention&quot;</span>]},
    },
)`,lang:"py",wrap:!1});var i=e(r,6);a(i,{code:"JTIzJTIwTGF5ZXIlMjAwJTIwZG9lcyUyMG5vdCUyMGRlZmluZSUyMG92ZXJyaWRlcyUyQyUyMHNvJTIwaXQlMjBpbmhlcml0cyUyMHRoZSUyMGdsb2JhbCUyMHZhbHVlcy4lMEFjb25maWcucGVyX2xheWVyX2NvbmZpZyU1QjAlNUQuaW50ZXJtZWRpYXRlX3NpemUlMEElMjMlMjAxNDMzNiUwQSUwQWNvbmZpZy5wZXJfbGF5ZXJfY29uZmlnJTVCMCU1RC5udW1fa2V5X3ZhbHVlX2hlYWRzJTBBJTIzJTIwOCUwQSUwQSUyMyUyMExheWVyJTIwNSUyMG92ZXJyaWRlcyUyMHRoZSUyME1MUCUyMGludGVybWVkaWF0ZSUyMHNpemUuJTBBY29uZmlnLnBlcl9sYXllcl9jb25maWclNUI1JTVELmludGVybWVkaWF0ZV9zaXplJTBBJTIzJTIwODE5MiUwQSUwQSUyMyUyMExheWVyJTIwMTElMjBvdmVycmlkZXMlMjB0aGUlMjBudW1iZXIlMjBvZiUyMGtleS12YWx1ZSUyMGhlYWRzLiUwQWNvbmZpZy5wZXJfbGF5ZXJfY29uZmlnJTVCMTElNUQubnVtX2tleV92YWx1ZV9oZWFkcyUwQSUyMyUyMDQlMEElMEElMjMlMjBMYXllciUyMDIzJTIwcmVjb3JkcyUyMGFuJTIwTUxQJTIwc2tpcC4lMEFjb25maWcucGVyX2xheWVyX2NvbmZpZyU1QjIzJTVELnNraXAlMEElMjMlMjAlNUIlMjJtbHAlMjIlNUQlMEElMEElMjMlMjBMYXllciUyMDI3JTIwcmVjb3JkcyUyMGFuJTIwYXR0ZW50aW9uJTIwc2tpcC4lMEFjb25maWcucGVyX2xheWVyX2NvbmZpZyU1QjI3JTVELnNraXAlMEElMjMlMjAlNUIlMjJhdHRlbnRpb24lMjIlNUQ=",highlighted:`<span class="hljs-comment"># Layer 0 does not define overrides, so it inherits the global values.</span>
config.per_layer_config[<span class="hljs-number">0</span>].intermediate_size
<span class="hljs-comment"># 14336</span>
config.per_layer_config[<span class="hljs-number">0</span>].num_key_value_heads
<span class="hljs-comment"># 8</span>
<span class="hljs-comment"># Layer 5 overrides the MLP intermediate size.</span>
config.per_layer_config[<span class="hljs-number">5</span>].intermediate_size
<span class="hljs-comment"># 8192</span>
<span class="hljs-comment"># Layer 11 overrides the number of key-value heads.</span>
config.per_layer_config[<span class="hljs-number">11</span>].num_key_value_heads
<span class="hljs-comment"># 4</span>
<span class="hljs-comment"># Layer 23 records an MLP skip.</span>
config.per_layer_config[<span class="hljs-number">23</span>].skip
<span class="hljs-comment"># [&quot;mlp&quot;]</span>
<span class="hljs-comment"># Layer 27 records an attention skip.</span>
config.per_layer_config[<span class="hljs-number">27</span>].skip
<span class="hljs-comment"># [&quot;attention&quot;]</span>`,lang:"py",wrap:!1});var c=e(i,6);l(c,{title:"Global attribute access",local:"global-attribute-access",headingTag:"h2"});var y=e(c,8);a(y,{code:"Y29uZmlnJTIwJTNEJTIwTGxhbWFDb25maWcoJTBBJTIwJTIwJTIwJTIwaGlkZGVuX3NpemUlM0Q0MDk2JTJDJTBBJTIwJTIwJTIwJTIwaW50ZXJtZWRpYXRlX3NpemUlM0QxNDMzNiUyQyUwQSUyMCUyMCUyMCUyMG51bV9oaWRkZW5fbGF5ZXJzJTNEMzIlMkMlMEElMjAlMjAlMjAlMjBudW1fYXR0ZW50aW9uX2hlYWRzJTNEMzIlMkMlMEElMjAlMjAlMjAlMjBudW1fa2V5X3ZhbHVlX2hlYWRzJTNEOCUyQyUwQSUyMCUyMCUyMCUyMGFsbG93X2dsb2JhbF9wZXJfbGF5ZXJfYXR0cmlidXRlX2FjY2VzcyUzRFRydWUlMkMlMEElMjAlMjAlMjAlMjBwZXJfbGF5ZXJfY29uZmlnJTNEJTdCJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwMTElM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwNCU3RCUyQyUwQSUyMCUyMCUyMCUyMCU3RCUyQyUwQSklMEElMEFjb25maWcubnVtX2tleV92YWx1ZV9oZWFkcyUwQSUyMyUyMDglMEElMjMlMjBFbWl0cyUyMGElMjBvbmUtdGltZSUyMHdhcm5pbmclMjBiZWNhdXNlJTIwbnVtX2tleV92YWx1ZV9oZWFkcyUyMGhhcyUyMGElMjBwZXItbGF5ZXIlMjBvdmVycmlkZS4=",highlighted:`config = LlamaConfig(
    hidden_size=<span class="hljs-number">4096</span>,
    intermediate_size=<span class="hljs-number">14336</span>,
    num_hidden_layers=<span class="hljs-number">32</span>,
    num_attention_heads=<span class="hljs-number">32</span>,
    num_key_value_heads=<span class="hljs-number">8</span>,
    allow_global_per_layer_attribute_access=<span class="hljs-literal">True</span>,
    per_layer_config={
        <span class="hljs-number">11</span>: {<span class="hljs-string">&quot;num_key_value_heads&quot;</span>: <span class="hljs-number">4</span>},
    },
)
config.num_key_value_heads
<span class="hljs-comment"># 8</span>
<span class="hljs-comment"># Emits a one-time warning because num_key_value_heads has a per-layer override.</span>`,lang:"py",wrap:!1});var M=e(y,2);l(M,{title:"Serialization",local:"serialization",headingTag:"h2"});var p=e(M,4);a(p,{code:"ZnJvbSUyMHRyYW5zZm9ybWVycyUyMGltcG9ydCUyMExsYW1hQ29uZmlnJTBBJTBBJTBBY29uZmlnJTIwJTNEJTIwTGxhbWFDb25maWcoJTBBJTIwJTIwJTIwJTIwaGlkZGVuX3NpemUlM0Q0MDk2JTJDJTBBJTIwJTIwJTIwJTIwaW50ZXJtZWRpYXRlX3NpemUlM0QxNDMzNiUyQyUwQSUyMCUyMCUyMCUyMG51bV9oaWRkZW5fbGF5ZXJzJTNENCUyQyUwQSUyMCUyMCUyMCUyMG51bV9hdHRlbnRpb25faGVhZHMlM0QzMiUyQyUwQSUyMCUyMCUyMCUyMG51bV9rZXlfdmFsdWVfaGVhZHMlM0Q4JTJDJTBBJTIwJTIwJTIwJTIwcGVyX2xheWVyX2NvbmZpZyUzRCU3QiUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMDAlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwOCU3RCUyQyUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMDIlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwNCU3RCUyQyUwQSUyMCUyMCUyMCUyMCU3RCUyQyUwQSklMEElMEFjb25maWcudG9fZGljdCgpJTVCJTIycGVyX2xheWVyX2NvbmZpZyUyMiU1RCUwQSUyMyUyMCU3QiUyMjIlMjIlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwNCU3RCU3RA==",highlighted:`<span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> LlamaConfig
config = LlamaConfig(
    hidden_size=<span class="hljs-number">4096</span>,
    intermediate_size=<span class="hljs-number">14336</span>,
    num_hidden_layers=<span class="hljs-number">4</span>,
    num_attention_heads=<span class="hljs-number">32</span>,
    num_key_value_heads=<span class="hljs-number">8</span>,
    per_layer_config={
        <span class="hljs-number">0</span>: {<span class="hljs-string">&quot;num_key_value_heads&quot;</span>: <span class="hljs-number">8</span>},
        <span class="hljs-number">2</span>: {<span class="hljs-string">&quot;num_key_value_heads&quot;</span>: <span class="hljs-number">4</span>},
    },
)
config.to_dict()[<span class="hljs-string">&quot;per_layer_config&quot;</span>]
<span class="hljs-comment"># {&quot;2&quot;: {&quot;num_key_value_heads&quot;: 4}}</span>`,lang:"py",wrap:!1});var h=e(p,4);a(h,{code:"ZXhwbGljaXRfY29uZmlnJTIwJTNEJTIwTGxhbWFDb25maWcoJTBBJTIwJTIwJTIwJTIwaGlkZGVuX3NpemUlM0Q0MDk2JTJDJTBBJTIwJTIwJTIwJTIwaW50ZXJtZWRpYXRlX3NpemUlM0QxNDMzNiUyQyUwQSUyMCUyMCUyMCUyMG51bV9oaWRkZW5fbGF5ZXJzJTNENCUyQyUwQSUyMCUyMCUyMCUyMG51bV9hdHRlbnRpb25faGVhZHMlM0QzMiUyQyUwQSUyMCUyMCUyMCUyMG51bV9rZXlfdmFsdWVfaGVhZHMlM0Q4JTJDJTBBJTIwJTIwJTIwJTIwc2VyaWFsaXplX2V4cGxpY2l0X3Blcl9sYXllcl9jb25maWclM0RUcnVlJTJDJTBBJTIwJTIwJTIwJTIwcGVyX2xheWVyX2NvbmZpZyUzRCU3QiUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMDAlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwOCU3RCUyQyUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMDIlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwNCU3RCUyQyUwQSUyMCUyMCUyMCUyMCU3RCUyQyUwQSklMEElMEFzZXJpYWxpemVkX3Blcl9sYXllcl9jb25maWclMjAlM0QlMjBleHBsaWNpdF9jb25maWcudG9fZGljdCgpJTVCJTIycGVyX2xheWVyX2NvbmZpZyUyMiU1RCUwQSUwQXNlcmlhbGl6ZWRfcGVyX2xheWVyX2NvbmZpZyUwQSUyMyUyMCU3QiUwQSUyMyUyMCUyMCUyMCUyMCUyMCUyMjAlMjIlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwOCU3RCUyQyUwQSUyMyUyMCUyMCUyMCUyMCUyMCUyMjElMjIlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwOCU3RCUyQyUwQSUyMyUyMCUyMCUyMCUyMCUyMCUyMjIlMjIlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwNCU3RCUyQyUwQSUyMyUyMCUyMCUyMCUyMCUyMCUyMjMlMjIlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwOCU3RCUyQyUwQSUyMyUyMCU3RA==",highlighted:`explicit_config = LlamaConfig(
    hidden_size=<span class="hljs-number">4096</span>,
    intermediate_size=<span class="hljs-number">14336</span>,
    num_hidden_layers=<span class="hljs-number">4</span>,
    num_attention_heads=<span class="hljs-number">32</span>,
    num_key_value_heads=<span class="hljs-number">8</span>,
    serialize_explicit_per_layer_config=<span class="hljs-literal">True</span>,
    per_layer_config={
        <span class="hljs-number">0</span>: {<span class="hljs-string">&quot;num_key_value_heads&quot;</span>: <span class="hljs-number">8</span>},
        <span class="hljs-number">2</span>: {<span class="hljs-string">&quot;num_key_value_heads&quot;</span>: <span class="hljs-number">4</span>},
    },
)
serialized_per_layer_config = explicit_config.to_dict()[<span class="hljs-string">&quot;per_layer_config&quot;</span>]
serialized_per_layer_config
<span class="hljs-comment"># {</span>
<span class="hljs-comment">#     &quot;0&quot;: {&quot;num_key_value_heads&quot;: 8},</span>
<span class="hljs-comment">#     &quot;1&quot;: {&quot;num_key_value_heads&quot;: 8},</span>
<span class="hljs-comment">#     &quot;2&quot;: {&quot;num_key_value_heads&quot;: 4},</span>
<span class="hljs-comment">#     &quot;3&quot;: {&quot;num_key_value_heads&quot;: 8},</span>
<span class="hljs-comment"># }</span>`,lang:"py",wrap:!1});var j=e(h,4);g(j,{source:"https://github.com/huggingface/transformers/blob/main/docs/source/en/heterogeneous_configurations.md"}),B(2),U(J,s),X()}export{G as component};

Xet Storage Details

Size:: 18.8 kB
Xet hash:: eb39fe4a391a7267fc08a92cd20195db063d38d34a5f27e514efeb7b42bd70e4

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	import"../chunks/DsnmJJEf.js";import{i as T,h as w,C as b,H as l,a,E as g,s as _}from"../chunks/qxQfBDOH.js";import{p as C,o as I,s as e,f as Z,a as U,b as X,c as m,n as B}from"../chunks/Dpn4EqEf.js";const W='{"title":"Per-layer configurations","local":"per-layer-configurations","sections":[{"title":"Define per-layer overrides","local":"define-per-layer-overrides","sections":[],"depth":2},{"title":"Global attribute access","local":"global-attribute-access","sections":[],"depth":2},{"title":"Serialization","local":"serialization","sections":[],"depth":2}],"depth":1}';var V=m('<meta name="hf:doc:metadata"/>'),v=m(`<p></p> <!> <!> <p>Some checkpoints are heterogeneous (not layer-uniform). A smaller MLP in one layer, fewer key-value heads in another, or a different
	layout on selected layers means a single global config doesn’t accurately describe the stack.</p> <p>Use <code>per_layer_config</code> on <a href="/docs/transformers/pr_46007/en/main_classes/configuration#transformers.PreTrainedConfig">PreTrainedConfig</a> to record those diffs when you’re authoring or
	inspecting a config, and when model code will consume them. Each entry stores only what differs from the
	global configuration, the rest inherits.</p> <blockquote class="note"><p>Heterogeneous configurations are a power feature. If a heterogeneous layout becomes a common or prominent
	architecture, we will strive to model it explicitly in the architecture implementation rather than rely on <code>per_layer_config</code>. Prefer the explicit architecture when one exists.</p></blockquote> <p>The models below are heterogeneous checkpoints. Layers are not uniform across the stack. They use a dedicated architecture with <code>block_configs</code> and their own <code>model_type</code>, rather than <code>per_layer_config</code> on a standard architecture.</p> <table><thead><tr><th>Model</th><th>Derived from</th></tr></thead><tbody><tr><td><a href="https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1_5" rel="nofollow">nvidia/Llama-3_3-Nemotron-Super-49B-v1_5</a></td><td><a href="https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct" rel="nofollow">meta-llama/Llama-3.3-70B-Instruct</a></td></tr><tr><td><a href="https://huggingface.co/nvidia/Llama-3_1-Nemotron-Ultra-253B-v1" rel="nofollow">nvidia/Llama-3_1-Nemotron-Ultra-253B-v1</a></td><td><a href="https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct" rel="nofollow">meta-llama/Llama-3.1-405B-Instruct</a></td></tr><tr><td><a href="https://huggingface.co/nvidia/gpt-oss-puzzle-88B" rel="nofollow">nvidia/gpt-oss-puzzle-88B</a></td><td><a href="https://huggingface.co/openai/gpt-oss-120b" rel="nofollow">openai/gpt-oss-120b</a></td></tr><tr><td><a href="https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16" rel="nofollow">nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16</a></td><td><a href="https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16" rel="nofollow">nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16</a></td></tr></tbody></table> <!> <p>Pass <code>per_layer_config</code> to <a href="/docs/transformers/pr_46007/en/model_doc/llama2#transformers.LlamaConfig">LlamaConfig</a> as a mapping from layer indices to attribute overrides. Layer
	indices are zero-based. Only attributes that differ from the global configuration need to be specified.</p> <p><code>per_layer_config</code> records and resolves configuration values. It does not by itself change the modules a model creates
	or how those modules run. Applying a size override or a <code>skip</code> requires model code that reads the resolved per-layer
	configuration when constructing or running each layer. The current <code>LlamaModel</code> constructs every <code>LlamaDecoderLayer</code> with the global configuration, so the <code>skip</code> entries below remain configuration values rather than removing modules
	after <code>from_pretrained</code>.</p> <p>The following example records overrides for four layers: layer 5 uses a smaller MLP, layer 11 uses
	fewer key-value heads, and layers 23 and 27 record <code>skip</code> values for architectures that support them.</p> <!> <p>The submodules that an architecture can skip (for example, <code>"mlp"</code> and <code>"attention"</code>) are defined per architecture. <code>skip</code> accepts a list, so a layer can record more than one submodule override.</p> <p>Accessing <code>config.per_layer_config[layer_idx]</code> returns a resolved layer configuration. The resolved configuration
	combines the global configuration with the overrides for that layer.</p> <!> <p>Configurations that use <code>per_layer_config</code> support the same <a href="/docs/transformers/pr_46007/en/main_classes/configuration#transformers.PreTrainedConfig.save_pretrained">save_pretrained()</a> and <a href="/docs/transformers/pr_46007/en/main_classes/configuration#transformers.PreTrainedConfig.from_pretrained">from_pretrained()</a> round trip as other configurations.</p> <p>Each architecture defines in its code which attributes it consumes at the layer level. <code>per_layer_config</code> provides the
	mechanism for recording those layer-level differences and resolving them against the global config.</p> <!> <p>An attribute with per-layer overrides does not have a single model-wide value. <code>num_key_value_heads</code> may be <code>8</code> on
	most layers and <code>4</code> on selected layers. Reading <code>config.num_key_value_heads</code> outside a layer context is ambiguous.</p> <p>By default that access raises <code>AmbiguousGlobalPerLayerAttributeError</code> and points you to <code>config.per_layer_config[layer_idx]</code>. The attribute still exists on the global config, so this is not an <code>AttributeError</code>. Reading that global value without a layer index is still wrong. Code that builds a key-value cache from a global <code>num_key_value_heads</code> would size the wrong layers incorrectly.</p> <p>Set <code>allow_global_per_layer_attribute_access=True</code> only when you intentionally need the global fallback and can handle
	heterogeneous configs. Global access is then allowed. A warning is emitted once.</p> <!> <!> <p><code>per_layer_config</code> serializes sparsely by default, and layers without overrides are omitted. Overridden attributes that
	match the global value are omitted too.</p> <!> <p>Set <code>serialize_explicit_per_layer_config=True</code> to include every layer for the attributes represented in <code>per_layer_config</code>. That makes the layer layout easier to inspect when some values still match the global
	configuration.</p> <!> <p>Use sparse serialization for compact configs, and explicit serialization when you need the full per-layer layout for
	readability or tooling.</p> <!> <p></p>`,1);function G(J,f){C(f,!1),I(()=>{new URLSearchParams(window.location.search).get("fw")}),T();var s=v();w("pkalzc",d=>{var u=V();_(u,"content",W),U(d,u)});var n=e(Z(s),2);b(n,{containerStyle:"float: right; margin-left: 10px; display: inline-flex; position: relative; z-index: 10;"});var o=e(n,2);l(o,{title:"Per-layer configurations",local:"per-layer-configurations",headingTag:"h1"});var t=e(o,12);l(t,{title:"Define per-layer overrides",local:"define-per-layer-overrides",headingTag:"h2"});var r=e(t,8);a(r,{code:"ZnJvbSUyMHRyYW5zZm9ybWVycyUyMGltcG9ydCUyMExsYW1hQ29uZmlnJTBBJTBBJTBBY29uZmlnJTIwJTNEJTIwTGxhbWFDb25maWcoJTBBJTIwJTIwJTIwJTIwaGlkZGVuX3NpemUlM0Q0MDk2JTJDJTBBJTIwJTIwJTIwJTIwaW50ZXJtZWRpYXRlX3NpemUlM0QxNDMzNiUyQyUwQSUyMCUyMCUyMCUyMG51bV9oaWRkZW5fbGF5ZXJzJTNEMzIlMkMlMEElMjAlMjAlMjAlMjBudW1fYXR0ZW50aW9uX2hlYWRzJTNEMzIlMkMlMEElMjAlMjAlMjAlMjBudW1fa2V5X3ZhbHVlX2hlYWRzJTNEOCUyQyUwQSUyMCUyMCUyMCUyMHBlcl9sYXllcl9jb25maWclM0QlN0IlMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjMlMjBVc2UlMjBhJTIwc21hbGxlciUyME1MUCUyMGluJTIwb25lJTIwbGF5ZXIuJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwNSUzQSUyMCU3QiUyMmludGVybWVkaWF0ZV9zaXplJTIyJTNBJTIwODE5MiU3RCUyQyUwQSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMyUyMFVzZSUyMGZld2VyJTIwa2V5LXZhbHVlJTIwaGVhZHMlMjBpbiUyMGFub3RoZXIlMjBsYXllci4lMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAxMSUzQSUyMCU3QiUyMm51bV9rZXlfdmFsdWVfaGVhZHMlMjIlM0ElMjA0JTdEJTJDJTBBJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIzJTIwUmVjb3JkJTIwYSUyMHJlcXVlc3QlMjB0byUyMHNraXAlMjB0aGUlMjBNTFAlMjBpbiUyMGFyY2hpdGVjdHVyZXMlMjB0aGF0JTIwc3VwcG9ydCUyMGl0LiUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMDIzJTNBJTIwJTdCJTIyc2tpcCUyMiUzQSUyMCU1QiUyMm1scCUyMiU1RCU3RCUyQyUwQSUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMyUyMFJlY29yZCUyMGElMjByZXF1ZXN0JTIwdG8lMjBza2lwJTIwYXR0ZW50aW9uJTIwaW4lMjBhcmNoaXRlY3R1cmVzJTIwdGhhdCUyMHN1cHBvcnQlMjBpdC4lMEElMjAlMjAlMjAlMjAlMjAlMjAlMjAlMjAyNyUzQSUyMCU3QiUyMnNraXAlMjIlM0ElMjAlNUIlMjJhdHRlbnRpb24lMjIlNUQlN0QlMkMlMEElMjAlMjAlMjAlMjAlN0QlMkMlMEEp",highlighted:`<span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> LlamaConfig


	config = LlamaConfig(
	hidden_size=<span class="hljs-number">4096</span>,
	intermediate_size=<span class="hljs-number">14336</span>,
	num_hidden_layers=<span class="hljs-number">32</span>,
	num_attention_heads=<span class="hljs-number">32</span>,
	num_key_value_heads=<span class="hljs-number">8</span>,
	per_layer_config={
	<span class="hljs-comment"># Use a smaller MLP in one layer.</span>
	<span class="hljs-number">5</span>: {<span class="hljs-string">"intermediate_size"</span>: <span class="hljs-number">8192</span>},

	<span class="hljs-comment"># Use fewer key-value heads in another layer.</span>
	<span class="hljs-number">11</span>: {<span class="hljs-string">"num_key_value_heads"</span>: <span class="hljs-number">4</span>},

	<span class="hljs-comment"># Record a request to skip the MLP in architectures that support it.</span>
	<span class="hljs-number">23</span>: {<span class="hljs-string">"skip"</span>: [<span class="hljs-string">"mlp"</span>]},

	<span class="hljs-comment"># Record a request to skip attention in architectures that support it.</span>
	<span class="hljs-number">27</span>: {<span class="hljs-string">"skip"</span>: [<span class="hljs-string">"attention"</span>]},
	},
	)`,lang:"py",wrap:!1});var i=e(r,6);a(i,{code:"JTIzJTIwTGF5ZXIlMjAwJTIwZG9lcyUyMG5vdCUyMGRlZmluZSUyMG92ZXJyaWRlcyUyQyUyMHNvJTIwaXQlMjBpbmhlcml0cyUyMHRoZSUyMGdsb2JhbCUyMHZhbHVlcy4lMEFjb25maWcucGVyX2xheWVyX2NvbmZpZyU1QjAlNUQuaW50ZXJtZWRpYXRlX3NpemUlMEElMjMlMjAxNDMzNiUwQSUwQWNvbmZpZy5wZXJfbGF5ZXJfY29uZmlnJTVCMCU1RC5udW1fa2V5X3ZhbHVlX2hlYWRzJTBBJTIzJTIwOCUwQSUwQSUyMyUyMExheWVyJTIwNSUyMG92ZXJyaWRlcyUyMHRoZSUyME1MUCUyMGludGVybWVkaWF0ZSUyMHNpemUuJTBBY29uZmlnLnBlcl9sYXllcl9jb25maWclNUI1JTVELmludGVybWVkaWF0ZV9zaXplJTBBJTIzJTIwODE5MiUwQSUwQSUyMyUyMExheWVyJTIwMTElMjBvdmVycmlkZXMlMjB0aGUlMjBudW1iZXIlMjBvZiUyMGtleS12YWx1ZSUyMGhlYWRzLiUwQWNvbmZpZy5wZXJfbGF5ZXJfY29uZmlnJTVCMTElNUQubnVtX2tleV92YWx1ZV9oZWFkcyUwQSUyMyUyMDQlMEElMEElMjMlMjBMYXllciUyMDIzJTIwcmVjb3JkcyUyMGFuJTIwTUxQJTIwc2tpcC4lMEFjb25maWcucGVyX2xheWVyX2NvbmZpZyU1QjIzJTVELnNraXAlMEElMjMlMjAlNUIlMjJtbHAlMjIlNUQlMEElMEElMjMlMjBMYXllciUyMDI3JTIwcmVjb3JkcyUyMGFuJTIwYXR0ZW50aW9uJTIwc2tpcC4lMEFjb25maWcucGVyX2xheWVyX2NvbmZpZyU1QjI3JTVELnNraXAlMEElMjMlMjAlNUIlMjJhdHRlbnRpb24lMjIlNUQ=",highlighted:`<span class="hljs-comment"># Layer 0 does not define overrides, so it inherits the global values.</span>
	config.per_layer_config[<span class="hljs-number">0</span>].intermediate_size
	<span class="hljs-comment"># 14336</span>

	config.per_layer_config[<span class="hljs-number">0</span>].num_key_value_heads
	<span class="hljs-comment"># 8</span>

	<span class="hljs-comment"># Layer 5 overrides the MLP intermediate size.</span>
	config.per_layer_config[<span class="hljs-number">5</span>].intermediate_size
	<span class="hljs-comment"># 8192</span>

	<span class="hljs-comment"># Layer 11 overrides the number of key-value heads.</span>
	config.per_layer_config[<span class="hljs-number">11</span>].num_key_value_heads
	<span class="hljs-comment"># 4</span>

	<span class="hljs-comment"># Layer 23 records an MLP skip.</span>
	config.per_layer_config[<span class="hljs-number">23</span>].skip
	<span class="hljs-comment"># ["mlp"]</span>

	<span class="hljs-comment"># Layer 27 records an attention skip.</span>
	config.per_layer_config[<span class="hljs-number">27</span>].skip
	<span class="hljs-comment"># ["attention"]</span>`,lang:"py",wrap:!1});var c=e(i,6);l(c,{title:"Global attribute access",local:"global-attribute-access",headingTag:"h2"});var y=e(c,8);a(y,{code:"Y29uZmlnJTIwJTNEJTIwTGxhbWFDb25maWcoJTBBJTIwJTIwJTIwJTIwaGlkZGVuX3NpemUlM0Q0MDk2JTJDJTBBJTIwJTIwJTIwJTIwaW50ZXJtZWRpYXRlX3NpemUlM0QxNDMzNiUyQyUwQSUyMCUyMCUyMCUyMG51bV9oaWRkZW5fbGF5ZXJzJTNEMzIlMkMlMEElMjAlMjAlMjAlMjBudW1fYXR0ZW50aW9uX2hlYWRzJTNEMzIlMkMlMEElMjAlMjAlMjAlMjBudW1fa2V5X3ZhbHVlX2hlYWRzJTNEOCUyQyUwQSUyMCUyMCUyMCUyMGFsbG93X2dsb2JhbF9wZXJfbGF5ZXJfYXR0cmlidXRlX2FjY2VzcyUzRFRydWUlMkMlMEElMjAlMjAlMjAlMjBwZXJfbGF5ZXJfY29uZmlnJTNEJTdCJTBBJTIwJTIwJTIwJTIwJTIwJTIwJTIwJTIwMTElM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwNCU3RCUyQyUwQSUyMCUyMCUyMCUyMCU3RCUyQyUwQSklMEElMEFjb25maWcubnVtX2tleV92YWx1ZV9oZWFkcyUwQSUyMyUyMDglMEElMjMlMjBFbWl0cyUyMGElMjBvbmUtdGltZSUyMHdhcm5pbmclMjBiZWNhdXNlJTIwbnVtX2tleV92YWx1ZV9oZWFkcyUyMGhhcyUyMGElMjBwZXItbGF5ZXIlMjBvdmVycmlkZS4=",highlighted:`config = LlamaConfig(
	hidden_size=<span class="hljs-number">4096</span>,
	intermediate_size=<span class="hljs-number">14336</span>,
	num_hidden_layers=<span class="hljs-number">32</span>,
	num_attention_heads=<span class="hljs-number">32</span>,
	num_key_value_heads=<span class="hljs-number">8</span>,
	allow_global_per_layer_attribute_access=<span class="hljs-literal">True</span>,
	per_layer_config={
	<span class="hljs-number">11</span>: {<span class="hljs-string">"num_key_value_heads"</span>: <span class="hljs-number">4</span>},
	},
	)

	config.num_key_value_heads
	<span class="hljs-comment"># 8</span>
	<span class="hljs-comment"># Emits a one-time warning because num_key_value_heads has a per-layer override.</span>`,lang:"py",wrap:!1});var M=e(y,2);l(M,{title:"Serialization",local:"serialization",headingTag:"h2"});var p=e(M,4);a(p,{code:"ZnJvbSUyMHRyYW5zZm9ybWVycyUyMGltcG9ydCUyMExsYW1hQ29uZmlnJTBBJTBBJTBBY29uZmlnJTIwJTNEJTIwTGxhbWFDb25maWcoJTBBJTIwJTIwJTIwJTIwaGlkZGVuX3NpemUlM0Q0MDk2JTJDJTBBJTIwJTIwJTIwJTIwaW50ZXJtZWRpYXRlX3NpemUlM0QxNDMzNiUyQyUwQSUyMCUyMCUyMCUyMG51bV9oaWRkZW5fbGF5ZXJzJTNENCUyQyUwQSUyMCUyMCUyMCUyMG51bV9hdHRlbnRpb25faGVhZHMlM0QzMiUyQyUwQSUyMCUyMCUyMCUyMG51bV9rZXlfdmFsdWVfaGVhZHMlM0Q4JTJDJTBBJTIwJTIwJTIwJTIwcGVyX2xheWVyX2NvbmZpZyUzRCU3QiUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMDAlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwOCU3RCUyQyUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMDIlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwNCU3RCUyQyUwQSUyMCUyMCUyMCUyMCU3RCUyQyUwQSklMEElMEFjb25maWcudG9fZGljdCgpJTVCJTIycGVyX2xheWVyX2NvbmZpZyUyMiU1RCUwQSUyMyUyMCU3QiUyMjIlMjIlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwNCU3RCU3RA==",highlighted:`<span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> LlamaConfig


	config = LlamaConfig(
	hidden_size=<span class="hljs-number">4096</span>,
	intermediate_size=<span class="hljs-number">14336</span>,
	num_hidden_layers=<span class="hljs-number">4</span>,
	num_attention_heads=<span class="hljs-number">32</span>,
	num_key_value_heads=<span class="hljs-number">8</span>,
	per_layer_config={
	<span class="hljs-number">0</span>: {<span class="hljs-string">"num_key_value_heads"</span>: <span class="hljs-number">8</span>},
	<span class="hljs-number">2</span>: {<span class="hljs-string">"num_key_value_heads"</span>: <span class="hljs-number">4</span>},
	},
	)

	config.to_dict()[<span class="hljs-string">"per_layer_config"</span>]
	<span class="hljs-comment"># {"2": {"num_key_value_heads": 4}}</span>`,lang:"py",wrap:!1});var h=e(p,4);a(h,{code:"ZXhwbGljaXRfY29uZmlnJTIwJTNEJTIwTGxhbWFDb25maWcoJTBBJTIwJTIwJTIwJTIwaGlkZGVuX3NpemUlM0Q0MDk2JTJDJTBBJTIwJTIwJTIwJTIwaW50ZXJtZWRpYXRlX3NpemUlM0QxNDMzNiUyQyUwQSUyMCUyMCUyMCUyMG51bV9oaWRkZW5fbGF5ZXJzJTNENCUyQyUwQSUyMCUyMCUyMCUyMG51bV9hdHRlbnRpb25faGVhZHMlM0QzMiUyQyUwQSUyMCUyMCUyMCUyMG51bV9rZXlfdmFsdWVfaGVhZHMlM0Q4JTJDJTBBJTIwJTIwJTIwJTIwc2VyaWFsaXplX2V4cGxpY2l0X3Blcl9sYXllcl9jb25maWclM0RUcnVlJTJDJTBBJTIwJTIwJTIwJTIwcGVyX2xheWVyX2NvbmZpZyUzRCU3QiUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMDAlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwOCU3RCUyQyUwQSUyMCUyMCUyMCUyMCUyMCUyMCUyMCUyMDIlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwNCU3RCUyQyUwQSUyMCUyMCUyMCUyMCU3RCUyQyUwQSklMEElMEFzZXJpYWxpemVkX3Blcl9sYXllcl9jb25maWclMjAlM0QlMjBleHBsaWNpdF9jb25maWcudG9fZGljdCgpJTVCJTIycGVyX2xheWVyX2NvbmZpZyUyMiU1RCUwQSUwQXNlcmlhbGl6ZWRfcGVyX2xheWVyX2NvbmZpZyUwQSUyMyUyMCU3QiUwQSUyMyUyMCUyMCUyMCUyMCUyMCUyMjAlMjIlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwOCU3RCUyQyUwQSUyMyUyMCUyMCUyMCUyMCUyMCUyMjElMjIlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwOCU3RCUyQyUwQSUyMyUyMCUyMCUyMCUyMCUyMCUyMjIlMjIlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwNCU3RCUyQyUwQSUyMyUyMCUyMCUyMCUyMCUyMCUyMjMlMjIlM0ElMjAlN0IlMjJudW1fa2V5X3ZhbHVlX2hlYWRzJTIyJTNBJTIwOCU3RCUyQyUwQSUyMyUyMCU3RA==",highlighted:`explicit_config = LlamaConfig(
	hidden_size=<span class="hljs-number">4096</span>,
	intermediate_size=<span class="hljs-number">14336</span>,
	num_hidden_layers=<span class="hljs-number">4</span>,
	num_attention_heads=<span class="hljs-number">32</span>,
	num_key_value_heads=<span class="hljs-number">8</span>,
	serialize_explicit_per_layer_config=<span class="hljs-literal">True</span>,
	per_layer_config={
	<span class="hljs-number">0</span>: {<span class="hljs-string">"num_key_value_heads"</span>: <span class="hljs-number">8</span>},
	<span class="hljs-number">2</span>: {<span class="hljs-string">"num_key_value_heads"</span>: <span class="hljs-number">4</span>},
	},
	)

	serialized_per_layer_config = explicit_config.to_dict()[<span class="hljs-string">"per_layer_config"</span>]

	serialized_per_layer_config
	<span class="hljs-comment"># {</span>
	<span class="hljs-comment"># "0": {"num_key_value_heads": 8},</span>
	<span class="hljs-comment"># "1": {"num_key_value_heads": 8},</span>
	<span class="hljs-comment"># "2": {"num_key_value_heads": 4},</span>
	<span class="hljs-comment"># "3": {"num_key_value_heads": 8},</span>
	<span class="hljs-comment"># }</span>`,lang:"py",wrap:!1});var j=e(h,4);g(j,{source:"https://github.com/huggingface/transformers/blob/main/docs/source/en/heterogeneous_configurations.md"}),B(2),U(J,s),X()}export{G as component};