Hugging Face's logo

EasyDeL
/
gemma-4-E4B

Image-to-Text
JAX
easydel
gemma4
ImageTextToText
vanilla
Model card Files
xet
 Community
erfanzar commited on
Commit
d9c20a0
·
verified ·
1 Parent(s): 3fcfa1e

Add EasyDeL checkpoint for Gemma-4-E4B

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. .gitattributes +0 -0
  2. README.md +159 -0
  3. checkpoint_metadata.json +6 -0
  4. config.json +510 -0
  5. generation_config.json +72 -0
  6. model/model/embed_vision/embedding_projection/kernel/.zarray +1 -0
  7. model/model/embed_vision/embedding_projection/kernel/0.0 +3 -0
  8. model/model/language_model/embed_tokens/embedding/.zarray +1 -0
  9. model/model/language_model/embed_tokens/embedding/0.0 +3 -0
  10. model/model/language_model/embed_tokens/embedding/1.0 +3 -0
  11. model/model/language_model/embed_tokens/embedding/2.0 +3 -0
  12. model/model/language_model/embed_tokens/embedding/3.0 +3 -0
  13. model/model/language_model/embed_tokens_per_layer/embedding/.zarray +1 -0
  14. model/model/language_model/embed_tokens_per_layer/embedding/0.0 +3 -0
  15. model/model/language_model/embed_tokens_per_layer/embedding/1.0 +3 -0
  16. model/model/language_model/embed_tokens_per_layer/embedding/2.0 +3 -0
  17. model/model/language_model/embed_tokens_per_layer/embedding/3.0 +3 -0
  18. model/model/language_model/layers/0/input_layernorm/kernel/.zarray +1 -0
  19. model/model/language_model/layers/0/input_layernorm/kernel/0 +0 -0
  20. model/model/language_model/layers/0/layer_scalar/.zarray +1 -0
  21. model/model/language_model/layers/0/layer_scalar/0 +0 -0
  22. model/model/language_model/layers/0/mlp/down_proj/kernel/.zarray +1 -0
  23. model/model/language_model/layers/0/mlp/down_proj/kernel/0.0 +3 -0
  24. model/model/language_model/layers/0/mlp/down_proj/kernel/0.1 +3 -0
  25. model/model/language_model/layers/0/mlp/down_proj/kernel/0.2 +3 -0
  26. model/model/language_model/layers/0/mlp/down_proj/kernel/0.3 +3 -0
  27. model/model/language_model/layers/0/mlp/gate_proj/kernel/.zarray +1 -0
  28. model/model/language_model/layers/0/mlp/gate_proj/kernel/0.0 +3 -0
  29. model/model/language_model/layers/0/mlp/gate_proj/kernel/1.0 +3 -0
  30. model/model/language_model/layers/0/mlp/gate_proj/kernel/2.0 +3 -0
  31. model/model/language_model/layers/0/mlp/gate_proj/kernel/3.0 +3 -0
  32. model/model/language_model/layers/0/mlp/up_proj/kernel/.zarray +1 -0
  33. model/model/language_model/layers/0/mlp/up_proj/kernel/0.0 +3 -0
  34. model/model/language_model/layers/0/mlp/up_proj/kernel/1.0 +3 -0
  35. model/model/language_model/layers/0/mlp/up_proj/kernel/2.0 +3 -0
  36. model/model/language_model/layers/0/mlp/up_proj/kernel/3.0 +3 -0
  37. model/model/language_model/layers/0/per_layer_input_gate/kernel/.zarray +1 -0
  38. model/model/language_model/layers/0/per_layer_input_gate/kernel/0.0 +3 -0
  39. model/model/language_model/layers/0/per_layer_projection/kernel/.zarray +1 -0
  40. model/model/language_model/layers/0/per_layer_projection/kernel/0.0 +3 -0
  41. model/model/language_model/layers/0/post_attention_layernorm/kernel/.zarray +1 -0
  42. model/model/language_model/layers/0/post_attention_layernorm/kernel/0 +0 -0
  43. model/model/language_model/layers/0/post_feedforward_layernorm/kernel/.zarray +1 -0
  44. model/model/language_model/layers/0/post_feedforward_layernorm/kernel/0 +0 -0
  45. model/model/language_model/layers/0/post_per_layer_input_norm/kernel/.zarray +1 -0
  46. model/model/language_model/layers/0/post_per_layer_input_norm/kernel/0 +0 -0
  47. model/model/language_model/layers/0/pre_feedforward_layernorm/kernel/.zarray +1 -0
  48. model/model/language_model/layers/0/pre_feedforward_layernorm/kernel/0 +0 -0
  49. model/model/language_model/layers/0/self_attn/k_norm/kernel/.zarray +1 -0
  50. model/model/language_model/layers/0/self_attn/k_norm/kernel/0 +0 -0
.gitattributes CHANGED
The diff for this file is too large to render. See raw diff
 
README.md ADDED
@@ -0,0 +1,159 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: easydel
3
+ pipeline_tag: image-to-text
4
+ tags:
5
+ - easydel
6
+ - jax
7
+ - "gemma4"
8
+ - "ImageTextToText"
9
+ - "vanilla"
10
+ ---
11
+
12
+ <p align="center">
13
+ <img alt="EasyDeL" src="https://raw.githubusercontent.com/erfanzar/easydel/main/images/easydel-logo-with-text.png" height="80">
14
+ </p>
15
+
16
+ <h1 align="center">google/gemma-4-E4B</h1>
17
+
18
+ <div align="center">
19
+ EasyDeL checkpoint converted from google/gemma-4-E4B.
20
+ </div>
21
+
22
+ ## Overview
23
+
24
+ This checkpoint is intended to be loaded with EasyDeL on JAX (CPU/GPU/TPU). It supports sharded loading with `auto_shard_model=True` and configurable precision via `dtype`, `param_dtype`, and `precision`.
25
+
26
+ ## Quickstart
27
+
28
+ ```python
29
+ import easydel as ed
30
+ from jax import numpy as jnp, lax
31
+
32
+ repo_id = "/dev/shm/conv/gemma-4-E4B"
33
+
34
+ dtype = jnp.bfloat16 # try jnp.float16 on many GPUs
35
+
36
+ model = ed.AutoEasyDeLModelForImageTextToText.from_pretrained(
37
+ repo_id,
38
+ dtype=dtype,
39
+ param_dtype=dtype,
40
+ precision=lax.Precision("fastest"),
41
+ sharding_axis_names=("dp", "fsdp", "ep", "tp", "sp"),
42
+ sharding_axis_dims=(1, -1, 1, 1, 1),
43
+ config_kwargs=ed.EasyDeLBaseConfigDict(
44
+ attn_dtype=dtype,
45
+ attn_mechanism=ed.AttentionMechanisms.VANILLA,
46
+ fsdp_is_ep_bound=True,
47
+ sp_is_ep_bound=True,
48
+ moe_method=ed.MoEMethods.FUSED_MOE,
49
+ ),
50
+ auto_shard_model=True,
51
+ partition_axis=ed.PartitionAxis(),
52
+ )
53
+ ```
54
+
55
+ If the repository only provides PyTorch weights, pass `from_torch=True` to `from_pretrained(...)`.
56
+
57
+ ## Sharding & Parallelism (Multi-Device)
58
+
59
+ EasyDeL can scale to multiple devices by creating a logical device mesh. Most EasyDeL loaders use a 5D mesh:
60
+
61
+ - `dp`: data parallel (replicated parameters, different batch shards)
62
+ - `fsdp`: parameter sharding (memory saver; often the biggest axis)
63
+ - `ep`: expert parallel (MoE; keep `1` for non-MoE models)
64
+ - `tp`: tensor parallel (splits large matmuls)
65
+ - `sp`: sequence parallel (splits sequence dimension)
66
+
67
+ Use `sharding_axis_names=("dp","fsdp","ep","tp","sp")` and choose `sharding_axis_dims` so that their product equals your device count.
68
+ You can use `-1` in `sharding_axis_dims` to let EasyDeL infer the remaining dimension.
69
+
70
+ <details>
71
+ <summary>Example sharding configs</summary>
72
+
73
+ ```python
74
+ # 8 devices, pure FSDP
75
+ sharding_axis_dims = (1, 8, 1, 1, 1)
76
+
77
+ # 8 devices, 2-way DP x 4-way FSDP
78
+ sharding_axis_dims = (2, 4, 1, 1, 1)
79
+
80
+ # 8 devices, 4-way FSDP x 2-way TP
81
+ sharding_axis_dims = (1, 4, 1, 2, 1)
82
+ ```
83
+ </details>
84
+
85
+ ## Using via `eLargeModel` (ELM)
86
+
87
+ `eLargeModel` is a higher-level interface that wires together loading, sharding, training, and eSurge inference from a single config.
88
+
89
+ ```python
90
+ from easydel import eLargeModel
91
+
92
+ repo_id = "/dev/shm/conv/gemma-4-E4B"
93
+
94
+ elm = eLargeModel.from_pretrained(repo_id) # task is auto-detected
95
+ elm.set_dtype("bf16")
96
+ elm.set_sharding(axis_names=("dp", "fsdp", "ep", "tp", "sp"), axis_dims=(1, -1, 1, 1, 1))
97
+
98
+ model = elm.build_model()
99
+ # Optional: build an inference engine
100
+ # engine = elm.build_esurge()
101
+ ```
102
+
103
+ <details>
104
+ <summary>ELM YAML config example</summary>
105
+
106
+ ```yaml
107
+ model:
108
+ name_or_path: "/dev/shm/conv/gemma-4-E4B"
109
+
110
+ loader:
111
+ dtype: bf16
112
+ param_dtype: bf16
113
+
114
+ sharding:
115
+ axis_dims: [1, -1, 1, 1, 1]
116
+ auto_shard_model: true
117
+ ```
118
+ </details>
119
+
120
+ ## Features
121
+
122
+ **EasyDeL:**
123
+ - JAX native implementation and sharded execution
124
+ - Configurable attention backends via `AttentionMechanisms.*`
125
+ - Precision control via `dtype`, `param_dtype`, and `precision`
126
+
127
+ ## Installation
128
+
129
+ ```bash
130
+ pip install easydel
131
+ ```
132
+
133
+ ## Links
134
+
135
+ - EasyDeL GitHub: https://github.com/erfanzar/EasyDeL
136
+ - Docs: https://easydel.readthedocs.io/en/latest/
137
+
138
+ ## Supported Tasks
139
+
140
+ - ImageTextToText
141
+
142
+ ## Limitations
143
+
144
+ - Refer to the original model card for training data, evaluation, and intended use.
145
+
146
+ ## License
147
+
148
+ EasyDeL is released under the Apache-2.0 license. The license for this model's weights may differ; please consult the original repository.
149
+
150
+ ## Citation
151
+
152
+ ```bibtex
153
+ @misc{Zare Chavoshi_2023,
154
+ title={EasyDeL: An open-source library for enhancing and streamlining the training process of machine learning models},
155
+ url={https://github.com/erfanzar/EasyDeL},
156
+ author={Zare Chavoshi, Erfan},
157
+ year={2023}
158
+ }
159
+ ```
checkpoint_metadata.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "timestamp": "2026-04-09T03:44:41.226750",
3
+ "custom_metadata": {
4
+ "step": 0
5
+ }
6
+ }
config.json ADDED
@@ -0,0 +1,510 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_external_rope_config_kwargs": {},
3
+ "add_cross_attention": false,
4
+ "architectures": [
5
+ "Gemma4ForConditionalGeneration"
6
+ ],
7
+ "attn_mechanism": "vanilla",
8
+ "audio_config": {
9
+ "_name_or_path": "",
10
+ "architectures": null,
11
+ "attention_chunk_size": 12,
12
+ "attention_context_left": 13,
13
+ "attention_context_right": 0,
14
+ "attention_invalid_logits_value": -1000000000.0,
15
+ "attention_logit_cap": 50.0,
16
+ "chunk_size_feed_forward": 0,
17
+ "conv_kernel_size": 5,
18
+ "dtype": "bfloat16",
19
+ "gradient_clipping": 10000000000.0,
20
+ "hidden_act": "silu",
21
+ "hidden_size": 1024,
22
+ "id2label": {
23
+ "0": "LABEL_0",
24
+ "1": "LABEL_1"
25
+ },
26
+ "initializer_range": 0.02,
27
+ "is_encoder_decoder": false,
28
+ "label2id": {
29
+ "LABEL_0": 0,
30
+ "LABEL_1": 1
31
+ },
32
+ "model_type": "gemma4_audio",
33
+ "num_attention_heads": 8,
34
+ "num_hidden_layers": 12,
35
+ "output_attentions": false,
36
+ "output_hidden_states": false,
37
+ "output_proj_dims": 1536,
38
+ "problem_type": null,
39
+ "residual_weight": 0.5,
40
+ "return_dict": true,
41
+ "rms_norm_eps": 1e-06,
42
+ "subsampling_conv_channels": [
43
+ 128,
44
+ 32
45
+ ],
46
+ "use_clipped_linears": true
47
+ },
48
+ "audio_token_id": 258881,
49
+ "backend": null,
50
+ "bits": null,
51
+ "blocksize_b": 1,
52
+ "blocksize_k": 512,
53
+ "blocksize_q": 512,
54
+ "boa_token_id": 256000,
55
+ "boi_token_id": 255999,
56
+ "bos_token_id": null,
57
+ "cross_attention_hidden_size": null,
58
+ "decode_attn_mechanism": null,
59
+ "decoder_start_token_id": null,
60
+ "dtype": "bfloat16",
61
+ "easy_method": "train",
62
+ "eoa_token_id": 258883,
63
+ "eoa_token_index": 258883,
64
+ "eoi_token_id": 258882,
65
+ "eos_token_id": null,
66
+ "fcm_max_ratio": 0.0,
67
+ "fcm_min_ratio": 0.0,
68
+ "flash_attention_backward_pass_impl": "triton",
69
+ "fsdp_is_ep_bound": true,
70
+ "gradient_checkpointing": "",
71
+ "gradient_checkpointing_targets": null,
72
+ "hardware_abstraction": false,
73
+ "image_token_id": 258880,
74
+ "initializer_range": 0.02,
75
+ "is_decoder": false,
76
+ "kv_cache_quantization_config": null,
77
+ "kv_cache_sharding_sequence_axis_name": "sp",
78
+ "kvdtype": "bfloat16",
79
+ "lmhead_chunksize": null,
80
+ "max_position_embeddings": null,
81
+ "mla_attn_dtype": "bfloat16",
82
+ "mla_attn_mechanism": "auto",
83
+ "mla_attn_softmax_dtype": "float32",
84
+ "model_type": "gemma4",
85
+ "moe_force_xla_gmm": false,
86
+ "moe_method": "fused_moe",
87
+ "moe_tiling_size_batch": 4,
88
+ "moe_tiling_size_dim": 128,
89
+ "moe_tiling_size_seqlen": 128,
90
+ "operation_configs": null,
91
+ "pad_token_id": null,
92
+ "pallas_k_block_size": 128,
93
+ "pallas_m_block_size": 128,
94
+ "pallas_n_block_size": 128,
95
+ "partition_axis": {
96
+ "attention_dim_axis": null,
97
+ "attention_kv_dim_axis": null,
98
+ "batch_axis": [
99
+ "fsdp",
100
+ "dp"
101
+ ],
102
+ "bias_head_sequence_axis": null,
103
+ "bias_key_sequence_axis": null,
104
+ "data_parallel_axis": "dp",
105
+ "decode_attention_dim_axis": null,
106
+ "decode_attention_kv_dim_axis": null,
107
+ "decode_batch_axis": [
108
+ "fsdp",
109
+ "dp"
110
+ ],
111
+ "decode_head_axis": "tp",
112
+ "decode_key_sequence_axis": "sp",
113
+ "decode_kv_head_axis": "tp",
114
+ "decode_query_sequence_axis": null,
115
+ "expert_axis": "ep",
116
+ "expert_gate_axis": null,
117
+ "expert_parallel_axis": "ep",
118
+ "fully_sharded_data_parallel_axis": "fsdp",
119
+ "head_axis": "tp",
120
+ "hidden_state_axis": "tp",
121
+ "key_sequence_axis": "sp",
122
+ "kv_head_axis": "tp",
123
+ "mlp_intermediate_axis": "tp",
124
+ "query_sequence_axis": "sp",
125
+ "sequence_axis": "sp",
126
+ "sequence_parallel_axis": "sp",
127
+ "tensor_parallel_axis": "tp",
128
+ "vocab_axis": "tp"
129
+ },
130
+ "platform": null,
131
+ "precompute_masks": true,
132
+ "pretraining_tp": 1,
133
+ "qmm_platform_override": null,
134
+ "qmm_tpu_path_override": null,
135
+ "quantization_config": null,
136
+ "scan_attention_layers": false,
137
+ "scan_mlp_chunk_size": 1024,
138
+ "scan_ring_attention": true,
139
+ "sep_token_id": null,
140
+ "sequence_axis_name": "sp",
141
+ "sharding_axis_dims": [
142
+ 1,
143
+ -1,
144
+ 1,
145
+ 1,
146
+ 1
147
+ ],
148
+ "sharding_axis_names": [
149
+ "dp",
150
+ "fsdp",
151
+ "ep",
152
+ "tp",
153
+ "sp"
154
+ ],
155
+ "sharding_dcn_axis_dims": null,
156
+ "sp_is_ep_bound": true,
157
+ "text_config": {
158
+ "_external_rope_config_kwargs": {},
159
+ "add_cross_attention": false,
160
+ "attention_bias": false,
161
+ "attention_dropout": 0.0,
162
+ "attention_k_eq_v": false,
163
+ "attn_dtype": "bfloat16",
164
+ "attn_mechanism": "vanilla",
165
+ "attn_softmax_dtype": "float32",
166
+ "backend": null,
167
+ "bits": null,
168
+ "blocksize_b": 1,
169
+ "blocksize_k": 512,
170
+ "blocksize_q": 512,
171
+ "bos_token_id": 2,
172
+ "cross_attention_hidden_size": null,
173
+ "decode_attn_mechanism": null,
174
+ "decoder_start_token_id": null,
175
+ "dtype": "bfloat16",
176
+ "easy_method": "train",
177
+ "enable_moe_block": false,
178
+ "eos_token_id": 1,
179
+ "expert_intermediate_size": null,
180
+ "fcm_max_ratio": 0.0,
181
+ "fcm_min_ratio": 0.0,
182
+ "final_logit_softcapping": 30.0,
183
+ "flash_attention_backward_pass_impl": "triton",
184
+ "fsdp_is_ep_bound": true,
185
+ "global_head_dim": 512,
186
+ "gradient_checkpointing": "",
187
+ "gradient_checkpointing_targets": null,
188
+ "hardware_abstraction": false,
189
+ "head_dim": 256,
190
+ "hidden_activation": "gelu_pytorch_tanh",
191
+ "hidden_size": 2560,
192
+ "hidden_size_per_layer_input": 256,
193
+ "initializer_range": 0.02,
194
+ "intermediate_size": 10240,
195
+ "is_decoder": false,
196
+ "kv_cache_quantization_config": null,
197
+ "kv_cache_sharding_sequence_axis_name": "sp",
198
+ "kvdtype": "bfloat16",
199
+ "layer_types": [
200
+ "sliding_attention",
201
+ "sliding_attention",
202
+ "sliding_attention",
203
+ "sliding_attention",
204
+ "sliding_attention",
205
+ "full_attention",
206
+ "sliding_attention",
207
+ "sliding_attention",
208
+ "sliding_attention",
209
+ "sliding_attention",
210
+ "sliding_attention",
211
+ "full_attention",
212
+ "sliding_attention",
213
+ "sliding_attention",
214
+ "sliding_attention",
215
+ "sliding_attention",
216
+ "sliding_attention",
217
+ "full_attention",
218
+ "sliding_attention",
219
+ "sliding_attention",
220
+ "sliding_attention",
221
+ "sliding_attention",
222
+ "sliding_attention",
223
+ "full_attention",
224
+ "sliding_attention",
225
+ "sliding_attention",
226
+ "sliding_attention",
227
+ "sliding_attention",
228
+ "sliding_attention",
229
+ "full_attention",
230
+ "sliding_attention",
231
+ "sliding_attention",
232
+ "sliding_attention",
233
+ "sliding_attention",
234
+ "sliding_attention",
235
+ "full_attention",
236
+ "sliding_attention",
237
+ "sliding_attention",
238
+ "sliding_attention",
239
+ "sliding_attention",
240
+ "sliding_attention",
241
+ "full_attention"
242
+ ],
243
+ "lmhead_chunksize": null,
244
+ "max_position_embeddings": 131072,
245
+ "mla_attn_dtype": "bfloat16",
246
+ "mla_attn_mechanism": "auto",
247
+ "mla_attn_softmax_dtype": "float32",
248
+ "model_type": "gemma4_text",
249
+ "moe_force_xla_gmm": false,
250
+ "moe_intermediate_size": null,
251
+ "moe_method": "fused_moe",
252
+ "moe_tiling_size_batch": 4,
253
+ "moe_tiling_size_dim": 128,
254
+ "moe_tiling_size_seqlen": 128,
255
+ "num_attention_heads": 8,
256
+ "num_experts": null,
257
+ "num_global_key_value_heads": null,
258
+ "num_hidden_layers": 42,
259
+ "num_key_value_heads": 2,
260
+ "num_kv_shared_layers": 18,
261
+ "num_local_experts": null,
262
+ "operation_configs": null,
263
+ "pad_token_id": 0,
264
+ "pallas_k_block_size": 128,
265
+ "pallas_m_block_size": 128,
266
+ "pallas_n_block_size": 128,
267
+ "partition_axis": {
268
+ "attention_dim_axis": null,
269
+ "attention_kv_dim_axis": null,
270
+ "batch_axis": [
271
+ "fsdp",
272
+ "dp"
273
+ ],
274
+ "bias_head_sequence_axis": null,
275
+ "bias_key_sequence_axis": null,
276
+ "data_parallel_axis": "dp",
277
+ "decode_attention_dim_axis": null,
278
+ "decode_attention_kv_dim_axis": null,
279
+ "decode_batch_axis": [
280
+ "fsdp",
281
+ "dp"
282
+ ],
283
+ "decode_head_axis": "tp",
284
+ "decode_key_sequence_axis": "sp",
285
+ "decode_kv_head_axis": "tp",
286
+ "decode_query_sequence_axis": null,
287
+ "expert_axis": "ep",
288
+ "expert_gate_axis": null,
289
+ "expert_parallel_axis": "ep",
290
+ "fully_sharded_data_parallel_axis": "fsdp",
291
+ "head_axis": "tp",
292
+ "hidden_state_axis": "tp",
293
+ "key_sequence_axis": "sp",
294
+ "kv_head_axis": "tp",
295
+ "mlp_intermediate_axis": "tp",
296
+ "query_sequence_axis": "sp",
297
+ "sequence_axis": "sp",
298
+ "sequence_parallel_axis": "sp",
299
+ "tensor_parallel_axis": "tp",
300
+ "vocab_axis": "tp"
301
+ },
302
+ "platform": null,
303
+ "precompute_masks": true,
304
+ "pretraining_tp": 1,
305
+ "qmm_platform_override": null,
306
+ "qmm_tpu_path_override": null,
307
+ "quantization_config": null,
308
+ "rms_norm_eps": 1e-06,
309
+ "rope_parameters": {
310
+ "full_attention": {
311
+ "partial_rotary_factor": 0.25,
312
+ "rope_theta": 1000000.0,
313
+ "rope_type": "proportional",
314
+ "type": "proportional"
315
+ },
316
+ "sliding_attention": {
317
+ "rope_theta": 10000.0,
318
+ "rope_type": "default",
319
+ "type": "default"
320
+ }
321
+ },
322
+ "scan_attention_layers": false,
323
+ "scan_layers": false,
324
+ "scan_mlp_chunk_size": 1024,
325
+ "scan_ring_attention": true,
326
+ "sep_token_id": null,
327
+ "sequence_axis_name": "sp",
328
+ "sharding_axis_dims": [
329
+ 1,
330
+ -1,
331
+ 1,
332
+ 1,
333
+ 1
334
+ ],
335
+ "sharding_axis_names": [
336
+ "dp",
337
+ "fsdp",
338
+ "ep",
339
+ "tp",
340
+ "sp"
341
+ ],
342
+ "sharding_dcn_axis_dims": null,
343
+ "sliding_window": 512,
344
+ "sp_is_ep_bound": true,
345
+ "tie_encoder_decoder": false,
346
+ "tie_word_embeddings": true,
347
+ "top_k_experts": null,
348
+ "use_bidirectional_attention": null,
349
+ "use_cache": true,
350
+ "use_double_wide_mlp": false,
351
+ "use_expert_tensor_mode": false,
352
+ "use_qmm_best_config": false,
353
+ "use_ring_of_experts": false,
354
+ "use_scan_mlp": false,
355
+ "use_sharded_kv_caching": false,
356
+ "use_sharding_constraint": false,
357
+ "vocab_size": 262144,
358
+ "vocab_size_per_layer_input": 262144
359
+ },
360
+ "tie_encoder_decoder": false,
361
+ "tie_word_embeddings": true,
362
+ "transformers_version": "5.5.0",
363
+ "use_expert_tensor_mode": false,
364
+ "use_qmm_best_config": false,
365
+ "use_ring_of_experts": false,
366
+ "use_scan_mlp": false,
367
+ "use_sharded_kv_caching": false,
368
+ "use_sharding_constraint": false,
369
+ "video_token_id": 258884,
370
+ "vision_config": {
371
+ "_external_rope_config_kwargs": {},
372
+ "add_cross_attention": false,
373
+ "attention_bias": false,
374
+ "attention_dropout": 0.0,
375
+ "attn_dtype": "bfloat16",
376
+ "attn_mechanism": "vanilla",
377
+ "attn_softmax_dtype": "float32",
378
+ "backend": null,
379
+ "bits": null,
380
+ "blocksize_b": 1,
381
+ "blocksize_k": 512,
382
+ "blocksize_q": 512,
383
+ "bos_token_id": null,
384
+ "cross_attention_hidden_size": null,
385
+ "decode_attn_mechanism": null,
386
+ "decoder_start_token_id": null,
387
+ "default_output_length": 280,
388
+ "dtype": "bfloat16",
389
+ "easy_method": "train",
390
+ "eos_token_id": null,
391
+ "fcm_max_ratio": 0.0,
392
+ "fcm_min_ratio": 0.0,
393
+ "flash_attention_backward_pass_impl": "triton",
394
+ "fsdp_is_ep_bound": true,
395
+ "global_head_dim": 64,
396
+ "gradient_checkpointing": "",
397
+ "gradient_checkpointing_targets": null,
398
+ "hardware_abstraction": false,
399
+ "head_dim": 64,
400
+ "hidden_activation": "gelu_pytorch_tanh",
401
+ "hidden_size": 768,
402
+ "initializer_range": 0.02,
403
+ "intermediate_size": 3072,
404
+ "is_decoder": false,
405
+ "kv_cache_quantization_config": null,
406
+ "kv_cache_sharding_sequence_axis_name": "sp",
407
+ "kvdtype": "bfloat16",
408
+ "lmhead_chunksize": null,
409
+ "max_position_embeddings": 131072,
410
+ "mla_attn_dtype": "bfloat16",
411
+ "mla_attn_mechanism": "auto",
412
+ "mla_attn_softmax_dtype": "float32",
413
+ "model_type": "gemma4_vision",
414
+ "moe_force_xla_gmm": false,
415
+ "moe_method": "fused_moe",
416
+ "moe_tiling_size_batch": 4,
417
+ "moe_tiling_size_dim": 128,
418
+ "moe_tiling_size_seqlen": 128,
419
+ "num_attention_heads": 12,
420
+ "num_hidden_layers": 16,
421
+ "num_key_value_heads": 12,
422
+ "operation_configs": null,
423
+ "pad_token_id": null,
424
+ "pallas_k_block_size": 128,
425
+ "pallas_m_block_size": 128,
426
+ "pallas_n_block_size": 128,
427
+ "partition_axis": {
428
+ "attention_dim_axis": null,
429
+ "attention_kv_dim_axis": null,
430
+ "batch_axis": [
431
+ "fsdp",
432
+ "dp"
433
+ ],
434
+ "bias_head_sequence_axis": null,
435
+ "bias_key_sequence_axis": null,
436
+ "data_parallel_axis": "dp",
437
+ "decode_attention_dim_axis": null,
438
+ "decode_attention_kv_dim_axis": null,
439
+ "decode_batch_axis": [
440
+ "fsdp",
441
+ "dp"
442
+ ],
443
+ "decode_head_axis": "tp",
444
+ "decode_key_sequence_axis": "sp",
445
+ "decode_kv_head_axis": "tp",
446
+ "decode_query_sequence_axis": null,
447
+ "expert_axis": "ep",
448
+ "expert_gate_axis": null,
449
+ "expert_parallel_axis": "ep",
450
+ "fully_sharded_data_parallel_axis": "fsdp",
451
+ "head_axis": "tp",
452
+ "hidden_state_axis": "tp",
453
+ "key_sequence_axis": "sp",
454
+ "kv_head_axis": "tp",
455
+ "mlp_intermediate_axis": "tp",
456
+ "query_sequence_axis": "sp",
457
+ "sequence_axis": "sp",
458
+ "sequence_parallel_axis": "sp",
459
+ "tensor_parallel_axis": "tp",
460
+ "vocab_axis": "tp"
461
+ },
462
+ "patch_size": 16,
463
+ "platform": null,
464
+ "pooling_kernel_size": 3,
465
+ "position_embedding_size": 10240,
466
+ "precompute_masks": true,
467
+ "pretraining_tp": 1,
468
+ "qmm_platform_override": null,
469
+ "qmm_tpu_path_override": null,
470
+ "quantization_config": null,
471
+ "rms_norm_eps": 1e-06,
472
+ "rope_parameters": {
473
+ "rope_theta": 100.0,
474
+ "rope_type": "default",
475
+ "type": "default"
476
+ },
477
+ "scan_attention_layers": false,
478
+ "scan_mlp_chunk_size": 1024,
479
+ "scan_ring_attention": true,
480
+ "sep_token_id": null,
481
+ "sequence_axis_name": "sp",
482
+ "sharding_axis_dims": [
483
+ 1,
484
+ -1,
485
+ 1,
486
+ 1,
487
+ 1
488
+ ],
489
+ "sharding_axis_names": [
490
+ "dp",
491
+ "fsdp",
492
+ "ep",
493
+ "tp",
494
+ "sp"
495
+ ],
496
+ "sharding_dcn_axis_dims": null,
497
+ "sp_is_ep_bound": true,
498
+ "standardize": false,
499
+ "tie_encoder_decoder": false,
500
+ "tie_word_embeddings": true,
501
+ "use_clipped_linears": true,
502
+ "use_expert_tensor_mode": false,
503
+ "use_qmm_best_config": false,
504
+ "use_ring_of_experts": false,
505
+ "use_scan_mlp": false,
506
+ "use_sharded_kv_caching": false,
507
+ "use_sharding_constraint": false
508
+ },
509
+ "vision_soft_tokens_per_image": 280
510
+ }
generation_config.json ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": null,
3
+ "assistant_confidence_threshold": null,
4
+ "assistant_early_exit": null,
5
+ "assistant_lookbehind": null,
6
+ "bad_words_ids": null,
7
+ "begin_suppress_tokens": null,
8
+ "bos_token_id": 2,
9
+ "cache_config": null,
10
+ "cache_implementation": null,
11
+ "compile_config": null,
12
+ "constraints": null,
13
+ "continuous_batching_config": null,
14
+ "decoder_start_token_id": null,
15
+ "disable_compile": null,
16
+ "diversity_penalty": null,
17
+ "do_sample": true,
18
+ "dola_layers": null,
19
+ "early_stopping": null,
20
+ "encoder_no_repeat_ngram_size": null,
21
+ "encoder_repetition_penalty": null,
22
+ "eos_token_id": 1,
23
+ "epsilon_cutoff": null,
24
+ "eta_cutoff": null,
25
+ "exponential_decay_length_penalty": null,
26
+ "force_words_ids": null,
27
+ "forced_bos_token_id": null,
28
+ "forced_eos_token_id": null,
29
+ "guidance_scale": null,
30
+ "is_assistant": null,
31
+ "length_penalty": null,
32
+ "low_memory": null,
33
+ "max_length": null,
34
+ "max_matching_ngram_size": null,
35
+ "max_new_tokens": null,
36
+ "max_time": null,
37
+ "min_length": null,
38
+ "min_new_tokens": null,
39
+ "min_p": null,
40
+ "no_repeat_ngram_size": null,
41
+ "num_assistant_tokens": null,
42
+ "num_assistant_tokens_schedule": null,
43
+ "num_beam_groups": null,
44
+ "num_beams": null,
45
+ "num_return_sequences": null,
46
+ "output_attentions": null,
47
+ "output_hidden_states": null,
48
+ "output_logits": null,
49
+ "output_scores": null,
50
+ "pad_token_id": 0,
51
+ "penalty_alpha": null,
52
+ "prefill_chunk_size": null,
53
+ "prompt_lookup_num_tokens": null,
54
+ "remove_invalid_values": null,
55
+ "renormalize_logits": null,
56
+ "repetition_penalty": null,
57
+ "return_dict_in_generate": null,
58
+ "sequence_bias": null,
59
+ "stop_strings": null,
60
+ "suppress_tokens": null,
61
+ "target_lookbehind": null,
62
+ "temperature": 1.0,
63
+ "token_healing": null,
64
+ "top_h": null,
65
+ "top_k": 64,
66
+ "top_p": 0.95,
67
+ "transformers_version": "5.5.0",
68
+ "trust_remote_code": false,
69
+ "typical_p": null,
70
+ "use_cache": null,
71
+ "watermarking_config": null
72
+ }
model/model/embed_vision/embedding_projection/kernel/.zarray ADDED
@@ -0,0 +1 @@
 
 
1
+ {"chunks":[768,2560],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[768,2560],"zarr_format":2}
model/model/embed_vision/embedding_projection/kernel/0.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:46e582998f9770b4002d56554ed9e2a3b4e133d38959b918a65298a86db905f8
3
+ size 3087949
model/model/language_model/embed_tokens/embedding/.zarray ADDED
@@ -0,0 +1 @@
 
 
1
+ {"chunks":[65536,2560],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[262144,2560],"zarr_format":2}
model/model/language_model/embed_tokens/embedding/0.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b09e748a1ff0e516995af5db82de199035820cbf5bfe7ddc60e8ad0354d0ac6e
3
+ size 261575439
model/model/language_model/embed_tokens/embedding/1.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:43e52561d226de6672fa2769d9ba3311262206f650552dd3153693c03223be76
3
+ size 261620470
model/model/language_model/embed_tokens/embedding/2.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ee08d9bd155fc131cd108ba5df4aeb93171ed94fda3d05ec6c5497ff9a1e0b14
3
+ size 261634046
model/model/language_model/embed_tokens/embedding/3.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3113f6c98a34214a5f3f8c7c0a8866d9b332607ab00c6e6fe68d0786a7d174b5
3
+ size 261637947
model/model/language_model/embed_tokens_per_layer/embedding/.zarray ADDED
@@ -0,0 +1 @@
 
 
1
+ {"chunks":[65536,10752],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[262144,10752],"zarr_format":2}
model/model/language_model/embed_tokens_per_layer/embedding/0.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:03f71c0a0cc7e19fa3ef9af0ab249aef942a96f69fc0207836afdc88d2bb70af
3
+ size 1097885172
model/model/language_model/embed_tokens_per_layer/embedding/1.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0c4dd9dd52bfb6aea69fc6c615cbaba0b887d8927ff0a769485e885db06e2083
3
+ size 1097857879
model/model/language_model/embed_tokens_per_layer/embedding/2.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b0622a6c498031ebf5f147751c27a82278231d98b3196b6f3ac43a8d7950a375
3
+ size 1097690355
model/model/language_model/embed_tokens_per_layer/embedding/3.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9c79321e3b15857deadd43025871a914d502590c8f13bc525cd19259cc6142f9
3
+ size 1097087046
model/model/language_model/layers/0/input_layernorm/kernel/.zarray ADDED
@@ -0,0 +1 @@
 
 
1
+ {"chunks":[2560],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[2560],"zarr_format":2}
model/model/language_model/layers/0/input_layernorm/kernel/0 ADDED
Binary file (3.37 kB). View file
 
model/model/language_model/layers/0/layer_scalar/.zarray ADDED
@@ -0,0 +1 @@
 
 
1
+ {"chunks":[1],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[1],"zarr_format":2}
model/model/language_model/layers/0/layer_scalar/0 ADDED
Binary file (11 Bytes). View file
 
model/model/language_model/layers/0/mlp/down_proj/kernel/.zarray ADDED
@@ -0,0 +1 @@
 
 
1
+ {"chunks":[10240,640],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[10240,2560],"zarr_format":2}
model/model/language_model/layers/0/mlp/down_proj/kernel/0.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:695ed765f35b402617dc1838897ac9c3ad21cbb7f666db6f4df8106573271bf7
3
+ size 10245764
model/model/language_model/layers/0/mlp/down_proj/kernel/0.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7c6a6967fd7ef34ab6c0a8e944097c1be2593178f1d7910ec855683c11652a18
3
+ size 10246484
model/model/language_model/layers/0/mlp/down_proj/kernel/0.2 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:713569ba0fb8f36e5db79be093c5245838367056e9eebb1ece85019cef07f675
3
+ size 10242181
model/model/language_model/layers/0/mlp/down_proj/kernel/0.3 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:30377474b5ca2430442aac516ae2332187ce5b36d4d37c02198b53cd66d4262a
3
+ size 10242850
model/model/language_model/layers/0/mlp/gate_proj/kernel/.zarray ADDED
@@ -0,0 +1 @@
 
 
1
+ {"chunks":[640,10240],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[2560,10240],"zarr_format":2}
model/model/language_model/layers/0/mlp/gate_proj/kernel/0.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f32f6416c701958f4a2d955d298a2cc99724e81cf9135b8f22475ce4828d2f72
3
+ size 10228660
model/model/language_model/layers/0/mlp/gate_proj/kernel/1.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4348521f5fe2ffe278e7ec334995abf3f2577bda04d86ee0b64c5b1aec23a521
3
+ size 10226583
model/model/language_model/layers/0/mlp/gate_proj/kernel/2.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:62d27932d2e63b6a45d57b66af33bc076b42d3450b23e510a7fc19a53d5a3c67
3
+ size 10229132
model/model/language_model/layers/0/mlp/gate_proj/kernel/3.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bec38cdc83f2cb683753dca7729846b6ba2ff8fb270b785f98ea88a62e684c61
3
+ size 10226450
model/model/language_model/layers/0/mlp/up_proj/kernel/.zarray ADDED
@@ -0,0 +1 @@
 
 
1
+ {"chunks":[640,10240],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[2560,10240],"zarr_format":2}
model/model/language_model/layers/0/mlp/up_proj/kernel/0.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:853bed5756e09f7cd06b23bbc7d3b42abdef0ba1160f3eef181bbd252687e3f2
3
+ size 10233338
model/model/language_model/layers/0/mlp/up_proj/kernel/1.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6824f04f4490a35643c15fc1102d43f35e6d9fdffba8d300ce7872ae9b762651
3
+ size 10233480
model/model/language_model/layers/0/mlp/up_proj/kernel/2.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1299bfb9280519c28c941705a6df13407fe1f0504de706891fde2f7262849e83
3
+ size 10233254
model/model/language_model/layers/0/mlp/up_proj/kernel/3.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c14eed2803894b468bebf0b1b08cc0c3572be839f8ad757bea72983bc52b6013
3
+ size 10232235
model/model/language_model/layers/0/per_layer_input_gate/kernel/.zarray ADDED
@@ -0,0 +1 @@
 
 
1
+ {"chunks":[2560,256],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[2560,256],"zarr_format":2}
model/model/language_model/layers/0/per_layer_input_gate/kernel/0.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:13568133fe196c1223808ca1cf38cb6acf84d14e152ae314785dc3ef8a7c4127
3
+ size 1032916
model/model/language_model/layers/0/per_layer_projection/kernel/.zarray ADDED
@@ -0,0 +1 @@
 
 
1
+ {"chunks":[256,2560],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[256,2560],"zarr_format":2}
model/model/language_model/layers/0/per_layer_projection/kernel/0.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d0ac17db0746d6f064aa60665954d48ab2fb955ad0b95566763da1d82f8e97a7
3
+ size 1067568
model/model/language_model/layers/0/post_attention_layernorm/kernel/.zarray ADDED
@@ -0,0 +1 @@
 
 
1
+ {"chunks":[2560],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[2560],"zarr_format":2}
model/model/language_model/layers/0/post_attention_layernorm/kernel/0 ADDED
Binary file (3.62 kB). View file
 
model/model/language_model/layers/0/post_feedforward_layernorm/kernel/.zarray ADDED
@@ -0,0 +1 @@
 
 
1
+ {"chunks":[2560],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[2560],"zarr_format":2}
model/model/language_model/layers/0/post_feedforward_layernorm/kernel/0 ADDED
Binary file (3.23 kB). View file
 
model/model/language_model/layers/0/post_per_layer_input_norm/kernel/.zarray ADDED
@@ -0,0 +1 @@
 
 
1
+ {"chunks":[2560],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[2560],"zarr_format":2}
model/model/language_model/layers/0/post_per_layer_input_norm/kernel/0 ADDED
Binary file (3.07 kB). View file
 
model/model/language_model/layers/0/pre_feedforward_layernorm/kernel/.zarray ADDED
@@ -0,0 +1 @@
 
 
1
+ {"chunks":[2560],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[2560],"zarr_format":2}
model/model/language_model/layers/0/pre_feedforward_layernorm/kernel/0 ADDED
Binary file (3.15 kB). View file
 
model/model/language_model/layers/0/self_attn/k_norm/kernel/.zarray ADDED
@@ -0,0 +1 @@
 
 
1
+ {"chunks":[256],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[256],"zarr_format":2}
model/model/language_model/layers/0/self_attn/k_norm/kernel/0 ADDED
Binary file (18 Bytes). View file