Upload KimiVLForConditionalGeneration

README.md

@@ -0,0 +1,123 @@
+---
+tags:
+- EasyDeL
+- KimiVLForConditionalGeneration
+- TaskType.IMAGE_TEXT_TO_TEXT
+- AttentionMechanisms.RAGGED_PAGE_ATTENTION_V3
+- safetensors
+- TPU
+- GPU
+- XLA
+- Flax
+---
+<p align="center">
+  <a href="https://github.com/erfanzar/EasyDeL">
+    <img src="https://raw.githubusercontent.com/erfanzar/easydel/main/images/easydel-logo-with-text.png" height="80">
+  </a>
+</p>
+
+<p align="center">
+  <a href="https://github.com/erfanzar/EasyDeL">
+    <img src="https://img.shields.io/badge/🤗_EasyDeL-0.2.0-blue.svg" />
+  </a>
+  <a href="https://github.com/erfanzar/EasyDeL">
+    <img src="https://img.shields.io/badge/Model_Type-KimiVLForConditionalGeneration-green.svg" />
+  </a>
+</p>
+
+# EasyDeL/Kimi-VL-A3B-Instruct
+
+A model implemented using the EasyDeL framework, designed to deliver optimal performance for large-scale natural language processing tasks.
+
+## Overview
+
+This model is built using [EasyDeL](https://github.com/erfanzar/EasyDeL), an open-source framework designed to enhance and streamline the training and serving process of machine learning models, with a primary focus on Jax/Flax on TPU/GPU at scale.
+
+EasyDeL provides an efficient, highly-optimized, and customizable machine learning model compatible with both GPU and TPU environments. Built with JAX, this model supports advanced features such as sharded model parallelism, making it suitable for distributed training and inference and customized kernels.
+
+## Features Provided by EasyDeL
+
+**EasyDeL Framework Features:**
+
+- **Efficient Implementation**: Built with JAX/Flax for high-performance computation.
+- **Modern Architecture**: Built on Flax NNX for better integration, modularity, and performance.
+- **Multi-Device Support**: Optimized to run on TPU, GPU, and CPU environments.
+- **Sharded Model Parallelism**: Supports model parallelism across multiple devices for scalability (using `auto_shard_model=True`).
+- **Customizable Precision**: Allows specification of `dtype`, `param_dtype`, and `precision`.
+- **Advanced Serving**: Includes `eSurge` LLM serving engine, `vWhisper` speech endpoints, and OpenAI-compatible APIs.
+- **Optimized Kernels**: Integrates multiple attention mechanisms (like `AttentionMechanisms.RAGGED_PAGE_ATTENTION_V3`) and platform-specific optimizations.
+
+## Installation
+
+To use this model via EasyDeL, first install EasyDeL:
+
+```bash
+pip install easydel
+```
+
+## Usage
+
+### Loading the Pre-trained Model
+
+To load this pre-trained model with EasyDeL:
+
+```python
+from easydel import AutoEasyDeLModelForCausalLM, EasyDeLBaseConfigDict, AttentionMechanisms
+from jax import numpy as jnp, lax
+
+# Define max_length if needed for memory optimization
+max_length = None
+
+# Load model and parameters
+# Set auto_shard_model=True to automatically distribute across devices
+model = AutoEasyDeLModelForCausalLM.from_pretrained(
+    "EasyDeL/Kimi-VL-A3B-Instruct",
+    config_kwargs=EasyDeLBaseConfigDict(
+        # use_scan_mlp=False, # Set to True to potentially reduce memory usage
+        attn_dtype=jnp.float16, # Or jnp.bfloat16
+        # freq_max_position_embeddings=max_length, # Set if using RoPE and need truncation
+        # mask_max_position_embeddings=max_length, # Set if max length is defined
+        attn_mechanism=AttentionMechanisms.RAGGED_PAGE_ATTENTION_V3 # Matches the mechanism used by this model
+    ),
+    dtype=jnp.float16, # Or jnp.bfloat16 - Computation data type
+    param_dtype=jnp.float16, # Or jnp.bfloat16 - Parameter data type
+    precision=lax.Precision("fastest"), # Like "default", "fastest", "high", "highest"
+    auto_shard_model=True, # Auto-shard across available devices
+)
+```
+
+## Supported Tasks
+
+The primary task for this model is **TaskType.IMAGE_TEXT_TO_TEXT**. Further specific supported tasks are not explicitly listed.
+
+## Limitations
+
+**General Limitations:**
+
+- **Hardware Dependency**: Performance can vary significantly based on the hardware (TPU/GPU) used.
+- **JAX/Flax Setup Required**: The environment must support JAX/Flax for optimal use.
+- **Experimental Features**: Some EasyDeL features (like custom kernels) may require additional configuration.
+
+## License 📜
+
+EasyDeL is released under the Apache v2 license. The license for this specific model might differ; please consult the original model repository or documentation.
+
+```code
+# Apache License 2.0 (referring to EasyDeL Framework)
+# ... (Full license text usually included in the main repo) ...
+```
+
+## Citation
+
+If you use EasyDeL in your research or work, please cite it:
+
+```bibtex
+@misc{Zare Chavoshi_2023,
+    title={EasyDeL: An open-source library for enhancing and streamlining the training process of machine learning models},
+    url={https://github.com/erfanzar/EasyDeL},
+    author={Zare Chavoshi, Erfan},
+    year={2023}
+}
+```
+
+Please also consider citing the original paper or source for the **EasyDeL/Kimi-VL-A3B-Instruct** model architecture if applicable.

checkpoint_metadata.json

@@ -0,0 +1,8 @@
+{
+  "version": "0.0.85",
+  "timestamp": "2025-12-15T11:21:29.648456",
+  "checksum": {},
+  "array_metadata": {},
+  "framework_version": null,
+  "custom_metadata": {}
+}

config.json

@@ -0,0 +1,399 @@
+{
+  "architectures": [
+    "KimiVLForConditionalGeneration"
+  ],
+  "attn_mechanism": "ragged_page_attention_v3",
+  "auto_map": {
+    "AutoConfig": "configuration_kimi_vl.KimiVLConfig",
+    "AutoModel": "modeling_kimi_vl.KimiVLForConditionalGeneration",
+    "AutoModelForCausalLM": "modeling_kimi_vl.KimiVLForConditionalGeneration"
+  },
+  "backend": null,
+  "bits": null,
+  "blocksize_b": 1,
+  "blocksize_k": 128,
+  "blocksize_q": 128,
+  "decode_attn_mechanism": null,
+  "dtype": "bfloat16",
+  "easy_method": "train",
+  "fcm_max_ratio": 0.0,
+  "fcm_min_ratio": 0.0,
+  "flash_attention_backward_pass_impl": "triton",
+  "freq_max_position_embeddings": 40960,
+  "fsdp_is_ep_bound": true,
+  "gradient_checkpointing": "",
+  "gradient_checkpointing_targets": null,
+  "hardware_abstraction": true,
+  "ignore_index": -100,
+  "kv_cache_quantization_config": null,
+  "kv_cache_sharding_sequence_axis_name": "sp",
+  "mask_max_position_embeddings": 40960,
+  "media_placeholder_token_id": 163605,
+  "model_type": "kimi_vl",
+  "moe_force_xla_gmm": false,
+  "moe_method": "fused_moe",
+  "moe_tiling_size_batch": 4,
+  "moe_tiling_size_dim": 128,
+  "moe_tiling_size_seqlen": 128,
+  "operation_configs": null,
+  "pad_token_id": 0,
+  "pallas_k_block_size": 128,
+  "pallas_m_block_size": 128,
+  "pallas_n_block_size": 128,
+  "partition_axis": {
+    "attention_dim_axis": null,
+    "attention_kv_dim_axis": null,
+    "batch_axis": [
+      "fsdp",
+      "dp"
+    ],
+    "bias_head_sequence_axis": null,
+    "bias_key_sequence_axis": null,
+    "data_parallel_axis": "dp",
+    "decode_attention_dim_axis": null,
+    "decode_attention_kv_dim_axis": null,
+    "decode_batch_axis": [
+      "fsdp",
+      "dp"
+    ],
+    "decode_head_axis": "tp",
+    "decode_key_sequence_axis": "sp",
+    "decode_kv_head_axis": "tp",
+    "decode_query_sequence_axis": null,
+    "expert_axis": "ep",
+    "expert_gate_axis": null,
+    "expert_parallel_axis": "ep",
+    "fully_sharded_data_parallel_axis": "fsdp",
+    "head_axis": "tp",
+    "hidden_state_axis": "tp",
+    "key_sequence_axis": "sp",
+    "kv_head_axis": "tp",
+    "mlp_intermediate_axis": "tp",
+    "query_sequence_axis": "sp",
+    "sequence_axis": "sp",
+    "sequence_parallel_axis": "sp",
+    "tensor_parallel_axis": "tp",
+    "vocab_axis": "tp"
+  },
+  "platform": null,
+  "precompute_masks": true,
+  "pretraining_tp": 1,
+  "quantization_config": null,
+  "scan_attention_layers": false,
+  "scan_mlp_chunk_size": 1024,
+  "scan_ring_attention": true,
+  "sequence_axis_name": "sp",
+  "sharding_axis_dims": [
+    1,
+    1,
+    1,
+    -1,
+    1
+  ],
+  "sharding_axis_names": [
+    "dp",
+    "fsdp",
+    "ep",
+    "tp",
+    "sp"
+  ],
+  "sharding_dcn_axis_dims": null,
+  "sp_is_ep_bound": true,
+  "text_config": {
+    "architectures": [
+      "KimiVLForConditionalGeneration"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_mechanism": "ragged_page_attention_v3",
+    "auto_map": {
+      "AutoConfig": "configuration_kimi_vl.KimiVLConfig",
+      "AutoModel": "modeling_kimi_vl.KimiVLForConditionalGeneration",
+      "AutoModelForCausalLM": "modeling_kimi_vl.KimiVLForConditionalGeneration"
+    },
+    "aux_loss_alpha": 0.001,
+    "backend": null,
+    "bits": null,
+    "blocksize_b": 1,
+    "blocksize_k": 128,
+    "blocksize_q": 128,
+    "bos_token_id": 163584,
+    "decode_attn_mechanism": null,
+    "dtype": "bfloat16",
+    "easy_method": "train",
+    "eos_token_id": 163585,
+    "ep_size": 1,
+    "fcm_max_ratio": 0.0,
+    "fcm_min_ratio": 0.0,
+    "first_k_dense_replace": 1,
+    "flash_attention_backward_pass_impl": "triton",
+    "freq_max_position_embeddings": 40960,
+    "fsdp_is_ep_bound": true,
+    "gradient_checkpointing": "",
+    "gradient_checkpointing_targets": null,
+    "hardware_abstraction": true,
+    "head_dim": 192,
+    "hidden_act": "silu",
+    "hidden_size": 2048,
+    "initializer_range": 0.02,
+    "intermediate_size": 11264,
+    "kv_cache_quantization_config": null,
+    "kv_cache_sharding_sequence_axis_name": "sp",
+    "kv_lora_rank": 512,
+    "layer_types": [
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention"
+    ],
+    "mask_max_position_embeddings": 40960,
+    "max_position_embeddings": 131072,
+    "model_type": "deepseek_v3",
+    "moe_force_xla_gmm": false,
+    "moe_intermediate_size": 1408,
+    "moe_layer_freq": 1,
+    "moe_method": "fused_moe",
+    "moe_tiling_size_batch": 4,
+    "moe_tiling_size_dim": 128,
+    "moe_tiling_size_seqlen": 128,
+    "n_group": 1,
+    "n_routed_experts": 64,
+    "n_shared_experts": 2,
+    "norm_topk_prob": true,
+    "num_attention_heads": 16,
+    "num_experts_per_tok": 6,
+    "num_hidden_layers": 27,
+    "num_key_value_heads": 16,
+    "num_nextn_predict_layers": 1,
+    "operation_configs": null,
+    "pad_token_id": 163839,
+    "pallas_k_block_size": 128,
+    "pallas_m_block_size": 128,
+    "pallas_n_block_size": 128,
+    "partition_axis": {
+      "attention_dim_axis": null,
+      "attention_kv_dim_axis": null,
+      "batch_axis": [
+        "fsdp",
+        "dp"
+      ],
+      "bias_head_sequence_axis": null,
+      "bias_key_sequence_axis": null,
+      "data_parallel_axis": "dp",
+      "decode_attention_dim_axis": null,
+      "decode_attention_kv_dim_axis": null,
+      "decode_batch_axis": [
+        "fsdp",
+        "dp"
+      ],
+      "decode_head_axis": "tp",
+      "decode_key_sequence_axis": "sp",
+      "decode_kv_head_axis": "tp",
+      "decode_query_sequence_axis": null,
+      "expert_axis": "ep",
+      "expert_gate_axis": null,
+      "expert_parallel_axis": "ep",
+      "fully_sharded_data_parallel_axis": "fsdp",
+      "head_axis": "tp",
+      "hidden_state_axis": "tp",
+      "key_sequence_axis": "sp",
+      "kv_head_axis": "tp",
+      "mlp_intermediate_axis": "tp",
+      "query_sequence_axis": "sp",
+      "sequence_axis": "sp",
+      "sequence_parallel_axis": "sp",
+      "tensor_parallel_axis": "tp",
+      "vocab_axis": "tp"
+    },
+    "platform": null,
+    "precompute_masks": true,
+    "pretraining_tp": 1,
+    "q_lora_rank": null,
+    "qk_nope_head_dim": 128,
+    "qk_rope_head_dim": 64,
+    "quantization_config": null,
+    "rms_norm_eps": 1e-05,
+    "rope_scaling": null,
+    "rope_theta": 800000.0,
+    "routed_scaling_factor": 2.446,
+    "scan_attention_layers": false,
+    "scan_mlp_chunk_size": 1024,
+    "scan_ring_attention": true,
+    "scoring_func": "sigmoid",
+    "seq_aux": true,
+    "sequence_axis_name": "sp",
+    "sharding_axis_dims": [
+      1,
+      1,
+      1,
+      -1,
+      1
+    ],
+    "sharding_axis_names": [
+      "dp",
+      "fsdp",
+      "ep",
+      "tp",
+      "sp"
+    ],
+    "sharding_dcn_axis_dims": null,
+    "sp_is_ep_bound": true,
+    "topk_group": 1,
+    "topk_method": "noaux_tc",
+    "use_cache": true,
+    "use_expert_tensor_mode": false,
+    "use_ring_of_experts": false,
+    "use_scan_mlp": false,
+    "use_sharded_kv_caching": false,
+    "use_sharding_constraint": false,
+    "v_head_dim": 128,
+    "vocab_size": 163840
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.3",
+  "use_expert_tensor_mode": false,
+  "use_ring_of_experts": false,
+  "use_scan_mlp": false,
+  "use_sharded_kv_caching": false,
+  "use_sharding_constraint": false,
+  "vision_config": {
+    "architectures": [
+      "KimiVLForConditionalGeneration"
+    ],
+    "attn_mechanism": "ragged_page_attention_v3",
+    "auto_map": {
+      "AutoConfig": "configuration_kimi_vl.KimiVLConfig",
+      "AutoModel": "modeling_kimi_vl.KimiVLForConditionalGeneration",
+      "AutoModelForCausalLM": "modeling_kimi_vl.KimiVLForConditionalGeneration"
+    },
+    "backend": null,
+    "bits": null,
+    "blocksize_b": 1,
+    "blocksize_k": 128,
+    "blocksize_q": 128,
+    "decode_attn_mechanism": null,
+    "dtype": "bfloat16",
+    "easy_method": "train",
+    "fcm_max_ratio": 0.0,
+    "fcm_min_ratio": 0.0,
+    "flash_attention_backward_pass_impl": "triton",
+    "freq_max_position_embeddings": 40960,
+    "fsdp_is_ep_bound": true,
+    "gradient_checkpointing": "",
+    "gradient_checkpointing_targets": null,
+    "hardware_abstraction": true,
+    "hidden_size": 1152,
+    "init_pos_emb_height": 64,
+    "init_pos_emb_width": 64,
+    "intermediate_size": 4304,
+    "kv_cache_quantization_config": null,
+    "kv_cache_sharding_sequence_axis_name": "sp",
+    "mask_max_position_embeddings": 40960,
+    "merge_kernel_size": [
+      2,
+      2
+    ],
+    "model_type": "moonvit",
+    "moe_force_xla_gmm": false,
+    "moe_method": "fused_moe",
+    "moe_tiling_size_batch": 4,
+    "moe_tiling_size_dim": 128,
+    "moe_tiling_size_seqlen": 128,
+    "num_attention_heads": 16,
+    "num_hidden_layers": 27,
+    "operation_configs": null,
+    "pallas_k_block_size": 128,
+    "pallas_m_block_size": 128,
+    "pallas_n_block_size": 128,
+    "partition_axis": {
+      "attention_dim_axis": null,
+      "attention_kv_dim_axis": null,
+      "batch_axis": [
+        "fsdp",
+        "dp"
+      ],
+      "bias_head_sequence_axis": null,
+      "bias_key_sequence_axis": null,
+      "data_parallel_axis": "dp",
+      "decode_attention_dim_axis": null,
+      "decode_attention_kv_dim_axis": null,
+      "decode_batch_axis": [
+        "fsdp",
+        "dp"
+      ],
+      "decode_head_axis": "tp",
+      "decode_key_sequence_axis": "sp",
+      "decode_kv_head_axis": "tp",
+      "decode_query_sequence_axis": null,
+      "expert_axis": "ep",
+      "expert_gate_axis": null,
+      "expert_parallel_axis": "ep",
+      "fully_sharded_data_parallel_axis": "fsdp",
+      "head_axis": "tp",
+      "hidden_state_axis": "tp",
+      "key_sequence_axis": "sp",
+      "kv_head_axis": "tp",
+      "mlp_intermediate_axis": "tp",
+      "query_sequence_axis": "sp",
+      "sequence_axis": "sp",
+      "sequence_parallel_axis": "sp",
+      "tensor_parallel_axis": "tp",
+      "vocab_axis": "tp"
+    },
+    "patch_size": 14,
+    "platform": null,
+    "precompute_masks": true,
+    "pretraining_tp": 1,
+    "quantization_config": null,
+    "scan_attention_layers": false,
+    "scan_mlp_chunk_size": 1024,
+    "scan_ring_attention": true,
+    "sequence_axis_name": "sp",
+    "sharding_axis_dims": [
+      1,
+      1,
+      1,
+      -1,
+      1
+    ],
+    "sharding_axis_names": [
+      "dp",
+      "fsdp",
+      "ep",
+      "tp",
+      "sp"
+    ],
+    "sharding_dcn_axis_dims": null,
+    "sp_is_ep_bound": true,
+    "use_expert_tensor_mode": false,
+    "use_ring_of_experts": false,
+    "use_scan_mlp": false,
+    "use_sharded_kv_caching": false,
+    "use_sharding_constraint": false,
+    "vocab_size": 163840
+  },
+  "vocab_size": 163840
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "bos_token_id": 163584,
+  "do_sample": true,
+  "eos_token_id": [
+    163585
+  ],
+  "pad_token_id": 163838,
+  "temperature": 0.2,
+  "transformers_version": "4.57.3"
+}

model/language_model/lm_head/kernel/.zarray

@@ -0,0 +1 @@
+{"chunks":[2048,40960],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[2048,163840],"zarr_format":2}

model/language_model/lm_head/kernel/0.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a63e9c3ff3401c20facd55ecb94d60847d1006607efbd5c7e8aceff55d02084b
+size 131102355

model/language_model/lm_head/kernel/0.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:33e75e80569aa4c19cde796fe58f236fbf968f3a2237fefe33394ebd50ad10f7
+size 130719145

model/language_model/lm_head/kernel/0.2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:438cf01dcd64730af247c6c8128d2e54cf4ee8fd214923f8e436c9d7d136bf78
+size 130532818

model/language_model/lm_head/kernel/0.3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:58ee279bd18ec439d7f228e3d1df025a67098ad34e8c2b52c67e96a065d92187
+size 130384513

model/language_model/model/embed_tokens/embedding/.zarray

@@ -0,0 +1 @@
+{"chunks":[163840,512],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[163840,2048],"zarr_format":2}

model/language_model/model/embed_tokens/embedding/0.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be8065b661be9ad3be95867fb8089352c2305de4cca9aa40f5e7d0806c37ac1a
+size 130810030

model/language_model/model/embed_tokens/embedding/0.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5064efeb57188fc540e2e2c9598eb03f909d685c2f53de470d9c46d832cb7e76
+size 130810992

model/language_model/model/embed_tokens/embedding/0.2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:366f6841e6e5abbc9ff762ebbf87427760ace33e335342a4a69ec3bbfada9512
+size 130807493

model/language_model/model/embed_tokens/embedding/0.3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37545eb2886fe601aeb669c4a35890e135032a15353f8716142db70bd5472b19
+size 130807472

model/language_model/model/layers/0/input_layernorm/kernel/.zarray

@@ -0,0 +1 @@
+{"chunks":[2048],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[2048],"zarr_format":2}

model/language_model/model/layers/0/mlp/down_proj/kernel/.zarray

@@ -0,0 +1 @@
+{"chunks":[2816,2048],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[11264,2048],"zarr_format":2}

model/language_model/model/layers/0/mlp/down_proj/kernel/0.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bf60a32f0b3214fe9b7ac01a6d1e2f0c5be77f0d856be9b3cc711638b577e5d1
+size 9021981

model/language_model/model/layers/0/mlp/down_proj/kernel/1.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59638343feb8361352694bb7c87f91bcd140d29619658bd77cc1e0101ecb0c92
+size 9021753

model/language_model/model/layers/0/mlp/down_proj/kernel/2.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b302ffe8b6b1551439721f113ff717beeab8029a0013a6fe36fa58de348fd0d0
+size 9020679

model/language_model/model/layers/0/mlp/down_proj/kernel/3.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d7d7804f3d4b74fce03c521a6c5aff8267679141d62dbb9a15a03e91d812f65
+size 9021529

model/language_model/model/layers/0/mlp/gate_proj/kernel/.zarray

@@ -0,0 +1 @@
+{"chunks":[2048,2816],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[2048,11264],"zarr_format":2}

model/language_model/model/layers/0/mlp/gate_proj/kernel/0.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e942350103be8aa516671d8b9f4857054d62f4bb59b6f1cc0c0d30a1b3337226
+size 9029129

model/language_model/model/layers/0/mlp/gate_proj/kernel/0.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e431343bda4d6c5ac45225b8a8903a70e52363c599c7ee8fed3a3b2813092b6
+size 9029287

model/language_model/model/layers/0/mlp/gate_proj/kernel/0.2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f35b8a4b06bcf1afae6fc1a5c41cb1393f710d19fa280e8ea55af6f05f2f609
+size 9029213

model/language_model/model/layers/0/mlp/gate_proj/kernel/0.3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4ed0c0c6779b9e6c6197c54f5fd89aa2c3ca99531dda4a1b7201af571e64173a
+size 9028435

model/language_model/model/layers/0/mlp/up_proj/kernel/.zarray

@@ -0,0 +1 @@
+{"chunks":[2048,2816],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[2048,11264],"zarr_format":2}

model/language_model/model/layers/0/mlp/up_proj/kernel/0.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:10850d1789777f84472564c7a1659081e6672ec35b6b018b2f65e5ac47a0c780
+size 9028870

model/language_model/model/layers/0/mlp/up_proj/kernel/0.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:11416ff25247b61577a28c9d77d85ce52b5de3e70c58adb9270905bb304a388a
+size 9028826

model/language_model/model/layers/0/mlp/up_proj/kernel/0.2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17e97b3e55ce2e988fc58caab9a2ebdacf984702d70d21b485b0cac75bcf69db
+size 9028315

model/language_model/model/layers/0/mlp/up_proj/kernel/0.3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe6d299d6c14199a0601a540dabaeadca817c58b690b3815393e96f0013af14a
+size 9027448

model/language_model/model/layers/0/post_attention_layernorm/kernel/.zarray

@@ -0,0 +1 @@
+{"chunks":[2048],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[2048],"zarr_format":2}

model/language_model/model/layers/0/self_attn/kv_a_layernorm/kernel/.zarray

@@ -0,0 +1 @@
+{"chunks":[512],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[512],"zarr_format":2}

model/language_model/model/layers/0/self_attn/kv_a_proj_with_mqa/kernel/.zarray

@@ -0,0 +1 @@
+{"chunks":[2048,144],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[2048,576],"zarr_format":2}

model/language_model/model/layers/0/self_attn/kv_a_proj_with_mqa/kernel/0.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:051dcdda963f127302981ccc4151c884488ea8eeddd946c1b537ab4ac750936f
+size 461611

model/language_model/model/layers/0/self_attn/kv_a_proj_with_mqa/kernel/0.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4b31e64bae7db92df30e0c60b23633def93f2a9162c974df9db0677f45abf5ff
+size 461422

model/language_model/model/layers/0/self_attn/kv_a_proj_with_mqa/kernel/0.2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:75a22b7852e0cfd23ee23203234daec00b5187ab0531277a9cc7dc19275f55ea
+size 461724

model/language_model/model/layers/0/self_attn/kv_a_proj_with_mqa/kernel/0.3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38166fc7a4fb13c84176a7e1fe1ad3fb17b59387e206fc7ce7af213177ef6c07
+size 465596

model/language_model/model/layers/0/self_attn/kv_b_proj/kernel/.zarray

@@ -0,0 +1 @@
+{"chunks":[512,1024],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[512,4096],"zarr_format":2}

model/language_model/model/layers/0/self_attn/kv_b_proj/kernel/0.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eca0650432a15207c76ab6ee550f0894bc84cb22c6cd3306e375b47d10236a97
+size 842700

model/language_model/model/layers/0/self_attn/kv_b_proj/kernel/0.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:03e9dde9879a68cd8689de0c0e284a531ebb5a6b61924d259d26b43352d1feb6
+size 851802

model/language_model/model/layers/0/self_attn/kv_b_proj/kernel/0.2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:221683d111f1969998441fdb2a17411b031e13e7eff596796249172626660d23
+size 846798

model/language_model/model/layers/0/self_attn/kv_b_proj/kernel/0.3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5d32510b19ba51e931249a92756bddba5d4e12cdfd4d3ec6c582bce063c50ea
+size 852033

model/language_model/model/layers/0/self_attn/o_proj/kernel/.zarray

@@ -0,0 +1 @@
+{"chunks":[512,2048],"compressor":{"id":"zstd","level":1},"dimension_separator":".","dtype":"bfloat16","fill_value":null,"filters":null,"order":"C","shape":[2048,2048],"zarr_format":2}

model/language_model/model/layers/0/self_attn/o_proj/kernel/0.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bf995252924f618313b1b47735d0afde9638e64a164d8a2692968832bc42f454
+size 1632089

model/language_model/model/layers/0/self_attn/o_proj/kernel/1.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:daf927088d1e489cbdae6c365dc53b1ec819b37d191c76fac337858a75972dd0
+size 1629443

model/language_model/model/layers/0/self_attn/o_proj/kernel/2.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b982469cf6789ad1418136046eaefaa5c6bf824d03fe4729a2894d057bfcdd5a
+size 1629410