Hugging Face's logo

nvidia
/
4D-RGPT-8B

Video-Text-to-Text
Transformers
Safetensors
English
llava_llama
multimodal
video-understanding
region-grounding
3d-reasoning
4d-reasoning
perceptual-distillation
nvila
vila
Model card Files
xet
 Community
1
ca-joe-yang commited on
Commit
cc76ee3
·
verified ·
1 Parent(s): e4cb1ee

Initial release: 4D-RGPT-8B v4 inference weights

Browse files
Files changed (19) hide show
  1. README.md +158 -0
  2. config.json +351 -0
  3. llm/added_tokens.json +10 -0
  4. llm/config.json +32 -0
  5. llm/generation_config.json +14 -0
  6. llm/merges.txt +0 -0
  7. llm/model-00001-of-00004.safetensors +3 -0
  8. llm/model-00002-of-00004.safetensors +3 -0
  9. llm/model-00003-of-00004.safetensors +3 -0
  10. llm/model-00004-of-00004.safetensors +3 -0
  11. llm/model.safetensors.index.json +346 -0
  12. llm/special_tokens_map.json +30 -0
  13. llm/tokenizer_config.json +88 -0
  14. llm/vocab.json +0 -0
  15. mm_projector/config.json +10 -0
  16. mm_projector/model.safetensors +3 -0
  17. vision_tower/config.json +23 -0
  18. vision_tower/model.safetensors +3 -0
  19. vision_tower/preprocessor_config.json +24 -0
README.md ADDED
@@ -0,0 +1,158 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Model Overview
2
+
3
+ ### Description:
4
+ 4D-RGPT is a specialized multimodal large language model that improves region-level 4D (i.e., 3D + time) video understanding by distilling latent and explicit 4D perceptual signals (for example, depth and optical flow) from a frozen expert model into an NVILA-based student model. 4D-RGPT was developed by NVIDIA as part of the NVILA visual-language model family and introduces Perceptual 4D Distillation (P4D), Timestamp Positional Encoding (TPE), and the companion R4D-Bench benchmark for region-level 4D VQA.
5
+
6
+ This model is for research and development only.
7
+
8
+ ### License/Terms of Use:
9
+ Use of this model is governed by the [CC-BY-NC-4.0 License](https://creativecommons.org/licenses/by-nc/4.0/legalcode.en).
10
+
11
+ ### Deployment Geography:
12
+ Global
13
+
14
+ ### Use Case:
15
+ Expected users are multimodal AI researchers, applied research teams, and developers studying video understanding, region grounding, 3D/4D reasoning, and physical AI. Representative use cases include region-level video question answering, model benchmarking, research on depth-and-time-aware MLLMs, and prototyping for domains such as robotics, autonomous driving, and industrial inspection.
16
+
17
+ ### Release Date:
18
+ Hugging Face [06/01/2026] via [https://huggingface.co/nvidia/4D-RGPT-8B]
19
+
20
+ ## References(s):
21
+ * Paper: https://arxiv.org/abs/2512.17012 <br>
22
+ * GitHub: https://github.com/NVlabs/4D-RGPT <br>
23
+ * Project page: https://www.ca-joe-yang.com/resource/projects/4D_RGPT/ <br>
24
+ * R4D-Bench: https://huggingface.co/datasets/nvidia/R4D-Bench <br>
25
+
26
+ ## Model Architecture:
27
+ **Architecture Type:** Transformer <br>
28
+
29
+ **Network Architecture:** NVILA-Lite-based MLLM using a SigLIP vision encoder, multimodal projector, and language model. <br>
30
+
31
+ **This model was developed based on:** NVILA-Lite-based MLLM <br>
32
+
33
+ **Number of model parameters:** 8.0\*10^9 for 4D-RGPT-8B <br>
34
+
35
+ **Describe design choices related to initialization techniques, hyperparameter tuning, regularization techniques, model optimization, damping, and training parameters:** 4D-RGPT adds a lightweight training-only MLP 4D perception decoder (hidden size 2,560) with GELU activations, Xavier weight initialization, and zero bias initialization. Training begins from pretrained NVILA weights.Ttotal loss combines SFT, latent distillation, and explicit distillation with Timestamp Positional Encoding uses T=10,000.
36
+
37
+ ## Input(s): <br>
38
+ **Input Type(s):** Image, Text, Video <br>
39
+
40
+ **Input Format(s):**
41
+ - Image: RGB <br>
42
+ - Text: String <br>
43
+ - Video: .mp4 <br>
44
+
45
+ **Input Parameters:**
46
+ - Image: Two-Dimensional (2D) <br>
47
+ - Text: One-Dimensional (1D) <br>
48
+ - Video: Three-Dimensional (3D) <br>
49
+
50
+ **Other Properties Related to Input:** The model is designed for video-question answering with explicit temporal cues encoded through timestamps for sampled frames. The paper uses sampled frame timestamps for TPE and, for fair comparison on R4D-Bench, evaluates open-source models using 16 sampled frames. Region-level evaluation uses region prompts represented through Set-of-Marks (SoM) or region masks in benchmark workflows. <br>
51
+
52
+ ## Output(s)
53
+
54
+ **Output Type(s):** Text <br>
55
+
56
+ **Output Format(s):**
57
+ - Text: String <br>
58
+
59
+ **Output Parameters:**
60
+ - Text: One-Dimensional (1D) <br>
61
+
62
+ **Other Properties Related to Output:** Outputs are text answers for 3D/4D VQA tasks, commonly multiple-choice selections, short phrases, or short numeric answers. The paper focuses on accuracy benchmarks rather than production API formatting. This model is designed to run on NVIDIA GPU-accelerated systems; the public training setup uses NVIDIA A100-SXM4-80GB GPUs. <br>
63
+
64
+ Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
65
+
66
+ ## Software Integration:
67
+ **Runtime Engine(s):**
68
+ * Not Applicable (N/A)- inference using [NVILA](https://github.com/NVlabs/VILA) <br>
69
+
70
+ **Supported Hardware Microarchitecture Compatibility:** <br>
71
+ * NVIDIA Ampere — A100-SXM4-80GB <br>
72
+
73
+ **Supported Operating System(s):**
74
+ * [Linux] <br>
75
+
76
+ The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
77
+
78
+ This AI model can be embedded as an Application Programming Interface (API) call into the software environment described above.
79
+
80
+ ## Model Version(s):
81
+ - **4D-RGPT-8B** — main paper model based on NVILA-Lite-8B; this is the primary reported configuration in the main results tables. <br>
82
+
83
+ ## Training, Testing, and Evaluation Datasets:
84
+
85
+ ### Dataset Overview
86
+ **Total Size:** Approximately **3.8e5** supervision examples / QA pairs / conversations across the disclosed training mixture, based on the paper-reported counts. This corresponds to approximately **2.06e5** unique visual items (about 190k images plus about 16.2k videos). <br>
87
+ **Total Number of Datasets:** 4 training datasets. <br>
88
+
89
+ **General description of data processing:** The training mixture comprises VSTI-Bench training data, the NuScenes portion of Wolf, RoboFAC, and SAT. For evaluation, this release reports results on the companion R4D-Bench benchmark, VLM4D-real, and VSTI-Bench. <br>
90
+
91
+ ## Public Datasets
92
+ **Training datasets:** <br>
93
+ - **VSTI-Bench (training split):** ~1.2k unique videos and ~130k QA pairs. Source videos are from ScanNet and ScanNet++. <br>
94
+ - **Wolf (NuScenes portion):** ~5k unique videos and ~15k QA pairs derived from dense captions. <br>
95
+ - **RoboFAC:** ~10k unique videos and ~65k conversations; simulated robotic-arm videos. <br>
96
+ - **SAT (training split):** ~190k unique simulated images and ~170k QA pairs. <br>
97
+
98
+ **Evaluation datasets:** <br>
99
+ - R4D-Bench <br>
100
+ - VLM4D-real <br>
101
+ - VSTI-Bench <br>
102
+
103
+
104
+ ## Training Dataset:
105
+
106
+ ### Data Modality:
107
+ * [Image] <br>
108
+ * [Text] <br>
109
+ * [Video] <br>
110
+
111
+ ### Training Data Size:
112
+ Approx. **3.8e5** supervision examples / QA pairs / conversations across the disclosed training mixture. <br>
113
+
114
+ **Image Training Data Size** <br>
115
+ * Less than a Million Images <br>
116
+
117
+ **Text Training Data Size** <br>
118
+ * Less than a Billion Tokens <br>
119
+
120
+ **Video Training Data Size** <br>
121
+ * Less than 10,000 Hours <br>
122
+
123
+ **Data Collection Method by dataset** <br>
124
+ * Hybrid: Automatic/Sensors, Human, Synthetic <br>
125
+
126
+ **Labeling Method by dataset** <br>
127
+ * Hybrid: Human, Automated, Synthetic <br>
128
+
129
+ **Properties:** The training data are English-language multimodal supervision examples spanning indoor scenes, autonomous driving, robotics, and simulated 3D/4D reasoning tasks. The mixture includes real-world videos, simulated videos, simulated images, and model-generated QA from captions. <br>
130
+
131
+ ### Evaluation Dataset:
132
+
133
+ Data Collection Method by dataset: <br>
134
+ * Hybrid: Automatic/Sensors, Human, Synthetic <br>
135
+
136
+ Labeling Method by dataset: <br>
137
+ * Hybrid: Human, Automated, Synthetic <br>
138
+
139
+ **Properties:** Public evaluation is reported on R4D-Bench, VLM4D-real, and VSTI-Bench. They are testing-only and disjoint from the training data. R4D-Bench is region-prompted and is curated from STI-Bench and VLM4D through keyword extraction, segmentation, Set-of-Marks prompting, automated matching, and human verification. <br>
140
+
141
+ **Quantitative Evaluation Benchmarks** <br>
142
+
143
+ **4D-RGPT-8B:** <br>
144
+ - R4D-Bench: 46.2 <br>
145
+ - VLM4D-real: 53.8 <br>
146
+ - VSTI-Bench: 59.8 <br>
147
+
148
+ ## Inference:
149
+ **Acceleration Engine:** N/A <br>
150
+ **Test Hardware:**
151
+ * NVIDIA A100
152
+
153
+ ## Ethical Considerations:
154
+ NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>
155
+
156
+ Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included. <br>
157
+
158
+ Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/). <br>
config.json ADDED
@@ -0,0 +1,351 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "Ubit": 100,
3
+ "_attn_implementation_autoset": true,
4
+ "_name_or_path": "runs/train/NVILA-Lite-8B-4D-dev_v11/model/tmp-checkpoint-300",
5
+ "architectures": [
6
+ "LlavaLlamaModel"
7
+ ],
8
+ "babit": 100,
9
+ "bobit": "E5M2",
10
+ "bwbit": 100,
11
+ "chat_template": null,
12
+ "col_blocksize": -1,
13
+ "col_blocksize_optimizer": 128,
14
+ "disable_distillation": false,
15
+ "distill_tasks": [
16
+ "depth",
17
+ "flow_2d_backward",
18
+ "dyn_mask",
19
+ "camray"
20
+ ],
21
+ "distillation_cfg": null,
22
+ "draw_distribution_backward": false,
23
+ "draw_distribution_forward": false,
24
+ "drop_path_rate": 0.0,
25
+ "dynamic_s2": false,
26
+ "ed_weights": "depth=0.1,flow_2d_backward=0.001,dyn_mask=0.01,camray=0.00001",
27
+ "epsilon": 1e-10,
28
+ "epsilon_optimizer": 1e-15,
29
+ "fabit": "E4M3",
30
+ "first_order_bit": null,
31
+ "first_order_quant_type": null,
32
+ "fobit": 100,
33
+ "fps": 0.0,
34
+ "fwbit": "E4M3",
35
+ "hidden_size": 3584,
36
+ "image_aspect_ratio": "resize",
37
+ "image_encoder": {
38
+ "_target_": "llava.model.encoders.BasicImageEncoder"
39
+ },
40
+ "interpolate_mode": "linear",
41
+ "ld_weight": 0.01,
42
+ "llm_cfg": {
43
+ "_attn_implementation_autoset": false,
44
+ "_name_or_path": "runs/train/NVILA-Lite-8B-4D-dev_v11/model/tmp-checkpoint-300/llm",
45
+ "add_cross_attention": false,
46
+ "architectures": [
47
+ "Qwen2ForCausalLM"
48
+ ],
49
+ "attention_dropout": 0.0,
50
+ "bad_words_ids": null,
51
+ "begin_suppress_tokens": null,
52
+ "bos_token_id": 151643,
53
+ "chunk_size_feed_forward": 0,
54
+ "cross_attention_hidden_size": null,
55
+ "decoder_start_token_id": null,
56
+ "diversity_penalty": 0.0,
57
+ "do_sample": false,
58
+ "early_stopping": false,
59
+ "encoder_no_repeat_ngram_size": 0,
60
+ "eos_token_id": 151645,
61
+ "exponential_decay_length_penalty": null,
62
+ "finetuning_task": null,
63
+ "forced_bos_token_id": null,
64
+ "forced_eos_token_id": null,
65
+ "hidden_act": "silu",
66
+ "hidden_size": 3584,
67
+ "id2label": {
68
+ "0": "LABEL_0",
69
+ "1": "LABEL_1"
70
+ },
71
+ "initializer_range": 0.02,
72
+ "intermediate_size": 18944,
73
+ "is_decoder": false,
74
+ "is_encoder_decoder": false,
75
+ "label2id": {
76
+ "LABEL_0": 0,
77
+ "LABEL_1": 1
78
+ },
79
+ "length_penalty": 1.0,
80
+ "max_length": 20,
81
+ "max_position_embeddings": 32768,
82
+ "max_window_layers": 28,
83
+ "min_length": 0,
84
+ "model_max_length": 16384,
85
+ "model_type": "qwen2",
86
+ "no_repeat_ngram_size": 0,
87
+ "num_attention_heads": 28,
88
+ "num_beam_groups": 1,
89
+ "num_beams": 1,
90
+ "num_hidden_layers": 28,
91
+ "num_key_value_heads": 4,
92
+ "num_return_sequences": 1,
93
+ "output_attentions": false,
94
+ "output_hidden_states": false,
95
+ "output_scores": false,
96
+ "pad_token_id": null,
97
+ "prefix": null,
98
+ "problem_type": null,
99
+ "pruned_heads": {},
100
+ "remove_invalid_values": false,
101
+ "repetition_penalty": 1.0,
102
+ "return_dict": true,
103
+ "return_dict_in_generate": false,
104
+ "rms_norm_eps": 1e-06,
105
+ "rope_scaling": null,
106
+ "rope_theta": 1000000.0,
107
+ "sep_token_id": null,
108
+ "sliding_window": null,
109
+ "suppress_tokens": null,
110
+ "task_specific_params": null,
111
+ "temperature": 1.0,
112
+ "tf_legacy_loss": false,
113
+ "tie_encoder_decoder": false,
114
+ "tie_word_embeddings": false,
115
+ "tokenizer_class": null,
116
+ "tokenizer_model_max_length": 4096,
117
+ "tokenizer_padding_side": "right",
118
+ "top_k": 50,
119
+ "top_p": 1.0,
120
+ "torch_dtype": "bfloat16",
121
+ "torchscript": false,
122
+ "typical_p": 1.0,
123
+ "use_bfloat16": false,
124
+ "use_cache": false,
125
+ "use_sliding_window": false,
126
+ "vocab_size": 151651
127
+ },
128
+ "low_level_perception": "l4p",
129
+ "low_level_perception_cfg": null,
130
+ "max_tiles": 12,
131
+ "min_blockunit_col": 4,
132
+ "min_blockunit_row": 4,
133
+ "min_tiles": 1,
134
+ "mlp_path": null,
135
+ "mm_hidden_size": 1152,
136
+ "mm_projector": "mlp_downsample",
137
+ "mm_projector_cfg": {
138
+ "_attn_implementation_autoset": false,
139
+ "_name_or_path": "runs/train/NVILA-Lite-8B-4D-dev_v11/model/tmp-checkpoint-300/mm_projector",
140
+ "add_cross_attention": false,
141
+ "architectures": [
142
+ "MultimodalProjector"
143
+ ],
144
+ "bad_words_ids": null,
145
+ "begin_suppress_tokens": null,
146
+ "bos_token_id": null,
147
+ "chunk_size_feed_forward": 0,
148
+ "cross_attention_hidden_size": null,
149
+ "decoder_start_token_id": null,
150
+ "diversity_penalty": 0.0,
151
+ "do_sample": false,
152
+ "early_stopping": false,
153
+ "encoder_no_repeat_ngram_size": 0,
154
+ "eos_token_id": null,
155
+ "exponential_decay_length_penalty": null,
156
+ "finetuning_task": null,
157
+ "forced_bos_token_id": null,
158
+ "forced_eos_token_id": null,
159
+ "id2label": {
160
+ "0": "LABEL_0",
161
+ "1": "LABEL_1"
162
+ },
163
+ "is_decoder": false,
164
+ "is_encoder_decoder": false,
165
+ "label2id": {
166
+ "LABEL_0": 0,
167
+ "LABEL_1": 1
168
+ },
169
+ "length_penalty": 1.0,
170
+ "max_length": 20,
171
+ "min_length": 0,
172
+ "mm_projector_type": "mlp_downsample_3x3_fix",
173
+ "model_type": "v2l_projector",
174
+ "no_repeat_ngram_size": 0,
175
+ "num_beam_groups": 1,
176
+ "num_beams": 1,
177
+ "num_return_sequences": 1,
178
+ "output_attentions": false,
179
+ "output_hidden_states": false,
180
+ "output_scores": false,
181
+ "pad_token_id": null,
182
+ "prefix": null,
183
+ "problem_type": null,
184
+ "pruned_heads": {},
185
+ "remove_invalid_values": false,
186
+ "repetition_penalty": 1.0,
187
+ "return_dict": true,
188
+ "return_dict_in_generate": false,
189
+ "sep_token_id": null,
190
+ "suppress_tokens": null,
191
+ "task_specific_params": null,
192
+ "temperature": 1.0,
193
+ "tf_legacy_loss": false,
194
+ "tie_encoder_decoder": false,
195
+ "tie_word_embeddings": true,
196
+ "tokenizer_class": null,
197
+ "top_k": 50,
198
+ "top_p": 1.0,
199
+ "torch_dtype": "bfloat16",
200
+ "torchscript": false,
201
+ "typical_p": 1.0,
202
+ "use_bfloat16": false
203
+ },
204
+ "mm_projector_lr": null,
205
+ "mm_use_im_patch_token": false,
206
+ "mm_use_im_start_end": false,
207
+ "mm_vision_select_feature": "cls_patch",
208
+ "mm_vision_select_layer": -2,
209
+ "model_dtype": "torch.bfloat16",
210
+ "model_name_or_path": "Efficient-Large-Model/NVILA-Lite-8B",
211
+ "model_type": "llava_llama",
212
+ "num_time_tokens": 0,
213
+ "num_video_frames": 16,
214
+ "output_dir": "runs/train/NVILA-Lite-8B-4D-dev_v11/model",
215
+ "pad_block": false,
216
+ "pad_to_multiple_of": 0,
217
+ "pe": "rope",
218
+ "pe_cfg": "rope",
219
+ "qchoice": [
220
+ "none",
221
+ "all",
222
+ "linear",
223
+ "mlp",
224
+ "attn",
225
+ "gelu",
226
+ "layernorm",
227
+ "backbone",
228
+ "residual",
229
+ "backbone"
230
+ ],
231
+ "quantize_model": false,
232
+ "refine_attn_blocksize": false,
233
+ "refine_col_blocksize": 4,
234
+ "refine_ln_blocksize": false,
235
+ "refine_ln_blocksize_but_only_backward": false,
236
+ "refine_ln_blocksize_but_only_forward": false,
237
+ "refine_ln_pertoken": false,
238
+ "refine_mlp_blocksize": false,
239
+ "refine_residual_fp": false,
240
+ "refine_row_blocksize": 4,
241
+ "region_extractor": "regiongpt",
242
+ "region_extractor_cfg": "regiongpt",
243
+ "resume_path": "/lustre/fs12/portfolios/nvr/projects/nvr_taiwan_rvos/users/joeya/projects/4d_rgpt/runs/train/NVILA-Lite-8B-4D-dev_v11/model/checkpoint-150",
244
+ "row_blocksize": -1,
245
+ "row_blocksize_optimizer": 1,
246
+ "s2": false,
247
+ "s2_max_split_size": 336,
248
+ "s2_resize_output_to_scale_idx": 0,
249
+ "s2_scales": "336,672,1008",
250
+ "second_order_bit": null,
251
+ "second_order_quant_type": null,
252
+ "soft_ce_std": 1.0,
253
+ "symm": true,
254
+ "time_token_format": "<t{t}>",
255
+ "time_token_ids": [],
256
+ "transformers_version": "4.46.3",
257
+ "tune_language_model": true,
258
+ "tune_mm_projector": false,
259
+ "tune_pe": false,
260
+ "tune_region_extractor": true,
261
+ "tune_vision_tower": false,
262
+ "use_quantize_optimizer": false,
263
+ "use_time_pe": false,
264
+ "version": "auto",
265
+ "video_encoder": {
266
+ "_target_": "llava.model.encoders.BasicVideoEncoder"
267
+ },
268
+ "vision_resolution": -1,
269
+ "vision_tower": "google/siglip-so400m-patch14-384",
270
+ "vision_tower_cfg": {
271
+ "_attn_implementation_autoset": false,
272
+ "_name_or_path": "runs/train/NVILA-Lite-8B-4D-dev_v11/model/tmp-checkpoint-300/vision_tower",
273
+ "add_cross_attention": false,
274
+ "architectures": [
275
+ "SiglipVisionModel"
276
+ ],
277
+ "attention_dropout": 0.0,
278
+ "bad_words_ids": null,
279
+ "begin_suppress_tokens": null,
280
+ "bos_token_id": null,
281
+ "chunk_size_feed_forward": 0,
282
+ "cross_attention_hidden_size": null,
283
+ "decoder_start_token_id": null,
284
+ "diversity_penalty": 0.0,
285
+ "do_sample": false,
286
+ "early_stopping": false,
287
+ "encoder_no_repeat_ngram_size": 0,
288
+ "eos_token_id": null,
289
+ "exponential_decay_length_penalty": null,
290
+ "finetuning_task": null,
291
+ "forced_bos_token_id": null,
292
+ "forced_eos_token_id": null,
293
+ "hidden_act": "gelu_pytorch_tanh",
294
+ "hidden_size": 1152,
295
+ "id2label": {
296
+ "0": "LABEL_0",
297
+ "1": "LABEL_1"
298
+ },
299
+ "image_size": 448,
300
+ "intermediate_size": 4304,
301
+ "is_decoder": false,
302
+ "is_encoder_decoder": false,
303
+ "label2id": {
304
+ "LABEL_0": 0,
305
+ "LABEL_1": 1
306
+ },
307
+ "layer_norm_eps": 1e-06,
308
+ "length_penalty": 1.0,
309
+ "max_length": 20,
310
+ "min_length": 0,
311
+ "model_type": "siglip_vision_model",
312
+ "no_repeat_ngram_size": 0,
313
+ "num_attention_heads": 16,
314
+ "num_beam_groups": 1,
315
+ "num_beams": 1,
316
+ "num_channels": 3,
317
+ "num_hidden_layers": 27,
318
+ "num_image_tokens": 256,
319
+ "num_return_sequences": 1,
320
+ "output_attentions": false,
321
+ "output_hidden_states": false,
322
+ "output_scores": false,
323
+ "pad_token_id": null,
324
+ "patch_size": 14,
325
+ "prefix": null,
326
+ "problem_type": null,
327
+ "projection_dim": 2048,
328
+ "projector_hidden_act": "gelu_fast",
329
+ "pruned_heads": {},
330
+ "remove_invalid_values": false,
331
+ "repetition_penalty": 1.0,
332
+ "return_dict": true,
333
+ "return_dict_in_generate": false,
334
+ "sep_token_id": null,
335
+ "suppress_tokens": null,
336
+ "task_specific_params": null,
337
+ "temperature": 1.0,
338
+ "tf_legacy_loss": false,
339
+ "tie_encoder_decoder": false,
340
+ "tie_word_embeddings": true,
341
+ "tokenizer_class": null,
342
+ "top_k": 50,
343
+ "top_p": 1.0,
344
+ "torch_dtype": "bfloat16",
345
+ "torchscript": false,
346
+ "typical_p": 1.0,
347
+ "use_bfloat16": false,
348
+ "vision_use_head": false
349
+ },
350
+ "vision_tower_lr": null
351
+ }
llm/added_tokens.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "<image>": 151649,
3
+ "<vila/sentinel>": 151648,
4
+ "<vila/video>": 151650,
5
+ "<|endoftext|>": 151643,
6
+ "<|im_end|>": 151645,
7
+ "<|im_start|>": 151644,
8
+ "[BOS]": 151646,
9
+ "[PAD]": 151647
10
+ }
llm/config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "runs/train/NVILA-Lite-8B-4D-dev_v11/model/tmp-checkpoint-300/llm",
3
+ "architectures": [
4
+ "Qwen2ForCausalLM"
5
+ ],
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 151643,
8
+ "eos_token_id": 151645,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 3584,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 18944,
13
+ "max_position_embeddings": 32768,
14
+ "max_window_layers": 28,
15
+ "model_max_length": 16384,
16
+ "model_type": "qwen2",
17
+ "num_attention_heads": 28,
18
+ "num_hidden_layers": 28,
19
+ "num_key_value_heads": 4,
20
+ "rms_norm_eps": 1e-06,
21
+ "rope_scaling": null,
22
+ "rope_theta": 1000000.0,
23
+ "sliding_window": null,
24
+ "tie_word_embeddings": false,
25
+ "tokenizer_model_max_length": 4096,
26
+ "tokenizer_padding_side": "right",
27
+ "torch_dtype": "bfloat16",
28
+ "transformers_version": "4.46.3",
29
+ "use_cache": false,
30
+ "use_sliding_window": false,
31
+ "vocab_size": 151651
32
+ }
llm/generation_config.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 151643,
3
+ "do_sample": true,
4
+ "eos_token_id": [
5
+ 151645,
6
+ 151643
7
+ ],
8
+ "pad_token_id": 151643,
9
+ "repetition_penalty": 1.05,
10
+ "temperature": 0.7,
11
+ "top_k": 20,
12
+ "top_p": 0.8,
13
+ "transformers_version": "4.46.3"
14
+ }
llm/merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
llm/model-00001-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:666a12ec2557072f47355d646eff0a9b9cb625348cb9699d7a4390efb97b708e
3
+ size 4874700392
llm/model-00002-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:38d2a6835b693e7f2d580da64e75634702d63f6e404beb072868aa9c31e1f87a
3
+ size 4932751008
llm/model-00003-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:983a71d693625278303d9e29958bcfe969ec49e20807d3b37598651f662d6f62
3
+ size 4330865200
llm/model-00004-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d70fc103c5a4ee4b49d579412e16994b8d2b65eda6319ac44cb9181656493cca
3
+ size 1087034496
llm/model.safetensors.index.json ADDED
@@ -0,0 +1,346 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 15225312256
4
+ },
5
+ "weight_map": {
6
+ "lm_head.weight": "model-00004-of-00004.safetensors",
7
+ "model.embed_tokens.weight": "model-00001-of-00004.safetensors",
8
+ "model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
9
+ "model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
10
+ "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
11
+ "model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
12
+ "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
13
+ "model.layers.0.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
14
+ "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
15
+ "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
16
+ "model.layers.0.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
17
+ "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
18
+ "model.layers.0.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
19
+ "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
20
+ "model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
21
+ "model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
22
+ "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
23
+ "model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
24
+ "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
25
+ "model.layers.1.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
26
+ "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
27
+ "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
28
+ "model.layers.1.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
29
+ "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
30
+ "model.layers.1.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
31
+ "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
32
+ "model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
33
+ "model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
34
+ "model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
35
+ "model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
36
+ "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
37
+ "model.layers.10.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
38
+ "model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
39
+ "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
40
+ "model.layers.10.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
41
+ "model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
42
+ "model.layers.10.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
43
+ "model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
44
+ "model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
45
+ "model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
46
+ "model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
47
+ "model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
48
+ "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
49
+ "model.layers.11.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
50
+ "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
51
+ "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
52
+ "model.layers.11.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
53
+ "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
54
+ "model.layers.11.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
55
+ "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
56
+ "model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
57
+ "model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
58
+ "model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
59
+ "model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
60
+ "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
61
+ "model.layers.12.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
62
+ "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
63
+ "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
64
+ "model.layers.12.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
65
+ "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
66
+ "model.layers.12.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
67
+ "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
68
+ "model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
69
+ "model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
70
+ "model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
71
+ "model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
72
+ "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
73
+ "model.layers.13.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
74
+ "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
75
+ "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
76
+ "model.layers.13.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
77
+ "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
78
+ "model.layers.13.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
79
+ "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
80
+ "model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
81
+ "model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
82
+ "model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
83
+ "model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
84
+ "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
85
+ "model.layers.14.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
86
+ "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
87
+ "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
88
+ "model.layers.14.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
89
+ "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
90
+ "model.layers.14.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
91
+ "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
92
+ "model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
93
+ "model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
94
+ "model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
95
+ "model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
96
+ "model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
97
+ "model.layers.15.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
98
+ "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
99
+ "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
100
+ "model.layers.15.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
101
+ "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
102
+ "model.layers.15.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
103
+ "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
104
+ "model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
105
+ "model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
106
+ "model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
107
+ "model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
108
+ "model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
109
+ "model.layers.16.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
110
+ "model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
111
+ "model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
112
+ "model.layers.16.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
113
+ "model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
114
+ "model.layers.16.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
115
+ "model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
116
+ "model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
117
+ "model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
118
+ "model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
119
+ "model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
120
+ "model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
121
+ "model.layers.17.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
122
+ "model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
123
+ "model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
124
+ "model.layers.17.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
125
+ "model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
126
+ "model.layers.17.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
127
+ "model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
128
+ "model.layers.18.input_layernorm.weight": "model-00003-of-00004.safetensors",
129
+ "model.layers.18.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
130
+ "model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
131
+ "model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
132
+ "model.layers.18.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
133
+ "model.layers.18.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
134
+ "model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
135
+ "model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
136
+ "model.layers.18.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
137
+ "model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
138
+ "model.layers.18.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
139
+ "model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
140
+ "model.layers.19.input_layernorm.weight": "model-00003-of-00004.safetensors",
141
+ "model.layers.19.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
142
+ "model.layers.19.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
143
+ "model.layers.19.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
144
+ "model.layers.19.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
145
+ "model.layers.19.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
146
+ "model.layers.19.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
147
+ "model.layers.19.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
148
+ "model.layers.19.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
149
+ "model.layers.19.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
150
+ "model.layers.19.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
151
+ "model.layers.19.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
152
+ "model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
153
+ "model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
154
+ "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
155
+ "model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
156
+ "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
157
+ "model.layers.2.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
158
+ "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
159
+ "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
160
+ "model.layers.2.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
161
+ "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
162
+ "model.layers.2.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
163
+ "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
164
+ "model.layers.20.input_layernorm.weight": "model-00003-of-00004.safetensors",
165
+ "model.layers.20.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
166
+ "model.layers.20.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
167
+ "model.layers.20.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
168
+ "model.layers.20.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
169
+ "model.layers.20.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
170
+ "model.layers.20.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
171
+ "model.layers.20.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
172
+ "model.layers.20.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
173
+ "model.layers.20.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
174
+ "model.layers.20.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
175
+ "model.layers.20.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
176
+ "model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
177
+ "model.layers.21.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
178
+ "model.layers.21.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
179
+ "model.layers.21.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
180
+ "model.layers.21.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
181
+ "model.layers.21.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
182
+ "model.layers.21.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
183
+ "model.layers.21.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
184
+ "model.layers.21.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
185
+ "model.layers.21.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
186
+ "model.layers.21.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
187
+ "model.layers.21.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
188
+ "model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
189
+ "model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
190
+ "model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
191
+ "model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
192
+ "model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
193
+ "model.layers.22.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
194
+ "model.layers.22.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
195
+ "model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
196
+ "model.layers.22.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
197
+ "model.layers.22.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
198
+ "model.layers.22.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
199
+ "model.layers.22.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
200
+ "model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
201
+ "model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
202
+ "model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
203
+ "model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
204
+ "model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
205
+ "model.layers.23.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
206
+ "model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
207
+ "model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
208
+ "model.layers.23.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
209
+ "model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
210
+ "model.layers.23.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
211
+ "model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
212
+ "model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
213
+ "model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
214
+ "model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
215
+ "model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
216
+ "model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
217
+ "model.layers.24.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
218
+ "model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
219
+ "model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
220
+ "model.layers.24.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
221
+ "model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
222
+ "model.layers.24.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
223
+ "model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
224
+ "model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
225
+ "model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
226
+ "model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
227
+ "model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
228
+ "model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
229
+ "model.layers.25.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
230
+ "model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
231
+ "model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
232
+ "model.layers.25.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
233
+ "model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
234
+ "model.layers.25.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
235
+ "model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
236
+ "model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
237
+ "model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
238
+ "model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
239
+ "model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
240
+ "model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
241
+ "model.layers.26.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
242
+ "model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
243
+ "model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
244
+ "model.layers.26.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
245
+ "model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
246
+ "model.layers.26.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
247
+ "model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
248
+ "model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
249
+ "model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
250
+ "model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
251
+ "model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
252
+ "model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
253
+ "model.layers.27.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
254
+ "model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
255
+ "model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
256
+ "model.layers.27.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
257
+ "model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
258
+ "model.layers.27.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
259
+ "model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
260
+ "model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
261
+ "model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
262
+ "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
263
+ "model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
264
+ "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
265
+ "model.layers.3.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
266
+ "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
267
+ "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
268
+ "model.layers.3.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
269
+ "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
270
+ "model.layers.3.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
271
+ "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
272
+ "model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
273
+ "model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
274
+ "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
275
+ "model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
276
+ "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
277
+ "model.layers.4.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
278
+ "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
279
+ "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
280
+ "model.layers.4.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
281
+ "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
282
+ "model.layers.4.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
283
+ "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
284
+ "model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
285
+ "model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
286
+ "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
287
+ "model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
288
+ "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
289
+ "model.layers.5.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
290
+ "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
291
+ "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
292
+ "model.layers.5.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
293
+ "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
294
+ "model.layers.5.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
295
+ "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
296
+ "model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
297
+ "model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
298
+ "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
299
+ "model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
300
+ "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
301
+ "model.layers.6.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
302
+ "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
303
+ "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
304
+ "model.layers.6.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
305
+ "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
306
+ "model.layers.6.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
307
+ "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
308
+ "model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
309
+ "model.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
310
+ "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
311
+ "model.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
312
+ "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
313
+ "model.layers.7.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
314
+ "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
315
+ "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
316
+ "model.layers.7.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
317
+ "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
318
+ "model.layers.7.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
319
+ "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
320
+ "model.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
321
+ "model.layers.8.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
322
+ "model.layers.8.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
323
+ "model.layers.8.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
324
+ "model.layers.8.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
325
+ "model.layers.8.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
326
+ "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
327
+ "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
328
+ "model.layers.8.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
329
+ "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
330
+ "model.layers.8.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
331
+ "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
332
+ "model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
333
+ "model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
334
+ "model.layers.9.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
335
+ "model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
336
+ "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
337
+ "model.layers.9.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
338
+ "model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
339
+ "model.layers.9.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
340
+ "model.layers.9.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
341
+ "model.layers.9.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
342
+ "model.layers.9.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
343
+ "model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
344
+ "model.norm.weight": "model-00003-of-00004.safetensors"
345
+ }
346
+ }
llm/special_tokens_map.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<|im_start|>",
4
+ "<|im_end|>",
5
+ "<vila/sentinel>",
6
+ "<image>",
7
+ "<vila/video>"
8
+ ],
9
+ "bos_token": {
10
+ "content": "[BOS]",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "eos_token": {
17
+ "content": "<|im_end|>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "pad_token": {
24
+ "content": "[PAD]",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ }
30
+ }
llm/tokenizer_config.json ADDED
@@ -0,0 +1,88 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "added_tokens_decoder": {
4
+ "151643": {
5
+ "content": "<|endoftext|>",
6
+ "lstrip": false,
7
+ "normalized": false,
8
+ "rstrip": false,
9
+ "single_word": false,
10
+ "special": true
11
+ },
12
+ "151644": {
13
+ "content": "<|im_start|>",
14
+ "lstrip": false,
15
+ "normalized": false,
16
+ "rstrip": false,
17
+ "single_word": false,
18
+ "special": true
19
+ },
20
+ "151645": {
21
+ "content": "<|im_end|>",
22
+ "lstrip": false,
23
+ "normalized": false,
24
+ "rstrip": false,
25
+ "single_word": false,
26
+ "special": true
27
+ },
28
+ "151646": {
29
+ "content": "[BOS]",
30
+ "lstrip": false,
31
+ "normalized": false,
32
+ "rstrip": false,
33
+ "single_word": false,
34
+ "special": true
35
+ },
36
+ "151647": {
37
+ "content": "[PAD]",
38
+ "lstrip": false,
39
+ "normalized": false,
40
+ "rstrip": false,
41
+ "single_word": false,
42
+ "special": true
43
+ },
44
+ "151648": {
45
+ "content": "<vila/sentinel>",
46
+ "lstrip": false,
47
+ "normalized": false,
48
+ "rstrip": false,
49
+ "single_word": false,
50
+ "special": true
51
+ },
52
+ "151649": {
53
+ "content": "<image>",
54
+ "lstrip": false,
55
+ "normalized": false,
56
+ "rstrip": false,
57
+ "single_word": false,
58
+ "special": true
59
+ },
60
+ "151650": {
61
+ "content": "<vila/video>",
62
+ "lstrip": false,
63
+ "normalized": false,
64
+ "rstrip": false,
65
+ "single_word": false,
66
+ "special": true
67
+ }
68
+ },
69
+ "additional_special_tokens": [
70
+ "<|im_start|>",
71
+ "<|im_end|>",
72
+ "<vila/sentinel>",
73
+ "<image>",
74
+ "<vila/video>"
75
+ ],
76
+ "bos_token": "[BOS]",
77
+ "chat_template": "{% if messages[0]['role'] != 'system' %}{{ '<|im_start|>system\\nYou are a helpful assistant<|im_end|>\\n' }}{% endif %}{% for message in messages if message['content'] is not none %}{{ '<|im_start|>' + message['role'] + '\\n' + message['content'] + '<|im_end|>' + '\\n' }}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\\n' }}{% endif %}",
78
+ "clean_up_tokenization_spaces": false,
79
+ "eos_token": "<|im_end|>",
80
+ "errors": "replace",
81
+ "legacy": false,
82
+ "model_max_length": 16384,
83
+ "pad_token": "[PAD]",
84
+ "padding_side": "right",
85
+ "split_special_tokens": false,
86
+ "tokenizer_class": "Qwen2Tokenizer",
87
+ "unk_token": null
88
+ }
llm/vocab.json ADDED
The diff for this file is too large to render. See raw diff
 
mm_projector/config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "runs/train/NVILA-Lite-8B-4D-dev_v11/model/tmp-checkpoint-300/mm_projector",
3
+ "architectures": [
4
+ "MultimodalProjector"
5
+ ],
6
+ "mm_projector_type": "mlp_downsample_3x3_fix",
7
+ "model_type": "v2l_projector",
8
+ "torch_dtype": "bfloat16",
9
+ "transformers_version": "4.46.3"
10
+ }
mm_projector/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:be36150c27b7f9af417f1f1584353565ff68d6b43518d10da03417292ab36e4a
3
+ size 122203760
vision_tower/config.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "runs/train/NVILA-Lite-8B-4D-dev_v11/model/tmp-checkpoint-300/vision_tower",
3
+ "architectures": [
4
+ "SiglipVisionModel"
5
+ ],
6
+ "attention_dropout": 0.0,
7
+ "hidden_act": "gelu_pytorch_tanh",
8
+ "hidden_size": 1152,
9
+ "image_size": 448,
10
+ "intermediate_size": 4304,
11
+ "layer_norm_eps": 1e-06,
12
+ "model_type": "siglip_vision_model",
13
+ "num_attention_heads": 16,
14
+ "num_channels": 3,
15
+ "num_hidden_layers": 27,
16
+ "num_image_tokens": 256,
17
+ "patch_size": 14,
18
+ "projection_dim": 2048,
19
+ "projector_hidden_act": "gelu_fast",
20
+ "torch_dtype": "bfloat16",
21
+ "transformers_version": "4.46.3",
22
+ "vision_use_head": false
23
+ }
vision_tower/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5a46ef371610c7293e9d0b06b2e6f8f0644544c3307702adc64d3d4147a6acba
3
+ size 826707904
vision_tower/preprocessor_config.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "do_convert_rgb": null,
3
+ "do_normalize": true,
4
+ "do_rescale": true,
5
+ "do_resize": true,
6
+ "image_mean": [
7
+ 0.5,
8
+ 0.5,
9
+ 0.5
10
+ ],
11
+ "image_processor_type": "SiglipImageProcessor",
12
+ "image_std": [
13
+ 0.5,
14
+ 0.5,
15
+ 0.5
16
+ ],
17
+ "processor_class": "SiglipProcessor",
18
+ "resample": 3,
19
+ "rescale_factor": 0.00392156862745098,
20
+ "size": {
21
+ "height": 448,
22
+ "width": 448
23
+ }
24
+ }