ayushexel meta-bot commited on
Commit
627a1cf
·
verified ·
0 Parent(s):

Duplicate from facebook/vjepa2-vitg-fpc64-384-ssv2

Browse files

Co-authored-by: Meta Platforms <meta-bot@users.noreply.huggingface.co>

.gitattributes ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,94 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ pipeline_tag: video-classification
4
+ tags:
5
+ - video
6
+ library_name: transformers
7
+ datasets:
8
+ - HuggingFaceM4/something_something_v2
9
+ base_model:
10
+ - facebook/vjepa2-vitg-fpc64-384
11
+ ---
12
+
13
+ # V-JEPA 2
14
+
15
+ A frontier video understanding model developed by FAIR, Meta, which extends the pretraining objectives of [VJEPA](https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-joint-embedding-predictive-architecture/), resulting in state-of-the-art video understanding capabilities, leveraging data and model sizes at scale.
16
+ The code is released [in this repository](https://github.com/facebookresearch/vjepa2).
17
+
18
+ <div style="background-color: rgba(251, 255, 120, 0.4); padding: 10px; color: black; border-radius: 5px; box-shadow: 0 4px 8px rgba(0,0,0,0.1);">
19
+ 💡 This is V-JEPA 2 <a href="https://huggingface.co/facebook/vjepa2-vitg-fpc64-384">ViT-g 384</a> model with video classification head pretrained on <a href="https://paperswithcode.com/dataset/something-something-v2" style="color: black;">Something-Something-V2</a> dataset.
20
+ </div>
21
+ <br></br>
22
+
23
+ <img src="https://github.com/user-attachments/assets/914942d8-6a1e-409d-86ff-ff856b7346ab">&nbsp;
24
+
25
+ ## Installation
26
+
27
+ To run V-JEPA 2 model, ensure you have installed the latest transformers:
28
+
29
+ ```bash
30
+ pip install -U git+https://github.com/huggingface/transformers
31
+ ```
32
+
33
+ ## Video classification code snippet
34
+
35
+ ```python
36
+ import torch
37
+ import numpy as np
38
+
39
+ from torchcodec.decoders import VideoDecoder
40
+ from transformers import AutoVideoProcessor, AutoModelForVideoClassification
41
+
42
+ device = "cuda" if torch.cuda.is_available() else "cpu"
43
+
44
+ # Load model and video preprocessor
45
+ hf_repo = "facebook/vjepa2-vitg-fpc64-384-ssv2"
46
+
47
+ model = AutoModelForVideoClassification.from_pretrained(hf_repo).to(device)
48
+ processor = AutoVideoProcessor.from_pretrained(hf_repo)
49
+
50
+ # To load a video, sample the number of frames according to the model.
51
+ # For this model, we use 64.
52
+ video_url = "https://huggingface.co/datasets/nateraw/kinetics-mini/resolve/main/val/bowling/-WH-lxmGJVY_000005_000015.mp4"
53
+ vr = VideoDecoder(video_url)
54
+ frame_idx = np.arange(0, model.config.frames_per_clip, 2) # you can define more complex sampling strategy
55
+ video = vr.get_frames_at(indices=frame_idx).data # frames x channels x height x width
56
+
57
+ # Preprocess and run inference
58
+ inputs = processor(video, return_tensors="pt").to(model.device)
59
+ with torch.no_grad():
60
+ outputs = model(**inputs)
61
+ logits = outputs.logits
62
+
63
+ print("Top 5 predicted class names:")
64
+ top5_indices = logits.topk(5).indices[0]
65
+ top5_probs = torch.softmax(logits, dim=-1).topk(5).values[0]
66
+ for idx, prob in zip(top5_indices, top5_probs):
67
+ text_label = model.config.id2label[idx.item()]
68
+ print(f" - {text_label}: {prob:.2f}")
69
+ ```
70
+ Output:
71
+ ```
72
+ Top 5 predicted class names:
73
+ - Putting [something] onto [something]: 0.39
74
+ - Putting [something similar to other things that are already on the table]: 0.23
75
+ - Stacking [number of] [something]: 0.07
76
+ - Putting [something] into [something]: 0.04
77
+ - Putting [number of] [something] onto [something]: 0.03
78
+ ```
79
+
80
+ ## Citation
81
+
82
+ ```
83
+ @techreport{assran2025vjepa2,
84
+ title={V-JEPA~2: Self-Supervised Video Models Enable Understanding, Prediction and Planning},
85
+ author={Assran, Mahmoud and Bardes, Adrien and Fan, David and Garrido, Quentin and Howes, Russell and
86
+ Komeili, Mojtaba and Muckley, Matthew and Rizvi, Ammar and Roberts, Claire and Sinha, Koustuv and Zholus, Artem and
87
+ Arnaud, Sergio and Gejji, Abha and Martin, Ada and Robert Hogan, Francois and Dugas, Daniel and
88
+ Bojanowski, Piotr and Khalidov, Vasil and Labatut, Patrick and Massa, Francisco and Szafraniec, Marc and
89
+ Krishnakumar, Kapil and Li, Yong and Ma, Xiaodong and Chandar, Sarath and Meier, Franziska and LeCun, Yann and
90
+ Rabbat, Michael and Ballas, Nicolas},
91
+ institution={FAIR at Meta},
92
+ year={2025}
93
+ }
94
+ ```
config.json ADDED
@@ -0,0 +1,386 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "VJEPA2ForVideoClassification"
4
+ ],
5
+ "attention_dropout": 0.0,
6
+ "attention_probs_dropout_prob": 0.0,
7
+ "crop_size": 384,
8
+ "drop_path_rate": 0.0,
9
+ "frames_per_clip": 64,
10
+ "hidden_act": "gelu",
11
+ "hidden_dropout_prob": 0.0,
12
+ "hidden_size": 1408,
13
+ "id2label": {
14
+ "0": "Approaching [something] with your camera",
15
+ "1": "Attaching [something] to [something]",
16
+ "2": "Bending [something] so that it deforms",
17
+ "3": "Bending [something] until it breaks",
18
+ "4": "Burying [something] in [something]",
19
+ "5": "Closing [something]",
20
+ "6": "Covering [something] with [something]",
21
+ "7": "Digging [something] out of [something]",
22
+ "8": "Dropping [something] behind [something]",
23
+ "9": "Dropping [something] in front of [something]",
24
+ "10": "Dropping [something] into [something]",
25
+ "11": "Dropping [something] next to [something]",
26
+ "12": "Dropping [something] onto [something]",
27
+ "13": "Failing to put [something] into [something] because [something] does not fit",
28
+ "14": "Folding [something]",
29
+ "15": "Hitting [something] with [something]",
30
+ "16": "Holding [something]",
31
+ "17": "Holding [something] behind [something]",
32
+ "18": "Holding [something] in front of [something]",
33
+ "19": "Holding [something] next to [something]",
34
+ "20": "Holding [something] over [something]",
35
+ "21": "Laying [something] on the table on its side, not upright",
36
+ "22": "Letting [something] roll along a flat surface",
37
+ "23": "Letting [something] roll down a slanted surface",
38
+ "24": "Letting [something] roll up a slanted surface, so it rolls back down",
39
+ "25": "Lifting a surface with [something] on it but not enough for it to slide down",
40
+ "26": "Lifting a surface with [something] on it until it starts sliding down",
41
+ "27": "Lifting [something] up completely without letting it drop down",
42
+ "28": "Lifting [something] up completely, then letting it drop down",
43
+ "29": "Lifting [something] with [something] on it",
44
+ "30": "Lifting up one end of [something] without letting it drop down",
45
+ "31": "Lifting up one end of [something], then letting it drop down",
46
+ "32": "Moving away from [something] with your camera",
47
+ "33": "Moving [part] of [something]",
48
+ "34": "Moving [something] across a surface until it falls down",
49
+ "35": "Moving [something] across a surface without it falling down",
50
+ "36": "Moving [something] and [something] away from each other",
51
+ "37": "Moving [something] and [something] closer to each other",
52
+ "38": "Moving [something] and [something] so they collide with each other",
53
+ "39": "Moving [something] and [something] so they pass each other",
54
+ "40": "Moving [something] away from [something]",
55
+ "41": "Moving [something] away from the camera",
56
+ "42": "Moving [something] closer to [something]",
57
+ "43": "Moving [something] down",
58
+ "44": "Moving [something] towards the camera",
59
+ "45": "Moving [something] up",
60
+ "46": "Opening [something]",
61
+ "47": "Picking [something] up",
62
+ "48": "Piling [something] up",
63
+ "49": "Plugging [something] into [something]",
64
+ "50": "Plugging [something] into [something] but pulling it right out as you remove your hand",
65
+ "51": "Poking a hole into [some substance]",
66
+ "52": "Poking a hole into [something soft]",
67
+ "53": "Poking a stack of [something] so the stack collapses",
68
+ "54": "Poking a stack of [something] without the stack collapsing",
69
+ "55": "Poking [something] so it slightly moves",
70
+ "56": "Poking [something] so lightly that it doesn't or almost doesn't move",
71
+ "57": "Poking [something] so that it falls over",
72
+ "58": "Poking [something] so that it spins around",
73
+ "59": "Pouring [something] into [something]",
74
+ "60": "Pouring [something] into [something] until it overflows",
75
+ "61": "Pouring [something] onto [something]",
76
+ "62": "Pouring [something] out of [something]",
77
+ "63": "Pretending or failing to wipe [something] off of [something]",
78
+ "64": "Pretending or trying and failing to twist [something]",
79
+ "65": "Pretending to be tearing [something that is not tearable]",
80
+ "66": "Pretending to close [something] without actually closing it",
81
+ "67": "Pretending to open [something] without actually opening it",
82
+ "68": "Pretending to pick [something] up",
83
+ "69": "Pretending to poke [something]",
84
+ "70": "Pretending to pour [something] out of [something], but [something] is empty",
85
+ "71": "Pretending to put [something] behind [something]",
86
+ "72": "Pretending to put [something] into [something]",
87
+ "73": "Pretending to put [something] next to [something]",
88
+ "74": "Pretending to put [something] on a surface",
89
+ "75": "Pretending to put [something] onto [something]",
90
+ "76": "Pretending to put [something] underneath [something]",
91
+ "77": "Pretending to scoop [something] up with [something]",
92
+ "78": "Pretending to spread air onto [something]",
93
+ "79": "Pretending to sprinkle air onto [something]",
94
+ "80": "Pretending to squeeze [something]",
95
+ "81": "Pretending to take [something] from [somewhere]",
96
+ "82": "Pretending to take [something] out of [something]",
97
+ "83": "Pretending to throw [something]",
98
+ "84": "Pretending to turn [something] upside down",
99
+ "85": "Pulling [something] from behind of [something]",
100
+ "86": "Pulling [something] from left to right",
101
+ "87": "Pulling [something] from right to left",
102
+ "88": "Pulling [something] onto [something]",
103
+ "89": "Pulling [something] out of [something]",
104
+ "90": "Pulling two ends of [something] but nothing happens",
105
+ "91": "Pulling two ends of [something] so that it gets stretched",
106
+ "92": "Pulling two ends of [something] so that it separates into two pieces",
107
+ "93": "Pushing [something] from left to right",
108
+ "94": "Pushing [something] from right to left",
109
+ "95": "Pushing [something] off of [something]",
110
+ "96": "Pushing [something] onto [something]",
111
+ "97": "Pushing [something] so it spins",
112
+ "98": "Pushing [something] so that it almost falls off but doesn't",
113
+ "99": "Pushing [something] so that it falls off the table",
114
+ "100": "Pushing [something] so that it slightly moves",
115
+ "101": "Pushing [something] with [something]",
116
+ "102": "Putting [number of] [something] onto [something]",
117
+ "103": "Putting [something] and [something] on the table",
118
+ "104": "Putting [something] behind [something]",
119
+ "105": "Putting [something] in front of [something]",
120
+ "106": "Putting [something] into [something]",
121
+ "107": "Putting [something] next to [something]",
122
+ "108": "Putting [something] on a flat surface without letting it roll",
123
+ "109": "Putting [something] on a surface",
124
+ "110": "Putting [something] on the edge of [something] so it is not supported and falls down",
125
+ "111": "Putting [something] onto a slanted surface but it doesn't glide down",
126
+ "112": "Putting [something] onto [something]",
127
+ "113": "Putting [something] onto [something else that cannot support it] so it falls down",
128
+ "114": "Putting [something similar to other things that are already on the table]",
129
+ "115": "Putting [something] that can't roll onto a slanted surface, so it slides down",
130
+ "116": "Putting [something] that can't roll onto a slanted surface, so it stays where it is",
131
+ "117": "Putting [something that cannot actually stand upright] upright on the table, so it falls on its side",
132
+ "118": "Putting [something] underneath [something]",
133
+ "119": "Putting [something] upright on the table",
134
+ "120": "Putting [something], [something] and [something] on the table",
135
+ "121": "Removing [something], revealing [something] behind",
136
+ "122": "Rolling [something] on a flat surface",
137
+ "123": "Scooping [something] up with [something]",
138
+ "124": "Showing a photo of [something] to the camera",
139
+ "125": "Showing [something] behind [something]",
140
+ "126": "Showing [something] next to [something]",
141
+ "127": "Showing [something] on top of [something]",
142
+ "128": "Showing [something] to the camera",
143
+ "129": "Showing that [something] is empty",
144
+ "130": "Showing that [something] is inside [something]",
145
+ "131": "[Something] being deflected from [something]",
146
+ "132": "[Something] colliding with [something] and both are being deflected",
147
+ "133": "[Something] colliding with [something] and both come to a halt",
148
+ "134": "[Something] falling like a feather or paper",
149
+ "135": "[Something] falling like a rock",
150
+ "136": "Spilling [something] behind [something]",
151
+ "137": "Spilling [something] next to [something]",
152
+ "138": "Spilling [something] onto [something]",
153
+ "139": "Spinning [something] so it continues spinning",
154
+ "140": "Spinning [something] that quickly stops spinning",
155
+ "141": "Spreading [something] onto [something]",
156
+ "142": "Sprinkling [something] onto [something]",
157
+ "143": "Squeezing [something]",
158
+ "144": "Stacking [number of] [something]",
159
+ "145": "Stuffing [something] into [something]",
160
+ "146": "Taking [one of many similar things on the table]",
161
+ "147": "Taking [something] from [somewhere]",
162
+ "148": "Taking [something] out of [something]",
163
+ "149": "Tearing [something] into two pieces",
164
+ "150": "Tearing [something] just a little bit",
165
+ "151": "Throwing [something]",
166
+ "152": "Throwing [something] against [something]",
167
+ "153": "Throwing [something] in the air and catching it",
168
+ "154": "Throwing [something] in the air and letting it fall",
169
+ "155": "Throwing [something] onto a surface",
170
+ "156": "Tilting [something] with [something] on it slightly so it doesn't fall down",
171
+ "157": "Tilting [something] with [something] on it until it falls off",
172
+ "158": "Tipping [something] over",
173
+ "159": "Tipping [something] with [something in it] over, so [something in it] falls out",
174
+ "160": "Touching (without moving) [part] of [something]",
175
+ "161": "Trying but failing to attach [something] to [something] because it doesn't stick",
176
+ "162": "Trying to bend [something unbendable] so nothing happens",
177
+ "163": "Trying to pour [something] into [something], but missing so it spills next to it",
178
+ "164": "Turning [something] upside down",
179
+ "165": "Turning the camera downwards while filming [something]",
180
+ "166": "Turning the camera left while filming [something]",
181
+ "167": "Turning the camera right while filming [something]",
182
+ "168": "Turning the camera upwards while filming [something]",
183
+ "169": "Twisting (wringing) [something] wet until water comes out",
184
+ "170": "Twisting [something]",
185
+ "171": "Uncovering [something]",
186
+ "172": "Unfolding [something]",
187
+ "173": "Wiping [something] off of [something]"
188
+ },
189
+ "image_size": 384,
190
+ "in_chans": 3,
191
+ "initializer_range": 0.02,
192
+ "label2id": {
193
+ "LABEL_0": 0,
194
+ "LABEL_1": 1,
195
+ "LABEL_10": 10,
196
+ "LABEL_100": 100,
197
+ "LABEL_101": 101,
198
+ "LABEL_102": 102,
199
+ "LABEL_103": 103,
200
+ "LABEL_104": 104,
201
+ "LABEL_105": 105,
202
+ "LABEL_106": 106,
203
+ "LABEL_107": 107,
204
+ "LABEL_108": 108,
205
+ "LABEL_109": 109,
206
+ "LABEL_11": 11,
207
+ "LABEL_110": 110,
208
+ "LABEL_111": 111,
209
+ "LABEL_112": 112,
210
+ "LABEL_113": 113,
211
+ "LABEL_114": 114,
212
+ "LABEL_115": 115,
213
+ "LABEL_116": 116,
214
+ "LABEL_117": 117,
215
+ "LABEL_118": 118,
216
+ "LABEL_119": 119,
217
+ "LABEL_12": 12,
218
+ "LABEL_120": 120,
219
+ "LABEL_121": 121,
220
+ "LABEL_122": 122,
221
+ "LABEL_123": 123,
222
+ "LABEL_124": 124,
223
+ "LABEL_125": 125,
224
+ "LABEL_126": 126,
225
+ "LABEL_127": 127,
226
+ "LABEL_128": 128,
227
+ "LABEL_129": 129,
228
+ "LABEL_13": 13,
229
+ "LABEL_130": 130,
230
+ "LABEL_131": 131,
231
+ "LABEL_132": 132,
232
+ "LABEL_133": 133,
233
+ "LABEL_134": 134,
234
+ "LABEL_135": 135,
235
+ "LABEL_136": 136,
236
+ "LABEL_137": 137,
237
+ "LABEL_138": 138,
238
+ "LABEL_139": 139,
239
+ "LABEL_14": 14,
240
+ "LABEL_140": 140,
241
+ "LABEL_141": 141,
242
+ "LABEL_142": 142,
243
+ "LABEL_143": 143,
244
+ "LABEL_144": 144,
245
+ "LABEL_145": 145,
246
+ "LABEL_146": 146,
247
+ "LABEL_147": 147,
248
+ "LABEL_148": 148,
249
+ "LABEL_149": 149,
250
+ "LABEL_15": 15,
251
+ "LABEL_150": 150,
252
+ "LABEL_151": 151,
253
+ "LABEL_152": 152,
254
+ "LABEL_153": 153,
255
+ "LABEL_154": 154,
256
+ "LABEL_155": 155,
257
+ "LABEL_156": 156,
258
+ "LABEL_157": 157,
259
+ "LABEL_158": 158,
260
+ "LABEL_159": 159,
261
+ "LABEL_16": 16,
262
+ "LABEL_160": 160,
263
+ "LABEL_161": 161,
264
+ "LABEL_162": 162,
265
+ "LABEL_163": 163,
266
+ "LABEL_164": 164,
267
+ "LABEL_165": 165,
268
+ "LABEL_166": 166,
269
+ "LABEL_167": 167,
270
+ "LABEL_168": 168,
271
+ "LABEL_169": 169,
272
+ "LABEL_17": 17,
273
+ "LABEL_170": 170,
274
+ "LABEL_171": 171,
275
+ "LABEL_172": 172,
276
+ "LABEL_173": 173,
277
+ "LABEL_18": 18,
278
+ "LABEL_19": 19,
279
+ "LABEL_2": 2,
280
+ "LABEL_20": 20,
281
+ "LABEL_21": 21,
282
+ "LABEL_22": 22,
283
+ "LABEL_23": 23,
284
+ "LABEL_24": 24,
285
+ "LABEL_25": 25,
286
+ "LABEL_26": 26,
287
+ "LABEL_27": 27,
288
+ "LABEL_28": 28,
289
+ "LABEL_29": 29,
290
+ "LABEL_3": 3,
291
+ "LABEL_30": 30,
292
+ "LABEL_31": 31,
293
+ "LABEL_32": 32,
294
+ "LABEL_33": 33,
295
+ "LABEL_34": 34,
296
+ "LABEL_35": 35,
297
+ "LABEL_36": 36,
298
+ "LABEL_37": 37,
299
+ "LABEL_38": 38,
300
+ "LABEL_39": 39,
301
+ "LABEL_4": 4,
302
+ "LABEL_40": 40,
303
+ "LABEL_41": 41,
304
+ "LABEL_42": 42,
305
+ "LABEL_43": 43,
306
+ "LABEL_44": 44,
307
+ "LABEL_45": 45,
308
+ "LABEL_46": 46,
309
+ "LABEL_47": 47,
310
+ "LABEL_48": 48,
311
+ "LABEL_49": 49,
312
+ "LABEL_5": 5,
313
+ "LABEL_50": 50,
314
+ "LABEL_51": 51,
315
+ "LABEL_52": 52,
316
+ "LABEL_53": 53,
317
+ "LABEL_54": 54,
318
+ "LABEL_55": 55,
319
+ "LABEL_56": 56,
320
+ "LABEL_57": 57,
321
+ "LABEL_58": 58,
322
+ "LABEL_59": 59,
323
+ "LABEL_6": 6,
324
+ "LABEL_60": 60,
325
+ "LABEL_61": 61,
326
+ "LABEL_62": 62,
327
+ "LABEL_63": 63,
328
+ "LABEL_64": 64,
329
+ "LABEL_65": 65,
330
+ "LABEL_66": 66,
331
+ "LABEL_67": 67,
332
+ "LABEL_68": 68,
333
+ "LABEL_69": 69,
334
+ "LABEL_7": 7,
335
+ "LABEL_70": 70,
336
+ "LABEL_71": 71,
337
+ "LABEL_72": 72,
338
+ "LABEL_73": 73,
339
+ "LABEL_74": 74,
340
+ "LABEL_75": 75,
341
+ "LABEL_76": 76,
342
+ "LABEL_77": 77,
343
+ "LABEL_78": 78,
344
+ "LABEL_79": 79,
345
+ "LABEL_8": 8,
346
+ "LABEL_80": 80,
347
+ "LABEL_81": 81,
348
+ "LABEL_82": 82,
349
+ "LABEL_83": 83,
350
+ "LABEL_84": 84,
351
+ "LABEL_85": 85,
352
+ "LABEL_86": 86,
353
+ "LABEL_87": 87,
354
+ "LABEL_88": 88,
355
+ "LABEL_89": 89,
356
+ "LABEL_9": 9,
357
+ "LABEL_90": 90,
358
+ "LABEL_91": 91,
359
+ "LABEL_92": 92,
360
+ "LABEL_93": 93,
361
+ "LABEL_94": 94,
362
+ "LABEL_95": 95,
363
+ "LABEL_96": 96,
364
+ "LABEL_97": 97,
365
+ "LABEL_98": 98,
366
+ "LABEL_99": 99
367
+ },
368
+ "layer_norm_eps": 1e-06,
369
+ "mlp_ratio": 4.363636363636363,
370
+ "model_type": "vjepa2",
371
+ "num_attention_heads": 22,
372
+ "num_hidden_layers": 40,
373
+ "num_pooler_layers": 3,
374
+ "patch_size": 16,
375
+ "pred_hidden_size": 384,
376
+ "pred_mlp_ratio": 4.0,
377
+ "pred_num_attention_heads": 12,
378
+ "pred_num_hidden_layers": 12,
379
+ "pred_num_mask_tokens": 10,
380
+ "pred_zero_init_mask_tokens": true,
381
+ "qkv_bias": true,
382
+ "torch_dtype": "float32",
383
+ "transformers_version": "4.53.0.dev0",
384
+ "tubelet_size": 2,
385
+ "wide_SiLU": true
386
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fb15cc4a23907575ec2205e20e0b6997692a9ade3a12afdaf8a2f04f9ce5a72b
3
+ size 4512302192
notebook_finetuning.ipynb ADDED
The diff for this file is too large to render. See raw diff
 
video_preprocessor_config.json ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_valid_kwargs_names": [
3
+ "do_convert_rgb",
4
+ "do_resize",
5
+ "size",
6
+ "size_divisor",
7
+ "default_to_square",
8
+ "resample",
9
+ "do_rescale",
10
+ "rescale_factor",
11
+ "do_normalize",
12
+ "image_mean",
13
+ "image_std",
14
+ "do_pad",
15
+ "do_center_crop",
16
+ "crop_size",
17
+ "data_format",
18
+ "input_data_format",
19
+ "device"
20
+ ],
21
+ "crop_size": {
22
+ "height": 384,
23
+ "width": 384
24
+ },
25
+ "data_format": "channels_first",
26
+ "default_to_square": true,
27
+ "device": null,
28
+ "do_center_crop": true,
29
+ "do_convert_rgb": null,
30
+ "do_normalize": true,
31
+ "do_pad": null,
32
+ "do_rescale": true,
33
+ "do_resize": true,
34
+ "do_sample_frames": null,
35
+ "fps": null,
36
+ "image_mean": [
37
+ 0.485,
38
+ 0.456,
39
+ 0.406
40
+ ],
41
+ "image_std": [
42
+ 0.229,
43
+ 0.224,
44
+ 0.225
45
+ ],
46
+ "input_data_format": null,
47
+ "model_valid_processing_keys": [
48
+ "do_convert_rgb",
49
+ "do_resize",
50
+ "size",
51
+ "size_divisor",
52
+ "default_to_square",
53
+ "resample",
54
+ "do_rescale",
55
+ "rescale_factor",
56
+ "do_normalize",
57
+ "image_mean",
58
+ "image_std",
59
+ "do_pad",
60
+ "do_center_crop",
61
+ "crop_size",
62
+ "data_format",
63
+ "input_data_format",
64
+ "device",
65
+ "do_sample_frames",
66
+ "video_metadata",
67
+ "fps",
68
+ "num_frames"
69
+ ],
70
+ "num_frames": null,
71
+ "resample": 2,
72
+ "rescale_factor": 0.00392156862745098,
73
+ "size": {
74
+ "shortest_edge": 438
75
+ },
76
+ "size_divisor": null,
77
+ "video_metadata": null,
78
+ "video_processor_type": "VJEPA2VideoProcessor"
79
+ }