metadata
license: other
license_name: openarlow
license_link: LICENSE
language:
- en
pipeline_tag: text-generation
library_name: transformers
THIS MODEL IS NOT OFFICIAL BUT RATHER A PROOF OF CONCEPT OF THE ARLOW VISION ARCHITECTURE
Arlow Vision is the standalone vision-pretraining stage for the Arlow multimodal stack. It trains the visual tower to produce visual tokens that match the Arlow text backbone width and can later be plugged into a full vision-language model.
This model requires a specific Transformers fork because the architecture code has not been merged into official Transformers yet.
Special transformers fork: https://github.com/yuchenxie4645/transformers/tree/ArlowVL
git clone --branch ArlowVL --single-branch https://github.com/yuchenxie4645/transformers
cd transformers
pip install -e .
Training Summary
| Item | Value |
|---|---|
| Objective | Masked autoencoding over visual patch tokens |
| Modalities | Images, with optional video mixed into training |
| Output width | 3072 |
| Next stage | Multimodal alignment with the Arlow text backbone |
Model
| Item | Value |
|---|---|
| Vision encoder | ArlowVLVisionModel |
| Depth | 48 |
| Embedding dimension | 1536 |
| Hidden size | 3072 |
| Attention heads | 24 |
| Patch size | 14 |
| Temporal patch size | 2 |
| Spatial merge size | 2 |
| Activation | gelu_pytorch_tanh |
| Deformable attention | Enabled |
| Progressive patches | Enabled |
| DeepStack visual features | Enabled |
| M-ROPE | Enabled |
Data
| Item | Value |
|---|---|
| Primary modality | Images |
| Optional modality | Video |
| Default video sampling probability | 0.25 |
| Default image data | ILSVRC/imagenet-1k train split |
| Default video data | ucf101 train split |
| Recommended larger-scale direction | YFCC-style image data and OpenVid-style video data |
Optimization
| Item | Value |
|---|---|
| Hardware target | 8x RTX 8000 with 48 GB each |
| System RAM target | 200 GB |
| Precision | fp16 |
| Attention backend | sdpa |
| Distributed strategy | DeepSpeed ZeRO-2 |
| Epochs | 1 |
| Steps per epoch cap | 2621440 |
| Per-device batch size | 2 |
| Gradient accumulation | 16 |
| Effective global batch size on 8 GPUs | 256 |
| Learning rate | 1.5e-4 |
| Weight decay | 0.05 |
| Warmup steps | 40000 |
| Max grad norm | 1.0 |
MAE Objective
| Item | Value |
|---|---|
| Mask ratio | 0.75 |
| Decoder embedding size | 512 |
| Decoder depth | 8 |
| Decoder heads | 8 |
| Normalized pixel loss | Enabled |
Exported Artifacts
| Item | Value |
|---|---|
| Main artifact to keep | checkpoint-*/vision_encoder/ |
| Matching preprocessing artifacts | image_processor/, video_processor/, processor_config.json |