Improve model card for Don't Blind Your VLA: Aligning Visual Representations for OOD Generalization

This PR significantly enhances the model card for the `Don't Blind Your VLA` LoRA adapter.

Key improvements include:
* Adding `robotics` as the `pipeline_tag` to correctly categorize the model for its domain.
* Specifying the `apache-2.0` license.
* Populating the model card content with detailed descriptions from the paper abstract and GitHub README.
* Adding direct links to the Hugging Face paper page, the project page, the GitHub repository, and the Hugging Face collection.
* Including the installation instructions and relevant code snippets from the GitHub README for Visual Representation Alignment integration and LoRA fine-tuning.
* Adding the academic citation and acknowledgements.
* Adding the primary image from the GitHub README for visual context.

These updates make the model card more comprehensive, discoverable, and user-friendly for the community.

README.md

@@ -1,202 +1,193 @@
 ---
 base_model: openvla/openvla-7b
 library_name: peft
+pipeline_tag: robotics
+license: apache-2.0
 ---
 
-# Model Card for Model ID
+# Don't Blind Your VLA: Aligning Visual Representations for OOD Generalization
 
-<!-- Provide a quick summary of what the model is/does. -->
+This model is a LoRA adapter presented in the paper [Don't Blind Your VLA: Aligning Visual Representations for OOD Generalization](https://huggingface.co/papers/2510.25616).
 
+To address the degradation of visual-language (VL) representations during VLA supervised fine-tuning (SFT), we introduce **Visual Representation Alignment**. During SFT, we pull a VLA’s visual tokens toward a frozen teacher’s patch features using cosine similarity through a lightweight frozen projector. This keeps perception anchored while the model learns to act — improving OOD generalization with almost no added cost.
 
+<img width="5567" height="4133" alt="method_1" src="https://huggingface.co/CognitiveAISystems/BlindVLA/resolve/main/figs/method_1.png" />
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
+The growing success of Vision-Language-Action (VLA) models stems from the promise that pretrained Vision-Language Models (VLMs) can endow agents with transferable world knowledge and vision-language (VL) grounding, laying a foundation for action models with broader generalization. Yet when these VLMs are adapted to the action modality, it remains unclear to what extent their original VL representations and knowledge are preserved. This work systematically studies representation retention during VLA fine-tuning, showing that naive action fine-tuning leads to degradation of visual representations. To characterize and measure these effects, the authors probe VLA's hidden representations and analyze attention maps, and design a set of targeted tasks and methods that contrast VLA models with their counterpart VLMs, isolating changes in VL capabilities induced by action fine-tuning. They introduce a simple yet effective method, **Visual Representation Alignment**, that mitigates degradation and yields improved generalization to out-of-distribution (OOD) scenarios.
 
+The paper also introduces the **VL-Think Task Suite**, a diagnostic suite assessing the transfer of VL understanding and knowledge from VLMs to VLAs independently of low-level control. This suite focuses on whether models retain the ability to interpret visual symbols, compositional cues, and categorical relations rather than pure manipulation skills.
 
+-   **Developed by:** Nikita Kachaev, Mikhail Kolosov, Daniil Zelezetsky, Alexey K. Kovalev, Aleksandr I. Panov
+-   **Model type:** Vision-Language-Action (VLA) model (LoRA adapter)
+-   **Language(s):** English
+-   **License:** Apache-2.0
+-   **Finetuned from model:** [openvla/openvla-7b](https://huggingface.co/openvla/openvla-7b)
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+### Model Sources
 
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+-   **Repository:** [https://github.com/CognitiveAISystems/BlindVLA](https://github.com/CognitiveAISystems/BlindVLA)
+-   **Paper:** [https://huggingface.co/papers/2510.25616](https://huggingface.co/papers/2510.25616)
+-   **Project Page:** [https://blind-vla-paper.github.io/](https://blind-vla-paper.github.io/)
+-   **Hugging Face Collection:** [https://huggingface.co/collections/tttonyalpha/dont-blind-your-vla](https://huggingface.co/collections/tttonyalpha/dont-blind-your-vla)
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
+This model is intended for research in Vision-Language-Action (VLA) models, particularly for understanding and improving out-of-distribution (OOD) generalization in robotic and agent control tasks through visual representation alignment. Researchers can use this adapter and methodology to fine-tune base VLA models and explore the impact of representation degradation.
 
-### Downstream Use [optional]
+### Out-of-Scope Use
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+As a research artifact, this model is not intended for deployment in real-world, safety-critical applications without further rigorous testing, validation, and adaptation. It is focused on studying and mitigating specific representation issues in VLAs, rather than serving as a production-ready agent.
 
-[More Information Needed]
+## How to Get Started with the Model
 
-### Out-of-Scope Use
+### Installation
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+Use the environment setup commands below to get started:
 
-[More Information Needed]
+```bash
+# Create and activate conda environment
+conda create -n blindvla python=3.10 -y
+conda activate blindvla
 
-## Bias, Risks, and Limitations
+# Install PyTorch. Below is a sample command to do this, but you should check the following link
+# to find installation instructions that are specific to your compute platform:
+# https://pytorch.org/get-started/locally/
+pip install torch torchvision torchaudio
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+# Clone and install the BlindVLA repo
+git clone https://github.com/CognitiveAISystems/BlindVLA.git
+cd BlindVLA
+pip install -e ./openvla
 
-[More Information Needed]
+# Install Flash Attention 2 for training (https://github.com/Dao-AILab/flash-attention)
+#   =>> If you run into difficulty, try `pip cache remove flash_attn` first
+pip3 install packaging ninja
+ninja --version; echo $?  # Verify Ninja --> should return exit code "0"
+pip install "flash-attn==2.5.5" --no-build-isolation
+pip install diffusers==0.33.0
 
-### Recommendations
+pip install -e ./ManiSkill
+pip install -e ./SimplerEnv
+pip install -U "typeguard>=3"
+```
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+The pretrained OpenVLA model is warmed up using 140 episodes collected with Octo-Small and a motion planner for 2k steps.
+You can download the training dataset (1.4k episodes) [here](https://huggingface.co/datasets/tttonyalpha/openvla_1k-dataset) and the warm-up checkpoint [here](https://huggingface.co/tttonyalpha/openvla-7b-warmup-checkpoint_lora_002000).
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+### Sample Usage (LoRA Fine-tuning with Visual Representation Alignment)
 
-## How to Get Started with the Model
+Below is a minimal example from the GitHub README of how you can integrate Visual Representation Alignment into your VLA’s training pipeline. Just plug in these few lines right after your forward pass — no architecture changes are needed.
 
-Use the code below to get started with the model.
+```python
+# ....
+# out = vla.forward(..., output_hidden_states=True)
+# pixel_values = preprocessor(image, ...)
+# ....
+#
 
-[More Information Needed]
+n_vis = out.projector_features.shape[1]
+pos, pos_end = 1,
 
-## Training Details
+# 1. Extract VLA's visual features from specific layer and project to visual teacher dimention
+vla_features = out.hidden_states[align_layer][:, pos:pos_end]
+vla_features = alignment_projector(vla_feats)
 
-### Training Data
+# 2. Get teacher patch features
+with torch.no_grad():
+    teacher_features = teacher_vision_backbone(pixel_values)
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+# 3. Compute cosine alignment loss
+emb_t = F.normalize(teacher_features, dim=-1)
+emb_s = F.normalize(vla_features, dim=-1)
 
-[More Information Needed]
+cossim = (emb_t * emb_s).sum(dim=-1)
+align_loss = (-cossim).mean()
 
-### Training Procedure
+loss += cfg.align_coeff * align_loss
+```
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+You can run LoRA fine-tuning with Visual Representation Alignment using this script:
 
-#### Preprocessing [optional]
+```bash
+openvla_path="tttonyalpha/openvla-7b-warmup-checkpoint_merged_002000_lora_002000"
 
-[More Information Needed]
+torchrun --standalone --nnodes 1 --nproc-per-node 1 vla-scripts/finetune.py \
+  --vla_path "$openvla_path" \
+  --data_root_dir "datasets" \
+  --dataset_name "sft" \
+  --run_root_dir "runs" \
+  --lora_rank 32 \
+  --batch_size 8 \
+  --max_steps 60000 \
+  --eval_steps 200 \
+  --save_steps "0,5000,10000,20000,30000,40000,50000,60000" \
+  --grad_accumulation_steps 1 \
+  --learning_rate 5e-4 \
+  --image_aug True
+```
 
+## Training Details
 
-#### Training Hyperparameters
+### Training Data
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+The model is warmed up using 140 episodes collected with Octo-Small and a motion planner for 2k steps. A larger training dataset (1.4k episodes) is available [here](https://huggingface.co/datasets/tttonyalpha/openvla_1k-dataset).
 
-#### Speeds, Sizes, Times [optional]
+### Training Procedure
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+#### Training Hyperparameters
 
-[More Information Needed]
+The model is fine-tuned using LoRA with the following hyperparameters, as described in the provided `finetune.py` script:
+-   **LoRA rank:** 32
+-   **Batch size:** 8
+-   **Max steps:** 60000
+-   **Evaluation steps:** 200
+-   **Save steps:** 0, 5000, 10000, 20000, 30000, 40000, 50000, 60000
+-   **Gradient accumulation steps:** 1
+-   **Learning rate:** 5e-4
+-   **Image augmentation:** True
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
 ### Testing Data, Factors & Metrics
 
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+The model is evaluated using the **VL-Think Task Suite**, a diagnostic suite assessing the transfer of VL understanding and knowledge from VLMs to VLAs independently of low-level control. The suite includes various tasks, focusing on the ability to interpret visual symbols, compositional cues, and categorical relations.
+Examples of tasks include:
+*   `PutOnShapeInSceneMultiColor-v1` (13 shapes)
+*   `PutOnColorInSceneMulti-v1` (8 colors)
+*   `PutOnLaundryIconInSceneMulti-v1` (17 laundry icons)
+*   `PutOnNumberInSceneParity-v1` (8 numbers)
+*   `PutOnPublicInfoSignInSceneMulti-v1` (14 public info signs)
+*   `PutOnSignTrafficInSceneMulti-v1` (24 traffic signs)
+*   `PutOnWeatherIconInSceneMulti-v1` (9 weather icons)
+*   `PutOnArrowSignInSceneMulti-v1` (4 directions)
 
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
+Evaluation is performed using batched environments for efficient parallel processing.
 
 ### Results
 
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
+The paper demonstrates that Visual Representation Alignment mitigates degradation of visual representations and yields improved generalization to out-of-distribution (OOD) scenarios. For detailed results, refer to the [paper](https://huggingface.co/papers/2510.25616).
 
-[More Information Needed]
+## Citation
 
-## Model Card Contact
+If you find our code useful, please cite [our paper](https://arxiv.org/abs/2510.25616):
 
-[More Information Needed]
-### Framework versions
+```BibTeX
+@misc{kachaev2025dontblindvlaaligning,
+      title={Don't Blind Your VLA: Aligning Visual Representations for OOD Generalization},
+      author={Nikita Kachaev and Mikhail Kolosov and Daniil Zelezetsky and Alexey K. Kovalev and Aleksandr I. Panov},
+      year={2025},
+      eprint={2510.25616},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2510.25616},
+}
+```
 
-- PEFT 0.11.1
+## Acknowledgement
+BlindVLA is built with reference on: [RL4VLA](https://github.com/gen-robot/RL4VLA), [Simpler](https://github.com/simpler-env/SimplerEnv), [REPA](https://github.com/sihyun-yu/REPA), [OpenVLA](https://github.com/openvla/openvla). Many thanks for their awesome work!