---
license: apache-2.0
tags:
- vision-language
- multimodal
- episodic-memory
- fiber-alignment
- qwen2
- deit
- pytorch
library_name: transformers
datasets:
- conceptual-12m
pipeline_tag: image-to-text
---

# MicroVLM-V: Vision-Language Model with FIBER Alignment & Episodic Memory

## 📋 Model Overview

MicroVLM-V is a compact vision-language model (~215 MB) that combines:
- **Vision Encoder**: DeiT-Tiny (5.7M params)
- **Language Model**: Qwen2.5-0.5B (4-bit quantized, 315M params)
- **Alignment**: FIBER fusion at layers [6, 8, 10]
- **Episodic Memory**: Larimar GPM (512 slots, 4.8M params)

**Checkpoint**: `best` (Best alignment model)

---

## 📊 Model Architecture

### Parameter Distribution

| Component | Total Parameters | Trainable | Status |
|-----------|-----------------|-----------|--------|
| **Total Model** | **334.5M** | **13.8M** | **4.1% trainable** |
| Vision Encoder | 8.8M | 3.3M | FIBER fusion trainable |
| Language Model | 315.1M | 0 | Frozen (4-bit) |
| Multimodal Adapter | 5.0M | 5.0M | Fully trainable |
| Episodic Memory | 4.8M | 4.8M | Fully trainable |

### Quantization Status

| Component | Quantization |
|-----------|-------------|
| Vision Encoder | FP16 |
| Language Model | 4-bit ✓ |
| Episodic Memory | FP32 |

**Estimated Model Size**: ~214.6 MB

---

## 🏋️ Training Details

### Configuration
- **Dataset**: CC12M (Conceptual 12M) - 3M training samples
- **Batch Size**: 512
- **Training Time**: ~0.64 hours on 2x A100 80GB
- **Throughput**: ~332 samples/sec
- **Total FLOPs**: 2088 PFLOPs

### FIBER Alignment
- **Mode**: Fusion-in-Backbone (FIBER-style)
- **Fusion Layers**: [6, 8, 10]
- **ITC Weight**: 1.0
- **ITM Weight**: 0.5
- **ITC Queue Size**: 256

### Training Metrics (Best Checkpoint)
- **Best Alignment Similarity**: 0.0249 (step 25)
- **Final ITM Loss**: ~0.53
- **Final Token Loss**: ~0.056
- **Training stopped**: Early stopping at step 1500 (alignment plateau)

---

## 💻 Usage

### Loading the Model

```python
import torch

# Load checkpoint
checkpoint = torch.load('model.pt', map_location='cpu')

# Access model state dict
model_state = checkpoint['model_state_dict']

# Get training info
print(f"Global step: {checkpoint.get('global_step', 'N/A')}")
print(f"Best alignment: {checkpoint.get('best_correct_sim', 'N/A')}")
```

### Inference Example

```python
from PIL import Image
import torchvision.transforms as transforms

# Prepare image
transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                        std=[0.229, 0.224, 0.225])
])

image = Image.open('example.jpg').convert('RGB')
image_tensor = transform(image).unsqueeze(0)

# Forward pass (after loading model)
with torch.no_grad():
    outputs = model(
        images=image_tensor,
        input_ids=tokens['input_ids'],
        attention_mask=tokens['attention_mask']
    )
```

---

## 📁 Repository Contents

- `model.pt` - Best alignment checkpoint
- `statistics.json` - Training statistics
- `config.json` - Model configuration
- `README.md` - This model card

---

## ⚙️ Requirements

```bash
pip install torch>=2.0.0
pip install transformers>=4.30.0
pip install timm  # For DeiT vision encoder
pip install bitsandbytes  # For 4-bit quantization
```

---

## 📜 License

Apache 2.0 License

---

## 🔗 Links

- **GitHub Repository**: [euhidaman/MicroVLM-V](https://github.com/euhidaman/MicroVLM-V)
- **Branch**: FocusedAttention

---

## ⚠️ Limitations

- This is the **Stage 1 alignment checkpoint** - focuses on vision-language alignment
- Best for: Image-text matching, alignment tasks
- May need further fine-tuning for generation tasks

---

*Uploaded: 2025-12-08 14:53:01 UTC*