|
|
--- |
|
|
language: |
|
|
- en |
|
|
license: apache-2.0 |
|
|
tags: |
|
|
- vision-language |
|
|
- multimodal |
|
|
- robotics |
|
|
- edge-deployment |
|
|
- tiny-vlm |
|
|
- repvit |
|
|
- tinyllm |
|
|
- stage2 |
|
|
base_model: |
|
|
- tinyllm |
|
|
library_name: transformers |
|
|
pipeline_tag: image-text-to-text |
|
|
--- |
|
|
|
|
|
# EmberVLM: Tiny (~35M parameters) |
|
|
|
|
|
**🔥 Efficient Vision-Language Model for Edge Deployment & Robotic Applications** |
|
|
|
|
|
This model is currently in training - **STAGE2 (Epoch 1)**. |
|
|
|
|
|
## 📊 Current Training Status |
|
|
|
|
|
- **Stage**: Multimodal Instruction Tuning - Following complex instructions |
|
|
- **Epoch**: 1 |
|
|
- **Last Updated**: 2026-02-01 16:01:18 UTC |
|
|
|
|
|
### Latest Metrics |
|
|
- **instruction_loss**: 0.0000 |
|
|
- **loss**: 5.2714 |
|
|
|
|
|
## 🏗️ Model Architecture |
|
|
|
|
|
- **Size**: Tiny (~35M parameters) |
|
|
- **Total Parameters**: 40,196,257 |
|
|
- **Trainable Parameters**: 26,212,929 (65.2%) |
|
|
- **Vision Encoder**: RepViT-M0.9 (~5M params) |
|
|
- **Language Model**: TinyLLM-30M (30M params) |
|
|
|
|
|
## 🎯 Training Curriculum |
|
|
|
|
|
EmberVLM follows a 4-stage training curriculum: |
|
|
|
|
|
1. ✅ **Stage 1: Visual-Language Alignment** - Grounding vision and language |
|
|
2. ✅ **Stage 2: Multimodal Instruction Tuning** - Following instructions |
|
|
3. ✅ **Stage 3: Robot Fleet Selection** - Task-robot matching |
|
|
4. ⏳ **Stage 4: Chain-of-Thought Reasoning** - Reasoning generation |
|
|
|
|
|
**Current Stage**: STAGE2 |
|
|
|
|
|
## 💻 Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer |
|
|
from embervlm import EmberVLM |
|
|
from PIL import Image |
|
|
|
|
|
# Load model and tokenizer |
|
|
model = EmberVLM.from_pretrained("euhidaman/embervlm-tiny") |
|
|
tokenizer = AutoTokenizer.from_pretrained("euhidaman/embervlm-tiny") |
|
|
|
|
|
# Load image |
|
|
image = Image.open("scene.jpg") |
|
|
|
|
|
# Generate response |
|
|
prompt = "<image>Describe what you see and select the best robot for this task." |
|
|
outputs = model.generate( |
|
|
image=image, |
|
|
prompt=prompt, |
|
|
tokenizer=tokenizer, |
|
|
max_new_tokens=256 |
|
|
) |
|
|
|
|
|
print(outputs) |
|
|
``` |
|
|
|
|
|
## 🎓 Training Details |
|
|
|
|
|
- **Vision Backbone**: repvit |
|
|
- **Language Backbone**: tinyllm |
|
|
- **Optimization**: AdamW with cosine learning rate schedule |
|
|
- **Mixed Precision**: bfloat16 |
|
|
- **Distributed Training**: Multi-GPU with DDP |
|
|
- **Class Balancing**: Focal loss for robot selection (Stage 3) |
|
|
- **Reasoning**: Chain-of-thought with reinforcement learning (Stage 4) |
|
|
|
|
|
## 🌍 Environmental Impact |
|
|
|
|
|
This model is designed for edge deployment to minimize energy consumption. |
|
|
|
|
|
## 🎯 Intended Use |
|
|
|
|
|
- **Primary**: Edge deployment on resource-constrained devices |
|
|
- **Applications**: |
|
|
- Robotic vision-language understanding |
|
|
- Real-time multimodal reasoning |
|
|
- Robot fleet selection and task planning |
|
|
- Mobile/embedded AI systems |
|
|
|
|
|
## ⚠️ Limitations |
|
|
|
|
|
- Model is still in training - performance will improve as training progresses |
|
|
- Optimized for efficiency over maximum accuracy |
|
|
- Best suited for edge/mobile deployment scenarios |
|
|
- Training focused on robot-centric scenarios |
|
|
|
|
|
## 📚 Citation |
|
|
|
|
|
```bibtex |
|
|
@software{embervlm_2026, |
|
|
title = {EmberVLM: Efficient Vision-Language Model for Edge Deployment}, |
|
|
author = {EmberVLM Team}, |
|
|
year = {2026}, |
|
|
url = {https://huggingface.co/euhidaman/embervlm-tiny} |
|
|
} |
|
|
``` |
|
|
|
|
|
## 📝 License |
|
|
|
|
|
Apache 2.0 |
|
|
|
|
|
--- |
|
|
|
|
|
**Note**: This is a checkpoint from stage2 training (epoch 1). |
|
|
The model will be updated after each epoch with improved performance. |
|
|
|
