metadata
language: en
license: mit
tags:
- bitmar
- multimodal
- babylm
- cross-modal
datasets:
- babylm_multimodal
metrics:
- bleu
- cross_modal_similarity
BitMar 100M Token Model
This model was trained on exactly 100 million tokens for the BabyLM Challenge.
Key Features
BitNet Quantization: 1.58-bit quantized text encoder/decoder for efficient inference.
Vision Processing: Quantized Vision Transformer using pre-computed DiNOv2 features.
Episodic Memory: Human Brain inspired memory mechanism for learning, without fine-tuning.
Training Details
- Total tokens: 100,000,000
- Epochs completed: 10
- Tokens processed: 996,862,184
- Cross-modal similarity: 0.4552
Model Architecture
- Text encoder: 4 layers, 128 hidden size
- Vision encoder: DiNOv2 features compressed to 128
- Episodic memory: 32 slots
Usage
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("euhidaman/bitmar-attention-multimodal")
tokenizer = AutoTokenizer.from_pretrained("euhidaman/bitmar-attention-multimodal")
Training Status
- Status: Completed
- Tokens Processed: 996,862,184
- Best Cross-modal Similarity: 0.4552