Instructions to use VantoraLabs/Vantora-Micro-Hybrid with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use VantoraLabs/Vantora-Micro-Hybrid with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="VantoraLabs/Vantora-Micro-Hybrid", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("VantoraLabs/Vantora-Micro-Hybrid", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use VantoraLabs/Vantora-Micro-Hybrid with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "VantoraLabs/Vantora-Micro-Hybrid"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "VantoraLabs/Vantora-Micro-Hybrid",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/VantoraLabs/Vantora-Micro-Hybrid

SGLang

How to use VantoraLabs/Vantora-Micro-Hybrid with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "VantoraLabs/Vantora-Micro-Hybrid" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "VantoraLabs/Vantora-Micro-Hybrid",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "VantoraLabs/Vantora-Micro-Hybrid" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "VantoraLabs/Vantora-Micro-Hybrid",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use VantoraLabs/Vantora-Micro-Hybrid with Docker Model Runner:
```
docker model run hf.co/VantoraLabs/Vantora-Micro-Hybrid
```

Vantora Micro Hybrid

A 11,256-parameter hybrid language model combining a Mamba-2 SSM with attention (Falcon-H1 "SA_M" pattern), trained on a 100M-token slice of FineWeb-Edu. This is the hybrid counterpart to Vantora-Micro — the same size, data, and token budget, but with the SSM's linear-time sequence memory added.

Model Details

Property	Value
Architecture	HybridLM (Mamba-2 SSM + attention, Falcon-H1 SA_M)
Parameters	11,256
Vocab size	1024 (ByteLevel BPE)
Hidden size (d_model)	8
SSM expand	2 (d_inner = 16)
SSM state (d_state)	4
SSM conv (d_conv)	4
MLP ratio	2.77 (hidden = 22)
Hidden layers	2 (block sharing)
Attention heads	1
Meta tokens	2 (Hymba-style)
Context length	256
RoPE theta	500000.0
Tied embeddings	Yes
Dtype	float32

Block structure (Falcon-H1 "SA_M")

r' = r + F_attn(Norm(r)) + F_ssm(Norm(r))    # parallel attention + SSM
r  = r' + F_mlp(Norm(r'))                     # sequential SwiGLU MLP

At small scale, attention heads are too narrow to compress sequence information. The Mamba-2 SSM provides linear-time sequence memory that carries context forward across tokens — something attention at d_head=8 cannot do well.

Training

Data: first 100M tokens of HuggingFaceFW/fineweb-edu sample-10BT
Epochs: 1 (100M total tokens seen)
Batch: 128 × seq 256 (3,051 steps)
Optimizer: Muon (2D weights) + AdamW (1D/embed), lr 5e-3
LR schedule: WSD (warmup-stable-decay)
EMA: enabled (decay 0.999)
Grad clip: 0.5, seed 42
Hardware: NVIDIA GTX 750 (Maxwell, 4 GB VRAM)
Time: ~49.5 minutes

Benchmark: BananaMind Base Bench 1.1

Evaluated with the official BananaMind benchmark.py runner (official_complete_run: true, exact 350-item split, SHA-256 verified).

Metric	Value
Overall Elo	863
Accuracy	30.29% (106/350)
Weighted accuracy	31.22%

Category	Elo	Accuracy
Language Completion	807	36.0%
Commonsense	850	34.0%
World Knowledge	767	26.0%
Context Tracking	733	18.0%
Quantitative	918	30.0%
Logical Reasoning	944	30.0%
Code Completion	1014	38.0%

⚠️ Length-bias caveat on Code Completion

The high Code Completion score (Elo 1014, 38%) is not evidence the model can code. It is a benchmark artifact:

In the code_completion category, the correct answer is the longest continuation 68% of the time (vs 12-34% in every other category).
This model has a length bias: it picks the longest continuation 50% of the time on code (vs 25% random), because its token distribution is near uniform and longer sequences accumulate more probability.
The two effects line up, so the length bias coincidentally matches the correct answer most of the time.

The BananaMind README itself warns: "Mean token log-probability reduces direct continuation-length bias but does not eliminate every tokenizer-dependent effect." Treat the Code Completion Elo as a length-bias artifact, not a real coding skill.

vs. Vantora-Micro (pure transformer, same size/data)

	Vantora-Micro	Vantora Micro Hybrid
Params	9,800	11,256
Overall Elo	810	863
Accuracy	26.00%	30.29%
Val loss (edu)	4.9097	4.8584
Training time	~4.3 min	~49.5 min

The hybrid beats the pure transformer by +53 Elo and +4.3% accuracy, winning 6 of 7 categories. The biggest edges are in Commonsense (+192 Elo), Logical Reasoning (+105), and Context Tracking (+68) — the categories that need sequence memory and reasoning across context, which the Mamba-2 SSM provides.

Honest cost/benefit

The hybrid's edge is real but small, and it costs a lot to get:

11.5× slower to train (49.5 min vs 4.3 min on the same GTX 750) because the Mamba-2 selective scan is a Python loop over timesteps.
+53 Elo and +4.3% accuracy on BananaMind, but most of that comes from the length-bias artifact on Code Completion, not real reasoning.
On PIQA / HellaSwag / ARC-Easy, the hybrid beats the pure transformer by only 0.5-2% — within the noise floor at this scale.

For this task (a 10K-param model on a 100M-token slice of web text), the hybrid pipeline is not worth it. The pure transformer trains 11.5× faster and gets within noise of the same result. The hybrid architecture was a clear win on TinyStories (where the SSM's narrative memory mattered), but on this benchmark the extra training time buys almost nothing.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "VantoraLabs/Vantora-Micro-Hybrid", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("VantoraLabs/Vantora-Micro-Hybrid")

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Note: this model uses a custom architecture, so trust_remote_code=True is required. The modeling_hybridlm.py file is self-contained (no external dependencies).

Files

config.json              # HybridLMConfig (with auto_map)
modeling_hybridlm.py     # self-contained custom architecture (trust_remote_code)
model.safetensors        # 11,256-param weights
tokenizer.json          # ByteLevel BPE (1024 vocab)
tokenizer_config.json   # tokenizer settings
special_tokens_map.json # special token mapping

Notes

This is an extremely small model — a research artifact for studying scaling laws and architecture comparisons at the sub-10K parameter scale, not a production language model. Its BananaMind score (Elo 863) is modest but meaningfully above the four-choice random baseline (25%), and it demonstrates that the hybrid SSM+attention architecture generalizes better than a pure transformer at the same size and token budget.

Downloads last month: 32

Safetensors

Model size

21.5k params

Tensor type

F32

VantoraLabs
/

Vantora-Micro-Hybrid