| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - llama |
| - text-generation |
| - micro |
| - nano |
| - small |
| - base-model |
| - banana-mind |
| pipeline_tag: text-generation |
| library_name: transformers |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| --- |
| |
| # Vantora Micro |
|
|
| A **9,800-parameter** pure Llama-style causal language model, trained on a |
| 100M-token slice of FineWeb-Edu. This is the "pure transformer" baseline in a |
| head-to-head comparison against a hybrid Mamba-2 + attention model of the same |
| size. |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| |---|---| |
| | Architecture | LlamaForCausalLM (pure transformer) | |
| | Parameters | 9,800 | |
| | Vocab size | 1024 (ByteLevel BPE) | |
| | Hidden size (d_model) | 8 | |
| | Intermediate size | 22 (MLP ratio 2.77) | |
| | Hidden layers | 2 | |
| | Attention heads | 1 | |
| | Head dim | 8 | |
| | Context length | 512 | |
| | RoPE theta | 10000.0 | |
| | RMSNorm eps | 1e-6 | |
| | Tied embeddings | Yes | |
| | Dtype | float32 | |
| |
| ## Training |
| |
| - **Data:** first 100M tokens of `HuggingFaceFW/fineweb-edu` `sample-10BT` |
| - **Epochs:** 1 (100M total tokens seen) |
| - **Batch:** 128 × seq 256 (3,051 steps) |
| - **Optimizer:** AdamW, lr 5e-3, cosine schedule + 15% warmup |
| - **Grad clip:** 1.0, seed 42 |
| - **Hardware:** NVIDIA GTX 750 (Maxwell, 4 GB VRAM) |
| - **Time:** ~4.3 minutes |
| |
| ## Benchmark: BananaMind Base Bench 1.1 |
| |
| Evaluated with the **official** BananaMind `benchmark.py` runner |
| (`official_complete_run: true`, exact 350-item split, SHA-256 verified). |
| |
| | Metric | Value | |
| |---|---| |
| | **Overall Elo** | **810** | |
| | **Accuracy** | 26.00% (91/350) | |
| | Weighted accuracy | 25.48% | |
| |
| | Category | Elo | Accuracy | |
| |---|---|---| |
| | Language Completion | 919 | 52.0% | |
| | Commonsense | 658 | 16.0% | |
| | World Knowledge | 702 | 20.0% | |
| | Context Tracking | 665 | 12.0% | |
| | Quantitative | 875 | 26.0% | |
| | Logical Reasoning | 839 | 22.0% | |
| | Code Completion | 982 | 34.0% | |
| |
| ### ⚠️ Length-bias caveat on Code Completion |
| |
| The Code Completion score (Elo 982, 34%) is **not evidence the model can |
| code**. It is a benchmark artifact: |
| |
| - In the `code_completion` category, the **correct answer is the longest |
| continuation 68% of the time** (vs 12-34% in every other category). |
| - This model has a **length bias**: it picks the longest continuation more |
| often than random, because its token distribution is near uniform and |
| longer sequences accumulate more probability. |
| - The two effects line up, so the length bias coincidentally matches the |
| correct answer most of the time. |
|
|
| The BananaMind README itself warns: *"Mean token log-probability reduces |
| direct continuation-length bias but does not eliminate every |
| tokenizer-dependent effect."* Treat the Code Completion Elo as a length-bias |
| artifact, not a real coding skill. |
|
|
| ### vs. Vantora-Micro-Hybrid (same size/data) |
|
|
| | | Vantora-Micro | Vantora Micro Hybrid | |
| |---|---|---| |
| | Params | 9,800 | 11,256 | |
| | **Overall Elo** | **810** | **863** | |
| | **Accuracy** | 26.00% | 30.29% | |
| | Val loss (edu) | 4.9097 | 4.8584 | |
| | **Training time** | **~4.3 min** | **~49.5 min** | |
|
|
| The hybrid edges out this model by +53 Elo and +4.3% accuracy, but most of |
| that gap comes from the length-bias artifact on Code Completion, not real |
| reasoning. On PIQA / HellaSwag / ARC-Easy the two are within noise (0.5-2%). |
|
|
| ### Why this model is the practical choice |
|
|
| For a 10K-param model on a 100M-token slice of web text, the pure transformer |
| is the better tradeoff: |
|
|
| - **11.5× faster to train** (4.3 min vs 49.5 min on the same GTX 750). |
| - Within noise of the hybrid on every benchmark that measures real ability. |
| - No custom architecture, no `trust_remote_code`, loads with stock |
| `AutoModelForCausalLM`. |
|
|
| The hybrid's SSM sequence memory was a clear win on TinyStories (where |
| narrative memory mattered), but on this benchmark the extra training time |
| buys almost nothing. This model gets the same result in a fraction of the |
| time. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained("VantoraLabs/Vantora-Micro") |
| tokenizer = AutoTokenizer.from_pretrained("VantoraLabs/Vantora-Micro") |
| |
| prompt = "Once upon a time" |
| inputs = tokenizer(prompt, return_tensors="pt") |
| outputs = model.generate(**inputs, max_new_tokens=50) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Files |
|
|
| ``` |
| config.json # LlamaForCausalLM config |
| model.safetensors # 9,800-param weights |
| tokenizer.json # ByteLevel BPE (1024 vocab) |
| tokenizer_config.json # tokenizer settings |
| special_tokens_map.json # special token mapping |
| generation_config.json # generation defaults |
| ``` |
|
|
| ## Notes |
|
|
| This is an extremely small model — it is a research artifact for studying |
| scaling laws and architecture comparisons at the sub-10K parameter scale, not |
| a production language model. Its BananaMind score (Elo 810) is near the |
| four-choice random baseline (25%), as expected for a model this size. |
|
|
