VantoraLabs

We build tiny language models on hardware most people would call a potato.

The lab bench

Everything here is trained and evaluated on a single desktop:

• GPU: NVIDIA GTX 750 (Maxwell, 4 GB VRAM, 512 cores, no tensor cores)
• CPU: AMD Ryzen 5 3500X
• RAM: 16 GB

Just one machine that has been running experiments for months and refuses to quit.

Why tiny models

Big models are expensive to train and expensive to run. Tiny models are cheap enough to train a dozen of them in an afternoon, which means we can actually test ideas instead of just talking about them.

The GTX 750 forces a specific kind of discipline. Every parameter has to earn its place. Every architecture choice has to justify itself against a baseline. You cannot hide behind scale, so you have to be honest about what actually works.

What we study

• Scaling laws at the extreme low end. What happens to the tokens-per-parameter ratio when a model has only a few thousand parameters? Where is the knee? When does repetition stop helping and start hurting?
• Architecture comparisons at fixed size. Pure transformer versus hybrid Mamba-2 + attention, same parameter count, same data, same token budget. The hybrid wins on BananaMind by a real margin, and we want to know exactly why.
• Optimizers that matter at small scale. Muon versus AdamW, and how much of the gap is the optimizer versus the architecture.
• What a 10K-parameter model can actually learn. The answer is more than you would think, and less than you would hope.

The models

Vantora-Micro

A 9,800-parameter pure Llama-style transformer. The baseline. Trained on 100M tokens of FineWeb-Edu. BananaMind Elo 810.

Vantora-Micro-Hybrid

An 11,256-parameter hybrid that pairs a Mamba-2 SSM with attention. Same data, same budget, same hardware. BananaMind Elo 863, which beats the pure transformer by 53 points and wins six of seven categories.

These two are the first. They are not the last.

The plan

The GTX 750 is slow, so we think in terms of what fits in a night. A 10K model over 100M tokens takes about five minutes. A 20K model over 900M tokens takes about an hour. That is enough to run real sweeps, real ablations, and real head-to-heads.

Up next:

• Larger slices of the dataset, one full epoch at a time
• More architecture variants at the same parameter budget
• Muon-only versus AdamW-only ablations to separate optimizer from architecture
• Whatever breaks the current best score on the leaderboard

The rules

• Every model gets a real benchmark run, not a hand-waved number.
• Every claim gets a baseline to compare against.
• If it does not fit in VRAM, it does not get built.

Contact

Open an issue on any model repo, or reach out through the Hugging Face organization. We read everything.

VantoraLabs. Small models, honest numbers, one very patient GPU.