Quazim0t0/Byrne-VLM-131M

Byrne-VLM (131M)

Model will be ungated for open download once I am done with the base..

Byrne-VLM is a compact (~130.9M-param) vision-language model built entirely from the SpikeWhale / Byrne family: a from-scratch 39M vision encoder, a lightweight connector, and the ~90M Byrne language model — wired LLaVA-style and trained by distillation. It is a research / architecture artifact, not a SOTA captioner (see Honest evaluation below).

image → Byrne-VE (39.3M, frozen) → 196 patch tokens (512-d)
      → Connector MLP (512→640, 1.18M)
      → spliced at <image> placeholders into Byrne-LM (90M + Family-LoRA)
      → caption

Components

part	params	what it is
Byrne-VE (vision)	39.34M	ViT-style encoder — RMSNorm, 2D-axial RoPE, QK-Norm, SwiGLU, HRM refinement. 224px / patch16 / 196 tokens / dim512 / depth12. Distilled from DINOv2-base, then improved teacher-free with DINO-style self-distillation.
Connector	1.18M	2-layer MLP (512→1024→640) projecting patch tokens into the LM space.
Byrne-LM	~90M (+Family-LoRA)	SpikeWhale LM — MLA attention, DERF, XSA, Engram n-gram, hyper-connections, MoE, MTP, HRM refine, QK-Norm, partial RoPE. Custom SpikeTokenizer (vocab 16512), 4096 context.
Family-LoRA	5.04M	Custom adapter whose bottleneck is a full family block: HRM iterative gated refinement + MoE-SwiGLU (shared+routed experts, sqrtsoftplus routing). r=16, zero-init → exact no-op at start, preserving the LM.
Total	130.9M

How it was trained

1 · Vision encoder (Byrne-VE). Distilled from a frozen facebook/dinov2-base teacher (cosine on CLS + patch grid), then improved teacher-free with DINO-style EMA self-distillation. Reaches ~88% of DINOv2's k-NN accuracy at ~45% of its params; the HRM gate is alive (tanh ≈ 0.35).

2 · Connector grounding (10 streamed rounds). The connector was trained to map vision features into the LM, streaming image–caption pairs live from a broad mix: CC3M, CC12M, Conceptual-Captions-12M, LLaVA-ReCap-558K/118K/CC3M, TextCaps, Flickr8k, LLaVA-NeXT-Data. (Vision encoder + LM frozen throughout.)

3 · Style fine-tune. Connector fine-tuned on ~25k images captioned by HuggingFaceTB/SmolVLM-256M-Instruct — sequence-level distillation (the teacher and the SpikeTokenizer have different vocabularies, so we distill the generated caption text, not logits).

4 · Stage-2 Family-LoRA. A Family-LoRA (HRM + MoE-SwiGLU adapter) is added to the LM decoder and trained with the connector. Base LM weights stay frozen; the zero-init adapter wakes during training (gates climb off zero, MoE routing learns, no collapse).

Honest evaluation

Byrne-VLM is a demonstration of the SpikeWhale family as a VLM, not a competitive captioner. On 500 COCO-val images it scores CIDEr ≈ 0.06 / BLEU-4 ≈ 4.8 — far below usable captioners (its own teacher, SmolVLM-256M, is much stronger). It reliably gets coarse scene gist and is correct on clear subjects (e.g. lion, tower) but cannot produce specific, reference-grade captions. This is a capacity-and-scale limit (90M from-scratch LM, 39M encoder, ~25k caption pairs), not a wiring bug. Example outputs:

image	Byrne-VLM
lion	"a lion on the ground. In the background there are trees."
tower	"a tower. In the background there is a sky."
cheetah	"a tiger in the water." (right family)

Usage

import torch
from generate import load_vlm, caption
from spike_tokenizer import SpikeTokenizer

device = "cuda" if torch.cuda.is_available() else "cpu"
tok = SpikeTokenizer(vocab_file="tokenizer.json")
vlm = load_vlm("weights/byrne_vlm.pt", "lm", "weights/byrne_ve.pt", device)
print(caption(vlm, tok, "photo.jpg", device))

AnyRes tiling (higher effective resolution for dense images / documents) is supported via load_vlm(..., anyres_grid=(2,2)) — fits the 4096 context (980 image tokens) and needs a matching fine-tune.

Files

weights/byrne_vlm.pt (connector + Family-LoRA) · weights/byrne_ve.pt (vision encoder) · lm/ (Byrne base LM) · model code (vlm_model.py, modeling_byrne_embed.py, family_lora.py, anyres.py, …).

Citation

@misc{byrne_vlm_2026,
  title  = {Byrne-VLM: A Tiny From-Scratch SpikeWhale Vision-Language Model},
  author = {Quazim0t0},
  year   = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Quazim0t0/Byrne-VLM-131M}}
}

License

Apache-2.0.

Escarda vs Byrne — vision family comparison

The Byrne family uses HRM refinement. Escarda = Byrne + JEPA (Joint-Embedding Predictive head) added alongside HRM in both the vision encoder and the LM trunk — auxiliary only, zero inference cost.

Vision encoder (DINOv2 teacher-alignment, n=1024 held-out):

	Byrne-VE	Escarda-VE
Params	39.34M	39.60M (+JEPA head)
CLS cosine	0.776	0.771
PATCH cosine	0.600	0.584
JEPA self-consistency	—	0.040

Docling (same held-out doc images, atomic DocTags): both emit well-formed DocTags; Byrne-Docling is marginally more complete on the hardest samples (closes </formula>, includes the <code> wrapper), consistent with its slightly higher teacher-alignment. Escarda-Docling is structurally on par and adds the JEPA representation-learning trait.

Pros/cons. Byrne (HRM): higher teacher-alignment, all capacity on distillation fidelity; no self-supervised objective. Escarda (HRM+JEPA): self-supervised neighbour-prediction (richer spatial structure) at zero inference cost, trading ~1–3% teacher-alignment. Same size class.

Family repos: Byrne-VE · Escarda-VE · Byrne-Docling-131M · Escarda-Docling-126M

v2 update (448px + repaired engram + terse captions)

This model was upgraded: the 448px vision encoder (was 224px), the engram-repaired base LM (the n-gram memory was degenerate — all tokens hashed to bucket 0 — now repaired bit-exact and populated), and retrained on terse COCO-style captions.

COCO CIDEr (val, n=500): 0.061 -> 0.140 (2.3x). The intermediate verbose-style v2 scored 0.029; switching the training captions from the verbose "In this image we can see..." style to terse COCO-style captions is what unlocked the gain.

Use this as a base — keep training on it

Byrne-VLM-131M is meant to be a base VLM you continue training, not a finished end-model. It ships with the vision encoder (weights/byrne_ve.pt, 448px), the engram-repaired base LM (lm/), and a trained caption LoRA + connector (weights/byrne_vlm.pt). Fine-tune the LoRA on your own images/captions (or a new task) and it will pick up from here.

Two training scripts are included: train_connector.py (stage 1, vision→LM grounding) and train_lora.py (stage 2, the Family-LoRA that does the actual task learning).

Data format

A directory with a captions.jsonl and an images/ folder:

my_data/
  captions.jsonl        # one JSON object per line
  images/
    0001.jpg
    0002.jpg

Each line of captions.jsonl:

{"image": "0001.jpg", "caption": "A cat sitting on a windowsill."}

Keep captions terse and COCO-style — verbose "In this image we can see..." phrasing hurts CIDEr badly (it cost this model 0.140 → 0.029 in an ablation).

A. Continue this model's LoRA (recommended)

Picks up the existing caption LoRA + connector and keeps training:

pip install -r requirements.txt

python train_lora.py --lora-type family \
    --lm-dir lm \
    --vision-ckpt weights/byrne_ve.pt \
    --connector   weights/byrne_vlm.pt \
    --resume-vlm  weights/byrne_vlm.pt \
    --data-dir my_data \
    --unfreeze-engram \
    --steps 4000 --batch-size 4 --lr 2e-4 \
    --save-every 1000 --out my_ckpt

--unfreeze-engram keeps the n-gram Engram memory trainable (it was repaired so every bucket is reachable — leave this on so the memory keeps populating). The result is my_ckpt/vlm_stage2_last.pt.

B. Train a fresh LoRA on your own task/style

Reuse the grounding (connector) but start the LoRA from scratch — good for a different caption style or a new task. Just drop --resume-vlm:

python train_lora.py --lora-type family \
    --lm-dir lm --vision-ckpt weights/byrne_ve.pt \
    --connector weights/byrne_vlm.pt \
    --data-dir my_data --unfreeze-engram \
    --steps 8000 --batch-size 4 --save-every 2000 --out my_ckpt

C. (Optional) Retrain the connector first

Only needed if you change the vision encoder or want fresh grounding:

python train_connector.py --data-dir my_data \
    --lm-dir lm --vision-ckpt weights/byrne_ve.pt \
    --steps 6000 --batch-size 4 --save-every 2000 --out my_ckpt
# then point train_lora.py --connector my_ckpt/connector_last.pt

Run your fine-tuned model

python generate.py --image my_data/images/0001.jpg \
    --ckpt my_ckpt/vlm_stage2_last.pt \
    --lm-dir lm --vision-ckpt weights/byrne_ve.pt \
    --tokenizer tokenizer.json --max-new 40 --repetition-penalty 1.2

Notes

• Document images / DocTags? Add --doctags --tokenizer tokenizer_doctags.json --letterbox and use structured-DocTags data — see the Byrne-Docling model for that path.
• Batch size / VRAM: batch 4 at 448px fits ~8–10 GB. Drop to 2 if you OOM.
• What trains: the Family-LoRA (HRM + MoE-SwiGLU adapter, ~5M) + the connector + (with --unfreeze-engram) the Engram tables. The base LM stays frozen otherwise.