Escarda-Docling-126M
Escarda-Docling is a compact (~126M) document-understanding VLM that converts a
page image into structured DocTags (OTSL tables, <chart>/<formula>/<code>,
<loc_*> boxes, language tags). It is the document specialization of the Escarda
model family.
Escarda = the Byrne family plus JEPA added alongside HRM (not replacing it). Every place Byrne uses an HRM refinement block, Escarda adds a JEPA (Joint-Embedding Predictive) head next to it: from a token it predicts a neighbouring token's own representation (stop-gradient target, 1−cosine loss). It is auxiliary only — active during training, zero cost at inference.
Architecture
| Component | Params | Notes |
|---|---|---|
| Escarda-VE (vision encoder) | 39.60M | ViT-style, 448px / patch16 / 784 tokens, RMSNorm, 2D-axial RoPE, QK-Norm, SwiGLU, HRM refine + JEPA head |
| Connector | 1.18M | MLP 512→640 |
| Escarda-86M LM | 85.7M | SpikeWhale LM, hidden 640, 16 layers, 10 heads, vocab 16652 (140 atomic DocTags tokens added), 4096 ctx, HRM + JEPA |
| Family-LoRA | ~5M | HRM + MoE-SwiGLU adapter (JEPA lives in the encoder + LM trunk, not the per-token LoRA) |
| Total | ~126M |
How it was built (identical recipe to Byrne-Docling)
- VE distillation — 50k steps from a frozen DINOv2-base teacher (448px), HRM + JEPA aux.
- VE self-distillation — 20k steps DINO-style (EMA teacher, cosine prototypes), no collapse (teacher peak-prob stayed ~0.15).
- Connector — 6k steps aligning Escarda-VE → Escarda-LM on document images (letterbox).
- DocTags stage-2 — 8k steps Family-LoRA (HRM+MoE) with 140 atomic DocTags tokens added to the tokenizer + extended LM embeddings, letterbox preprocessing (whole page visible), on 6k ground-truth SmolDocling pairs (SynthChart/Formula/Code).
Escarda vs Byrne — head to head
Vision encoder (DINOv2 teacher-alignment, n=1024 held-out):
| Byrne-VE | Escarda-VE | |
|---|---|---|
| Params | 39.34M | 39.60M (+JEPA head) |
| CLS cosine | 0.776 | 0.771 |
| PATCH cosine | 0.600 | 0.584 |
| JEPA self-consistency | — | 0.040 |
Docling (same 3 held-out doc images, atomic DocTags):
| Sample | Byrne-Docling | Escarda-Docling |
|---|---|---|
| chart | well-formed <chart>…<bar_chart><fcel>…</chart> |
well-formed <chart>…<bar_chart><fcel>…</chart> |
| formula | compact LaTeX, closed </formula> |
valid LaTeX, longer/rambled |
| code | includes <code> wrapper + <_Java_> |
<_SQL_> lang tag, omitted <code> wrapper |
Pros / cons
Byrne (HRM only)
- ➕ Marginally higher teacher-alignment; slightly cleaner/more complete DocTags on these samples (closes formula, includes
<code>wrapper). - ➕ All encoder capacity spent on distillation fidelity.
- ➖ Pure supervised-mimicry representation; no self-supervised spatial objective.
Escarda (HRM + JEPA)
- ➕ Adds a self-supervised neighbour-prediction (JEPA) objective in both the encoder and LM trunk → richer spatial/representation structure, at zero inference cost.
- ➕ Structurally sound DocTags, essentially on par with Byrne.
- ➖ Trades ~1–3% teacher-alignment (JEPA diverts a little capacity from pure mimicry); slightly less complete on the hardest samples at this scale.
Bottom line: on raw DocTags quality the two are comparable, with Byrne holding a small edge on completeness; Escarda's value is the added JEPA representation-learning trait for free at inference. Both are the same size class.
Usage
pip install -r requirements.txt
python generate.py --image examples/chart.png --ckpt weights/escarda_docling.pt \
--lm-dir lm --vision-ckpt weights/escarda_ve.pt --tokenizer tokenizer_doctags.json \
--letterbox --max-new 256 --repetition-penalty 1.2 --no-repeat-ngram 3
Related
- Escarda-86M-Base — the JEPA+HRM base LM
- Byrne-Docling-131M — the HRM-only sibling
- Byrne-VLM-131M
Weights are bundled here for a self-contained release. Auxiliary JEPA adds no inference cost.