Sakatepon/Brujula-450M-DocQA

Brújula-450M-DocQA

Pass it a document, it answers from the document. A LoRA fine-tune of Brújula-450M for extractive question answering over papers and multi-paragraph contexts, trained on ~4,800 samples in 6 h 40 m on a single Intel Arc B580 (12 GB), at a cost of $0.

What this gets right, and what it does not

It scores 22.7% on held-out documents, up from 0.0% for the base model. So: it is right about one answer in four, and you should expect to see it wrong more often than right.

It does: pull single facts verbatim out of a document, and abstain (71% of the time) when the answer genuinely is not there, instead of inventing one.

It does not:

• Multi-hop reasoning. Composition sits at ~6% across every checkpoint measured. It never learned this at all — not a tuning problem, the ceiling of a 512M model.
• Improve with more data. The curve runs to 7,646 samples: p = 0.189, so an 8× increase is not statistically detectable. A second epoch actively overfits.
• Transfer off-domain. 8.9% on papers outside its training distribution.

The honest claim here is sample efficiency — 55% of a 1.2B RAG model's score from ~200× less data — not that this is a capable document-QA system. Full numbers below; nothing is buried.

Results

150 held-out rows (documents unseen in training), greedy decoding, same grader for every model:

model	overall	fact	comp (multi-hop)	abstain
Brújula-450M (base)	0.0%	0/107	0/36	0/7
Brújula-450M-DocQA	22.7%	23%	11%	71%
LFM2.5-1.2B-Base (yardstick)	32.0%	32%	25%	71%
LFM2-1.2B-RAG (yardstick, 1M+ samples)	41.3%	40%	50%	14%

The two Liquid models are yardsticks, not competitors — they are 2.3× the size and trained on orders of magnitude more data. They are here so the number has a scale.

What the comparison actually shows:

• Sample efficiency. 22.7% from ~4,800 samples against 41.3% from 1M+ — roughly 55% of the score from ~200× less data at 37% of the parameters.
• Abstention survived the fine-tune. 71% vs the RAG model's 14%. Liquid's own base abstains at 71% too; its RAG fine-tune destroyed that. Ours kept it, having been trained with 373 unanswerable rows.
• No lost-in-the-middle. Accuracy by evidence depth is flat (early 23 / middle 19 / late 19) at a size the literature treats as purely recency-biased.
• The base floor is real. 0.0% is not an artifact: 123 of 150 base generations were non-empty, just wrong.

What it cannot do — please read this

• Multi-hop composition does not work. comp sits at ~6% across every checkpoint we measured. All of the learning is in single-fact extraction.

• More data does not help. We ran the scaling curve, 950 → 7,646 samples:

  samples	950	1900	2478	2850	3800	4750	5700	6650	7646
  score	16.7	17.3	18.7	16.0	16.0	17.3	20.0	20.0	21.3%

  Paired McNemar, 950 vs 7,646: gained 14, lost 7, two-sided p = 0.189. An 8× data increase is not statistically detectable, and the dips at 2,850 / 3,800 confirm the drift is noise.

• More compute does not help either. A 2-epoch control halved train loss (1.235 → 0.621) while validation worsened (1.4635 → 1.5362) and the score fell to 19.3%. One epoch does not memorize; two does.

• It generalizes worse off-domain. 8.9% on papers about our own architecture (trained on NLP papers + Wikipedia). The drop from 22.7% is a generalization gap, not a context-length one — the in-domain length effect is mild (24% under 4K → 19% at 4–8K).

Taken together, the bottleneck is a capability boundary of a 512M model, not data volume or training budget. The honest claim here is sample efficiency, not scale-with-data.

Usage

The prompt format is load-bearing — it must match the training render exactly, and decoding must be greedy. Both are how the 22.7% was measured; sampling will do worse.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Sakatepon/Brujula-450M-DocQA"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16
).eval()

document = "..."          # the paper / passages to answer from
question = "What dataset was used for evaluation?"

prompt = f"User: {document.strip()}\n\nQuestion: {question.strip()}\nAssistant: "
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=24, do_sample=False, use_cache=False,
                     eos_token_id=50256, pad_token_id=50256)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))

Answers are terse — the median generation is 4 tokens. That is trained behaviour, not truncation. If it cannot find the answer it is designed to say so rather than invent one.

The YaRN context regime (yarn_ms, rope_trained_len=1024, rope_scale_len=32768) is baked into config.json; it is the regime the model was fine-tuned and evaluated in. Changing it will degrade results. use_cache=False is required (Block Attention-Residuals mix across layers).

Training

LoRA rank 16 / alpha 32 on attention + FFN projections, base frozen, adapters merged for release. AdamW, LR 2e-4, block 8192, answer-masked loss, 1 epoch, bf16, yarn_ms RoPE scaling at scale_len 32768 from a pre-train length of 1024. Final train loss 1.3332 / val 1.4467 — the validation loss did not diverge, which is why this checkpoint was chosen.

Training data

A 27,150-row corpus (23,761 train / 3,389 val) built from public sources, split at document level so no document appears in both halves:

• QASPER — questions over NLP papers (CC BY 4.0)
• MuSiQue — multi-hop questions over paragraph sets (CC BY 4.0)

99% of answers are verbatim-supported by the context; evidence depth is balanced early/middle/late by construction; 523 rows are deliberately unanswerable so abstention is trained rather than hoped for.

License

Apache-2.0 (weights). Training data is CC BY 4.0 — please credit QASPER and MuSiQue.