brasa-12m
Code: https://github.com/yanzino6/brasa
Brasa is a small GPT-style language model written and trained from scratch in PyTorch: the transformer, the byte-level BPE tokenizer, the training loop, the retriever and the browser inference engine are all hand-written, with no pretrained weights and no model libraries. It answers questions about one person, in Portuguese or English, from a small curated knowledge base, and it is small enough to run entirely in a web browser.
Still from scratch: the weights start random and are first trained by this repository's own code as a plain language model on public text (10,000 steps, 246M tokens seen, 55 minutes on the same laptop, validation perplexity 20.4), then on the question corpus. No pretrained weights or tokenizers from anyone else are involved. Text: https://huggingface.co/datasets/wikimedia/wikipedia at revision b04c8d1c, dump 2023-11-01, licence CC BY-SA 3.0 and GFDL (Wikipedia text); pt: 22,631 articles, 160 MB (20231101.pt/train-00000-of-00006.parquet), en: 32,049 articles, 160 MB (20231101.en/train-00000-of-00041.parquet).
It is a narrow model on purpose. It is not a general assistant: it knows nothing outside its knowledge base and is trained to say so.
| Parameters | 12,194,688 |
| Architecture | decoder-only transformer, 6 layers, width 384, 6 heads, RMSNorm (pre-norm), rotary position embeddings, SwiGLU, tied embeddings, no biases |
| Context length | 384 tokens |
| Tokenizer | byte-level BPE trained from scratch, 4096 tokens (7 control tokens) |
| Languages | Portuguese, English |
| Training | 6,000 steps, AdamW, cosine schedule, 77 min on mps |
| License | MIT |
Files
| File | Bytes | What |
|---|---|---|
config.json |
1,182 | model configuration, prompt format, special tokens, context length |
model-card.json |
1,308,665 | machine-readable evaluation (every number of this card comes from it) |
model.safetensors |
48,782,920 | float32 weights of the released checkpoint, tied output matrix stored once (rows labelled float32 in the evaluation) |
retrieval.json |
10,175,004 | BM25 index, fact vectors and the curated facts the model answers from |
tokenizer.json |
38,945 | BPE merges and control tokens |
web/manifest.json |
2,397 | manifest of the browser bundle (configuration, sizes, checksums) |
web/model.f16.bin |
24,442,308 | float16 weights in the TGPT container (rows labelled float16 in the evaluation) |
web/model.i8.bin |
12,364,896 | int8 weights, one scale per row: the bundle the browser engine loads (rows labelled int8) |
The evaluation tables below have a Weights column: float32 is model.safetensors, float16 is web/model.f16.bin and int8 is web/model.i8.bin, each evaluated as stored (the quantised files de-quantised to float32 for scoring). The two web/*.bin files use the single-file "TGPT" container described in the code repository (JSON header with the configuration, the prompt format and the tokenizer, followed by the tensors); int8 stores every matrix with one scale per output row.
How to use
In Python, with the brasa package from the code repository (https://github.com/yanzino6/brasa) and PyTorch:
pip install "git+https://github.com/yanzino6/brasa" safetensors huggingface_hub
import json
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from brasa.config import GPTConfig, PromptConfig
from brasa.encoder import DenseRetriever
from brasa.model import GPT
from brasa.retrieval import Bm25Retriever
from brasa.sample import answer
from brasa.tokenizer import BPETokenizer
folder = snapshot_download("Yanzino6/brasa-12m") # or the path of a local copy of this repository
config = json.load(open(f"{folder}/config.json"))
model = GPT(GPTConfig(**{k: v for k, v in config["model"].items() if k != "params"}))
state = load_file(f"{folder}/model.safetensors")
state["lm_head.weight"] = state["tok_emb.weight"] # tied embeddings
model.load_state_dict(state)
model.eval()
tok = BPETokenizer.load(f"{folder}/tokenizer.json")
index = Bm25Retriever.load(f"{folder}/retrieval.json")
retriever = DenseRetriever.from_index(model, tok, index) # BM25 fused with the model's own ranking, as on the site
# Retrieval, open-book prompt, then greedy decoding restricted to the retrieved facts and the fixed answers,
# with the retriever's ranking as a prior (the prior stored in retrieval.json): the site's decoding.
out = answer(model, tok, retriever, "Onde o Yan estuda?", PromptConfig(**config["prompt"]), constrain="retrieved")
print(out["answer"])
for hit in out["hits"]:
print(f" [{hit.score:.2f}] {hit.text}")
In the browser, with the engine from the code repository (no dependencies, WebAssembly SIMD with a plain JavaScript fallback); the int8 bundle is web/model.i8.bin:
import { BrasaClient } from './engine/index.js';
const client = new BrasaClient(new Worker(new URL('./engine/worker.js', import.meta.url), { type: 'module' }));
await client.load({ model: 'web/model.i8.bin', wasm: 'kernel.wasm', retrieval: 'retrieval.json' });
const run = client.ask('Onde ele mora?', {}, { onToken: (t) => console.log(t.delta) });
console.log((await run.done).text);
Prompt format
open-book <|bos|><|ctx|>fact<|sep|>fact<|sep|>fact<|q|>question<|a|>answer<|eos|>
closed-book <|bos|><|q|>question<|a|>answer<|eos|>
Facts are ranked by two signals added together: BM25 over the knowledge base (retrieval.json: words and character trigrams of the facts, their topic names and the training questions) and the model's own opinion. The same network that writes the answer is also trained, with a contrastive loss, to put a question next to the fact that answers it: the browser runs it once over the question, pools its final hidden states into one vector and compares it with the precomputed vectors of the facts (int8, inside retrieval.json; fusion weight 1.5). On held-out test questions the gold fact is the first hit for 66.2% (BM25 alone: 49.8%) and among the first four for 89.8% (76.9%). Up to 4 facts are put in the prompt. Out-of-scope questions are answered
with a short refusal in the language of the question.
Evaluation
Greedy decoding. "Seen" are training questions; "held-out" are questions whose wording never occurs in training (written to be phrased differently: most share less than a third of their words with every training phrasing of the same fact), plus small talk and out-of-scope questions never seen.
Decoding modes. Free: the model writes whatever it wants. Any curated sentence: every step may only continue one of the curated sentences of the knowledge base (or a fixed greeting / refusal), so the answer is always a real fact, possibly the wrong one. Retrieved facts only: the same, restricted to the facts the retriever put in the prompt; every statement and number in the answer is then one the visitor can see in the sources. Retrieved facts + ranking prior (the site's default): the same, and the retriever's ranking counts as a prior on the model's choice, so a fact the retriever put lower has to be that much more likely for the model to win; it costs no extra computation. Retriever's first fact uses no model: it is the baseline the open-book rows have to beat, and it can never decline a question.
EM is normalised exact match against the one gold answer, F1 is token-level, "key spans" is the share of required names and numbers present in the answer, "bad numbers" is the share of numbers in the answers that appear neither in the gold answer, nor in the retrieved facts, nor in the question (with constrained decoding these are numbers of other real facts, not inventions).
| Weights | Mode | Questions | n | EM | F1 | EM in scope | Key spans | Refusal P | Refusal R | Language | Bad numbers |
|---|---|---|---|---|---|---|---|---|---|---|---|
| float32 | open-book, free | seen (training) | 300 | 97.7% | 98.3% | 97.6% | 99.4% | 100.0% | 100.0% | 100.0% | 0.0% |
| float32 | open-book, free | held-out (validation) | 1250 | 64.8% | 75.3% | 63.5% | 78.6% | 88.3% | 82.9% | 100.0% | 0.7% |
| float32 | open-book, free | held-out (test) | 1278 | 63.0% | 74.4% | 60.8% | 80.0% | 86.5% | 87.3% | 99.8% | 1.4% |
| float32 | open-book, free | held-out (final, read once) | 1210 | 65.5% | 75.2% | 63.8% | 83.1% | 84.9% | 81.8% | 100.0% | 0.6% |
| float32 | closed-book, free | seen (training) | 300 | 93.0% | 95.1% | 92.8% | 96.7% | 100.0% | 100.0% | 100.0% | 1.2% |
| float32 | closed-book, free | held-out (validation) | 1250 | 46.0% | 61.4% | 43.1% | 66.7% | 80.2% | 89.0% | 99.8% | 33.1% |
| float32 | closed-book, free | held-out (test) | 1278 | 46.5% | 62.1% | 42.6% | 69.4% | 82.3% | 89.1% | 99.5% | 31.8% |
| float32 | closed-book, free | held-out (final, read once) | 1210 | 46.0% | 61.1% | 42.4% | 70.2% | 81.2% | 82.7% | 99.8% | 29.4% |
| float32 | open-book, any curated sentence | seen (training) | 300 | 97.7% | 98.3% | 97.6% | 99.4% | 100.0% | 100.0% | 100.0% | 0.0% |
| float32 | open-book, any curated sentence | held-out (validation) | 1250 | 64.8% | 75.2% | 63.5% | 78.6% | 88.3% | 82.9% | 100.0% | 0.8% |
| float32 | open-book, any curated sentence | held-out (test) | 1278 | 63.0% | 74.4% | 60.8% | 80.0% | 86.5% | 87.3% | 99.8% | 1.4% |
| float32 | open-book, any curated sentence | held-out (final, read once) | 1210 | 65.5% | 75.2% | 63.9% | 83.1% | 84.9% | 81.8% | 100.0% | 0.6% |
| float32 | closed-book, any curated sentence | seen (training) | 300 | 93.0% | 95.1% | 92.8% | 96.7% | 100.0% | 100.0% | 100.0% | 1.2% |
| float32 | closed-book, any curated sentence | held-out (validation) | 1250 | 46.1% | 61.2% | 43.1% | 66.6% | 80.2% | 89.0% | 99.8% | 33.7% |
| float32 | closed-book, any curated sentence | held-out (test) | 1278 | 46.5% | 61.9% | 42.6% | 69.3% | 82.3% | 89.1% | 99.5% | 32.3% |
| float32 | closed-book, any curated sentence | held-out (final, read once) | 1210 | 46.2% | 61.0% | 42.5% | 69.9% | 82.0% | 82.7% | 99.8% | 29.0% |
| float32 | open-book, retrieved facts only | seen (training) | 300 | 97.7% | 98.3% | 97.6% | 99.4% | 100.0% | 100.0% | 100.0% | 0.0% |
| float32 | open-book, retrieved facts only | held-out (validation) | 1250 | 65.0% | 75.4% | 63.7% | 78.6% | 88.3% | 82.9% | 100.0% | 0.0% |
| float32 | open-book, retrieved facts only | held-out (test) | 1278 | 63.1% | 74.6% | 61.0% | 80.3% | 86.5% | 87.3% | 99.8% | 0.0% |
| float32 | open-book, retrieved facts only | held-out (final, read once) | 1210 | 66.0% | 75.5% | 64.4% | 83.2% | 84.1% | 81.8% | 100.0% | 0.0% |
| float32 | open-book, retrieved facts + ranking prior (site default) | seen (training) | 300 | 99.3% | 99.6% | 99.3% | 100.0% | 100.0% | 100.0% | 100.0% | 0.0% |
| float32 | open-book, retrieved facts + ranking prior (site default) | held-out (validation) | 1250 | 70.8% | 79.7% | 69.9% | 83.1% | 88.5% | 84.2% | 100.0% | 0.0% |
| float32 | open-book, retrieved facts + ranking prior (site default) | held-out (test) | 1278 | 67.4% | 77.5% | 65.6% | 82.7% | 86.5% | 87.3% | 99.8% | 0.0% |
| float32 | open-book, retrieved facts + ranking prior (site default) | held-out (final, read once) | 1210 | 70.2% | 78.5% | 69.1% | 85.7% | 84.1% | 81.8% | 100.0% | 0.0% |
| float32 | retriever's first fact (no model) | seen (training) | 300 | 94.0% | 95.7% | 96.2% | 99.0% | n/a | 0.0% | 100.0% | 0.0% |
| float32 | retriever's first fact (no model) | held-out (validation) | 1250 | 62.8% | 73.3% | 67.2% | 83.2% | n/a | 0.0% | 100.0% | 0.0% |
| float32 | retriever's first fact (no model) | held-out (test) | 1278 | 57.9% | 69.6% | 63.4% | 81.8% | n/a | 0.0% | 99.9% | 0.0% |
| float32 | retriever's first fact (no model) | held-out (final, read once) | 1210 | 62.6% | 71.2% | 68.8% | 86.0% | n/a | 0.0% | 99.9% | 0.0% |
| float16 | open-book, free | held-out (validation) | 1250 | 64.8% | 75.3% | 63.5% | 78.6% | 88.3% | 82.9% | 100.0% | 0.7% |
| float16 | open-book, free | held-out (test) | 1278 | 63.0% | 74.4% | 60.8% | 80.0% | 86.5% | 87.3% | 99.8% | 1.4% |
| float16 | open-book, free | held-out (final, read once) | 1210 | 65.4% | 75.1% | 63.7% | 83.1% | 84.1% | 81.8% | 100.0% | 0.6% |
| float16 | closed-book, free | held-out (validation) | 1250 | 46.0% | 61.4% | 43.1% | 66.7% | 80.2% | 89.0% | 99.8% | 33.1% |
| float16 | closed-book, free | held-out (test) | 1278 | 46.5% | 62.1% | 42.6% | 69.4% | 82.3% | 89.1% | 99.5% | 31.8% |
| float16 | closed-book, free | held-out (final, read once) | 1210 | 46.0% | 61.1% | 42.4% | 70.2% | 81.2% | 82.7% | 99.8% | 29.4% |
| float16 | open-book, any curated sentence | held-out (validation) | 1250 | 64.8% | 75.2% | 63.5% | 78.6% | 88.3% | 82.9% | 100.0% | 0.8% |
| float16 | open-book, any curated sentence | held-out (test) | 1278 | 63.0% | 74.4% | 60.8% | 80.0% | 86.5% | 87.3% | 99.8% | 1.4% |
| float16 | open-book, any curated sentence | held-out (final, read once) | 1210 | 65.5% | 75.2% | 63.8% | 83.1% | 84.1% | 81.8% | 100.0% | 0.6% |
| float16 | closed-book, any curated sentence | held-out (validation) | 1250 | 46.1% | 61.2% | 43.1% | 66.6% | 80.2% | 89.0% | 99.8% | 33.7% |
| float16 | closed-book, any curated sentence | held-out (test) | 1278 | 46.5% | 61.9% | 42.6% | 69.3% | 82.3% | 89.1% | 99.5% | 32.3% |
| float16 | closed-book, any curated sentence | held-out (final, read once) | 1210 | 46.2% | 61.0% | 42.5% | 69.9% | 82.0% | 82.7% | 99.8% | 29.0% |
| float16 | open-book, retrieved facts only | held-out (validation) | 1250 | 65.0% | 75.4% | 63.7% | 78.6% | 88.3% | 82.9% | 100.0% | 0.0% |
| float16 | open-book, retrieved facts only | held-out (test) | 1278 | 63.1% | 74.6% | 61.0% | 80.3% | 86.5% | 87.3% | 99.8% | 0.0% |
| float16 | open-book, retrieved facts only | held-out (final, read once) | 1210 | 65.9% | 75.4% | 64.3% | 83.2% | 83.3% | 81.8% | 100.0% | 0.0% |
| float16 | open-book, retrieved facts + ranking prior (site default) | held-out (validation) | 1250 | 70.8% | 79.7% | 69.9% | 83.1% | 88.5% | 84.2% | 100.0% | 0.0% |
| float16 | open-book, retrieved facts + ranking prior (site default) | held-out (test) | 1278 | 67.4% | 77.5% | 65.6% | 82.7% | 86.5% | 87.3% | 99.8% | 0.0% |
| float16 | open-book, retrieved facts + ranking prior (site default) | held-out (final, read once) | 1210 | 70.2% | 78.4% | 69.0% | 85.7% | 83.3% | 81.8% | 100.0% | 0.0% |
| float16 | retriever's first fact (no model) | held-out (validation) | 1250 | 62.8% | 73.3% | 67.2% | 83.2% | n/a | 0.0% | 100.0% | 0.0% |
| float16 | retriever's first fact (no model) | held-out (test) | 1278 | 57.8% | 69.5% | 63.3% | 81.7% | n/a | 0.0% | 99.9% | 0.0% |
| float16 | retriever's first fact (no model) | held-out (final, read once) | 1210 | 62.6% | 71.2% | 68.8% | 86.0% | n/a | 0.0% | 99.9% | 0.0% |
| int8 | open-book, free | held-out (validation) | 1250 | 64.6% | 75.2% | 63.4% | 78.6% | 88.3% | 82.9% | 100.0% | 0.7% |
| int8 | open-book, free | held-out (test) | 1278 | 62.8% | 74.3% | 60.6% | 79.9% | 85.7% | 87.3% | 99.8% | 1.4% |
| int8 | open-book, free | held-out (final, read once) | 1210 | 65.5% | 75.2% | 63.9% | 83.1% | 84.9% | 81.8% | 100.0% | 0.6% |
| int8 | closed-book, free | held-out (validation) | 1250 | 46.2% | 61.5% | 43.2% | 66.7% | 82.0% | 89.0% | 99.8% | 33.1% |
| int8 | closed-book, free | held-out (test) | 1278 | 46.8% | 62.3% | 43.0% | 69.5% | 82.3% | 89.1% | 99.5% | 31.4% |
| int8 | closed-book, free | held-out (final, read once) | 1210 | 46.1% | 61.1% | 42.6% | 70.4% | 80.2% | 80.9% | 99.8% | 28.8% |
| int8 | open-book, any curated sentence | held-out (validation) | 1250 | 64.6% | 75.1% | 63.4% | 78.6% | 88.3% | 82.9% | 100.0% | 0.8% |
| int8 | open-book, any curated sentence | held-out (test) | 1278 | 62.8% | 74.3% | 60.6% | 79.9% | 85.7% | 87.3% | 99.8% | 1.4% |
| int8 | open-book, any curated sentence | held-out (final, read once) | 1210 | 65.6% | 75.3% | 64.0% | 83.1% | 84.9% | 81.8% | 100.0% | 0.6% |
| int8 | closed-book, any curated sentence | held-out (validation) | 1250 | 46.2% | 61.4% | 43.3% | 66.6% | 82.0% | 89.0% | 99.8% | 33.6% |
| int8 | closed-book, any curated sentence | held-out (test) | 1278 | 46.8% | 62.1% | 43.0% | 69.4% | 82.3% | 89.1% | 99.5% | 31.9% |
| int8 | closed-book, any curated sentence | held-out (final, read once) | 1210 | 46.3% | 61.1% | 42.8% | 70.0% | 80.9% | 80.9% | 99.8% | 28.5% |
| int8 | open-book, retrieved facts only | held-out (validation) | 1250 | 64.8% | 75.3% | 63.5% | 78.6% | 88.3% | 82.9% | 100.0% | 0.0% |
| int8 | open-book, retrieved facts only | held-out (test) | 1278 | 63.0% | 74.5% | 60.8% | 80.0% | 85.7% | 87.3% | 99.8% | 0.0% |
| int8 | open-book, retrieved facts only | held-out (final, read once) | 1210 | 66.0% | 75.5% | 64.4% | 83.2% | 84.1% | 81.8% | 100.0% | 0.0% |
| int8 | open-book, retrieved facts + ranking prior (site default) | held-out (validation) | 1250 | 70.7% | 79.6% | 69.8% | 83.1% | 88.5% | 84.2% | 100.0% | 0.0% |
| int8 | open-book, retrieved facts + ranking prior (site default) | held-out (test) | 1278 | 67.3% | 77.5% | 65.5% | 82.7% | 85.7% | 87.3% | 99.8% | 0.0% |
| int8 | open-book, retrieved facts + ranking prior (site default) | held-out (final, read once) | 1210 | 70.3% | 78.5% | 69.2% | 85.7% | 84.1% | 81.8% | 100.0% | 0.0% |
| int8 | retriever's first fact (no model) | held-out (validation) | 1250 | 62.7% | 73.3% | 67.1% | 83.2% | n/a | 0.0% | 100.0% | 0.0% |
| int8 | retriever's first fact (no model) | held-out (test) | 1278 | 57.7% | 69.4% | 63.1% | 81.4% | n/a | 0.0% | 99.9% | 0.0% |
| int8 | retriever's first fact (no model) | held-out (final, read once) | 1210 | 62.6% | 71.2% | 68.8% | 85.9% | n/a | 0.0% | 99.9% | 0.0% |
Exact match is strict: with dozens of closely related facts per project, a neighbouring fact of the right topic is often a reasonable answer. The next table adds what a visitor experiences: whether the answer is a curated sentence at all, whether it is about the right topic, and how often the retriever had the gold fact among its first hits.
| Weights | Mode | Questions | Exact | Curated sentence | Right topic | Gold fact in top 1 / 2 / 4 | Exact when retrieved / when missed |
|---|---|---|---|---|---|---|---|
| float32 | open-book, free | held-out (validation) | 64.8% | 99.6% | 93.5% | 70.2% / 85.7% / 92.0% | 69.2% / 3.3% |
| float32 | open-book, free | held-out (test) | 63.0% | 99.9% | 93.1% | 66.2% / 81.5% / 89.8% | 68.7% / 3.5% |
| float32 | open-book, free | held-out (final, read once) | 65.5% | 99.9% | 94.1% | 72.1% / 85.6% / 93.2% | 69.9% / 1.4% |
| float32 | closed-book, free | held-out (validation) | 46.0% | 98.7% | 89.5% | n/a | n/a |
| float32 | closed-book, free | held-out (test) | 46.5% | 99.1% | 88.4% | n/a | n/a |
| float32 | closed-book, free | held-out (final, read once) | 46.0% | 98.7% | 89.7% | n/a | n/a |
| float32 | open-book, any curated sentence | held-out (validation) | 64.8% | 99.9% | 93.7% | 70.2% / 85.7% / 92.0% | 69.2% / 3.3% |
| float32 | open-book, any curated sentence | held-out (test) | 63.0% | 100.0% | 93.2% | 66.2% / 81.5% / 89.8% | 68.7% / 3.5% |
| float32 | open-book, any curated sentence | held-out (final, read once) | 65.5% | 100.0% | 94.2% | 72.1% / 85.6% / 93.2% | 70.0% / 1.4% |
| float32 | closed-book, any curated sentence | held-out (validation) | 46.1% | 99.9% | 90.1% | n/a | n/a |
| float32 | closed-book, any curated sentence | held-out (test) | 46.5% | 100.0% | 89.0% | n/a | n/a |
| float32 | closed-book, any curated sentence | held-out (final, read once) | 46.2% | 99.9% | 90.3% | n/a | n/a |
| float32 | open-book, retrieved facts only | held-out (validation) | 65.0% | 99.8% | 93.3% | 70.2% / 85.7% / 92.0% | 69.7% / 0.0% |
| float32 | open-book, retrieved facts only | held-out (test) | 63.1% | 99.9% | 93.2% | 66.2% / 81.5% / 89.8% | 69.3% / 0.0% |
| float32 | open-book, retrieved facts only | held-out (final, read once) | 66.0% | 100.0% | 94.1% | 72.1% / 85.6% / 93.2% | 70.6% / 0.0% |
| float32 | open-book, retrieved facts + ranking prior (site default) | held-out (validation) | 70.8% | 99.8% | 93.9% | 70.2% / 85.7% / 92.0% | 76.6% / 0.0% |
| float32 | open-book, retrieved facts + ranking prior (site default) | held-out (test) | 67.4% | 99.9% | 93.5% | 66.2% / 81.5% / 89.8% | 74.7% / 0.0% |
| float32 | open-book, retrieved facts + ranking prior (site default) | held-out (final, read once) | 70.2% | 100.0% | 94.6% | 72.1% / 85.6% / 93.2% | 75.9% / 0.0% |
| float32 | retriever's first fact (no model) | held-out (validation) | 62.8% | 100.0% | 94.1% | 70.2% / 85.7% / 92.0% | 76.4% / 0.0% |
| float32 | retriever's first fact (no model) | held-out (test) | 57.9% | 100.0% | 93.2% | 66.2% / 81.5% / 89.8% | 73.7% / 0.0% |
| float32 | retriever's first fact (no model) | held-out (final, read once) | 62.6% | 100.0% | 94.4% | 72.1% / 85.6% / 93.2% | 77.3% / 0.0% |
| float16 | open-book, free | held-out (validation) | 64.8% | 99.6% | 93.5% | 70.2% / 85.7% / 92.0% | 69.2% / 3.3% |
| float16 | open-book, free | held-out (test) | 63.0% | 99.9% | 93.1% | 66.1% / 81.5% / 89.8% | 68.7% / 3.5% |
| float16 | open-book, free | held-out (final, read once) | 65.4% | 99.9% | 93.9% | 72.1% / 85.6% / 93.2% | 69.8% / 1.4% |
| float16 | closed-book, free | held-out (validation) | 46.0% | 98.7% | 89.5% | n/a | n/a |
| float16 | closed-book, free | held-out (test) | 46.5% | 99.1% | 88.4% | n/a | n/a |
| float16 | closed-book, free | held-out (final, read once) | 46.0% | 98.7% | 89.7% | n/a | n/a |
| float16 | open-book, any curated sentence | held-out (validation) | 64.8% | 99.9% | 93.7% | 70.2% / 85.7% / 92.0% | 69.2% / 3.3% |
| float16 | open-book, any curated sentence | held-out (test) | 63.0% | 100.0% | 93.2% | 66.1% / 81.5% / 89.8% | 68.7% / 3.5% |
| float16 | open-book, any curated sentence | held-out (final, read once) | 65.5% | 100.0% | 94.0% | 72.1% / 85.6% / 93.2% | 69.9% / 1.4% |
| float16 | closed-book, any curated sentence | held-out (validation) | 46.1% | 99.9% | 90.1% | n/a | n/a |
| float16 | closed-book, any curated sentence | held-out (test) | 46.5% | 100.0% | 89.0% | n/a | n/a |
| float16 | closed-book, any curated sentence | held-out (final, read once) | 46.2% | 99.9% | 90.3% | n/a | n/a |
| float16 | open-book, retrieved facts only | held-out (validation) | 65.0% | 99.8% | 93.3% | 70.2% / 85.7% / 92.0% | 69.7% / 0.0% |
| float16 | open-book, retrieved facts only | held-out (test) | 63.1% | 99.9% | 93.2% | 66.1% / 81.5% / 89.8% | 69.3% / 0.0% |
| float16 | open-book, retrieved facts only | held-out (final, read once) | 65.9% | 100.0% | 93.9% | 72.1% / 85.6% / 93.2% | 70.5% / 0.0% |
| float16 | open-book, retrieved facts + ranking prior (site default) | held-out (validation) | 70.8% | 99.8% | 93.9% | 70.2% / 85.7% / 92.0% | 76.6% / 0.0% |
| float16 | open-book, retrieved facts + ranking prior (site default) | held-out (test) | 67.4% | 99.9% | 93.5% | 66.1% / 81.5% / 89.8% | 74.7% / 0.0% |
| float16 | open-book, retrieved facts + ranking prior (site default) | held-out (final, read once) | 70.2% | 100.0% | 94.5% | 72.1% / 85.6% / 93.2% | 75.8% / 0.0% |
| float16 | retriever's first fact (no model) | held-out (validation) | 62.8% | 100.0% | 94.1% | 70.2% / 85.7% / 92.0% | 76.4% / 0.0% |
| float16 | retriever's first fact (no model) | held-out (test) | 57.8% | 100.0% | 93.2% | 66.1% / 81.5% / 89.8% | 73.6% / 0.0% |
| float16 | retriever's first fact (no model) | held-out (final, read once) | 62.6% | 100.0% | 94.4% | 72.1% / 85.6% / 93.2% | 77.3% / 0.0% |
| int8 | open-book, free | held-out (validation) | 64.6% | 99.6% | 93.5% | 70.1% / 85.7% / 91.9% | 69.0% / 3.3% |
| int8 | open-book, free | held-out (test) | 62.8% | 99.9% | 93.1% | 65.9% / 81.5% / 89.8% | 68.6% / 2.6% |
| int8 | open-book, free | held-out (final, read once) | 65.5% | 99.9% | 94.0% | 72.1% / 85.7% / 93.1% | 70.0% / 1.4% |
| int8 | closed-book, free | held-out (validation) | 46.2% | 98.6% | 89.3% | n/a | n/a |
| int8 | closed-book, free | held-out (test) | 46.8% | 99.1% | 88.3% | n/a | n/a |
| int8 | closed-book, free | held-out (final, read once) | 46.1% | 98.8% | 90.1% | n/a | n/a |
| int8 | open-book, any curated sentence | held-out (validation) | 64.6% | 99.9% | 93.7% | 70.1% / 85.7% / 91.9% | 69.0% / 3.3% |
| int8 | open-book, any curated sentence | held-out (test) | 62.8% | 100.0% | 93.2% | 65.9% / 81.5% / 89.8% | 68.6% / 2.6% |
| int8 | open-book, any curated sentence | held-out (final, read once) | 65.6% | 100.0% | 94.1% | 72.1% / 85.7% / 93.1% | 70.1% / 1.4% |
| int8 | closed-book, any curated sentence | held-out (validation) | 46.2% | 99.9% | 90.0% | n/a | n/a |
| int8 | closed-book, any curated sentence | held-out (test) | 46.8% | 100.0% | 89.0% | n/a | n/a |
| int8 | closed-book, any curated sentence | held-out (final, read once) | 46.3% | 99.9% | 90.6% | n/a | n/a |
| int8 | open-book, retrieved facts only | held-out (validation) | 64.8% | 99.8% | 93.3% | 70.1% / 85.7% / 91.9% | 69.5% / 0.0% |
| int8 | open-book, retrieved facts only | held-out (test) | 63.0% | 99.9% | 93.2% | 65.9% / 81.5% / 89.8% | 69.1% / 0.0% |
| int8 | open-book, retrieved facts only | held-out (final, read once) | 66.0% | 100.0% | 94.1% | 72.1% / 85.7% / 93.1% | 70.7% / 0.0% |
| int8 | open-book, retrieved facts + ranking prior (site default) | held-out (validation) | 70.7% | 99.8% | 93.9% | 70.1% / 85.7% / 91.9% | 76.6% / 0.0% |
| int8 | open-book, retrieved facts + ranking prior (site default) | held-out (test) | 67.3% | 99.9% | 93.5% | 65.9% / 81.5% / 89.8% | 74.6% / 0.0% |
| int8 | open-book, retrieved facts + ranking prior (site default) | held-out (final, read once) | 70.3% | 100.0% | 94.6% | 72.1% / 85.7% / 93.1% | 76.1% / 0.0% |
| int8 | retriever's first fact (no model) | held-out (validation) | 62.7% | 100.0% | 94.1% | 70.1% / 85.7% / 91.9% | 76.3% / 0.0% |
| int8 | retriever's first fact (no model) | held-out (test) | 57.7% | 100.0% | 93.1% | 65.9% / 81.5% / 89.8% | 73.4% / 0.0% |
| int8 | retriever's first fact (no model) | held-out (final, read once) | 62.6% | 100.0% | 94.3% | 72.1% / 85.7% / 93.1% | 77.4% / 0.0% |
Retriever used in the open-book rows: BM25 fused with the model's own ranking of the facts (cosine of its pooled hidden states, weight 1.5). Ranking prior of the site default: a fact loses weight x (1 - its score / the best score) nats, weight 24.0, chosen on the validation questions. Open-book, held-out test questions: the retriever returned the gold facts for 89.8% of the in-scope questions; exact match was 68.7% when it did and 3.5% when it did not.
Breakdown of open-book, retrieved facts + ranking prior (site default), float32, held-out (final, read once):
| Language | n | Exact | F1 | Curated sentence | Right topic |
|---|---|---|---|---|---|
| en | 605 | 71.2% | 79.0% | 100.0% | 94.1% |
| pt | 605 | 69.3% | 78.0% | 100.0% | 95.0% |
| Kind of question | n | Exact | F1 | Curated sentence | Right topic |
|---|---|---|---|---|---|
| small talk | 50 | 34.0% | 48.1% | 100.0% | n/a |
| about the model | 44 | 75.0% | 80.2% | 100.0% | 100.0% |
| out of scope | 110 | 81.8% | 86.0% | 100.0% | n/a |
| one fact | 1006 | 70.6% | 79.1% | 100.0% | 94.3% |
| Topic | n | Exact | F1 | Curated sentence | Right topic |
|---|---|---|---|---|---|
| (none) | 160 | 66.9% | 74.2% | 100.0% | n/a |
| breakhis | 112 | 69.6% | 78.8% | 100.0% | 100.0% |
| pad | 106 | 78.3% | 85.0% | 100.0% | 100.0% |
| crm | 96 | 70.8% | 79.1% | 100.0% | 97.9% |
| seer | 86 | 76.7% | 83.0% | 100.0% | 100.0% |
| skills | 66 | 57.6% | 67.6% | 100.0% | 72.7% |
| isabela | 58 | 56.9% | 69.4% | 100.0% | 98.3% |
| nina | 56 | 73.2% | 80.9% | 100.0% | 98.2% |
| prospecting | 48 | 64.6% | 77.5% | 100.0% | 100.0% |
| brasa | 44 | 75.0% | 80.2% | 100.0% | 100.0% |
| gjdb | 34 | 73.5% | 78.2% | 100.0% | 100.0% |
| armv7a | 30 | 66.7% | 77.9% | 100.0% | 100.0% |
| c-coursework | 30 | 86.7% | 90.7% | 100.0% | 100.0% |
| simmer | 30 | 76.7% | 82.2% | 100.0% | 96.7% |
| experience | 28 | 67.9% | 80.1% | 100.0% | 92.9% |
| algorithms-c | 24 | 91.7% | 94.4% | 100.0% | 100.0% |
| boundaries | 24 | 62.5% | 74.7% | 100.0% | 70.8% |
| esp32-cam-car | 24 | 66.7% | 77.9% | 100.0% | 100.0% |
| process | 22 | 54.5% | 65.9% | 100.0% | 72.7% |
| profile | 22 | 54.5% | 65.2% | 100.0% | 63.6% |
| ml-coursework | 20 | 75.0% | 83.9% | 100.0% | 95.0% |
| projeto-piloto | 20 | 60.0% | 70.8% | 100.0% | 100.0% |
| ct-junior-clients | 16 | 87.5% | 92.6% | 100.0% | 93.8% |
| services | 16 | 75.0% | 80.8% | 100.0% | 75.0% |
| operating-systems | 14 | 71.4% | 78.5% | 100.0% | 78.6% |
| studies | 14 | 71.4% | 83.2% | 100.0% | 92.9% |
| contact | 10 | 90.0% | 91.2% | 100.0% | 90.0% |
The held-out test questions were also used to choose between training settings, so the test rows are optimistic by selection; the "final" rows, when present, come from a slice of questions that was read once, for this card.
Quantisation: f16: relative weight error 0.000207, largest logit drift 0.00391; i8: relative weight error 0.007449, largest logit drift 0.13656; float16 answers identical to float32: 26153/26166; int8 answers identical to float32: 25941/26166
Training data
525 facts in 26 topics, each in Portuguese and English; 57947 training, 1250 validation and 1278 test question/answer records (52 training questions per fact and language, median); 347,682 training examples of 56,081,323 tokens after open-book / closed-book expansion.
Limitations
- The model only knows the facts of its knowledge base, and only in the phrasings it can generalise to from its training questions. Unusual wordings can retrieve the wrong fact or get a wrong answer that sounds confident.
- Answers are short and close to the wording of the knowledge base. It does not reason, calculate, hold a conversation across turns or follow instructions.
- With 12,194,688 parameters and a 384-token context it is not comparable to general language models and is not meant to be.
- The refusal behaviour is learned from examples, not guaranteed: an out-of-scope question can still get an in-scope-looking answer.
Reproduce
See the code repository: uv sync, scripts/smoke.sh for the end-to-end rehearsal, and the README
section "Reproduce" for the release recipe.
- Downloads last month
- 71
Evaluation results
- exact_match (%, float32, open-book, retrieved facts + ranking prior) on Brasa held-out questions, final split (read once)self-reported70.200
- f1 (%, float32, open-book, retrieved facts + ranking prior) on Brasa held-out questions, final split (read once)self-reported78.500