brasa-12m

Code: https://github.com/yanzino6/brasa

Brasa is a small GPT-style language model written and trained from scratch in PyTorch: the transformer, the byte-level BPE tokenizer, the training loop, the retriever and the browser inference engine are all hand-written, with no pretrained weights and no model libraries. It answers questions about one person, in Portuguese or English, from a small curated knowledge base, and it is small enough to run entirely in a web browser.

Still from scratch: the weights start random and are first trained by this repository's own code as a plain language model on public text (10,000 steps, 246M tokens seen, 55 minutes on the same laptop, validation perplexity 20.4), then on the question corpus. No pretrained weights or tokenizers from anyone else are involved. Text: https://huggingface.co/datasets/wikimedia/wikipedia at revision b04c8d1c, dump 2023-11-01, licence CC BY-SA 3.0 and GFDL (Wikipedia text); pt: 22,631 articles, 160 MB (20231101.pt/train-00000-of-00006.parquet), en: 32,049 articles, 160 MB (20231101.en/train-00000-of-00041.parquet).

It is a narrow model on purpose. It is not a general assistant: it knows nothing outside its knowledge base and is trained to say so.

Parameters 12,194,688
Architecture decoder-only transformer, 6 layers, width 384, 6 heads, RMSNorm (pre-norm), rotary position embeddings, SwiGLU, tied embeddings, no biases
Context length 384 tokens
Tokenizer byte-level BPE trained from scratch, 4096 tokens (7 control tokens)
Languages Portuguese, English
Training 6,000 steps, AdamW, cosine schedule, 77 min on mps
License MIT

Files

File Bytes What
config.json 1,182 model configuration, prompt format, special tokens, context length
model-card.json 1,308,665 machine-readable evaluation (every number of this card comes from it)
model.safetensors 48,782,920 float32 weights of the released checkpoint, tied output matrix stored once (rows labelled float32 in the evaluation)
retrieval.json 10,175,004 BM25 index, fact vectors and the curated facts the model answers from
tokenizer.json 38,945 BPE merges and control tokens
web/manifest.json 2,397 manifest of the browser bundle (configuration, sizes, checksums)
web/model.f16.bin 24,442,308 float16 weights in the TGPT container (rows labelled float16 in the evaluation)
web/model.i8.bin 12,364,896 int8 weights, one scale per row: the bundle the browser engine loads (rows labelled int8)

The evaluation tables below have a Weights column: float32 is model.safetensors, float16 is web/model.f16.bin and int8 is web/model.i8.bin, each evaluated as stored (the quantised files de-quantised to float32 for scoring). The two web/*.bin files use the single-file "TGPT" container described in the code repository (JSON header with the configuration, the prompt format and the tokenizer, followed by the tensors); int8 stores every matrix with one scale per output row.

How to use

In Python, with the brasa package from the code repository (https://github.com/yanzino6/brasa) and PyTorch:

pip install "git+https://github.com/yanzino6/brasa" safetensors huggingface_hub
import json

from huggingface_hub import snapshot_download
from safetensors.torch import load_file

from brasa.config import GPTConfig, PromptConfig
from brasa.encoder import DenseRetriever
from brasa.model import GPT
from brasa.retrieval import Bm25Retriever
from brasa.sample import answer
from brasa.tokenizer import BPETokenizer

folder = snapshot_download("Yanzino6/brasa-12m")  # or the path of a local copy of this repository
config = json.load(open(f"{folder}/config.json"))

model = GPT(GPTConfig(**{k: v for k, v in config["model"].items() if k != "params"}))
state = load_file(f"{folder}/model.safetensors")
state["lm_head.weight"] = state["tok_emb.weight"]  # tied embeddings
model.load_state_dict(state)
model.eval()

tok = BPETokenizer.load(f"{folder}/tokenizer.json")
index = Bm25Retriever.load(f"{folder}/retrieval.json")
retriever = DenseRetriever.from_index(model, tok, index)  # BM25 fused with the model's own ranking, as on the site

# Retrieval, open-book prompt, then greedy decoding restricted to the retrieved facts and the fixed answers,
# with the retriever's ranking as a prior (the prior stored in retrieval.json): the site's decoding.
out = answer(model, tok, retriever, "Onde o Yan estuda?", PromptConfig(**config["prompt"]), constrain="retrieved")
print(out["answer"])
for hit in out["hits"]:
    print(f"  [{hit.score:.2f}] {hit.text}")

In the browser, with the engine from the code repository (no dependencies, WebAssembly SIMD with a plain JavaScript fallback); the int8 bundle is web/model.i8.bin:

import { BrasaClient } from './engine/index.js';

const client = new BrasaClient(new Worker(new URL('./engine/worker.js', import.meta.url), { type: 'module' }));
await client.load({ model: 'web/model.i8.bin', wasm: 'kernel.wasm', retrieval: 'retrieval.json' });
const run = client.ask('Onde ele mora?', {}, { onToken: (t) => console.log(t.delta) });
console.log((await run.done).text);

Prompt format

open-book    <|bos|><|ctx|>fact<|sep|>fact<|sep|>fact<|q|>question<|a|>answer<|eos|>
closed-book  <|bos|><|q|>question<|a|>answer<|eos|>

Facts are ranked by two signals added together: BM25 over the knowledge base (retrieval.json: words and character trigrams of the facts, their topic names and the training questions) and the model's own opinion. The same network that writes the answer is also trained, with a contrastive loss, to put a question next to the fact that answers it: the browser runs it once over the question, pools its final hidden states into one vector and compares it with the precomputed vectors of the facts (int8, inside retrieval.json; fusion weight 1.5). On held-out test questions the gold fact is the first hit for 66.2% (BM25 alone: 49.8%) and among the first four for 89.8% (76.9%). Up to 4 facts are put in the prompt. Out-of-scope questions are answered with a short refusal in the language of the question.

Evaluation

Greedy decoding. "Seen" are training questions; "held-out" are questions whose wording never occurs in training (written to be phrased differently: most share less than a third of their words with every training phrasing of the same fact), plus small talk and out-of-scope questions never seen.

Decoding modes. Free: the model writes whatever it wants. Any curated sentence: every step may only continue one of the curated sentences of the knowledge base (or a fixed greeting / refusal), so the answer is always a real fact, possibly the wrong one. Retrieved facts only: the same, restricted to the facts the retriever put in the prompt; every statement and number in the answer is then one the visitor can see in the sources. Retrieved facts + ranking prior (the site's default): the same, and the retriever's ranking counts as a prior on the model's choice, so a fact the retriever put lower has to be that much more likely for the model to win; it costs no extra computation. Retriever's first fact uses no model: it is the baseline the open-book rows have to beat, and it can never decline a question.

EM is normalised exact match against the one gold answer, F1 is token-level, "key spans" is the share of required names and numbers present in the answer, "bad numbers" is the share of numbers in the answers that appear neither in the gold answer, nor in the retrieved facts, nor in the question (with constrained decoding these are numbers of other real facts, not inventions).

Weights Mode Questions n EM F1 EM in scope Key spans Refusal P Refusal R Language Bad numbers
float32 open-book, free seen (training) 300 97.7% 98.3% 97.6% 99.4% 100.0% 100.0% 100.0% 0.0%
float32 open-book, free held-out (validation) 1250 64.8% 75.3% 63.5% 78.6% 88.3% 82.9% 100.0% 0.7%
float32 open-book, free held-out (test) 1278 63.0% 74.4% 60.8% 80.0% 86.5% 87.3% 99.8% 1.4%
float32 open-book, free held-out (final, read once) 1210 65.5% 75.2% 63.8% 83.1% 84.9% 81.8% 100.0% 0.6%
float32 closed-book, free seen (training) 300 93.0% 95.1% 92.8% 96.7% 100.0% 100.0% 100.0% 1.2%
float32 closed-book, free held-out (validation) 1250 46.0% 61.4% 43.1% 66.7% 80.2% 89.0% 99.8% 33.1%
float32 closed-book, free held-out (test) 1278 46.5% 62.1% 42.6% 69.4% 82.3% 89.1% 99.5% 31.8%
float32 closed-book, free held-out (final, read once) 1210 46.0% 61.1% 42.4% 70.2% 81.2% 82.7% 99.8% 29.4%
float32 open-book, any curated sentence seen (training) 300 97.7% 98.3% 97.6% 99.4% 100.0% 100.0% 100.0% 0.0%
float32 open-book, any curated sentence held-out (validation) 1250 64.8% 75.2% 63.5% 78.6% 88.3% 82.9% 100.0% 0.8%
float32 open-book, any curated sentence held-out (test) 1278 63.0% 74.4% 60.8% 80.0% 86.5% 87.3% 99.8% 1.4%
float32 open-book, any curated sentence held-out (final, read once) 1210 65.5% 75.2% 63.9% 83.1% 84.9% 81.8% 100.0% 0.6%
float32 closed-book, any curated sentence seen (training) 300 93.0% 95.1% 92.8% 96.7% 100.0% 100.0% 100.0% 1.2%
float32 closed-book, any curated sentence held-out (validation) 1250 46.1% 61.2% 43.1% 66.6% 80.2% 89.0% 99.8% 33.7%
float32 closed-book, any curated sentence held-out (test) 1278 46.5% 61.9% 42.6% 69.3% 82.3% 89.1% 99.5% 32.3%
float32 closed-book, any curated sentence held-out (final, read once) 1210 46.2% 61.0% 42.5% 69.9% 82.0% 82.7% 99.8% 29.0%
float32 open-book, retrieved facts only seen (training) 300 97.7% 98.3% 97.6% 99.4% 100.0% 100.0% 100.0% 0.0%
float32 open-book, retrieved facts only held-out (validation) 1250 65.0% 75.4% 63.7% 78.6% 88.3% 82.9% 100.0% 0.0%
float32 open-book, retrieved facts only held-out (test) 1278 63.1% 74.6% 61.0% 80.3% 86.5% 87.3% 99.8% 0.0%
float32 open-book, retrieved facts only held-out (final, read once) 1210 66.0% 75.5% 64.4% 83.2% 84.1% 81.8% 100.0% 0.0%
float32 open-book, retrieved facts + ranking prior (site default) seen (training) 300 99.3% 99.6% 99.3% 100.0% 100.0% 100.0% 100.0% 0.0%
float32 open-book, retrieved facts + ranking prior (site default) held-out (validation) 1250 70.8% 79.7% 69.9% 83.1% 88.5% 84.2% 100.0% 0.0%
float32 open-book, retrieved facts + ranking prior (site default) held-out (test) 1278 67.4% 77.5% 65.6% 82.7% 86.5% 87.3% 99.8% 0.0%
float32 open-book, retrieved facts + ranking prior (site default) held-out (final, read once) 1210 70.2% 78.5% 69.1% 85.7% 84.1% 81.8% 100.0% 0.0%
float32 retriever's first fact (no model) seen (training) 300 94.0% 95.7% 96.2% 99.0% n/a 0.0% 100.0% 0.0%
float32 retriever's first fact (no model) held-out (validation) 1250 62.8% 73.3% 67.2% 83.2% n/a 0.0% 100.0% 0.0%
float32 retriever's first fact (no model) held-out (test) 1278 57.9% 69.6% 63.4% 81.8% n/a 0.0% 99.9% 0.0%
float32 retriever's first fact (no model) held-out (final, read once) 1210 62.6% 71.2% 68.8% 86.0% n/a 0.0% 99.9% 0.0%
float16 open-book, free held-out (validation) 1250 64.8% 75.3% 63.5% 78.6% 88.3% 82.9% 100.0% 0.7%
float16 open-book, free held-out (test) 1278 63.0% 74.4% 60.8% 80.0% 86.5% 87.3% 99.8% 1.4%
float16 open-book, free held-out (final, read once) 1210 65.4% 75.1% 63.7% 83.1% 84.1% 81.8% 100.0% 0.6%
float16 closed-book, free held-out (validation) 1250 46.0% 61.4% 43.1% 66.7% 80.2% 89.0% 99.8% 33.1%
float16 closed-book, free held-out (test) 1278 46.5% 62.1% 42.6% 69.4% 82.3% 89.1% 99.5% 31.8%
float16 closed-book, free held-out (final, read once) 1210 46.0% 61.1% 42.4% 70.2% 81.2% 82.7% 99.8% 29.4%
float16 open-book, any curated sentence held-out (validation) 1250 64.8% 75.2% 63.5% 78.6% 88.3% 82.9% 100.0% 0.8%
float16 open-book, any curated sentence held-out (test) 1278 63.0% 74.4% 60.8% 80.0% 86.5% 87.3% 99.8% 1.4%
float16 open-book, any curated sentence held-out (final, read once) 1210 65.5% 75.2% 63.8% 83.1% 84.1% 81.8% 100.0% 0.6%
float16 closed-book, any curated sentence held-out (validation) 1250 46.1% 61.2% 43.1% 66.6% 80.2% 89.0% 99.8% 33.7%
float16 closed-book, any curated sentence held-out (test) 1278 46.5% 61.9% 42.6% 69.3% 82.3% 89.1% 99.5% 32.3%
float16 closed-book, any curated sentence held-out (final, read once) 1210 46.2% 61.0% 42.5% 69.9% 82.0% 82.7% 99.8% 29.0%
float16 open-book, retrieved facts only held-out (validation) 1250 65.0% 75.4% 63.7% 78.6% 88.3% 82.9% 100.0% 0.0%
float16 open-book, retrieved facts only held-out (test) 1278 63.1% 74.6% 61.0% 80.3% 86.5% 87.3% 99.8% 0.0%
float16 open-book, retrieved facts only held-out (final, read once) 1210 65.9% 75.4% 64.3% 83.2% 83.3% 81.8% 100.0% 0.0%
float16 open-book, retrieved facts + ranking prior (site default) held-out (validation) 1250 70.8% 79.7% 69.9% 83.1% 88.5% 84.2% 100.0% 0.0%
float16 open-book, retrieved facts + ranking prior (site default) held-out (test) 1278 67.4% 77.5% 65.6% 82.7% 86.5% 87.3% 99.8% 0.0%
float16 open-book, retrieved facts + ranking prior (site default) held-out (final, read once) 1210 70.2% 78.4% 69.0% 85.7% 83.3% 81.8% 100.0% 0.0%
float16 retriever's first fact (no model) held-out (validation) 1250 62.8% 73.3% 67.2% 83.2% n/a 0.0% 100.0% 0.0%
float16 retriever's first fact (no model) held-out (test) 1278 57.8% 69.5% 63.3% 81.7% n/a 0.0% 99.9% 0.0%
float16 retriever's first fact (no model) held-out (final, read once) 1210 62.6% 71.2% 68.8% 86.0% n/a 0.0% 99.9% 0.0%
int8 open-book, free held-out (validation) 1250 64.6% 75.2% 63.4% 78.6% 88.3% 82.9% 100.0% 0.7%
int8 open-book, free held-out (test) 1278 62.8% 74.3% 60.6% 79.9% 85.7% 87.3% 99.8% 1.4%
int8 open-book, free held-out (final, read once) 1210 65.5% 75.2% 63.9% 83.1% 84.9% 81.8% 100.0% 0.6%
int8 closed-book, free held-out (validation) 1250 46.2% 61.5% 43.2% 66.7% 82.0% 89.0% 99.8% 33.1%
int8 closed-book, free held-out (test) 1278 46.8% 62.3% 43.0% 69.5% 82.3% 89.1% 99.5% 31.4%
int8 closed-book, free held-out (final, read once) 1210 46.1% 61.1% 42.6% 70.4% 80.2% 80.9% 99.8% 28.8%
int8 open-book, any curated sentence held-out (validation) 1250 64.6% 75.1% 63.4% 78.6% 88.3% 82.9% 100.0% 0.8%
int8 open-book, any curated sentence held-out (test) 1278 62.8% 74.3% 60.6% 79.9% 85.7% 87.3% 99.8% 1.4%
int8 open-book, any curated sentence held-out (final, read once) 1210 65.6% 75.3% 64.0% 83.1% 84.9% 81.8% 100.0% 0.6%
int8 closed-book, any curated sentence held-out (validation) 1250 46.2% 61.4% 43.3% 66.6% 82.0% 89.0% 99.8% 33.6%
int8 closed-book, any curated sentence held-out (test) 1278 46.8% 62.1% 43.0% 69.4% 82.3% 89.1% 99.5% 31.9%
int8 closed-book, any curated sentence held-out (final, read once) 1210 46.3% 61.1% 42.8% 70.0% 80.9% 80.9% 99.8% 28.5%
int8 open-book, retrieved facts only held-out (validation) 1250 64.8% 75.3% 63.5% 78.6% 88.3% 82.9% 100.0% 0.0%
int8 open-book, retrieved facts only held-out (test) 1278 63.0% 74.5% 60.8% 80.0% 85.7% 87.3% 99.8% 0.0%
int8 open-book, retrieved facts only held-out (final, read once) 1210 66.0% 75.5% 64.4% 83.2% 84.1% 81.8% 100.0% 0.0%
int8 open-book, retrieved facts + ranking prior (site default) held-out (validation) 1250 70.7% 79.6% 69.8% 83.1% 88.5% 84.2% 100.0% 0.0%
int8 open-book, retrieved facts + ranking prior (site default) held-out (test) 1278 67.3% 77.5% 65.5% 82.7% 85.7% 87.3% 99.8% 0.0%
int8 open-book, retrieved facts + ranking prior (site default) held-out (final, read once) 1210 70.3% 78.5% 69.2% 85.7% 84.1% 81.8% 100.0% 0.0%
int8 retriever's first fact (no model) held-out (validation) 1250 62.7% 73.3% 67.1% 83.2% n/a 0.0% 100.0% 0.0%
int8 retriever's first fact (no model) held-out (test) 1278 57.7% 69.4% 63.1% 81.4% n/a 0.0% 99.9% 0.0%
int8 retriever's first fact (no model) held-out (final, read once) 1210 62.6% 71.2% 68.8% 85.9% n/a 0.0% 99.9% 0.0%

Exact match is strict: with dozens of closely related facts per project, a neighbouring fact of the right topic is often a reasonable answer. The next table adds what a visitor experiences: whether the answer is a curated sentence at all, whether it is about the right topic, and how often the retriever had the gold fact among its first hits.

Weights Mode Questions Exact Curated sentence Right topic Gold fact in top 1 / 2 / 4 Exact when retrieved / when missed
float32 open-book, free held-out (validation) 64.8% 99.6% 93.5% 70.2% / 85.7% / 92.0% 69.2% / 3.3%
float32 open-book, free held-out (test) 63.0% 99.9% 93.1% 66.2% / 81.5% / 89.8% 68.7% / 3.5%
float32 open-book, free held-out (final, read once) 65.5% 99.9% 94.1% 72.1% / 85.6% / 93.2% 69.9% / 1.4%
float32 closed-book, free held-out (validation) 46.0% 98.7% 89.5% n/a n/a
float32 closed-book, free held-out (test) 46.5% 99.1% 88.4% n/a n/a
float32 closed-book, free held-out (final, read once) 46.0% 98.7% 89.7% n/a n/a
float32 open-book, any curated sentence held-out (validation) 64.8% 99.9% 93.7% 70.2% / 85.7% / 92.0% 69.2% / 3.3%
float32 open-book, any curated sentence held-out (test) 63.0% 100.0% 93.2% 66.2% / 81.5% / 89.8% 68.7% / 3.5%
float32 open-book, any curated sentence held-out (final, read once) 65.5% 100.0% 94.2% 72.1% / 85.6% / 93.2% 70.0% / 1.4%
float32 closed-book, any curated sentence held-out (validation) 46.1% 99.9% 90.1% n/a n/a
float32 closed-book, any curated sentence held-out (test) 46.5% 100.0% 89.0% n/a n/a
float32 closed-book, any curated sentence held-out (final, read once) 46.2% 99.9% 90.3% n/a n/a
float32 open-book, retrieved facts only held-out (validation) 65.0% 99.8% 93.3% 70.2% / 85.7% / 92.0% 69.7% / 0.0%
float32 open-book, retrieved facts only held-out (test) 63.1% 99.9% 93.2% 66.2% / 81.5% / 89.8% 69.3% / 0.0%
float32 open-book, retrieved facts only held-out (final, read once) 66.0% 100.0% 94.1% 72.1% / 85.6% / 93.2% 70.6% / 0.0%
float32 open-book, retrieved facts + ranking prior (site default) held-out (validation) 70.8% 99.8% 93.9% 70.2% / 85.7% / 92.0% 76.6% / 0.0%
float32 open-book, retrieved facts + ranking prior (site default) held-out (test) 67.4% 99.9% 93.5% 66.2% / 81.5% / 89.8% 74.7% / 0.0%
float32 open-book, retrieved facts + ranking prior (site default) held-out (final, read once) 70.2% 100.0% 94.6% 72.1% / 85.6% / 93.2% 75.9% / 0.0%
float32 retriever's first fact (no model) held-out (validation) 62.8% 100.0% 94.1% 70.2% / 85.7% / 92.0% 76.4% / 0.0%
float32 retriever's first fact (no model) held-out (test) 57.9% 100.0% 93.2% 66.2% / 81.5% / 89.8% 73.7% / 0.0%
float32 retriever's first fact (no model) held-out (final, read once) 62.6% 100.0% 94.4% 72.1% / 85.6% / 93.2% 77.3% / 0.0%
float16 open-book, free held-out (validation) 64.8% 99.6% 93.5% 70.2% / 85.7% / 92.0% 69.2% / 3.3%
float16 open-book, free held-out (test) 63.0% 99.9% 93.1% 66.1% / 81.5% / 89.8% 68.7% / 3.5%
float16 open-book, free held-out (final, read once) 65.4% 99.9% 93.9% 72.1% / 85.6% / 93.2% 69.8% / 1.4%
float16 closed-book, free held-out (validation) 46.0% 98.7% 89.5% n/a n/a
float16 closed-book, free held-out (test) 46.5% 99.1% 88.4% n/a n/a
float16 closed-book, free held-out (final, read once) 46.0% 98.7% 89.7% n/a n/a
float16 open-book, any curated sentence held-out (validation) 64.8% 99.9% 93.7% 70.2% / 85.7% / 92.0% 69.2% / 3.3%
float16 open-book, any curated sentence held-out (test) 63.0% 100.0% 93.2% 66.1% / 81.5% / 89.8% 68.7% / 3.5%
float16 open-book, any curated sentence held-out (final, read once) 65.5% 100.0% 94.0% 72.1% / 85.6% / 93.2% 69.9% / 1.4%
float16 closed-book, any curated sentence held-out (validation) 46.1% 99.9% 90.1% n/a n/a
float16 closed-book, any curated sentence held-out (test) 46.5% 100.0% 89.0% n/a n/a
float16 closed-book, any curated sentence held-out (final, read once) 46.2% 99.9% 90.3% n/a n/a
float16 open-book, retrieved facts only held-out (validation) 65.0% 99.8% 93.3% 70.2% / 85.7% / 92.0% 69.7% / 0.0%
float16 open-book, retrieved facts only held-out (test) 63.1% 99.9% 93.2% 66.1% / 81.5% / 89.8% 69.3% / 0.0%
float16 open-book, retrieved facts only held-out (final, read once) 65.9% 100.0% 93.9% 72.1% / 85.6% / 93.2% 70.5% / 0.0%
float16 open-book, retrieved facts + ranking prior (site default) held-out (validation) 70.8% 99.8% 93.9% 70.2% / 85.7% / 92.0% 76.6% / 0.0%
float16 open-book, retrieved facts + ranking prior (site default) held-out (test) 67.4% 99.9% 93.5% 66.1% / 81.5% / 89.8% 74.7% / 0.0%
float16 open-book, retrieved facts + ranking prior (site default) held-out (final, read once) 70.2% 100.0% 94.5% 72.1% / 85.6% / 93.2% 75.8% / 0.0%
float16 retriever's first fact (no model) held-out (validation) 62.8% 100.0% 94.1% 70.2% / 85.7% / 92.0% 76.4% / 0.0%
float16 retriever's first fact (no model) held-out (test) 57.8% 100.0% 93.2% 66.1% / 81.5% / 89.8% 73.6% / 0.0%
float16 retriever's first fact (no model) held-out (final, read once) 62.6% 100.0% 94.4% 72.1% / 85.6% / 93.2% 77.3% / 0.0%
int8 open-book, free held-out (validation) 64.6% 99.6% 93.5% 70.1% / 85.7% / 91.9% 69.0% / 3.3%
int8 open-book, free held-out (test) 62.8% 99.9% 93.1% 65.9% / 81.5% / 89.8% 68.6% / 2.6%
int8 open-book, free held-out (final, read once) 65.5% 99.9% 94.0% 72.1% / 85.7% / 93.1% 70.0% / 1.4%
int8 closed-book, free held-out (validation) 46.2% 98.6% 89.3% n/a n/a
int8 closed-book, free held-out (test) 46.8% 99.1% 88.3% n/a n/a
int8 closed-book, free held-out (final, read once) 46.1% 98.8% 90.1% n/a n/a
int8 open-book, any curated sentence held-out (validation) 64.6% 99.9% 93.7% 70.1% / 85.7% / 91.9% 69.0% / 3.3%
int8 open-book, any curated sentence held-out (test) 62.8% 100.0% 93.2% 65.9% / 81.5% / 89.8% 68.6% / 2.6%
int8 open-book, any curated sentence held-out (final, read once) 65.6% 100.0% 94.1% 72.1% / 85.7% / 93.1% 70.1% / 1.4%
int8 closed-book, any curated sentence held-out (validation) 46.2% 99.9% 90.0% n/a n/a
int8 closed-book, any curated sentence held-out (test) 46.8% 100.0% 89.0% n/a n/a
int8 closed-book, any curated sentence held-out (final, read once) 46.3% 99.9% 90.6% n/a n/a
int8 open-book, retrieved facts only held-out (validation) 64.8% 99.8% 93.3% 70.1% / 85.7% / 91.9% 69.5% / 0.0%
int8 open-book, retrieved facts only held-out (test) 63.0% 99.9% 93.2% 65.9% / 81.5% / 89.8% 69.1% / 0.0%
int8 open-book, retrieved facts only held-out (final, read once) 66.0% 100.0% 94.1% 72.1% / 85.7% / 93.1% 70.7% / 0.0%
int8 open-book, retrieved facts + ranking prior (site default) held-out (validation) 70.7% 99.8% 93.9% 70.1% / 85.7% / 91.9% 76.6% / 0.0%
int8 open-book, retrieved facts + ranking prior (site default) held-out (test) 67.3% 99.9% 93.5% 65.9% / 81.5% / 89.8% 74.6% / 0.0%
int8 open-book, retrieved facts + ranking prior (site default) held-out (final, read once) 70.3% 100.0% 94.6% 72.1% / 85.7% / 93.1% 76.1% / 0.0%
int8 retriever's first fact (no model) held-out (validation) 62.7% 100.0% 94.1% 70.1% / 85.7% / 91.9% 76.3% / 0.0%
int8 retriever's first fact (no model) held-out (test) 57.7% 100.0% 93.1% 65.9% / 81.5% / 89.8% 73.4% / 0.0%
int8 retriever's first fact (no model) held-out (final, read once) 62.6% 100.0% 94.3% 72.1% / 85.7% / 93.1% 77.4% / 0.0%

Retriever used in the open-book rows: BM25 fused with the model's own ranking of the facts (cosine of its pooled hidden states, weight 1.5). Ranking prior of the site default: a fact loses weight x (1 - its score / the best score) nats, weight 24.0, chosen on the validation questions. Open-book, held-out test questions: the retriever returned the gold facts for 89.8% of the in-scope questions; exact match was 68.7% when it did and 3.5% when it did not.

Breakdown of open-book, retrieved facts + ranking prior (site default), float32, held-out (final, read once):

Language n Exact F1 Curated sentence Right topic
en 605 71.2% 79.0% 100.0% 94.1%
pt 605 69.3% 78.0% 100.0% 95.0%
Kind of question n Exact F1 Curated sentence Right topic
small talk 50 34.0% 48.1% 100.0% n/a
about the model 44 75.0% 80.2% 100.0% 100.0%
out of scope 110 81.8% 86.0% 100.0% n/a
one fact 1006 70.6% 79.1% 100.0% 94.3%
Topic n Exact F1 Curated sentence Right topic
(none) 160 66.9% 74.2% 100.0% n/a
breakhis 112 69.6% 78.8% 100.0% 100.0%
pad 106 78.3% 85.0% 100.0% 100.0%
crm 96 70.8% 79.1% 100.0% 97.9%
seer 86 76.7% 83.0% 100.0% 100.0%
skills 66 57.6% 67.6% 100.0% 72.7%
isabela 58 56.9% 69.4% 100.0% 98.3%
nina 56 73.2% 80.9% 100.0% 98.2%
prospecting 48 64.6% 77.5% 100.0% 100.0%
brasa 44 75.0% 80.2% 100.0% 100.0%
gjdb 34 73.5% 78.2% 100.0% 100.0%
armv7a 30 66.7% 77.9% 100.0% 100.0%
c-coursework 30 86.7% 90.7% 100.0% 100.0%
simmer 30 76.7% 82.2% 100.0% 96.7%
experience 28 67.9% 80.1% 100.0% 92.9%
algorithms-c 24 91.7% 94.4% 100.0% 100.0%
boundaries 24 62.5% 74.7% 100.0% 70.8%
esp32-cam-car 24 66.7% 77.9% 100.0% 100.0%
process 22 54.5% 65.9% 100.0% 72.7%
profile 22 54.5% 65.2% 100.0% 63.6%
ml-coursework 20 75.0% 83.9% 100.0% 95.0%
projeto-piloto 20 60.0% 70.8% 100.0% 100.0%
ct-junior-clients 16 87.5% 92.6% 100.0% 93.8%
services 16 75.0% 80.8% 100.0% 75.0%
operating-systems 14 71.4% 78.5% 100.0% 78.6%
studies 14 71.4% 83.2% 100.0% 92.9%
contact 10 90.0% 91.2% 100.0% 90.0%

The held-out test questions were also used to choose between training settings, so the test rows are optimistic by selection; the "final" rows, when present, come from a slice of questions that was read once, for this card.

Quantisation: f16: relative weight error 0.000207, largest logit drift 0.00391; i8: relative weight error 0.007449, largest logit drift 0.13656; float16 answers identical to float32: 26153/26166; int8 answers identical to float32: 25941/26166

Training data

525 facts in 26 topics, each in Portuguese and English; 57947 training, 1250 validation and 1278 test question/answer records (52 training questions per fact and language, median); 347,682 training examples of 56,081,323 tokens after open-book / closed-book expansion.

Limitations

  • The model only knows the facts of its knowledge base, and only in the phrasings it can generalise to from its training questions. Unusual wordings can retrieve the wrong fact or get a wrong answer that sounds confident.
  • Answers are short and close to the wording of the knowledge base. It does not reason, calculate, hold a conversation across turns or follow instructions.
  • With 12,194,688 parameters and a 384-token context it is not comparable to general language models and is not meant to be.
  • The refusal behaviour is learned from examples, not guaranteed: an out-of-scope question can still get an in-scope-looking answer.

Reproduce

See the code repository: uv sync, scripts/smoke.sh for the end-to-end rehearsal, and the README section "Reproduce" for the release recipe.

Downloads last month
71
Safetensors
Model size
12.2M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Evaluation results

  • exact_match (%, float32, open-book, retrieved facts + ranking prior) on Brasa held-out questions, final split (read once)
    self-reported
    70.200
  • f1 (%, float32, open-book, retrieved facts + ranking prior) on Brasa held-out questions, final split (read once)
    self-reported
    78.500