LAM8B / README.md

docs: LAM8B branding + full eval metrics (base vs FT)

0cd4c43 verified about 18 hours ago

17.2 kB

license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
thumbnail: https://huggingface.co/Akhrots/LAM8B/resolve/main/logo.png
language:
  - en
tags:
  - memory
  - long-term-memory
  - retrieval-augmented-generation
  - qlora
  - abstention
  - hallucination-reduction
  - locomo
  - longmemeval
datasets:
  - snap-research/locomo
  - xiaowu0162/longmemeval
model-index:
  - name: LAM8B
    results:
      - task:
          type: question-answering
          name: Memory QA (answer task)
        dataset:
          name: LoCoMo (held-out QA)
          type: locomo
          split: test
        metrics:
          - type: f1
            value: 0.6599
            name: Token F1
          - type: exact_match
            value: 0.365
            name: Exact Match
          - type: f1
            value: 0.8472
            name: Abstention F1
          - type: precision
            value: 0.8026
            name: Abstention Precision
          - type: recall
            value: 0.8971
            name: Abstention Recall
      - task:
          type: question-answering
          name: Memory QA (answer task)
        dataset:
          name: LongMemEval (reserved, never trained on)
          type: longmemeval
          split: reserved
        metrics:
          - type: f1
            value: 0.4843
            name: Token F1
          - type: exact_match
            value: 0.3264
            name: Exact Match
          - type: f1
            value: 0.381
            name: Abstention F1
      - task:
          type: question-answering
          name: Memory QA (answer task)
        dataset:
          name: GML memory eval (LoCoMo + LongMemEval reserved)
          type: mixed
          split: test
        metrics:
          - type: f1
            value: 0.6061
            name: Token F1
          - type: exact_match
            value: 0.3532
            name: Exact Match
          - type: f1
            value: 0.7879
            name: Abstention F1
          - type: accuracy
            value: 1
            name: Output-contract validity

LAM8B 🐏

the memory model that knows when it doesn't know.
_{Large Akhrots Model · by Tronocity Labs}
_{most models would rather hallucinate than admit a gap. this one won't. 💅}

✨ the vibe

your assistant has a memory problem. it either forgets what you told it, or worse — it makes something up and says it with its whole chest.

LAM8B does both halves of the job in one set of weights:

	what it does
✍️ writes	decides what's worth storing, and whether a new fact supersedes an old one
🔍 reads	answers from what's stored — or tells you it's not in there

that second half is the whole point. ⬇️

🧩 where it sits: LAM8B lives behind your retriever. your system pulls candidate memories, LAM8B reads them and answers or abstains. it is not a general chat model — don't drop it in as one.

📈 the glow-up

same prompts, same decoding, same harness. only the weights changed.

metric	un-tuned base	LAM8B
token F1	`█████░░░░░░░` 0.376	`███████░░░░░` 0.606	🟢 +61%
exact match	`██░░░░░░░░░░` 0.177	`████░░░░░░░░` 0.353	🟢 2.0×
abstention F1	`██████░░░░░░` 0.541	`█████████░░░` 0.788	🟢 +0.247
abstention precision	`█████░░░░░░░` 0.378	`█████████░░░` 0.714	🔥 +0.336
abstention recall	`███████████░` 0.946	`███████████░` 0.878	🟡 −0.068
wrongly refused	115 / 544	26 / 544	🔥 −77%
contract-valid output	100%	100%	✅ tie
speed	2.1 s/ex	1.6 s/ex	🟢 −24%

_{n = 544 held-out answer-task examples.}

🎯 the party trick: it shuts up when it should

here's the trap. the un-tuned base has 94.6% abstention recall — it catches almost every unanswerable question. sounds elite. it is not.

precision is 37.8%. it refuses constantly. so of course it catches the real gaps — it's refusing everything. 115 out of 544 questions it could have answered, it didn't. that's not "safe." that's a model that shrugs at you.

LAM8B trades 6.7 points of recall for 33.6 points of precision:

                  precision                     recall
base   0.378  █████░░░░░░░░░░░       0.946  ███████████████░
LAM8B  0.714  ███████████░░░░░       0.878  ██████████████░░
                 ▲ +0.336                       ▼ −0.068

wrongly-refused questions: 115 → 26. on the dedicated abstention split (n=74) both models hold 100% precision — base scores 0.972 abstain-F1 there, LAM8B 0.935. so the base model's whole problem is on questions that had answers. 💀

🧾 receipts (per category)

sorted by how hard it glowed up. over = answerable questions it wrongly refused.

category	n	base	LAM8B	Δ F1	over
🥇 adversarial	76	`██░░░░░░░░░░` 0.183	`████████░░░░` 0.673	+0.489	44 → 6
🥈 temporal	93	`████░░░░░░░░` 0.303	`███████░░░░░` 0.586	+0.282	18 → 2
🥉 single_hop	140	`███████░░░░░` 0.545	`█████████░░░` 0.734	+0.189	16 → 3
open_domain	13	`████░░░░░░░░` 0.326	`██████░░░░░░` 0.468	+0.141	3 → 3
multi_hop	94	`███░░░░░░░░░` 0.283	`█████░░░░░░░` 0.418	+0.135	21 → 3
knowledge_update	20	`██████░░░░░░` 0.508	`████████░░░░` 0.634	+0.125	2 → 0
single_hop_assistant	21	`█████████░░░` 0.785	`██████████░░` 0.842	+0.056	0 → 0
💀 preference	13	`█░░░░░░░░░░░` 0.060	`█░░░░░░░░░░░` 0.067	+0.007	11 → 9

adversarial is the flex. 🥇 same-domain distractors, where refusing takes actual judgement instead of vibes. base model basically gave up (0.183). LAM8B: 0.673, and 44 wrong refusals collapse to 6.

preference is the L. 💀 it did not improve. see keeping it real.

📊 exact-match numbers too (click)

category	n	EM (base)	EM (LAM8B)
adversarial	76	0.1053	0.4211
temporal	93	0.0753	0.2473
single_hop	140	0.2643	0.4714
open_domain	13	0.2308	0.3846
multi_hop	94	0.0957	0.1809
knowledge_update	20	0.4000	0.4500
single_hop_assistant	21	0.5238	0.6667
preference	13	0.0000	0.0000

🔬 the honest split

source	n	F1 (base)	F1 (LAM8B)	EM	abstain P	abstain R	abstain F1	over
LoCoMo	394	`████░░░░░░░░` 0.372	`████████░░░░` 0.660	0.150 → 0.365	0.444 → 0.803	0.941 → 0.897	0.604 → 0.847	80 → 15
LongMemEval 🔒	150	`█████░░░░░░░` 0.384	`██████░░░░░░` 0.484	0.236 → 0.326	0.146 → 0.267	1.000 → 0.667	0.255 → 0.381	35 → 11

🔒 quote this one. those 150 LongMemEval questions were held in reserve and never trained on — longmemeval_oracle and _s share the same 500 questions, so training on either contaminates the other. +0.10 F1 on genuinely unseen data is the real generalization number, not the +0.29 on LoCoMo.

also notice base's 1.000 abstention recall on LongMemEval with 0.146 precision. that's not skill. that's a model saying "idk" to everything. 🫠

⚡ quickstart

pip install transformers torch

from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "Akhrots/LAM8B"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(
    MODEL,
    trust_remote_code=True,   # ← required: LAM registers its own model class
    torch_dtype="float16",
    device_map="auto",
)

SYSTEM = (
    "You answer questions using only the user's stored memories, which are "
    "gathered from every app they use.\n"
    "Rules:\n"
    "- Use only the memories provided. Never invent or assume.\n"
    "- When memories conflict, trust the most recent and answer with that.\n"
    "- Combine memories when the answer needs more than one.\n"
    "- Respect timestamps for questions about when, before, after or latest.\n"
    "- Read negation carefully: a memory saying something stopped is not "
    "evidence that it is still true.\n"
    "- If the memories do not contain the answer, reply exactly: "
    "I don't have that in your memory.\n"
    "Answer concisely — no preamble, no restating the question."
)

# memories render as:  [id] (timestamp via source) text
memories = (
    "[m1] (2024-03-02 via slack) Priya moved the launch review to Thursday.\n"
    "[m2] (2024-02-11 via gmail) Priya scheduled the launch review for Tuesday."
)
user = f"Memories:\n{memories}\n\nQuestion: When is the launch review?"

msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}]
prompt = tok.apply_chat_template(
    msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device), max_new_tokens=128)
print(tok.decode(out[0][len(tok(prompt).input_ids):], skip_special_tokens=True))
# -> "Thursday."   ← m1 is newer, so m2 loses. that's the supersession behaviour.

serve it (weights are merged fp16, no adapter juggling):

vllm serve Akhrots/LAM8B --served-model-name lam8b --trust-remote-code

🚨 three rules, don't skip

0. trust_remote_code=True is mandatory. LAM registers its own model class via modeling_lam.py. without the flag, loading fails outright.

1. the prompt is load-bearing. system prompt, [id] (time via source) memory format, and the exact abstention string I don't have that in your memory. are all part of the trained contract. drift the prompt → quality drops silently, no error.

2. thinking stays OFF. LAM8B is a hybrid-thinking architecture and was trained with enable_thinking=False. flip it on at serve time and you're measuring a different model than the one that was trained.

🧰 four tasks, one model

task	mode	output
`answer`	read	text, or the exact abstention string
`write_op`	write	JSON — `insert` / `supersede` / `merge` / `skip`
`extract`	write	JSON — entities + attributes
`resolve`	read	JSON — `{"kept": [...], "superseded": [...], "reason": str}`

_{only answer is scored above — see keeping it real for why the write-path numbers aren't published.}

🔧 how it was cooked

training config


base	open Apache-2.0 weights
method	QLoRA, 4-bit NF4, fp16 compute
LoRA r / α / dropout	32 / 64 / 0.05
target modules	`q,k,v,o,gate,up,down_proj` (all of them)
LR	1e-4
batch × grad-accum	1 × 16 (effective 16)
epochs	1
hardware	a single free Kaggle T4 😭 ~9 h/epoch
release format	merged fp16 (vLLM / llama.cpp / TGI ready)

the corpus — 12,783 train / 1,640 test

shard	train	source
`write`	7,921	synthetic `write_op` + `extract`
`resolve`	2,250	synthetic read-time conflict resolution
`locomo`	1,582	LoCoMo QA
`abstain`	720	synthetic hard abstention + answerable controls
`longmemeval`	315	LongMemEval oracle (MIT)

the one design choice that matters: every hard-abstention example ships with a twin — identical topical clutter, but the queried fact is present. without those controls the model just learns a new shortcut ("in-domain clutter → refuse") instead of learning to actually check. the only signal separating the labels is whether the fact is in the set.

everything here is --visibility public — synthetic + public benchmarks only. models memorise training data, so real user memories live behind a separate flag and a separate private adapter trained on top of these weights. no one's DMs are in these weights. 🔐

how it was scored

deterministic, lexical, no LLM judge, no API key — reproducible and free.

token F1 / EM after normalization (lowercase, strip articles + punctuation)
abstention P/R matched against the exact contract string, case- and trailing-period-insensitive
contract validity = fraction of outputs parsing against their task schema. both models hit 1.000 — the base model's problem is content, not format
test file is shuffled before slicing (it's concatenated by source, so slicing raw would make every per-category number a lie)

python scripts/memory_sft/eval_gate.py \
    --model Akhrots/LAM8B \
    --data data/sft/mixed/test.jsonl \
    --out reports/ft.json --compare reports/base.json

🫡 keeping it real

no model card should be all W's. here are the L's, on the record:

	the L
💀 preference questions are cooked	F1 0.067, EM 0.000, 9 of 13 still wrongly refused. no shard targets subjective recall and it shows. don't use this for "what does the user like."
🪫 multi-hop is the weakest working category	0.418. up 48% from base, still way under single-hop's 0.734.
🎯 *these scores are the reader, not the retriever*	every number here measures answering given retrieved memories. your end-to-end quality also depends on your retrieval stack — a bad retriever will bottleneck this model no matter how good it is.
🔍 lexical metrics undersell it	token F1 scores "Stripe" and "we use Stripe" as different answers. treat these as run-to-run comparisons, not absolute quality.
✍️ write-path numbers aren't published	the `write_op`/`extract`/`resolve` eval shares generators with its training data, so it measures the generator as much as the model. validating against real production memories is the top open risk.
🧩 abstention doesn't fully transfer	0.381 abstain-F1 on LongMemEval vs 0.847 on LoCoMo. training teaches total absence; LongMemEval tests partial information ("you mentioned X but not Y"). different problem shape.
⏳ one epoch	a T4 caps one Kaggle session at ~1 epoch over this corpus. whether epochs 2–3 leave gains on the table is untested.
🌍 english only	both benchmarks and every synthetic generator are English.

📜 license & credits

Apache-2.0. LAM8B is a fine-tune of open, Apache-2.0-licensed weights and the derivative is released under the same license. training data is synthetic + LoCoMo + LongMemEval (MIT).

@misc{lam8b,
  title  = {LAM8B: a joint read/write memory model with calibrated abstention},
  author = {Tronocity Labs},
  year   = {2026},
  url    = {https://huggingface.co/Akhrots/LAM8B}
}

_{one T4 · zero GPU budget 🐏}

_{if it saved you from a hallucinated answer, leave a ❤️ on the repo.}