--- license: apache-2.0 library_name: transformers pipeline_tag: text-generation thumbnail: https://huggingface.co/Akhrots/LAM8B/resolve/main/logo.png language: - en tags: - memory - long-term-memory - retrieval-augmented-generation - qlora - abstention - hallucination-reduction - locomo - longmemeval datasets: - snap-research/locomo - xiaowu0162/longmemeval model-index: - name: LAM8B results: - task: type: question-answering name: Memory QA (answer task) dataset: name: LoCoMo (held-out QA) type: locomo split: test metrics: - type: f1 value: 0.6599 name: Token F1 - type: exact_match value: 0.3650 name: Exact Match - type: f1 value: 0.8472 name: Abstention F1 - type: precision value: 0.8026 name: Abstention Precision - type: recall value: 0.8971 name: Abstention Recall - task: type: question-answering name: Memory QA (answer task) dataset: name: LongMemEval (reserved, never trained on) type: longmemeval split: reserved metrics: - type: f1 value: 0.4843 name: Token F1 - type: exact_match value: 0.3264 name: Exact Match - type: f1 value: 0.3810 name: Abstention F1 - task: type: question-answering name: Memory QA (answer task) dataset: name: GML memory eval (LoCoMo + LongMemEval reserved) type: mixed split: test metrics: - type: f1 value: 0.6061 name: Token F1 - type: exact_match value: 0.3532 name: Exact Match - type: f1 value: 0.7879 name: Abstention F1 - type: accuracy value: 1.0 name: Output-contract validity ---

LAM8B

LAM8B ๐Ÿ

the memory model that knows when it doesn't know.
Large Akhrots Model ยท by Tronocity Labs
most models would rather hallucinate than admit a gap. this one won't. ๐Ÿ’…

F1 0.606 +61% vs base -77% over-abstentions

8B params Apache 2.0 QLoRA r32 alpha64 1x T4 100% contract valid

--- ## โœจ the vibe your assistant has a memory problem. it either **forgets** what you told it, or worse โ€” it **makes something up** and says it with its whole chest. LAM8B does both halves of the job in one set of weights: | | what it does | |:--|:--| | โœ๏ธ **writes** | decides what's worth storing, and whether a new fact *supersedes* an old one | | ๐Ÿ” **reads** | answers from what's stored โ€” **or tells you it's not in there** | that second half is the whole point. โฌ‡๏ธ > ๐Ÿงฉ **where it sits:** LAM8B lives *behind* your retriever. your system pulls candidate memories, LAM8B reads them and answers or abstains. **it is not a general chat model** โ€” don't drop it in as one. --- ## ๐Ÿ“ˆ the glow-up same prompts, same decoding, same harness. **only the weights changed.** | metric | un-tuned base | **LAM8B** | | |:--|:--|:--|:--| | **token F1** | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` 0.376 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘` **0.606** | ๐ŸŸข **+61%** | | **exact match** | `โ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` 0.177 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` **0.353** | ๐ŸŸข **2.0ร—** | | **abstention F1** | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` 0.541 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘` **0.788** | ๐ŸŸข **+0.247** | | **abstention precision** | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` 0.378 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘` **0.714** | ๐Ÿ”ฅ **+0.336** | | **abstention recall** | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘` 0.946 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘` 0.878 | ๐ŸŸก โˆ’0.068 | | **wrongly refused** | 115 / 544 | **26 / 544** | ๐Ÿ”ฅ **โˆ’77%** | | **contract-valid output** | 100% | 100% | โœ… tie | | **speed** | 2.1 s/ex | **1.6 s/ex** | ๐ŸŸข โˆ’24% | n = 544 held-out answer-task examples. --- ## ๐ŸŽฏ the party trick: it shuts up when it should here's the trap. the un-tuned base has **94.6% abstention recall** โ€” it catches almost every unanswerable question. sounds elite. it is not. > **precision is 37.8%.** it refuses *constantly*. so of course it catches the real gaps โ€” it's refusing everything. **115 out of 544 questions it could have answered, it didn't.** that's not "safe." that's a model that shrugs at you. LAM8B trades **6.7 points of recall** for **33.6 points of precision**: ``` precision recall base 0.378 โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘ 0.946 โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘ LAM8B 0.714 โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘ 0.878 โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘ โ–ฒ +0.336 โ–ผ โˆ’0.068 ``` wrongly-refused questions: **115 โ†’ 26.** on the dedicated abstention split (n=74) both models hold **100% precision** โ€” base scores 0.972 abstain-F1 there, LAM8B 0.935. so the base model's whole problem is on questions that *had* answers. ๐Ÿ’€ --- ## ๐Ÿงพ receipts (per category) sorted by how hard it glowed up. `over` = answerable questions it wrongly refused. | category | n | base | **LAM8B** | ฮ” F1 | over | |:--|--:|:--|:--|--:|:--| | ๐Ÿฅ‡ **adversarial** | 76 | `โ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` 0.183 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘` **0.673** | **+0.489** | 44 โ†’ **6** | | ๐Ÿฅˆ **temporal** | 93 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` 0.303 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘` **0.586** | **+0.282** | 18 โ†’ **2** | | ๐Ÿฅ‰ **single_hop** | 140 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘` 0.545 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘` **0.734** | +0.189 | 16 โ†’ **3** | | **open_domain** | 13 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` 0.326 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` **0.468** | +0.141 | 3 โ†’ **3** | | **multi_hop** | 94 | `โ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` 0.283 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` **0.418** | +0.135 | 21 โ†’ **3** | | **knowledge_update** | 20 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` 0.508 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘` **0.634** | +0.125 | 2 โ†’ **0** | | **single_hop_assistant** | 21 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘` 0.785 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘` **0.842** | +0.056 | 0 โ†’ **0** | | ๐Ÿ’€ **preference** | 13 | `โ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` 0.060 | `โ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` 0.067 | +0.007 | 11 โ†’ **9** | **adversarial is the flex.** ๐Ÿฅ‡ same-domain distractors, where refusing takes actual judgement instead of vibes. base model basically gave up (0.183). LAM8B: **0.673**, and 44 wrong refusals collapse to 6. **preference is the L.** ๐Ÿ’€ it did not improve. see [keeping it real](#-keeping-it-real).
๐Ÿ“Š exact-match numbers too (click) | category | n | EM (base) | **EM (LAM8B)** | |:--|--:|--:|--:| | adversarial | 76 | 0.1053 | **0.4211** | | temporal | 93 | 0.0753 | **0.2473** | | single_hop | 140 | 0.2643 | **0.4714** | | open_domain | 13 | 0.2308 | **0.3846** | | multi_hop | 94 | 0.0957 | **0.1809** | | knowledge_update | 20 | 0.4000 | **0.4500** | | single_hop_assistant | 21 | 0.5238 | **0.6667** | | preference | 13 | 0.0000 | 0.0000 |
--- ## ๐Ÿ”ฌ the honest split | source | n | F1 (base) | **F1 (LAM8B)** | EM | abstain P | abstain R | abstain F1 | over | |:--|--:|:--|:--|:--|:--|:--|:--|:--| | **LoCoMo** | 394 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` 0.372 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘` **0.660** | 0.150 โ†’ **0.365** | 0.444 โ†’ **0.803** | 0.941 โ†’ 0.897 | 0.604 โ†’ **0.847** | 80 โ†’ **15** | | **LongMemEval** ๐Ÿ”’ | 150 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` 0.384 | `โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘` **0.484** | 0.236 โ†’ **0.326** | 0.146 โ†’ **0.267** | 1.000 โ†’ 0.667 | 0.255 โ†’ **0.381** | 35 โ†’ **11** | > ๐Ÿ”’ **quote this one.** those 150 LongMemEval questions were **held in reserve and never trained on** โ€” `longmemeval_oracle` and `_s` share the same 500 questions, so training on either contaminates the other. **+0.10 F1 on genuinely unseen data** is the real generalization number, not the +0.29 on LoCoMo. also notice base's **1.000** abstention recall on LongMemEval with **0.146** precision. that's not skill. that's a model saying "idk" to everything. ๐Ÿซ  --- ## โšก quickstart ```bash pip install transformers torch ``` ```python from transformers import AutoModelForCausalLM, AutoTokenizer MODEL = "Akhrots/LAM8B" tok = AutoTokenizer.from_pretrained(MODEL) model = AutoModelForCausalLM.from_pretrained( MODEL, trust_remote_code=True, # โ† required: LAM registers its own model class torch_dtype="float16", device_map="auto", ) SYSTEM = ( "You answer questions using only the user's stored memories, which are " "gathered from every app they use.\n" "Rules:\n" "- Use only the memories provided. Never invent or assume.\n" "- When memories conflict, trust the most recent and answer with that.\n" "- Combine memories when the answer needs more than one.\n" "- Respect timestamps for questions about when, before, after or latest.\n" "- Read negation carefully: a memory saying something stopped is not " "evidence that it is still true.\n" "- If the memories do not contain the answer, reply exactly: " "I don't have that in your memory.\n" "Answer concisely โ€” no preamble, no restating the question." ) # memories render as: [id] (timestamp via source) text memories = ( "[m1] (2024-03-02 via slack) Priya moved the launch review to Thursday.\n" "[m2] (2024-02-11 via gmail) Priya scheduled the launch review for Tuesday." ) user = f"Memories:\n{memories}\n\nQuestion: When is the launch review?" msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}] prompt = tok.apply_chat_template( msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False ) out = model.generate(**tok(prompt, return_tensors="pt").to(model.device), max_new_tokens=128) print(tok.decode(out[0][len(tok(prompt).input_ids):], skip_special_tokens=True)) # -> "Thursday." โ† m1 is newer, so m2 loses. that's the supersession behaviour. ``` serve it (weights are merged fp16, no adapter juggling): ```bash vllm serve Akhrots/LAM8B --served-model-name lam8b --trust-remote-code ``` ### ๐Ÿšจ three rules, don't skip **0. `trust_remote_code=True` is mandatory.** LAM registers its own model class via `modeling_lam.py`. without the flag, loading fails outright. **1. the prompt is load-bearing.** system prompt, `[id] (time via source)` memory format, and the exact abstention string `I don't have that in your memory.` are all part of the trained contract. drift the prompt โ†’ quality drops silently, no error. **2. thinking stays OFF.** LAM8B is a hybrid-thinking architecture and was trained with `enable_thinking=False`. flip it on at serve time and you're measuring a different model than the one that was trained. ### ๐Ÿงฐ four tasks, one model | task | mode | output | |:--|:--|:--| | `answer` | read | text, or the exact abstention string | | `write_op` | write | JSON โ€” `insert` / `supersede` / `merge` / `skip` | | `extract` | write | JSON โ€” entities + attributes | | `resolve` | read | JSON โ€” `{"kept": [...], "superseded": [...], "reason": str}` | only answer is scored above โ€” see keeping it real for why the write-path numbers aren't published. --- ## ๐Ÿ”ง how it was cooked
training config | | | |:--|:--| | base | open Apache-2.0 weights | | method | QLoRA, 4-bit NF4, fp16 compute | | LoRA r / ฮฑ / dropout | **32 / 64 / 0.05** | | target modules | `q,k,v,o,gate,up,down_proj` (all of them) | | LR | 1e-4 | | batch ร— grad-accum | 1 ร— 16 (effective 16) | | epochs | 1 | | hardware | **a single free Kaggle T4** ๐Ÿ˜ญ ~9 h/epoch | | release format | merged fp16 (vLLM / llama.cpp / TGI ready) |
the corpus โ€” 12,783 train / 1,640 test | shard | train | source | |:--|--:|:--| | `write` | 7,921 | synthetic `write_op` + `extract` | | `resolve` | 2,250 | synthetic read-time conflict resolution | | `locomo` | 1,582 | LoCoMo QA | | `abstain` | 720 | synthetic **hard** abstention + answerable controls | | `longmemeval` | 315 | LongMemEval oracle (MIT) | **the one design choice that matters:** every hard-abstention example ships with a **twin** โ€” identical topical clutter, but the queried fact *is* present. without those controls the model just learns a new shortcut ("in-domain clutter โ†’ refuse") instead of learning to actually check. the only signal separating the labels is whether the fact is in the set. **everything here is `--visibility public`** โ€” synthetic + public benchmarks only. models memorise training data, so real user memories live behind a separate flag and a separate private adapter trained on top of these weights. no one's DMs are in these weights. ๐Ÿ”
how it was scored deterministic, lexical, no LLM judge, no API key โ€” reproducible and free. - **token F1 / EM** after normalization (lowercase, strip articles + punctuation) - **abstention P/R** matched against the exact contract string, case- and trailing-period-insensitive - **contract validity** = fraction of outputs parsing against their task schema. both models hit **1.000** โ€” the base model's problem is content, not format - test file is **shuffled before slicing** (it's concatenated by source, so slicing raw would make every per-category number a lie) ```bash python scripts/memory_sft/eval_gate.py \ --model Akhrots/LAM8B \ --data data/sft/mixed/test.jsonl \ --out reports/ft.json --compare reports/base.json ```
--- ## ๐Ÿซก keeping it real no model card should be all W's. here are the L's, on the record: | | the L | |:--|:--| | ๐Ÿ’€ **preference questions are cooked** | F1 **0.067**, EM **0.000**, 9 of 13 still wrongly refused. no shard targets subjective recall and it shows. don't use this for "what does the user like." | | ๐Ÿชซ **multi-hop is the weakest working category** | 0.418. up 48% from base, still way under single-hop's 0.734. | | ๐ŸŽฏ **these scores are the *reader*, not the retriever** | every number here measures answering **given** retrieved memories. your end-to-end quality also depends on your retrieval stack โ€” a bad retriever will bottleneck this model no matter how good it is. | | ๐Ÿ” **lexical metrics undersell it** | token F1 scores "Stripe" and "we use Stripe" as different answers. treat these as run-to-run comparisons, not absolute quality. | | โœ๏ธ **write-path numbers aren't published** | the `write_op`/`extract`/`resolve` eval shares generators with its training data, so it measures the generator as much as the model. validating against real production memories is the top open risk. | | ๐Ÿงฉ **abstention doesn't fully transfer** | 0.381 abstain-F1 on LongMemEval vs 0.847 on LoCoMo. training teaches *total absence*; LongMemEval tests *partial information* ("you mentioned X but not Y"). different problem shape. | | โณ **one epoch** | a T4 caps one Kaggle session at ~1 epoch over this corpus. whether epochs 2โ€“3 leave gains on the table is **untested**. | | ๐ŸŒ **english only** | both benchmarks and every synthetic generator are English. | --- ## ๐Ÿ“œ license & credits **Apache-2.0**. LAM8B is a fine-tune of open, Apache-2.0-licensed weights and the derivative is released under the same license. training data is synthetic + [LoCoMo](https://github.com/snap-research/locomo) + [LongMemEval](https://github.com/xiaowu0162/LongMemEval) (MIT). ```bibtex @misc{lam8b, title = {LAM8B: a joint read/write memory model with calibrated abstention}, author = {Tronocity Labs}, year = {2026}, url = {https://huggingface.co/Akhrots/LAM8B} } ```

one T4 ยท zero GPU budget ๐Ÿ

if it saved you from a hallucinated answer, leave a โค๏ธ on the repo.