README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
thumbnail: https://huggingface.co/Akhrots/LAM8B/resolve/main/logo.png
language:
- en
tags:
- memory
- long-term-memory
- retrieval-augmented-generation
- qlora
- abstention
- hallucination-reduction
- locomo
- longmemeval
datasets:
- snap-research/locomo
- xiaowu0162/longmemeval
model-index:
- name: LAM8B
  results:
  - task:
      type: question-answering
      name: Memory QA (answer task)
    dataset:
      name: LoCoMo (held-out QA)
      type: locomo
      split: test
    metrics:
    - type: f1
      value: 0.6599
      name: Token F1
    - type: exact_match
      value: 0.3650
      name: Exact Match
    - type: f1
      value: 0.8472
      name: Abstention F1
    - type: precision
      value: 0.8026
      name: Abstention Precision
    - type: recall
      value: 0.8971
      name: Abstention Recall
  - task:
      type: question-answering
      name: Memory QA (answer task)
    dataset:
      name: LongMemEval (reserved, never trained on)
      type: longmemeval
      split: reserved
    metrics:
    - type: f1
      value: 0.4843
      name: Token F1
    - type: exact_match
      value: 0.3264
      name: Exact Match
    - type: f1
      value: 0.3810
      name: Abstention F1
  - task:
      type: question-answering
      name: Memory QA (answer task)
    dataset:
      name: GML memory eval (LoCoMo + LongMemEval reserved)
      type: mixed
      split: test
    metrics:
    - type: f1
      value: 0.6061
      name: Token F1
    - type: exact_match
      value: 0.3532
      name: Exact Match
    - type: f1
      value: 0.7879
      name: Abstention F1
    - type: accuracy
      value: 1.0
      name: Output-contract validity
---

<p align="center">
  <img src="https://huggingface.co/Akhrots/LAM8B/resolve/main/logo.png" alt="LAM8B" width="440">
</p>

<h1 align="center">LAM8B 🐏</h1>

<p align="center">
  <strong>the memory model that knows when it doesn't know.</strong><br>
  <sub>Large Akhrots Model · by <b>Tronocity Labs</b></sub><br>
  <sub>most models would rather hallucinate than admit a gap. this one won't. 💅</sub>
</p>

<p align="center">
  <img src="https://img.shields.io/badge/token_F1-0.606-00E5A0?style=for-the-badge&labelColor=101010" alt="F1 0.606">
  <img src="https://img.shields.io/badge/vs_base-%2B61%25-FF2E93?style=for-the-badge&labelColor=101010" alt="+61% vs base">
  <img src="https://img.shields.io/badge/hallucinated_refusals-−77%25-8B5CF6?style=for-the-badge&labelColor=101010" alt="-77% over-abstentions">
</p>

<p align="center">
  <img src="https://img.shields.io/badge/params-8B-FF6B00?style=flat-square&labelColor=101010" alt="8B params">
  <img src="https://img.shields.io/badge/license-Apache_2.0-00B4D8?style=flat-square&labelColor=101010" alt="Apache 2.0">
  <img src="https://img.shields.io/badge/QLoRA-r32_·_α64-FFD60A?style=flat-square&labelColor=101010" alt="QLoRA r32 alpha64">
  <img src="https://img.shields.io/badge/trained_on-1×_T4_😭-EF476F?style=flat-square&labelColor=101010" alt="1x T4">
  <img src="https://img.shields.io/badge/contract_valid-100%25-06D6A0?style=flat-square&labelColor=101010" alt="100% contract valid">
</p>

---

## ✨ the vibe

your assistant has a memory problem. it either **forgets** what you told it, or worse — it **makes something up** and says it with its whole chest.

LAM8B does both halves of the job in one set of weights:

| | what it does |
|:--|:--|
| ✍️ **writes** | decides what's worth storing, and whether a new fact *supersedes* an old one |
| 🔍 **reads** | answers from what's stored — **or tells you it's not in there** |

that second half is the whole point. ⬇️

> 🧩 **where it sits:** LAM8B lives *behind* your retriever. your system pulls candidate memories, LAM8B reads them and answers or abstains. **it is not a general chat model** — don't drop it in as one.

---

## 📈 the glow-up

same prompts, same decoding, same harness. **only the weights changed.**

| metric | un-tuned base | **LAM8B** | |
|:--|:--|:--|:--|
| **token F1** | `█████░░░░░░░` 0.376 | `███████░░░░░` **0.606** | 🟢 **+61%** |
| **exact match** | `██░░░░░░░░░░` 0.177 | `████░░░░░░░░` **0.353** | 🟢 **2.0×** |
| **abstention F1** | `██████░░░░░░` 0.541 | `█████████░░░` **0.788** | 🟢 **+0.247** |
| **abstention precision** | `█████░░░░░░░` 0.378 | `█████████░░░` **0.714** | 🔥 **+0.336** |
| **abstention recall** | `███████████░` 0.946 | `███████████░` 0.878 | 🟡 −0.068 |
| **wrongly refused** | 115 / 544 | **26 / 544** | 🔥 **−77%** |
| **contract-valid output** | 100% | 100% | ✅ tie |
| **speed** | 2.1 s/ex | **1.6 s/ex** | 🟢 −24% |

<sub>n = 544 held-out answer-task examples.</sub>

---

## 🎯 the party trick: it shuts up when it should

here's the trap. the un-tuned base has **94.6% abstention recall** — it catches almost every unanswerable question. sounds elite. it is not.

> **precision is 37.8%.** it refuses *constantly*. so of course it catches the real gaps — it's refusing everything. **115 out of 544 questions it could have answered, it didn't.** that's not "safe." that's a model that shrugs at you.

LAM8B trades **6.7 points of recall** for **33.6 points of precision**:

```
                  precision                     recall
base   0.378  █████░░░░░░░░░░░       0.946  ███████████████░
LAM8B  0.714  ███████████░░░░░       0.878  ██████████████░░
                 ▲ +0.336                       ▼ −0.068
```

wrongly-refused questions: **115 → 26.** on the dedicated abstention split (n=74) both models hold **100% precision** — base scores 0.972 abstain-F1 there, LAM8B 0.935. so the base model's whole problem is on questions that *had* answers. 💀

---

## 🧾 receipts (per category)

sorted by how hard it glowed up. `over` = answerable questions it wrongly refused.

| category | n | base | **LAM8B** | Δ F1 | over |
|:--|--:|:--|:--|--:|:--|
| 🥇 **adversarial** | 76 | `██░░░░░░░░░░` 0.183 | `████████░░░░` **0.673** | **+0.489** | 44 → **6** |
| 🥈 **temporal** | 93 | `████░░░░░░░░` 0.303 | `███████░░░░░` **0.586** | **+0.282** | 18 → **2** |
| 🥉 **single_hop** | 140 | `███████░░░░░` 0.545 | `█████████░░░` **0.734** | +0.189 | 16 → **3** |
| **open_domain** | 13 | `████░░░░░░░░` 0.326 | `██████░░░░░░` **0.468** | +0.141 | 3 → **3** |
| **multi_hop** | 94 | `███░░░░░░░░░` 0.283 | `█████░░░░░░░` **0.418** | +0.135 | 21 → **3** |
| **knowledge_update** | 20 | `██████░░░░░░` 0.508 | `████████░░░░` **0.634** | +0.125 | 2 → **0** |
| **single_hop_assistant** | 21 | `█████████░░░` 0.785 | `██████████░░` **0.842** | +0.056 | 0 → **0** |
| 💀 **preference** | 13 | `█░░░░░░░░░░░` 0.060 | `█░░░░░░░░░░░` 0.067 | +0.007 | 11 → **9** |

**adversarial is the flex.** 🥇 same-domain distractors, where refusing takes actual judgement instead of vibes. base model basically gave up (0.183). LAM8B: **0.673**, and 44 wrong refusals collapse to 6.

**preference is the L.** 💀 it did not improve. see [keeping it real](#-keeping-it-real).

<details>
<summary><b>📊 exact-match numbers too (click)</b></summary>

| category | n | EM (base) | **EM (LAM8B)** |
|:--|--:|--:|--:|
| adversarial | 76 | 0.1053 | **0.4211** |
| temporal | 93 | 0.0753 | **0.2473** |
| single_hop | 140 | 0.2643 | **0.4714** |
| open_domain | 13 | 0.2308 | **0.3846** |
| multi_hop | 94 | 0.0957 | **0.1809** |
| knowledge_update | 20 | 0.4000 | **0.4500** |
| single_hop_assistant | 21 | 0.5238 | **0.6667** |
| preference | 13 | 0.0000 | 0.0000 |

</details>

---

## 🔬 the honest split

| source | n | F1 (base) | **F1 (LAM8B)** | EM | abstain P | abstain R | abstain F1 | over |
|:--|--:|:--|:--|:--|:--|:--|:--|:--|
| **LoCoMo** | 394 | `████░░░░░░░░` 0.372 | `████████░░░░` **0.660** | 0.150 → **0.365** | 0.444 → **0.803** | 0.941 → 0.897 | 0.604 → **0.847** | 80 → **15** |
| **LongMemEval** 🔒 | 150 | `█████░░░░░░░` 0.384 | `██████░░░░░░` **0.484** | 0.236 → **0.326** | 0.146 → **0.267** | 1.000 → 0.667 | 0.255 → **0.381** | 35 → **11** |

> 🔒 **quote this one.** those 150 LongMemEval questions were **held in reserve and never trained on** — `longmemeval_oracle` and `_s` share the same 500 questions, so training on either contaminates the other. **+0.10 F1 on genuinely unseen data** is the real generalization number, not the +0.29 on LoCoMo.

also notice base's **1.000** abstention recall on LongMemEval with **0.146** precision. that's not skill. that's a model saying "idk" to everything. 🫠

---

## ⚡ quickstart

```bash
pip install transformers torch
```

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "Akhrots/LAM8B"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(
    MODEL,
    trust_remote_code=True,   # ← required: LAM registers its own model class
    torch_dtype="float16",
    device_map="auto",
)

SYSTEM = (
    "You answer questions using only the user's stored memories, which are "
    "gathered from every app they use.\n"
    "Rules:\n"
    "- Use only the memories provided. Never invent or assume.\n"
    "- When memories conflict, trust the most recent and answer with that.\n"
    "- Combine memories when the answer needs more than one.\n"
    "- Respect timestamps for questions about when, before, after or latest.\n"
    "- Read negation carefully: a memory saying something stopped is not "
    "evidence that it is still true.\n"
    "- If the memories do not contain the answer, reply exactly: "
    "I don't have that in your memory.\n"
    "Answer concisely — no preamble, no restating the question."
)

# memories render as:  [id] (timestamp via source) text
memories = (
    "[m1] (2024-03-02 via slack) Priya moved the launch review to Thursday.\n"
    "[m2] (2024-02-11 via gmail) Priya scheduled the launch review for Tuesday."
)
user = f"Memories:\n{memories}\n\nQuestion: When is the launch review?"

msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}]
prompt = tok.apply_chat_template(
    msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device), max_new_tokens=128)
print(tok.decode(out[0][len(tok(prompt).input_ids):], skip_special_tokens=True))
# -> "Thursday."   ← m1 is newer, so m2 loses. that's the supersession behaviour.
```

serve it (weights are merged fp16, no adapter juggling):

```bash
vllm serve Akhrots/LAM8B --served-model-name lam8b --trust-remote-code
```

### 🚨 three rules, don't skip

**0. `trust_remote_code=True` is mandatory.** LAM registers its own model class via `modeling_lam.py`. without the flag, loading fails outright.

**1. the prompt is load-bearing.** system prompt, `[id] (time via source)` memory format, and the exact abstention string `I don't have that in your memory.` are all part of the trained contract. drift the prompt → quality drops silently, no error.

**2. thinking stays OFF.** LAM8B is a hybrid-thinking architecture and was trained with `enable_thinking=False`. flip it on at serve time and you're measuring a different model than the one that was trained.

### 🧰 four tasks, one model

| task | mode | output |
|:--|:--|:--|
| `answer` | read | text, or the exact abstention string |
| `write_op` | write | JSON — `insert` / `supersede` / `merge` / `skip` |
| `extract` | write | JSON — entities + attributes |
| `resolve` | read | JSON — `{"kept": [...], "superseded": [...], "reason": str}` |

<sub>only <code>answer</code> is scored above — see <a href="#-keeping-it-real">keeping it real</a> for why the write-path numbers aren't published.</sub>

---

## 🔧 how it was cooked

<details open>
<summary><b>training config</b></summary>

| | |
|:--|:--|
| base | open Apache-2.0 weights |
| method | QLoRA, 4-bit NF4, fp16 compute |
| LoRA r / α / dropout | **32 / 64 / 0.05** |
| target modules | `q,k,v,o,gate,up,down_proj` (all of them) |
| LR | 1e-4 |
| batch × grad-accum | 1 × 16 (effective 16) |
| epochs | 1 |
| hardware | **a single free Kaggle T4** 😭 ~9 h/epoch |
| release format | merged fp16 (vLLM / llama.cpp / TGI ready) |

</details>

<details>
<summary><b>the corpus — 12,783 train / 1,640 test</b></summary>

| shard | train | source |
|:--|--:|:--|
| `write` | 7,921 | synthetic `write_op` + `extract` |
| `resolve` | 2,250 | synthetic read-time conflict resolution |
| `locomo` | 1,582 | LoCoMo QA |
| `abstain` | 720 | synthetic **hard** abstention + answerable controls |
| `longmemeval` | 315 | LongMemEval oracle (MIT) |

**the one design choice that matters:** every hard-abstention example ships with a **twin** — identical topical clutter, but the queried fact *is* present. without those controls the model just learns a new shortcut ("in-domain clutter → refuse") instead of learning to actually check. the only signal separating the labels is whether the fact is in the set.

**everything here is `--visibility public`** — synthetic + public benchmarks only. models memorise training data, so real user memories live behind a separate flag and a separate private adapter trained on top of these weights. no one's DMs are in these weights. 🔐

</details>

<details>
<summary><b>how it was scored</b></summary>

deterministic, lexical, no LLM judge, no API key — reproducible and free.

- **token F1 / EM** after normalization (lowercase, strip articles + punctuation)
- **abstention P/R** matched against the exact contract string, case- and trailing-period-insensitive
- **contract validity** = fraction of outputs parsing against their task schema. both models hit **1.000** — the base model's problem is content, not format
- test file is **shuffled before slicing** (it's concatenated by source, so slicing raw would make every per-category number a lie)

```bash
python scripts/memory_sft/eval_gate.py \
    --model Akhrots/LAM8B \
    --data data/sft/mixed/test.jsonl \
    --out reports/ft.json --compare reports/base.json
```

</details>

---

## 🫡 keeping it real

no model card should be all W's. here are the L's, on the record:

| | the L |
|:--|:--|
| 💀 **preference questions are cooked** | F1 **0.067**, EM **0.000**, 9 of 13 still wrongly refused. no shard targets subjective recall and it shows. don't use this for "what does the user like." |
| 🪫 **multi-hop is the weakest working category** | 0.418. up 48% from base, still way under single-hop's 0.734. |
| 🎯 **these scores are the *reader*, not the retriever** | every number here measures answering **given** retrieved memories. your end-to-end quality also depends on your retrieval stack — a bad retriever will bottleneck this model no matter how good it is. |
| 🔍 **lexical metrics undersell it** | token F1 scores "Stripe" and "we use Stripe" as different answers. treat these as run-to-run comparisons, not absolute quality. |
| ✍️ **write-path numbers aren't published** | the `write_op`/`extract`/`resolve` eval shares generators with its training data, so it measures the generator as much as the model. validating against real production memories is the top open risk. |
| 🧩 **abstention doesn't fully transfer** | 0.381 abstain-F1 on LongMemEval vs 0.847 on LoCoMo. training teaches *total absence*; LongMemEval tests *partial information* ("you mentioned X but not Y"). different problem shape. |
| ⏳ **one epoch** | a T4 caps one Kaggle session at ~1 epoch over this corpus. whether epochs 2–3 leave gains on the table is **untested**. |
| 🌍 **english only** | both benchmarks and every synthetic generator are English. |

---

## 📜 license & credits

**Apache-2.0**. LAM8B is a fine-tune of open, Apache-2.0-licensed weights and the derivative is released under the same license. training data is synthetic + [LoCoMo](https://github.com/snap-research/locomo) + [LongMemEval](https://github.com/xiaowu0162/LongMemEval) (MIT).

```bibtex
@misc{lam8b,
  title  = {LAM8B: a joint read/write memory model with calibrated abstention},
  author = {Tronocity Labs},
  year   = {2026},
  url    = {https://huggingface.co/Akhrots/LAM8B}
}
```

<p align="center">
  <sub>one T4 · zero GPU budget 🐏</sub>
</p>

<p align="center">
  <sub>if it saved you from a hallucinated answer, leave a ❤️ on the repo.</sub>
</p>