Compaction summariser: 0/45 -> 45/45 task lines, fact recall 0.157 -> 0.661

README.md

@@ -0,0 +1,105 @@
+---
+base_model: Lazarus-Ai/ReAligned-Qwen3.5-4B
+library_name: peft
+license: apache-2.0
+tags:
+  - lora
+  - sft
+  - summarization
+  - context-compaction
+  - agents
+  - egirl
+---
+
+# Compactor-Qwen3.5-4B
+
+A context-compaction summariser for [egirl](https://github.com/Schneewolf-Labs/egirl): given a run
+of agentic conversation about to be dropped, write the summary that lets the agent continue.
+
+Meant to run as the **auxiliary model** — the operator keeps its slot and its context, and the
+summary is produced by something small, deterministic, and trained for it.
+
+## The failure it fixes
+
+egirl compacts by dropping middle messages and replacing them with a summary. When that summary
+loses the task, the agent does not get confused — **it invents a new task**:
+
+```
+Interior compaction: dropped 19 middle messages, kept head + 0 tail groups
+Generated summary (396 chars) from 19 messages
+```
+
+Sixteen web searches of real research compressed to 396 characters, after which the agent answered
+a question about agent-harness architecture with *"Fresh project scaffolded at
+~/projects/fresh_project."* The request was gone from its context.
+
+So the objective is not fluency. It is **preservation**: the task, the findings, the state.
+
+## Measured
+
+45 held-out compaction windows, from transcripts that contributed no training data. Deterministic
+checks, no judge:
+
+| | base | **+ Compactor** |
+|---|---|---|
+| has a `Task:` line | **0/45 (0%)** | **45/45 (100%)** |
+| task overlap (word F1 vs reference) | 0.000 | **0.509** |
+| **fact recall** (paths, identifiers, numbers, errors) | **0.157** | **0.661** |
+| compression | 0.043 | **0.129** |
+| mean output | 884 chars | 2,689 chars |
+
+Reference summaries (GPT-5) compress to ~0.12×. The base compresses to 0.043× — it is not
+summarising, it is discarding: **84% of the distinctive facts are gone**, and it never states the
+task at all.
+
+**Fact recall is the number to read.** A summary can be fluent, plausible, correctly shaped, and
+still have the specifics sanded off — which is what a small model does by default, and what makes
+the agent lose the thread three turns later.
+
+## Use
+
+```python
+model = AutoModelForImageTextToText.from_pretrained(
+    "Lazarus-Ai/ReAligned-Qwen3.5-4B", dtype=torch.bfloat16, device_map="auto")
+model = PeftModel.from_pretrained(model, "schneewolflabs/Compactor-Qwen3.5-4B")
+```
+
+Render prompts with `enable_thinking=False` — training used the pre-closed think block, and the
+model answers directly rather than deliberating. Serving with thinking open puts it in a state it
+never saw.
+
+In egirl, point `[local.auxiliary]` at a server running this and compaction routes to it
+automatically.
+
+## Training
+
+| | |
+|---|---|
+| base | `Lazarus-Ai/ReAligned-Qwen3.5-4B` |
+| method | SFT, LoRA **r=64 α=128** dropout 0.05 |
+| data | [egirl-compaction-SFT](https://huggingface.co/datasets/schneewolflabs/egirl-compaction-SFT) — 1,059 train / 45 val |
+| tokens | 9.34M |
+| schedule | 1 epoch, 133 steps, lr 1e-4 cosine |
+| sequence | 16,384 — an 8192 cap would have dropped 55% of rows |
+| hardware | one RTX A6000, 11h44m |
+
+Train loss 0.738 → 0.584; held-out 0.754 → **0.561**, below train at every checkpoint. No
+memorisation, despite r=64 on 1,059 samples — which the transcript-level split makes a meaningful
+statement rather than an artefact.
+
+**This architecture is expensive to train.** 265 s/step at 16k sequence, against 15.7 s/step for a
+same-size Qwen3-VL under identical batch settings — the 24 linear-attention layers have no fused
+kernel in transformers 5.3.0 and run a sequential scan whose cost grows with sequence length.
+Inference is cheap; training is not.
+
+## Limits
+
+- **One epoch, one checkpoint.** No epoch-2 comparison, which has mattered elsewhere in this
+  family: a sibling model won overall at epoch 2 while losing a specific behaviour it had at
+  epoch 1.
+- **Targets are one model's judgement.** GPT-5 decided what mattered in each window. Where it
+  dropped something, this learned to drop it.
+- **Task overlap is 0.509, not 0.9.** The task line is always present and usually about the right
+  thing; it is not word-for-word the reference. Treat it as "states the task" rather than "states
+  it identically".
+- **One operator's transcripts**, 120 conversations of agentic coding. Other domains are untested.

adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Lazarus-Ai/ReAligned-Qwen3.5-4B",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "down_proj",
+    "gate_proj",
+    "o_proj",
+    "up_proj",
+    "v_proj",
+    "k_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:195427fa658f09867e777ec9b97674b52122f240bd98e4b6fc3283c2ca9dece0
+size 339776632

build_train.py

@@ -0,0 +1,69 @@
+"""Render the compaction dataset for ReAligned-Qwen3.5-4B.
+
+Two things this has to get right, both learned on sibling runs in this project.
+
+**Thinking is closed in the prompt.** Qwen3.5's template opens `<think>` on every generation.
+Rendered naively the target would begin `\n</think>\n\n`, teaching the model to emit a closing think
+tag as the first thing in its answer — the broken hybrid-thinking gate. This template does honour
+`enable_thinking=False`, which emits the empty block as part of the prompt, where the trainer masks
+it. The summariser should answer directly anyway: egirl calls it for a summary, not a deliberation.
+
+**Over-length rows are dropped, not truncated.** The trainer truncates prompts from the right,
+which here removes the most recent messages in the window — the current state, the part a summary
+most needs. A row that cannot fit is worth less than a row that teaches summarising from a
+mutilated transcript.
+"""
+
+import argparse
+import json
+import statistics as st
+
+from transformers import AutoTokenizer
+
+MODEL = "Lazarus-Ai/ReAligned-Qwen3.5-4B"
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--max-tokens", type=int, default=16384)
+    args = ap.parse_args()
+
+    tok = AutoTokenizer.from_pretrained(MODEL)
+
+    for split in ("train", "val"):
+        rows = [json.loads(line) for line in open(f"{split}.jsonl")]
+        out, lens, dropped = [], [], 0
+        for r in rows:
+            prompt = tok.apply_chat_template(
+                [{"role": "system", "content": r["system"]},
+                 {"role": "user", "content": r["prompt"]}],
+                add_generation_prompt=True, enable_thinking=False, tokenize=False)
+            response = r["chosen"] + tok.eos_token
+            n = len(tok(prompt)["input_ids"]) + len(
+                tok(response, add_special_tokens=False)["input_ids"])
+            lens.append(n)
+            if n > args.max_tokens:
+                dropped += 1
+                continue
+            out.append({"prompt": prompt, "chosen": response,
+                        "source_group": r["source_group"]})
+
+        with open(f"compaction_{split}.jsonl", "w") as f:
+            for o in out:
+                f.write(json.dumps(o) + "\n")
+
+        lens.sort()
+        print(f"{split}: {len(rows)} -> {len(out)} kept, {dropped} over {args.max_tokens}")
+        print(f"  tokens mean {st.mean(lens):.0f}  median {lens[len(lens)//2]}  "
+              f"p95 {lens[int(.95*len(lens))]}  max {max(lens)}")
+        for cap in (8192, 12288, 16384, 24576):
+            print(f"    cap {cap:6d} would drop {sum(1 for x in lens if x > cap):4d}/{len(lens)}")
+        print(f"  total tokens: {sum(x for x in lens if x <= args.max_tokens):,}")
+
+    sample = json.loads(open("compaction_train.jsonl").readline())
+    print("\nprompt tail:", repr(sample["prompt"][-70:]))
+    print("response head:", repr(sample["chosen"][:90]))
+
+
+if __name__ == "__main__":
+    main()

eval_compaction.py

@@ -0,0 +1,133 @@
+"""Does the summary preserve what is needed to continue?
+
+Loss is the wrong headline. The failure this tune exists to fix is specific: a summary that drops
+the task, after which the agent invents a new one — sixteen searches of research compacted into 396
+characters, answered with "fresh project scaffolded". So the metric is preservation, not fluency.
+
+Four checks, all deterministic, no judge:
+
+- **has_task** — is there a `Task:` line at all. The reference summaries all have one.
+- **task_overlap** — word-level F1 between the generated task line and the reference's. Catches a
+  summary that states *a* task, just not this one.
+- **fact_recall** — fraction of the reference's distinctive tokens (paths, identifiers, numbers,
+  errors) that survive into the generated summary. This is the one that catches a fluent summary
+  with the specifics sanded off, which is what a small model tends to produce.
+- **compression** — length ratio. A "summary" that is 90% of the input has not compacted anything,
+  and one that is 2% has thrown the state away.
+
+Run base and tuned through the identical path; the difference is the adapter.
+
+Usage:
+    python3 eval_compaction.py --out base_eval.json
+    python3 eval_compaction.py --adapter out/adapter --out tuned_eval.json
+    python3 eval_compaction.py --compare base_eval.json tuned_eval.json
+"""
+
+import argparse
+import json
+import re
+import sys
+
+MODEL = "Lazarus-Ai/ReAligned-Qwen3.5-4B"
+
+# Tokens worth preserving: paths, dotted identifiers, CamelCase, numbers with units, errors.
+DISTINCTIVE = re.compile(
+    r"(?:/[\w.\-/]{4,})"
+    r"|(?:\b\w+\.(?:ts|py|json|jsonl|toml|md|sh|jinja|gguf|safetensors)\b)"
+    r"|(?:\b[A-Z][a-z]+[A-Z]\w+\b)"
+    r"|(?:\b\d{2,}(?:\.\d+)?[kKmMgG]?\b)"
+    r"|(?:\b[A-Z][A-Z_]{3,}\b)"
+)
+
+
+def task_line(text: str) -> str:
+    m = re.search(r"^\s*task:\s*(.+)$", text, re.I | re.M)
+    return m.group(1).strip() if m else ""
+
+
+def f1(a: str, b: str) -> float:
+    ta = {w.lower() for w in re.findall(r"\w+", a) if len(w) > 2}
+    tb = {w.lower() for w in re.findall(r"\w+", b) if len(w) > 2}
+    if not ta or not tb:
+        return 0.0
+    hit = len(ta & tb)
+    p, r = hit / len(ta), hit / len(tb)
+    return 0.0 if p + r == 0 else 2 * p * r / (p + r)
+
+
+def score(generated: str, reference: str, prompt: str) -> dict:
+    ref_facts = set(DISTINCTIVE.findall(reference))
+    gen_facts = set(DISTINCTIVE.findall(generated))
+    return {
+        "has_task": bool(task_line(generated)),
+        "task_overlap": round(f1(task_line(generated), task_line(reference)), 3),
+        "fact_recall": round(len(ref_facts & gen_facts) / len(ref_facts), 3) if ref_facts else None,
+        "compression": round(len(generated) / max(len(prompt), 1), 4),
+        "chars": len(generated),
+    }
+
+
+def run(args) -> None:
+    import torch
+    from transformers import AutoModelForImageTextToText, AutoTokenizer
+
+    tok = AutoTokenizer.from_pretrained(MODEL)
+    tok.padding_side = "left"
+    if tok.pad_token_id is None:
+        tok.pad_token = tok.eos_token
+
+    model = AutoModelForImageTextToText.from_pretrained(
+        MODEL, dtype=torch.bfloat16, device_map="cuda:0")
+    if args.adapter:
+        from peft import PeftModel
+        model = PeftModel.from_pretrained(model, args.adapter)
+    model.eval()
+
+    rows = [json.loads(l) for l in open(args.val)][: args.limit]
+    results = []
+    for i, r in enumerate(rows):
+        enc = tok(r["prompt"], return_tensors="pt").to(model.device)
+        with torch.no_grad():
+            out = model.generate(**enc, max_new_tokens=1200, do_sample=False,
+                                 pad_token_id=tok.pad_token_id)
+        gen = tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True).strip()
+        s = score(gen, r["chosen"], r["prompt"])
+        s["text"] = gen[:1500]
+        results.append(s)
+        print(f"[{i+1}/{len(rows)}] task={s['has_task']} overlap={s['task_overlap']} "
+              f"facts={s['fact_recall']} chars={s['chars']}", flush=True)
+
+    summarise(results, args.adapter or "base")
+    json.dump({"adapter": args.adapter, "results": results}, open(args.out, "w"), indent=1)
+
+
+def summarise(results: list, label: str) -> None:
+    n = len(results)
+    has = sum(r["has_task"] for r in results)
+    ov = [r["task_overlap"] for r in results]
+    fr = [r["fact_recall"] for r in results if r["fact_recall"] is not None]
+    comp = [r["compression"] for r in results]
+    print(f"\n--- {label} ({n} held-out windows) ---")
+    print(f"  has Task: line   {has}/{n}  ({100*has/n:.0f}%)")
+    print(f"  task overlap     {sum(ov)/n:.3f} mean")
+    print(f"  fact recall      {sum(fr)/len(fr):.3f} mean" if fr else "  fact recall n/a")
+    print(f"  compression      {sum(comp)/n:.3f} mean  (reference ~0.12)")
+    print(f"  mean chars       {sum(r['chars'] for r in results)//n}")
+
+
+def compare(a: str, b: str) -> int:
+    for path in (a, b):
+        d = json.load(open(path))
+        summarise(d["results"], d.get("adapter") or "base")
+    return 0
+
+
+if __name__ == "__main__":
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--adapter")
+    ap.add_argument("--val", default="compaction_val.jsonl")
+    ap.add_argument("--limit", type=int, default=45)
+    ap.add_argument("--out", default="compaction_eval.json")
+    ap.add_argument("--compare", nargs=2)
+    a = ap.parse_args()
+    sys.exit(compare(*a.compare) if a.compare else run(a))