| --- |
| license: apache-2.0 |
| base_model: |
| - Qwen/Qwen3-4B-Base |
| - Qwen/Qwen3-8B-Base |
| tags: [research-backup, memory-graft, product-key-memory, capacity-expansion] |
| --- |
| |
| # memsplit-checkpoints |
|
|
| Research checkpoints from the FCUO / post-training capacity-expansion project (Qwen3), backed up |
| before B200 server release. Weights-only + sparse learned-memory (optimizer state and regenerable |
| calib caches excluded). |
|
|
| ## License |
| Weights derive from **Qwen3 (Apache-2.0)** → Apache-2.0. **Exception:** `ckd_*` (agentic) were |
| trained partly on **APIGen-MT-5k (CC-BY-NC-4.0)** → those subdirs are **non-commercial / research-only**. |
| Training data: OpenR1-Math-220k (Apache-2.0), openwebmath (ODC-By), xlam-function-calling-60k (CC-BY-4.0), |
| ToolACE (Apache-2.0), APIGen-MT-5k (CC-BY-NC-4.0), PopQA. Third-party datasets are NOT redistributed here. |
|
|
| ## Headline result |
| `pk_p2_frozen` — a FROZEN Qwen3-4B-Base backbone + a large content-addressed product-key memory |
| (dq512 / 65536 keys / 1.6B params) reaches **52.5% direct fact recall (full 14k set; 77.5 on the |
| first-3000 subset)** vs small-PK 24.3 and bare-frozen 0 — the project's strongest near-zero-inference |
| capacity positive (direct-probe only; generalization untested). |
|
|
| ## Format |
| - Weights: standard HF `*.safetensors` + config/tokenizer (+ `ctx_gate.pt` for grafts, `pk_memory.pt` for PK). |
| - `sparse_memory/layer_*.pt` (learned memory grafts only): nonzero rows only. Reconstruct dense: |
| ```python |
| import torch |
| d = torch.load("sparse_memory/layer_00.pt") |
| dense = torch.zeros(d["V"], d["I"], dtype=d["mean_rows"].dtype) |
| dense[d["rows"].long()] = d["mean_rows"] |
| ``` |
| - Graft checkpoints need their calib table to run; calib dirs are regenerable via |
| `fcuo/calibrate_selfgen_math.py` (see code repo). Frozen-graft `trained_memory` (= calib copy) is omitted. |
|
|
| Code: github.com/hyunseoklee-ai/memory_split. Full result docs: private backup repo. |
| |
