DNA-DiskChat-2B-PEER-v27

Recurrent PEER code model — 2.0B logical parameters, 40M active per token, 185 MB deployable checkpoint.

v27 is a code-focused continuation of the v25/v26 DNA-DiskChat-2B-PEER controller. It trains the same architecture on 7.5B additional tokens (~50% code) and keeps the PEER expert pool unfrozen during SFT (a change from v25, where the pool was frozen and contributed nothing to instruction tuning).

Model Details

Field	Value
Logical parameters	1,999,254,018
Active parameters / token	40,189,952 (~2%)
PEER experts	1,893,376 (nk=1376)
Layers / heads	9 / 1
Pretraining tokens	7.5B (code-focused continuation)
SFT steps / data	5,000 / code_instructions_120k (filtered)
Checkpoint (fp32/fp16)	8.84 GB
Compressed deployment (controller.v27q)	185.2 MB
Effective bits / parameter (compressed)	0.741
Compression validation CE delta	+0.0072
Compression quality gate (<= +0.01)	PASS
License	Apache 2.0

Pretraining

Code-focused continuation on 7.5B tokens (github-code-clean + fineweb-edu, ~50% code). The run survived several instance restarts and resumed from checkpoint each time.

Metric	Value
Final step CE	2.04
Mean throughput	61,529 tok/s
Median throughput	62,624 tok/s
Wall-clock (final segment)	7.8 h
Hardware	1x RTX 3090 (Vast AI)

Supervised Fine-Tuning

v27 unfreezes the PEER pool during SFT (sparse index_add_ SGD for the pool, AdamW for the dense controller), so the 1.9B logical pool parameters actually participate in instruction tuning.

Metric	Value
Train assistant CE (final)	2.32
Validation assistant CE	2.49
Data	code_instructions_120k, contamination-filtered
Train / validation records	20,337 / 408
Assistant tokens	7.15M

Code Benchmarks

EvalPlus 0.3.1, greedy pass@1, evaluated on the SFT checkpoint (not the compressed file). Reported honestly.

Benchmark	Base pass@1	Plus pass@1	Tasks
HumanEval	0/164 (0.0%)	0.0%	164
MBPP	0/378 (0.0%)	0.0%	378

Honest note: pass@1 is 0.0 on both, matching the v25 family behavior at 40M active parameters. This model is a research artifact for compact recurrent routing and low-bit MoE deployment, not a production code model.

Compression

The SFT checkpoint is product-quantized (8 groups, 12 bits) to a 185 MB controller.v27q file that fits comfortably under the 195 MiB deployment cap, with a negligible CE penalty.

Field	Value
controller_v27.v27q	185.2 MB
controller_g8b12.v27q	174.7 MB
Effective bits / param	0.741
Base validation CE	5.288
Compressed validation CE	5.296
CE delta	+0.0072
Quality gate (<= +0.01)	PASS

Deployment

• deployment/controller.v27q — PQ checkpoint (product-quantized pool + fp16/fp32 dense).
• checkpoints/sft.pt — full SFT checkpoint (8.84 GB).
• checkpoints/v27_base.pt — final 7.5B pretraining checkpoint.
• tokenizer.json — 32k BPE (FineWeb).
• Reader: scripts/v26_pq/pq_page_format.py (O_DIRECT NVMe page lookups).

# load the PQ controller (see pq_page_format.py for the full reader)
python -c "
from pq_page_format import load_controller
ctrl = load_controller('controller.v27q')
print(ctrl.summary())
"

Reproducibility

• Validation set: fineweb-edu docs 2000:2199 (16,416 tokens, sha256 7e30af1a4412d2fe990294d0dfb158bbd987ecbc6e707af4bdd30e1b816e7568).
• SFT data: sahil2801/code_instructions_120k, contamination-filtered vs HumanEval/MBPP, 20,337 train / 408 validation records, 7.15M assistant tokens.
• All experiments on a Vast AI instance (RTX 3090).
• Training config: batch 24, seq 512, nk 1376, pool-lr 0.05 (pretrain) / 0.02 (SFT), AdamW lr 2e-5.

Intended Use

Research on compact recurrent codecs and low-bit MoE deployment. Generated code is untrusted; review it and execute only in a sandbox.

Limitations

• pass@1 is 0.0 on HumanEval/MBPP; the model does not produce working code solutions at this scale.
• Benchmark scores do not establish security, correctness, or production readiness.
• Missing metrics were not measured and are not estimated from plans or earlier model versions.
• EvalPlus execution runs untrusted model-generated Python and requires isolation.