You need to agree to share your Hugging Face information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
Research checkpoint shared on request. Access is approved manually.
Log in or Sign Up to review the conditions and access this model content.
CODI on Ministral-3-8B-Instruct (research checkpoints)
Plain CODI (no step grounding) on mistralai/Ministral-3-8B-Instruct-2512-BF16 @
f6fae9795746f63c9be8344932f01275f3c63734, lr 2e-4 (picked over 4e-4 in a 1,500-step pilot). LoRA + projection
checkpoints for interpretability research, not a merged model, not for production use. Training finished at epoch 3.
| epoch (step) | files | latent / removed / direct / teacher | latent − removed | transplant follow vs random | mean-fill |
|---|---|---|---|---|---|
| 0.5 (6,088) | — | 36.4 / 35.9 / 36.2 / 56.0 | +0.5 [−0.4, +1.4] | 0/101 vs 0/101 | — |
| 1 (12,175) | light + full state | 50.6 / 44.2 / 43.8 / 67.7 | +6.4 [+4.5, +8.2] | 35/162 vs 6/162 | — |
| 1.5 (18,263) | — | 54.2 / 47.8 / 48.6 / 67.2 | +6.4 [+4.5, +8.3] | 53/192 vs 2/192 | 45.2 |
| 2 (24,350) | light + full state | 57.2 / 49.7 / 50.2 / 69.1 | +7.5 [+5.6, +9.4] | 54/202 vs 5/202 | 47.6 |
| 3 (36,525) | light + full state | 63.4 / 54.4 / 54.0 / 70.0 | +9.0 [+7.1, +10.8] | 83/226 vs 5/226 | 51.3 |
Use still rises at epoch 3 (unlike the 3B, which peaks at epoch 2). A linear probe finds intermediate step values better encoded in the latents than in the question state (step 1 +4.7 [+1.9, +7.5], step 2 +5.6 [+2.4, +8.9] at epoch 3).
Recipe (all checkpoints in this repo)
- 3-mode CODI (latent student, written-CoT teacher, direct answer), batch 32, LoRA r128 / alpha 32 on q/k/v/o/gate/up/down, 6 latent thoughts, projection + LayerNorm, answer-only distillation.
- Data: GSM8k-Aug-NL + CommonsenseQA. 3% linear warmup then cosine over 10 epochs (121,750 steps); training stopped
at epoch 3. Text-only language model (vision tower dropped). Special tokens
<|bocot|>= 131072,<|eocot|>= 131073. - Prompt (template-free): latent mode
<s> {question} Output only the answer and nothing else.</s><|bocot|>; direct mode... </s><|eocot|>.
Metrics: GSM8k test, n = 1,319, greedy. "latent − removed" = accuracy with the 6 latent thoughts minus the same checkpoint with the thoughts removed (paired bootstrap 95% CI). Transplant = latents taken from a copy of the problem with one number changed; "follow" = the answer becomes the copy's answer, on pairs both versions solve, vs norm-matched random latents. Mean-fill = each latent replaced by its across-problem mean (no problem information).
Files
ckpt_<step>.pt: light eval checkpoint (trainable LoRA + projection weights).checkpoint-<step>/: full Hugging Face Trainer state for exact resume (trainable.safetensors,optimizer.pt,scheduler.pt,rng_state.pth,trainer_state.json).md5.txt: md5 of every file; all match the training pod's records.
Loading and resuming
The training and evaluation scripts (train62.py, eval29.py / eval62.sh, make_text_base.py) are in the authors'
research repository and are shared together with access. To resume exactly, pass checkpoint-<step>/ as --resume
with the same flags as the original job and the same data order; the Trainer restores optimizer, scheduler and RNG and
skips the data already seen.
Model tree for mjkenney/codi-ministral3-8b-instruct
Base model
mistralai/Ministral-3-8B-Base-2512