# pip install moshi
# Run the interactive web server
python -m moshi.server --hf-repo "MagicLuke/oif-instok-sft"
# Then open https://localhost:8998 in your browser# pip install moshi
import torch
from moshi.models import loaders
# Load checkpoint info from HuggingFace
checkpoint = loaders.CheckpointInfo.from_hf_repo("MagicLuke/oif-instok-sft")
# Load the Mimi audio codec
mimi = checkpoint.get_mimi(device="cuda")
mimi.set_num_codebooks(8)
# Encode audio (24kHz, mono)
wav = torch.randn(1, 1, 24000 * 10) # [batch, channels, samples]
with torch.no_grad():
codes = mimi.encode(wav.cuda())
decoded = mimi.decode(codes)You need to agree to share your contact information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
These adapters are trained on Fisher (LDC) and Seamless Interaction. The corpora themselves are not redistributed here. By requesting access you agree not to attempt to reconstruct or redistribute the underlying corpora.
Log in or Sign Up to review the conditions and access this model content.
oif β online-instruction-following SFT checkpoints
LoRA adapters for nvidia/personaplex-7b-v1
that make the model follow instructions injected mid-conversation by a separate text LLM.
A directive is spliced into the model's inner-monologue stream as extra timesteps
(ins_token mode) with the audio held silent, so the model reads the directive and acts on
its next turn without ever speaking it.
Project: MagicLuke/duplex-online-if Β·
recipe of record: docs/two_stage_training.md Β· per-recipe lab notes: docs/lab/ Β·
RL post-training of the same base: MagicLuke/oif-instok-rl
Contents
One subfolder per published checkpoint, <run_tag>/checkpoint_<step>/.
| run | step | what it is | downstream eval |
|---|---|---|---|
stage1f_final_ins |
4000 | Stage 1 only β Fisher + Seamless directive injection, never saw a goal card. The checkpoint stage 2 was initialised from, and the peak of the stage-1 conditioning probe | none |
stage2_ins_stage1f |
337 | The shipped SFT checkpoint. 337 steps of conv-synth goal arcs continued from stage 1 via lora_init_from |
IF-Bench v1.3 + FDB v1.0 |
Read that eval column before choosing. stage1f_final_ins/4000 is published as the controlled
stage-1 baseline for "what did stage 2 add" β it has never been benchmarked and it cannot
initiate speech from a cold prompt (see below). If you want a usable model, use step 337, or
the RL checkpoints.
stage2_ins_stage1f/checkpoint_000337
The SFT checkpoint of record. lora_init_from stage 1's step 4000, then 337 steps on conv-synth
goal arcs; instr_lr 0.0 (delimiter rows frozen), instr_dropout 0.0, lr 1e-6, bs 2 Γ 8 ranks,
240 s windows. Eval loss 2.097 β 1.597 (text 0.715 β 0.365, audio 1.382 β 1.233).
lora.safetensors is 2,328,712,528 bytes, sha256
6ccafd8707471d89620a2767bec1efe866b4527fb60461e451705730ab5ce4d8, folded at --scaling 2.0.
IF-Bench v1.3 β 200 tasks, judge Qwen/Qwen3.8-27B
| examiner | topic 1β5 | completion | either-side | order | seat-flip β | naturalness |
|---|---|---|---|---|---|---|
| stage2-337 + brain | 3.79 | 49.1% | 58.4% | 9.5% | 4.6% | 4.05 |
| stage2-337 brain off | 2.19 | 14.6% | 19.6% | 0.0% | 3.6% | 4.54 |
oif-instok-rl gdpo-duplex v1200 + brain |
4.80 | 85.8% | 91.4% | 47.5% | 42.5% | 3.81 |
| PersonaPlex-7B base | 2.19 | 7.6% | 10.6% | 0.0% | 21.5% | 4.45 |
This checkpoint's real strength is role stability, not completion. At 3.6β4.6% quote-verified seat flips it is an order of magnitude steadier than either RL checkpoint (40β66%) and better than the base model (21.5%), and its naturalness is the highest of our checkpoints. It completes far fewer tasks. Pick accordingly.
FDB v1.0 β 727 stimuli
| condition | pause TOR syn/candor β | backchannel TOR/freq/JSD | smooth TOR/lat | interrupt TOR/rating/lat |
|---|---|---|---|---|
| brain off, training card + voice | 0.124 / 0.236 | 0.073 / 0.011 / 0.953 | 0.857 / 0.992 | 0.960 / 2.312 / 0.908 |
| brain off, VARF0 + FDB prompts | 0.153 / 0.245 | 0.018 / 0.056 / 0.795 | 0.756 / 0.007 | 0.950 / 4.037 / 0.859 |
| raw adapter, NATF2 + FDB prompts | 0.000 / 0.009 | 0.255 / 0.045 / 0.786 | 0.244 / 0.997 | 0.930 / 3.914 / 0.977 |
| PersonaPlex (ours, VARF0) | 0.642 / 0.718 | 0.418 / 0.176 / 0.700 | 0.983 / 0.020 | 0.935 / 4.684 / 0.214 |
The finetune regressed turn-taking on this benchmark β smooth-turn TOR fell 0.96 β 0.24 in the raw-adapter condition. Low pause TOR is good (it holds silence), but taken together with the smooth-turn collapse the picture is a model that has become reluctant to take the floor.
Training-data overlap with IF-Bench β measured
Disclosure: conv-synth v1 was seeded from the 200 FDB-v2 staged tasks and split by scenario into 180 train / 10 val / 10 test β and those 200 scenarios are the 200 IF-Bench task ids. So stage 2 trained on scenarios matching 180 of the 200 benchmark tasks. (Stage 1 did not; its corpus is Fisher + Seamless.)
Verified completion on the 20 genuinely held-out scenarios minus the 180 trained ones, bootstrap CI over tasks:
| arm | held-out (n=19) | trained (n=172) | gap | 95% CI |
|---|---|---|---|---|
| stage2-337 brain off | 18.4% | 14.8% | +3.6 | [β5.7, +14.3] |
| stage2-337 brain on | 50.0% | 49.9% | +0.1 | [β13.0, +13.4] |
No memorization advantage is detectable β both CIs span zero and the point estimates are
flat-to-positive. The overlap is real in provenance but does not show up in the score. The test
has limited power at nβ20 (roughly Β±10 points), so it rules out a large effect, not a small one.
Held-out ids: fdb-v2-IF-bench/benchmark/oif_splits/convsynth_overlap.json; score both halves
with slurm/arm_compare.py --subsets.
stage1f_final_ins/checkpoint_004000
Stage 1 of the two-stage recipe, stopped for publication at the step stage 2 branched from.
- Base
personaplex-7b-v1-native-dq8-instr32002β dep_q sliced 16β8, text vocab 32000β32002 for the two directive markers. The adapter does not load onto stock PersonaPlex unchanged; see the repo'sscripts/convert/. - Data
instr_full/v2β Fisher-train + Seamless improvised + Seamless naturalistic, weighted, 240 s windows. 221,247 examples / 1,652,922 injections. - Recipe LoRA r512 +
depth_rank64, scaling 2.0 Β· lr 2e-6,instr_lr2e-5 Β·instr_dropout0.5β0.3 Β·instr_near_p0.7 /near_frames25 Β·text_padding_weight0.3 Β·first_codebook_weight_multiplier100 Β· bs 8.
Measured behaviour, including what is wrong with it
Directive probe on 200 held-out Fisher dialogues (1,850 scored injections), wrong β correct
cross-entropy, base-corrected:
| checkpoint | base | 1000 | 2000 | 3000 | 4000 | 5000 | 6000 | 7000 |
|---|---|---|---|---|---|---|---|---|
| base-corrected Ξ | β | +0.088 | +0.085 | +0.088 | +0.100 | +0.091 | +0.092 | +0.081 |
Steering saturates at the first saved checkpoint. Steps 1000 β 7000 move the conditioning delta by less than 0.02, and the peak is at 4000, not at the end.
This checkpoint cannot initiate speech from a cold prompt. Free-running from a voice+persona prompt with a real partner talking and no teacher-forced history, it produces 0 words in 120 s where the base model produces 41. The same checkpoint looks better than base under any fixture that supplies 30 s of real conversation first (conditionality delta +0.257 vs base's +0.116) β it can continue speech but not start it. Treat teacher-forced numbers for this checkpoint as measuring conditioning only, never conversational competence.
It is published because it is the controlled stage-1 baseline for "what did stage 2 add", not because it is a usable conversational model on its own.
Why ins_token. Every variant that routes the instruction through a trainable projection
measured at chance, including on this exact corpus: add_sum reached Ξ = +0.0002 Β± 0.0003
after 4000 steps β correct, wrong, and no instruction at all were identical to three decimals,
while its eval loss fell normally. Only mechanisms carrying real token ids (inline,
ins_token) ever learned. That is the single most useful negative result here.
Usage
Instructions ride as real text-token spans spliced in as extra timesteps:
span = [32000, *spm.encode("ask them about their job"), 32001] # <instr_start> β¦ <instr_end>
Fold into NVIDIA's base with the tested recipe from
duplex-brain:
huggingface-cli download MagicLuke/oif-instok-sft \
--include "stage2_ins_stage1f/checkpoint_000337/*" --local-dir adapter
python -m duplex_brain.ckpt.build_checkpoint \
--hf-repo nvidia/personaplex-7b-v1 \
--adapter adapter/stage2_ins_stage1f/checkpoint_000337/lora.safetensors --scaling 2.0 \
--out ckpt/duplex-brain-stage2.safetensors
--scaling 2.0 is the run's lora_scaling, applied directly, not divided by rank. Keep the
generated .depq.json and .vocab.json sidecars beside the weights.
β οΈ Two failure modes that load cleanly and are silently wrong
- The sidecars are not optional. Loading the
dep_q=8weights without.depq.jsonsilently duplicates agent heads into partner heads; without.vocab.json,<instr_start>silently aliases BOS. Keep both files beside the.safetensors. - The injection mode is required and cannot be inferred. An adapter's tensors cannot
distinguish
ins_tokenfrominline. Always passmode=ins_token; a wrong mode produces plausible-sounding, wrong output rather than an error.
Ids β₯ 32000 are not SentencePiece pieces β id_to_piece(32000) raises. Render them from the
.vocab.json sidecar, never through the tokenizer.
Caveats
- Free-run speech quality was never systematically evaluated during training. The documented
failure mode for this recipe family is the model going near-mute (13.2 β 3.9 words/clip) while
eval loss improves. Step 337 has downstream benchmark numbers, which is stronger evidence than
any offline metric here β but run
audio_health(β₯6 seeds; the failure is ~50% per-rollout stochastic) before trusting a checkpoint for generation. - No directive probe was run on any stage-2 checkpoint. Whether stage 2 preserved stage-1's steering is unmeasured, and stage 2 drops Fisher replay by design.
- Both runs were cut by a 3-day wall clock. Stage 1 stopped at 7000 of a planned 10000 steps,
so the OneCycle schedule annealed to ~4.5e-7 rather than to 0. Stage 2 used
batch_size 2(vs 4 in the reference recipe). - Stage-1-only models never saw a conv-synth goal card. Prompt them with a caller-seat task prompt (they are the user side, not an assistant) and expect loose adherence to goal lists.
- Eval delivers all instructions deterministically (no dropout), so eval loss is self-consistent across checkpoints but not comparable to training loss.
- Trained on Fisher (LDC) and Seamless Interaction; neither corpus is redistributed here.
- Base weights are NVIDIA's PersonaPlex β their licence governs the base model.
Provenance
Code: MagicLuke/duplex-online-if. Training data derived from Fisher (LDC), Seamless Interaction
and the conv-synth goal-arc corpus; instruction annotations from
MagicLuke/{fisher,seamless}-turn-instructions-v1. Benchmark outputs:
MagicLuke/ifbench-conversations-v1
and MagicLuke/fdb-v1-outputs-v1.
- Downloads last month
- -
# Gated model: Login with a HF token with gated access permission hf auth login