johnbean393/fluid-2-qwen3.5-2b-beta

Fluid 2 Qwen3.5 2B Beta

This private full-precision release is the final step-344 checkpoint of a full-parameter Fluid 2 dictation-cleaning SFT run.

Training snapshot

• Full fine-tune: 1,881,825,088 / 1,881,825,088 parameters trainable
• Training represented by this checkpoint: exactly 1 epoch(s)
• LR horizon: 688 optimizer steps (2 epochs), while training stopped after 344 steps
• LR schedule: cosine, peak 3e-5, 21 warmup steps; this is exactly the first half of the prior two-epoch schedule
• Dataset split: deterministic document-grouped 96/2/2
• Train / validation / test rows: 340,236 / 7,161 / 7,152
• Packing: padding-free best-fit decreasing at 8,192 tokens
• Objective: completion-only loss on the cleaned dictation target
• Configured in-training validation events / captured private checkpoints: 20 / 5
• Final validation loss: 0.5301376581192017
• Final validation mean token accuracy: 0.873393036665455
• Audio was excluded from loading and training

The five checkpoint-* folders include optimizer, scheduler, RNG, trainer state, and a hashed run contract. The model and tokenizer at the repository root are the final inference checkpoint.

SGLang development-set evaluation

The following strict scores use the public 7,161-row Fluid 2 development set. They are model-selection metrics, not a blind-test result.

Metric	Result
Scored text rows	7,016
Exact match	30.3449%
CER	17.4199%
WER	27.4544%
Excluded EOS-only rows	121
Excluded generation-capped rows	24

Metric scope: EM, CER, and WER exclude both rows with an empty clean_target and non-empty-target generations whose finish_reason is length. Capped requests remain in the failure and throughput census. Scores are strict and remain case-, punctuation-, and formatting-sensitive.

• Evaluated model revision: f71a0445a8f1f4e9bc682ae532c6a69660eb3361
• Eval dataset revision: c1711ac9eec8706c776c58148ae03e643b7fd77f
• Evaluation W&B run: https://wandb.ai/johnbean393-new-york-university/fluid-2/runs/fluid2-eval-158489c101b767b7

Prompt template

Do not apply a chat template. Send this raw completion prompt and stop on <|end_target_text|>:

<|dictation_clean_v1|>
<|start_prev_text|>{previous context}<|end_prev_text|>
<|start_post_text|>{following context}<|end_post_text|>
<|start_asr_text|>{ASR transcript to clean}<|end_asr_text|>
<|start_target_text|>

Previous and following context may be empty, but retain every marker pair. The nine Fluid control tokens occupy existing spare vocabulary rows 248077..248085; the embedding tables were not resized.

Run with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "johnbean393/fluid-2-qwen3.5-2b-beta"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    dtype="auto",
    device_map="auto",
)

prompt = '''<|dictation_clean_v1|>
<|start_prev_text|><|end_prev_text|>
<|start_post_text|><|end_post_text|>
<|start_asr_text|>hello world<|end_asr_text|>
<|start_target_text|>'''
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

MTP / NextN

The base configuration declares one optional MTP/NextN speculative draft layer, but the causal-LM checkpoint contains only its trained decoder layers and no draft-layer tensors. Normal next-token inference uses all trained decoder weights. GGUF releases therefore use --no-mtp only to avoid advertising an absent speculative layer.