File size: 2,225 Bytes
262fa3f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | # Method
LatentASR adds continuous latent test-time scaling to a frozen ASR backbone.
The implementation wraps `Qwen/Qwen3-ASR-0.6B` with two lightweight modules:
- **Latent Adapter**: converts decoder hidden states into bounded latent
deltas.
- **Value Head**: predicts whether additional latent compute is useful and
enables dynamic halting.
## Prefix Layout
The default layout is prefix mode:
```text
system/user/audio prompt
assistant language <asr_text>
<|latent|> <|latent|> <|latent|> <|latent|>
transcript tokens
```
Transcript loss masks all prompt and latent positions.
## Stable Injection
Each latent delta is stabilized by three mechanisms:
1. **Bounded delta**: `delta_proj(h_k)` is L2-normalized and scaled by a
learned per-step scalar.
2. **Sigmoid gate**: a zero-initialized gate starts at `0.5` and learns how
much of the delta to apply.
3. **Fixed embedding anchor**: the injected vector is
`embedding(<|latent|>) + gate * delta`, keeping the input near a real token
embedding.
These mechanisms correspond to the ablation flags:
```bash
LATENT_USE_BOUNDED_DELTA=1
LATENT_USE_INJECTION_GATE=1
LATENT_USE_EMBEDDING_ANCHOR=1
```
## Value Head Target
For each training utterance, the value head predicts the latent-vs-baseline
accuracy gain. The default target is:
```text
0.9 * tanh(3.0 * (latent_accuracy - baseline_accuracy))
```
If both baseline and latent accuracies are zero for an utterance, the target
uses a clamped CE-difference fallback:
```text
0.9 * tanh(0.5 * clamp(baseline_ce - latent_ce, -2, 2))
```
With probability `VALUE_FORCED_NEG_PROB=0.3` per minibatch, the target is
flipped to `-|target|` for conservative calibration.
The current implementation supervises the value prediction at every produced
latent state, matching the fact that inference can halt after any step.
## Dynamic Halting
At inference time:
1. The initial value prediction gates the whole loop.
2. If `v_0 < theta`, all latent tokens are removed and generation falls back to
the frozen baseline.
3. After each latent step, the value head is re-evaluated.
4. If `v_k < theta`, unused latent tokens are removed before decoding.
The paper uses `theta=0.0` for the deployed setting.
|