VoiceMem-Jev

VoiceMem-Jev selects the reasoning depth for a voice conversation turn. It combines the current transcript, recent dialogue and optional speech-state observations to return ordinary or deep, together with their probabilities.

The model supports Chinese and English and uses a 140.8M-parameter jhu-clsp/mmBERT-small encoder with a custom classification head. Speech-state inputs use shared physical quantities, so integration does not depend on a particular ASR provider.

Inputs and outputs

Input	Description
text	Current ASR transcript
history	Recent user and assistant messages; optional
eot_probability	End-of-turn probability in [0, 1]; optional
silence_s	Silence duration in seconds; optional
span_s	Voiced-span duration in seconds; optional

The output contains label, p_ordinary and p_deep. A deep probability of 0.5 or greater selects deep. Omit unavailable observations; an observed zero and a missing value have different masks.

ASR, turn detection, memory retrieval, reply generation and speech synthesis are provided by the host application.

Repository layout

Path	Contents
eager/	FP32 Safetensors weights, configuration and tokenizer
onnx/	FP32 ONNX weights, configuration and tokenizer
python/	Python runtime and VoiceMem Studio adapter
EVALUATION.json	Evaluation results and measurement conditions
TRAINING.json	Training method and calibration settings
SOURCE.json	Model provenance and runtime source hashes
SHA256SUMS	Release file checksums

Keep each weight file with its matching configuration and tokenizer. The classification head requires the supplied runtime.

CPU inference

hf download Sorbett/VoiceMem-Jev --revision main \
  --include 'onnx/*' 'python/*' 'README.md' 'LICENSE' 'NOTICE.txt' 'licenses/*' \
  --local-dir ./VoiceMem-Jev
python -m pip install './VoiceMem-Jev/python[onnx]'

from voicemem_jev import OnnxRouter

router = OnnxRouter('./VoiceMem-Jev/onnx')
decision = router.predict(
    'Can you compare these options and explain the trade-offs?',
    history=[{'role': 'user', 'content': 'I am choosing an apartment.'}],
    observations={'eot_probability': 0.9, 'silence_s': 0.4, 'span_s': 3.0},
)
print(decision.label, decision.p_deep)

CPU inference uses NumPy, Tokenizers and ONNX Runtime. The standalone CPU runtime does not require Studio, PyTorch or Transformers.

CUDA inference

hf download Sorbett/VoiceMem-Jev --revision main \
  --include 'eager/*' 'python/*' 'README.md' 'LICENSE' 'NOTICE.txt' 'licenses/*' \
  --local-dir ./VoiceMem-Jev
python -m pip install './VoiceMem-Jev/python[eager]'

from voicemem_jev import Router

router = Router(
    './VoiceMem-Jev/eager', backend='eager', device='cuda:0',
)
print(router.predict('How are you today?'))

For reproducible deployments, replace the branch name in download commands with the published commit SHA.

VoiceMem Studio integration

Install the runtime in Studio's Python environment. In Settings > Components > Router > Adapter, configure:

Setting	Value
Factory	voicemem_jev.studio_adapter:create
Model	Absolute path to the downloaded onnx/ or eager/ directory
API Key / Base URL	Leave empty

CPU ONNX options:

{"backend":"onnx","device":"cpu","allow_prototype":true}

CUDA options with compilation:

{"backend":"eager","device":"cuda:0","allow_prototype":true,"compiled":true}

The adapter prewarms five sequence-length buckets before serving. Compilation requires a CUDA device. The compiled runtime can recover to the same-weight eager CUDA model when its execution environment changes.

The adapter requires Studio's studio.core.utils.reply_modes.encoder_depth.EncoderDepthRouter interface. Studio owns asynchronous execution, cancellation, caching and device admission. The checkpoint retains its evaluation-stage deployment flag, so the adapter requires allow_prototype=true.

Model and training

The current transcript and recent dialogue are encoded separately. Their representations and elementwise interaction are combined with speech-state features and missing-value masks for classification.

Training uses cross-entropy supervision followed by RLCD optimization, with public and authored dialogue, synthetic ASR variants and Qwen3-8B supervision. Temperature calibration is fitted on a separate calibration split. Training settings are recorded in TRAINING.json.

The runtime retains the last four user/assistant messages, with 80 characters per message, and encodes up to 384 tokens per text. The calibrated temperature is 0.6. Keep these preprocessing and calibration settings with the weights.

Evaluation

The development regression contains 664 machine-labelled examples: 303 English and 361 Chinese, with 150 deep and 514 ordinary labels.

Metric	Result
Overall accuracy	92.92% (617 / 664)
English accuracy	90.43%
Chinese accuracy	95.01%
Deep recall	78.00%
Ordinary false-deep rate	2.72%

The regression was used during development and contains related translations and variants. It does not establish accuracy on independent natural speech. An authored context diagnostic with 12 scenario families and 360 variants reports 80.83% accuracy and 8.33% context-deep recall; context-dependent follow-up routing remains a limitation.

Warm single-decision latency includes tokenization and disables result caching:

Runtime	p50 / p95	Measurement
NVIDIA L20, eager FP32	10.60 / 11.42 ms	128 inputs, two passes
CPU ONNX, 2 threads	47.48 / 121.95 ms	128 inputs, two passes
NVIDIA L20, compiled FP32	2.21 / 3.22 ms	664 inputs, two passes

Initial CUDA compilation and prewarming take approximately 61 seconds, excluding model loading. Measurements use Linux and exclude queueing and the rest of the voice pipeline. Native macOS, Windows and concurrent ASR/TTS performance have not been benchmarked. See EVALUATION.json for details.

License

VoiceMem-Jev is distributed under Apache-2.0. The mmBERT base model is MIT-licensed. Attribution and source revisions are provided in LICENSE, NOTICE.txt, SOURCE.json and licenses/.