kcz358/aero-realtime-4B

Aero Realtime

Aero Realtime is a native proactive audio-video language model whose input and output share one aligned autoregressive stream. The released 4B checkpoint is available at kcz358/aero-realtime-4B.

Realtime Interaction

Turn-based models separate input prefill from output decoding. Micro-turn systems shorten that boundary but repeatedly restart interaction. Aero Realtime instead advances audio, video, silence, and lexical output on one causal clock.

Model Architecture

Each approximately 80 ms audio slot is fused with the preceding output token. The language-model head predicts either the next lexical token or the learned silence token, jointly modeling when and what to generate.

Examples

Offline video demo

Run the bundled 60-second video through the realtime engine. The script feeds 80-ms audio chunks and 1-fps video frames in timestamp order and writes the per-interval outputs to results/offline_demo.jsonl.

python examples/offline/demo.py \
  --model kcz358/aero-realtime-4B \
  --deploy-config /path/to/vllm-omni/vllm_omni/deploy/aero_realtime.yaml

Pass --video /path/to/video.mp4 to use another video. Add --tensor-parallel-size N when the model should span multiple GPUs.

Online WebSocket demo

Start the online server in the first terminal. It keeps one Aero generation session alive and accepts chunkwise audio/video input over WebSocket.

python examples/online/server.py \
  --model kcz358/aero-realtime-4B \
  --deploy-config /path/to/vllm-omni/vllm_omni/deploy/aero_realtime.yaml \
  --port 8090

In a second terminal, stream the bundled video at its original wall-clock rate:

python examples/online/client.py --video assets/demo.mp4

Use --url ws://HOST:8090/v1/aero/realtime when the server runs remotely. For B200 inference, add --mm-encoder-attn-backend TRITON_ATTN to the server command.

Interactive 4B outputs, including seekable video, chunked audio waveforms, and per-slot silence/lexical predictions, are available on the project website.

Training

Training uses the main branch of LMMs-Engine. The two-stage 4B configs and launcher are under scripts/train/.

export AERO_WORKSPACE=/path/to/aero-workspace
bash scripts/train/launch.sh stage1
export AERO_STAGE1_MODEL=/path/to/merged-stage1
bash scripts/train/launch.sh stage2

The workspace must contain the dataset and initialization paths referenced by the YAML files. The launcher only starts training; environment and data setup remain external.

Evaluation

Evaluation uses the main branch of LMMs-Eval and the shard runner under scripts/eval/.

export AERO_MODEL_PATH=kcz358/aero-realtime-4B
export AERO_OUTPUT_DIR=$PWD/results/ovobench
bash scripts/eval/launch.sh

See scripts/eval/README.md for sharding and hardware overrides.