Audio logits (talker head) can contain NaN from FP16 overflow when
vision features are present. Guard with nan_to_num before softmax
and multinomial, matching existing text path protection.
Revert camera prompt changes; add NaN guard and CUDA sync; add engineering challenges doc
- prepare_turn: restored original '请描述这张图片' prompt + image tokens after text
- build_ids: removed system prompt and image_context parameter
- model.py stream_generate: added nan_to_num guard on logits/probs
- omni_o_call.py: added torch.cuda.synchronize() after ASR before generation
- docs/interview/challenges.md: new document covering real engineering challenges
- docs/interview/index.md: added link to challenges.md
Cast logits to float32 before softmax/sampling to prevent fp16 overflow
- Text logits: .clone().float() / temperature
- Audio logits: .clone().float() / 0.2
fp16 overflow in attention or lm_head can produce inf logits, making
softmax output NaN and crashing multinomial with CUDA device-side assert.
Fix from_pretrained: use SenseVoice/Siglip encoder wrappers instead of raw models
VAM.__init__ wraps encoders in SenseVoiceAudioEncoder / SiglipVisionEncoder
(expected by encode_audio_inputs / get_image_embeddings), but from_pretrained
override used load_sensevoice/load_vision which returned raw encoder modules.
Convert omni-o.pth to HF format + add from_pretrained support
- scripts/convert_omni_o_to_hf.py: convert .pth -> config.json + model.safetensors
- src/models/vam/model.py: from_pretrained override loads encoders after meta init
- src/models/vam/model.py: __init__ skips encoder loading on meta device
- scripts/omni_o_call.py: auto-detect HF dir and load via VAM.from_pretrained
- auto_map in config.json points to modeling_omni_o.py for trust_remote_code
Fix generate() to return last text tokens instead of (None, audio_frame) tuple
When stream=False, generate() now finds the last yield with non-None
text tokens instead of returning tokens[-1] which is (None, audio_frame)
after text generation finishes.
rename minimind→omni in model_type, eval/convert scripts, and docs
