Instructions to use wudq/UEmo-D1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use wudq/UEmo-D1 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="wudq/UEmo-D1", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("wudq/UEmo-D1", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
UEmo-D1
Official implementation of UEmo-D1: An Efficient and Calibrated Decision Model for Unified Multimodal Emotion Recognition.
This repository contains the final, self-contained UEmo-D1 inference checkpoint, its tokenizer and media processor, and the code needed to predict an emotion from user-supplied candidate options. It supports text, images, audio, and video paired with audio. The checkpoint includes the complete text decoder, vision encoder, audio encoder, and final trained low-rank parameters; no separate base-model download is required. Low-rank parameters remain unmerged in their original precision to preserve the validated model outputs. Optimizer states, training data, evaluation data, intermediate checkpoints, and speech-generation components are not included.
Input template
Multimodal content [MEDIA], an instruction [INST], and candidate options [CHOICES] are assembled into a shared decision input:
[MEDIA]
[INST]
Choose your answer from: A. [CHOICE 1]; B. [CHOICE 2]; C. [CHOICE 3]; ...
Answer with only the single option letter, without explanation.
The inference script expands the options onto separate lines and applies the original system message, media markers, and chat template. Optional preceding context appears before the target. Supply 1–26 unique options in the desired order; the output is their corresponding letter, not a fixed global emotion ID.
Installation
The validated runtime uses Linux, Python 3.12, CUDA, and FlashAttention 2. The release was checked on an NVIDIA RTX 4090. Memory needs increase with audio duration, video length, and context size.
pip install huggingface_hub==0.35.3
hf download wudq/UEmo-D1 --local-dir ./UEmo-D1
cd UEmo-D1
pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt
pip install flash-attn==2.8.3 --no-build-isolation
Use the pinned Transformers version: the bidirectional-attention integration checks the model implementation it was validated against. The loader uses trust_remote_code=True to load the included UEmo-D1 architecture.
Inference
Create a text input and a JSON request:
printf '%s\n' 'I am delighted that we can celebrate together today.' > sample.txt
cat > sample.json <<'JSON'
{
"multimodal": {
"text": ["sample.txt"],
"image": [],
"audio": [],
"video": []
},
"instruction": "Identify the emotion expressed in the text.",
"choices": ["happy", "sad", "angry", "neutral"]
}
JSON
python inference.py --model . --input sample.json
The command prints only one option letter. To also return the selected label and calibrated option probabilities:
python inference.py --model . --input sample.json --probabilities
Paths are resolved relative to the input JSON file, or to an explicit --media-root. All modality entries are file paths, including UTF-8 text files. Populate the corresponding lists for other modalities:
- Image:
"image": ["face.jpg"]. - Audio:
"audio": ["speech.wav"]. - Video with audio:
"video": ["clip.mp4"]and"audio": ["clip.wav"], where each audio file is aligned with the video at the same list position. The script does not silently extract or discard video audio. - Context: add a
"context"object inside"multimodal", containing its owntext,image,audio, andvideolists.
The preprocessing retains full audio, converts it to mono 16 kHz, and samples video at 2 FPS. Image and video pixel budgets follow the final inference pipeline (448² and 224² maximum pixels respectively). No fixed 16-frame API cap is applied.
Decision probabilities
UEmo-D1 reads the next-token logits for the valid option letters and computes:
probabilities = softmax(option_logits / 1.0790077648040843)
answer = option_letters[argmax(probabilities)]
The fixed temperature is applied exactly once to the valid-choice logits. It is already handled by inference.py; do not apply another temperature or replace this step with sampling-temperature settings in generate(). The model uses bidirectional attention over the complete input prefix. Use the included loader rather than loading these weights as an ordinary causal Qwen model.
License
Built with Qwen. This derivative retains the Qwen Research License and the attribution in NOTICE. Use is for non-commercial research or evaluation under that license; commercial use requires the upstream licensor's authorization.
- Downloads last month
- 22
Model tree for wudq/UEmo-D1
Base model
Qwen/Qwen2.5-Omni-3B