LoveMachine / README.md
Koppen9029's picture
Publish LoveMachine research preview
05fbb7d verified
|
Raw History Blame Contribute Delete
3.64 kB
metadata
language:
  - ja
base_model: Qwen/Qwen2.5-Omni-3B
license: other
license_name: qwen-research
license_link: https://huggingface.co/Qwen/Qwen2.5-Omni-3B/blob/main/LICENSE
tags:
  - multimodal
  - streaming
  - research

LoveMachine · Research Preview

Built with Qwen. LoveMachine explores whether a local agent can keep updating a numerical state while receiving RGB video and PCM audio, revisit earlier evidence, and decide when a Japanese text response would help.

The full research system uses one Qwen2.5-Omni-3B Thinker on the GPU and smaller numerical components for recurrent state, evidence selection, and action decisions. These components operate on numerical features. Text is the outward action. This preview includes selected source modules, the trained evidence-action policy, and evaluation summaries. The policy accepts 13 numerical features produced by the evidence comparison code and chooses among waiting, reusing evidence, and seeking evidence. The complete streaming application requires additional runtime code and checkpoints.

Try the preview

python -m pip install torch safetensors
python demo.py

The demo runs the released policy on nine saved numerical inputs from three development episodes and prints the recorded drum comparison. The examples were selected to show all three actions. The full development counts are below. It runs on CPU with the included weights.

Results and limitations

Setting Observed result
224 development episodes with three matched evidence conditions All three actions correct in 176 episodes. The correct prior reference ranked first in 212 episodes.
Eight controlled scenes assembled from real recordings All eight generated requests received a new reference that the agent selected. Seven subsequent reports mentioned the annotated source. The unchanged-sound condition was silent in all eight.
Paced drum comparison The agent reported the drums after the responding environment supplied footage. Without the footage it made no follow-up report. The two conditions used the same input through the request.
Three unedited public benchmark videos Two utterances. Neither specifically described an annotated reference event. This diagnostic run omitted benchmark task prompts.

The controlled trials show behavior that depends on prior evidence and on what arrives after the agent acts. Open-ended curiosity, useful timing across arbitrary scenes, and sustained learning from experience remain unverified. See the recorded comparisons for counts, conditions, and the actual drum utterances.

Discussion

What comparison would convince you that an agent is following a meaningful uncertainty across time? Where would the same behavior be explainable by a simple change detector? Examples where the model should remain silent are especially welcome.

This preview is for research and evaluation. See the Qwen Research License for upstream terms. The upstream model files are available from the official Qwen repository.


LoveMachineは、映像と音声を受け続けながら、過去の観測を参照し直し、沈黙・報告・確認を選ぶ研究試作です。公開版では、判断に使う小型の学習済み重みと、同じ入力で応答を返す場合・返さない場合の実験記録を読めます。自由な場面での好奇心や発言の有用性は未確認です。「どういう対照なら、意味のある関心を追っていると言えるか」をぜひ議論してください。