character-animate-4b

Qwen3.5-4B, fine-tuned to animate a character. You give it one sentence of action, such as "walks about 4 metres, then looks at a spot 2.5 m to the right". It writes a short JSON program that says how the body moves. A compiler in Blender turns the program into an animation clip and checks it.

The model drives the two people in a live scene: app.hedwigai.com/try/live-scene. The code that runs it is at github.com/un-seen/character-scene.

What is in this repository

• The fine-tuned weights, merged into Qwen3.5-4B (bf16).
• unmerge/: a LoRA adapter that takes the merged model back to Qwen3.5-4B. The scene server loads one engine and uses this adapter for the speaking model, which is the base model with a persona push. Without it you would need two copies of the model.

How to prompt it

The system prompt is the one it was trained with. It names no one and describes the body:

You write rigctl animation programs for one character as compact JSON, nothing else. Body: 1.79 m tall; eyes 1.69 m, shoulders 1.40 m, chest 1.28 m, waist 1.10 m; arm 0.50 m; leg 0.99 m. Face keys: blink, smile, brows_up, frown, jaw_open. Write plan first, then clip, style (only what differs from neutral), beats and keys.

The user message is the action, one sentence:

The character walks a few steps, then stops and stands, then looks at a spot 2.5 m to the right, then walks about 4 metres, then walks about 2 metres, then walks about 3 metres.

A program for it (trimmed to its first four beats, and shown expanded):

{"plan": "happy at 0.84: walk; idle; look_at; walk; walk; walk",
 "clip": {"fps": 24, "frames": 479, "seed": 5, "variation": 0.3},
 "style": {"speed_ms": 0.98, "cadence_spm": 109.4, "hip_bob_cm": 2.61, "arm_swing": 1.25, "lift_cm": 6.68},
 "beats": [{"id": "b0", "verb": "walk", "args": {"distance": 1.3}, "start": 0},
           {"id": "b1", "verb": "idle", "args": {}, "after": "b0", "dur": 36},
           {"id": "b2", "verb": "look_at", "args": {"target": [-2.5, 0.2, 1.55]}, "after": "b1", "dur": 36},
           {"id": "b3", "verb": "walk", "args": {"distance": 4.4}, "after": "b2"}]}

Decode it under the program's grammar (character/rigctl/compact.py and grammar.py in the code repository); the scene server does this with vLLM's structured outputs. A spoken line can come before the action, as the record of what was said ("[s1 I] "Whoa, just a little bit there, okay?""). The model then times gestures and expressions to the words.

How it was trained

• LoRA, rank 32, alpha 64, on every attention, linear-attention and MLP projection, then merged.
• Supervised fine-tuning on programs written for the action sentences of generated scenes. Each program was compiled in Blender and kept only if it passed the checker.
• Then rounds of rejection sampling. The model wrote several programs per prompt. The compiler and checker scored each one, and the passing programs went back into training. Hints were added for prompts that had no passing program. The last round used a learning rate of 5e-5 with one epoch.

How it scores

On our test suite of 196 action prompts (walking, running, turning, looking, reaching, gestures, sitting, kneeling, crouching, lying down and posed beats):

Prompts whose program does what was asked	0.99
Programs that compile	1.00
Clips that pass every check (balance, lean, feet, the body passing through itself, joint limits, pops)	1.00

These numbers are for our rig. The program's control names are that rig's (torso, hand_ik.L, f_index.02.L and so on). The rig itself is not published, so another rig needs those names mapped onto its own bones.

Limits

• It writes programs, not motion. Without the compiler and a rig, the output is only JSON.
• It knows one body. The body line in the prompt can change, but it was trained on one body's numbers.
• It moves one character at a time. The scene code decides where each one stands and whom they face.

Credits

This builds on open work, and we thank the people behind it.

• Qwen3.5-4B by the Qwen team (Apache 2.0), the base model.
• MPFB, MakeHuman for Blender (GPL 3), and MakeHuman's CC0 assets built the people the clips play on.
• Blender (GPL) compiled and checked every training program.
• PyTorch (BSD), Transformers and PEFT (Apache 2.0) trained it.
• vLLM (Apache 2.0) and flash-linear-attention (MIT) served it while it was sampled and scored.
• three.js (MIT) plays the clips in the browser.

License

Apache 2.0, as the base model is.