what about --audio
to give voices to subjects, can we add --audio reference?
An example of 'stage directing an actor' to deliver his lines. audio only output
Many famous people in the model can be 'summoned' by prompt directions. here is obama reading green eggs and ham
Those clips are a great demonstration of something real: the voice lives in prompt space. There is no separate TTS stage to feed — the DiT denoises one joint [text | audio | video] stream, so everything the audio becomes is steered by the same conditioning the video gets.
On --audio reference.wav as an input: I have to be straight about the architecture. H3's conditioning interface is text tokens plus (optionally) first/last frame latents. There is no audio encoder pathway into the conditioning — audio exists only on the output side of the DiT, decoded by the audio VAE. So a reference-audio flag isn't a matter of wiring up a CLI option; it would need a trained audio tower + projection into the conditioning stream — the audio twin of what ClipProj does for vision. It's an interesting direction and I've noted it, but I won't pretend it's one flag away.
What works today, and your own clips show it working:
- Name the voice and direct it. Speaker identity, timbre, pacing, emotion, accent — as stage directions in the prompt ("a gravelly baritone, slow, weary", "reads in the cadence of a bedtime story"). The MiniMax spec-style prompts you're already writing are the right shape.
- Keep the seed and the description fixed across takes of the same subject; the voice is a function of both, and changing either changes the actor.
If you want to push it further, the strongest lever is the same one as for faces: the fuller the description, the tighter the DiT's posterior around the voice you meant.
"On --audio reference.wav as an input: I have to be straight about the architecture. H3's conditioning interface is text tokens plus (optionally) first/last frame latents. "
i think mini max h3 supports this . you can pass in a reference audio and video to it.
but they may have limited this in their open source version so people have to use the api. Because their documentation shows they support it