GreenBitAI/Drama-3-8bit-paged

Drama 3 8-bit, paged for gbx-lm

Built with Fish Audio. Fish Audio Drama 3, the expressive multi-speaker text-to-speech model, packaged for gbx-lm on Apple silicon: an OpenAI-compatible /v1/audio/speech server that streams audio, and runs on Macs that cannot hold the whole model in memory.

This is a community build by GreenBitAI. It is not made, sponsored or endorsed by Fish Audio.

• Where the weights come from. Fish Audio's release, as converted to MLX 8-bit by mlx-community/fish-audio-drama-3-8bit (routed experts 8-bit, everything else as released). This repository changes how those weights are laid out on disk, not their values: the routed experts move into experts.bin, one record per expert, so that a machine can read just the experts it needs. Checked when the repository was built, against the mlx-community weights: identical output up to 127-token prompts, and every layer identical.
• Real time on 64 GB and up. 46-47 frames per second on an M3 Ultra, 2.2x real time, first audio in under 0.3 s when streaming.
• Runs on 32-48 GB, for offline use. With less memory the experts stream from the SSD: about half real time on a 48 GB Mac, a third on 32 GB. Generate the file, then play it.

What you need

Mac	Apple silicon; 64 GB or more for real time, 32 GB at least
Disk	40 GB for the model
gbx-lm	a release with speech support
Python	3.10 or newer, for the audio codec (below)

gbx-lm generates the model's audio codes; turning codes into sound is the job of a small companion package, gbx-codecs, which gbx-lm starts in its own process. It needs only mlx, numpy and miniaudio -- no PyTorch, no ffmpeg -- and about 3 GB of memory while it decodes. The codec weights are in this repository.

Setup

1. gbx-lm. The prebuilt binary, as in the gbx-lm README:

curl -fL -o gbx_lm-darwin-arm64.tar.gz 'https://github.com/GreenBitAI/gbx-lm/releases/latest/download/gbx_lm-darwin-arm64.tar.gz' \
  && tar -xzf gbx_lm-darwin-arm64.tar.gz gbx_lm

2. The codec, in a virtual environment of its own (this location is the one gbx-lm looks in by default):

python3 -m venv ~/.gbx-codecs
~/.gbx-codecs/bin/pip install gbx-codecs

3. Voices (optional). To speak in a particular voice, put a short clean recording (5-30 s; wav, flac or mp3) and its exact transcript side by side in a folder:

~/voices/
    narrator.wav
    narrator.txt      # the words spoken in narrator.wav

The request's voice is the file name, narrator. Use only voices you have the right to use -- see License. Without a voice ("voice": "default") the model chooses one; fix it with a seed.

4. Start the server. The model downloads on first start:

./gbx_lm --model GreenBitAI/Drama-3-8bit-paged --voices_dir ~/voices

If the codec is not in ~/.gbx-codecs, add --codec_python /path/to/that/python. To check the codec on its own once the model is downloaded:

~/.gbx-codecs/bin/gbx-codecs check ~/.libra/cache/models/hub/models--GreenBitAI--Drama-3-8bit-paged/snapshots/*/

Use

curl http://127.0.0.1:11688/v1/audio/speech -H "Content-Type: application/json" -d '{
  "model": "GreenBitAI/Drama-3-8bit-paged",
  "input": "[sighs] Fine. If we miss the train, that is on you. [laughs] Okay, okay, I am going.",
  "voice": "narrator",
  "response_format": "wav"
}' -o out.wav

With the OpenAI SDK, streaming as it is generated:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:11688/v1", api_key="local")
with client.audio.speech.with_streaming_response.create(
    model="GreenBitAI/Drama-3-8bit-paged",
    voice="narrator",
    input="<|speaker:0|> Are you coming tonight? <|speaker:1|> I wouldn't miss it.",
    response_format="pcm",                # 24 kHz, 16-bit, mono
    extra_body={"stream_format": "audio", "seed": 7},
) as response:
    for chunk in response.iter_bytes():
        ...                               # play or write it

field
input	the text. Inline cues such as [laughs], [sighs], [whispering]; speakers `<
voice	a name from --voices_dir, or default
response_format	wav (44.1 kHz, the default) or pcm (24 kHz, as OpenAI's). Other formats are refused
stream_format	audio to stream; absent, the whole file is returned
seed, temperature, top_p, sample_rate	extensions; from the OpenAI SDK, pass them in extra_body

speed other than 1.0 and instructions are not supported by this model: emotion and delivery come from the inline cues.

Pauses longer than a second are shortened in the output: the model occasionally goes silent mid-sentence for many seconds before finishing it.

Quality

32 texts in English and Chinese (plain, cued, two-speaker, and long enough to be split), one take each, transcribed by Whisper large-v3-turbo:

	English word error	Chinese character error
mlx-audio reference implementation, mlx-community 8-bit	0.0%	1.1-1.5%
this repository on gbx-lm, in memory	0.0-0.1%	1.3%
this repository on gbx-lm, streaming experts	0.1%	1.3%

No take had an error rate over 20%.

Speed

M3 Ultra, experts read from a cold SSD where they stream; the model makes 21.5 frames for each second of audio:

Mac	how	frames/s	x real time
64 GB and up	in memory	46-47	2.2
48 GB	streaming experts	10-13	0.5-0.6
32 GB	streaming experts	7-8.5	0.35-0.4

The smaller tiers are measured with the expert cache gbx-lm would give those Macs, on a Mac Studio's SSD; a laptop's is slower, so expect less. 24 GB Macs can load the model but run at under a quarter of real time. gbx-lm decides how to load the model from the memory it finds; GBX_PAGING=off forces it into memory.

License

The Fish Audio Community License, Version 1.0 applies to this model, including the outputs you make with it. In short, and the license itself governs:

• Free for research and non-commercial use, and for commercial use by organizations with annual revenue under USD 500,000. Above that, or to offer text-to-speech to third parties as a hosted service (an API or app whose purpose is speech generation from their text or voices), you need a separate license from Fish Audio (business@fish.audio).
• Do not generate speech imitating an identifiable person without their consent, impersonate anyone to deceive, or use the outputs to train other speech or foundation models.
• NOTICE records the attribution and every change made to Fish Audio's weights, including this repository's.

gbx-lm is Apache 2.0 and gbx-codecs is MIT. The codec implementation in gbx-codecs and the model implementation in gbx-lm are adapted from mlx-audio (MIT).

Credits

Fish Audio for Drama 3. cguldogan for the MLX 8-bit conversion published as mlx-community/fish-audio-drama-3-8bit. Prince Canuma and the mlx-audio contributors for the MLX implementation this build and its codec are based on.