Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,362 @@
+---
+license: other
+license_name: lfm1.0
+license_link: LICENSE
+language:
+- en
+pipeline_tag: text-to-audio
+tags:
+- liquid
+- edge
+- lfm2.5-audio
+- lfm2.5
+- onnx
+- onnxruntime
+- webgpu
+- tts
+- asr
+- speech
+base_model:
+- LiquidAI/LFM2.5-Audio-1.5B
+---
+
+<div align="center">
+  <img
+    src="https://cdn-uploads.huggingface.co/production/uploads/61b8e2ba285851687028d395/2b08LKpev0DNEk6DlnWkY.png"
+    alt="Liquid AI"
+    style="width: 100%; max-width: 100%; height: auto; display: inline-block; margin-bottom: 0.5em; margin-top: 0.5em;"
+  />
+  <div style="display: flex; justify-content: center; gap: 0.5em; margin-bottom: 1em;">
+    <a href="https://playground.liquid.ai/"><strong>Try LFM</strong></a> •
+    <a href="https://docs.liquid.ai/lfm"><strong>Documentation</strong></a> •
+    <a href="https://leap.liquid.ai/"><strong>LEAP</strong></a>
+  </div>
+</div>
+
+# LFM2.5-Audio-1.5B-ONNX
+
+ONNX export of [LFM2.5-Audio-1.5B](https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B) for cross-platform inference.
+
+LFM2.5-Audio is a multimodal model supporting three modes:
+- **ASR** (Automatic Speech Recognition): Audio → Text
+- **TTS** (Text-to-Speech): Text → Audio
+- **Interleaved**: Mixed text and audio input/output
+
+## Recommended Variants
+
+| Decoder | Vocoder | Size | Platform | Use Case |
+|---------|---------|------|----------|----------|
+| Q4 | Q4 | ~1.5GB | WebGPU, Server | Recommended for most uses |
+| FP16 | FP16 | ~3.2GB | Server | Higher quality |
+
+- **WebGPU**: Use Q4 decoder + Q4 vocoder (Q8 not supported)
+- **Server**: Q4 for efficiency, FP16 for quality
+
+## Model Files
+
+```
+onnx/
+├── decoder.onnx                    # LFM2 backbone (FP32)
+├── decoder.onnx_data*
+├── decoder_fp16.onnx               # LFM2 backbone (FP16)
+├── decoder_fp16.onnx_data*
+├── decoder_q4.onnx                 # LFM2 backbone (Q4, recommended)
+├── decoder_q4.onnx_data
+├── audio_encoder.onnx              # Conformer encoder for ASR (FP32)
+├── audio_encoder.onnx_data
+├── audio_encoder_fp16.onnx         # Conformer encoder (FP16)
+├── audio_encoder_fp16.onnx_data
+├── audio_encoder_q4.onnx           # Conformer encoder (Q4)
+├── audio_encoder_q4.onnx_data
+├── audio_embedding.onnx            # Audio code embeddings (FP32)
+├── audio_embedding_fp16.onnx       # Audio code embeddings (FP16)
+├── audio_embedding_q4.onnx         # Audio code embeddings (Q4)
+├── audio_detokenizer.onnx          # Neural vocoder STFT (FP32)
+├── audio_detokenizer.onnx_data
+├── audio_detokenizer_fp16.onnx     # Neural vocoder (FP16)
+├── audio_detokenizer_fp16.onnx_data
+├── audio_detokenizer_q4.onnx       # Neural vocoder (Q4)
+├── audio_detokenizer_q4.onnx_data
+├── vocoder_depthformer.onnx        # Audio codebook prediction (FP32)
+├── vocoder_depthformer.onnx_data
+├── vocoder_depthformer_fp16.onnx   # Audio codebook prediction (FP16)
+├── vocoder_depthformer_fp16.onnx_data
+├── vocoder_depthformer_q4.onnx     # Audio codebook prediction (Q4)
+├── vocoder_depthformer_q4.onnx_data
+├── embed_tokens.bin                # Text embeddings (binary)
+├── embed_tokens.json               # Text embeddings metadata
+├── audio_embedding.bin             # Audio embeddings (binary, for direct lookup)
+├── audio_embedding.json            # Audio embeddings metadata
+└── mel_config.json                 # Mel spectrogram configuration
+
+* Large models (>2GB) split weights across multiple files:
+  decoder.onnx_data, decoder.onnx_data_1, decoder.onnx_data_2, etc.
+  All data files must be in the same directory as the .onnx file.
+```
+
+## Python
+
+Use the [onnx-export](https://github.com/Liquid4All/onnx-export) repository for inference.
+
+### Installation
+
+```bash
+git clone https://github.com/Liquid4All/onnx-export.git
+cd onnx-export
+uv sync
+```
+
+### ASR (Speech Recognition)
+
+Transcribe audio to text:
+
+```bash
+uv run lfm2-audio-infer /path/to/LFM2.5-Audio-1.5B-ONNX \
+    --mode asr \
+    --audio input.wav \
+    --precision q4
+```
+
+### TTS (Text-to-Speech)
+
+Generate audio from text:
+
+```bash
+uv run lfm2-audio-infer /path/to/LFM2.5-Audio-1.5B-ONNX \
+    --mode tts \
+    --prompt "Hello, this is a test of text to speech synthesis." \
+    --output output.wav \
+    --precision q4
+```
+
+Options:
+- `--system "Perform TTS. Use the UK female voice."` - Custom system prompt
+- `--audio-temperature 0.8` - Audio sampling temperature
+- `--audio-top-k 64` - Top-k sampling for audio
+
+### Interleaved (Mixed Audio/Text)
+
+Generate interleaved text and audio response from audio input:
+
+```bash
+uv run lfm2-audio-infer /path/to/LFM2.5-Audio-1.5B-ONNX \
+    --mode interleaved \
+    --audio input.wav \
+    --output output.wav \
+    --precision q4
+```
+
+Or from text prompt:
+
+```bash
+uv run lfm2-audio-infer /path/to/LFM2.5-Audio-1.5B-ONNX \
+    --mode interleaved \
+    --prompt "Respond with audio" \
+    --output output.wav \
+    --precision q4
+```
+
+### CLI Options
+
+```bash
+uv run lfm2-audio-infer --help
+```
+
+| Option | Description |
+|--------|-------------|
+| `--mode` | `asr`, `tts`, or `interleaved` |
+| `--precision` | `fp16`, `q4`, or `q8` (default: fp32) |
+| `--audio` | Input audio file (WAV) |
+| `--output` | Output audio file (WAV) |
+| `--prompt` | Text prompt |
+| `--system` | System prompt |
+| `--max-tokens` | Maximum tokens to generate |
+| `--temperature` | Text sampling temperature |
+| `--audio-temperature` | Audio sampling temperature |
+| `--audio-top-k` | Top-k sampling for audio |
+| `--seed` | Random seed for reproducibility |
+
+## WebGPU (Browser)
+
+### Installation
+
+```bash
+npm install onnxruntime-web @huggingface/transformers
+```
+
+### Enable WebGPU
+
+WebGPU is required for browser inference. To enable:
+
+1. **Chrome/Edge**: Navigate to `chrome://flags/#enable-unsafe-webgpu`, enable, and restart
+2. **Verify**: Check `chrome://gpu` for "WebGPU" status
+3. **Test**: Run `navigator.gpu.requestAdapter()` in DevTools console
+
+### Inference
+
+```javascript
+import * as ort from "onnxruntime-web/webgpu";
+import { AutoTokenizer } from "@huggingface/transformers";
+
+// Check WebGPU availability
+if (!navigator.gpu) {
+  throw new Error("WebGPU not available. Enable at chrome://flags/#enable-unsafe-webgpu");
+}
+
+ort.env.wasm.numThreads = 1;
+
+const modelId = "LiquidAI/LFM2.5-Audio-1.5B-ONNX";
+const modelBase = `https://huggingface.co/${modelId}/resolve/main`;
+
+// Load tokenizer
+const tokenizer = await AutoTokenizer.from_pretrained(modelId);
+
+// Load ONNX sessions
+async function loadSession(name, dataFiles = 1) {
+  const onnxPath = `${modelBase}/onnx/${name}.onnx`;
+  const externalData = [];
+  for (let i = 0; i < dataFiles; i++) {
+    const suffix = i === 0 ? "" : `_${i}`;
+    const fileName = `${name}.onnx_data${suffix}`;
+    externalData.push({ path: fileName, data: `${modelBase}/onnx/${fileName}` });
+  }
+  return ort.InferenceSession.create(onnxPath, {
+    executionProviders: ["webgpu"],
+    externalData,
+  });
+}
+
+// Load models (Q4 recommended for WebGPU)
+const decoder = await loadSession("decoder_q4");
+const audioEmbedding = await loadSession("audio_embedding_q4");
+const detokenizer = await loadSession("audio_detokenizer_q4");
+const depthformer = await loadSession("vocoder_depthformer_q4");
+
+// Load text embeddings binary
+const embedResponse = await fetch(`${modelBase}/onnx/embed_tokens.bin`);
+const embedBuffer = await embedResponse.arrayBuffer();
+const embedMetaResponse = await fetch(`${modelBase}/onnx/embed_tokens.json`);
+const embedMeta = await embedMetaResponse.json();
+const embedWeight = new Float32Array(embedBuffer);
+
+function getTextEmbeddings(ids) {
+  const hiddenSize = embedMeta.hidden_size;
+  const embeds = new Float32Array(ids.length * hiddenSize);
+  for (let i = 0; i < ids.length; i++) {
+    const offset = ids[i] * hiddenSize;
+    embeds.set(embedWeight.subarray(offset, offset + hiddenSize), i * hiddenSize);
+  }
+  return new ort.Tensor("float32", embeds, [1, ids.length, hiddenSize]);
+}
+
+// Model config
+const hiddenSize = 2048;
+const numCodebooks = 8;
+const codebookVocab = 2049;
+
+// TTS example
+const text = "Hello, this is a test.";
+const prompt = `<|startoftext|><|im_start|>system
+Perform TTS. Use the UK female voice.<|im_end|>
+<|im_start|>user
+${text}<|im_end|>
+<|im_start|>assistant
+`;
+
+const inputIds = tokenizer.encode(prompt);
+let embeds = getTextEmbeddings(inputIds);
+
+// Initialize KV cache
+const cache = {};
+for (const name of decoder.inputNames) {
+  if (name.startsWith("past_conv")) {
+    cache[name] = new ort.Tensor("float32", new Float32Array(hiddenSize * 3), [1, hiddenSize, 3]);
+  } else if (name.startsWith("past_key_values")) {
+    cache[name] = new ort.Tensor("float32", new Float32Array(0), [1, 8, 0, 64]);
+  }
+}
+
+// Generation loop
+const audioCodes = [];
+let inAudioMode = false;
+let curLen = inputIds.length;
+
+for (let step = 0; step < 1024; step++) {
+  const attentionMask = new ort.Tensor("int64", new BigInt64Array(curLen).fill(1n), [1, curLen]);
+  const outputs = await decoder.run({ inputs_embeds: embeds, attention_mask: attentionMask, ...cache });
+
+  // Update cache
+  for (const [name, tensor] of Object.entries(outputs)) {
+    if (name.startsWith("present_conv")) {
+      cache[name.replace("present_conv", "past_conv")] = tensor;
+    } else if (name.startsWith("present.")) {
+      cache[name.replace("present.", "past_key_values.")] = tensor;
+    }
+  }
+
+  if (inAudioMode) {
+    // Use depthformer to generate audio codes
+    const hiddenStates = outputs.hidden_states;
+    const lastHidden = /* extract last position */;
+
+    // Autoregressive codebook generation (8 steps per frame)
+    const frameCodes = await generateAudioFrame(depthformer, lastHidden);
+
+    if (frameCodes[0] === 2048) {
+      // End of audio
+      break;
+    }
+
+    audioCodes.push(frameCodes);
+
+    // Get audio embeddings for feedback
+    const audioTokens = frameCodes.map((code, cb) => cb * codebookVocab + code);
+    const audioEmbedsResult = await audioEmbedding.run({
+      audio_codes: new ort.Tensor("int64", new BigInt64Array(audioTokens.map(BigInt)), [1, 8])
+    });
+    // Sum embeddings across codebooks
+    embeds = sumEmbeddings(audioEmbedsResult.audio_embeds);
+  } else {
+    // Text generation
+    const logits = outputs.logits;
+    const nextToken = argmax(logits);
+
+    if (nextToken === 128) {
+      // <|audio_start|> - switch to audio mode
+      inAudioMode = true;
+    }
+
+    embeds = getTextEmbeddings([nextToken]);
+  }
+
+  curLen++;
+}
+
+// Decode audio codes to waveform using detokenizer + ISTFT
+const waveform = await decodeAudio(detokenizer, audioCodes);
+```
+
+### WebGPU Notes
+
+- Recommended: Q4 models for all components
+- Audio generation is autoregressive: 8 depthformer calls per audio frame
+- Each audio frame = 80ms of audio (24kHz, 320 hop length, 6x upsampling)
+- End-of-audio token is 2048 in any codebook
+- Large models (>2GB) split weights across multiple files
+
+## Audio Processing Details
+
+### Input (ASR)
+- Sample rate: 16kHz
+- Mel spectrogram: 128 bins, 512 FFT, 160 hop, 400 window
+- Pre-emphasis: 0.97
+
+### Output (TTS)
+- Sample rate: 24kHz
+- 8 codebooks with 2049 tokens each (0-2047 audio, 2048 end-of-audio)
+- STFT reconstruction: 1280 FFT, 320 hop
+- Detokenizer provides 6x temporal upsampling
+
+## License
+
+This model is released under the [LFM 1.0 License](LICENSE).