| --- |
| library_name: transformers |
| pipeline_tag: any-to-any |
| base_model: |
| - LiquidAI/LFM3-Omni-2B |
| --- |
| |
| This is a randomly initialized tiny model based on [LiquidAI/LFM3-Omni-2B](https://huggingface.co/LiquidAI/LFM3-Omni-2B) for testing and debugging. |
|
|
| | Feature | Source | This repo | |
| |---|---|---| |
| | Checkpoint tensor elements | 2.24B BF16, 1.28K F32 | 13.91M BF16, 1.28K F32 | |
| | Safetensors weight files | 4.48 GB (2 files) | 27.88 MB (2 files) | |
| | Text layers / cycle | 24 / two-conv prefix then full/conv/sliding/conv | 10 / same prefix and two complete cycles | |
| | Text hidden / MLP width | 2,048 / 7,680 | 64 / 128 | |
| | Text Q / KV heads / head dim | 16 / 4 / 128 | 16 / 4 / 32 | |
| | Audio encoder layers / width | 17 / 768 | 6 / 64 (two conv/conv/full cycles) | |
| | Audio mel bins / stride / window | 128 / 8 / 32 frames | Same | |
| | Depth decoder layers / width | 3 / 1,024 | 3 / 32 | |
| | Codebooks / codebook size / maskgit | 12 / 2,017 / [4, 4, 4] | Same | |
| | Vision layers / width / Q-KV heads | 14 / 1,024 / 16-8 | 6 / 128 / 4-2 | |
| | Patch / temporal patch / merge | 16 / 2 / 2 | Same | |
| | Projector MLP hidden / downsample | 4,096 / 2 | 128 / 2 | |
| | Codec sample rate / frame samples | 24,000 / 1,920 | Same | |
| | Codec STFT n_fft / hop / frame stack | 1,280 / 320 / 6 | Same | |
| | Codec FSQ levels / codebooks | [8, 7, 6, 6] / 12 | Same | |
| | Codec LFM3 layers / width | 8 + 8 / 512 | 4 + 4 / 64 | |
| | Vocabulary rows / tying | 128,000 / tied | Same | |
| | Stored precision | BF16; codec ISTFT window FP32 | Same | |
| |
| Text/vision geometry follows [tiny-random/lfm3-vl](https://huggingface.co/tiny-random/lfm3-vl). The only runtime change lets the bundled codec LFM3 backbone read its dimensions from config. |
| |
| | File path | Size | |
| |------|------| |
| | audio_tokenizer/model.safetensors | 1.0MB | |
| | model.safetensors | 26.9MB | |
|
|
| ### Example usage |
|
|
| #### Transformers (unverified) |
|
|
| Load the included pinned upstream implementation with `trust_remote_code=True`. The example sends a 16 kHz spoken question together with an image and four predecoded video frames, answers as text, then synthesizes 24 kHz speech through the bundled LiquidCodec. The Triton short-conv path requires CUDA. |
|
|
| ```python |
| def run_transformers(model_id: str): |
| import numpy as np |
| import torch |
| from PIL import Image |
| from transformers import AutoModelForImageTextToText, AutoProcessor |
| |
| model_id = str(model_id) |
| processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True) |
| model = ( |
| AutoModelForImageTextToText.from_pretrained( |
| model_id, trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa" |
| ) |
| .eval() |
| .cuda() |
| ) |
| codec = processor.audio_tokenizer.to(device=model.device).eval() |
| |
| image = Image.fromarray(np.random.randint(0, 255, (256, 256, 3), dtype=np.uint8), "RGB") |
| frames = np.stack( |
| [ |
| np.asarray(Image.new("RGB", (256, 256), color)) |
| for color in ((220, 30, 30), (30, 220, 30), (30, 30, 220), (220, 220, 30)) |
| ] |
| ) |
| sample_rate = 16000 |
| speech = (np.sin(2 * np.pi * 440 * np.arange(sample_rate) / sample_rate) * 0.1).astype(np.float32) |
| conversation = [ |
| {"role": "system", "content": "Respond in text."}, |
| { |
| "role": "user", |
| "content": [ |
| {"type": "image", "image": image}, |
| {"type": "audio", "audio": speech}, |
| {"type": "video", "video": frames}, |
| {"type": "text", "text": "Transcribe the question, describe the image and the video."}, |
| ], |
| }, |
| ] |
| inputs = processor.apply_chat_template( |
| conversation, |
| add_generation_prompt=True, |
| tokenize=True, |
| return_dict=True, |
| return_tensors="pt", |
| sampling_rate=sample_rate, |
| processor_kwargs={"video_metadata": [{"total_num_frames": 4, "fps": 2, "frames_indices": [0, 1, 2, 3]}]}, |
| ).to(model.device) |
| assert ( |
| inputs["input_features"].shape[-1] > 0 |
| and inputs["pixel_values"].shape[0] > 0 |
| and inputs["pixel_values_videos"].shape[0] > 0 |
| ) |
| with torch.no_grad(): |
| answered = model.generate(**inputs, max_new_tokens=32, generation_mode="text") |
| text = processor.decode( |
| answered.sequences[0, inputs["input_ids"].shape[1] : answered.sequence_lengths[0]], |
| skip_special_tokens=True, |
| ) |
| print(text) |
| |
| tts_conversation = [ |
| {"role": "system", "content": "Perform TTS in American English using a female voice."}, |
| {"role": "user", "content": "The quick brown fox jumps over the lazy dog."}, |
| ] |
| tts_inputs = processor.apply_chat_template( |
| tts_conversation, |
| add_generation_prompt=True, |
| tokenize=True, |
| return_dict=True, |
| return_tensors="pt", |
| ).to(model.device) |
| with torch.no_grad(): |
| spoken = model.generate( |
| **tts_inputs, |
| max_new_tokens=64, |
| generation_mode="audio", |
| audio_temperature=0.4, |
| audio_repetition_penalty=1.2, |
| ) |
| assert spoken.audio_codes.shape[-1] > 0 |
| waveform = codec.decode(spoken.audio_codes[:1, :, : spoken.audio_lengths[0]]).audio_values[0] |
| assert waveform.numel() > 0 and torch.isfinite(waveform).all() |
| print(waveform.shape) |
| |
| |
| if __name__ == "__main__": |
| run_transformers(model_id="scratch/lfm3-omni") |
| ``` |
|
|
| ### Creating this repo |
|
|
| - Code: [.meta/create_model.py](.meta/create_model.py) |
| - Model structure: [.meta/model.log](.meta/model.log) |
| - Environment: [.meta/env.json](.meta/env.json) |
|
|
| ### Change log |
|
|
| - 2026-11-03: Initial version: omni text/vision/16kHz-audio encoder, maskgit depth decoder and a config-driven tiny bundled 24 kHz LiquidCodec. |
|
|
| *This repo is generated by GPT-6.1-Sol.* |
