File size: 5,771 Bytes
c8adba0 b04a81e c8adba0 b04a81e c8adba0 b04a81e c8adba0 b04a81e c8adba0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 | ---
library_name: transformers
pipeline_tag: any-to-any
base_model:
- LiquidAI/LFM3-Omni-2B
---
This is a randomly initialized tiny model based on [LiquidAI/LFM3-Omni-2B](https://huggingface.co/LiquidAI/LFM3-Omni-2B) for testing and debugging.
| Feature | Source | This repo |
|---|---|---|
| Checkpoint tensor elements | 2.24B BF16, 1.28K F32 | 13.91M BF16, 1.28K F32 |
| Safetensors weight files | 4.48 GB (2 files) | 27.88 MB (2 files) |
| Text layers / cycle | 24 / two-conv prefix then full/conv/sliding/conv | 10 / same prefix and two complete cycles |
| Text hidden / MLP width | 2,048 / 7,680 | 64 / 128 |
| Text Q / KV heads / head dim | 16 / 4 / 128 | 16 / 4 / 32 |
| Audio encoder layers / width | 17 / 768 | 6 / 64 (two conv/conv/full cycles) |
| Audio mel bins / stride / window | 128 / 8 / 32 frames | Same |
| Depth decoder layers / width | 3 / 1,024 | 3 / 32 |
| Codebooks / codebook size / maskgit | 12 / 2,017 / [4, 4, 4] | Same |
| Vision layers / width / Q-KV heads | 14 / 1,024 / 16-8 | 6 / 128 / 4-2 |
| Patch / temporal patch / merge | 16 / 2 / 2 | Same |
| Projector MLP hidden / downsample | 4,096 / 2 | 128 / 2 |
| Codec sample rate / frame samples | 24,000 / 1,920 | Same |
| Codec STFT n_fft / hop / frame stack | 1,280 / 320 / 6 | Same |
| Codec FSQ levels / codebooks | [8, 7, 6, 6] / 12 | Same |
| Codec LFM3 layers / width | 8 + 8 / 512 | 4 + 4 / 64 |
| Vocabulary rows / tying | 128,000 / tied | Same |
| Stored precision | BF16; codec ISTFT window FP32 | Same |
Text/vision geometry follows [tiny-random/lfm3-vl](https://huggingface.co/tiny-random/lfm3-vl). The only runtime change lets the bundled codec LFM3 backbone read its dimensions from config.
| File path | Size |
|------|------|
| audio_tokenizer/model.safetensors | 1.0MB |
| model.safetensors | 26.9MB |
### Example usage
#### Transformers (unverified)
Load the included pinned upstream implementation with `trust_remote_code=True`. The example sends a 16 kHz spoken question together with an image and four predecoded video frames, answers as text, then synthesizes 24 kHz speech through the bundled LiquidCodec. The Triton short-conv path requires CUDA.
```python
def run_transformers(model_id: str):
import numpy as np
import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor
model_id = str(model_id)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = (
AutoModelForImageTextToText.from_pretrained(
model_id, trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa"
)
.eval()
.cuda()
)
codec = processor.audio_tokenizer.to(device=model.device).eval()
image = Image.fromarray(np.random.randint(0, 255, (256, 256, 3), dtype=np.uint8), "RGB")
frames = np.stack(
[
np.asarray(Image.new("RGB", (256, 256), color))
for color in ((220, 30, 30), (30, 220, 30), (30, 30, 220), (220, 220, 30))
]
)
sample_rate = 16000
speech = (np.sin(2 * np.pi * 440 * np.arange(sample_rate) / sample_rate) * 0.1).astype(np.float32)
conversation = [
{"role": "system", "content": "Respond in text."},
{
"role": "user",
"content": [
{"type": "image", "image": image},
{"type": "audio", "audio": speech},
{"type": "video", "video": frames},
{"type": "text", "text": "Transcribe the question, describe the image and the video."},
],
},
]
inputs = processor.apply_chat_template(
conversation,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
sampling_rate=sample_rate,
processor_kwargs={"video_metadata": [{"total_num_frames": 4, "fps": 2, "frames_indices": [0, 1, 2, 3]}]},
).to(model.device)
assert (
inputs["input_features"].shape[-1] > 0
and inputs["pixel_values"].shape[0] > 0
and inputs["pixel_values_videos"].shape[0] > 0
)
with torch.no_grad():
answered = model.generate(**inputs, max_new_tokens=32, generation_mode="text")
text = processor.decode(
answered.sequences[0, inputs["input_ids"].shape[1] : answered.sequence_lengths[0]],
skip_special_tokens=True,
)
print(text)
tts_conversation = [
{"role": "system", "content": "Perform TTS in American English using a female voice."},
{"role": "user", "content": "The quick brown fox jumps over the lazy dog."},
]
tts_inputs = processor.apply_chat_template(
tts_conversation,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
with torch.no_grad():
spoken = model.generate(
**tts_inputs,
max_new_tokens=64,
generation_mode="audio",
audio_temperature=0.4,
audio_repetition_penalty=1.2,
)
assert spoken.audio_codes.shape[-1] > 0
waveform = codec.decode(spoken.audio_codes[:1, :, : spoken.audio_lengths[0]]).audio_values[0]
assert waveform.numel() > 0 and torch.isfinite(waveform).all()
print(waveform.shape)
if __name__ == "__main__":
run_transformers(model_id="scratch/lfm3-omni")
```
### Creating this repo
- Code: [.meta/create_model.py](.meta/create_model.py)
- Model structure: [.meta/model.log](.meta/model.log)
- Environment: [.meta/env.json](.meta/env.json)
### Change log
- 2026-11-03: Initial version: omni text/vision/16kHz-audio encoder, maskgit depth decoder and a config-driven tiny bundled 24 kHz LiquidCodec.
*This repo is generated by GPT-6.1-Sol.* |