---
library_name: transformers
pipeline_tag: any-to-any
base_model:
- LiquidAI/LFM3-Omni-2B
---

This is a randomly initialized tiny model based on [LiquidAI/LFM3-Omni-2B](https://huggingface.co/LiquidAI/LFM3-Omni-2B) for testing and debugging.

| Feature | Source | This repo |
|---|---|---|
| Checkpoint tensor elements | 2.24B BF16, 1.28K F32 | 13.91M BF16, 1.28K F32 |
| Safetensors weight files | 4.48 GB (2 files) | 27.88 MB (2 files) |
| Text layers / cycle | 24 / two-conv prefix then full/conv/sliding/conv | 10 / same prefix and two complete cycles |
| Text hidden / MLP width | 2,048 / 7,680 | 64 / 128 |
| Text Q / KV heads / head dim | 16 / 4 / 128 | 16 / 4 / 32 |
| Audio encoder layers / width | 17 / 768 | 6 / 64 (two conv/conv/full cycles) |
| Audio mel bins / stride / window | 128 / 8 / 32 frames | Same |
| Depth decoder layers / width | 3 / 1,024 | 3 / 32 |
| Codebooks / codebook size / maskgit | 12 / 2,017 / [4, 4, 4] | Same |
| Vision layers / width / Q-KV heads | 14 / 1,024 / 16-8 | 6 / 128 / 4-2 |
| Patch / temporal patch / merge | 16 / 2 / 2 | Same |
| Projector MLP hidden / downsample | 4,096 / 2 | 128 / 2 |
| Codec sample rate / frame samples | 24,000 / 1,920 | Same |
| Codec STFT n_fft / hop / frame stack | 1,280 / 320 / 6 | Same |
| Codec FSQ levels / codebooks | [8, 7, 6, 6] / 12 | Same |
| Codec LFM3 layers / width | 8 + 8 / 512 | 4 + 4 / 64 |
| Vocabulary rows / tying | 128,000 / tied | Same |
| Stored precision | BF16; codec ISTFT window FP32 | Same |

Text/vision geometry follows [tiny-random/lfm3-vl](https://huggingface.co/tiny-random/lfm3-vl). The only runtime change lets the bundled codec LFM3 backbone read its dimensions from config.

| File path | Size |
|------|------|
| audio_tokenizer/model.safetensors | 1.0MB |
| model.safetensors | 26.9MB |

### Example usage

#### Transformers (unverified)

Load the included pinned upstream implementation with `trust_remote_code=True`. The example sends a 16 kHz spoken question together with an image and four predecoded video frames, answers as text, then synthesizes 24 kHz speech through the bundled LiquidCodec. The Triton short-conv path requires CUDA.

```python
def run_transformers(model_id: str):
    import numpy as np
    import torch
    from PIL import Image
    from transformers import AutoModelForImageTextToText, AutoProcessor

    model_id = str(model_id)
    processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
    model = (
        AutoModelForImageTextToText.from_pretrained(
            model_id, trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa"
        )
        .eval()
        .cuda()
    )
    codec = processor.audio_tokenizer.to(device=model.device).eval()

    image = Image.fromarray(np.random.randint(0, 255, (256, 256, 3), dtype=np.uint8), "RGB")
    frames = np.stack(
        [
            np.asarray(Image.new("RGB", (256, 256), color))
            for color in ((220, 30, 30), (30, 220, 30), (30, 30, 220), (220, 220, 30))
        ]
    )
    sample_rate = 16000
    speech = (np.sin(2 * np.pi * 440 * np.arange(sample_rate) / sample_rate) * 0.1).astype(np.float32)
    conversation = [
        {"role": "system", "content": "Respond in text."},
        {
            "role": "user",
            "content": [
                {"type": "image", "image": image},
                {"type": "audio", "audio": speech},
                {"type": "video", "video": frames},
                {"type": "text", "text": "Transcribe the question, describe the image and the video."},
            ],
        },
    ]
    inputs = processor.apply_chat_template(
        conversation,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
        sampling_rate=sample_rate,
        processor_kwargs={"video_metadata": [{"total_num_frames": 4, "fps": 2, "frames_indices": [0, 1, 2, 3]}]},
    ).to(model.device)
    assert (
        inputs["input_features"].shape[-1] > 0
        and inputs["pixel_values"].shape[0] > 0
        and inputs["pixel_values_videos"].shape[0] > 0
    )
    with torch.no_grad():
        answered = model.generate(**inputs, max_new_tokens=32, generation_mode="text")
    text = processor.decode(
        answered.sequences[0, inputs["input_ids"].shape[1] : answered.sequence_lengths[0]],
        skip_special_tokens=True,
    )
    print(text)

    tts_conversation = [
        {"role": "system", "content": "Perform TTS in American English using a female voice."},
        {"role": "user", "content": "The quick brown fox jumps over the lazy dog."},
    ]
    tts_inputs = processor.apply_chat_template(
        tts_conversation,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
    ).to(model.device)
    with torch.no_grad():
        spoken = model.generate(
            **tts_inputs,
            max_new_tokens=64,
            generation_mode="audio",
            audio_temperature=0.4,
            audio_repetition_penalty=1.2,
        )
        assert spoken.audio_codes.shape[-1] > 0
        waveform = codec.decode(spoken.audio_codes[:1, :, : spoken.audio_lengths[0]]).audio_values[0]
    assert waveform.numel() > 0 and torch.isfinite(waveform).all()
    print(waveform.shape)


if __name__ == "__main__":
    run_transformers(model_id="scratch/lfm3-omni")
```

### Creating this repo

- Code: [.meta/create_model.py](.meta/create_model.py)
- Model structure: [.meta/model.log](.meta/model.log)
- Environment: [.meta/env.json](.meta/env.json)

### Change log

- 2026-11-03: Initial version: omni text/vision/16kHz-audio encoder, maskgit depth decoder and a config-driven tiny bundled 24 kHz LiquidCodec.

*This repo is generated by GPT-6.1-Sol.*