README.md

metadata

library_name: transformers
pipeline_tag: any-to-any
base_model:
  - LiquidAI/LFM3-Omni-2B

This is a randomly initialized tiny model based on LiquidAI/LFM3-Omni-2B for testing and debugging.

Feature	Source	This repo
Checkpoint tensor elements	2.24B BF16, 1.28K F32	13.91M BF16, 1.28K F32
Safetensors weight files	4.48 GB (2 files)	27.88 MB (2 files)
Text layers / cycle	24 / two-conv prefix then full/conv/sliding/conv	10 / same prefix and two complete cycles
Text hidden / MLP width	2,048 / 7,680	64 / 128
Text Q / KV heads / head dim	16 / 4 / 128	16 / 4 / 32
Audio encoder layers / width	17 / 768	6 / 64 (two conv/conv/full cycles)
Audio mel bins / stride / window	128 / 8 / 32 frames	Same
Depth decoder layers / width	3 / 1,024	3 / 32
Codebooks / codebook size / maskgit	12 / 2,017 / [4, 4, 4]	Same
Vision layers / width / Q-KV heads	14 / 1,024 / 16-8	6 / 128 / 4-2
Patch / temporal patch / merge	16 / 2 / 2	Same
Projector MLP hidden / downsample	4,096 / 2	128 / 2
Codec sample rate / frame samples	24,000 / 1,920	Same
Codec STFT n_fft / hop / frame stack	1,280 / 320 / 6	Same
Codec FSQ levels / codebooks	[8, 7, 6, 6] / 12	Same
Codec LFM3 layers / width	8 + 8 / 512	4 + 4 / 64
Vocabulary rows / tying	128,000 / tied	Same
Stored precision	BF16; codec ISTFT window FP32	Same

Text/vision geometry follows tiny-random/lfm3-vl. The only runtime change lets the bundled codec LFM3 backbone read its dimensions from config.

File path	Size
audio_tokenizer/model.safetensors	1.0MB
model.safetensors	26.9MB

Example usage

Transformers (unverified)

Load the included pinned upstream implementation with trust_remote_code=True. The example sends a 16 kHz spoken question together with an image and four predecoded video frames, answers as text, then synthesizes 24 kHz speech through the bundled LiquidCodec. The Triton short-conv path requires CUDA.

def run_transformers(model_id: str):
    import numpy as np
    import torch
    from PIL import Image
    from transformers import AutoModelForImageTextToText, AutoProcessor

    model_id = str(model_id)
    processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
    model = (
        AutoModelForImageTextToText.from_pretrained(
            model_id, trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa"
        )
        .eval()
        .cuda()
    )
    codec = processor.audio_tokenizer.to(device=model.device).eval()

    image = Image.fromarray(np.random.randint(0, 255, (256, 256, 3), dtype=np.uint8), "RGB")
    frames = np.stack(
        [
            np.asarray(Image.new("RGB", (256, 256), color))
            for color in ((220, 30, 30), (30, 220, 30), (30, 30, 220), (220, 220, 30))
        ]
    )
    sample_rate = 16000
    speech = (np.sin(2 * np.pi * 440 * np.arange(sample_rate) / sample_rate) * 0.1).astype(np.float32)
    conversation = [
        {"role": "system", "content": "Respond in text."},
        {
            "role": "user",
            "content": [
                {"type": "image", "image": image},
                {"type": "audio", "audio": speech},
                {"type": "video", "video": frames},
                {"type": "text", "text": "Transcribe the question, describe the image and the video."},
            ],
        },
    ]
    inputs = processor.apply_chat_template(
        conversation,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
        sampling_rate=sample_rate,
        processor_kwargs={"video_metadata": [{"total_num_frames": 4, "fps": 2, "frames_indices": [0, 1, 2, 3]}]},
    ).to(model.device)
    assert (
        inputs["input_features"].shape[-1] > 0
        and inputs["pixel_values"].shape[0] > 0
        and inputs["pixel_values_videos"].shape[0] > 0
    )
    with torch.no_grad():
        answered = model.generate(**inputs, max_new_tokens=32, generation_mode="text")
    text = processor.decode(
        answered.sequences[0, inputs["input_ids"].shape[1] : answered.sequence_lengths[0]],
        skip_special_tokens=True,
    )
    print(text)

    tts_conversation = [
        {"role": "system", "content": "Perform TTS in American English using a female voice."},
        {"role": "user", "content": "The quick brown fox jumps over the lazy dog."},
    ]
    tts_inputs = processor.apply_chat_template(
        tts_conversation,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
    ).to(model.device)
    with torch.no_grad():
        spoken = model.generate(
            **tts_inputs,
            max_new_tokens=64,
            generation_mode="audio",
            audio_temperature=0.4,
            audio_repetition_penalty=1.2,
        )
        assert spoken.audio_codes.shape[-1] > 0
        waveform = codec.decode(spoken.audio_codes[:1, :, : spoken.audio_lengths[0]]).audio_values[0]
    assert waveform.numel() > 0 and torch.isfinite(waveform).all()
    print(waveform.shape)


if __name__ == "__main__":
    run_transformers(model_id="scratch/lfm3-omni")

Creating this repo

• Code: .meta/create_model.py
• Model structure: .meta/model.log
• Environment: .meta/env.json

Change log

• 2026-11-03: Initial version: omni text/vision/16kHz-audio encoder, maskgit depth decoder and a config-driven tiny bundled 24 kHz LiquidCodec.

This repo is generated by GPT-6.1-Sol.