| --- |
| library_name: transformers |
| pipeline_tag: image-text-to-text |
| base_model: |
| - LiquidAI/LFM3-VL-2B |
| --- |
| |
| This is a randomly initialized tiny model based on [LiquidAI/LFM3-VL-2B](https://huggingface.co/LiquidAI/LFM3-VL-2B) for testing and debugging. |
|
|
| | Feature | Source | This repo | |
| |---|---|---| |
| | Checkpoint tensor elements | 1.92B BF16 | 13.81M BF16 | |
| | Safetensors weight files | 3.84 GB (model.safetensors) | 27.64 MB (model.safetensors) | |
| | Text layers / layer-type cycle | 24 / two-conv prefix, then full/conv/sliding/conv | 10 / same prefix and two complete cycles | |
| | Text hidden width / MLP width | 2,048 / 7,680 | 64 / 128 | |
| | Text Q heads / KV heads / head dim | 16 / 4 / 128 | 16 / 4 / 32 | |
| | Vision layers / layer-type cycle | 14 / two-conv prefix, then six full/conv pairs | 6 / same prefix and two complete full/conv cycles | |
| | Vision hidden width / MLP width | 1,024 / 2,560 | 256 / 640 | |
| | Vision Q heads / KV heads / head dim | 16 / 8 / 64 | 8 / 4 / 32 | |
| | Patch / temporal patch / spatial merge | 16 / 2 / 2 | Same | |
| | Vision depthwise conv kernel | 3 x 3 | Same | |
| | RoPE | vision axial theta 1e6; text full/sliding dual theta | Same | |
| | Projector MLP hidden / downsample factor | 4,096 / 2 | 128 / 2 | |
| | Sandwich norms / QK RMSNorm / sigmoid head gate | present | Same | |
| | Vocabulary rows / embedding tying | 128,000 / tied | Same | |
| | Text maximum context length | 128,000 | Same | |
| | Stored precision | BF16 | Same | |
|
|
| Text-backbone widths follow [tiny-random/lfm3](https://huggingface.co/tiny-random/lfm3), which follows [tiny-random/lfm2](https://huggingface.co/tiny-random/lfm2). The pinned upstream implementation is included for Transformers loading. This repo provides a Transformers example with image and video inputs; no serving-backend validation is claimed. |
|
|
| | File path | Size | |
| |------|------| |
| | model.safetensors | 27.6MB | |
|
|
| ### Example usage |
|
|
| #### Transformers (✅ verified) |
|
|
| Load the included pinned upstream implementation with `trust_remote_code=True`. The example sends one image and four predecoded video frames with complete `VideoMetadata`; the Triton short-conv path requires CUDA. |
|
|
| ```python |
| def run_transformers(model_id: str): |
| import numpy as np |
| import torch |
| from PIL import Image |
| from transformers import AutoModelForImageTextToText, AutoProcessor |
| from transformers.video_utils import VideoMetadata |
| |
| model_id = str(model_id) |
| processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True) |
| model = ( |
| AutoModelForImageTextToText.from_pretrained( |
| model_id, trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa" |
| ) |
| .eval() |
| .cuda() |
| ) |
| image = Image.fromarray(np.random.randint(0, 255, (256, 256, 3), dtype=np.uint8), "RGB") |
| frames = np.stack( |
| [ |
| np.asarray(Image.new("RGB", (256, 256), color)) |
| for color in ((220, 30, 30), (30, 220, 30), (30, 30, 220), (220, 220, 30)) |
| ] |
| ) |
| messages = [ |
| { |
| "role": "user", |
| "content": [ |
| {"type": "image", "image": image}, |
| {"type": "video", "video": frames}, |
| {"type": "text", "text": "Describe the image and the video."}, |
| ], |
| } |
| ] |
| inputs = processor.apply_chat_template( |
| messages, |
| add_generation_prompt=True, |
| tokenize=True, |
| return_dict=True, |
| return_tensors="pt", |
| enable_thinking=False, |
| processor_kwargs={"video_metadata": [VideoMetadata(total_num_frames=4, fps=2, frames_indices=[0, 1, 2, 3])]}, |
| ).to(model.device) |
| assert inputs["pixel_values"].shape[0] > 0 and inputs["pixel_values_videos"].shape[0] > 0 |
| with torch.no_grad(): |
| logits = model(**inputs).logits |
| assert torch.isfinite(logits).all() |
| output = model.generate(**inputs, do_sample=False, max_new_tokens=32) |
| print(processor.decode(output[0, inputs["input_ids"].shape[1] :], skip_special_tokens=False)) |
| |
| |
| if __name__ == "__main__": |
| run_transformers(model_id="tiny-random/lfm3-vl") |
| ``` |
|
|
| ### Creating this repo |
|
|
| - Code: [.meta/create_model.py](.meta/create_model.py) |
| - Model structure: [.meta/model.log](.meta/model.log) |
| - Environment: [.meta/env.json](.meta/env.json) |
|
|
| ### Change log |
|
|
| - 2026-11-01: Initial version. Text backbone matches tiny-random/lfm3's residual/MLP widths (64/128, ten layers with the two-conv prefix and two full/conv/sliding/conv cycles) but keeps the source 4:1 Q/KV ratio with 16 Q heads and at least four KV heads (4) at head dim 32 for TP=4. The vision tower keeps the source 2:1 Q/KV ratio with 8 Q heads / 4 KV heads, which requires width 256 for its derived head dim of 32; it retains the conv/conv prefix, two full/conv cycles, patch 16, temporal patch 2, merge size 2 and axial RoPE. |
|
|
| *This repo is generated by GPT-6.1-Sol.* |
