File size: 4,852 Bytes
bd2c39a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 | ---
library_name: transformers
pipeline_tag: image-text-to-text
base_model:
- LiquidAI/LFM3-VL-2B
---
This is a randomly initialized tiny model based on [LiquidAI/LFM3-VL-2B](https://huggingface.co/LiquidAI/LFM3-VL-2B) for testing and debugging.
| Feature | Source | This repo |
|---|---|---|
| Checkpoint tensor elements | 1.92B BF16 | 13.81M BF16 |
| Safetensors weight files | 3.84 GB (model.safetensors) | 27.64 MB (model.safetensors) |
| Text layers / layer-type cycle | 24 / two-conv prefix, then full/conv/sliding/conv | 10 / same prefix and two complete cycles |
| Text hidden width / MLP width | 2,048 / 7,680 | 64 / 128 |
| Text Q heads / KV heads / head dim | 16 / 4 / 128 | 16 / 4 / 32 |
| Vision layers / layer-type cycle | 14 / two-conv prefix, then six full/conv pairs | 6 / same prefix and two complete full/conv cycles |
| Vision hidden width / MLP width | 1,024 / 2,560 | 256 / 640 |
| Vision Q heads / KV heads / head dim | 16 / 8 / 64 | 8 / 4 / 32 |
| Patch / temporal patch / spatial merge | 16 / 2 / 2 | Same |
| Vision depthwise conv kernel | 3 x 3 | Same |
| RoPE | vision axial theta 1e6; text full/sliding dual theta | Same |
| Projector MLP hidden / downsample factor | 4,096 / 2 | 128 / 2 |
| Sandwich norms / QK RMSNorm / sigmoid head gate | present | Same |
| Vocabulary rows / embedding tying | 128,000 / tied | Same |
| Text maximum context length | 128,000 | Same |
| Stored precision | BF16 | Same |
Text-backbone widths follow [tiny-random/lfm3](https://huggingface.co/tiny-random/lfm3), which follows [tiny-random/lfm2](https://huggingface.co/tiny-random/lfm2). The pinned upstream implementation is included for Transformers loading. This repo provides a Transformers example with image and video inputs; no serving-backend validation is claimed.
| File path | Size |
|------|------|
| model.safetensors | 27.6MB |
### Example usage
#### Transformers (✅ verified)
Load the included pinned upstream implementation with `trust_remote_code=True`. The example sends one image and four predecoded video frames with complete `VideoMetadata`; the Triton short-conv path requires CUDA.
```python
def run_transformers(model_id: str):
import numpy as np
import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor
from transformers.video_utils import VideoMetadata
model_id = str(model_id)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = (
AutoModelForImageTextToText.from_pretrained(
model_id, trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa"
)
.eval()
.cuda()
)
image = Image.fromarray(np.random.randint(0, 255, (256, 256, 3), dtype=np.uint8), "RGB")
frames = np.stack(
[
np.asarray(Image.new("RGB", (256, 256), color))
for color in ((220, 30, 30), (30, 220, 30), (30, 30, 220), (220, 220, 30))
]
)
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": image},
{"type": "video", "video": frames},
{"type": "text", "text": "Describe the image and the video."},
],
}
]
inputs = processor.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
enable_thinking=False,
processor_kwargs={"video_metadata": [VideoMetadata(total_num_frames=4, fps=2, frames_indices=[0, 1, 2, 3])]},
).to(model.device)
assert inputs["pixel_values"].shape[0] > 0 and inputs["pixel_values_videos"].shape[0] > 0
with torch.no_grad():
logits = model(**inputs).logits
assert torch.isfinite(logits).all()
output = model.generate(**inputs, do_sample=False, max_new_tokens=32)
print(processor.decode(output[0, inputs["input_ids"].shape[1] :], skip_special_tokens=False))
if __name__ == "__main__":
run_transformers(model_id="tiny-random/lfm3-vl")
```
### Creating this repo
- Code: [.meta/create_model.py](.meta/create_model.py)
- Model structure: [.meta/model.log](.meta/model.log)
- Environment: [.meta/env.json](.meta/env.json)
### Change log
- 2026-11-01: Initial version. Text backbone matches tiny-random/lfm3's residual/MLP widths (64/128, ten layers with the two-conv prefix and two full/conv/sliding/conv cycles) but keeps the source 4:1 Q/KV ratio with 16 Q heads and at least four KV heads (4) at head dim 32 for TP=4. The vision tower keeps the source 2:1 Q/KV ratio with 8 Q heads / 4 KV heads, which requires width 256 for its derived head dim of 32; it retains the conv/conv prefix, two full/conv cycles, patch 16, temporal patch 2, merge size 2 and axial RoPE.
*This repo is generated by GPT-6.1-Sol.* |