File size: 2,963 Bytes
2afc1a1 92d119a 2afc1a1 92d119a 2afc1a1 92d119a 2afc1a1 92d119a 2afc1a1 92d119a 2afc1a1 92d119a 2afc1a1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 | ---
library_name: transformers
pipeline_tag: text-generation
base_model:
- LiquidAI/LFM3-1.7B
---
This is a randomly initialized tiny model based on [LiquidAI/LFM3-1.7B](https://huggingface.co/LiquidAI/LFM3-1.7B) for testing and debugging.
| Feature | Source | This repo |
|---|---|---|
| Checkpoint tensor elements | 1.73B BF16 | 8.62M BF16 |
| Safetensors weight files | 3.46 GB (model.safetensors) | 17.26 MB (model.safetensors) |
| Layers / layer-type cycle | 24 / two-conv prefix, then full/conv/sliding/conv | 10 / same prefix and two complete cycles |
| Hidden width / MLP width | 2,048 / 7,680 | 64 / 128 |
| Q heads / KV heads / head_dim | 16 / 4 / 128 | 4 / 1 / 32 |
| Short-conv kernel / cache | 7 / 7 | Same |
| RoPE | full 0.5 partial, theta 1e6; sliding theta 1e4 | Same |
| Sandwich norms / QK RMSNorm / sigmoid head gate | present | Same |
| Vocabulary rows / embedding tying | 128,000 / tied | Same |
| Maximum context length | 262,144 | 128,000 |
| Stored precision | BF16 | Same |
Residual width 64 and MLP width 128 follow [tiny-random/lfm2](https://huggingface.co/tiny-random/lfm2). The pinned upstream implementation is included for Transformers loading. This repo provides a Transformers example; no serving-backend validation is claimed.
| File path | Size |
|------|------|
| model.safetensors | 17.3MB |
### Example usage
#### Transformers (✅ verified)
Load the included pinned upstream implementation with `trust_remote_code=True`.
```python
def run_transformers(model_id: str):
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = str(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = (
AutoModelForCausalLM.from_pretrained(
model_id, trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa"
)
.eval()
.cuda()
)
inputs = tokenizer.apply_chat_template(
[{"role": "user", "content": "What is C. elegans?"}],
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
enable_thinking=False,
).to(model.device)
with torch.no_grad():
logits = model(**inputs).logits
assert torch.isfinite(logits).all()
output = model.generate(**inputs, do_sample=False, max_new_tokens=32)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1] :], skip_special_tokens=False))
if __name__ == "__main__":
run_transformers(model_id="tiny-random/lfm3")
```
### Creating this repo
- Code: [.meta/create_model.py](.meta/create_model.py)
- Model structure: [.meta/model.log](.meta/model.log)
- Environment: [.meta/env.json](.meta/env.json)
### Change log
- 2026-10-30: Initial version.
- 2026-10-31: Resize to hidden width 64 and MLP width 128, following tiny-random/lfm2; use four Q heads and one KV head while preserving the LFM3 layer cadence.
*This repo is generated by GPT-6.1-Sol.* |