Instructions to use tiny-random/lfm3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tiny-random/lfm3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tiny-random/lfm3", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tiny-random/lfm3", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("tiny-random/lfm3", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tiny-random/lfm3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tiny-random/lfm3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/lfm3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tiny-random/lfm3

SGLang

How to use tiny-random/lfm3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tiny-random/lfm3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/lfm3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tiny-random/lfm3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/lfm3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use tiny-random/lfm3 with Docker Model Runner:
```
docker model run hf.co/tiny-random/lfm3
```

lfm3 / .meta /model.log

yujiepan

Upload folder using huggingface_hub

f0e7fe0 verified 6 days ago

Raw History Blame Contribute Delete

8.36 kB

Lfm3ForCausalLM(
  (model): Lfm3Model(
    (embed_tokens): Embedding(128000, 64, padding_idx=124893)
    (layers): ModuleList(
      (0-1): 2 x Lfm3DecoderLayer(
        (conv): Lfm3ShortConv(
          (conv): Conv1d(64, 64, kernel_size=(7,), stride=(1,), padding=(6,), groups=64, bias=False)
          (in_proj): Linear(in_features=64, out_features=192, bias=False)
          (out_proj): Linear(in_features=64, out_features=64, bias=False)
        )
        (feed_forward): Lfm3MLP(
          (w1): Linear(in_features=64, out_features=128, bias=False)
          (w3): Linear(in_features=64, out_features=128, bias=False)
          (w2): Linear(in_features=128, out_features=64, bias=False)
        )
        (operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
      )
      (2): Lfm3DecoderLayer(
        (self_attn): Lfm3Attention(
          (q_proj): Linear(in_features=64, out_features=128, bias=False)
          (k_proj): Linear(in_features=64, out_features=32, bias=False)
          (v_proj): Linear(in_features=64, out_features=32, bias=False)
          (o_proj): Linear(in_features=128, out_features=64, bias=False)
          (q_norm): Lfm3RMSNorm((32,), eps=1e-05)
          (k_norm): Lfm3RMSNorm((32,), eps=1e-05)
          (gate_proj): Linear(in_features=64, out_features=4, bias=False)
        )
        (feed_forward): Lfm3MLP(
          (w1): Linear(in_features=64, out_features=128, bias=False)
          (w3): Linear(in_features=64, out_features=128, bias=False)
          (w2): Linear(in_features=128, out_features=64, bias=False)
        )
        (operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
      )
      (3): Lfm3DecoderLayer(
        (conv): Lfm3ShortConv(
          (conv): Conv1d(64, 64, kernel_size=(7,), stride=(1,), padding=(6,), groups=64, bias=False)
          (in_proj): Linear(in_features=64, out_features=192, bias=False)
          (out_proj): Linear(in_features=64, out_features=64, bias=False)
        )
        (feed_forward): Lfm3MLP(
          (w1): Linear(in_features=64, out_features=128, bias=False)
          (w3): Linear(in_features=64, out_features=128, bias=False)
          (w2): Linear(in_features=128, out_features=64, bias=False)
        )
        (operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
      )
      (4): Lfm3DecoderLayer(
        (self_attn): Lfm3Attention(
          (q_proj): Linear(in_features=64, out_features=128, bias=False)
          (k_proj): Linear(in_features=64, out_features=32, bias=False)
          (v_proj): Linear(in_features=64, out_features=32, bias=False)
          (o_proj): Linear(in_features=128, out_features=64, bias=False)
          (q_norm): Lfm3RMSNorm((32,), eps=1e-05)
          (k_norm): Lfm3RMSNorm((32,), eps=1e-05)
          (gate_proj): Linear(in_features=64, out_features=4, bias=False)
        )
        (feed_forward): Lfm3MLP(
          (w1): Linear(in_features=64, out_features=128, bias=False)
          (w3): Linear(in_features=64, out_features=128, bias=False)
          (w2): Linear(in_features=128, out_features=64, bias=False)
        )
        (operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
      )
      (5): Lfm3DecoderLayer(
        (conv): Lfm3ShortConv(
          (conv): Conv1d(64, 64, kernel_size=(7,), stride=(1,), padding=(6,), groups=64, bias=False)
          (in_proj): Linear(in_features=64, out_features=192, bias=False)
          (out_proj): Linear(in_features=64, out_features=64, bias=False)
        )
        (feed_forward): Lfm3MLP(
          (w1): Linear(in_features=64, out_features=128, bias=False)
          (w3): Linear(in_features=64, out_features=128, bias=False)
          (w2): Linear(in_features=128, out_features=64, bias=False)
        )
        (operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
      )
      (6): Lfm3DecoderLayer(
        (self_attn): Lfm3Attention(
          (q_proj): Linear(in_features=64, out_features=128, bias=False)
          (k_proj): Linear(in_features=64, out_features=32, bias=False)
          (v_proj): Linear(in_features=64, out_features=32, bias=False)
          (o_proj): Linear(in_features=128, out_features=64, bias=False)
          (q_norm): Lfm3RMSNorm((32,), eps=1e-05)
          (k_norm): Lfm3RMSNorm((32,), eps=1e-05)
          (gate_proj): Linear(in_features=64, out_features=4, bias=False)
        )
        (feed_forward): Lfm3MLP(
          (w1): Linear(in_features=64, out_features=128, bias=False)
          (w3): Linear(in_features=64, out_features=128, bias=False)
          (w2): Linear(in_features=128, out_features=64, bias=False)
        )
        (operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
      )
      (7): Lfm3DecoderLayer(
        (conv): Lfm3ShortConv(
          (conv): Conv1d(64, 64, kernel_size=(7,), stride=(1,), padding=(6,), groups=64, bias=False)
          (in_proj): Linear(in_features=64, out_features=192, bias=False)
          (out_proj): Linear(in_features=64, out_features=64, bias=False)
        )
        (feed_forward): Lfm3MLP(
          (w1): Linear(in_features=64, out_features=128, bias=False)
          (w3): Linear(in_features=64, out_features=128, bias=False)
          (w2): Linear(in_features=128, out_features=64, bias=False)
        )
        (operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
      )
      (8): Lfm3DecoderLayer(
        (self_attn): Lfm3Attention(
          (q_proj): Linear(in_features=64, out_features=128, bias=False)
          (k_proj): Linear(in_features=64, out_features=32, bias=False)
          (v_proj): Linear(in_features=64, out_features=32, bias=False)
          (o_proj): Linear(in_features=128, out_features=64, bias=False)
          (q_norm): Lfm3RMSNorm((32,), eps=1e-05)
          (k_norm): Lfm3RMSNorm((32,), eps=1e-05)
          (gate_proj): Linear(in_features=64, out_features=4, bias=False)
        )
        (feed_forward): Lfm3MLP(
          (w1): Linear(in_features=64, out_features=128, bias=False)
          (w3): Linear(in_features=64, out_features=128, bias=False)
          (w2): Linear(in_features=128, out_features=64, bias=False)
        )
        (operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
      )
      (9): Lfm3DecoderLayer(
        (conv): Lfm3ShortConv(
          (conv): Conv1d(64, 64, kernel_size=(7,), stride=(1,), padding=(6,), groups=64, bias=False)
          (in_proj): Linear(in_features=64, out_features=192, bias=False)
          (out_proj): Linear(in_features=64, out_features=64, bias=False)
        )
        (feed_forward): Lfm3MLP(
          (w1): Linear(in_features=64, out_features=128, bias=False)
          (w3): Linear(in_features=64, out_features=128, bias=False)
          (w2): Linear(in_features=128, out_features=64, bias=False)
        )
        (operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
        (post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
      )
    )
    (rotary_emb): Lfm3RotaryEmbedding()
    (embedding_norm): Lfm3RMSNorm((64,), eps=1e-05)
  )
  (lm_head): Linear(in_features=64, out_features=128000, bias=False)
)

	Lfm3ForCausalLM(
	(model): Lfm3Model(
	(embed_tokens): Embedding(128000, 64, padding_idx=124893)
	(layers): ModuleList(
	(0-1): 2 x Lfm3DecoderLayer(
	(conv): Lfm3ShortConv(
	(conv): Conv1d(64, 64, kernel_size=(7,), stride=(1,), padding=(6,), groups=64, bias=False)
	(in_proj): Linear(in_features=64, out_features=192, bias=False)
	(out_proj): Linear(in_features=64, out_features=64, bias=False)
	)
	(feed_forward): Lfm3MLP(
	(w1): Linear(in_features=64, out_features=128, bias=False)
	(w3): Linear(in_features=64, out_features=128, bias=False)
	(w2): Linear(in_features=128, out_features=64, bias=False)
	)
	(operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	)
	(2): Lfm3DecoderLayer(
	(self_attn): Lfm3Attention(
	(q_proj): Linear(in_features=64, out_features=128, bias=False)
	(k_proj): Linear(in_features=64, out_features=32, bias=False)
	(v_proj): Linear(in_features=64, out_features=32, bias=False)
	(o_proj): Linear(in_features=128, out_features=64, bias=False)
	(q_norm): Lfm3RMSNorm((32,), eps=1e-05)
	(k_norm): Lfm3RMSNorm((32,), eps=1e-05)
	(gate_proj): Linear(in_features=64, out_features=4, bias=False)
	)
	(feed_forward): Lfm3MLP(
	(w1): Linear(in_features=64, out_features=128, bias=False)
	(w3): Linear(in_features=64, out_features=128, bias=False)
	(w2): Linear(in_features=128, out_features=64, bias=False)
	)
	(operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	)
	(3): Lfm3DecoderLayer(
	(conv): Lfm3ShortConv(
	(conv): Conv1d(64, 64, kernel_size=(7,), stride=(1,), padding=(6,), groups=64, bias=False)
	(in_proj): Linear(in_features=64, out_features=192, bias=False)
	(out_proj): Linear(in_features=64, out_features=64, bias=False)
	)
	(feed_forward): Lfm3MLP(
	(w1): Linear(in_features=64, out_features=128, bias=False)
	(w3): Linear(in_features=64, out_features=128, bias=False)
	(w2): Linear(in_features=128, out_features=64, bias=False)
	)
	(operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	)
	(4): Lfm3DecoderLayer(
	(self_attn): Lfm3Attention(
	(q_proj): Linear(in_features=64, out_features=128, bias=False)
	(k_proj): Linear(in_features=64, out_features=32, bias=False)
	(v_proj): Linear(in_features=64, out_features=32, bias=False)
	(o_proj): Linear(in_features=128, out_features=64, bias=False)
	(q_norm): Lfm3RMSNorm((32,), eps=1e-05)
	(k_norm): Lfm3RMSNorm((32,), eps=1e-05)
	(gate_proj): Linear(in_features=64, out_features=4, bias=False)
	)
	(feed_forward): Lfm3MLP(
	(w1): Linear(in_features=64, out_features=128, bias=False)
	(w3): Linear(in_features=64, out_features=128, bias=False)
	(w2): Linear(in_features=128, out_features=64, bias=False)
	)
	(operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	)
	(5): Lfm3DecoderLayer(
	(conv): Lfm3ShortConv(
	(conv): Conv1d(64, 64, kernel_size=(7,), stride=(1,), padding=(6,), groups=64, bias=False)
	(in_proj): Linear(in_features=64, out_features=192, bias=False)
	(out_proj): Linear(in_features=64, out_features=64, bias=False)
	)
	(feed_forward): Lfm3MLP(
	(w1): Linear(in_features=64, out_features=128, bias=False)
	(w3): Linear(in_features=64, out_features=128, bias=False)
	(w2): Linear(in_features=128, out_features=64, bias=False)
	)
	(operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	)
	(6): Lfm3DecoderLayer(
	(self_attn): Lfm3Attention(
	(q_proj): Linear(in_features=64, out_features=128, bias=False)
	(k_proj): Linear(in_features=64, out_features=32, bias=False)
	(v_proj): Linear(in_features=64, out_features=32, bias=False)
	(o_proj): Linear(in_features=128, out_features=64, bias=False)
	(q_norm): Lfm3RMSNorm((32,), eps=1e-05)
	(k_norm): Lfm3RMSNorm((32,), eps=1e-05)
	(gate_proj): Linear(in_features=64, out_features=4, bias=False)
	)
	(feed_forward): Lfm3MLP(
	(w1): Linear(in_features=64, out_features=128, bias=False)
	(w3): Linear(in_features=64, out_features=128, bias=False)
	(w2): Linear(in_features=128, out_features=64, bias=False)
	)
	(operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	)
	(7): Lfm3DecoderLayer(
	(conv): Lfm3ShortConv(
	(conv): Conv1d(64, 64, kernel_size=(7,), stride=(1,), padding=(6,), groups=64, bias=False)
	(in_proj): Linear(in_features=64, out_features=192, bias=False)
	(out_proj): Linear(in_features=64, out_features=64, bias=False)
	)
	(feed_forward): Lfm3MLP(
	(w1): Linear(in_features=64, out_features=128, bias=False)
	(w3): Linear(in_features=64, out_features=128, bias=False)
	(w2): Linear(in_features=128, out_features=64, bias=False)
	)
	(operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	)
	(8): Lfm3DecoderLayer(
	(self_attn): Lfm3Attention(
	(q_proj): Linear(in_features=64, out_features=128, bias=False)
	(k_proj): Linear(in_features=64, out_features=32, bias=False)
	(v_proj): Linear(in_features=64, out_features=32, bias=False)
	(o_proj): Linear(in_features=128, out_features=64, bias=False)
	(q_norm): Lfm3RMSNorm((32,), eps=1e-05)
	(k_norm): Lfm3RMSNorm((32,), eps=1e-05)
	(gate_proj): Linear(in_features=64, out_features=4, bias=False)
	)
	(feed_forward): Lfm3MLP(
	(w1): Linear(in_features=64, out_features=128, bias=False)
	(w3): Linear(in_features=64, out_features=128, bias=False)
	(w2): Linear(in_features=128, out_features=64, bias=False)
	)
	(operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	)
	(9): Lfm3DecoderLayer(
	(conv): Lfm3ShortConv(
	(conv): Conv1d(64, 64, kernel_size=(7,), stride=(1,), padding=(6,), groups=64, bias=False)
	(in_proj): Linear(in_features=64, out_features=192, bias=False)
	(out_proj): Linear(in_features=64, out_features=64, bias=False)
	)
	(feed_forward): Lfm3MLP(
	(w1): Linear(in_features=64, out_features=128, bias=False)
	(w3): Linear(in_features=64, out_features=128, bias=False)
	(w2): Linear(in_features=128, out_features=64, bias=False)
	)
	(operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_operator_norm): Lfm3RMSNorm((64,), eps=1e-05)
	(post_ffn_norm): Lfm3RMSNorm((64,), eps=1e-05)
	)
	)
	(rotary_emb): Lfm3RotaryEmbedding()
	(embedding_norm): Lfm3RMSNorm((64,), eps=1e-05)
	)
	(lm_head): Linear(in_features=64, out_features=128000, bias=False)
	)