API Try Docs Blog

LFM3-1.7B-ONNX

ONNX version of LFM3-1.7B for cross-platform inference with ONNX Runtime.

Find architecture and usage details in the original model card.

Files

---

Precision	File	Size	Device
q4	model_q4	1.63 GB	CPU
q4f32	model_q4f32	2.24 GB	CPU
q8	model_q8	2.27 GB	CPU
q4f16	model_q4f16	1.38 GB	GPU
q8f16	model_q8f16	2.03 GB	GPU
fp16	model_fp16	3.59 GB	GPU
fp32	model	7.18 GB	Reference

fp16, q4f16 and q8f16 run with fp16 activations, the others with fp32. q4 and q4f16 keep the embedding, the LM head and 27 of the 153 layer projections in int8 and the rest in int4, while q4f32 quantizes all 153 projections to int4 and keeps the embedding and LM head in fp32.

Use with ONNX Runtime

---

The graphs run on ONNX Runtime. Each step takes input_ids, attention_mask, position_ids and logits_to_keep, plus the convolution and attention caches (past_conv.N, past_key_values.N.key and .value). It returns logits and the updated caches (present_conv.N, present.N.key and .value). The fp16, q4f16 and q8f16 graphs keep their caches in fp16.

On GPU, use fp16, q4f16 or q8f16. The CUDA execution provider has no fp32 GroupQueryAttention kernel, so the other graphs run attention on the CPU.

Recommended sampling: temperature=0.1, top_k=50, repetition_penalty=1.1. For math and science, use temperature=1.0, top_p=0.95, top_k=20 instead. More options in the ONNX guide.