File size: 4,971 Bytes
56fce82 7d7035b 56fce82 7d7035b 56fce82 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 | ---
license: other
license_name: lfm1.0
license_link: LICENSE
base_model: LiquidAI/LFM2.5-Encoder-350M-Prompt-Router
pipeline_tag: zero-shot-classification
library_name: onnx
tags:
- onnx
- lfm2
- zero-shot-classification
- lemonade
language:
- en
- de
- es
- fr
- it
- nl
- pl
- pt
- ar
- hi
- ja
- ru
- tr
- vi
- zh
---
# LFM2.5-Encoder-350M-Prompt-Router-ONNX
An ONNX export of
[LiquidAI/LFM2.5-Encoder-350M-Prompt-Router](https://huggingface.co/LiquidAI/LFM2.5-Encoder-350M-Prompt-Router),
for [Lemonade](https://github.com/lemonade-sdk/lemonade) and its
[ort-server](https://github.com/lemonade-sdk/ort-server) backend.
**All credit for the model goes to [Liquid AI](https://huggingface.co/LiquidAI).**
This repository only changes the file format. The weights are the original
float32 weights, and the LFM Open License v1.0 applies (see `LICENSE`).
The model is a zero-shot prompt router. You give it a text and a list of labels
at request time. It scores the text against every label in one encoder pass, and
the scores sum to 1.
## Changes from the original
| File | Origin |
|---|---|
| `model.onnx` | Converted from `model.safetensors` at revision `35ca4a0469f180f1cf05a630df8842fa17ac18e3`. |
| `config.json`, `tokenizer.json`, `LICENSE` | Unchanged copies. |
| `manifest.json` | New. ort-server settings: `task: zero-shot-classification`, `max_length: 2048`. |
The window is 2,048 tokens. The position table holds 128,000, but this export
uses eager attention, so memory grows with the square of the window: about
1.4 GB extra at 2,048 tokens and 5.3 GB at 4,096. A longer text is cut, and the
response then has `"truncated": true`. To raise the window, pass ort-server's
`--max-length` (in Lemonade: `--onnxruntime-args "--max-length 4096"`).
The graph takes `input_ids`, `attention_mask`, `text_pool` `[b, 1, s]` and
`category_pool` `[b, r, s]`, and returns `logits` `[b, r]`. The label count `r`
and the sequence length `s` are dynamic. The two pooling inputs mark which tokens
belong to the text and to each label. ort-server builds them from the
tokenizer's character offsets.
## Use with Lemonade
Lemonade needs ort-server 0.4.0 or later. The model is built in:
```bash
lemonade pull LFM2.5-Encoder-350M-Prompt-Router-ONNX
```
Classify a text:
```bash
curl -X POST http://localhost:13305/api/v1/classify \
-H "Content-Type: application/json" \
-d '{"model": "LFM2.5-Encoder-350M-Prompt-Router-ONNX",
"input": "Can you help me debug a failing Python unit test?",
"labels": ["Coding", "Sales", "Creative writing", "General knowledge"]}'
```
```json
{"labels": {"Coding": 0.838, "Creative writing": 0.054, "General knowledge": 0.054, "Sales": 0.054}, "truncated": false}
```
In a router policy, use the model in a `zero_shot` classifier. See Lemonade's
`docs/dev/router-policy.md`.
## How this export was made
The scripts are in ort-server's
[`tools/zero_shot_router/`](https://github.com/lemonade-sdk/ort-server/tree/main/tools/zero_shot_router).
From an ort-server checkout:
```bash
pip install torch transformers onnx onnxruntime huggingface_hub
python tools/zero_shot_router/export.py <out_dir> --revision 35ca4a0469f180f1cf05a630df8842fa17ac18e3
python tools/zero_shot_router/parity.py <out_dir> --revision 35ca4a0469f180f1cf05a630df8842fa17ac18e3 --binary <ort-server>
```
`export.py` loads the checkpoint with `trust_remote_code=True`. This is required,
because `Lfm2BidirForSequenceRouting` and `route()` are in the checkpoint's
`modeling_lfm2_bidirectional.py`, not in `transformers`. The script then:
1. Replaces the short-conv forward, whose padding and groups come from tensor
shapes that tracing cannot export.
2. Exports `forward()` at opset 17, not `route()`, so that the label count stays dynamic.
3. Compares the graph with `route()` on 16 cases with 2 to 8 labels, and stops
on any score difference above `1e-4` or any changed top label.
Tested with transformers 4.57, torch 2.8 and onnxruntime 1.24.
## Parity with the original model
`parity.py` runs 95 labelled dummy cases through three paths:
- the original safetensors model, with its own `route()` in PyTorch
- this ONNX graph in ONNX Runtime
- ort-server over HTTP, which builds the pooling inputs in C++
The cases cover routing, PII, prompt injection, support topics, French,
Spanish, German and Japanese, and three long texts of 900 to 1,900 tokens.
| Path | Accuracy on the dummy labels | Top-1 agreement with PyTorch | Max score difference |
|---|---|---|---|
| PyTorch (safetensors) | 76.84% | - | - |
| ONNX Runtime | 76.84% | 100% | 2.3e-6 |
| ort-server | 76.84% | 100% | 2.3e-6 |
The accuracy column describes the model on hand-written labels, not the
conversion: all three paths get the same cases right and wrong.
## License
LFM Open License v1.0. See `LICENSE` and the
[original model](https://huggingface.co/LiquidAI/LFM2.5-Encoder-350M-Prompt-Router).
Commercial use above the license's threshold is not licensed.
|