| --- |
| license: other |
| license_name: lfm1.0 |
| license_link: LICENSE |
| base_model: LiquidAI/LFM2.5-Encoder-350M-Prompt-Router |
| pipeline_tag: zero-shot-classification |
| library_name: onnx |
| tags: |
| - onnx |
| - lfm2 |
| - zero-shot-classification |
| - lemonade |
| language: |
| - en |
| - de |
| - es |
| - fr |
| - it |
| - nl |
| - pl |
| - pt |
| - ar |
| - hi |
| - ja |
| - ru |
| - tr |
| - vi |
| - zh |
| --- |
| |
| # LFM2.5-Encoder-350M-Prompt-Router-ONNX |
|
|
| An ONNX export of |
| [LiquidAI/LFM2.5-Encoder-350M-Prompt-Router](https://huggingface.co/LiquidAI/LFM2.5-Encoder-350M-Prompt-Router), |
| for [Lemonade](https://github.com/lemonade-sdk/lemonade) and its |
| [ort-server](https://github.com/lemonade-sdk/ort-server) backend. |
|
|
| **All credit for the model goes to [Liquid AI](https://huggingface.co/LiquidAI).** |
| This repository only changes the file format. The weights are the original |
| float32 weights, and the LFM Open License v1.0 applies (see `LICENSE`). |
|
|
| The model is a zero-shot prompt router. You give it a text and a list of labels |
| at request time. It scores the text against every label in one encoder pass, and |
| the scores sum to 1. |
|
|
| ## Changes from the original |
|
|
| | File | Origin | |
| |---|---| |
| | `model.onnx` | Converted from `model.safetensors` at revision `35ca4a0469f180f1cf05a630df8842fa17ac18e3`. | |
| | `config.json`, `tokenizer.json`, `LICENSE` | Unchanged copies. | |
| | `manifest.json` | New. ort-server settings: `task: zero-shot-classification`, `max_length: 2048`. | |
|
|
| The window is 2,048 tokens. The position table holds 128,000, but this export |
| uses eager attention, so memory grows with the square of the window: about |
| 1.4 GB extra at 2,048 tokens and 5.3 GB at 4,096. A longer text is cut, and the |
| response then has `"truncated": true`. To raise the window, pass ort-server's |
| `--max-length` (in Lemonade: `--onnxruntime-args "--max-length 4096"`). |
|
|
| The graph takes `input_ids`, `attention_mask`, `text_pool` `[b, 1, s]` and |
| `category_pool` `[b, r, s]`, and returns `logits` `[b, r]`. The label count `r` |
| and the sequence length `s` are dynamic. The two pooling inputs mark which tokens |
| belong to the text and to each label. ort-server builds them from the |
| tokenizer's character offsets. |
|
|
| ## Use with Lemonade |
|
|
| Lemonade needs ort-server 0.4.0 or later. The model is built in: |
|
|
| ```bash |
| lemonade pull LFM2.5-Encoder-350M-Prompt-Router-ONNX |
| ``` |
|
|
| Classify a text: |
|
|
| ```bash |
| curl -X POST http://localhost:13305/api/v1/classify \ |
| -H "Content-Type: application/json" \ |
| -d '{"model": "LFM2.5-Encoder-350M-Prompt-Router-ONNX", |
| "input": "Can you help me debug a failing Python unit test?", |
| "labels": ["Coding", "Sales", "Creative writing", "General knowledge"]}' |
| ``` |
|
|
| ```json |
| {"labels": {"Coding": 0.838, "Creative writing": 0.054, "General knowledge": 0.054, "Sales": 0.054}, "truncated": false} |
| ``` |
|
|
| In a router policy, use the model in a `zero_shot` classifier. See Lemonade's |
| `docs/dev/router-policy.md`. |
|
|
| ## How this export was made |
|
|
| The scripts are in ort-server's |
| [`tools/zero_shot_router/`](https://github.com/lemonade-sdk/ort-server/tree/main/tools/zero_shot_router). |
| From an ort-server checkout: |
|
|
| ```bash |
| pip install torch transformers onnx onnxruntime huggingface_hub |
| python tools/zero_shot_router/export.py <out_dir> --revision 35ca4a0469f180f1cf05a630df8842fa17ac18e3 |
| python tools/zero_shot_router/parity.py <out_dir> --revision 35ca4a0469f180f1cf05a630df8842fa17ac18e3 --binary <ort-server> |
| ``` |
|
|
| `export.py` loads the checkpoint with `trust_remote_code=True`. This is required, |
| because `Lfm2BidirForSequenceRouting` and `route()` are in the checkpoint's |
| `modeling_lfm2_bidirectional.py`, not in `transformers`. The script then: |
|
|
| 1. Replaces the short-conv forward, whose padding and groups come from tensor |
| shapes that tracing cannot export. |
| 2. Exports `forward()` at opset 17, not `route()`, so that the label count stays dynamic. |
| 3. Compares the graph with `route()` on 16 cases with 2 to 8 labels, and stops |
| on any score difference above `1e-4` or any changed top label. |
|
|
| Tested with transformers 4.57, torch 2.8 and onnxruntime 1.24. |
|
|
| ## Parity with the original model |
|
|
| `parity.py` runs 95 labelled dummy cases through three paths: |
|
|
| - the original safetensors model, with its own `route()` in PyTorch |
| - this ONNX graph in ONNX Runtime |
| - ort-server over HTTP, which builds the pooling inputs in C++ |
|
|
| The cases cover routing, PII, prompt injection, support topics, French, |
| Spanish, German and Japanese, and three long texts of 900 to 1,900 tokens. |
|
|
| | Path | Accuracy on the dummy labels | Top-1 agreement with PyTorch | Max score difference | |
| |---|---|---|---| |
| | PyTorch (safetensors) | 76.84% | - | - | |
| | ONNX Runtime | 76.84% | 100% | 2.3e-6 | |
| | ort-server | 76.84% | 100% | 2.3e-6 | |
|
|
| The accuracy column describes the model on hand-written labels, not the |
| conversion: all three paths get the same cases right and wrong. |
|
|
| ## License |
|
|
| LFM Open License v1.0. See `LICENSE` and the |
| [original model](https://huggingface.co/LiquidAI/LFM2.5-Encoder-350M-Prompt-Router). |
| Commercial use above the license's threshold is not licensed. |
|
|
