Set the serving window to 2048 tokens

README.md

@@ -49,7 +49,13 @@ the scores sum to 1.
 |---|---|
 | `model.onnx` | Converted from `model.safetensors` at revision `35ca4a0469f180f1cf05a630df8842fa17ac18e3`. |
 | `config.json`, `tokenizer.json`, `LICENSE` | Unchanged copies. |
-| `manifest.json` | New. ort-server settings: `task: zero-shot-classification`, `max_length: 128000`. |
+| `manifest.json` | New. ort-server settings: `task: zero-shot-classification`, `max_length: 2048`. |
+
+The window is 2,048 tokens. The position table holds 128,000, but this export
+uses eager attention, so memory grows with the square of the window: about
+1.4 GB extra at 2,048 tokens and 5.3 GB at 4,096. A longer text is cut, and the
+response then has `"truncated": true`. To raise the window, pass ort-server's
+`--max-length` (in Lemonade: `--onnxruntime-args "--max-length 4096"`).
 
 The graph takes `input_ids`, `attention_mask`, `text_pool` `[b, 1, s]` and
 `category_pool` `[b, r, s]`, and returns `logits` `[b, r]`. The label count `r`
@@ -76,7 +82,7 @@ curl -X POST http://localhost:13305/api/v1/classify \
 ```
 
 ```json
-{"labels": {"Coding": 0.838, "Creative writing": 0.054, "General knowledge": 0.054, "Sales": 0.054}}
+{"labels": {"Coding": 0.838, "Creative writing": 0.054, "General knowledge": 0.054, "Sales": 0.054}, "truncated": false}
 ```
 
 In a router policy, use the model in a `zero_shot` classifier. See Lemonade's

manifest.json

@@ -1,4 +1,4 @@
-{
-  "task": "zero-shot-classification",
-  "max_length": 128000
-}
+{
+  "task": "zero-shot-classification",
+  "max_length": 2048
+}