Add desktop LiteRT-LM CLI section (import/run/serve)
Browse files
README.md
CHANGED
|
@@ -48,6 +48,19 @@ The `.litertlm` bundle carries the tokenizer and the prompt template (Falcon3's
|
|
| 48 |
native `<|user|>` / `<|assistant|>` format, stop token `<|endoftext|>`), so no
|
| 49 |
separate tokenizer files are needed.
|
| 50 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 51 |
## Quality — GSM8K parity
|
| 52 |
|
| 53 |
Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought asking for `#### <n>`,
|
|
|
|
| 48 |
native `<|user|>` / `<|assistant|>` format, stop token `<|endoftext|>`), so no
|
| 49 |
separate tokenizer files are needed.
|
| 50 |
|
| 51 |
+
## Run on desktop (LiteRT-LM CLI)
|
| 52 |
+
|
| 53 |
+
The same `.litertlm` bundle runs on macOS / Linux / Windows with the official
|
| 54 |
+
[LiteRT-LM CLI](https://github.com/google-ai-edge/LiteRT-LM) — including as a
|
| 55 |
+
local **OpenAI-compatible API server**:
|
| 56 |
+
|
| 57 |
+
```bash
|
| 58 |
+
pip install litert-lm
|
| 59 |
+
litert-lm import --from-huggingface-repo mlboydaisuke/Falcon3-3B-Instruct-LiteRT model.litertlm falcon3-3b-instruct-litert
|
| 60 |
+
litert-lm run falcon3-3b-instruct-litert # interactive chat in the terminal
|
| 61 |
+
litert-lm serve # local OpenAI-compatible API server
|
| 62 |
+
```
|
| 63 |
+
|
| 64 |
## Quality — GSM8K parity
|
| 65 |
|
| 66 |
Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought asking for `#### <n>`,
|