Instructions to use mlboydaisuke/Falcon3-3B-Instruct-LiteRT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT-LM
How to use mlboydaisuke/Falcon3-3B-Instruct-LiteRT with LiteRT-LM:
# LiteRT-LM runs on various platforms (Android, iOS, Windows, Linux, macOS, IoT, Web/WASM) # and supports many APIs (C++, Python, Kotlin, Swift, JavaScript, Flutter). # For platform-specific integration guides, please refer to the official developer website: # https://ai.google.dev/edge/litert-lm # To try LiteRT-LM, the easiest way is to use our CLI tool. # 1. Install the LiteRT-LM CLI tool: pip install -U litert-lm # 2. Download and run this model locally: # See: https://ai.google.dev/edge/litert-lm/cli litert-lm run \ --from-huggingface-repo=mlboydaisuke/Falcon3-3B-Instruct-LiteRT \ --prompt="Write me a poem"
- LiteRT
How to use mlboydaisuke/Falcon3-3B-Instruct-LiteRT with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Card: add 'Reproduce (official tools only)' section
Browse files
README.md
CHANGED
|
@@ -74,6 +74,28 @@ standard `LlamaForCausalLM` architecture, so it rides the existing converter and
|
|
| 74 |
runtime with no custom code. Blockwise (not channelwise) int4 is what preserves
|
| 75 |
reasoning accuracy.
|
| 76 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 77 |
## License
|
| 78 |
|
| 79 |
Falcon LLM License (TII), inherited from the base model
|
|
|
|
| 74 |
runtime with no custom code. Blockwise (not channelwise) int4 is what preserves
|
| 75 |
reasoning accuracy.
|
| 76 |
|
| 77 |
+
## Reproduce (official tools only)
|
| 78 |
+
|
| 79 |
+
Built with **stock `litert-torch`** — no custom code, no graph patches. The only
|
| 80 |
+
non-default choice is the int4 recipe: the tool's default named int4 is
|
| 81 |
+
*channelwise* (which degrades small models), so this uses **blockwise-128** (the
|
| 82 |
+
scheme the official models ship), passed as a recipe file to the standard export:
|
| 83 |
+
|
| 84 |
+
```python
|
| 85 |
+
from litert_torch.generative.export_hf.export import export
|
| 86 |
+
export(
|
| 87 |
+
model="tiiuae/Falcon3-3B-Instruct",
|
| 88 |
+
output_dir="out",
|
| 89 |
+
quantization_recipe="falcon_int4_block128.json", # included in this repo
|
| 90 |
+
cache_length=2048,
|
| 91 |
+
trust_remote_code=True,
|
| 92 |
+
)
|
| 93 |
+
```
|
| 94 |
+
|
| 95 |
+
`falcon_int4_block128.json` is included in this repo. (If the export errors with a
|
| 96 |
+
missing `ai_edge_quantizer/recipes/` directory, create it empty — a packaging gap
|
| 97 |
+
in some releases that trips the `.json`-recipe path.)
|
| 98 |
+
|
| 99 |
## License
|
| 100 |
|
| 101 |
Falcon LLM License (TII), inherited from the base model
|