Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,88 @@
+---
+license: other
+license_name: falcon-llm-license
+license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
+base_model: tiiuae/Falcon3-3B-Instruct
+tags:
+  - litert
+  - litert-lm
+  - litertlm
+  - on-device
+  - edge
+  - falcon3
+pipeline_tag: text-generation
+library_name: litert-lm
+---
+
+# Falcon3-3B-Instruct — LiteRT-LM (mixed int4)
+
+[tiiuae/Falcon3-3B-Instruct](https://huggingface.co/tiiuae/Falcon3-3B-Instruct)
+converted to the **LiteRT-LM** (`.litertlm`) format for on-device inference with
+Google's [LiteRT-LM](https://github.com/google-ai-edge/litert-lm) runtime (the
+engine behind the official `litert-community/*` models).
+
+Text-only conversion (the Falcon3 decoder; no vision/audio towers).
+
+| | |
+|---|---|
+| **File** | `model.litertlm` (~1.7 GB) |
+| **Quantization** | mixed int4 — weights INT4 (channelwise, symmetric), embeddings INT8 |
+| **Compute** | integer |
+| **Base model** | tiiuae/Falcon3-3B-Instruct |
+| **Decode speed** | ~144 tok/s (Apple Silicon, LiteRT-LM) · ~33 tok/s cold on iPhone 17 Pro GPU |
+| **Context** | 32K |
+
+## Usage
+
+Run with the LiteRT-LM runtime:
+
+```bash
+# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
+litert_lm_main \
+  --model_path model.litertlm \
+  --backend gpu \
+  --input_prompt "Explain on-device AI in one sentence."
+```
+
+The `.litertlm` bundle carries the tokenizer and the prompt template (Falcon3's
+native `<|user|>` / `<|assistant|>` format, stop token `<|endoftext|>`), so no
+separate tokenizer files are needed.
+
+## Quality
+
+Scored with an 8-question local quality gate (correctness + degeneracy, the same
+checks as the iOS LLM benchmark), each asked separately on Apple Silicon:
+
+| Check | Result |
+|---|---|
+| 17 + 25 = 42 | ✅ |
+| capital of Japan = Tokyo | ✅ |
+| opposite of "hot" = cold | ✅ |
+| days in a week = 7 | ✅ |
+| "thank you" in French = merci | ✅ |
+| 8 × 7 = 56 | ✅ |
+| larger of 0.9 / 0.11 | ✅ |
+| "roses are red, violets are…" = blue | ✅ |
+
+**8 / 8 correct, non-degenerate**, and it terminates cleanly at `<|endoftext|>`.
+This is a direct-answering instruct model (no `<think>` block). As expected for a
+3B model at int4, long free-form explanations can show occasional minor artifacts
+(a stray word or brief language mix); short-form answers and reasoning are clean.
+
+## On-device
+
+Validated on **iPhone 17 Pro** via the LiteRT-LM Swift sample app: loads in ~5 s,
+generates coherent text, ~33 tok/s decode (cold) on the GPU backend.
+
+## Conversion
+
+Converted with [`litert-torch`](https://github.com/google-ai-edge/litert) using a
+mixed-int4 recipe (INT4 weights channelwise + INT8 embeddings) and Falcon3's
+native chat template. Falcon3-3B is a standard `LlamaForCausalLM` architecture, so
+it rides the existing converter and runtime with no custom code.
+
+## License
+
+Falcon LLM License (TII), inherited from the base model
+[tiiuae/Falcon3-3B-Instruct](https://huggingface.co/tiiuae/Falcon3-3B-Instruct).
+See https://falconllm.tii.ae/falcon-terms-and-conditions.html