| --- |
| license: apache-2.0 |
| --- |
| |
| # Introduction |
|
|
| This repository hosts the [distiluse-base-multilingual-cased-v2](https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased-v2/tree/main) model for the [React Native ExecuTorch](https://www.npmjs.com/package/react-native-executorch) library. It includes the model exported for both the **XNNPACK** (Android / generic CPU) and **CoreML** (Apple) delegates, in multiple precisions, ready for use in the **ExecuTorch** runtime. |
|
|
| If you'd like to run these models in your own ExecuTorch runtime, refer to the [official documentation](https://pytorch.org/executorch/stable/index.html) for setup instructions. |
|
|
| ## Compatibility |
|
|
| If you intend to use this model outside of React Native ExecuTorch, make sure your runtime is compatible with the **ExecuTorch** version used to export the `.pte` files. For more details, see the compatibility note in the [ExecuTorch GitHub repository](https://github.com/pytorch/executorch/blob/11d1742fdeddcf05bc30a6cfac321d2a2e3b6768/runtime/COMPATIBILITY.md?plain=1#L4). If you work with React Native ExecuTorch, the constants from the library will guarantee compatibility with the runtime used behind the scenes. |
|
|
| These models were exported using React Native ExecuTorch `v0.9.0`, which ships an ExecuTorch runtime derived from the `v1.2.0` release branch. **No forward compatibility** is guaranteed — older versions of the runtime may not work with these files. |
|
|
| ## Variant Matrix |
|
|
| | Delegate | Precision | File | Size | RMSE vs eager | Notes | |
| |----------|-----------|------------------------------------------------------------------------------|--------|---------------|--------------------------------------------------------------------| |
| | XNNPACK | fp32 | `xnnpack/distiluse-base-multilingual-cased-v2_xnnpack_fp32.pte` | 516 MB | 0.0 | Baseline. Works on Android / iOS / generic CPU. | |
| | XNNPACK | 8da4w | `xnnpack/distiluse-base-multilingual-cased-v2_xnnpack_8da4w.pte` | 375 MB | 5.4e-4 | Int8 dynamic activation + Int4 weight (torchao), group_size=32. Embeddings stay fp32 — the bulk of the size reduction comes from linear layers. | |
| | CoreML | fp32 | `coreml/distiluse-base-multilingual-cased-v2_coreml_fp32.pte` | 516 MB | 0.0 | Apple Neural Engine / GPU / CPU, float32 compute. | |
| | CoreML | fp16 | `coreml/distiluse-base-multilingual-cased-v2_coreml_fp16.pte` | 258 MB | 1.9e-4 | Half-sized via `compute_precision=FLOAT16` at CoreML compile. Cleanest size win on iOS. | |
|
|
| Pick the variant that matches your platform + size/quality trade-off. The CoreML variants only load on Apple platforms; the XNNPACK variants load everywhere. |
|
|
| ## Repository Structure |
|
|
| - `xnnpack/` — `.pte` files partitioned for the XNNPACK delegate. |
| - `coreml/` — `.pte` files partitioned for the CoreML delegate (iOS / macOS only). |
| - `tokenizer.json` — HuggingFace fast-tokenizer dump (WordPiece + BertNormalizer). Wire this to `tokenizerSource`. |
| - `config.json`, `tokenizer_config.json` — upstream model/tokenizer configs, kept for reference and for non-RNE consumers. |
|
|
| The `.pte` path goes to `modelSource`; `tokenizer.json` is shared across all variants. |
|
|
| ## Model details |
|
|
| - Architecture: DistilBERT multilingual cased + mean pooling + Dense (768→512, Tanh) + L2 norm. |
| - Output dimension: **512**. |
| - Max sequence length: **126** tokens (128 − 2 for `[CLS]` / `[SEP]`). |
| - Languages: 50+ (multilingual). |
| - Typical strength: cross-lingual sentence similarity and medium-length sentence retrieval. Short single-word queries in non-English languages are this model's weakest case — for those, longer sentences and/or English inputs give markedly better ranking. |
|
|
| ## Export notes |
|
|
| The exported program skips HuggingFace's internal attention-mask-to-4D conversion because the RNE runtime never pads at inference (single sentence, no batching). This preserves bit-exactness with the PyTorch reference (RMSE 0 on fp32 random input) while trimming ~27% off the XNNPACK forward wall-time and keeping XNNPACK delegation around 89–91% of graph runtime. |
|
|
| Unsupported combinations (rejected by the exporter, documented for reference): |
|
|
| - **XNNPACK + fp16** — `model.to(torch.float16)` causes softmax / LayerNorm overflow and the runtime output is NaN. XNNPACK's size wins come from quantization, not fp16. |
| - **CoreML + 8da4w** — `coremltools` has no MIL mapping for the `torch.int8` tensors torchao emits (`KeyError: torch.int8`). The CoreML-native way to shrink further is `ct.optimize.coreml` palette/linear quantization, not torchao source transforms. |
|
|
