File size: 4,835 Bytes
225456f 11fe867 225456f 11fe867 225456f 11fe867 225456f 11fe867 225456f 11fe867 225456f 11fe867 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | ---
license: apache-2.0
---
# Introduction
This repository hosts the [distiluse-base-multilingual-cased-v2](https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased-v2/tree/main) model for the [React Native ExecuTorch](https://www.npmjs.com/package/react-native-executorch) library. It includes the model exported for both the **XNNPACK** (Android / generic CPU) and **CoreML** (Apple) delegates, in multiple precisions, ready for use in the **ExecuTorch** runtime.
If you'd like to run these models in your own ExecuTorch runtime, refer to the [official documentation](https://pytorch.org/executorch/stable/index.html) for setup instructions.
## Compatibility
If you intend to use this model outside of React Native ExecuTorch, make sure your runtime is compatible with the **ExecuTorch** version used to export the `.pte` files. For more details, see the compatibility note in the [ExecuTorch GitHub repository](https://github.com/pytorch/executorch/blob/11d1742fdeddcf05bc30a6cfac321d2a2e3b6768/runtime/COMPATIBILITY.md?plain=1#L4). If you work with React Native ExecuTorch, the constants from the library will guarantee compatibility with the runtime used behind the scenes.
These models were exported using React Native ExecuTorch `v0.9.0`, which ships an ExecuTorch runtime derived from the `v1.2.0` release branch. **No forward compatibility** is guaranteed — older versions of the runtime may not work with these files.
## Variant Matrix
| Delegate | Precision | File | Size | RMSE vs eager | Notes |
|----------|-----------|------------------------------------------------------------------------------|--------|---------------|--------------------------------------------------------------------|
| XNNPACK | fp32 | `xnnpack/distiluse-base-multilingual-cased-v2_xnnpack_fp32.pte` | 516 MB | 0.0 | Baseline. Works on Android / iOS / generic CPU. |
| XNNPACK | 8da4w | `xnnpack/distiluse-base-multilingual-cased-v2_xnnpack_8da4w.pte` | 375 MB | 5.4e-4 | Int8 dynamic activation + Int4 weight (torchao), group_size=32. Embeddings stay fp32 — the bulk of the size reduction comes from linear layers. |
| CoreML | fp32 | `coreml/distiluse-base-multilingual-cased-v2_coreml_fp32.pte` | 516 MB | 0.0 | Apple Neural Engine / GPU / CPU, float32 compute. |
| CoreML | fp16 | `coreml/distiluse-base-multilingual-cased-v2_coreml_fp16.pte` | 258 MB | 1.9e-4 | Half-sized via `compute_precision=FLOAT16` at CoreML compile. Cleanest size win on iOS. |
Pick the variant that matches your platform + size/quality trade-off. The CoreML variants only load on Apple platforms; the XNNPACK variants load everywhere.
## Repository Structure
- `xnnpack/` — `.pte` files partitioned for the XNNPACK delegate.
- `coreml/` — `.pte` files partitioned for the CoreML delegate (iOS / macOS only).
- `tokenizer.json` — HuggingFace fast-tokenizer dump (WordPiece + BertNormalizer). Wire this to `tokenizerSource`.
- `config.json`, `tokenizer_config.json` — upstream model/tokenizer configs, kept for reference and for non-RNE consumers.
The `.pte` path goes to `modelSource`; `tokenizer.json` is shared across all variants.
## Model details
- Architecture: DistilBERT multilingual cased + mean pooling + Dense (768→512, Tanh) + L2 norm.
- Output dimension: **512**.
- Max sequence length: **126** tokens (128 − 2 for `[CLS]` / `[SEP]`).
- Languages: 50+ (multilingual).
- Typical strength: cross-lingual sentence similarity and medium-length sentence retrieval. Short single-word queries in non-English languages are this model's weakest case — for those, longer sentences and/or English inputs give markedly better ranking.
## Export notes
The exported program skips HuggingFace's internal attention-mask-to-4D conversion because the RNE runtime never pads at inference (single sentence, no batching). This preserves bit-exactness with the PyTorch reference (RMSE 0 on fp32 random input) while trimming ~27% off the XNNPACK forward wall-time and keeping XNNPACK delegation around 89–91% of graph runtime.
Unsupported combinations (rejected by the exporter, documented for reference):
- **XNNPACK + fp16** — `model.to(torch.float16)` causes softmax / LayerNorm overflow and the runtime output is NaN. XNNPACK's size wins come from quantization, not fp16.
- **CoreML + 8da4w** — `coremltools` has no MIL mapping for the `torch.int8` tensors torchao emits (`KeyError: torch.int8`). The CoreML-native way to shrink further is `ct.optimize.coreml` palette/linear quantization, not torchao source transforms.
|