Add model card: overview, models, ExecuTorch quickstart

README.md

@@ -0,0 +1,174 @@
+---
+license: other
+license_name: futo-model-weights-license-1.0
+license_link: LICENSE.md
+library_name: executorch
+pipeline_tag: token-classification
+tags:
+  - swipe-typing
+  - gesture-typing
+  - shape-writing
+  - keyboard
+  - on-device
+  - executorch
+  - ctc
+  - mobile
+---
+
+# FUTO Swipe
+
+Mobile-oriented models for decoding swipe gestures into text.
+
+<p align="center">
+  <img src="https://huggingface.co/futo-org/futo-swipe/resolve/main/animations/swipe_demo_computer.gif" width="640" alt="Swipe decode of the word 'computer'">
+</p>
+
+See the paper, [*coming soon*](https://huggingface.co/futo-org/futo-swipe),
+for details.
+
+## Models
+
+This repository contains 3 models that compose together.
+Only the encoder is required; the decoder and language model are
+additional refinements, leveraging specific layout and language information.
+The encoder can decode for **any** keyboard layout,
+while the decoder is English/QWERTY-only and the language model is English-only.
+
+| Model | Codename | Role | Params | Size (fp32) |
+|-------|----------|------|-------:|------------:|
+| **Encoder** | `honorable_sturgeon` | Maps a swipe trajectory to per-timestep character emissions. Layout-agnostic — works on any keyboard supplied at runtime. | 635 K | 2.4 MB |
+| **Decoder** | `magic_macaw` | (Optional) per-layout refinement over the encoder's frozen features. Lifts top-k where layout-specific training data exists (English here). | 304 K | 1.2 MB |
+| **Context LM** | `hungry_jellyfish` | (Optional) next-word and beam-rerank language model that blends sentence context into candidate ranking. | — | — |
+
+### Encoder — `honorable_sturgeon`
+
+A 1D temporal convolutional network (TCN) reads the raw `(x, y)` touch trajectory
+and emits a 64-coefficient spectral pattern and a scalar
+"intention" gate for each timestep. Per-key character scores are read off by evaluating a
+fixed cosine (DCT) basis at the layout key centers. Switching layouts on
+device requires no retraining, just a different key-coordinate tensor.
+
+### Decoder (English/QWERTY) — `magic_macaw`
+
+A small DFSMN decoder over the frozen encoder features. It refines the
+character distribution on specific layouts.
+Currently we only have data for training an English/QWERTY decoder.
+
+### Context LM (English) — `hungry_jellyfish`
+
+A causal DFSMN language model with a hash embedding for large vocabularies.
+It can supply a rerank score and perform next-word prediction. This model
+can be used with or without a decoder.
+
+## Getting started
+
+The example below demonstrates the encoder on CPU (x86) with the ExecuTorch
+runtime and greedy-decodes a swipe into characters. Note that greedy decoding
+is fairly inaccurate and should generally be improved by constraining to a vocabulary
+lexicon (eg. trie, WFST).
+
+```python
+import numpy as np
+import torch
+from huggingface_hub import hf_hub_download
+from executorch.runtime import Runtime
+
+# QWERTY letter centers in the normalized [0,1] keyboard frame. This is the
+# only layout-specific input — swap in another layout's key coordinates to
+# decode a keyboard the encoder never saw at training time.
+QWERTY = {
+    "a": (0.10, 0.500), "b": (0.60, 0.833), "c": (0.40, 0.833), "d": (0.30, 0.500),
+    "e": (0.25, 0.167), "f": (0.40, 0.500), "g": (0.50, 0.500), "h": (0.60, 0.500),
+    "i": (0.75, 0.167), "j": (0.70, 0.500), "k": (0.80, 0.500), "l": (0.90, 0.500),
+    "m": (0.80, 0.833), "n": (0.70, 0.833), "o": (0.85, 0.167), "p": (0.95, 0.167),
+    "q": (0.05, 0.167), "r": (0.35, 0.167), "s": (0.20, 0.500), "t": (0.45, 0.167),
+    "u": (0.65, 0.167), "v": (0.50, 0.833), "w": (0.15, 0.167), "x": (0.30, 0.833),
+    "y": (0.55, 0.167), "z": (0.20, 0.833),
+}
+LETTERS = sorted(QWERTY)
+MAX_KEYS = 64  # export-time padding bound
+
+# A real swipe for the word "computer": normalized x, y and timestamps (ms).
+PX = [0.4141, 0.4478, 0.5, 0.5741, 0.6599, 0.7256, 0.7744, 0.8098, 0.8485, 0.867,
+      0.8737, 0.8653, 0.8418, 0.8182, 0.8098, 0.7963, 0.7946, 0.8081, 0.8418, 0.8704,
+      0.9057, 0.9259, 0.9545, 0.9697, 0.968, 0.9529, 0.9141, 0.8468, 0.7811, 0.7273,
+      0.6869, 0.6616, 0.6582, 0.6431, 0.6061, 0.5572, 0.5067, 0.4663, 0.4495, 0.4461,
+      0.4411, 0.4192, 0.3872, 0.362, 0.3283, 0.2795, 0.2391, 0.2323, 0.2407, 0.2593,
+      0.2879, 0.3249, 0.3468, 0.3569]
+PY = [0.8991, 0.858, 0.7876, 0.6702, 0.5352, 0.4237, 0.3357, 0.2653, 0.1655, 0.142,
+      0.142, 0.2183, 0.3709, 0.588, 0.7347, 0.8462, 0.8697, 0.811, 0.6115, 0.4707,
+      0.3122, 0.2066, 0.1303, 0.1068, 0.1068, 0.1068, 0.1185, 0.1596, 0.1772, 0.1772,
+      0.1772, 0.189, 0.189, 0.189, 0.1831, 0.189, 0.189, 0.189, 0.189, 0.189,
+      0.1831, 0.1831, 0.1831, 0.1831, 0.1831, 0.1948, 0.189, 0.1948, 0.189, 0.189,
+      0.189, 0.1831, 0.1831, 0.1831]
+PT = [0.0, 100, 149, 197, 246, 297, 348, 399, 449, 498, 548, 598, 648, 698, 749, 799,
+      849, 949, 999, 1047, 1100, 1152, 1197, 1248, 1314, 1364, 1414, 1465, 1515, 1565,
+      1614, 1666, 1715, 1851, 1898, 1951, 1998, 2049, 2097, 2165, 2231, 2279, 2331,
+      2382, 2431, 2481, 2532, 2584, 2649, 2700, 2751, 2798, 2848, 2899]
+
+
+def resample(px, py, pt, T=64):
+    """Resample a variable-length trajectory to T evenly-spaced points -> [2, T]."""
+    x, y, t = map(np.asarray, (px, py, pt))
+    t = t - t[0]
+    if t[-1] > 1e-3:  # uniform 60 Hz resample, then to T points
+        n60 = max(2, round(t[-1] / (1000.0 / 60.0)) + 1)
+        tt = np.linspace(0.0, t[-1], n60)
+        x, y = np.interp(tt, t, x), np.interp(tt, t, y)
+    idx = np.linspace(0, len(x) - 1, T)
+    rx = np.interp(idx, np.arange(len(x)), x)
+    ry = np.interp(idx, np.arange(len(y)), y)
+    return np.stack([rx, ry], axis=0).astype(np.float32)
+
+
+def greedy_ctc(log_emissions):
+    """Collapse the per-timestep argmax into a string (blank is the last class)."""
+    blank = log_emissions.shape[-1] - 1
+    out, prev = [], -1
+    for c in log_emissions[0].argmax(axis=-1):
+        if c != prev and c != blank and c < len(LETTERS):
+            out.append(LETTERS[c])
+        prev = c
+    return "".join(out)
+
+
+# Load the encoder .pte and run one forward pass.
+pte = hf_hub_download("futo-org/futo-swipe", "honorable_sturgeon/model_fp32.pte")
+encoder = Runtime.get().load_program(pte).load_method("forward")
+
+features = torch.from_numpy(resample(PX, PY, PT)[None])     # [1, 2, 64]
+keys = torch.zeros(1, MAX_KEYS, 2)                          # [1, 64, 2]
+mask = torch.zeros(1, MAX_KEYS, dtype=torch.bool)           # [1, 64]
+for i, ch in enumerate(LETTERS):
+    keys[0, i] = torch.tensor(QWERTY[ch])
+    mask[0, i] = True
+
+log_emissions, coefficients, lambda_ = encoder.execute((features, keys, mask))
+print("greedy decode:", greedy_ctc(log_emissions.numpy()))  # -> "computer"
+```
+
+Example output:
+
+```
+greedy decode: computer
+```
+
+### Encoder inputs and outputs
+
+| | Tensor | Shape | Meaning |
+|---|--------|-------|---------|
+| **in** | `features` | `[1, 2, 64]` | Swipe trajectory `(x, y)` resampled to 64 points |
+| **in** | `layout_keys` | `[1, 64, 2]` | Per-key `(x, y)` centers, padded to 64 keys |
+| **in** | `layout_mask` | `[1, 64]` | Boolean mask of valid keys |
+| **out** | `log_emissions` | `[1, 32, 65]` | Log-probabilities over 64 keys + blank |
+| **out** | `coefficients` | `[1, 32, 64]` | Spectral coefficients |
+| **out** | `lambda` | `[1, 32, 1]` | *Intention* gate |
+
+The output time dimension is **32**, half the 64 input points: the encoder
+applies a 2× temporal downsample (a stride-2 adapter) inside the network, so
+the 64 trajectory steps become 32 emission steps.
+
+## License
+
+Released under the [FUTO Model Weights License 1.0](LICENSE.md).
+