| --- |
| license: other |
| license_name: futo-model-weights-license-1.0 |
| license_link: LICENSE.md |
| library_name: executorch |
| tags: |
| - swipe-typing |
| - gesture-typing |
| - shape-writing |
| - keyboard |
| - on-device |
| - executorch |
| - ctc |
| - mobile |
| datasets: |
| - futo-org/swipe.futo.org |
| - futo-org/swipe-negatives |
| --- |
| |
| # FUTO Swipe |
|
|
| Mobile-oriented models for decoding swipe gestures into text. |
|
|
| <p align="center"> |
| <img src="https://huggingface.co/futo-org/futo-swipe/resolve/main/animations/swipe_demo_computer.gif" width="640" alt="Swipe decode of the word 'computer'"> |
| </p> |
|
|
| See the paper [(*coming soon*)](https://huggingface.co/futo-org/futo-swipe) |
| for more details. |
|
|
| ## Models |
|
|
| This repository contains 3 CNN models that compose together. |
| Only the encoder is required. The decoder and language model are |
| additional refinements, leveraging specific layout and language information. |
| The encoder can decode for **any** keyboard layout, |
| while the decoder is English/QWERTY-only and the language model is English-only. |
|
|
| | Model | Codename | Role | Params | Size (fp32) | |
| |-------|----------|------|-------:|------------:| |
| | **Encoder** | `honorable_sturgeon` | Maps a swipe trajectory to per-timestep character emissions. Layout-agnostic — works on any keyboard supplied at runtime. | 635 K | 2.65 MB | |
| | **Decoder** | `magic_macaw` | (Optional) per-layout refinement over the encoder's frozen features. Lifts top-k where layout-specific training data exists (English here). | 304 K | 1.25 MB | |
| | **Context LM** | `hungry_jellyfish` | (Optional) next-word and beam-rerank language model that blends sentence context into candidate ranking. | 1.5M | 6.25 MB | |
|
|
| ### Encoder — `honorable_sturgeon` |
| |
| A 1D temporal convolutional network (TCN) reads the raw `(x, y)` touch trajectory |
| and emits a 64-coefficient spectral pattern and a scalar |
| "intention" gate for each timestep. Per-key character scores are read off by evaluating a |
| fixed cosine (DCT) basis at the layout key centers. Switching layouts on |
| device requires no retraining, just a different key-coordinate tensor. |
| |
| ### Decoder (English/QWERTY) — `magic_macaw` |
|
|
| A small DFSMN decoder over the frozen encoder features. It refines the |
| character distribution on specific layouts. |
| Currently we only have data for training an English/QWERTY decoder. |
|
|
| ### Context LM (English) — `hungry_jellyfish` |
| |
| A causal DFSMN language model with a hash embedding for large vocabularies. |
| It can supply a rerank score and perform next-word prediction. This model |
| can be used with or without a decoder. |
| |
| ## Getting started |
| |
| The example below demonstrates the encoder on CPU (x86) with the ExecuTorch |
| runtime and greedy-decodes a swipe into characters. Note that greedy decoding |
| is fairly inaccurate and should generally be improved by constraining to a |
| lexicon (eg. trie, WFST). |
| |
| ```python |
| import numpy as np |
| import torch |
| from huggingface_hub import hf_hub_download |
| from executorch.runtime import Runtime |
|
|
| # QWERTY letter centers in the normalized [0,1] keyboard frame. This is the |
| # only layout-specific input (can swap in another layout's key coordinates to |
| # decode a keyboard the encoder never saw at training time). |
| QWERTY = { |
| "a": (0.10, 0.500), "b": (0.60, 0.833), "c": (0.40, 0.833), "d": (0.30, 0.500), |
| "e": (0.25, 0.167), "f": (0.40, 0.500), "g": (0.50, 0.500), "h": (0.60, 0.500), |
| "i": (0.75, 0.167), "j": (0.70, 0.500), "k": (0.80, 0.500), "l": (0.90, 0.500), |
| "m": (0.80, 0.833), "n": (0.70, 0.833), "o": (0.85, 0.167), "p": (0.95, 0.167), |
| "q": (0.05, 0.167), "r": (0.35, 0.167), "s": (0.20, 0.500), "t": (0.45, 0.167), |
| "u": (0.65, 0.167), "v": (0.50, 0.833), "w": (0.15, 0.167), "x": (0.30, 0.833), |
| "y": (0.55, 0.167), "z": (0.20, 0.833), |
| } |
| LETTERS = sorted(QWERTY) |
| MAX_KEYS = 64 # export-time padding bound |
| |
| # A real swipe for the word "computer": normalized x, y and timestamps (ms). |
| PX = [0.4141, 0.4478, 0.5, 0.5741, 0.6599, 0.7256, 0.7744, 0.8098, 0.8485, 0.867, |
| 0.8737, 0.8653, 0.8418, 0.8182, 0.8098, 0.7963, 0.7946, 0.8081, 0.8418, 0.8704, |
| 0.9057, 0.9259, 0.9545, 0.9697, 0.968, 0.9529, 0.9141, 0.8468, 0.7811, 0.7273, |
| 0.6869, 0.6616, 0.6582, 0.6431, 0.6061, 0.5572, 0.5067, 0.4663, 0.4495, 0.4461, |
| 0.4411, 0.4192, 0.3872, 0.362, 0.3283, 0.2795, 0.2391, 0.2323, 0.2407, 0.2593, |
| 0.2879, 0.3249, 0.3468, 0.3569] |
| PY = [0.8991, 0.858, 0.7876, 0.6702, 0.5352, 0.4237, 0.3357, 0.2653, 0.1655, 0.142, |
| 0.142, 0.2183, 0.3709, 0.588, 0.7347, 0.8462, 0.8697, 0.811, 0.6115, 0.4707, |
| 0.3122, 0.2066, 0.1303, 0.1068, 0.1068, 0.1068, 0.1185, 0.1596, 0.1772, 0.1772, |
| 0.1772, 0.189, 0.189, 0.189, 0.1831, 0.189, 0.189, 0.189, 0.189, 0.189, |
| 0.1831, 0.1831, 0.1831, 0.1831, 0.1831, 0.1948, 0.189, 0.1948, 0.189, 0.189, |
| 0.189, 0.1831, 0.1831, 0.1831] |
| PT = [0.0, 100, 149, 197, 246, 297, 348, 399, 449, 498, 548, 598, 648, 698, 749, 799, |
| 849, 949, 999, 1047, 1100, 1152, 1197, 1248, 1314, 1364, 1414, 1465, 1515, 1565, |
| 1614, 1666, 1715, 1851, 1898, 1951, 1998, 2049, 2097, 2165, 2231, 2279, 2331, |
| 2382, 2431, 2481, 2532, 2584, 2649, 2700, 2751, 2798, 2848, 2899] |
| |
|
|
| def resample(px, py, pt, T=64): |
| """Resample a variable-length trajectory to T evenly-spaced points -> [2, T].""" |
| x, y, t = map(np.asarray, (px, py, pt)) |
| t = t - t[0] |
| if t[-1] > 1e-3: # uniform 60 Hz resample, then to T points |
| n60 = max(2, round(t[-1] / (1000.0 / 60.0)) + 1) |
| tt = np.linspace(0.0, t[-1], n60) |
| x, y = np.interp(tt, t, x), np.interp(tt, t, y) |
| idx = np.linspace(0, len(x) - 1, T) |
| rx = np.interp(idx, np.arange(len(x)), x) |
| ry = np.interp(idx, np.arange(len(y)), y) |
| return np.stack([rx, ry], axis=0).astype(np.float32) |
| |
|
|
| def greedy_ctc(log_emissions): |
| """Collapse the per-timestep argmax into a string (blank is the last class).""" |
| blank = log_emissions.shape[-1] - 1 |
| out, prev = [], -1 |
| for c in log_emissions[0].argmax(axis=-1): |
| if c != prev and c != blank and c < len(LETTERS): |
| out.append(LETTERS[c]) |
| prev = c |
| return "".join(out) |
| |
|
|
| # Load the encoder .pte and run one forward pass. |
| pte = hf_hub_download("futo-org/futo-swipe", "honorable_sturgeon/model_fp32.pte") |
| encoder = Runtime.get().load_program(pte).load_method("forward") |
|
|
| features = torch.from_numpy(resample(PX, PY, PT)[None]) # [1, 2, 64] |
| keys = torch.zeros(1, MAX_KEYS, 2) # [1, 64, 2] |
| mask = torch.zeros(1, MAX_KEYS, dtype=torch.bool) # [1, 64] |
| for i, ch in enumerate(LETTERS): |
| keys[0, i] = torch.tensor(QWERTY[ch]) |
| mask[0, i] = True |
| |
| log_emissions, coefficients, lambda_ = encoder.execute((features, keys, mask)) |
| print("greedy decode:", greedy_ctc(log_emissions.numpy())) # -> "computer" |
| ``` |
| |
| Example output: |
| |
| ``` |
| greedy decode: computer |
| ``` |
| |
| ### Encoder inputs and outputs |
| |
| | | Tensor | Shape | Meaning | |
| |---|--------|-------|---------| |
| | **input** | `features` | `[1, 2, 64]` | Swipe trajectory `(x, y)` resampled to 64 points | |
| | **input** | `layout_keys` | `[1, 64, 2]` | Per-key `(x, y)` centers, padded to 64 keys | |
| | **input** | `layout_mask` | `[1, 64]` | Boolean mask of valid keys | |
| | **output** | `log_emissions` | `[1, 32, 65]` | Log-probabilities over 64 keys + blank | |
| | **output** | `coefficients` | `[1, 32, 64]` | Spectral coefficients (decoder features) | |
| | **output** | `lambda` | `[1, 32, 1]` | *Intention* gate (decoder features) | |
| |
| The output time dimension is 32, half the 64 input points. The encoder |
| applies a 2× temporal downsample (a stride-2 adapter) inside the network, so |
| the 64 trajectory steps become 32 emission steps. |
| |
| ## License |
| |
| Released under the [FUTO Model Weights License 1.0](LICENSE.md). |
