Halfword 100M: Llama-architecture AAC predictor with Core ML, ONNX and GGUF exports

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+halfword.gguf filter=lfs diff=lfs merge=lfs -text
+ngram.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,139 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- aac
+- augmentative-and-alternative-communication
+- assistive-technology
+- accessibility
+- text-entry
+- word-prediction
+- on-device
+- eye-gaze
+- switch-scanning
+- coreml
+- onnx
+- gguf
+datasets:
+- allenai/soda
+- google/Synthetic-Persona-Chat
+- pfb30/multi_woz_v22
+- Helsinki-NLP/tatoeba
+---
+
+# Halfword 100M
+
+A 0.1 B-parameter Llama-architecture language model for **AAC word prediction**, trained to run on the device and paired with a decoder that optimises **seconds to utterance** rather than keystrokes saved.
+
+The model on its own is unremarkable — a small LM trained on public conversation. What it is *for* is the interesting part: it feeds a decoder that knows what a candidate costs to reach on the person's actual access method, and chooses the candidate list accordingly.
+
+- **Code, benchmark and figures:** https://github.com/NagaYu/halfword
+- **Interactive demo:** https://huggingface.co/spaces/NagaYu/halfword
+- **Benchmark dataset:** https://huggingface.co/datasets/NagaYu/halfword-bench
+
+## What ships here
+
+| file | what it is |
+|---|---|
+| `model.safetensors`, `config.json` | the 101 M-parameter Llama-architecture model |
+| `tokenizer.json` | 16 k byte-level BPE, trained on the same corpus |
+| `halfword.onnx` | ONNX export, dynamic batch and sequence, verified in onnxruntime |
+| `halfword.mlpackage` | Core ML export, float16 |
+| `halfword.gguf` | GGUF under the stock `llama` architecture, for llama.cpp |
+| `ngram.json` | 4-gram fallback predictor — runs with no accelerator and no torch |
+| `calibrator.json` | acceptance calibrator (see below; the decoder does not work without it) |
+
+## Architecture
+
+Llama-architecture (RMSNorm, RoPE, SwiGLU, tied embeddings). That is an **export decision, not a preference**: it is what Core ML, ONNX and llama.cpp's GGUF path all already understand, so one definition leaves in three formats without a bespoke converter for each.
+
+| | |
+|---|---|
+| parameters | 100.7 M |
+| layers / hidden / heads / KV heads | 14 / 768 / 12 / 4 |
+| intermediate | 2048 |
+| vocabulary | 16 384 |
+| context | 512 |
+
+## Usage
+
+```python
+from halfword.predictor import NeuralPredictor, Context
+
+pred = NeuralPredictor.load("NaGaYu/halfword-100m")   # local dir or snapshot
+pred.warmup()
+for c in pred.predict(Context(history=("how are you today",), committed="i would like some wa")):
+    print(c.insert, round(c.probability, 4), c.kind)
+```
+
+Predictions are **prefix-constrained**: having typed `wa`, every candidate extends `wa`. Tokens that cannot are masked before the softmax is inspected, which both improves the candidates and bounds the work — a word finishes in one or two tokens far more often than a free continuation does.
+
+To get the point of the project, use it with the decoder:
+
+```python
+from halfword.costs import CostModel, Surface, load_keyboards
+from halfword.decoder import CostAwareDecoder
+from halfword.calibrate import AcceptanceCalibrator
+
+cost    = CostModel.from_name("scanning", scan_interval_s=1.2)
+surface = Surface(load_keyboards()["abc_6x6"])
+decoder = CostAwareDecoder(max_candidates=8,
+                           calibrator=AcceptanceCalibrator.load("calibrator.json"))
+shown = decoder.present(pred.predict(ctx), surface, cost)   # may legitimately be 0 candidates
+```
+
+## How good is it? Worse than the n-gram beside it.
+
+Measured on 8 000 utterances the model never saw (rows beyond the training cut of each source), top-5, 120 word boundaries:
+
+| | next word, nothing typed | word completion, 2 letters typed | latency |
+|---|---|---|---|
+| `ngram.json` (4-gram, 15 MB) | **59.2%** | **90.2%** | 0.7 ms |
+| this model (101 M) | 9.2% | 63.9% | 197 ms |
+
+That is not a typo and it is not a bug hunt left unfinished. **The trivial baseline shipped alongside this model beats it comfortably**, and you should use `ngram.json` unless you have a reason not to.
+
+The reason is ordinary: the model saw roughly 9 M tokens of training — about three passes over a 3 M-token corpus, in 100 minutes on a laptop GPU. Small language models that are actually good see three to four orders of magnitude more. This one has learned English morphology (`pl` → `please`, `wi` → `with`, `m` → `much`) but not much about what people say next, which is exactly the shape you would predict from the token budget.
+
+**What it is good for.** It is a working reference for the parts that are hard to get right and easy to get silently wrong: the Llama-architecture sizing, the three exports with their gotchas, the prefix-constrained decoding with token healing, and the KV-cache reuse that keeps latency inside a scan step. All of that is correct and tested. Swap in better weights and the rest of the stack does not change.
+
+**What would fix it.** More tokens, in this order: a larger and more conversational corpus (the current mix is 40% Tatoeba single sentences, which teach grammar but not dialogue), 10–50× the training steps, and a held-out early-stopping signal instead of a fixed step count. None of that is research; it is compute.
+
+## Latency
+## The calibrator is not optional
+
+The decoder converts probabilities into seconds. A raw model score is not "the probability this candidate is what the person means", and fed raw scores the decoder concludes prediction is *always* right — which makes long candidate lists free and collapses the entire trade-off the project exists to model. `calibrator.json` fits "is this the first correct candidate in the pool", so the residual mass is a genuine "you will have to spell another character". Held-out expected calibration error is reported in `provenance.json`.
+
+## Training data
+
+Public, openly-licensed conversational corpora only. Licences were checked individually and are recorded per source with a verification date.
+
+| source | licence |
+|---|---|
+| [`allenai/soda`](https://huggingface.co/datasets/allenai/soda) | CC BY 4.0 |
+| [`google/Synthetic-Persona-Chat`](https://huggingface.co/datasets/google/Synthetic-Persona-Chat) | CC BY 4.0 |
+| [`pfb30/multi_woz_v22`](https://huggingface.co/datasets/pfb30/multi_woz_v22) | Apache-2.0 |
+| [`Helsinki-NLP/tatoeba`](https://huggingface.co/datasets/Helsinki-NLP/tatoeba) | CC BY 2.0 FR |
+
+**No AAC user data.** There is, as far as we know, no public openly-licensed corpus of real AAC communication, and this project did not create one. The register of open-domain chat is not the register of AAC use, and that gap is a real limitation of everything here.
+
+Non-commercial corpora (e.g. DailyDialog, CC BY-NC-SA) are excluded by default. Including them would make this model non-commercial too.
+
+## Limitations
+
+- **It loses to a 4-gram.** See the table above. This is a reference implementation with real weights, not a good predictor, and the card says so where you will read it rather than in a footnote.
+- **Small model, narrow corpus.** 0.1 B parameters, 3 M training tokens, 100 minutes. It will not know your family's names, your medications, or your jokes — and on this evidence it does not know much about ordinary conversation either. On-device personalisation addresses the first of those, and never sends anything anywhere.
+- **The seconds are simulated.** Nobody has been timed using this. The cost models price mechanical scan, dwell and pointing time under stated assumptions.
+- **Visual search cost is not modelled.** The AAC literature argues the cost of *reading* a prediction list is a large part of why prediction can slow people down. Ignoring it makes long lists look cheaper than they are, so the reported gaps are a lower bound.
+- **Every duration is a property of an interface, never of a person.** A long scan interval means *this interface costs more per selection* and nothing about who is using it.
+
+## If you use AAC, or support someone who does
+
+This was built without AAC users in the room, which is a real limitation and not a small one. Blasko et al. (2025) put the problem in a title: *["Nothing about AAC users without AAC users"](https://doi.org/10.1080/07434618.2025.2514748)*. If the defaults look wrong to you — the scan intervals, the dwell times, the assumption that seconds are the right thing to minimise at all — please open a discussion here or an issue on GitHub. We would rather be corrected than cited.
+
+## Citation
+
+The finding that keystroke savings and communication rate come apart is **not ours**. See Koester & Levine (1994, 1996), Higginbotham (1992), Trnka et al. (2007–2009), Quinn & Zhai (2016), and especially Cai et al. ([2024, *Nature Communications*](https://doi.org/10.1038/s41467-024-53873-3)). What is new here is narrower: identifying *positional cost* rather than slowness as the mechanism, and making candidate-list length a computed decision variable in a time objective. Full positioning in the [repository README](https://github.com/NagaYu/halfword).

calibrator.json

@@ -0,0 +1 @@
+{"weights": [3.5384301057400043, 0.10019243562855197, 0.2035921170264405, 0.11479055941771045, -1.9041843052800418, 0.03140271699474312, -0.6681733880820065, -3.49499685786082, -0.7230667556147661], "features": ["bias", "logp", "margin_from_top", "rank", "log1p_rank", "new_chars", "log1p_new_chars", "is_phrase", "pool_size"]}

config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 2048,
+  "max_position_embeddings": 512,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 14,
+  "num_key_value_heads": 4,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": true,
+  "transformers_version": "4.57.6",
+  "use_cache": true,
+  "vocab_size": 16384
+}

export_report.json

@@ -0,0 +1,50 @@
+{
+ "model": "runs/halfword-100m",
+ "parameters": 100685568,
+ "torch_cpu_median_ms": 46.064916999966954,
+ "exports": [
+  {
+   "format": "onnx",
+   "path": "runs/exports/halfword.onnx",
+   "ok": true,
+   "bytes": 404918466,
+   "latency_ms_median": 15.054291498017847,
+   "latency_ms_p90": 16.251625002041692,
+   "max_abs_error": 2.956390380859375e-05,
+   "notes": "",
+   "detail": {
+    "dynamic_batch_verified": true
+   }
+  },
+  {
+   "format": "coreml",
+   "path": "runs/exports/halfword.mlpackage",
+   "ok": true,
+   "bytes": 201964284,
+   "latency_ms_median": 32.850979499926325,
+   "latency_ms_p90": 33.795082999859005,
+   "max_abs_error": 0.03583550453186035,
+   "notes": "",
+   "detail": {
+    "precision": "fp16",
+    "compute_units": "CPU_ONLY"
+   }
+  },
+  {
+   "format": "gguf",
+   "path": "runs/exports/halfword.gguf",
+   "ok": true,
+   "bytes": 453451168,
+   "latency_ms_median": 0.0,
+   "latency_ms_p90": 0.0,
+   "max_abs_error": NaN,
+   "notes": "",
+   "detail": {
+    "tensors_written": 129,
+    "arch": "llama",
+    "kv_keys": 27,
+    "tensors_read": 129
+   }
+  }
+ ]
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 0,
+  "transformers_version": "4.57.6"
+}

halfword.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de44722a8edfad750d3703675f292e624d67e7183108667a087e5fa4a21ebf9d
+size 453451168

halfword.mlpackage/Data/com.apple.CoreML/model.mlmodel

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:70fbae6be568ce7adcaeff3d8ae2ae6e23bde9ef94487e73d4467dc6622314c1
+size 270325

halfword.mlpackage/Data/com.apple.CoreML/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f82f3cea164447deed6084d4c2b93efe2bcb6f3403f2c80c02b739029e0c33ef
+size 201418688

halfword.mlpackage/Manifest.json

@@ -0,0 +1,18 @@
+{
+    "fileFormatVersion": "1.0.0",
+    "itemInfoEntries": {
+        "4DAF3277-28BC-46B2-9652-165CB7080110": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Specification",
+            "name": "model.mlmodel",
+            "path": "com.apple.CoreML/model.mlmodel"
+        },
+        "58A426E7-F772-4C0B-9CA1-D20E50BD8627": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Weights",
+            "name": "weights",
+            "path": "com.apple.CoreML/weights"
+        }
+    },
+    "rootModelIdentifier": "4DAF3277-28BC-46B2-9652-165CB7080110"
+}

halfword.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd2fedfc4db04239d90cf136fb5ac768ff1a674b1f0b9621839214bd368c82a4
+size 404918466

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:600887c9c760a51f4c283434563129f8b636f6218cdb77c1d71cedc330eeeafd
+size 402756480

ngram.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0f9e3f3eb2925d4843366c1012582cdff49e627f55dde19b9eca79b40278fdcb
+size 15180085

provenance.json

@@ -0,0 +1,55 @@
+{
+ "ngram": {
+  "order": 4,
+  "pruned_min_count": 2,
+  "bytes": 15180085,
+  "word_types": 29273
+ },
+ "calibration": {
+  "ece": 0.0041261140127787236,
+  "n": 349872,
+  "pools": 103977
+ },
+ "config": "halfword-100m",
+ "parameters": 100685568,
+ "train_steps": 3000,
+ "final_loss": 2.892653913497925,
+ "train_tokens": 3014077,
+ "batch_size": 16,
+ "seq_len": 192,
+ "lr": 0.0004,
+ "device": "mps",
+ "sources": [
+  {
+   "name": "halfword-seed",
+   "license": "CC0-1.0",
+   "url": "halfword/data/seed_corpus.json",
+   "verified_on": "2026-08-31"
+  },
+  {
+   "name": "allenai/soda",
+   "license": "cc-by-4.0",
+   "url": "https://huggingface.co/datasets/allenai/soda",
+   "verified_on": "2026-08-31"
+  },
+  {
+   "name": "google/Synthetic-Persona-Chat",
+   "license": "cc-by-4.0",
+   "url": "https://huggingface.co/datasets/google/Synthetic-Persona-Chat",
+   "verified_on": "2026-08-31"
+  },
+  {
+   "name": "pfb30/multi_woz_v22",
+   "license": "apache-2.0",
+   "url": "https://huggingface.co/datasets/pfb30/multi_woz_v22",
+   "verified_on": "2026-08-31"
+  },
+  {
+   "name": "Helsinki-NLP/tatoeba",
+   "license": "cc-by-2.0",
+   "url": "https://huggingface.co/datasets/Helsinki-NLP/tatoeba",
+   "verified_on": "2026-08-31"
+  }
+ ],
+ "commercial_use_permitted": true
+}

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>"
+}

tokenizer_config.json

@@ -0,0 +1,44 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<unk>"
+}