| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - sg-transit |
| - autocomplete |
| - siamese |
| - character-level |
| - text-matching |
| - singapore |
| - bus-stop |
| - mrt |
| pipeline_tag: feature-extraction |
| --- |
| |
| # SG Transit Prefix Encoder |
|
|
| A lightweight (~110k params) **character-level Siamese encoder** that maps |
| messy keyboard input to official Singapore bus-stop and MRT station names. |
|
|
| Built for on-device autocomplete in a mobile transit app. |
|
|
| ## What it handles |
|
|
| | User types | Matches | |
| |----------------------------|------------------| |
| | `bed` | Bedok | |
| | `choachukang` (no spaces) | Choa Chu Kang | |
| | `cck`, `amk`, `ns4` | the right station| |
| | `block 209` | Blk 209 | |
| | `rd`, `stn`, `int` | Road, Station, Interchange | |
| | `bedik` (typo) | Bedok | |
|
|
| ## Architecture |
|
|
| - **Tokenizer**: custom character-level, fixed alphabet of 47 chars (`a-z`, |
| `0-9`, space, `'-.&/,`). Max length = 40 chars. |
| - **Encoder**: `Embedding(48) -> 1-layer BiGRU(96) -> masked mean-pool -> |
| Linear(128)`. Shared (Siamese) weights for query and candidate. |
| - **Training**: in-batch InfoNCE contrastive loss over 260k `(input, official)` |
| pairs generated from LTA DataMall bus stops and MRT/LRT station data. |
|
|
| ## Usage |
|
|
| Two approaches are available, both suitable for Android / mobile apps. |
|
|
| ### Option A: embeddings.bin (no ML runtime required) |
|
|
| Just `embeddings.bin` + `embeddings_names.json`. Precomputed vectors — no |
| TorchScript, no model inference. Ideal when you want a simple exact/prefix |
| lookup without running a neural net on every keystroke. |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| |
| REPO = "Luke-Yong/sg-transit-prefix-encoder" |
| bin_path = hf_hub_download(REPO, "embeddings.bin") |
| names_path = hf_hub_download(REPO, "embeddings_names.json") |
| ``` |
|
|
| `embeddings.bin` layout: `uint32 N`, `uint32 D` (little-endian), then `N * D` float32. |
|
|
| ### Option B: TorchScript model (full embedding inference) |
|
|
| `prefix_encoder_scripted.pt` + `char_vocab.json` + `busstops_and_stations_words_prefixes.csv`. |
| Run the encoder on-device via PyTorch Mobile / LibTorch for real-time embedding |
| inference and typo-tolerant search. |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| |
| REPO = "Luke-Yong/sg-transit-prefix-encoder" |
| model_path = hf_hub_download(REPO, "prefix_encoder_scripted.pt") |
| vocab_path = hf_hub_download(REPO, "char_vocab.json") |
| csv_path = hf_hub_download(REPO, "busstops_and_stations_words_prefixes.csv") |
| ``` |
|
|
| ### Python example (full pipeline) |
|
|
| ```python |
| import json, struct |
| import torch |
| import numpy as np |
| from char_encoder import CharEncoder, CharTokenizer |
| from huggingface_hub import hf_hub_download |
| |
| REPO = "Luke-Yong/sg-transit-prefix-encoder" |
| |
| # Load model and tokenizer from Hugging Face |
| model = CharEncoder.from_pretrained(REPO) |
| tokenizer = CharTokenizer.from_pretrained(REPO) |
| |
| # Load precomputed embeddings + name lookup |
| bin_path = hf_hub_download(REPO, "embeddings.bin") |
| names_path = hf_hub_download(REPO, "embeddings_names.json") |
| |
| with open(bin_path, "rb") as f: |
| n, d = struct.unpack("<II", f.read(8)) |
| embeddings = np.frombuffer(f.read(), dtype=np.float32).reshape(n, d) |
| |
| with open(names_path, encoding="utf-8") as f: |
| display_to_index = json.load(f) |
| |
| # Build alias -> (official_name, index) lookup |
| idx_to_name = {} |
| alias_to_officials = {} |
| for disp, idx in display_to_index.items(): |
| parts = [p.strip() for p in disp.split(":") if p.strip()] |
| if len(parts) == 2: |
| alias, official = parts |
| alias_to_officials.setdefault(alias.lower(), []).append(official) |
| idx = int(idx) |
| if idx not in idx_to_name and len(parts) == 2: |
| idx_to_name[idx] = parts[1] |
| |
| # Encode a query |
| def search(query, top_k=5): |
| enc = tokenizer.encode_one(query) |
| with torch.no_grad(): |
| q = model(enc["input_ids"].unsqueeze(0), enc["attention_mask"].unsqueeze(0)) |
| q = q.squeeze(0).numpy() |
| q = q / np.linalg.norm(q) |
| sims = embeddings @ q # cosine similarity (embeddings are already normalized) |
| order = np.argsort(sims)[::-1] |
| seen = set() |
| results = [] |
| for i in order: |
| name = idx_to_name.get(i) |
| if name and name not in seen: |
| seen.add(name) |
| results.append((name, float(sims[i]))) |
| if len(results) >= top_k: |
| break |
| return results |
| |
| # Try it |
| for q in ["choachukang", "cck", "block 209", "blk 209", "bedok"]: |
| print(f"\n{q}:") |
| for name, score in search(q): |
| print(f" {name:<30} {score:.4f}") |
| ``` |
|
|
| ## Files |
|
|
| | File | Purpose | |
| |------|---------| |
| | `prefix_encoder.pt` | Trained model weights | |
| | `prefix_encoder_scripted.pt` | TorchScript export (for Android / PyTorch Mobile) | |
| | `char_vocab.json` | Tokenizer vocabulary (45-char alphabet) | |
| | `char_encoder.py` | Model + tokenizer Python definition | |
| | `config.json` | Hyperparameters | |
| | `embeddings.bin` | Precomputed float32 vectors for all officials | |
| | `embeddings_names.json` | Alias → index lookup for the above | |
| | `busstops_and_stations_words_prefixes.csv` | Training data (alias → official name pairs) | |
|
|
| ### What you need on Android |
|
|
| | App approach | Files needed | |
| |---|---| |
| | Simple lookup (no ML) | `embeddings.bin` + `embeddings_names.json` | |
| | Full inference (typo-tolerant) | `prefix_encoder_scripted.pt` + `char_vocab.json` + `busstops_and_stations_words_prefixes.csv` |
|
|
| ## Inference strategies |
|
|
| Three-tier lookup shared by both deployment options: |
|
|
| | Strategy | Trigger | What it does | |
| |----------|---------|--------------| |
| | 1. Exact alias | `amk`, `cck`, `blk 209`, `ns4` | Returns all mapped officials with score 1.0. No vector math. | |
| | 2. Restricted embedding | prefix/substring match (`bed`, `choachu`) | Narrows candidates to officials whose aliases contain the query, then ranks by cosine similarity. | |
| | 3. Full embedding search | typo / unknown (`tauseng` → `Tai Seng`) | Encodes the query through the model and searches all precomputed embeddings. | |
|
|
| ## Data |
|
|
| Trained on Singapore bus stops from [LTA DataMall](https://datamall.lta.gov.sg) |
| and MRT/LRT station data. The training CSV is generated by applying |
| deterministic augmentation rules (prefixes, joined words, initialisms, |
| abbreviation expansion/reduction, vowel-drop, and keyboard-adjacent typos) to |
| ~5.6k official names, producing ~260k positive pairs. |
|
|
| ## Acknowledgments |
|
|
| Data sourced from the Land Transport Authority (LTA) of Singapore. |
|
|
