| --- |
| language: |
| - af |
| - am |
| - ar |
| - hy |
| - az |
| - bn |
| - ba |
| - be |
| - bs |
| - bg |
| - ca |
| - zh |
| - hr |
| - cs |
| - da |
| - nl |
| - en |
| - eo |
| - et |
| - eu |
| - fil |
| - fi |
| - fr |
| - fur |
| - gl |
| - ka |
| - de |
| - el |
| - gu |
| - ha |
| - haw |
| - he |
| - hi |
| - hu |
| - is |
| - id |
| - ga |
| - it |
| - ja |
| - kn |
| - kk |
| - km |
| - ko |
| - ky |
| - lo |
| - la |
| - lv |
| - lt |
| - mk |
| - mg |
| - ms |
| - ml |
| - mt |
| - mr |
| - mn |
| - my |
| - ne |
| - nn |
| - oc |
| - om |
| - ps |
| - fa |
| - pl |
| - pt |
| - ro |
| - ru |
| - sa |
| - sat |
| - sr |
| - sn |
| - si |
| - sk |
| - sl |
| - es |
| - sw |
| - sv |
| - gsw |
| - ta |
| - tt |
| - te |
| - th |
| - bo |
| - tr |
| - udm |
| - uk |
| - ur |
| - uz |
| - vi |
| - cy |
| - xh |
| - yi |
| - yo |
| - zu |
| tags: |
| - keyboardrage |
| - semantic-search |
| - embeddings |
| - typing-game |
| - multilingual |
| - faiss |
| - granite-embedding |
| license: mit |
| datasets: |
| - wiktionary |
| - monkeytype |
| --- |
| |
| # KeyboardRage Semantic Models |
|
|
| Precomputed multilingual semantic embeddings and neighbor indices for the [KeyboardRage](https://github.com/EMRD95/keyboardrage) typing game's Galaxy visualization. |
|
|
| ## Overview |
|
|
| This repository contains the trained semantic models that power the 3D semantic word galaxy in KeyboardRage. 2,250,636 words across 108 languages are embedded into a 384-dimensional space using IBM's Granite multilingual embedding model, then projected to 3D via UMAP. Precomputed nearest-neighbor indices enable real-time similarity queries. |
|
|
| ## Contents |
|
|
| ### Global embeddings & index (15 GB) |
| | File | Size | Description | |
| |------|------|-------------| |
| | `semantic_embeddings.f32.npy` | 3.3 GB | Float32 embeddings for all 2.25M words (384-dim, inner-product normalized) | |
| | `semantic_faiss_hnsw.index` | 3.3 GB | FAISS flat inner-product index (exact cosine similarity over normalized vectors) | |
| | `neighbor_ids.npy` | 1.7 GB | Precomputed global top-200 neighbor IDs (rows × 200, int64) | |
| | `neighbor_scores.npy` | 1.7 GB | Precomputed global top-200 cosine similarity scores (float32) | |
| | `semantic_index_meta.json` | ~1 KB | Model metadata (embedding model, dimensions, row count) | |
|
|
| ### Per-language neighbor indices |
| 108 languages, each with 4 files: |
| - `neighbor_ids_{lang}.npy` — precomputed within-language top-200 neighbor IDs |
| - `neighbor_scores_{lang}.npy` — cosine similarity scores |
| - `lang_index_{lang}.npy` — global-ID → local-ID mapping |
| - `neighbor_meta_{lang}.json` — per-language statistics |
|
|
| ### 3D projection metadata |
| | File | Size | Description | |
| |------|------|-------------| |
| | `atlas_data.parquet` | 88 MB | Word metadata: 3D UMAP coordinates (x, y, z), word, language, definition | |
|
|
| ### Raw word embeddings (834 MB) |
| `words_emb_merged/{lang}.json` — raw embedding vectors per language, used for regeneration workflows. |
|
|
| ## Model Details |
|
|
| - **Embedding model**: [ibm-granite/granite-embedding-97m-multilingual-r2](https://huggingface.co/ibm-granite/granite-embedding-97m-multilingual-r2) |
| - **Embedding dimension**: 384 |
| - **Total words embedded**: 2,250,636 |
| - **Languages**: 108 |
| - **Similarity metric**: Cosine similarity via normalized inner product |
| - **3D projection**: UMAP (n_components=3, metric='cosine') |
| - **Neighbor count**: Top 200 per word (global + per-language) |
| |
| ## 108 Supported Languages |
| |
| afrikaans, albanian, amharic, arabic (+egypt, +morocco), armenian (+western), azerbaijani, bangla, bashkir, belarusian (+lacinka), bosnian, bulgarian, catalan, chinese_simplified, chinese_traditional, croatian, czech, danish, dutch, english, esperanto (+h_sistemo, +x_sistemo), estonian, euskera, filipino, finnish, french, friulian, galician, georgian, german, greek, gujarati, hausa, hawaiian, hebrew, hindi, hungarian, icelandic, indonesian, irish, italian, japanese (hiragana, katakana, romaji), kannada, kazakh, khmer, korean, kyrgyz, lao, latin, latvian, lithuanian, macedonian, malagasy, malay, malayalam, maltese, marathi, mongolian, myanmar, nepali, norwegian_nynorsk, occitan, oromo, pashto, persian, polish, portuguese (+acentos_e_cedilha), romanian, russian, sanskrit, santali, serbian (+latin), shona, sinhala, slovak, slovenian, spanish, swahili, swedish, swiss_german, tamil, tatar (+crimean, +crimean_cyrillic), telugu, thai, tibetan, turkish, udmurt, ukrainian (+latynka), urdu, uzbek, vietnamese, welsh, xhosa, yiddish, yoruba, zulu |
|
|
| ## On-Premise Deployment |
|
|
| ### Prerequisites |
| - Python 3.10+ |
| - FastAPI, NumPy, DuckDB, PyArrow |
|
|
| ### Quick start |
|
|
| ```bash |
| # 1. Clone the game code |
| git clone https://github.com/EMRD95/keyboardrage |
| cd keyboardrage |
| |
| # 2. Download models from HuggingFace |
| ./setup.sh |
| |
| # 3. Run the semantic neighbors API |
| cd galaxy/semantic |
| pip install fastapi uvicorn numpy duckdb pyarrow |
| python semantic_neighbors_server.py |
| # API available at http://localhost:8703 |
| ``` |
|
|
| ### API Endpoints |
|
|
| | Endpoint | Description | |
| |----------|-------------| |
| | `GET /health` | Server status, available languages, row count | |
| | `GET /point/{id}` | Get word metadata by global ID | |
| | `GET /neighbors/{id}?k=10&language=french` | Get nearest neighbors (global or per-language) | |
| | `GET /search?q=mot&language=french` | Full-text word search | |
|
|
| ## Source Code |
|
|
| The KeyboardRage game source code and visualization themes are at: |
| **[github.com/EMRD95/keyboardrage](https://github.com/EMRD95/keyboardrage)** |
|
|
| ## Regeneration |
|
|
| To rebuild these models from scratch: |
|
|
| ```bash |
| # 1. Rebuild embeddings from merged word lists |
| cd galaxy && ./rebuild_from_merged_words.sh |
| |
| # 2. Rebuild semantic index |
| cd semantic && python build_semantic_index.py |
| |
| # 3. Precompute neighbors |
| python precompute_neighbors.py --per-language |
| |
| # 4. Rebuild 3D projection |
| cd ../3D_galaxy && ./run_umap50_projection_rebuild.sh |
| ``` |
|
|
| All rebuild scripts are in the [GitHub repository](https://github.com/EMRD95/keyboardrage/tree/develop/galaxy). |
|
|
| ## License |
|
|
| MIT — same as KeyboardRage. |
|
|
