| --- |
| license: mit |
| tags: |
| - micro |
| - tiny |
| - tinyword |
| - microword |
| - word-generation |
| - word |
| - words |
| - little |
| - small |
| - harley-ml |
| - ml |
| - llm |
| - slm |
| - small-langauge-model |
| - tlm |
| datasets: |
| - Harley-ml/es-en-words |
| --- |
| |
| # MicroWord |
|
|
| You wouldn't believe me if I told you this, but we scaled down TinyWord. Meet MicroWord, which is smaller than Tiny, bigger than Pico, and somehow still trying its best. |
| MicroWord is a twenty-three thousand parameter transformer trained on seven-hundred and fifty-three thousand words. Its goal is to generate plausible-looking words based on the morphology of the English and Spanish languages. |
|
|
| ## Architecture |
|
|
| | Parameter | Value | |
| |------------------------|-------| |
| | Hidden Layers | 1 | |
| | Hidden Size | 16 | |
| | Attention Heads | 1 | |
| | KV Heads | 1 | |
| | Intermediate Size | 64 | |
| | RoPE Theta | 1000.0| |
| | Max Position Embeddings| 32 | |
| | Tie Word Embeddings | True | |
| | Vocab Size | 1200 | |
|
|
| Note: 1 attention head and a RoPE Theta of 1000 (vs Qwen3's 1,000,000) are intentional reductions for this scale. Max sequence length is 32, so positional generalization at range isn't a concern. |
|
|
| ## Training |
|
|
| ### Dataset |
|
|
| | Key | Value | |
| | :---------------------: | :-------: | |
| | Entries (words) | 753,232 | |
| | Tokens | 3,225,398 | |
| | Characters | 7,022,310 | |
| | Avg. Tokens Per Entry | ~4.2 | |
| | Avg. Words Per Entry | 1 | |
| | Avg. Chars Per Entry | ~9.3 | |
| | Longest Entry (Tokens) | 36 | |
| | Shortest Entry (Tokens) | 1 | |
| | English Words | ~660k | |
| | Spanish Words | ~90k | |
|
|
| ### Hardware |
|
|
| MicroWord trained on one NVIDIA RTX 2060 GPU for 6 epochs with a batch size of 32. |
|
|
| ### Training Results |
|
|
| | Epoch | Train Loss | Val Loss | Train PPL | Val PPL | |
| |-------|------------|----------|-----------|---------| |
| | 0.78 | 4.4464 | 4.3641 | 85.33 | 78.57 | |
| | 1.56 | 3.9422 | 3.8500 | 51.53 | 47.00 | |
| | 2.34 | 3.6247 | 3.5422 | 37.51 | 34.55 | |
| | 3.12 | 3.3900 | 3.3500 | 29.66 | 28.50 | |
| | 3.90 | 3.2822 | 3.2389 | 26.64 | 25.51 | |
| | 4.68 | 3.2115 | 3.1787 | 24.82 | 24.01 | |
| | 5.45 | 3.1607 | 3.1448 | 23.59 | 23.22 | |
| | 5.97 | 3.1623 | 3.1395 | 23.63 | 23.09 | |
|
|
| As you can see, the loss curve slows down quite a bit around epoch 4.68. This is normal and expected behavior. |
|
|
| ## Generations |
|
|
| ##### Generation1 |
|
|
| Prompt: `app` |
| Output: |
| ``` |
| appisalies |
| ``` |
|
|
| ##### Generation2 |
|
|
| Prompt: `b` |
| Output: |
| ``` |
| bcuntiber's |
| ``` |
|
|
| ##### Generation3: |
|
|
| Prompt: `wh` |
| Output: |
| ``` |
| agings's |
| ``` |
|
|
| All of the generated words are fabricated. This is expected, because the model does not have the neccesary parameters to memeorize specific words, like [MediumWord](https://huggingface.co/Harley-ml/MediumWord-559k) can. |
|
|
| ## Limitations |
|
|
| 1. It does not generate sentences, prose, code, or anything besides a single word-like sequence. |
| 2. It cannot reason or produce complex language. |
| 3. Generated words may not be real. The goal isn't real word generation but reflecting the lexicon and morphology of the English and Spanish languages through tiny language models. |
| 4. Output is non-deterministic. The same prompt can produce very different completions across runs. |
|
|
| # Inference |
|
|
| ```python |
| # ============================================================================= |
| # Inference |
| # ============================================================================= |
| |
| MODEL_DIR = "Harley-ml/microword-28k" # path |
| TOKENIZER_PATH = "Harley-ml/microword-28k" |
| |
| # --- Generation settings --- |
| PROMPT = "b" # prompt |
| MAX_NEW_TOKENS = 32 |
| TEMPERATURE = 1.2 |
| TOP_P = 0.95 |
| TOP_K = 50 |
| REPETITION_PENALTY = 1.1 |
| DO_SAMPLE = True |
| |
| # ============================================================================= |
| |
| import torch |
| from pathlib import Path |
| from transformers import ( |
| AutoModelForCausalLM, |
| PreTrainedTokenizerFast, |
| AddedToken, |
| ) |
| |
| # --------------------------------------------------------------------------- |
| # Device |
| # --------------------------------------------------------------------------- |
| |
| device = ( |
| "cuda" if torch.cuda.is_available() else |
| "mps" if torch.backends.mps.is_available() else |
| "cpu" |
| ) |
| print(f"Device : {device}") |
| |
| # --------------------------------------------------------------------------- |
| # Tokenizer (mirrors training setup) |
| # --------------------------------------------------------------------------- |
| |
| def load_tokenizer(path: str): |
| p = Path(path).resolve() |
| if not p.exists(): |
| raise FileNotFoundError(f"Tokenizer not found: {p}") |
| tok = PreTrainedTokenizerFast(tokenizer_file=str(p)) |
| specials = {} |
| if tok.bos_token is None: specials["bos_token"] = AddedToken("<|bos|>", special=True) |
| if tok.eos_token is None: specials["eos_token"] = AddedToken("<|eos|>", special=True) |
| if tok.unk_token is None: specials["unk_token"] = AddedToken("<|unk|>", special=True) |
| if tok.pad_token is None: |
| if tok.eos_token is not None: |
| tok.pad_token = tok.eos_token |
| else: |
| specials["pad_token"] = AddedToken("<|pad|>", special=True) |
| if specials: |
| tok.add_special_tokens(specials) |
| tok.padding_side = "left" # left-pad for batched generation |
| return tok |
| |
| print("Loading tokenizer...") |
| tokenizer = load_tokenizer(TOKENIZER_PATH) |
| print(f" Vocab size : {tokenizer.vocab_size}") |
| print(f" BOS : {tokenizer.bos_token!r}") |
| print(f" EOS : {tokenizer.eos_token!r}") |
| print(f" PAD : {tokenizer.pad_token!r} (id={tokenizer.pad_token_id})") |
| |
| # --------------------------------------------------------------------------- |
| # Model |
| # --------------------------------------------------------------------------- |
| |
| print(f"\nLoading model from {MODEL_DIR} ...") |
| model = AutoModelForCausalLM.from_pretrained( |
| MODEL_DIR, |
| dtype=torch.float16 if device == "cuda" else torch.float32, |
| low_cpu_mem_usage=True, |
| ) |
| model.eval() |
| model.to(device) |
| |
| total_params = sum(p.numel() for p in model.parameters()) |
| print(f" Parameters : {total_params:,}") |
| |
| # --------------------------------------------------------------------------- |
| # Generation helper |
| # --------------------------------------------------------------------------- |
| |
| def generate( |
| prompt: str = PROMPT, |
| max_new_tokens: int = MAX_NEW_TOKENS, |
| temperature: float = TEMPERATURE, |
| top_p: float = TOP_P, |
| top_k: int = TOP_K, |
| repetition_penalty: float = REPETITION_PENALTY, |
| do_sample: bool = DO_SAMPLE, |
| ) -> str: |
| |
| bos = tokenizer.bos_token or "" |
| full_prompt = bos + prompt |
| |
| inputs = tokenizer( |
| full_prompt, |
| return_tensors="pt", |
| add_special_tokens=False, |
| ).to(device) |
| inputs.pop("token_type_ids", None) # Qwen3 doesn't use this |
| |
| gen_kwargs = dict( |
| max_new_tokens = max_new_tokens, |
| do_sample = do_sample, |
| repetition_penalty = repetition_penalty, |
| eos_token_id = tokenizer.eos_token_id, |
| pad_token_id = tokenizer.pad_token_id, |
| ) |
| if do_sample: |
| gen_kwargs["temperature"] = temperature |
| gen_kwargs["top_p"] = top_p |
| gen_kwargs["top_k"] = top_k |
| |
| with torch.inference_mode(): |
| output_ids = model.generate(**inputs, **gen_kwargs) |
| |
| # Strip the prompt tokens so we only return what was generated |
| prompt_len = inputs["input_ids"].shape[-1] |
| new_ids = output_ids[0][prompt_len:] |
| return tokenizer.decode(new_ids, skip_special_tokens=True) |
| |
| |
| # --------------------------------------------------------------------------- |
| # Run |
| # --------------------------------------------------------------------------- |
| |
| if __name__ == "__main__": |
| print(f"\nPrompt : {PROMPT!r}") |
| print("-" * 60) |
| |
| output = generate(PROMPT) |
| |
| print("Generated:") |
| print(output) |
| ``` |
|
|
| ### Related Models |
|
|
| 1. [PicoWord](https://huggingface.co/Harley-ml/PicoWord-5k) |
| 2. [TinyWord](https://huggingface.co/Harley-ml/TinyWord-134k) |
| 3. [TinyWord2](https://huggingface.co/Harley-ml/TinyWord2-128k) |
| 4. [MediumWord](https://huggingface.co/Harley-ml/MediumWord-559k) |
| 5. [LargeWord](https://huggingface.co/Harley-ml/LargeWord-1.5M |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{microword-23k, |
| title = {MicroWord-23k: A Test of Morphological Compression in TLMs}, |
| author = {Harley-ml}, |
| year = {2026}, |
| url = {https://huggingface.co/Harley-ml/MicroWord-23k} |
| } |
| ``` |
