--- license: mit tags: - micro - tiny - tinyword - microword - word-generation - word - words - little - small - harley-ml - ml - llm - slm - small-langauge-model - tlm datasets: - Harley-ml/es-en-words --- # MicroWord You wouldn't believe me if I told you this, but we scaled down TinyWord. Meet MicroWord, which is smaller than Tiny, bigger than Pico, and somehow still trying its best. MicroWord is a twenty-three thousand parameter transformer trained on seven-hundred and fifty-three thousand words. Its goal is to generate plausible-looking words based on the morphology of the English and Spanish languages. ## Architecture | Parameter | Value | |------------------------|-------| | Hidden Layers | 1 | | Hidden Size | 16 | | Attention Heads | 1 | | KV Heads | 1 | | Intermediate Size | 64 | | RoPE Theta | 1000.0| | Max Position Embeddings| 32 | | Tie Word Embeddings | True | | Vocab Size | 1200 | Note: 1 attention head and a RoPE Theta of 1000 (vs Qwen3's 1,000,000) are intentional reductions for this scale. Max sequence length is 32, so positional generalization at range isn't a concern. ## Training ### Dataset | Key | Value | | :---------------------: | :-------: | | Entries (words) | 753,232 | | Tokens | 3,225,398 | | Characters | 7,022,310 | | Avg. Tokens Per Entry | ~4.2 | | Avg. Words Per Entry | 1 | | Avg. Chars Per Entry | ~9.3 | | Longest Entry (Tokens) | 36 | | Shortest Entry (Tokens) | 1 | | English Words | ~660k | | Spanish Words | ~90k | ### Hardware MicroWord trained on one NVIDIA RTX 2060 GPU for 6 epochs with a batch size of 32. ### Training Results | Epoch | Train Loss | Val Loss | Train PPL | Val PPL | |-------|------------|----------|-----------|---------| | 0.78 | 4.4464 | 4.3641 | 85.33 | 78.57 | | 1.56 | 3.9422 | 3.8500 | 51.53 | 47.00 | | 2.34 | 3.6247 | 3.5422 | 37.51 | 34.55 | | 3.12 | 3.3900 | 3.3500 | 29.66 | 28.50 | | 3.90 | 3.2822 | 3.2389 | 26.64 | 25.51 | | 4.68 | 3.2115 | 3.1787 | 24.82 | 24.01 | | 5.45 | 3.1607 | 3.1448 | 23.59 | 23.22 | | 5.97 | 3.1623 | 3.1395 | 23.63 | 23.09 | As you can see, the loss curve slows down quite a bit around epoch 4.68. This is normal and expected behavior. ## Generations ##### Generation1 Prompt: `app` Output: ``` appisalies ``` ##### Generation2 Prompt: `b` Output: ``` bcuntiber's ``` ##### Generation3: Prompt: `wh` Output: ``` agings's ``` All of the generated words are fabricated. This is expected, because the model does not have the neccesary parameters to memeorize specific words, like [MediumWord](https://huggingface.co/Harley-ml/MediumWord-559k) can. ## Limitations 1. It does not generate sentences, prose, code, or anything besides a single word-like sequence. 2. It cannot reason or produce complex language. 3. Generated words may not be real. The goal isn't real word generation but reflecting the lexicon and morphology of the English and Spanish languages through tiny language models. 4. Output is non-deterministic. The same prompt can produce very different completions across runs. # Inference ```python # ============================================================================= # Inference # ============================================================================= MODEL_DIR = "Harley-ml/microword-28k" # path TOKENIZER_PATH = "Harley-ml/microword-28k" # --- Generation settings --- PROMPT = "b" # prompt MAX_NEW_TOKENS = 32 TEMPERATURE = 1.2 TOP_P = 0.95 TOP_K = 50 REPETITION_PENALTY = 1.1 DO_SAMPLE = True # ============================================================================= import torch from pathlib import Path from transformers import ( AutoModelForCausalLM, PreTrainedTokenizerFast, AddedToken, ) # --------------------------------------------------------------------------- # Device # --------------------------------------------------------------------------- device = ( "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu" ) print(f"Device : {device}") # --------------------------------------------------------------------------- # Tokenizer (mirrors training setup) # --------------------------------------------------------------------------- def load_tokenizer(path: str): p = Path(path).resolve() if not p.exists(): raise FileNotFoundError(f"Tokenizer not found: {p}") tok = PreTrainedTokenizerFast(tokenizer_file=str(p)) specials = {} if tok.bos_token is None: specials["bos_token"] = AddedToken("<|bos|>", special=True) if tok.eos_token is None: specials["eos_token"] = AddedToken("<|eos|>", special=True) if tok.unk_token is None: specials["unk_token"] = AddedToken("<|unk|>", special=True) if tok.pad_token is None: if tok.eos_token is not None: tok.pad_token = tok.eos_token else: specials["pad_token"] = AddedToken("<|pad|>", special=True) if specials: tok.add_special_tokens(specials) tok.padding_side = "left" # left-pad for batched generation return tok print("Loading tokenizer...") tokenizer = load_tokenizer(TOKENIZER_PATH) print(f" Vocab size : {tokenizer.vocab_size}") print(f" BOS : {tokenizer.bos_token!r}") print(f" EOS : {tokenizer.eos_token!r}") print(f" PAD : {tokenizer.pad_token!r} (id={tokenizer.pad_token_id})") # --------------------------------------------------------------------------- # Model # --------------------------------------------------------------------------- print(f"\nLoading model from {MODEL_DIR} ...") model = AutoModelForCausalLM.from_pretrained( MODEL_DIR, dtype=torch.float16 if device == "cuda" else torch.float32, low_cpu_mem_usage=True, ) model.eval() model.to(device) total_params = sum(p.numel() for p in model.parameters()) print(f" Parameters : {total_params:,}") # --------------------------------------------------------------------------- # Generation helper # --------------------------------------------------------------------------- def generate( prompt: str = PROMPT, max_new_tokens: int = MAX_NEW_TOKENS, temperature: float = TEMPERATURE, top_p: float = TOP_P, top_k: int = TOP_K, repetition_penalty: float = REPETITION_PENALTY, do_sample: bool = DO_SAMPLE, ) -> str: bos = tokenizer.bos_token or "" full_prompt = bos + prompt inputs = tokenizer( full_prompt, return_tensors="pt", add_special_tokens=False, ).to(device) inputs.pop("token_type_ids", None) # Qwen3 doesn't use this gen_kwargs = dict( max_new_tokens = max_new_tokens, do_sample = do_sample, repetition_penalty = repetition_penalty, eos_token_id = tokenizer.eos_token_id, pad_token_id = tokenizer.pad_token_id, ) if do_sample: gen_kwargs["temperature"] = temperature gen_kwargs["top_p"] = top_p gen_kwargs["top_k"] = top_k with torch.inference_mode(): output_ids = model.generate(**inputs, **gen_kwargs) # Strip the prompt tokens so we only return what was generated prompt_len = inputs["input_ids"].shape[-1] new_ids = output_ids[0][prompt_len:] return tokenizer.decode(new_ids, skip_special_tokens=True) # --------------------------------------------------------------------------- # Run # --------------------------------------------------------------------------- if __name__ == "__main__": print(f"\nPrompt : {PROMPT!r}") print("-" * 60) output = generate(PROMPT) print("Generated:") print(output) ``` ### Related Models 1. [PicoWord](https://huggingface.co/Harley-ml/PicoWord-5k) 2. [TinyWord](https://huggingface.co/Harley-ml/TinyWord-134k) 3. [TinyWord2](https://huggingface.co/Harley-ml/TinyWord2-128k) 4. [MediumWord](https://huggingface.co/Harley-ml/MediumWord-559k) 5. [LargeWord](https://huggingface.co/Harley-ml/LargeWord-1.5M ## Citation ```bibtex @misc{microword-23k, title = {MicroWord-23k: A Test of Morphological Compression in TLMs}, author = {Harley-ml}, year = {2026}, url = {https://huggingface.co/Harley-ml/MicroWord-23k} } ```