| ---
|
| library_name: transformers
|
| base_model: Qwen/Qwen3-0.6B
|
| tags:
|
| - qwen3
|
| - causal-lm
|
| - tiny-language-model
|
| - novelty-gated-attention
|
| - trust-remote-code
|
| ---
|
|
|
| # tinyLM-8M-exp
|
|
|
| Tiny 8M-class Qwen3-config causal LM with math-only novelty-gated GQA.
|
|
|
| ## Architecture
|
|
|
| | Item | Value |
|
| | --- | ---: |
|
| | Config type | `tinyqwen3_novelty` |
|
| | Parameters | 8.132M |
|
| | Layers | 8 |
|
| | Hidden size | 256 |
|
| | MLP size | 896 |
|
| | Query heads | 8 |
|
| | KV heads | 4 |
|
| | Head dim | 32 |
|
| | RoPE theta | 2500 |
|
| | Tied embeddings | yes |
|
|
|
| | Attention | Value |
|
| | --- | --- |
|
| | Type | GQA |
|
| | Novelty gate | math-only element-wise RMS-normalized abs-delta |
|
| | Gate floor | 0.05 |
|
|
|
| ## Training
|
|
|
| | Item | Value |
|
| | --- | --- |
|
| | Tokenizer | `AxiomicLabs/GPT-S2-5M` |
|
| | Sequence length | 512 |
|
| | Microbatch size | 1024 |
|
| | Gradient accumulation | 4 |
|
| | Effective batch size | 4096 |
|
| | Steps | 10,000 |
|
| | Validation cadence | every 1,000 steps |
|
| | Official lm-eval | after final Hub upload on ARC-Easy, ARC-Challenge, PIQA, HellaSwag |
|
| | LR schedule | warmup, cosine to min by 10,000 |
|
| | Optimizer | Muon for middle 2D weights, AdamW for the rest |
|
| | Special-token policy | BOS/EOS are document-level; `<|im_start|>`/`<|im_end|>` are sequence-level |
|
|
|
| | Dataset | Share | Config |
|
| | --- | ---: | --- |
|
| | `HuggingFaceFW/fineweb-edu` | 60.0% | `sample-100BT` |
|
| | `HuggingFaceTB/smollm-corpus` | 30.0% | `cosmopedia-v2` only |
|
| | `epfml/FineWeb-HQ` | 10.0% | `default` |
|
|
|
| ## Validation
|
|
|
| | Metric | Value |
|
| | --- | ---: |
|
| | Dataset | `Salesforce/wikitext`, `wikitext-103-raw-v1`, validation |
|
| | Context / stride | 512 / 256 |
|
| | Loss | 3.2769 |
|
| | Perplexity | 26.49 |
|
| | UTF-8 BPB | 1.4992 |
|
| | Scored tokens | 365,258 |
|
| | UTF-8 bytes | 1,151,766 |
|
|
|
| ## Evaluation
|
|
|
| Scores were run after Hub upload against revision
|
| `d95a00a6edafab4bc2d6b60a28e6893b00f52699`.
|
| ARC-Easy, ARC-Challenge, PIQA, and HellaSwag use official `lm_eval`
|
| 0-shot log-likelihood scoring. ArithMark-2.0 uses the same continuation
|
| NLL scoring style with a custom scorer because it is not available in
|
| `lm_eval`.
|
|
|
| | Task | n | acc | acc stderr | acc_norm | acc_norm stderr |
|
| | --- | ---: | ---: | ---: | ---: | ---: |
|
| | ARC-Easy | 2,376 | 37.04% | 0.99% | 35.86% | 0.98% |
|
| | ARC-Challenge | 1,172 | 18.77% | 1.14% | 22.87% | 1.23% |
|
| | PIQA | 1,838 | 57.67% | 1.15% | 57.89% | 1.15% |
|
| | HellaSwag | 10,042 | 26.88% | 0.44% | 27.88% | 0.45% |
|
| | ArithMark-2.0 | 2,500 | 25.12% | 0.87% | 24.44% | 0.86% |
|
|
|
| ## Load And Generate
|
|
|
| ```python
|
| from transformers import AutoModelForCausalLM, AutoTokenizer
|
| import torch
|
|
|
| repo = "User01110/tinyLM-8M-exp"
|
| tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
|
| model = AutoModelForCausalLM.from_pretrained(
|
| repo,
|
| trust_remote_code=True,
|
| torch_dtype="auto",
|
| device_map="auto",
|
| )
|
|
|
| prompt = "The future of AI is"
|
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
| print(inputs.input_ids[0][:2].tolist()) # auto-prefix: [<|im_start|>, <bos>]
|
|
|
| with torch.no_grad():
|
| output = model.generate(
|
| **inputs,
|
| max_new_tokens=512,
|
| do_sample=True,
|
| temperature=0.65,
|
| top_k=30,
|
| repetition_penalty=1.2,
|
| pad_token_id=tokenizer.pad_token_id,
|
| eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
|
| )
|
|
|
| print(tokenizer.decode(output[0], skip_special_tokens=True))
|
| ```
|
|
|
| This repo uses a self-contained remote `TinyQwen3NoveltyConfig` plus model code for a Qwen3-style dense decoder with a math-only novelty-gated attention block.
|
|
|
