nano-dates: 1M byte-level date->ISO parser (v3, honest data)

README.md

@@ -0,0 +1,124 @@
+---
+license: mit
+language:
+  - en
+tags:
+  - tiny
+  - nano-llm
+  - dates
+  - structured-output
+  - byte-level
+  - from-scratch
+pipeline_tag: text2text-generation
+library_name: safetensors
+---
+
+# nano-dates
+
+A **1,016,960-parameter** (~1M) byte-level transformer that converts a natural date
+phrase to an **ISO-8601** date. It is small enough to run on a CPU in milliseconds
+and was trained **entirely on code-generated data** — no scraping, no labelling, no
+distillation from a larger model.
+
+```
+2024-03-10 | the 3rd of July 2025 => 2025-07-03
+2024-03-10 | Jun 12 2023          => 2023-06-12
+2024-03-10 | next week            => 2024-03-17
+2024-03-10 | in 3 months          => 2024-06-10
+```
+
+The model is given a **reference date** (`today`) at the start of the prompt, so
+relative phrases ("tomorrow", "in 3 weeks") are computable from the input alone —
+it never needs a wall clock.
+
+## Why this exists
+
+A 1M-parameter model can't be a general assistant, but it *can* completely nail a
+task that is **narrow and formally specified**. Date→ISO is exactly that: the
+answer has a known structure, so you can **sample the answer first and render it in
+many natural forms**, producing perfectly-labelled training data for free and in
+unlimited quantity. That is strictly better than asking a big model to generate
+data — the label *is* the ground truth, not a guess.
+
+This model is a worked demonstration of that recipe (and an honest map of where a
+nano model's capability ends — see below).
+
+## What it can and can't do
+
+Held-out exact-match accuracy (2,000 unseen examples, greedy decode):
+
+| capability | category | accuracy |
+|---|---|---|
+| **Parse absolute dates** | `2023-06-12`, `June 12, 2023`, `Jun 12 2023`, `12 June 2023`, `the 12th of June 2023` | **100%** |
+| **Resolve simple relatives** | today, tomorrow, yesterday, next/last week, next month, in N months | **98–100%** |
+| **Variable-N day/week arithmetic** | in N days, N days ago, in N weeks | **77–81%** |
+| **Weekday resolution** | next/last \<weekday\> | **~12%** ❌ |
+| **Overall** | mixed | **85.4%** |
+
+The clean limitation: **weekday resolution** ("next friday") is unsolved at this
+size. It requires mapping an arbitrary date to its weekday and then doing modular
+arithmetic — the hardest computation in the set — and a 1M model doesn't get there.
+Everything else, including the absolute-form parsing and most relative arithmetic,
+it does reliably. The accuracy numbers reflect **genuine parsing**: absolute phrases
+are trained with a reference date *independent* of the answer, so the model cannot
+cheat by copying the prompt.
+
+## Usage
+
+The repo includes a **self-contained** model definition (`modeling_nano_dates.py`)
+— no training framework required, just `torch` and `safetensors`.
+
+```python
+from modeling_nano_dates import load, parse
+
+model = load("model.safetensors", "config.json")
+print(parse(model, "2024-03-10", "the 3rd of July 2025"))  # -> 2025-07-03
+print(parse(model, "2024-03-10", "in 3 weeks"))            # -> 2024-03-31
+```
+
+Or just run the file for a demo: `python modeling_nano_dates.py`.
+
+Prompt format the model was trained on (byte-for-byte):
+
+```
+<reference ISO date> | <phrase> => <answer ISO date>
+```
+
+`parse()` builds that prompt and greedily decodes exactly 10 characters.
+
+## Model details
+
+| | |
+|---|---|
+| Parameters | 1,016,960 |
+| Architecture | decoder-only transformer (pre-norm) |
+| Tokenizer | raw UTF-8 bytes (vocab 256, no vocab file) |
+| dim / layers / heads | 128 / 4 / 4 (2 KV heads, GQA) |
+| Norm / position / FFN | RMSNorm / RoPE / SwiGLU |
+| Context | 64 bytes |
+| Training | SFT, prompt-masked cross-entropy, 12k steps, AdamW, cosine LR 3e-3 |
+| Data | 100k code-generated pairs, 17 surface renderers |
+| Final val loss | 0.036 |
+
+## Limitations & honest scope
+
+- **Not a production date library.** For real software, `dateutil`/`chrono` are
+  exact and free. This model's value is as a *method demonstration* and a study of
+  what a nano model can learn from synthetic data, not as a dependency.
+- **Weekday phrases ("next friday") are unreliable** (~12%). Don't use them.
+- **English only**, and only the 17 surface forms it was trained on. It has not
+  seen "12/06/2023"-style numeric forms (deliberately — they're ambiguous).
+- Reference dates were drawn from **2015–2035**; far outside that, behaviour is
+  untested.
+
+## What should this method point at next?
+
+The interesting question isn't this model — it's the *recipe*. If you have a
+**narrow, formal, annoying task** you wish a tiny reliable model could do (parse,
+normalize, validate, convert), that's exactly the shape this approach fits. Tell me
+what it is — open a discussion on this repo.
+
+---
+
+*Built from scratch with [voidlab](https://github.com/vukrosic). Trained on a single
+GPU in ~30 seconds.*

config.json

@@ -0,0 +1,33 @@
+{
+  "model_type": "nano-dates",
+  "architecture": "decoder-only transformer (pre-norm)",
+  "vocab_size": 256,
+  "tokenizer": "byte (raw UTF-8 bytes, no vocab file)",
+  "dim": 128,
+  "n_layers": 4,
+  "n_heads": 4,
+  "n_kv_heads": 2,
+  "head_dim": 32,
+  "ffn": "swiglu",
+  "ffn_mult": 4,
+  "norm": "rmsnorm",
+  "norm_eps": 1e-05,
+  "positional": "rope",
+  "rope_theta": 10000.0,
+  "max_seq_len": 64,
+  "tie_word_embeddings": true,
+  "params": 1016960,
+  "training": {
+    "task": "natural date phrase -> ISO 8601",
+    "data": "100% code-generated (answer sampled first, then rendered in 17 natural forms)",
+    "objective": "SFT, prompt-masked cross-entropy (only the 10-char ISO target is supervised)",
+    "steps": 12000,
+    "batch_size": 64,
+    "seq_len": 64,
+    "optimizer": "adamw",
+    "lr": 0.003,
+    "schedule": "cosine",
+    "warmup_steps": 200,
+    "final_val_loss": 0.0358
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b59fd7f9cd229a302a748268bc96f8a04af580a38e6b808a4aa45fbfd0934bec
+size 4071464

modeling_nano_dates.py

@@ -0,0 +1,165 @@
+"""Self-contained nano-dates model — no dependencies beyond torch + safetensors.
+
+A 1M-parameter byte-level decoder-only transformer (RMSNorm, RoPE, GQA, SwiGLU)
+that converts a natural date phrase to an ISO-8601 date. This single file vendors
+the exact architecture the model was trained with, so you can load and run the
+published weights without installing the training lab.
+
+    python modeling_nano_dates.py            # runs a few examples
+    # or, from your own code:
+    from modeling_nano_dates import load, parse
+    model = load("model.safetensors", "config.json")
+    print(parse(model, "2024-03-10", "the 3rd of July 2025"))  # -> 2025-07-03
+
+Prompt format the model was trained on (byte-for-byte):
+
+    <today ISO> | <phrase> => <answer ISO>
+
+`today` is given so relative phrases ("tomorrow", "next week") are computable from
+the input alone — the model never needs a wall clock. `parse()` builds the prompt
+and greedily decodes exactly 10 characters.
+"""
+
+from __future__ import annotations
+
+import json
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, dim: int, eps: float = 1e-5):
+        super().__init__()
+        self.eps = eps
+        self.weight = nn.Parameter(torch.ones(dim))
+
+    def forward(self, x):
+        rms = x.float().pow(2).mean(-1, keepdim=True).add(self.eps).rsqrt()
+        return (x.float() * rms).type_as(x) * self.weight
+
+
+class RoPE(nn.Module):
+    def __init__(self, head_dim: int, max_seq_len: int, theta: float = 10000.0):
+        super().__init__()
+        inv_freq = 1.0 / (theta ** (torch.arange(0, head_dim, 2).float() / head_dim))
+        freqs = torch.outer(torch.arange(max_seq_len).float(), inv_freq)
+        self.register_buffer("cos", freqs.cos(), persistent=False)
+        self.register_buffer("sin", freqs.sin(), persistent=False)
+
+    def apply(self, x, offset: int = 0):
+        seq = x.size(-2)
+        cos = self.cos[offset:offset + seq]
+        sin = self.sin[offset:offset + seq]
+        x1, x2 = x[..., 0::2], x[..., 1::2]
+        rot1 = x1 * cos - x2 * sin
+        rot2 = x1 * sin + x2 * cos
+        return torch.stack((rot1, rot2), dim=-1).flatten(-2).type_as(x)
+
+
+class GQA(nn.Module):
+    def __init__(self, dim, n_heads, n_kv_heads, head_dim, positional):
+        super().__init__()
+        self.n_heads, self.n_kv_heads, self.head_dim = n_heads, n_kv_heads, head_dim
+        self.n_rep = n_heads // n_kv_heads
+        self.positional = positional
+        self.q_proj = nn.Linear(dim, n_heads * head_dim, bias=False)
+        self.k_proj = nn.Linear(dim, n_kv_heads * head_dim, bias=False)
+        self.v_proj = nn.Linear(dim, n_kv_heads * head_dim, bias=False)
+        self.o_proj = nn.Linear(n_heads * head_dim, dim, bias=False)
+
+    def forward(self, x, mask):
+        b, seq, _ = x.shape
+        q = self.q_proj(x).view(b, seq, self.n_heads, self.head_dim).transpose(1, 2)
+        k = self.k_proj(x).view(b, seq, self.n_kv_heads, self.head_dim).transpose(1, 2)
+        v = self.v_proj(x).view(b, seq, self.n_kv_heads, self.head_dim).transpose(1, 2)
+        q = self.positional.apply(q)
+        k = self.positional.apply(k)
+        if self.n_rep > 1:
+            k = k.repeat_interleave(self.n_rep, dim=1)
+            v = v.repeat_interleave(self.n_rep, dim=1)
+        scores = (q @ k.transpose(-2, -1)) / (self.head_dim ** 0.5)
+        if mask is not None:
+            scores = scores + mask
+        out = F.softmax(scores, dim=-1) @ v
+        out = out.transpose(1, 2).reshape(b, seq, self.n_heads * self.head_dim)
+        return self.o_proj(out)
+
+
+class SwiGLU(nn.Module):
+    def __init__(self, dim: int, hidden: int):
+        super().__init__()
+        self.gate = nn.Linear(dim, hidden, bias=False)
+        self.up = nn.Linear(dim, hidden, bias=False)
+        self.down = nn.Linear(hidden, dim, bias=False)
+
+    def forward(self, x):
+        return self.down(F.silu(self.gate(x)) * self.up(x))
+
+
+class Block(nn.Module):
+    def __init__(self, cfg, positional):
+        super().__init__()
+        hidden = int(cfg["dim"] * cfg["ffn_mult"])
+        self.attn_norm = RMSNorm(cfg["dim"], cfg["norm_eps"])
+        self.attn = GQA(cfg["dim"], cfg["n_heads"], cfg["n_kv_heads"], cfg["head_dim"], positional)
+        self.ffn_norm = RMSNorm(cfg["dim"], cfg["norm_eps"])
+        self.ffn = SwiGLU(cfg["dim"], hidden)
+
+    def forward(self, x, mask):
+        x = x + self.attn(self.attn_norm(x), mask)
+        x = x + self.ffn(self.ffn_norm(x))
+        return x
+
+
+class NanoDates(nn.Module):
+    def __init__(self, cfg: dict):
+        super().__init__()
+        self.cfg = cfg
+        self.tok_emb = nn.Embedding(cfg["vocab_size"], cfg["dim"])
+        self.positional = RoPE(cfg["head_dim"], cfg["max_seq_len"], cfg["rope_theta"])
+        self.blocks = nn.ModuleList([Block(cfg, self.positional) for _ in range(cfg["n_layers"])])
+        self.final_norm = RMSNorm(cfg["dim"], cfg["norm_eps"])
+        self.lm_head = nn.Linear(cfg["dim"], cfg["vocab_size"], bias=False)
+        self.lm_head.weight = self.tok_emb.weight  # tied
+
+    def forward(self, tokens):
+        seq = tokens.size(1)
+        x = self.tok_emb(tokens)
+        mask = torch.triu(torch.full((seq, seq), float("-inf"), device=tokens.device), diagonal=1)
+        for block in self.blocks:
+            x = block(x, mask)
+        return self.lm_head(self.final_norm(x))
+
+
+def load(weights="model.safetensors", config="config.json", device="cpu"):
+    from safetensors.torch import load_file
+    with open(config) as f:
+        cfg = json.load(f)
+    model = NanoDates(cfg).to(device)
+    sd = load_file(weights)
+    sd["lm_head.weight"] = sd["tok_emb.weight"]  # restore tied weight
+    model.load_state_dict(sd)
+    model.eval()
+    return model
+
+
+@torch.no_grad()
+def parse(model, today_iso: str, phrase: str, device="cpu") -> str:
+    """`today_iso` like '2024-03-10', `phrase` like 'next friday' -> 10-char ISO."""
+    prompt = f"{today_iso} | {phrase} => "
+    toks = torch.tensor([list(prompt.encode("utf-8"))], dtype=torch.long, device=device)
+    max_seq = model.cfg["max_seq_len"]
+    for _ in range(10):
+        nxt = model(toks[:, -max_seq:])[:, -1, :].argmax(-1, keepdim=True)
+        toks = torch.cat([toks, nxt], dim=1)
+    return bytes(int(b) & 0xFF for b in toks[0, -10:].tolist()).decode("utf-8", "replace")
+
+
+if __name__ == "__main__":
+    m = load()
+    today = "2024-03-10"
+    for phrase in ["the 3rd of July 2025", "Jun 12 2023", "tomorrow", "yesterday",
+                   "next week", "last week", "next month", "in 3 months"]:
+        print(f"{today} | {phrase:<22} -> {parse(m, today, phrase)}")