Quark-50M-v2: 43.8M Italian-first LM (loss 0.9409551193519028)

README.md

@@ -1,124 +1,141 @@
 ---
 language:
+- it
 - en
-- code
 license: apache-2.0
 tags:
-- smol
-- pretraining
-- instruct
-- 50M
+- italian
 - causal-lm
-- gqa
-- swiglu
-- rmsnorm
-datasets:
-- HuggingFaceTB/smollm-corpus
-metrics:
-- perplexity
-model-index:
-- name: Quark-50m-Instruct
-  results: []
+- small-language-model
+- trained-from-scratch
+- chatml
+- conversational
 pipeline_tag: text-generation
 ---
 
-# Quark-50m-Instruct
+# ⚛ Quark-50M-v2
 
-**Quark-50m-Instruct** is a small (≈56M parameters) decoder-only language model, fine-tuned for instruction following.
-It is built on the same architecture of “SmolLM” family and was fully pretrained on 5 billion tokens from
-[HuggingFaceTB/smollm‑corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus).
+**43.8M parameter Italian-first bilingual language model, trained from scratch by ThingAI.**
 
-- **Model type:** Causal Language Model (LLaMA‑style decoder)
-- **Architecture:** GQA · SwiGLU · RMSNorm · RoPE · Weight‑tying
-- **Pretraining tokens:** 5 B
-- **Fine‑tuning:** Instruction‑tuned (details below)
-- **Creators:** [OvercastLab](https://huggingface.co/OvercastLab) (research & development lab for ML/AI)
-- **Release date:** 22 April 2026
+Quark-50M is an ultra-compact causal language model that speaks fluent Italian. Designed as a proof-of-concept for small, efficient, Italian-centric AI.
 
-## Model Summary
+## Highlights
 
-Quark-50m-Instruct is designed to be an efficient assistant that can run on consumer GPUs (e.g., RTX 3070 with 8 GB VRAM)
-and even on CPU for light workloads. It is **not** competitive with large models on knowledge‑intensive tasks,
-but it excels at:
+- **43.8M parameters** — runs on any device, even CPU
+- **Italian-first** — trained on 60% Italian data (books, Wikipedia, web)
+- **ChatML format** — `<|im_start|>user`/`<|im_start|>assistant`
+- **Custom tokenizer** — 16k BPE, optimized for Italian (4.15 chars/token)
+- **Trained from scratch** — architecture, tokenizer, and training pipeline all custom
 
-- Simple conversational tasks
-- Code generation and explanation (Python)
-- Short text rewriting and summarisation
-- On‑device / edge inference
+## Architecture
 
-The architecture closely follows the efficient‑small‑LM blueprint popularised by SmolLM:
+| Component | Value |
+|-----------|-------|
+| Parameters | 43.8M |
+| Vocabulary | 16,384 (BPE) |
+| Dimensions | 512 |
+| Layers | 12 |
+| Heads | 8 (4 KV heads, GQA) |
+| FFN | 1,408 (SwiGLU) |
+| Context | 2,048 tokens |
+| Normalization | RMSNorm |
+| Position | RoPE |
+| Weight Tying | Yes |
 
-| Component   | Details                       |
-|-------------|-------------------------------|
-| Vocab size  | 49,152                        |
-| Hidden size | 384                           |
-| Layers      | 24                            |
-| Attention   | Grouped Query (6 Q heads, 2 KV heads) |
-| FFN         | SwiGLU with 1,024 intermediate |
-| Position    | RoPE (θ = 10,000)             |
-| Normalisation | RMSNorm (pre‑block)         |
+## Training
 
-Total trainable parameters: **≈48 M** (with weight tying).
+**Pretraining:** 5B tokens on a curated mix:
 
-### Benchmark Evaluation Metrics
+| Dataset | Weight | Type |
+|---------|--------|------|
+| PleIAs/Italian-PD | 25% | 171K Italian books (public domain) |
+| FineWeb-2 Italian | 20% | Cleaned, deduplicated web |
+| Wikipedia IT | 15% | Encyclopedia |
+| Cosmopedia | 15% | Synthetic educational |
+| SmolLM-Corpus | 10% | Curated mix |
+| StarCoder Python | 8% | Code |
+| OpenWebMath | 7% | Mathematics |
 
-| Category | Benchmark | Metric | Score / Value | Status |
-| :--- | :--- | :--- | :---: | :---: |
-| **Linguistics & Grammar** | BLiMP | Accuracy | 68.12% | Success |
-| **Commonsense & Reasoning** | PIQA | Normalized Accuracy | 57.83% | Success |
-| | COPA | Accuracy | 57.00% | Success |
-| | BoolQ | Accuracy | 52.17% | Success |
-| | WinoGrande | Accuracy | 47.36% | Success |
-| | HellaSwag | Normalized Accuracy | 28.49% | Success |
-| | RACE | Accuracy | 26.41% | Success |
-| | CommonsenseQA | Accuracy | 20.31% | Success |
-| **Academic & Knowledge** | SciQ | Normalized Accuracy | 49.00% | Success |
-| | ARC-Easy | Normalized Accuracy | 36.49% | Success |
-| | MMLU | Accuracy | 25.64% | Success |
-| | ARC-Challenge | Normalized Accuracy | 25.17% | Success |
-| | OpenBookQA | Normalized Accuracy | 25.40% | Success |
-| **Language Modeling** | LAMBADA | Accuracy | 15.87% | Success |
-| | WikiText-2 | Word Perplexity | 251.76 | Success |
+**SFT:** Fine-tuned on [quattro-chiacchiere](https://huggingface.co/datasets/ThingAI/quattro-chiacchiere), a synthetic Italian Q&A dataset generated with [Alembic](https://github.com/skein-labs/Alembic).
 
-*Note: The Arithmetic benchmark failed due to outdated script support (`arithmetic.py`), and SocialIQA failed due to a registration tag error (`siqa`). Total baseline execution completed successfully for all other 15 tasks.*
-
-
-## Uses
-
-### Direct Use
-The model can be used via the 🤗 Transformers library for standard text generation.
-It expects chat‑formatted input (see example below).
-
-### Downstream Use
-Because of the open Apache‑2.0 license, you may fine‑tune Quark-50m‑Instruct on your own data for
-domain‑specific tasks – for instance, a customer‑support bot, a code reviewer, or a story writer.
-
-### Limitations
-- Limited world knowledge (stopped at mid‑2025 pretraining data).
-- Short context window (2,048 tokens).
-- Small size means it can make more factual mistakes than larger models.
-
-## How to Get Started
+## Usage
 
 ```python
-from transformers import AutoTokenizer, AutoModelForCausalLM
-
-model_name = "ThingAI/Quark-50m-Instruct"
-
-tokenizer = AutoTokenizer.from_pretrained(model_name)
-model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
-
-messages = [
-    {"role": "system", "content": "You are Quark, a helpful assistant."},
-    {"role": "user", "content": "Explain group query attention in one sentence."}
-]
-
-inputs = tokenizer.apply_chat_template(
-    messages,
-    tokenize=True,
-    add_generation_prompt=True,
-    return_tensors="pt"
-).to(model.device)
-
-outputs = model.generate(inputs, max_new_tokens=128)
-print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+import torch
+from huggingface_hub import hf_hub_download
+from transformers import PreTrainedTokenizerFast
+
+# Load
+ckpt_path = hf_hub_download("ThingAI/Quark-50M", "model.pt")
+model_py = hf_hub_download("ThingAI/Quark-50M", "model.py")
+tok_file = hf_hub_download("ThingAI/Quark-50M", "tokenizer.json")
+
+# Tokenizer
+tokenizer = PreTrainedTokenizerFast(tokenizer_file=tok_file)
+tokenizer.eos_token = "<|endoftext|>"
+
+# Model
+import importlib.util
+spec = importlib.util.spec_from_file_location("model", model_py)
+mod = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(mod)
+
+ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)
+cfg = mod.ModelConfig(**ckpt["model_cfg"])
+model = mod.Quark(cfg).eval()
+model.load_state_dict(ckpt["model"])
+
+# Chat
+prompt = "<|im_start|>user\nQual è la capitale d'Italia?<|im_end|>\n<|im_start|>assistant\n"
+ids = tokenizer.encode(prompt, return_tensors="pt")
+with torch.no_grad():
+    for _ in range(100):
+        logits = model(ids)[1][:, -1, :].float()
+        nxt = logits.argmax(-1, keepdim=True)
+        if nxt.item() == tokenizer.convert_tokens_to_ids("<|im_end|>"): break
+        ids = torch.cat([ids, nxt], -1)
+print(tokenizer.decode(ids[0], skip_special_tokens=True))
+# La capitale d'Italia è Roma.
+```
+
+## Examples
+
+```
+Tu: Qual è la capitale d'Italia?
+Quark: La capitale d'Italia è Roma.
+
+Tu: Chi sei?
+Quark: Sono Quark, piacere di conoscerti.
+
+Tu: Come ti chiami?
+Quark: Mi chiamo Quark, piacere di conoscerti.
+```
+
+## Limitations
+
+- **43.8M parameters** — cannot perform complex reasoning or long-form generation
+- **Factual accuracy** — may hallucinate facts, especially on niche topics
+- **SFT dataset** — currently limited; more data will improve reliability
+- **No safety training** — not recommended for production without guardrails
+
+## Related
+
+- [Quark3Tokenizer](https://huggingface.co/ThingAI/Quark3Tokenizer) — the tokenizer
+- [quattro-chiacchiere](https://huggingface.co/datasets/ThingAI/quattro-chiacchiere) — the SFT dataset
+- [Alembic](https://github.com/skein-labs/Alembic) — the dataset distillation tool
+- [Glyph](https://huggingface.co/ThingAI/Glyph) — multi-task text classifier by ThingAI
+
+## Citation
+
+```bibtex
+@misc{quark50m,
+  author = {ThingAI},
+  title  = {Quark-50M-v2: Italian-First Small Language Model},
+  year   = {2026},
+  url    = {https://huggingface.co/ThingAI/Quark-50M}
+}
+```
+
+## License
+
+Apache 2.0

model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b7c2bae538e8df09945d4feb450fc6784194f243a6855408a2b5276773657ec
+size 175196411

model.py

@@ -0,0 +1,90 @@
+"""Quark-50M model definition — standalone."""
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from dataclasses import dataclass
+
+@dataclass
+class ModelConfig:
+    vocab_size: int = 16384; d_model: int = 512; n_heads: int = 8
+    n_kv_heads: int = 4; n_layers: int = 12; d_ff: int = 1408
+    head_dim: int = 64; max_seq_len: int = 2048; rope_theta: float = 10000.0
+    rms_eps: float = 1e-5; qkv_bias: bool = False; dropout: float = 0.0
+
+class RMSNorm(nn.Module):
+    def __init__(self, dim, eps=1e-5):
+        super().__init__(); self.eps = eps; self.scale = nn.Parameter(torch.ones(dim))
+    def forward(self, x):
+        return (x.float() * x.float().pow(2).mean(-1, keepdim=True).add(self.eps).rsqrt()).to(x.dtype) * self.scale
+
+class RotaryEmbedding(nn.Module):
+    def __init__(self, head_dim, max_seq_len, theta=10000.0):
+        super().__init__()
+        inv_freq = 1.0 / (theta ** (torch.arange(0, head_dim, 2).float() / head_dim))
+        self.register_buffer("inv_freq", inv_freq, persistent=False); self._build(max_seq_len)
+    def _build(self, seq_len):
+        t = torch.arange(seq_len, device=self.inv_freq.device).float()
+        freqs = torch.outer(t, self.inv_freq); emb = torch.cat([freqs, freqs], dim=-1)
+        self.register_buffer("cos_cache", emb.cos()[None, None], persistent=False)
+        self.register_buffer("sin_cache", emb.sin()[None, None], persistent=False); self._max = seq_len
+    @staticmethod
+    def _rot(x):
+        x1, x2 = x.chunk(2, dim=-1); return torch.cat([-x2, x1], dim=-1)
+    def forward(self, q, k):
+        T = q.size(2)
+        if T > self._max: self._build(T)
+        c, s = self.cos_cache[:,:,:T], self.sin_cache[:,:,:T]
+        return q*c + self._rot(q)*s, k*c + self._rot(k)*s
+
+class GQA(nn.Module):
+    def __init__(self, cfg):
+        super().__init__()
+        self.n_heads, self.n_kv_heads = cfg.n_heads, cfg.n_kv_heads
+        self.n_groups, self.head_dim = cfg.n_heads // cfg.n_kv_heads, cfg.head_dim
+        self.q_proj = nn.Linear(cfg.d_model, cfg.n_heads * cfg.head_dim, bias=cfg.qkv_bias)
+        self.k_proj = nn.Linear(cfg.d_model, cfg.n_kv_heads * cfg.head_dim, bias=cfg.qkv_bias)
+        self.v_proj = nn.Linear(cfg.d_model, cfg.n_kv_heads * cfg.head_dim, bias=cfg.qkv_bias)
+        self.o_proj = nn.Linear(cfg.n_heads * cfg.head_dim, cfg.d_model, bias=False)
+        self.rope = RotaryEmbedding(cfg.head_dim, cfg.max_seq_len, cfg.rope_theta)
+    def forward(self, x):
+        B, T, _ = x.shape
+        q = self.q_proj(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2)
+        k = self.k_proj(x).view(B, T, self.n_kv_heads, self.head_dim).transpose(1, 2)
+        v = self.v_proj(x).view(B, T, self.n_kv_heads, self.head_dim).transpose(1, 2)
+        q, k = self.rope(q, k)
+        if self.n_groups > 1:
+            B_r, _, T_r, D_r = k.shape
+            k = k[:,:,None,:,:].expand(B_r, self.n_kv_heads, self.n_groups, T_r, D_r).reshape(B_r, self.n_heads, T_r, D_r)
+            v = v[:,:,None,:,:].expand(B_r, self.n_kv_heads, self.n_groups, T_r, D_r).reshape(B_r, self.n_heads, T_r, D_r)
+        return self.o_proj(F.scaled_dot_product_attention(q, k, v, is_causal=True).transpose(1, 2).contiguous().view(B, T, -1))
+
+class Block(nn.Module):
+    def __init__(self, cfg):
+        super().__init__()
+        self.norm_attn = RMSNorm(cfg.d_model, cfg.rms_eps); self.attn = GQA(cfg)
+        self.norm_ffn = RMSNorm(cfg.d_model, cfg.rms_eps)
+        self.gate = nn.Linear(cfg.d_model, cfg.d_ff, bias=False)
+        self.up = nn.Linear(cfg.d_model, cfg.d_ff, bias=False)
+        self.down = nn.Linear(cfg.d_ff, cfg.d_model, bias=False)
+    def forward(self, x):
+        x = x + self.attn(self.norm_attn(x))
+        h = self.norm_ffn(x); x = x + self.down(F.silu(self.gate(h)) * self.up(h))
+        return x
+
+class Quark(nn.Module):
+    def __init__(self, cfg):
+        super().__init__(); self.cfg = cfg
+        self.embed_tokens = nn.Embedding(cfg.vocab_size, cfg.d_model)
+        self.layers = nn.ModuleList([Block(cfg) for _ in range(cfg.n_layers)])
+        self.norm = RMSNorm(cfg.d_model, cfg.rms_eps)
+        self.lm_head = nn.Linear(cfg.d_model, cfg.vocab_size, bias=False)
+        self.lm_head.weight = self.embed_tokens.weight
+    def forward(self, ids, labels=None):
+        x = self.embed_tokens(ids)
+        for layer in self.layers: x = layer(x)
+        logits = self.lm_head(self.norm(x))
+        loss = None
+        if labels is not None:
+            loss = F.cross_entropy(logits.view(-1, self.cfg.vocab_size), labels.view(-1), ignore_index=-100)
+        return loss, logits

special_tokens_map.json

@@ -0,0 +1,8 @@
+{
+  "bos_token": "<|im_start|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|pad|>",
+  "additional_special_tokens": [
+    "<|im_end|>"
+  ]
+}

tokenizer_config.json

@@ -1,34 +1,10 @@
 {
-  "add_prefix_space": false,
-  "backend": "tokenizers",
-  "bos_token": "<|endoftext|>",
-  "clean_up_tokenization_spaces": false,
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "bos_token": "<|im_start|>",
   "eos_token": "<|endoftext|>",
-  "errors": "replace",
-  "extra_special_tokens": [
-    "<|endoftext|>",
-    "<|im_start|>",
-    "<|im_end|>",
-    "<repo_name>",
-    "<reponame>",
-    "<file_sep>",
-    "<filename>",
-    "<gh_stars>",
-    "<issue_start>",
-    "<issue_comment>",
-    "<issue_closed>",
-    "<jupyter_start>",
-    "<jupyter_text>",
-    "<jupyter_code>",
-    "<jupyter_output>",
-    "<jupyter_script>",
-    "<empty_output>"
-  ],
-  "is_local": true,
-  "local_files_only": false,
-  "model_max_length": 1000000000000000019884624838656,
-  "pad_token": "<|endoftext|>",
-  "tokenizer_class": "GPT2Tokenizer",
-  "unk_token": "<|endoftext|>",
-  "vocab_size": 49152
-}
+  "pad_token": "<|pad|>",
+  "model_max_length": 4096,
+  "chat_template": "{% for message in messages %}<|im_start|>{{ message['role'] }}\n{{ message['content'] }}<|im_end|>\n{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}",
+  "add_bos_token": false,
+  "add_eos_token": false
+}