| --- |
| license: mit |
| language: [it, en] |
| tags: [italian, english, code, sparse-llm, dsa, pretraining] |
| --- |
| |
| # Pina 500M |
|
|
| A **536M**-parameter decoder-only language model for **Italian + English + code**, |
| trained from scratch. This is an **early / partial base-model checkpoint** — about |
| **9.19B tokens** (~23% of a planned 40B-token base run) — released for inspection. |
|
|
| This repo contains **model weights only** (`model.safetensors`, bf16) plus config and |
| inference code. It intentionally does **not** include optimizer state; resumable training |
| checkpoints live separately in `procmarco/pina-500m-ckpt`. |
|
|
| Current source checkpoint: `step=17520`, `token_offset=9186050048`. |
|
|
| ## Current quality |
|
|
| - Italian: grammatical fragments, but still prone to repetition/boilerplate. |
| - English: improving, but still weak/repetitive. |
| - Code: still weak; this is not yet a coding-capable model. |
| - This is a base LM, **not instruction-tuned**. Use continuation-style prompts. |
| - Greedy decoding loops badly at this stage; sampling is more informative. |
|
|
| Recent eval notes: |
|
|
| - New representative 16-batch strided eval across the mixed eval bin: |
| **4.466 bits/token**, **1.070 bits/byte**. |
| - A broader local 2.10M-token eval gave **3.90 bits/token**, **0.967 bits/byte**. |
| - Earlier offset-0 dashboard eval was noisy and not representative of the mixed eval bin. |
|
|
| ## Model |
|
|
| Custom `SparseLM` (not a 🤗 Transformers architecture): RMSNorm, RoPE, QK-norm, |
| z-loss, tied embeddings, ReLU-gated sparse FFN, and **DeepSeek-style sparse attention |
| (DSA) in warm-up mode** (dense flash + lightning-indexer distillation during training). |
| Inference here uses plain dense causal attention. |
|
|
| Config: `d_model=1536`, `n_layers=16`, `ffn_hidden=4096`, `n_heads=24`, |
| `vocab_size=49152`, `max_seq_len=4096`. |
|
|
| ## Data |
|
|
| 40% Italian ([FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) ita) / |
| 40% English ([FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)) / |
| 20% code ([github-code-clean](https://huggingface.co/datasets/codeparrot/github-code-clean)). |
|
|
| Tokenizer: **[procmarco/ita-en-code-bpe-48k](https://huggingface.co/procmarco/ita-en-code-bpe-48k)**. |
| Documents were packed as `<|bos|> ... <|eos|>`. |
|
|
| ## Usage |
|
|
| Self-contained — the modeling code (`modeling_pina.py`) ships in this repo, so you only |
| need `torch safetensors tiktoken huggingface_hub`: |
|
|
| ```python |
| import json, pickle, torch |
| from safetensors.torch import load_model |
| from huggingface_hub import hf_hub_download |
| from modeling_pina import SparseLM, SparseLMConfig |
| |
| REPO, TOK = "procmarco/pina-500m", "procmarco/ita-en-code-bpe-48k" |
| cfg = SparseLMConfig(**{k: v for k, v in json.load(open(hf_hub_download(REPO, "config.json"))).items() if k != "architecture"}) |
| model = SparseLM(cfg).eval() |
| if torch.cuda.is_available(): |
| model = model.cuda().to(torch.bfloat16) |
| load_model(model, hf_hub_download(REPO, "model.safetensors")) |
| |
| enc = pickle.load(open(hf_hub_download(TOK, "tokenizer.pkl"), "rb")) |
| bos, eos = enc.encode_single_token("<|bos|>"), enc.encode_single_token("<|eos|>") |
| ids = [bos] + enc.encode_ordinary("L'Italia è un paese famoso per") |
| out = model.generate(ids, max_new_tokens=60, temperature=0.8, top_k=40, eos_id=eos) |
| print(enc.decode(out[1:])) |
| ``` |
|
|
| See `example.py` for a runnable version. |
|
|
