---
license: mit
language: [it, en]
tags: [italian, english, code, sparse-llm, dsa, pretraining]
---

# Pina 500M

A **536M**-parameter decoder-only language model for **Italian + English + code**,
trained from scratch. This is an **early / partial base-model checkpoint** — about
**9.19B tokens** (~23% of a planned 40B-token base run) — released for inspection.

This repo contains **model weights only** (`model.safetensors`, bf16) plus config and
inference code. It intentionally does **not** include optimizer state; resumable training
checkpoints live separately in `procmarco/pina-500m-ckpt`.

Current source checkpoint: `step=17520`, `token_offset=9186050048`.

## Current quality

- Italian: grammatical fragments, but still prone to repetition/boilerplate.
- English: improving, but still weak/repetitive.
- Code: still weak; this is not yet a coding-capable model.
- This is a base LM, **not instruction-tuned**. Use continuation-style prompts.
- Greedy decoding loops badly at this stage; sampling is more informative.

Recent eval notes:

- New representative 16-batch strided eval across the mixed eval bin:
  **4.466 bits/token**, **1.070 bits/byte**.
- A broader local 2.10M-token eval gave **3.90 bits/token**, **0.967 bits/byte**.
- Earlier offset-0 dashboard eval was noisy and not representative of the mixed eval bin.

## Model

Custom `SparseLM` (not a 🤗 Transformers architecture): RMSNorm, RoPE, QK-norm,
z-loss, tied embeddings, ReLU-gated sparse FFN, and **DeepSeek-style sparse attention
(DSA) in warm-up mode** (dense flash + lightning-indexer distillation during training).
Inference here uses plain dense causal attention.

Config: `d_model=1536`, `n_layers=16`, `ffn_hidden=4096`, `n_heads=24`,
`vocab_size=49152`, `max_seq_len=4096`.

## Data

40% Italian ([FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) ita) /
40% English ([FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)) /
20% code ([github-code-clean](https://huggingface.co/datasets/codeparrot/github-code-clean)).

Tokenizer: **[procmarco/ita-en-code-bpe-48k](https://huggingface.co/procmarco/ita-en-code-bpe-48k)**.
Documents were packed as `<|bos|> ... <|eos|>`.

## Usage

Self-contained — the modeling code (`modeling_pina.py`) ships in this repo, so you only
need `torch safetensors tiktoken huggingface_hub`:

```python
import json, pickle, torch
from safetensors.torch import load_model
from huggingface_hub import hf_hub_download
from modeling_pina import SparseLM, SparseLMConfig

REPO, TOK = "procmarco/pina-500m", "procmarco/ita-en-code-bpe-48k"
cfg = SparseLMConfig(**{k: v for k, v in json.load(open(hf_hub_download(REPO, "config.json"))).items() if k != "architecture"})
model = SparseLM(cfg).eval()
if torch.cuda.is_available():
    model = model.cuda().to(torch.bfloat16)
load_model(model, hf_hub_download(REPO, "model.safetensors"))

enc = pickle.load(open(hf_hub_download(TOK, "tokenizer.pkl"), "rb"))
bos, eos = enc.encode_single_token("<|bos|>"), enc.encode_single_token("<|eos|>")
ids = [bos] + enc.encode_ordinary("L'Italia è un paese famoso per")
out = model.generate(ids, max_new_tokens=60, temperature=0.8, top_k=40, eos_id=eos)
print(enc.decode(out[1:]))
```

See `example.py` for a runnable version.