AlBERTone101

AlBERTone101 is a 450M-parameter Italian encoder model from the AlBERTurin family.

It was trained from scratch on approximately 101B Italian tokens using masked language modeling.

Model Description

AlBERTone101 is the largest model in the AlBERTurin family of encoder-only Transformer models for Italian.

The model incorporates several architectural improvements over the original BERT architecture, including Pre-RMSNorm, SwiGLU activations, ALiBi positional biases, and a mask-only pre-training objective.

AlBERTone101 uses:

• 28 Transformer layers
• hidden size of 1024
• 16 attention heads
• SwiGLU activations
• Pre-RMSNorm
• ALiBi positional biases
• 1,024-token training sequence length
• 28% mask-only MLM
• Muon optimizer

The model uses gettone, a 32,768-token BPE tokenizer optimized for Italian and shared across the AlBERTurin model family.

The model was trained using Matformer.

AlBERTurin Model Family

Model	Parameters	Training Tokens
AlBERTmini	95M	7B
AlBERTina	140M	14B
AlBERTone101	450M	~101B

Installation

python -m pip install \
  git+https://github.com/mrinaldi97/matformer.git@alberturin-v1

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "AlBERTurin/AlBERTone101"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

Citation

If you use AlBERTone101 in your research, please cite:

Matteo Rinaldi, Marco Madeddu, Calogero Jerik Scozzaro, Matteo Delsanto, Daniele Paolo Radicioni, and Viviana Patti.
AlBERTurin: A Fully Open Family of Italian Encoder Models with Modern Architectures.
CLiC-it 2026.