README.md

metadata

license: apache-2.0
language:
  - it
library_name: echo-1.58
tags:
  - masked-diffusion
  - ternary
  - 1.58-bit
  - quantization-aware-training
  - bitnet
  - italian
pipeline_tag: fill-mask

These are research artifacts accompanying the paper Native Ternary Quantization-Aware Training for Masked Diffusion Language Models. They are 341M-parameter masked-diffusion language models trained on Italian FineWeb-2 with a 32k SentencePiece tokenizer. This is not a production model. At roughly 12 tokens per parameter neither the ternary nor the full-precision model composes fluent text; free generation degenerates identically at both precisions. Use these checkpoints for reproduction, infilling analysis, and as paired baselines, not as downstream generators.

• Architecture: bidirectional masked-diffusion transformer, d_model 1024, 24 layers, 16 heads, d_ff 2816, tied embeddings, 32001 vocabulary (mask token id 32000).
• Format: bfloat16, verified to reproduce the full-precision evaluation within 0.0003 masked-CE of the fp32 master.
• Code and reproduction: https://github.com/lupodevelop/echo-1.58
• Tokenizer: spm_it.model (included).

Evaluation (common protocol, sequence length 1024, identical seeded masks)

341M model	masked-CE	perplexity	vs FP16 twin
FP16 twin	4.8100	122.7	ceiling
Ternary baseline	4.9852	146.2	+19.2%
+ continued distillation	4.9125	136.0	+10.8%
+ from-scratch recipe	4.9878	146.6	+19.5%

Echo-1.58 341M, Ternary, From-Scratch Recipe (Negative Control)

A ternary model trained from the first step with the full recovery recipe (distillation plus per-channel scale) for the full 4B-token budget. This is a released negative control. At 341M the recipe applied from step 0 recovers nothing measurable: the model lands at masked-CE 4.9878, within noise of the bare ternary baseline (4.9852), against the 70% recovery the identical recipe delivers at 27M. Post-hoc distillation scales; from-scratch distillation does not. Released because reproducible negative results are rarely shared and are useful for anyone studying scale-dependent distillation.