| --- |
| license: apache-2.0 |
| language: |
| - it |
| library_name: echo-1.58 |
| tags: |
| - masked-diffusion |
| - ternary |
| - 1.58-bit |
| - quantization-aware-training |
| - bitnet |
| - italian |
| pipeline_tag: fill-mask |
| --- |
| |
| These are research artifacts accompanying the paper *Native Ternary |
| Quantization-Aware Training for Masked Diffusion Language Models*. They are 341M-parameter |
| masked-diffusion language models trained on Italian FineWeb-2 with a 32k SentencePiece |
| tokenizer. **This is not a production model.** At roughly 12 tokens per parameter neither the |
| ternary nor the full-precision model composes fluent text; free generation degenerates |
| identically at both precisions. Use these checkpoints for reproduction, infilling analysis, |
| and as paired baselines, not as downstream generators. |
|
|
| - **Architecture:** bidirectional masked-diffusion transformer, `d_model` 1024, 24 layers, 16 |
| heads, `d_ff` 2816, tied embeddings, 32001 vocabulary (mask token id 32000). |
| - **Format:** bfloat16, verified to reproduce the full-precision evaluation within 0.0003 |
| masked-CE of the fp32 master. |
| - **Code and reproduction:** https://github.com/lupodevelop/echo-1.58 |
| - **Tokenizer:** `spm_it.model` (included). |
|
|
| ## Evaluation (common protocol, sequence length 1024, identical seeded masks) |
|
|
| | 341M model | masked-CE | perplexity | vs FP16 twin | |
| |---|---|---|---| |
| | FP16 twin | 4.8100 | 122.7 | ceiling | |
| | Ternary baseline | 4.9852 | 146.2 | +19.2% | |
| | + continued distillation | 4.9125 | 136.0 | +10.8% | |
| | + from-scratch recipe | 4.9878 | 146.6 | +19.5% | |
|
|
| # Echo-1.58 341M, Ternary, From-Scratch Recipe (Negative Control) |
|
|
| A ternary model trained from the first step with the full recovery recipe (distillation plus |
| per-channel scale) for the full 4B-token budget. **This is a released negative control.** At |
| 341M the recipe applied from step 0 recovers nothing measurable: the model lands at |
| masked-CE 4.9878, within noise of the bare ternary baseline (4.9852), against the 70% recovery |
| the identical recipe delivers at 27M. Post-hoc distillation scales; from-scratch distillation |
| does not. Released because reproducible negative results are rarely shared and are useful for |
| anyone studying scale-dependent distillation. |
|
|
