File size: 3,608 Bytes
1fba696 51107bb 1fba696 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | ---
license: apache-2.0
language:
- it
library_name: echo-1.58
tags:
- masked-diffusion
- ternary
- 1.58-bit
- quantization-aware-training
- bitnet
- italian
pipeline_tag: fill-mask
---
# Echo-1.58 · 341M · Ternary, Continued Distillation
The recommended ternary checkpoint from *Native Ternary Quantization-Aware Training for Masked
Diffusion Language Models*. A 341M-parameter masked-diffusion language model trained natively
at 1.58 bits on Italian FineWeb-2, then recovered by distilling from its full-precision twin.
> **Research artifact, not a production model.** At roughly 12 tokens per parameter neither the
> ternary nor the full-precision model composes fluent text; free generation degenerates
> identically at both precisions. Use it for reproduction, infilling analysis, and as a paired
> baseline.
## What this checkpoint is
Starting from the bare ternary baseline, this model was distilled from the FP16 twin for a
further 1B tokens with a learnable per-channel scale (0.07% extra memory). It recovers **41% of
the ternary-to-FP16 gap** in cross-entropy (48-49% on held-out test and out-of-domain Italian
Wikipedia) and retains about **92% of FP16 infilling accuracy** at every mask ratio, in a model
5.4x smaller than its twin. The paper's central practical finding is that this post-hoc route
scales while applying the same recipe from the first step does not.
## Evaluation (common protocol, sequence length 1024, identical seeded masks)
| 341M model | masked-CE | perplexity | vs FP16 twin |
|---|---|---|---|
| FP16 twin | 4.8100 | 122.7 | ceiling |
| Ternary baseline | 4.9852 | 146.2 | +19.2% |
| **This model (continued distillation)** | **4.9125** | **136.0** | **+10.8%** |
| From-scratch recipe (null control) | 4.9878 | 146.6 | +19.5% |
## Model details
- **Architecture:** bidirectional masked-diffusion transformer; `d_model` 1024, 24 layers, 16
heads, `d_ff` 2816, tied embeddings, 32001 vocabulary (mask token id 32000).
- **Format:** `model_bf16.pt`, verified within 0.0003 masked-CE of the fp32 master.
- **Tokenizer:** `spm_it.model` (SentencePiece unigram, included).
## Loading
The checkpoint uses the architecture in the [`echo-1.58`](https://github.com/lupodevelop/echo-1.58)
repository:
```python
import torch
from huggingface_hub import hf_hub_download
from model.arch import TernaryDiffusionLM, TernaryDiffusionConfig
from model.mitigations import MitigatedDiffusionLM
from model.quant import MitigationConfig, QATConfig
path = hf_hub_download("lupodevelop/echo-mdlm-341m-ternary-continued", "model_bf16.pt")
ck = torch.load(path, map_location="cpu")
cfg = TernaryDiffusionConfig(**ck["cfg"])
model = MitigatedDiffusionLM(cfg, MitigationConfig(), QATConfig(chan_scale=True))
model.load_state_dict({k: v.float() for k, v in ck["model"].items()})
model.eval()
```
See the repository for infilling (`eval/cloze.py`) and generation (`eval/sample.py`).
## Related models
Part of the [Echo-1.58 collection](https://huggingface.co/collections/lupodevelop/echo-158-ternary-masked-diffusion-lms-6a5dd179a5f7ad08cc277404):
the FP16 twin, the bare ternary baseline, the from-scratch null control, and the 27M research
ladder.
## Citation
```bibtex
@article{scaratti2026ternary,
author = {Daniele Scaratti},
title = {Recovering the Ternary Penalty in Masked Diffusion Language Models: What Adds Capacity, What Only Reorganizes, and What Repetition Costs},
year = {2026},
doi = {10.5281/zenodo.21455055}
}
```
|