AnCoder-1.0B-Base
Anchored bidirectional diffusion language model built on Qwen3-0.6B.
- Architecture: 28 anchor layers + 28 denoiser layers, hid connection, all weights tied
- Parameters: 1.04B unique
- Base model: Qwen/Qwen3-0.6B
- Training: 50k steps continued pretraining, token-packed streams (block_size=2048), uniform noise schedule, anchor_weight=0.1, all-position anchor supervision, shifted AR alignment (BOS-prepend trick on Qwen3 lm_head)
- Endpoint: SWA over the last 5 saved checkpoints (steps 46k–50k, 1k stride)
Training Progress (1000-entry SMA over final 5k steps, ending at 50k)
| Metric | Value |
|---|---|
| Loss | 1.1645 |
| DLM loss | 1.0037 |
| Anchor loss | 1.1015 |
| DLM accuracy | 60.84% |
| Anchor accuracy | 58.12% |
Architecture
AnCoder uses an anchor-denoiser architecture for absorbing-state diffusion language modeling:
- Anchor: Full bidirectional Qwen3 (28 layers) processes masked input
- Denoiser: Full bidirectional Qwen3 (28 layers) refines anchor's hidden states
- Connection: Anchor hidden states passed directly to denoiser (hid mode)
- Weight tying: All embeddings and lm_heads share the same weight matrix
- Shifted AR alignment: BOS is prepended at forward time and the trailing position is dropped before lm_head, so the AR-pretrained Qwen3 head operates on its native "predict-next" alignment under the bidirectional diffusion loss.
SWA Endpoint
Stage-1 training reaches a plateau around step ~25k. Training continues to 50k to give the trajectory time to settle in the basin, then tail-averages the last 5 saved checkpoints (steps 46000, 47000, 48000, 49000, 50000) to reduce noise in the final weights. The averaged model's tied weights are deduplicated at save time, yielding a single ~1.87 GB safetensors file.
Usage
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("EER6/AnCoder-1.0B-Base", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("EER6/AnCoder-1.0B-Base")
# For diffusion inference, pad short prompts with mask tokens (<|fim_middle|>, 151660)
# rather than the actual pad token (<|endoftext|>, 151643).
tokenizer.pad_token_id = 151660
inputs = tokenizer("def fibonacci(n):", return_tensors="pt", padding="max_length", max_length=2048)
outputs = model(**inputs)
outputs.logits # (B, L, V) denoiser predictions
outputs.anchor_logits # (B, L, V) anchor predictions