Devatri/escape-blt-entropy-student-alpha1

ESCAPE: no-teacher control for the distilled BLT patcher (14.3M, α = 1)

The control arm for the distilled student of Project ESCAPE (Team Trimax, ANLP, IIIT Hyderabad, October 2026). Same architecture (14,262,144 parameters; dim 384, 8 layers, 6 heads, byte vocabulary 260, ≤512-byte context), same data, seed and schedule, but trained with α = 1: plain next-byte cross-entropy, with no signal from the BLT patcher. It measures what the teacher adds.

Training

Two epochs (6,860 steps) on 27,153 files (164,627 windows) of the ESCAPE extension corpus, which is disjoint from the evaluation set; batch 48, AdamW (lr 5e-4, weight decay 0.1, 200 warm-up steps), seed 20260927, RTX 2080 Ti (IIIT Hyderabad Ada cluster).

Results (from the ESCAPE final report)

Boundary agreement with the teacher's patches on held-out extension windows:

Model	Boundary F1 vs. teacher	Bits/byte
This control (α = 1)	0.789 (Python 0.798, C++ 0.791, English 0.764)	1.17
Distilled student (α = 0.1)	0.871	1.24
A boundary at every word start	0.475	–

The control predicts the next byte better, but agrees less with the teacher's segmentation: distillation trades a little compression for that agreement. This control was not scored on the held-out constituent-alignment split.

Use

Clone the ESCAPE repository, then:

import torch
from huggingface_hub import hf_hub_download
from escape_distill.student_blt import StudentBLT, StudentConfig

ckpt = torch.load(hf_hub_download("Devatri/escape-blt-entropy-student-alpha1", "student.pt"), map_location="cpu", weights_only=False)
model = StudentBLT(StudentConfig(**ckpt["config"])).eval()
model.load_state_dict(ckpt["model_state"])

model.safetensors holds the same weights; config.json has the model configuration and training arguments, train_log.json the loss curve, and distill_eval.json the held-out agreement with the teacher per domain.

Citation

Teacher model: Pagnoni et al., 2025, Byte Latent Transformer: Patches Scale Better Than Tokens, ACL 2025.

No licence has been chosen for this model; the training data keep their own licences.