yharith/anlp-a1-transformer-ablation

ANLP Assignment 1: Transformer Ablation (C1-C5)

Encoder-decoder transformers trained from scratch to map encrypted binary sequences to plaintext. Five configurations differing by exactly one component.

Config	Positional	Attention	Norm	Tokenization
C1	Sinusoidal	MHA	LayerNorm	BPE subword
C2	RoPE	MHA	LayerNorm	BPE subword
C3	Sinusoidal	GQA (n_kv=2)	LayerNorm	BPE subword
C4	Sinusoidal	MHA	RMSNorm	BPE subword
C5	Sinusoidal	MHA	LayerNorm	BLT (token-free bytes)

All modules (scaled dot-product attention, MHA, GQA, sinusoidal, RoPE, LayerNorm, RMSNorm, BLT local encoder/decoder) are implemented from basic PyTorch operations.

Links

• Training runs (WandB): https://wandb.ai/harith-yerragolam-iiit-hyderabad/anlp-a1-transformers

Shared hyperparameters

Identical across all five configurations, only the ablated component differs.

{
  "d_model": 256,
  "n_heads": 8,
  "n_layers": 4,
  "d_ff": 1024,
  "dropout": 0.1,
  "lr": 0.0003,
  "batch_size": 64,
  "epochs": 40,
  "warmup_frac": 0.05,
  "label_smoothing": 0.1,
  "grad_clip": 1.0,
  "weight_decay": 0.01,
  "patch_size": 8,
  "d_local": 64,
  "n_local_layers": 2,
  "n_local_heads": 4
}

Files

• <CONFIG>_best.pt: best-validation checkpoint
• <CONFIG>_history.json: per-epoch loss, timing, peak memory
• <CONFIG>_eval.json: test-set metrics under greedy decoding
• tokenizer_cipher.json, tokenizer_plain.json: byte-level BPE (C1-C4 only)

Loading

import torch
from src.train import build_model, HP

ckpt = torch.load("C1_best.pt", map_location="cpu", weights_only=False)
model = build_model("C1", HP, src_vocab=4000, tgt_vocab=8000)
model.load_state_dict(ckpt["model"])

Reading the results

Two caveats decide whether these numbers mean what they look like.

Bit accuracy has a high floor, do not read it as "percent correct". ASCII letters share their high-order bits (a=01100001, e=01100101), so emitting any letters of the right length already scores 0.729. The baseline table below gives the floor. Sequence accuracy and Levenshtein distance are the metrics that actually discriminate between these models.

Loss is not comparable between C5 and C1-C4. C1-C4 predict over 8000 subword tokens, C5 over 260 bytes, so their cross-entropies are in different units. Random guessing alone gives ln(8000)=8.99 against ln(260)=5.56. C5's lower loss is not evidence that it is better. Only the task-level metrics compare across the tokenization boundary.

A consequence worth knowing is that C5 leads on bit accuracy (0.9718) but trails badly on sequence accuracy (0.0879). That is an alignment effect, not per-character accuracy. Bit accuracy compares bits position-by-position, so one inserted or deleted character shifts everything after it. The byte-level decoder emits exactly one byte per character and is the right length 98.0% of the time; the subword decoders manage 34.8-62.0%. Per character, C5 is in fact worse than C1 (~96.3% against ~98.5%).

Note also that C5 is not capacity-matched. It has 3.47M parameters against C1's 12.49M, because a vocabulary-free model carries no large embedding or output-projection matrices and its local decoder is small (d=64, 2 layers) next to C1's decoder (d=256, 4 layers). Its global encoder is identical to C1's encoder at 3,154,944 parameters.

Results

Quality

config	params	val loss	bit acc	seq acc	lev	len exact	BLEU	ROUGE-L
C1 base	12,489,536	1.7954	0.9055	0.3798	2.71	0.5615	79.89	0.8941
C2 RoPE	12,489,536	1.7540	0.9167	0.4114	2.33	0.6202	81.56	0.9028
C3 GQA	11,309,888	2.1370	0.8474	0.1704	5.95	0.3484	66.03	0.8147
C4 RMSNorm	12,483,904	1.8012	0.9063	0.3735	2.79	0.5625	79.46	0.8919
C5 BLT	3,470,916	1.0911	0.9718	0.0879	4.65	0.9800	-	-

Bit-accuracy baselines

baseline	bit acc
const_e	0.7287
const_space	0.6619
random_letters	0.6125
empty	0.0000

Efficiency

config	s/epoch	train peak MB	decode ms/seq	decode peak MB
C1 base	51.7	1612	2.21	239
C2 RoPE	71.7	1611	5.56	229
C3 GQA	56.1	1596	5.34	233
C4 RMSNorm	43.0	1526	4.79	230
C5 BLT	38.9	298	8.36	68