--- language: - en tags: - pytorch - mixture-of-experts - optimizer-comparison --- # ANLP Assignment 2 ## Links - **Weights & Biases:** [Task 1](https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part1), [Task 2](https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part2), [Task 3](https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part3). - **Checkpoints (Hugging Face):** https://huggingface.co/Chechaaa09/anlp-a2-models ## Setup Dependencies are pinned in `pyproject.toml` / `uv.lock`: ```bash uv sync ``` Tasks 1 and 3 load their datasets from Hugging Face. For Task 2, save `browndw/human-ai-parallel-corpus` as a Hugging Face Dataset with a `text` column at `src/part2/dataset/`. ## Usage Run from the repository root: ```bash uv run python -m src.part1.train --all # train the five FFN variants uv run python -m src.part1.evaluate # evaluate checkpoints and plot expert usage uv run python -m src.part2.train --all # train with all five optimizers uv run python -m src.part3.evaluate --num_samples 1000 --max_new_tokens 40 ``` Task 1 checkpoints are written to `checkpoints_part1/best_variant_.pt`, and evaluation results to `assets/part1_results/`. Task 2 logs metrics to Weights & Biases and saves `best.pt`, `last.pt`, and `metrics.json` under `checkpoints_part2///`. The weights are PyTorch state dictionaries; `metrics.json` includes the model configuration for loading them. Task 3 results are written to `task3_results.json`. Task 3 uses the original pretrained [EleutherAI/pythia-160m](https://huggingface.co/EleutherAI/pythia-160m) checkpoint. ## Loading a Task 2 checkpoint These are state dictionaries for the custom PyTorch architecture in this repository. Clone or download the source files, install the dependencies, and run from that checkout: ```python import json import torch from huggingface_hub import hf_hub_download from src.part2.model import DecoderOnlyTransformer repo_id = "Chechaaa09/anlp-a2-models" folder = "checkpoints_part2/adamw/oj3opt7s" with open(hf_hub_download(repo_id, f"{folder}/metrics.json")) as handle: metadata = json.load(handle) model = DecoderOnlyTransformer(**metadata["model_config"]) weights = torch.load( hf_hub_download(repo_id, f"{folder}/best.pt"), map_location="cpu", weights_only=True, ) model.load_state_dict(weights) model.eval() ``` Use the GPT-2 tokenizer for Task 2. Each run includes `best.pt` (lowest validation loss), `last.pt` (final weights), and `metrics.json` (configuration, all ten evaluations, and its W&B link). Task 1 uses the saved custom tokenizer in `tokenizer_part1/`. | Optimizer | Checkpoint folder | Validation loss | Validation perplexity | Test sentence BLEU | | --- | --- | ---: | ---: | ---: | | adamw | `checkpoints_part2/adamw/oj3opt7s/` | 5.2531 | 191.15 | 2.9645 | | cautious | `checkpoints_part2/cautious/p2yw4r11/` | 5.4965 | 243.84 | 2.5585 | | lion | `checkpoints_part2/lion/ydyarkxk/` | 4.9409 | 139.90 | 2.5707 | | scion | `checkpoints_part2/scion/89yiklpr/` | 5.5330 | 252.91 | 2.7522 | | sophia | `checkpoints_part2/sophia/3j32zwf3/` | 5.3892 | 219.02 | 2.5534 | Task 2 results are from the completed checkpoint-saving runs on 4 October 2026. BLEU is on the 0–100 scale.