code/README.md

# Code Overview

This workspace keeps only the code needed for the tokenizer study.

## Main Modules

### `code/tokenization/`

- `wl.py`: WL tokenizer wrapper
- `simhash.py`: fixed-dimensional SimHash tokenizer
- `shortest_path.py`: shortest-path histogram tokenizer
- `graphbpe.py`: graph BPE tokenizer
- `random.py`: weak compositional random baseline
- `multidomain.py`: pooled and union all-domain tokenizer utilities

### `code/encoding_generation/`

- `generate_multi_embeddings.py`: domain-dependent embedding generation
- `generate_all_domain_embeddings.py`: all-domain tokenizer fitting and embedding generation

### `code/modeling/`

- `train_lstm.py`
- `train_xgb.py`
- `inference_lstm.py`
- `inference_xgb.py`
- `dataset.py`
- `models.py`

### `code/experiments/`

- `config.py`: tokenizer/model/domain defaults
- `run_tokenizer_paper.py`: main fresh-study orchestrator
- `analyze_tokenizer_study.py`: aggregation, significance testing, paper outputs

### `code/downstream/`

- `build_validity_dataset.py`: recovers gold plans, generates corrupted candidate plans, labels them, and extracts frozen transition-model features
- `train_validity.py`: trains a small MLP head for binary valid-plan classification
- `run_validity_experiments.py`: runs the standard downstream experiment over the weighted-best frozen source families

### `code/common/`

- `utils.py`: seeding and VAL plan validation
- `wl_wrapper.py`: WL helper integration
- `fsf_wrapper.py`: retained only for compatibility in shared modeling code paths

## Recommended Entry Points

- Full study: `python -m code.experiments.run_tokenizer_paper`
- Analysis only: `python -m code.experiments.analyze_tokenizer_study`
- Downstream validity task: `python -m code.downstream.run_validity_experiments`