Spaces:
Sleeping
Sleeping
File size: 1,774 Bytes
dbc6675 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | # Code Overview
This workspace keeps only the code needed for the tokenizer study.
## Main Modules
### `code/tokenization/`
- `wl.py`: WL tokenizer wrapper
- `simhash.py`: fixed-dimensional SimHash tokenizer
- `shortest_path.py`: shortest-path histogram tokenizer
- `graphbpe.py`: graph BPE tokenizer
- `random.py`: weak compositional random baseline
- `multidomain.py`: pooled and union all-domain tokenizer utilities
### `code/encoding_generation/`
- `generate_multi_embeddings.py`: domain-dependent embedding generation
- `generate_all_domain_embeddings.py`: all-domain tokenizer fitting and embedding generation
### `code/modeling/`
- `train_lstm.py`
- `train_xgb.py`
- `inference_lstm.py`
- `inference_xgb.py`
- `dataset.py`
- `models.py`
### `code/experiments/`
- `config.py`: tokenizer/model/domain defaults
- `run_tokenizer_paper.py`: main fresh-study orchestrator
- `analyze_tokenizer_study.py`: aggregation, significance testing, paper outputs
### `code/downstream/`
- `build_validity_dataset.py`: recovers gold plans, generates corrupted candidate plans, labels them, and extracts frozen transition-model features
- `train_validity.py`: trains a small MLP head for binary valid-plan classification
- `run_validity_experiments.py`: runs the standard downstream experiment over the weighted-best frozen source families
### `code/common/`
- `utils.py`: seeding and VAL plan validation
- `wl_wrapper.py`: WL helper integration
- `fsf_wrapper.py`: retained only for compatibility in shared modeling code paths
## Recommended Entry Points
- Full study: `python -m code.experiments.run_tokenizer_paper`
- Analysis only: `python -m code.experiments.analyze_tokenizer_study`
- Downstream validity task: `python -m code.downstream.run_validity_experiments`
|