code/README.md

Code Overview

This workspace keeps only the code needed for the tokenizer study.

Main Modules

code/tokenization/

• wl.py: WL tokenizer wrapper
• simhash.py: fixed-dimensional SimHash tokenizer
• shortest_path.py: shortest-path histogram tokenizer
• graphbpe.py: graph BPE tokenizer
• random.py: weak compositional random baseline
• multidomain.py: pooled and union all-domain tokenizer utilities

code/encoding_generation/

• generate_multi_embeddings.py: domain-dependent embedding generation
• generate_all_domain_embeddings.py: all-domain tokenizer fitting and embedding generation

code/modeling/

• train_lstm.py
• train_xgb.py
• inference_lstm.py
• inference_xgb.py
• dataset.py
• models.py

code/experiments/

• config.py: tokenizer/model/domain defaults
• run_tokenizer_paper.py: main fresh-study orchestrator
• analyze_tokenizer_study.py: aggregation, significance testing, paper outputs

code/downstream/

• build_validity_dataset.py: recovers gold plans, generates corrupted candidate plans, labels them, and extracts frozen transition-model features
• train_validity.py: trains a small MLP head for binary valid-plan classification
• run_validity_experiments.py: runs the standard downstream experiment over the weighted-best frozen source families

code/common/

• utils.py: seeding and VAL plan validation
• wl_wrapper.py: WL helper integration
• fsf_wrapper.py: retained only for compatibility in shared modeling code paths

Recommended Entry Points

• Full study: python -m code.experiments.run_tokenizer_paper
• Analysis only: python -m code.experiments.analyze_tokenizer_study
• Downstream validity task: python -m code.downstream.run_validity_experiments