| # Supplementary: tokenizer characterization benchmark |
|
|
| Code, data, and results behind Supplementary Figure 2: |
| - **(a)** motif preservation — perfect match rate and motif fragmentation |
| - **(b)** Jensen–Shannon distance between functional-region token distributions and a matched random genomic background |
|
|
| Tokenizers compared: DNABERT-2 BPE, Grover BPE, cCRE-guided BPE, motif-guided BPE, DNAMotifTokenizer. |
|
|
| ## Layout |
|
|
| ### Shared |
| - `tokenizers/` — all five tokenizers used in both panels |
| - `DNAbert2_tokenizer.json`, `Grover_tokenizer.json` — original DNABERT-2 / Grover tokenizers |
| - `cCRE_region_BPE_tokenizer.json`, `motif_region_BPE_tokenizer.json` |
| - `DNAMotifTokenizer/` — trie, motif list, vocab (loaded via `code/dna_motif_tokenizer.py`, seed = 42) |
| - `code/dna_motif_tokenizer.py` — wrapper exposing DNAMotifTokenizer through a `tokenizers`-like interface |
|
|
| ### Panel a — motif preservation |
| - `data/motifs.txt` — 904 unique motifs; wildcard (`*`) motifs are expanded to all A/C/G/T variants (≤5 wildcards), giving 3,737 test sequences of length 5–12 |
| - `code/motif_coverage_eval.py` — tokenizes each variant (no special tokens) and computes per-tokenizer metrics |
| - `results/summary_by_tokenizer.csv` — per-tokenizer metrics plotted in the figure |
| - `results/motif_variant_results.csv`, `results/consistency_by_motif.csv` — per-variant and per-motif detail |
| - `code/plot_supp_fig2a_motif.py` → `figures/supp_fig2a_motif.{pdf,png}` |
|
|
| Metrics: |
| - **Perfect match rate** — % of motif variants encoded as exactly one token |
| - **Avg. tokens per motif** — mean number of tokens per motif variant (lower = less fragmentation) |
|
|
| ### Panel b — region vs random-background JS distance |
| - `data/regions/` — promoter (2 kb), enhancer, and exon interval sets (merged BED) |
| - `data/background/` — `chrom.sizes` (canonical hg38 chromosomes), `region_union.bed` (union of the three region sets, used as exclusion), `random.bed` (exon intervals shuffled into the non-region genome with `bedtools shuffle -seed 42`, so total bp matches exon) |
| - `data/fastas/` — hg38 sequences for promoter, enhancer, exon, and random (`bedtools getfasta`) |
| - `code/js_vs_random_heatmap.py` — builds the background, tokenizes each FASTA, and computes the Jensen–Shannon distance (base 2) between each region's token-frequency distribution and the random background |
| - `results/js_vs_random_matrix.csv` — 5 tokenizers × 3 regions matrix plotted in the figure |
| - `results/js_vs_random.log` — run log with token counts per region and tokenizer |
| - `figures/js_vs_random_heatmap.{pdf,png}` — heatmap from `js_vs_random_heatmap.py` |
| - `code/plot_combined_panel_a_b.py` → `figures/combined_panel_a_b.{pdf,png}` — combined draft of panels a and b |
|
|
| ## Reproduce |
| ``` |
| # Panel a |
| python code/motif_coverage_eval.py --motif-file data/motifs.txt --output-dir results |
| python code/plot_supp_fig2a_motif.py |
| |
| # Panel b (requires bedtools and hg38.fa from UCSC) |
| python code/js_vs_random_heatmap.py # full run |
| python code/js_vs_random_heatmap.py --plot-only # redraw from results/js_vs_random_matrix.csv |
| ``` |
| `motif_coverage_eval.py`, `js_vs_random_heatmap.py`, and `plot_combined_panel_a_b.py` are uploaded as run and resolve paths relative to the original project layout (`DEFAULT_TOKENIZER_PATHS`, `ROOT`, `REGION_BEDS`, etc.); point them at `tokenizers/` and `data/` before running. The hg38 reference genome is not included (public, ~3 GB); the extracted FASTAs in `data/fastas/` are what the analysis tokenized. |
|
|
