PlasmidLMEval

Comprehensive evaluation data for PlasmidLM models. Generated by the eval pipeline in McClain-Thiel/PlasmidLLM.

Structure

reference/                              # Addgene-500 reference panel
  addgene_500.fasta                     # 500 Addgene plasmid sequences
  addgene_reference_500.csv             # ID, name, sequence, length
  addgene_reference_metrics.csv         # Pre-computed: length, GC, longest ORF, MFE density
  addgene_500_3mer_freqs.json           # Canonical 3-mer frequency distribution
  addgene_500_mfe_per_seq.parquet       # Per-sequence MFE density (DNA Mathews 2004,
                                        #   5×2 kb random windows for >5 kb seqs)
  training_sigs.zip                     # Sourmash signature of training set (20644 plasmids,
                                        #   k=31, scaled=100)

config/                                 # Eval configuration files
  environment.yml                       # Pinned tool versions (plannotate 1.2.2,
                                        #   prodigal 2.6.3, etc.)
  feature_categories.yaml               # pLannotate feature → eval category mapping

runs/
  firstlight_20260408/                  # Phase 1: sampling config grid search
    sweep_results.csv
    sweep_summary.json
    t{temp}_p{top_p}_r{rep}/
      generations.{parquet,fasta}
      plannotate_results.json
      cell_summary.json

  grpo_plannotate_full_20260408/        # Headline GRPO eval (n=1000, seed 42)
    config.json
    generations.{parquet,fasta}
    matched_pairs.parquet               # Paired with closest real Addgene plasmid
    plannotate_results.json
    dustmasker/{dustmasker.intervals,input_filtered.fasta}
    orfs/{prodigal.gff,prodigal_proteins.faa}
    metrics/
      tier1_summary.json
      tier1_distributional.parquet      # Per-seq length, GC, ORF density, ...
      tier3_essentials.parquet          # Per-seq has_ori, has_marker, plausibility_pass, ...
      tier5_architecture.parquet
      extra_metrics.json                # Codon JSD, GC skew, MFE summary
      prompt_fidelity.json
      prompt_fidelity_per_seq.parquet   # Per-prompt sseqid-intersection fidelity
      mfe_per_seq.parquet               # NEW (2026-04-27): per-seq MFE density
      memorization.parquet              # Sourmash hits ≥ threshold (typically empty)
      gen_sigs.zip
      shap_summary.png
      diversity.json
      discriminator.json
    evo2/
      evo2_scores.parquet               # Per-pair PPL on n=200 matched real/gen
      evo2_summary.json
      gen_embeddings.npy                # 200×4096 mean-pooled embeddings
      real_embeddings.npy               # 200×4096

  base_kmer6_matched_20260420_1303/     # Pre-GRPO base eval (n=1000, seed 42; same prompts as GRPO)
    config.json
    generations.{parquet,fasta}
    matched_pairs_evo2.parquet          # NEW (2026-04-27): 200-pair subset for Evo2 scoring
    plannotate_results.json
    dustmasker/, orfs/
    metrics/                            # Same schema as GRPO
      mfe_per_seq.parquet               # NEW (2026-04-27)
      tier{1,3,5}_*, extra_metrics.json, prompt_fidelity_per_seq.parquet, ...
    evo2/                               # NEW (2026-04-27)
      evo2_scores.parquet               # 200 pairs, paired with same real plasmids as GRPO Evo2 run
      evo2_summary.json

  bon_{base,grpo}_seed{43,44,45}_20260420_1303/   # Best-of-K seeds (paired with seed-42 above)
    Same schema as base_kmer6_matched / grpo_plannotate_full minus the evo2/ subdir.

  base_kmer6_sweep_20260420_1303/       # Pre-GRPO base temperature sweep (n=100/cell)
    t{03,05,10}/                        # No t07 — seed 42 above already provides t=0.7 at n=1000
      Same schema, no evo2/

  forced_sp_grpo_20260427/              # NEW (2026-04-27): n=400 GRPO generations forced
                                        #   to SP_ECOLI/SP_YEAST prompts only (200/200 with
                                        #   replacement from 40 unique SP_ECOLI + 99 unique
                                        #   SP_YEAST training prompts). Same sampling config
                                        #   as the matched runs.
    generations.{parquet,fasta}
    orfs/prodigal.gff
    metrics/{cai_per_orf,cai_per_seq}.parquet

  forced_sp_base_20260427/              # Same as above for the supervised base.

reference/training_sp_sample/           # NEW (2026-04-27): n=300/SP × 10 SP classes from
                                        # training_pairs_v4. Real plasmids stratified by SP
                                        # token, used as the positive control for the
                                        # CAI-by-SP_token experiment.
    generations.parquet                 # (mis-named for layout convenience — these are real
                                        # training plasmid sequences with their original
                                        # prompt strings; "generations" matches the schema)
    orfs/prodigal.gff
    cai_per_orf.parquet
    cai_per_seq.parquet

figures/                                # NEW (2026-04-27): rendered paper figures, archived
                                        # alongside the data they were rendered from.
    cai_yeast_minus_ecoli_forest.{pdf,png}

eval_log.md                             # Free-form log of every run that produced data here

Models Evaluated

• McClain/PlasmidLM-kmer6 — pre-GRPO supervised base (19.3 M params, dense kmer6 transformer). Acts as the supervised promptable baseline.
• McClain/PlasmidLM-kmer6-GRPO-plannotate — GRPO post-trained from above with pLannotate composite reward.

Headline numbers (paired n=1000, seed-42 matched)

Note: GRPO improves prompt fidelity and the useful-plasmid rate, but is slightly less in-distribution under Evo 2 than base. MFE moves in the opposite direction — GRPO is more thermodynamically stable and closer to real Addgene than base. Evo 2 PPL captures global "naturalness" under a 7 B-parameter generic DNA LM; MFE captures local structure stability of folded RNA. The two move in opposite directions because they measure different things.

Host-specific codon usage (SP × CAI, 2026-04-27)

To test whether the model conditions on the soft SP_* token in a way pLannotate cannot detect, we score per-ORF Codon Adaptation Index (Sharp & Li, NAR 1987) against three host codon-usage tables (E. coli K-12 / S. cerevisiae / human; Kazusa via python_codon_tables) on:

1. Real plasmids stratified by their SP token (training subsample, 300/class). Establishes that real plasmids carry a host-specific codon signal.
2. Forced SP_ECOLI / SP_YEAST generations from base and post-GRPO (n=400 each, with-replacement sampling from the 40 unique SP_ECOLI and 99 unique SP_YEAST prompts in training).
3. The original n=1000 paired eval (only SP_HUMAN/SP_MOUSE have meaningful n in that set; SP_ECOLI/SP_YEAST appear in only ~10 prompts each).

Per-sequence Δ = $\overline{\textrm{CAI}}_\textrm{yeast} - \overline{\textrm{CAI}}_\textrm{ecoli}$, length-weighted across ORFs ≥ 100 codons in each sequence. Bootstrap 95% CIs on the mean of Δ within each (source, SP) cell.

Direction-matches (3/4 vs 2/4 SP classes for GRPO vs base) on a soft-token signal that GRPO did not directly reward. See figures/cai_yeast_minus_ecoli_forest.pdf.

Methodology notes

• MFE per-seq is computed with ViennaRNA RNA.fold under DNA Mathews 2004 parameters. Sequences ≤ 5 kb are folded full-length; longer sequences are summarised as the mean MFE density across 5 random 2 kb windows with a per-sequence deterministic seed. Earlier extra_metrics.json uses the older 200-sample × 500 bp-window method; the per-seq parquet uses the newer thorough method, so summary numbers will not match exactly between the two.
• Evo 2 uses evo2_7b_base. Sequences > 8 kb are scored on sliding windows with 50 % overlap, perplexities averaged on the log scale. Embeddings (when emitted by the Evo 2 forward call) are mean-pooled across positions to a single (4096,) vector. The 2026-04-27 base run produced PPL only — embeddings were not returned by the current evo2.Evo2.__call__ signature; the 2026-04-08 GRPO run did get them.

Generation Method

Uses the model's optimized generate_simple() with KV cache. Prompts sampled from training set (5,124 unique prompt templates). Each prompt is a sequence of special tokens encoding plasmid properties (vector type, species, resistance markers, origins, promoters, etc.) followed by <SEQ>.

Tools

Phase 1: First-Light Sweep (2026-04-08)

Grid search over 12 sampling configs (3 temperatures × 2 top_p × 2 repetition_penalty), 50 sequences per cell = 600 total generations.

Winner: temp=0.7, top_p=0.95, rep_pen=1.0

• 70.8% median pLannotate coverage
• 22.1 mean features per sequence
• 7.6 kb mean length, 50.5% GC

Reproducing the figures

The paper figure script lives at paper/generate_figures.py (in the PlasmidLLM repo). Populate a local cache from this bucket via

hf buckets sync hf://buckets/McClain/PlasmidLMEval /tmp/plasmidlm_eval_cache

then uv run --script paper/generate_figures.py.