loubnabnl HF Staff commited on
Commit
a8e4fca
·
verified ·
1 Parent(s): 04ca264

fix tata and syn

Browse files
Files changed (1) hide show
  1. README.md +8 -8
README.md CHANGED
@@ -46,8 +46,7 @@ Carbon-3B is the **flagship** model of the Carbon family. We also release [**Car
46
  - **Metadata-conditioned**: optional species-type and gene-type metadata tokens enable conditional generation.
47
  - **Efficient inference**: compatible with vLLM and other inference engines. Can generate over 100,000 base-pairs per second on a single H100 GPU.
48
 
49
- Across our zero-shot evaluation suite sequence recovery, four variant-effect-prediction (VEP) benchmarks (ClinVar coding, ClinVar non-coding, BRCA2, TraitGym Mendelian), and two sequence-level perturbation tasks (TATA-box and synonymous codon) Carbon-3B is competitive with Evo2-7B. It additionally works well on long context and retrieves needles reliably from up to ≈ 393 kbp of distal context on the Genome-NIAH long-context benchmark, while remaining several times faster than Evo2-7B.
50
-
51
  For full design rationale and ablations, see the Carbon technical report and the [Carbon GitHub repository](https://github.com/huggingface/carbon).
52
 
53
  ## How to use
@@ -139,7 +138,7 @@ def score(seq: str) -> float:
139
  return logp.mean().item()
140
  ```
141
 
142
- For batched scoring with attention masking and full reproducible evaluation pipelines (sequence recovery, ClinVar / BRCA2 / TraitGym VEP, TATA / synonymous-codon perturbation, Genome-NIAH), use the official scripts in the [Carbon evaluation directory](https://github.com/huggingface/carbon/tree/main/evaluation) — see [`perturbation_tasks.py`](https://github.com/huggingface/carbon/blob/main/evaluation/perturbation_tasks.py) for the canonical `score_hf` implementation and [`README.md`](https://github.com/huggingface/carbon/blob/main/evaluation/README.md) for run instructions across all tasks.
143
 
144
  ### Long context: extending to 65,536 tokens (≈ 393 kbp) with YaRN
145
 
@@ -277,11 +276,12 @@ All evaluations are zero-shot and use the [public Carbon evaluation pipeline](ht
277
  | | **ClinVar non-coding** | AUROC, AUPRC |
278
  | | **BRCA2** | AUROC, AUPRC, Spearman ρ (centered 8 kb window, full-LL delta) |
279
  | | **TraitGym Mendelian** | AUROC, AUPRC, Spearman ρ |
280
- | Sequence-level perturbation | **TATA-box perturbation v2** | Pairwise discrimination accuracy |
281
- | | **Uniform synonymous-codon substitution v2** | Pairwise discrimination accuracy |
282
- | Long-context retrieval | **Genome-NIAH** (4 task variants × 6 context lengths, up to 786 kbp) | `gen_exact_match`, `ll_correct` |
 
283
 
284
- Below we highlight the three short-context probes for which we report headline numbers in this card. Full results, including all VEP benchmarks and Genome-NIAH heatmaps, are in the Carbon technical report.
285
 
286
  ### Downstream tasks
287
 
@@ -297,7 +297,7 @@ Below we highlight the three short-context probes for which we report headline n
297
 
298
  Carbon-3B is competitive with Evo2-7B while being much faster to run.
299
 
300
- ### Long-context retrieval (Genome-NIAH)
301
 
302
  [Genomic-NIAH](https://huggingface.co/datasets/HuggingFaceBio/genomic-niah) is a long context benchmark, inspired from NIAH and RULER benchmarks for English. The model needs to retrieves a random 24 bp VALUE planted in a real-genome haystack at one of five depths, evaluated at six context lengths from 24 kbp to 786 kbp. The benchmark contains 500 examples per (task, context) cell.
303
 
 
46
  - **Metadata-conditioned**: optional species-type and gene-type metadata tokens enable conditional generation.
47
  - **Efficient inference**: compatible with vLLM and other inference engines. Can generate over 100,000 base-pairs per second on a single H100 GPU.
48
 
49
+ Across our zero-shot evaluation suite, sequence recovery, four variant-effect-prediction (VEP) benchmarks (ClinVar coding, ClinVar non-coding, BRCA2, TraitGym Mendelian), and two sequence-level perturbation tasks (nucleotide triplet-expansion and synonymous codon replacement), Carbon-3B is competitive with Evo2-7B. It additionally works well on long context and retrieves needles reliably from up to ≈ 393 kbp of distal context on the Genomic-NIAH long-context benchmark, while remaining several times faster than Evo2-7B.
 
50
  For full design rationale and ablations, see the Carbon technical report and the [Carbon GitHub repository](https://github.com/huggingface/carbon).
51
 
52
  ## How to use
 
138
  return logp.mean().item()
139
  ```
140
 
141
+ For batched scoring with attention masking and full reproducible evaluation pipelines (sequence recovery, ClinVar / BRCA2 / TraitGym VEP, triplet-expansion / synonymous codon replacement, Genomic-NIAH), use the official scripts in the [Carbon evaluation directory](https://github.com/huggingface/carbon/tree/main/evaluation) — see [`perturbation_tasks.py`](https://github.com/huggingface/carbon/blob/main/evaluation/perturbation_tasks.py) for the canonical `score_hf` implementation and [`README.md`](https://github.com/huggingface/carbon/blob/main/evaluation/README.md) for run instructions across all tasks.
142
 
143
  ### Long context: extending to 65,536 tokens (≈ 393 kbp) with YaRN
144
 
 
276
  | | **ClinVar non-coding** | AUROC, AUPRC |
277
  | | **BRCA2** | AUROC, AUPRC, Spearman ρ (centered 8 kb window, full-LL delta) |
278
  | | **TraitGym Mendelian** | AUROC, AUPRC, Spearman ρ |
279
+ | Sequence-level perturbation | **Nucleotide triplet-expansion** (insert 10 consecutive CAG triplets into a CDS; model should prefer the natural reference) | Pairwise discrimination accuracy |
280
+ | | **Synonymous codon replacement** (replace every codon with the highest-frequency synonym for the target species; model should prefer the natural reference) | Pairwise discrimination accuracy |
281
+ | Long-context retrieval | **Genomic-NIAH** (4 task variants × 6 context lengths, up to 786 kbp) | `gen_exact_match`, `ll_correct` |
282
+
283
 
284
+ Below we highlight the three short-context probes for which we report headline numbers in this card. Full results, including all VEP benchmarks and Genomic-NIAH heatmaps, are in the Carbon technical report.
285
 
286
  ### Downstream tasks
287
 
 
297
 
298
  Carbon-3B is competitive with Evo2-7B while being much faster to run.
299
 
300
+ ### Long-context retrieval (Genomic-NIAH)
301
 
302
  [Genomic-NIAH](https://huggingface.co/datasets/HuggingFaceBio/genomic-niah) is a long context benchmark, inspired from NIAH and RULER benchmarks for English. The model needs to retrieves a random 24 bp VALUE planted in a real-genome haystack at one of five depths, evaluated at six context lengths from 24 kbp to 786 kbp. The benchmark contains 500 examples per (task, context) cell.
303