Carbon-X

Co-authored-by: loubnabnl <loubnabnl@users.noreply.huggingface.co>
Co-authored-by: danaaubakirova <danaaubakirova@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,38 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
+figures/carbon-x-banner.png filter=lfs diff=lfs merge=lfs -text
+figures/partner-logos.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,320 @@
+---
+library_name: transformers
+license: apache-2.0
+base_model: Qwen/Qwen3-14B
+language:
+- en
+tags:
+- dna
+- genomics
+- biology
+- variant-effect-prediction
+- conversational
+pipeline_tag: text-generation
+---
+
+![](figures/carbon-x-banner.png)
+
+<p align="center">
+  <a href="https://huggingface.co/spaces/HuggingFaceBio/Carbon-X-Release"><b>Blog</b></a> |
+  <a href="https://huggingface.co/HuggingFaceBio"><b>Hugging Face Bio</b></a> |
+  <a href="https://github.com/huggingface/carbon/tree/main"><b>Code</b></a>
+</p>
+
+# Carbon-X
+
+An open 14B language model for DNA understanding and generation, from the **Carbon** family.
+
+## Table of Contents
+
+1. [Model Summary](#model-summary)
+2. [How to use](#how-to-use)
+3. [Evaluation](#evaluation)
+4. [Training](#training)
+5. [Using Carbon-X as a tool for an agent](#using-carbon-x-as-a-tool-for-an-agent)
+6. [Limitations](#limitations)
+7. [License](#license)
+8. [Acknowledgements](#acknowledgements)
+
+## Model Summary
+
+<p align="center">
+    <img src="figures/score-vs-cost.png" alt="Average DNA benchmark score versus cost per problem relative to Carbon-X" width="800">
+</p>
+
+<p align="center"><em>Each comparison model is shown at the reasoning effort with its highest average score. Costs use standard API pricing; for Carbon-X, measured throughput per H200-hour at $5/hour.</em></p>
+
+**Carbon-X** is a 14.8B-parameter decoder-only language model trained jointly on DNA sequences and natural-language text. You give it raw DNA inside an ordinary chat conversation and it answers in English: it predicts whether mutations are benign or pathogenic, identifies regions that control gene activity, classifies sequences by organism group, predicts the next bases of a sequence, and explains its reasoning. Responses can contain both DNA and natural language.
+
+DNA and text share a single token stream in a single decoder. There is no separate DNA encoder. A hybrid tokenizer keeps a BPE vocabulary for text and encodes DNA as non-overlapping 6-mers, so one DNA token covers 6 base pairs.
+
+Across five DNA benchmarks, which cover variant-effect prediction, sequence understanding and next-base prediction, Carbon-X averages **66.7%** with thinking disabled. That is 27 points above Qwen3-14B, the model it starts from, and ahead of every other open model we evaluated. It is close to GPT-6.1 Sol (67.5%) at about 1/1,800 of the inference cost per problem, and it leads every model we tested on coding-variant effects and next-base prediction ([Evaluation](#evaluation)). Carbon-X answers directly, without extended reasoning, so each problem costs few output tokens. We estimate that analysing the 2.4 million variants of uncertain significance in ClinVar would cost about $300 with Carbon-X, compared with $32,000 to $4 million with the frontier models we evaluated. The weights fit on a single 80 GB GPU in bfloat16, so researchers can keep sensitive or unpublished sequences on their own infrastructure.
+
+### Key features
+
+- **14.8B parameters**: 40 layers, hidden size 5120, GQA (40 query heads, 8 KV heads), SwiGLU, RMSNorm with QK-norm.
+- **Hybrid tokenizer**: the Qwen3 BPE vocabulary for text plus 5,515 DNA tokens. These are 4,096 6-mers, 1,364 tail tokens for sequences whose length is not a multiple of 6, and the `<dna>`, `</dna>` and `<oov>` markers. DNA in a prompt is detected and tagged automatically.
+- **Chat model**: post-trained and evaluated with thinking disabled.
+- **Context window**: 40,960 tokens.
+- **Training**: 302B tokens of mid-training on DNA, biology text, English and instruction data, then supervised fine-tuning and reinforcement learning with verifiable rewards on DNA tasks.
+- **Efficient inference**: serves with vLLM on a single GPU.
+
+## How to use
+
+Carbon-X uses a standard causal LM architecture. The DNA tokenizer is custom code, so pass `trust_remote_code=True` when you load the tokenizer. Carbon-X requires Transformers v5 or later.
+
+```bash
+pip install -U transformers
+```
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "HuggingFaceBio/Carbon-X"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda")
+
+messages = [
+    {
+        "role": "user",
+        "content": (
+            "Reference: ATGGTGCATCTGACTCCTGAGGAGAAGTCTGCCGTTACTGCCCTGTGGGGCAAGG\n"
+            "Variant:   ATGGTGCATCTGACTCCTGTGGAGAAGTCTGCCGTTACTGCCCTGTGGGGCAAGG\n\n"
+            "Is this variant likely to be benign or pathogenic? Explain your reasoning."
+        ),
+    }
+]
+inputs = tokenizer.apply_chat_template(
+    messages,
+    add_generation_prompt=True,
+    enable_thinking=False,
+    return_tensors="pt",
+    return_dict=True,
+).to(model.device)
+
+outputs = model.generate(**inputs, max_new_tokens=1024)
+print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
+```
+
+Carbon-X was post-trained and evaluated with `enable_thinking=False`, and we recommend keeping it that way. `generate()` uses the sampling defaults from `generation_config.json` (temperature 0.6, top-p 0.95, top-k 20), which are also the settings we used for every reported result.
+
+### Serving with vLLM
+
+```bash
+pip install -U vllm
+vllm serve HuggingFaceBio/Carbon-X --trust-remote-code
+```
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
+response = client.chat.completions.create(
+    model="HuggingFaceBio/Carbon-X",
+    messages=[{"role": "user", "content": "What does this sequence encode?\n\nATGGTGCATCTGACTCCTGAGGAGAAGTCTGCCGTT"}],
+    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
+)
+print(response.choices[0].message.content)
+```
+
+vLLM also applies the sampling defaults from `generation_config.json` unless you override them in the request.
+
+### Tokenizer: working with DNA inputs
+
+The tokenizer switches between BPE for text and 6-mers for DNA. You can paste sequences directly into a message:
+
+```python
+>>> ids = tokenizer("Describe this sequence: ACGTACGATTCA")["input_ids"]
+>>> tokenizer.convert_ids_to_tokens(ids)
+['Describe', 'Ġthis', 'Ġsequence', ':', 'Ġ', '<dna>', 'ACGTAC', 'GATTCA', '</dna>']
+```
+
+- **DNA is tagged automatically.** Runs of at least 6 A/C/G/T bases are wrapped in `<dna>…</dna>` and tokenized as 6-mers. Runs of at least 12 bases with up to 20% IUPAC ambiguity codes (`N`, `R`, `Y`, …) are tagged too. Lowercase bases are uppercased, and sequences displayed in space-separated blocks are joined. Text you have already wrapped in `<dna>…</dna>` is always tokenized as DNA. To turn off detection for a call, pass `auto_dna_tags=False` to the tokenizer or to `apply_chat_template()`.
+- **Only A, C, G and T form 6-mers.** A 6-mer that contains any other character, such as `N`, becomes a single `<oov>` token. To keep full resolution, filter to canonical bases first.
+- **Any length works.** When a sequence's length is not a multiple of 6, its last 1–5 bases are encoded with dedicated tail tokens, and decoding returns the original sequence exactly.
+- **DNA in responses is tagged.** The model writes DNA inside `<dna>…</dna>`. Decoding with `skip_special_tokens=True` removes the tags.
+- **First load needs Hub access.** On first load, the tokenizer downloads the pinned BPE vocabulary it builds on.
+
+### Prompting tips
+
+- Put the sequence and the question in the same user message.
+- Include any context you already have, such as the gene, strand, coordinates or annotations. Without it, the model may guess at a sequence's identity or function.
+- For variant questions, give the reference and alternate sequences, or a reference sequence with the position and the base change.
+
+## Evaluation
+
+We evaluate Carbon-X on five benchmarks that test whether a model can use DNA knowledge to predict mutation effects, identify biological features and recover DNA bases. Carbon-X results use thinking disabled, the default sampling settings and one sample per problem. Each comparison model is shown at the reasoning effort with its highest average score. The best score on each benchmark is in bold.
+
+| Task | Benchmark | Carbon-X<br><sub>No reasoning</sub> | GPT-6.1 Sol<br><sub>Max</sub> | Opus 5.5<br><sub>Max</sub> | Gemini 4 Argon<br><sub>Low</sub> | GLM-5.3<br><sub>Low</sub> | Kimi K3<br><sub>Low</sub> | DeepSeek V4.1 Flash<br><sub>Low</sub> |
+|---|---|---:|---:|---:|---:|---:|---:|---:|
+| Coding variant effects | BioReason Coding | **74.9** | 71.0 | 70.3 | 6.5 | 52.1 | 53.9 | 51.4 |
+| Complex variant effects | BioReason Non-SNV | 85.1 | 91.7 | **94.8** | 90.3 | 60.1 | 65.3 | 55.9 |
+| DNA understanding | GenomeQA | 63.6 | **67.2** | 62.2 | 63.1 | 44.6 | 52.0 | 49.7 |
+| DNA prediction | NextBase | **39.4** | 32.9 | 1.1 | 31.2 | 9.7 | 30.6 | 27.2 |
+| Single-base variant effects | FrontierVEP | 70.4 | 74.9 | **76.2** | 58.0 | 32.7 | 50.1 | 51.3 |
+| **Average** | | 66.7 | **67.5** | 60.9 | 49.8 | 39.8 | 50.4 | 47.1 |
+
+BioReason and GenomeQA come from the literature; we built FrontierVEP and NextBase.
+
+- **BioReason Coding** (1,018 variants, balanced accuracy): given original and mutated sequences from protein-coding regions, predict whether the mutation is benign or pathogenic. The gene name is removed from the prompt, so the model must work from the sequence alone.
+- **BioReason Non-SNV** (873 variants, balanced accuracy): predict whether insertions, deletions and other changes that are not single-nucleotide variants are benign or pathogenic.
+- **GenomeQA** (2,600 questions, accuracy averaged over six tasks): multiple-choice questions about sequence features: enhancer vs. promoter, splice sites, organism group, histone marks, transcription-factor binding sites and motifs.
+- **NextBase** (600 sequences, per-base accuracy): predict the next DNA bases of a sequence. The score is the fraction of bases predicted correctly.
+- **FrontierVEP** (1,000 variants, accuracy): given a long reference sequence (up to 8,192 bases) and a single-nucleotide variant, predict whether the variant is benign or pathogenic (500 benign, 500 pathogenic).
+
+<details>
+<summary>GenomeQA per-task accuracy</summary>
+
+| Task | Accuracy |
+|---|---:|
+| Enhancer vs. promoter | 70.0 |
+| Splice sites | 29.8 |
+| Organism group (eukaryote classification) | 93.2 |
+| Histone marks | 34.6 |
+| Transcription-factor binding sites | 64.0 |
+| Motifs | 90.0 |
+
+</details>
+
+Carbon-X also remains a capable assistant. On IFEval it reaches 79.3% prompt-level strict accuracy, or 84.0% averaged over the four IFEval metrics.
+
+All training data was decontaminated against these benchmarks with three checks: whole-span DNA overlap, text n-gram overlap and sequence homology search.
+
+## Training
+
+### Model
+
+- **Architecture:** decoder-only Transformer with 40 layers, hidden size 5120, FFN size 17408, 40 attention heads with GQA (8 KV heads), head dimension 128, SwiGLU, RMSNorm with QK-norm and untied embeddings.
+- **Positional embedding:** RoPE with base θ = 1 × 10^6. The context window is 40,960 tokens.
+- **Vocabulary:** 157,184 tokens: 151,669 BPE tokens and 5,515 DNA tokens, including padding to a multiple of 128.
+- **Precision:** bfloat16.
+
+### Vocabulary extension
+
+Carbon-X starts from [Qwen3-14B](https://huggingface.co/Qwen/Qwen3-14B). We add the DNA tokens to its vocabulary with randomly initialised embeddings. We then train for about 10B tokens with the transformer body frozen, so that only the new DNA embeddings learn, before unfreezing the whole model.
+
+### Mid-training
+
+The full model is then trained for **302B tokens** with a sequence length of 8,192 and a global batch size of 512 sequences (≈ 4.2M tokens per step, 72,000 steps). The learning rate follows a cosine schedule from 5e-5 to 5e-6 after 2,000 warmup steps. In our ablations, learning rate was the dominant knob at this stage. A low learning rate preserves English but leaves DNA at chance, and a high one learns DNA but forgets English. The data mixture is:
+
+| Component | Share |
+|---|---:|
+| **Raw DNA** | **55%** |
+| Eukaryotic genomes: other vertebrates 16.2%, mammals 10.3%, invertebrates 7.6%, plants 3.1%, fungi 1.1%, protozoa 0.2% | 38.5% |
+| mRNA (OpenGenome2) | 8.8% |
+| Prokaryotic genomes (OpenGenome2) | 5.5% |
+| mRNA with splice sites and promoters (OpenGenome2) | 2.2% |
+| **English** | **20%** |
+| FineWeb-Edu | 8% |
+| DCLM | 8% |
+| FinePDFs-Edu | 4% |
+| **Biology text** | **15%** |
+| Documents with long DNA sequences (FinePDFs, PubMed, bioRxiv, FineWeb, Biostars) | 7% |
+| Other biology documents from the same sources | 7% |
+| Synthetic descriptions of annotated RefSeq sequences | 1% |
+| **Instruction data** | **10%** |
+| Instruction following and chat | 10% |
+
+### Supervised fine-tuning
+
+The mid-trained model is fine-tuned for 3 epochs on a mixture of 20 dataset subsets. The learning rate follows a cosine schedule from 1e-5 to 1e-6, with a global batch size of 32, examples of up to 9,984 tokens and the loss computed on assistant tokens only. The mixture covers:
+
+- Variant-effect reasoning over several views of the same variant (about 30% of the mixture)
+- Sequence classification questions: enhancers vs. promoters, organism groups, transcription-factor motifs and binding sites
+- Basic DNA operations, plus procedurally generated DNA skills such as counting, searching and editing sequences (about 10%)
+- Sequence recovery and next-base prediction
+- Biology questions from community Q&A forums
+- Instruction-following, format-following and general chat data, which keep the model a good assistant
+
+### Reinforcement learning
+
+The fine-tuned model is then trained with asynchronous GRPO and verifiable rewards in two rounds, with thinking disabled:
+
+1. One epoch over a blend of 98,433 prompts, selected from the fine-tuned model's own pass rates, for 150 steps.
+2. A second round from that checkpoint, on 58,249 prompts re-selected from its rollouts. The released weights come after about 2.2 passes over the blend.
+
+Both rounds use 16 samples per prompt, completions of up to 4,096 tokens, a learning rate of 1e-6, asymmetric clipping (ε = 0.2 / 0.28) and a maximum gradient norm of 0.2. Rewards are task-specific, code-checked verifiers. They cover variant-effect prediction (with a penalty for ungrounded rationales), sequence-feature classification, basic DNA operations, DNA skills, locating positions within sequences, biology multiple-choice questions and instruction following.
+
+### Software
+
+- **Post-training:** [TRL](https://github.com/huggingface/trl) for supervised fine-tuning and asynchronous GRPO, with [vLLM](https://github.com/vllm-project/vllm) for generation.
+
+## Using Carbon-X as a tool for an agent
+
+A general-purpose assistant can call Carbon-X when a question involves DNA. Serve Carbon-X with vLLM
+([Serving with vLLM](#serving-with-vllm)) and wrap it in a function that takes a question and named DNA sequences:
+
+```python
+from openai import OpenAI
+
+carbon = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
+
+
+def ask_carbon(question: str, sequences: dict[str, str], context: str = "") -> str:
+    """Ask Carbon-X a question about one or more named DNA sequences."""
+    parts = [context, question] if context else [question]
+    parts += [f"{name}: <dna>{seq}</dna>" for name, seq in sequences.items()]
+    response = carbon.chat.completions.create(
+        model="HuggingFaceBio/Carbon-X",
+        messages=[{"role": "user", "content": "\n\n".join(parts)}],
+        max_tokens=4096,
+        extra_body={"chat_template_kwargs": {"enable_thinking": False}},
+    )
+    return response.choices[0].message.content
+```
+
+The snippet above is a core call. We provide the options below as examples, each one is self-contained, so pick the one that fits your agent:
+
+- **Any agent with tool calling:** [`examples/agent_tool.py`](examples/agent_tool.py) describes `ask_carbon` as an
+  OpenAI-style tool and runs a minimal agent loop. The agent refers to sequences by name and your code looks up the DNA,
+  because language models often miscopy long sequences.
+- **Inspect:** [`examples/inspect_agent.py`](examples/inspect_agent.py) wraps Carbon-X in an `@tool` for an
+  [Inspect](https://inspect.aisi.org.uk/) `react()` agent, with each sample's DNA kept in metadata.
+- **Coding agents (Claude Code, Codex CLI):** [`skills/carbon-dna/`](skills/carbon-dna/) is a ready-made skill. Its
+  script needs only the Python standard library and has two commands: `ask` (a question about named DNA files) and
+  `compare` (rank equal-length variants by sequence likelihood). You need a GPU with about 80 GB of memory and vLLM
+  (`pip install -U vllm`); the skill itself needs no extra packages or API keys. Start Carbon-X, install the skill and
+  point it at the server:
+
+  ```bash
+  vllm serve HuggingFaceBio/Carbon-X --trust-remote-code              # one 80 GB GPU
+  hf download HuggingFaceBio/Carbon-X --include "skills/*" --local-dir carbon-x
+  cp -r carbon-x/skills/carbon-dna ~/.claude/skills/                  # Codex CLI: ~/.codex/skills/
+  export CARBON_BASE_URL=http://localhost:8000/v1                     # or the address of your vLLM server
+  ```
+
+  Carbon-X runs with vLLM or Transformers. llama.cpp and other GGUF runtimes are not supported: the DNA tokenizer is
+  custom Python code that they cannot load, so DNA would be tokenized incorrectly.
+
+### Tips for agent use
+
+- **Pass DNA by reference.** Keep sequences in files or a lookup table and let the agent name them.
+- **Ask for Carbon-X explicitly.** Agents often answer DNA questions on their own instead of calling a tool. If you want Carbon-X's view, say so in the system prompt.
+- **Leave exact operations to code.** Counting, coordinates and edits are more reliable in Python than in either model. For variants, compute the position and the reference and alternate bases in code and pass them as `context`.
+- **Cap generation length.** For next-base and continuation questions, set `max_tokens` to what you need and a client timeout, since long DNA generations are slow (see [Limitations](#limitations)).
+
+## Limitations
+
+Carbon-X is a starting point for research on connecting DNA and language. It has **not been clinically validated**, and it should not be used to make medical decisions.
+
+- ⚠️ **Genetic data is highly sensitive.** Depending on how you use this model (local download, inference API or endpoints, third-party inference providers, Spaces demos or others), input and output data may be processed or handled differently by different providers or Space owners. Make sure you understand and agree with how your data is handled before using the model.
+- **Exact sequence operations.** The model can miscount bases, confuse coordinates or make incorrect edits, particularly when it locates exact positions in long sequences. For tasks that need exact operations, pair it with tools that perform them.
+- **Missing context and hallucinations.** The model may invent a sequence's identity or function when the prompt does not give enough information. Include known annotations and context where available, and verify its biological claims.
+- **Uneven task coverage.** Splice-site and histone-mark questions remain weak (29.8% and 34.6% on GenomeQA). Carbon-X is also less reliable on short DNA exercises and textbook biology than on its core variant-effect and sequence-recovery tasks.
+- **Long sequence generation.** Quality degrades when the model is asked to generate long DNA sequences. Carbon-X reads and explains existing sequences. It was not trained to design new ones.
+- **Long inputs.** Mid-training used 8,192-token sequences (up to ≈ 49 kbp of DNA), and post-training examples were shorter than 16k tokens. The model accepts up to 40,960 tokens, but we have not evaluated inputs much longer than those seen in training.
+- **Thinking mode.** The chat template supports `enable_thinking=True`, but Carbon-X was post-trained and evaluated with thinking disabled.
+
+## License
+
+Apache 2.0.
+
+## Acknowledgements
+
+Carbon-X is a joint collaboration between the research teams at [Hugging Face](https://huggingface.co), [Zhongguancun Academy](https://www.bza.edu.cn/en/), and the [Telethon Institute of Genetics and Medicine](https://www.tigem.it/)/[University of Naples “Federico II”](https://www.dicmapi.unina.it/).
+
+<p align="center">
+  <img src="figures/partner-logos.png" alt="Hugging Face, Zhongguancun Academy, Telethon Institute of Genetics and Medicine and University of Naples Federico II" width="800">
+</p>

chat_template.jinja

@@ -0,0 +1,91 @@
+{#- Training variant of the Qwen3 chat template (see qwen3.jinja for the original).
+    Modifications vs the original:
+    - Removed the loop.index0 > ns.last_query_index conditional; always include thinking block.
+      This makes the template prefix-preserving for the [user, assistant] → [user, assistant, tool] transition.
+    - Added {% generation %} / {% endgeneration %} around assistant message output to support
+      assistant-only loss masking in SFT training.
+-#}
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = '' %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {{- '<|im_start|>' + message.role + '\n' }}
+        {%- generation %}
+        {{- '<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+        {%- endgeneration %}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- endif %}
+{%- endif %}

config.json

@@ -0,0 +1,75 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": null,
+  "dtype": "bfloat16",
+  "eos_token_id": 151643,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 17408,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 40960,
+  "max_window_layers": 40,
+  "model_type": "qwen3",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 40,
+  "num_key_value_heads": 8,
+  "pad_token_id": 151643,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 1000000,
+    "rope_type": "default"
+  },
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.13.0.dev0",
+  "use_cache": false,
+  "use_sliding_window": false,
+  "vocab_size": 157184
+}

dna_config.json

@@ -0,0 +1,12 @@
+{
+  "k": 6,
+  "dna_start_id": 151669,
+  "dna_vocab_size": 5515,
+  "dna_special_tokens": [
+    "<dna>",
+    "</dna>",
+    "<oov>"
+  ],
+  "auto_dna_tags": true,
+  "tail": "pad_n"
+}

examples/agent_tool.py

@@ -0,0 +1,65 @@
+"""Use Carbon-X as a tool for any chat model with OpenAI-style tool calling.
+
+Serve Carbon-X first: vllm serve HuggingFaceBio/Carbon-X --trust-remote-code
+Then set the agent client and model below and run: python agent_tool.py
+"""
+import json
+
+from openai import OpenAI
+
+carbon = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
+
+
+def ask_carbon(question: str, sequences: dict[str, str], context: str = "") -> str:
+    """Ask Carbon-X a question about one or more named DNA sequences."""
+    parts = [context, question] if context else [question]
+    parts += [f"{name}: <dna>{seq}</dna>" for name, seq in sequences.items()]
+    response = carbon.chat.completions.create(
+        model="HuggingFaceBio/Carbon-X",
+        messages=[{"role": "user", "content": "\n\n".join(parts)}],
+        max_tokens=4096,
+        extra_body={"chat_template_kwargs": {"enable_thinking": False}},
+    )
+    return response.choices[0].message.content
+
+
+sequences = {"reference.fa": "ATGGTGCATCTGACTCCTGAGGAGAAGTCTGCCGTTACTGCCCTGTGGGGCAAGG",
+             "variant.fa": "ATGGTGCATCTGACTCCTGTGGAGAAGTCTGCCGTTACTGCCCTGTGGGGCAAGG"}
+
+carbon_tool = {
+    "type": "function",
+    "function": {
+        "name": "ask_carbon",
+        "description": (
+            "Ask Carbon-X, a DNA language model, a question about DNA sequences: variant effects, "
+            "regulatory elements, organism of origin, sequence features. Refer to sequences by name; "
+            "do not paste DNA. Its answers are model predictions, not verified facts."
+        ),
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "question": {"type": "string", "description": "One self-contained question."},
+                "sequences": {"type": "array", "items": {"type": "string", "enum": list(sequences)}},
+                "context": {"type": "string", "description": "Known gene, coordinates or annotations, if any."},
+            },
+            "required": ["question", "sequences"],
+        },
+    },
+}
+
+agent = OpenAI()  # the agent model's provider
+messages = [
+    {"role": "system", "content": "For questions about DNA, consult Carbon-X with the ask_carbon tool before answering."},
+    {"role": "user", "content": "variant.fa differs from reference.fa by one base. Is the change likely benign or pathogenic?"},
+]
+while True:
+    reply = agent.chat.completions.create(model="<agent model>", messages=messages, tools=[carbon_tool])
+    message = reply.choices[0].message
+    messages.append(message)
+    if not message.tool_calls:
+        break
+    for call in message.tool_calls:
+        args = json.loads(call.function.arguments)
+        answer = ask_carbon(args["question"], {name: sequences[name] for name in args["sequences"]}, args.get("context", ""))
+        messages.append({"role": "tool", "tool_call_id": call.id, "content": answer})
+print(message.content)

examples/inspect_agent.py

@@ -0,0 +1,67 @@
+"""Use Carbon-X as a tool for an Inspect react() agent.
+
+Serve Carbon-X first: vllm serve HuggingFaceBio/Carbon-X --trust-remote-code
+Then run: python inspect_agent.py
+"""
+from inspect_ai import Task, eval, task
+from inspect_ai.agent import as_solver, react
+from inspect_ai.dataset import Sample
+from inspect_ai.scorer import includes
+from inspect_ai.solver import solver
+from inspect_ai.tool import ToolError, tool
+from inspect_ai.util import store
+from openai import AsyncOpenAI
+
+carbon = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
+
+
+@tool
+def ask_carbon():
+    async def execute(question: str, sequences: list[str], context: str = "") -> str:
+        """Ask Carbon-X, a DNA language model, about this task's DNA sequences. Its answers are predictions.
+
+        Args:
+            question: One self-contained question.
+            sequences: Names of the sequences to send, as given in the task.
+            context: Known facts, such as the gene, positions or alleles.
+        """
+        dna = store().get("sequences")
+        if missing := [name for name in sequences if name not in dna]:
+            raise ToolError(f"Unknown sequence names: {missing}. Available: {list(dna)}")
+        parts = [context, question] if context else [question]
+        parts += [f"{name}: <dna>{dna[name]}</dna>" for name in sequences]
+        response = await carbon.chat.completions.create(
+            model="HuggingFaceBio/Carbon-X",
+            messages=[{"role": "user", "content": "\n\n".join(parts)}],
+            max_tokens=4096,
+            extra_body={"chat_template_kwargs": {"enable_thinking": False}},
+        )
+        return response.choices[0].message.content
+
+    return execute
+
+
+@solver
+def load_sequences():
+    async def solve(state, generate):
+        store().set("sequences", state.metadata["sequences"])  # the tool reads DNA from here
+        return state
+
+    return solve
+
+
+@task
+def carbon_agent():
+    sample = Sample(
+        input="Sequence `variant` differs from `reference` at position 20 (A>T). Is the variant benign or pathogenic?",
+        target="pathogenic",
+        metadata={"sequences": {
+            "reference": "ATGGTGCATCTGACTCCTGAGGAGAAGTCTGCCGTTACTGCCCTGTGGGGCAAGG",
+            "variant": "ATGGTGCATCTGACTCCTGTGGAGAAGTCTGCCGTTACTGCCCTGTGGGGCAAGG",
+        }},
+    )
+    agent = react(prompt="Consult Carbon-X with the ask_carbon tool before you answer.", tools=[ask_carbon()])
+    return Task(dataset=[sample], solver=[load_sequences(), as_solver(agent)], scorer=includes())
+
+
+eval(carbon_agent(), model="openai/gpt-5")  # any Inspect model with tool calling as the agent

generation_config.json

@@ -0,0 +1,14 @@
+{
+  "do_sample": true,
+  "eos_token_id": [
+    151643,
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "temperature": 0.6,
+  "top_k": 20,
+  "top_p": 0.95,
+  "transformers_version": "5.13.0.dev0",
+  "trust_remote_code": true
+}

model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7858d72aaf5d22e45e75196c4f7352b7423127329c7d1ece67ec072c980cfe4
+size 24956844000

model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a9f69350110babbfab047b6f18358bdf7b0e1f4790b584c57063ed1d2e1f8d7
+size 4687301152

model.safetensors.index.json

@@ -0,0 +1,451 @@
+{
+  "metadata": {
+    "total_parameters": 14822046720,
+    "total_size": 29644093440
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00001-of-00002.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.32.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.36.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.36.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.36.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.36.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.36.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.36.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.36.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.36.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.36.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.36.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.36.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.37.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.37.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.37.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.37.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.37.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.37.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.37.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.37.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.37.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.37.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.37.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.38.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.38.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.38.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.38.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.38.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.38.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.38.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.38.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.38.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.38.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.38.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.39.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.39.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.39.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.39.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.39.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.39.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.39.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.39.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.39.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.39.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.39.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.norm.weight": "model-00002-of-00002.safetensors"
+  }
+}

skills/carbon-dna/SKILL.md

@@ -0,0 +1,42 @@
+---
+name: carbon-dna
+description: "Consult Carbon-X, a DNA language model, about DNA sequences in files: variant effects, regulatory elements, organism of origin, sequence features, and likelihood comparisons between variants."
+---
+
+# Consult Carbon-X about DNA
+
+Carbon-X is a DNA language model. Use it for biological interpretation of DNA alongside your own analysis. Its answers
+are model predictions, not verified facts or clinical classifications.
+
+`scripts/carbon.py` (next to this file) talks to a server running Carbon-X. It needs only Python 3.10+. The user
+configures the server with `CARBON_BASE_URL` (default `http://localhost:8000/v1`), `CARBON_MODEL` and, if needed,
+`CARBON_API_KEY`. Resolve the script's absolute path from this skill's directory.
+
+Put each sequence in its own file (plain DNA or one FASTA record) and refer to files by name. Do not retype DNA into
+commands: copying long sequences by hand introduces errors.
+
+Ask a question about one or more sequences:
+
+```sh
+python3 /path/to/carbon-dna/scripts/carbon.py ask \
+  --sequence reference=reference.fa --sequence variant=variant.fa \
+  --question "variant differs from reference at one base. Is the change likely benign or pathogenic?" \
+  --context "Position 20 (1-based): A>T. Gene and coordinates, if known."
+```
+
+Rank equal-length variants by how typical Carbon-X finds each sequence (a likelihood, not a pathogenicity score):
+
+```sh
+python3 /path/to/carbon-dna/scripts/carbon.py compare --reference reference.fa --alternative variant=variant.fa
+```
+
+From Python, add `scripts/` to `sys.path` and call `ask(question, {name: dna}, context)` or
+`compare(reference_dna, {name: dna})`.
+
+Guidance:
+
+- Work out exact facts in code first (positions, reference and alternate bases, reading frame, lengths) and pass them
+  as `--context`. Carbon-X can miscount positions in long sequences.
+- Ask one self-contained question per call, and keep any answer choices from the user's question.
+- Carbon-X reads A/C/G/T only. The script trims `N` padding at the ends and refuses other characters.
+- Check `truncated` in the result; for next-base or continuation questions, ask for only the length you need.

skills/carbon-dna/scripts/carbon.py

@@ -0,0 +1,105 @@
+"""Ask Carbon-X about DNA files, or compare DNA variants by model likelihood.
+
+Talks to any OpenAI-compatible server running Carbon-X, such as `vllm serve HuggingFaceBio/Carbon-X --trust-remote-code`.
+Configure it with environment variables:
+    CARBON_BASE_URL  server URL including /v1 (default http://localhost:8000/v1)
+    CARBON_MODEL     served model name (default HuggingFaceBio/Carbon-X)
+    CARBON_API_KEY   bearer token, if the server needs one (for example an HF token for an Inference Endpoint)
+
+Usage:
+    python carbon.py ask --question "..." --sequence reference=ref.fa --sequence variant=alt.fa [--context "..."]
+    python carbon.py compare --reference ref.fa --alternative alt1=alt1.fa [--alternative alt2=alt2.fa]
+Only the Python standard library is needed. Output is JSON.
+"""
+
+import argparse
+import json
+import os
+import re
+import sys
+import urllib.request
+from pathlib import Path
+
+BASE_URL = os.environ.get("CARBON_BASE_URL", "http://localhost:8000/v1").rstrip("/")
+MODEL = os.environ.get("CARBON_MODEL", "HuggingFaceBio/Carbon-X")
+API_KEY = os.environ.get("CARBON_API_KEY", "")
+
+
+def read_dna(path: str) -> str:
+    """Read plain DNA or one FASTA record; keep only A/C/G/T after trimming N padding at the ends."""
+    lines = [line.strip() for line in Path(path).read_text().splitlines() if line.strip()]
+    if lines and lines[0].startswith(">"):
+        lines = lines[1:]
+    if any(line.startswith(">") for line in lines):
+        sys.exit(f"{path}: one sequence per file, please (multi-record FASTA found).")
+    dna = "".join(lines).upper().strip("N")
+    if not re.fullmatch("[ACGT]+", dna):
+        sys.exit(f"{path}: Carbon-X reads A/C/G/T; remove other characters first.")
+    return dna
+
+
+def post(route: str, payload: dict) -> dict:
+    request = urllib.request.Request(f"{BASE_URL}/{route}", json.dumps({"model": MODEL, **payload}).encode(),
+                                     {"Content-Type": "application/json", **({"Authorization": f"Bearer {API_KEY}"} if API_KEY else {})})
+    with urllib.request.urlopen(request, timeout=600) as response:
+        return json.load(response)
+
+
+def ask(question: str, sequences: dict[str, str], context: str = "", max_tokens: int = 4096) -> dict:
+    """Ask a question about named DNA sequences. Returns Carbon-X's answer, a model prediction."""
+    parts = [context, question] if context else [question]
+    parts += [f"{name}: <dna>{dna}</dna>" for name, dna in sequences.items()]
+    response = post("chat/completions", {"messages": [{"role": "user", "content": "\n\n".join(parts)}],
+                                         "max_tokens": max_tokens, "chat_template_kwargs": {"enable_thinking": False}})
+    choice = response["choices"][0]
+    return {"answer": choice["message"]["content"], "truncated": choice["finish_reason"] == "length"}
+
+
+def log_likelihood(dna: str) -> float:
+    """Total log-likelihood (nats) of the DNA tokens of `<dna>{dna}</dna>` under Carbon-X."""
+    response = post("completions", {"prompt": f"<dna>{dna}</dna>", "max_tokens": 1, "temperature": 1.0,
+                                    "prompt_logprobs": 0, "add_special_tokens": False, "return_token_ids": True})
+    choice = response["choices"][0]
+    ids, logprobs = choice["prompt_token_ids"], choice["prompt_logprobs"]
+    n = (len(dna) + 5) // 6  # one token per 6 bases; a shorter tail gets its own token
+    if len(ids) != n + 2:  # <dna>, n DNA tokens, </dna>
+        sys.exit("The server did not tokenize the DNA as expected; is it serving Carbon-X with --trust-remote-code?")
+    return sum(logprobs[i][str(ids[i])]["logprob"] for i in range(1, n + 1))
+
+
+def compare(reference: str, alternatives: dict[str, str]) -> dict:
+    """Rank equal-length alternatives by log-likelihood relative to the reference (higher is more typical DNA).
+
+    This is sequence likelihood, not a calibrated pathogenicity score.
+    """
+    if any(len(dna) != len(reference) for dna in alternatives.values()):
+        sys.exit("compare needs equal-length sequences; for insertions or deletions, use ask.")
+    base = log_likelihood(reference)
+    rows = [{"name": name, "delta_log_likelihood": log_likelihood(dna) - base,
+             "substitutions": sum(a != b for a, b in zip(reference, dna))} for name, dna in alternatives.items()]
+    return {"reference_log_likelihood": base, "alternatives": sorted(rows, key=lambda r: -r["delta_log_likelihood"])}
+
+
+def named(items: list[str]) -> dict[str, str]:
+    pairs = [item.split("=", 1) for item in items]
+    if any(len(pair) != 2 for pair in pairs):
+        sys.exit("Use NAME=FILE for each sequence.")
+    return {name: read_dna(path) for name, path in pairs}
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    sub = parser.add_subparsers(dest="command", required=True)
+    a = sub.add_parser("ask")
+    a.add_argument("--question", required=True)
+    a.add_argument("--sequence", action="append", default=[], metavar="NAME=FILE")
+    a.add_argument("--context", default="")
+    c = sub.add_parser("compare")
+    c.add_argument("--reference", required=True, metavar="FILE")
+    c.add_argument("--alternative", action="append", required=True, metavar="NAME=FILE")
+    args = parser.parse_args()
+    if args.command == "ask":
+        result = ask(args.question, named(args.sequence), args.context)
+    else:
+        result = compare(read_dna(args.reference), named(args.alternative))
+    print(json.dumps(result, indent=2))

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
+size 11422650

tokenizer.py

@@ -0,0 +1,835 @@
+"""
+HybridDNATokenizer: Combines Qwen3 BPE tokenization with DNA 6-mer tokenization.
+
+DNA spans are detected automatically or supplied in <dna>...</dna> tags and
+tokenized as 6-mers. All remaining text uses Qwen3's BPE tokenization.
+
+Supports token_mask for Fine-grained Nucleotide Supervision (FNS):
+  -2: padding token
+  -1: text token (BPE)
+   0: DNA special token (<dna>, </dna>, <oov>)
+  1-5: partial 6-mer token — valid_length real bases at positions [0, valid_length),
+       right-padded with 'A' at positions [valid_length, k) so loss can supervise
+       positions 0..valid_len-1 via pos_mask = (valid_len > pos)
+   6: full 6-mer
+"""
+
+import os
+import json
+import itertools
+import re
+from typing import List, Optional, Tuple, Dict, Union, Any
+
+from transformers import PreTrainedTokenizer, AutoTokenizer, BatchEncoding
+
+# The BPE vocabulary comes from this tokenizer, and its size sets where the DNA ids start. Pin it so that a change on the
+# Hub cannot silently shift the ids that checkpoints were trained with.
+BASE_TOKENIZER = "Qwen/Qwen3-4B-Base"
+BASE_TOKENIZER_REVISION = "906bfd4b4dc7f14ee4320094d8b41684abff8539"
+
+
+class HybridDNATokenizer(PreTrainedTokenizer):
+    """
+    Hybrid tokenizer combining Qwen3 BPE with DNA 6-mer tokenization.
+
+    Explicit <dna>...</dna> regions are always tokenized as 6-mers. By default,
+    untagged DNA spans embedded in text are detected and tagged automatically;
+    pass auto_dna_tags=False to encode() or __call__() to disable detection for
+    one call.
+    """
+
+    model_input_names = ["input_ids", "attention_mask"]
+
+    # Single-nucleotide alphabet for tail="char": standard bases (A C T G U) +
+    # IUPAC ambiguity codes (R Y S W K M B D H V N). These 16 chars match
+    # microsoft/NatureLM-8x7B nlm_tokenizer.py's single-char DNA set and the
+    # NatureLM paper's stated DNA vocabulary size of 16 (arXiv:2502.07527, S2.1).
+    SINGLE_BASE_ALPHABET = "ACTGURYSWKMBDHVN"
+    IUPAC_DNA_ALPHABET = "ACGTRYSWKMBDHVN"
+    STANDARD_DNA_ALPHABET = frozenset("ACGT")
+    IUPAC_RUN_RE = re.compile(f"[{IUPAC_DNA_ALPHABET}]+", re.IGNORECASE | re.ASCII)
+    SPACED_DNA_RE = re.compile(
+        r"(?<![A-Za-z])(?:[ACGT]{1,5}[ \t]+){5,}[ACGT]{1,5}(?![A-Za-z])"
+    )
+
+    def __init__(
+        self,
+        base_tokenizer_path: Optional[str] = None,
+        k: int = 6,
+        auto_dna_tags: bool = True,
+        tail: str = "pad_a",
+        **kwargs
+    ):
+        if tail not in ("pad_a", "pad_n", "char"):
+            raise ValueError(f"tail must be 'pad_a', 'pad_n' or 'char', got {tail!r}")
+        self.tail = tail
+        self.pad_char = {"pad_a": "A", "pad_n": "N"}.get(tail)
+        self.k = k
+
+        # Load base tokenizer (Qwen3-4B-Base) at the pinned revision
+        self._base_tokenizer = AutoTokenizer.from_pretrained(BASE_TOKENIZER, revision=BASE_TOKENIZER_REVISION)
+
+        # Get base vocabulary
+        self._base_vocab = self._base_tokenizer.get_vocab()
+        self._base_vocab_size = len(self._base_vocab)
+
+        # Initialize DNA vocabulary
+        self._init_dna_vocab()
+
+        # Build combined vocabulary
+        self._build_combined_vocab()
+
+        # Set special tokens
+        self._eos_token = kwargs.pop('eos_token', None) or "<|endoftext|>"
+        self._pad_token = kwargs.pop('pad_token', None) or self._base_tokenizer.pad_token or "<|endoftext|>"
+
+        # Initialize parent class
+        super().__init__(
+            eos_token=self._eos_token,
+            pad_token=self._pad_token,
+            **kwargs
+        )
+
+        # Preserve the base tokenizer's AddedToken metadata. These tokens are
+        # already present in the combined vocabulary at their original IDs, so
+        # this only registers their metadata and does not grow the vocabulary.
+        self.add_tokens(list(self._base_tokenizer.added_tokens_decoder.values()))
+
+        self.special_tokens = self.dna_special_tokens + [self._eos_token, self._pad_token]
+        self.auto_dna_tags = auto_dna_tags
+
+    def _init_dna_vocab(self):
+        """Initialize DNA vocabulary (special tokens + k-mers + padding for 128 alignment)."""
+        bases = ['A', 'T', 'C', 'G']
+
+        # DNA special tokens
+        self.dna_special_tokens = ["<dna>", "</dna>", "<oov>"]
+
+        # Generate all k-mer combinations (4^k = 4096 for k=6)
+        self.kmers = [''.join(kmer) for kmer in itertools.product(bases, repeat=self.k)]
+
+        # tail="pad_n": dedicated tokens for the N-padded tail ("CT" -> "CTNNNN"),
+        # one per real prefix of length 1..k-1, i.e. 4^1+...+4^(k-1) = 1364.
+        self.npad_tokens = []
+        if self.tail == "pad_n":
+            for prefix_len in range(1, self.k):
+                for prefix in itertools.product(bases, repeat=prefix_len):
+                    self.npad_tokens.append(''.join(prefix) + 'N' * (self.k - prefix_len))
+
+        # tail="char": single-base tokens for the trailing < k bases (NatureLM
+        # IUPAC alphabet), so N / ambiguity codes fall back to a token, not <oov>.
+        self.base_tokens = []
+        self.dna_base_token = {}         # base -> token, e.g. "A" -> "<b_A>"
+        self.dna_base_token_to_base = {}  # token -> base
+        if self.tail == "char":
+            for b in self.SINGLE_BASE_ALPHABET:
+                tok = f"<b_{b}>"
+                self.base_tokens.append(tok)
+                self.dna_base_token[b] = tok
+                self.dna_base_token_to_base[tok] = b
+
+        # DNA tokens start after base vocabulary
+        self.dna_start_id = self._base_vocab_size
+
+        # All DNA tokens get new IDs (no reuse of base vocab IDs, even for
+        # overlapping tokens like CCCCCC — they have different semantics in
+        # DNA context vs BPE context, per Qiuyi's recommendation). Tail-variant
+        # tokens are listed AFTER the 4^k k-mers so the k-mer ids are unchanged.
+        base_dna_tokens = self.dna_special_tokens + self.kmers + self.npad_tokens + self.base_tokens
+
+        # Calculate padding for 128 alignment
+        total_vocab_unpadded = self._base_vocab_size + len(base_dna_tokens)
+        target_vocab_size = ((total_vocab_unpadded + 127) // 128) * 128
+        num_padding_tokens = target_vocab_size - total_vocab_unpadded
+
+        # Add unused padding tokens
+        self.padding_tokens = [f"<unused_{i}>" for i in range(num_padding_tokens)]
+
+        # Create DNA token mappings — all get sequential new IDs
+        self.dna_token_to_id = {}
+        self.dna_id_to_token = {}
+
+        current_id = self.dna_start_id
+        for token in base_dna_tokens:
+            self.dna_token_to_id[token] = current_id
+            self.dna_id_to_token[current_id] = token
+            current_id += 1
+
+        # Add padding tokens
+        for token in self.padding_tokens:
+            self.dna_token_to_id[token] = current_id
+            self.dna_id_to_token[current_id] = token
+            current_id += 1
+
+        self.dna_vocab_size = len(base_dna_tokens) + len(self.padding_tokens)
+
+        # Set DNA special token IDs
+        self.dna_begin_token_id = self.dna_token_to_id["<dna>"]
+        self.dna_end_token_id = self.dna_token_to_id["</dna>"]
+        self.oov_token_id = self.dna_token_to_id["<oov>"]
+
+    def _build_combined_vocab(self):
+        """Build combined vocabulary (base + DNA)."""
+        self._vocab = self._base_vocab.copy()
+
+        for token, token_id in self.dna_token_to_id.items():
+            if token not in self._vocab:
+                self._vocab[token] = token_id
+
+        self._id_to_token = {v: k for k, v in self._vocab.items()}
+        for token_id, token in self.dna_id_to_token.items():
+            if token_id not in self._id_to_token:
+                self._id_to_token[token_id] = token
+
+    @property
+    def vocab_size(self) -> int:
+        return max(self._vocab.values()) + 1
+
+    def get_vocab(self) -> Dict[str, int]:
+        return self._vocab.copy()
+
+    def __len__(self):
+        # Override default (len(get_vocab())) because get_vocab() deduplicates
+        # CCCCCC which exists as both BPE (ID 91443) and DNA 6-mer (ID 154402).
+        return self.vocab_size
+
+    @classmethod
+    def _auto_tag_text_segment(cls, text: str) -> str:
+        """Tag conservative DNA candidates in a text-only segment.
+
+        Each maximal IUPAC nucleotide run is considered independently. Pure
+        A/C/G/T runs need at least six bases. Runs containing IUPAC ambiguity
+        codes need at least twelve bases and may contain at most 20% ambiguity.
+        Rejected ambiguous runs stay wholly on the BPE path rather than having
+        any A/C/G/T substring tagged.
+        """
+        parts = []
+        previous_end = 0
+
+        for match in cls.IUPAC_RUN_RE.finditer(text):
+            candidate = match.group(0)
+            normalized = candidate.upper()
+            ambiguous_count = sum(base not in cls.STANDARD_DNA_ALPHABET for base in normalized)
+            is_standard_dna = ambiguous_count == 0 and len(normalized) >= 6
+            is_conservative_iupac = (
+                ambiguous_count > 0
+                and len(normalized) >= 12
+                and ambiguous_count / len(normalized) <= 0.2
+            )
+
+            if not (is_standard_dna or is_conservative_iupac):
+                continue
+
+            parts.append(text[previous_end:match.start()])
+            parts.append(f"<dna>{normalized}</dna>")
+            previous_end = match.end()
+
+        if not parts:
+            return text
+
+        parts.append(text[previous_end:])
+        return "".join(parts)
+
+    @classmethod
+    def _normalize_spaced_dna(cls, text: str) -> str:
+        """Join uppercase DNA displays with at least six chunks and twelve bases.
+
+        Chunks contain one to five A/C/G/T bases, separated only by spaces or
+        tabs. Alphabetic boundaries prevent joining fragments of words.
+        """
+        def normalize(match):
+            sequence = "".join(match.group(0).split())
+            return sequence if len(sequence) >= 12 else match.group(0)
+
+        return cls.SPACED_DNA_RE.sub(normalize, text)
+
+    def _add_auto_dna_tags(self, text: str) -> str:
+        """Tag DNA outside existing <dna>...</dna> regions."""
+        return "".join(
+            segment if is_dna else self._auto_tag_text_segment(self._normalize_spaced_dna(segment))
+            for segment, is_dna in self._split_by_dna_tags(text)
+        )
+
+    def _split_by_dna_tags(self, text: str) -> List[Tuple[str, bool]]:
+        segments = []
+        i = 0
+        n = len(text)
+
+        while i < n:
+            start_pos = text.find('<dna>', i)
+            end_pos = text.find('</dna>', i)
+
+            if start_pos == -1 and end_pos == -1:
+                remaining = text[i:]
+                if remaining:
+                    segments.append((remaining, False))
+                break
+
+            if start_pos == -1 and end_pos != -1:
+                dna_region = text[i:end_pos + 6]
+                if dna_region:
+                    segments.append((dna_region, True))
+                i = end_pos + 6
+                continue
+
+            if start_pos != -1 and end_pos == -1:
+                if i < start_pos:
+                    normal_text = text[i:start_pos]
+                    if normal_text:
+                        segments.append((normal_text, False))
+                dna_region = text[start_pos:]
+                if dna_region:
+                    segments.append((dna_region, True))
+                break
+
+            if start_pos < end_pos:
+                if i < start_pos:
+                    normal_text = text[i:start_pos]
+                    if normal_text:
+                        segments.append((normal_text, False))
+                dna_region = text[start_pos:end_pos + 6]
+                if dna_region:
+                    segments.append((dna_region, True))
+                i = end_pos + 6
+            else:
+                dna_region = text[i:end_pos + 6]
+                if dna_region:
+                    segments.append((dna_region, True))
+                i = end_pos + 6
+
+        return segments
+
+    def _parse_dna_region(self, dna_region: str) -> Tuple[str, bool, bool]:
+        if dna_region == '<dna>':
+            return '', True, False
+        elif dna_region == '</dna>':
+            return '', False, True
+
+        has_start = dna_region.startswith('<dna>')
+        has_end = dna_region.endswith('</dna>')
+
+        content = dna_region
+        if has_start:
+            content = content[5:]
+        if has_end and content.endswith('</dna>'):
+            content = content[:-6]
+
+        return content.strip(), has_start, has_end
+
+    def _process_dna_sequence(self, dna_seq: str) -> Dict:
+        k = self.k
+        dna_seq = dna_seq.upper()
+
+        kmer_tokens = []
+        valid_bases = set('ATCG')
+
+        def is_valid_kmer(kmer):
+            return len(kmer) == k and all(base in valid_bases for base in kmer)
+
+        if self.tail == "char":
+            # Greedy NatureLM tokenization (NatureLM, arXiv:2502.07527), ported
+            # from microsoft/NatureLM-8x7B nlm_tokenizer.py::_tokenize_entity:
+            # take a full k-mer when the next k chars form one, else emit a single
+            # base token and advance by 1 (also re-aligns after any non-base char).
+            i = 0
+            n = len(dna_seq)
+            while i < n:
+                window = dna_seq[i:i + k]
+                if is_valid_kmer(window):
+                    kmer_tokens.append(window)
+                    i += k
+                else:
+                    base = dna_seq[i]
+                    kmer_tokens.append(self.dna_base_token.get(base, "<oov>"))
+                    i += 1
+            return {"kmer_tokens": kmer_tokens, "padding_length": 0, "valid_length": k}
+
+        for i in range(0, len(dna_seq) - k + 1, k):
+            kmer = dna_seq[i:i+k]
+            if is_valid_kmer(kmer):
+                kmer_tokens.append(kmer)
+            else:
+                kmer_tokens.append("<oov>")
+
+        processed_length = len(kmer_tokens) * k
+        remaining = dna_seq[processed_length:]
+        padding_length = 0
+        valid_length = k
+
+        if remaining:
+            padding_needed = k - len(remaining)
+            # Right-pad up to a full k-mer: real bases occupy positions
+            # [0, valid_length). The hybrid BP loss supervises positions
+            # 0..valid_len-1 via pos_mask = (valid_len > pos), so padding must be
+            # at the END. pad_char="A" reuses a real k-mer id ("CT"->"CTAAAA");
+            # "N" maps to a dedicated tail token ("CT"->"CTNNNN").
+            padded = remaining + self.pad_char * padding_needed
+
+            if all(base in valid_bases for base in remaining):
+                kmer_tokens.append(padded)
+            else:
+                kmer_tokens.append("<oov>")
+
+            padding_length = padding_needed
+            valid_length = len(remaining)
+
+        return {
+            "kmer_tokens": kmer_tokens,
+            "padding_length": padding_length,
+            "valid_length": valid_length,
+        }
+
+    def _tokenize(self, text: str, **kwargs) -> List[str]:
+        return list(text)
+
+    def _convert_token_to_id(self, token: str) -> int:
+        if token in self.dna_token_to_id:
+            return self.dna_token_to_id[token]
+        return self._base_vocab.get(token, self._base_tokenizer.unk_token_id or 0)
+
+    def _convert_id_to_token(self, index: int) -> str:
+        if index in self.dna_id_to_token:
+            return self.dna_id_to_token[index]
+        return self._id_to_token.get(index, "<oov>")
+
+    def convert_tokens_to_string(self, tokens: List[str]) -> str:
+        ids = [self._convert_token_to_id(t) for t in tokens]
+        return self.decode(ids, skip_special_tokens=False)
+    def encode(
+        self,
+        text: str,
+        add_special_tokens: bool = False,
+        return_token_mask: bool = False,
+        auto_dna_tags: Optional[bool] = None,
+        **kwargs
+    ) -> Union[List[int], Tuple[List[int], List[int]]]:
+        use_auto = self.auto_dna_tags if auto_dna_tags is None else auto_dna_tags
+        if use_auto:
+            text = self._add_auto_dna_tags(text)
+
+        segments = self._split_by_dna_tags(text)
+
+        token_ids = []
+        token_mask = [] if return_token_mask else None
+
+        for segment_content, is_dna in segments:
+            if is_dna:
+                dna_content, has_start, has_end = self._parse_dna_region(segment_content)
+
+                if has_start:
+                    token_ids.append(self.dna_begin_token_id)
+                    if return_token_mask:
+                        token_mask.append(0)
+
+                if dna_content:
+                    result = self._process_dna_sequence(dna_content)
+
+                    for idx, kmer in enumerate(result["kmer_tokens"]):
+                        token_id = self.dna_token_to_id.get(kmer, self.oov_token_id)
+                        token_ids.append(token_id)
+
+                        if return_token_mask:
+                            if kmer == "<oov>":
+                                token_mask.append(0)
+                            elif kmer in self.dna_base_token_to_base:
+                                token_mask.append(1)  # char-fallback single base
+                            elif idx == len(result["kmer_tokens"]) - 1 and result["padding_length"] > 0:
+                                token_mask.append(result["valid_length"])
+                            else:
+                                token_mask.append(self.k)
+
+                if has_end:
+                    token_ids.append(self.dna_end_token_id)
+                    if return_token_mask:
+                        token_mask.append(0)
+            else:
+                base_ids = self._base_tokenizer.encode(
+                    segment_content,
+                    add_special_tokens=False
+                )
+                token_ids.extend(base_ids)
+                if return_token_mask:
+                    token_mask.extend([-1] * len(base_ids))
+
+        # Do NOT append EOS when add_special_tokens=True. Qwen3 doesn't add
+        # BOS/EOS either, and appending EOS here breaks lighteval's
+        # tok_encode_pair: it relies on
+        #   len(encode(ctx)) + len(encode(answer)) == len(encode(ctx + answer))
+        # which the extra EOS violates by shifting the split by 1.
+
+        if return_token_mask:
+            return token_ids, token_mask
+        return token_ids
+
+    def _decode_dna_token(self, token_str: str) -> str:
+        # Surface text for a DNA-vocab token. Baseline (pad_a) returns it
+        # unchanged; char maps "<b_A>"->"A"; pad_n strips the N-pad tail fill
+        # ("CTNNNN"->"CT"). Real k-mers contain no N, so rstrip is a no-op there.
+        if token_str in self.dna_base_token_to_base:
+            return self.dna_base_token_to_base[token_str]
+        if self.tail == "pad_n" and len(token_str) == self.k and token_str[0] in "ATCG":
+            return token_str.rstrip("N")
+        return token_str
+
+    def decode(
+        self,
+        token_ids: Union[int, List[int]],
+        skip_special_tokens: bool = False,
+        **kwargs
+    ) -> str:
+        # A raw tensor/ndarray must become plain ints: dna_id_to_token is keyed by
+        # python ints and a 0-dim tensor scalar hashes by identity, so DNA lookups
+        # silently miss and the bases are dropped (e.g. decode(generate(...)[0])).
+        if hasattr(token_ids, "tolist"):
+            token_ids = token_ids.tolist()
+        if isinstance(token_ids, int):
+            token_ids = [token_ids]
+
+        if skip_special_tokens:
+            special_ids = {self.eos_token_id, self.pad_token_id}
+            token_ids = [tid for tid in token_ids if tid not in special_ids]
+
+        parts = []
+        i = 0
+
+        while i < len(token_ids):
+            tid = token_ids[i]
+
+            if tid == self.dna_begin_token_id:
+                dna_tokens = []
+                i += 1
+
+                while i < len(token_ids) and token_ids[i] != self.dna_end_token_id:
+                    if token_ids[i] in self.dna_id_to_token:
+                        dna_tokens.append(self._decode_dna_token(self.dna_id_to_token[token_ids[i]]))
+                    i += 1
+
+                dna_seq = ''.join(dna_tokens)
+
+                if skip_special_tokens:
+                    parts.append(dna_seq)
+                else:
+                    parts.append(f"<dna>{dna_seq}")
+                    if i < len(token_ids) and token_ids[i] == self.dna_end_token_id:
+                        parts.append("</dna>")
+                        i += 1
+
+            elif tid in self.dna_id_to_token:
+                # This branch handles k-mer tokens that appear without a <dna>
+                # wrapper — the common generation case where <dna> was in the
+                # prompt but only the generated portion is being decoded.
+                # K-mer tokens are content, not special tokens, so always decode
+                # them.  Only drop true DNA special tokens (<dna>, </dna>, <oov>)
+                # when skip_special_tokens=True.
+                is_dna_special = tid in (self.dna_begin_token_id, self.dna_end_token_id, self.oov_token_id)
+                if not (skip_special_tokens and is_dna_special):
+                    parts.append(self._decode_dna_token(self.dna_id_to_token[tid]))
+                i += 1
+
+            else:
+                text_ids = []
+                while i < len(token_ids):
+                    curr_id = token_ids[i]
+                    if curr_id in self.dna_id_to_token or curr_id == self.dna_begin_token_id:
+                        break
+                    text_ids.append(curr_id)
+                    i += 1
+
+                if text_ids:
+                    decoded = self._base_tokenizer.decode(text_ids, skip_special_tokens=skip_special_tokens)
+                    parts.append(decoded)
+
+        return ''.join(parts)
+
+    def batch_decode(
+        self,
+        sequences: Union[List[int], List[List[int]], "torch.Tensor"],
+        skip_special_tokens: bool = False,
+        **kwargs
+    ) -> List[str]:
+        return [
+            self.decode(
+                seq.tolist() if hasattr(seq, 'tolist') else list(seq),
+                skip_special_tokens=skip_special_tokens,
+                **kwargs
+            )
+            for seq in sequences
+        ]
+
+    def __call__(
+        self,
+        text: Union[str, List[str]],
+        add_special_tokens: bool = False,
+        padding: bool = False,
+        truncation: bool = False,
+        max_length: Optional[int] = None,
+        return_tensors: Optional[str] = None,
+        return_token_mask: bool = False,
+        auto_dna_tags: Optional[bool] = None,
+        **kwargs
+    ) -> Dict[str, Any]:
+        is_batch = isinstance(text, list)
+        texts = text if is_batch else [text]
+
+        all_ids = []
+        all_masks = [] if return_token_mask else None
+
+        for t in texts:
+            if return_token_mask:
+                ids, mask = self.encode(t, add_special_tokens=add_special_tokens, return_token_mask=True, auto_dna_tags=auto_dna_tags)
+                all_ids.append(ids)
+                all_masks.append(mask)
+            else:
+                ids = self.encode(t, add_special_tokens=add_special_tokens, return_token_mask=False, auto_dna_tags=auto_dna_tags)
+                all_ids.append(ids)
+
+        if padding:
+            max_len = max(len(ids) for ids in all_ids)
+            if max_length:
+                max_len = min(max_len, max_length)
+
+            padded_ids = []
+            attention_masks = []
+            padded_token_masks = [] if return_token_mask else None
+
+            for idx, ids in enumerate(all_ids):
+                pad_len = max_len - len(ids)
+
+                if pad_len > 0:
+                    ids = ids + [self.pad_token_id] * pad_len
+                    attn = [1] * (max_len - pad_len) + [0] * pad_len
+                    if return_token_mask:
+                        mask = all_masks[idx] + [-2] * pad_len
+                else:
+                    ids = ids[:max_len]
+                    attn = [1] * max_len
+                    if return_token_mask:
+                        mask = all_masks[idx][:max_len]
+
+                padded_ids.append(ids)
+                attention_masks.append(attn)
+                if return_token_mask:
+                    padded_token_masks.append(mask)
+
+            all_ids = padded_ids
+            all_masks = padded_token_masks
+        else:
+            attention_masks = [[1] * len(ids) for ids in all_ids]
+
+        result = {
+            "input_ids": all_ids if is_batch else all_ids[0],
+            "attention_mask": attention_masks if is_batch else attention_masks[0],
+        }
+
+        if return_token_mask:
+            result["token_mask"] = all_masks if is_batch else all_masks[0]
+
+        if return_tensors == "pt":
+            import torch
+            if is_batch:
+                result["input_ids"] = torch.tensor(result["input_ids"])
+                result["attention_mask"] = torch.tensor(result["attention_mask"])
+                if return_token_mask:
+                    result["token_mask"] = torch.tensor(result["token_mask"])
+            else:
+                result["input_ids"] = torch.tensor([result["input_ids"]])
+                result["attention_mask"] = torch.tensor([result["attention_mask"]])
+                if return_token_mask:
+                    result["token_mask"] = torch.tensor([result["token_mask"]])
+
+        return BatchEncoding(result, tensor_type=return_tensors)
+
+    # Markers delimiting an assistant turn, used to build the assistant-token mask.
+    # Qwen chat format; override on the instance for other bases (e.g. Gemma uses
+    # "<start_of_turn>model\n" / "<end_of_turn>").
+    ASSISTANT_HEADER = "<|im_start|>assistant\n"
+    TURN_END = "<|im_end|>\n"
+
+    def apply_chat_template(self, conversation, *args, return_assistant_tokens_mask: bool = False, **kwargs):
+        """Completion-only assistant mask WITHOUT a fast tokenizer.
+
+        HF derives the assistant mask from ``{% generation %}`` char-spans via
+        ``BatchEncoding.char_to_token()``, which only fast (Rust) tokenizers implement.
+        HybridDNATokenizer is a slow Python tokenizer (the DNA k-mer logic can't be a
+        ``tokenizer.json``), so that path raises
+        ``char_to_token() is not available when using Python based tokenizers``.
+
+        Instead we tokenize normally and mark the tokens of every assistant turn -- from
+        just after ``ASSISTANT_HEADER`` up to and including the next ``TURN_END``. This
+        makes ``return_assistant_tokens_mask=True`` (and thus TRL ``assistant_only_loss``)
+        work unchanged, and needs no ``{% generation %}`` markers in the template.
+        """
+        auto_dna_tags = kwargs.pop("auto_dna_tags", None)
+        if auto_dna_tags is not None:
+            tokenizer_kwargs = dict(kwargs.pop("tokenizer_kwargs", None) or {})
+            tokenizer_kwargs["auto_dna_tags"] = auto_dna_tags
+            kwargs["tokenizer_kwargs"] = tokenizer_kwargs
+
+        if not return_assistant_tokens_mask:
+            return super().apply_chat_template(conversation, *args, **kwargs)
+        kwargs.pop("return_assistant_tokens_mask", None)
+        want_tensors = kwargs.pop("return_tensors", None)
+        kwargs["tokenize"] = True
+        kwargs["return_dict"] = True
+        enc = super().apply_chat_template(conversation, *args, return_tensors=None, **kwargs)
+        seqs = enc["input_ids"]
+        batched = len(seqs) > 0 and isinstance(seqs[0], (list, tuple))
+        batch = seqs if batched else [seqs]
+        hdr = self(self.ASSISTANT_HEADER, add_special_tokens=False)["input_ids"]
+        end = self(self.TURN_END, add_special_tokens=False)["input_ids"]
+        masks = []
+        for ids in batch:
+            ids = list(ids)
+            m = [0] * len(ids)
+            i = 0
+            while i <= len(ids) - len(hdr):
+                if ids[i:i + len(hdr)] == hdr:
+                    j = i + len(hdr)
+                    while j < len(ids) and ids[j:j + len(end)] != end:
+                        m[j] = 1
+                        j += 1
+                    for k in range(len(end)):          # include the turn-end token(s)
+                        if j + k < len(ids):
+                            m[j + k] = 1
+                    i = j + len(end)
+                else:
+                    i += 1
+            masks.append(m)
+        enc["assistant_masks"] = masks if batched else masks[0]
+        if want_tensors is not None:
+            enc = enc.convert_to_tensors(tensor_type=want_tensors)
+        return enc
+
+    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
+        vocab_file = os.path.join(
+            save_directory,
+            (filename_prefix + "-" if filename_prefix else "") + "vocab.json"
+        )
+
+        with open(vocab_file, "w", encoding="utf-8") as f:
+            json.dump(self._vocab, f, ensure_ascii=False, indent=2)
+
+        return (vocab_file,)
+
+    def save_pretrained(self, save_directory: str, **kwargs):
+        os.makedirs(save_directory, exist_ok=True)
+
+        # Save base tokenizer files with the wrapper's effective chat template.
+        self._base_tokenizer.chat_template = self.chat_template
+        self._base_tokenizer.save_pretrained(save_directory)
+
+        # Save DNA config
+        dna_config = {
+            "k": self.k,
+            "dna_start_id": self.dna_start_id,
+            "dna_vocab_size": self.dna_vocab_size,
+            "dna_special_tokens": self.dna_special_tokens,
+            "auto_dna_tags": self.auto_dna_tags,
+            "tail": self.tail,
+        }
+
+        dna_config_path = os.path.join(save_directory, "dna_config.json")
+        with open(dna_config_path, "w", encoding="utf-8") as f:
+            json.dump(dna_config, f, indent=2)
+
+        # Update tokenizer_config.json with auto_map
+        config_path = os.path.join(save_directory, "tokenizer_config.json")
+
+        if os.path.exists(config_path):
+            with open(config_path, "r") as f:
+                config = json.load(f)
+        else:
+            config = {}
+
+        config.update({
+            "tokenizer_class": "HybridDNATokenizer",
+            "auto_map": {
+                "AutoTokenizer": ["tokenizer.HybridDNATokenizer", None]
+            },
+            "k": self.k,
+            "auto_dna_tags": self.auto_dna_tags,
+            "tail": self.tail,
+        })
+
+        with open(config_path, "w", encoding="utf-8") as f:
+            json.dump(config, f, indent=2, ensure_ascii=False)
+
+        # Copy this tokenizer.py to save directory
+        import shutil
+        src_py = os.path.abspath(__file__)
+        dst_py = os.path.join(save_directory, "tokenizer.py")
+        if os.path.exists(src_py) and src_py != dst_py:
+            shutil.copy2(src_py, dst_py)
+
+        return (save_directory,)
+
+    @classmethod
+    def from_pretrained(
+        cls,
+        pretrained_model_name_or_path,
+        *args,
+        cache_dir=None,
+        force_download=False,
+        local_files_only=False,
+        token=None,
+        revision="main",
+        trust_remote_code=False,
+        **kwargs,
+    ):
+        # Custom loader: PreTrainedTokenizer.from_pretrained is incompatible with this
+        # tokenizer's __init__ (it re-passes bos_token etc.), so construct directly from
+        # bio_config/dna_config. That bypasses standard metadata loading, so also read
+        # chat_template from tokenizer_config.json (vLLM / apply_chat_template need it).
+        from transformers.utils.hub import cached_file
+
+        proxies = kwargs.pop("proxies", None)
+        subfolder = kwargs.pop("subfolder", None)
+        commit_hash = kwargs.pop("_commit_hash", None)
+        hub_kwargs = {
+            "cache_dir": cache_dir,
+            "force_download": force_download,
+            "local_files_only": local_files_only,
+            "proxies": proxies,
+            "revision": revision,
+            "subfolder": subfolder,
+            "token": token,
+            "_commit_hash": commit_hash,
+            "_raise_exceptions_for_missing_entries": False,
+        }
+
+        cfg = {}
+        for _name in ("bio_config.json", "dna_config.json"):
+            _p = cached_file(pretrained_model_name_or_path, _name, **hub_kwargs)
+            if _p is not None:
+                with open(_p, encoding="utf-8") as _f:
+                    cfg = json.load(_f)
+                break
+
+        _init = {"base_tokenizer_path": pretrained_model_name_or_path}
+        for _key in ("k", "tail", "auto_dna_tags"):
+            if _key in cfg:
+                _init[_key] = cfg[_key]
+
+        tokenizer_config_path = cached_file(pretrained_model_name_or_path, "tokenizer_config.json", **hub_kwargs)
+        if tokenizer_config_path is not None:
+            with open(tokenizer_config_path, encoding="utf-8") as _f:
+                tokenizer_config = json.load(_f)
+            if tokenizer_config.get("chat_template"):
+                _init["chat_template"] = tokenizer_config["chat_template"]
+
+        chat_template_path = cached_file(pretrained_model_name_or_path, "chat_template.jinja", **hub_kwargs)
+        if chat_template_path is not None:
+            with open(chat_template_path, encoding="utf-8") as _f:
+                _init["chat_template"] = _f.read()
+
+        _init.update(kwargs)
+        tokenizer = cls(**_init)
+        # DNA ids start right after the base vocabulary, so a layout that differs from the saved one would misalign every
+        # DNA embedding. Fail instead of tokenizing with shifted ids.
+        for _key in ("dna_start_id", "dna_vocab_size"):
+            if _key in cfg and cfg[_key] != getattr(tokenizer, _key):
+                raise ValueError(
+                    f"{pretrained_model_name_or_path} was saved with {_key}={cfg[_key]}, but the base tokenizer gives "
+                    f"{getattr(tokenizer, _key)}, so its DNA token ids would not match the model."
+                )
+        return tokenizer

tokenizer_config.json

@@ -0,0 +1,39 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "HybridDNATokenizer",
+  "unk_token": null,
+  "auto_map": {
+    "AutoTokenizer": [
+      "tokenizer.HybridDNATokenizer",
+      null
+    ]
+  },
+  "k": 6,
+  "auto_dna_tags": true,
+  "tail": "pad_n"
+}