Blog | Carbon-X | Hugging Face Bio | Code

Carbon-X-Base

The mid-trained base model behind Carbon-X: Qwen3-14B after learning DNA as a new modality over 302B tokens of DNA and text, before any task-specific post-training.

Table of Contents

1. Model Summary
2. How to use
3. Evaluation
4. Training
5. Limitations
6. License
7. Acknowledgements

Model Summary

Carbon-X-Base is a 14.8B-parameter decoder-only language model trained jointly on DNA sequences and natural-language text. It starts from Qwen3-14B, extends its vocabulary with DNA tokens, and is then trained on a 302B-token mixture that is 55% raw DNA, 15% biology text, 20% English and 10% instruction data. This is the checkpoint we fine-tuned and trained with reinforcement learning to build Carbon-X.

DNA and text share a single token stream in a single decoder, with no separate DNA encoder. A hybrid tokenizer keeps the Qwen3 BPE vocabulary for text and encodes DNA as non-overlapping 6-mers, so one DNA token covers 6 base pairs.

Mid-training is where the model learns DNA. Carbon-X-Base nearly doubles Qwen3-14B's accuracy at recovering DNA sequences while keeping its English benchmark scores within a few points (Evaluation). It also made reinforcement learning work: RL applied directly to Qwen3-14B never rose above chance on our FrontierVEP variant-effect benchmark, while the same RL starting from Carbon-X-Base did.

Carbon-X-Base is meant as a starting point:

• Fine-tuning your own DNA-and-language model on new tasks.
• Scoring sequences by likelihood, for example to compare a variant with its reference.
• Continuing DNA sequences.

To ask questions about sequences in a chat, use Carbon-X instead.

Key features

• 14.8B parameters: 40 layers, hidden size 5120, GQA (40 query heads, 8 KV heads), SwiGLU, RMSNorm with QK-norm.
• Hybrid tokenizer: the Qwen3 BPE vocabulary for text plus 5,515 DNA tokens. These are 4,096 6-mers, 1,364 tail tokens for sequences whose length is not a multiple of 6, and the <dna>, </dna> and <oov> markers. DNA in a prompt is detected and tagged automatically.
• Chat template: the Qwen3 chat template, with generation markers for assistant-only loss when fine-tuning.
• Context window: 40,960 tokens; mid-training used 8,192-token sequences.
• Runs with Transformers and vLLM (--trust-remote-code for the DNA tokenizer).

How to use

Carbon-X-Base uses a standard causal LM architecture. The DNA tokenizer is custom code, so pass trust_remote_code=True when you load the tokenizer.

pip install -U transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "HuggingFaceBio/Carbon-X-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda")

Scoring sequences

The log-likelihood of a sequence measures how plausible the model finds it. Comparing a variant with its reference gives a log-likelihood ratio:

def sequence_log_likelihood(sequence: str) -> float:
    inputs = tokenizer(sequence, return_tensors="pt").to(model.device)
    with torch.no_grad():
        logits = model(**inputs).logits[:, :-1].float()
    targets = inputs["input_ids"][:, 1:]
    return torch.log_softmax(logits, dim=-1).gather(-1, targets.unsqueeze(-1)).sum().item()


reference = "ATGGTGCATCTGACTCCTGAGGAGAAGTCTGCCGTTACTGCCCTGTGGGGCAAGG"
variant = "ATGGTGCATCTGACTCCTGTGGAGAAGTCTGCCGTTACTGCCCTGTGGGGCAAGG"  # HBB E6V (sickle cell)
llr = sequence_log_likelihood(variant) - sequence_log_likelihood(reference)
print(f"log-likelihood ratio (variant - reference): {llr:.2f}")

A negative ratio means the model finds the variant less likely than the reference. Likelihood ratios are a useful zero-shot signal for variant effects, but they are not calibrated pathogenicity predictions. Compare sequences of equal length, so that both have the same number of tokens.

Continuing DNA

Open a <dna> tag and let the model extend the sequence:

inputs = tokenizer("<dna>ATGGTGCATCTGACTCCTGAGGAGAAGTCTGCCGTTACTGCCCTGTGG", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=16, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Each generated DNA token adds 6 bases, so max_new_tokens=16 yields up to 96 bases.

Tokenizer: working with DNA inputs

The tokenizer switches between BPE for text and 6-mers for DNA:

>>> ids = tokenizer("Describe this sequence: ACGTACGATTCA")["input_ids"]
>>> tokenizer.convert_ids_to_tokens(ids)
['Describe', 'Ġthis', 'Ġsequence', ':', 'Ġ', '<dna>', 'ACGTAC', 'GATTCA', '</dna>']

• DNA is tagged automatically. Runs of at least 6 A/C/G/T bases are wrapped in <dna>…</dna> and tokenized as 6-mers. Runs of at least 12 bases with up to 20% IUPAC ambiguity codes (N, R, Y, …) are tagged too. Lowercase bases are uppercased, and sequences displayed in space-separated blocks are joined. Text you have already wrapped in <dna>…</dna> is always tokenized as DNA. To turn off detection for a call, pass auto_dna_tags=False to the tokenizer or to apply_chat_template().
• Only A, C, G and T form 6-mers. A 6-mer that contains any other character, such as N, becomes a single <oov> token. To keep full resolution, filter to canonical bases first.
• Any length works. When a sequence's length is not a multiple of 6, its last 1–5 bases are encoded with dedicated tail tokens, and decoding returns the original sequence exactly.
• First load needs Hub access. On first load, the tokenizer downloads the pinned BPE vocabulary it builds on.

Fine-tuning with TRL

Carbon-X-Base keeps the Qwen3 chat template, so you can fine-tune it on conversational data with TRL. The settings below are the ones we used for Carbon-X's supervised fine-tuning, with a global batch size of 32:

import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTConfig, SFTTrainer

model_id = "HuggingFaceBio/Carbon-X-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16)

trainer = SFTTrainer(
    model=model,
    processing_class=tokenizer,
    train_dataset=load_dataset("your-org/your-dna-chat-dataset", split="train"),  # a "messages" column
    args=SFTConfig(
        assistant_only_loss=True,
        eos_token="<|im_end|>",
        learning_rate=1e-5,
        lr_scheduler_type="cosine_with_min_lr",
        lr_scheduler_kwargs={"min_lr_rate": 0.1},
        num_train_epochs=3,
        max_length=9984,
        bf16=True,
        gradient_checkpointing=True,
    ),
)
trainer.train()

Set eos_token="<|im_end|>": the tokenizer's default end-of-sequence token is <|endoftext|>, but chat turns end with <|im_end|>. Full fine-tuning of a 14B model needs several GPUs, so launch the script with accelerate launch and DeepSpeed ZeRO-3 or FSDP.

Evaluation

We tracked DNA and English benchmarks throughout mid-training. The table compares Qwen3-14B, the starting model, with Carbon-X-Base at the end of mid-training (accuracy, %):

Group	Benchmark	Qwen3-14B	Carbon-X-Base
DNA	Sequence recovery	29.5	56.2
DNA	Eukaryote species classification	28.2	36.7
DNA	SeqQA (3 DNA tasks)	65.8	70.8
English	MMLU-Pro biology	72.1	72.5
English	HellaSwag	79.0	79.2
English	OpenBookQA	88.2	85.8

The blog's mid-training figure shows how each score evolves over the run, with 95% confidence intervals.

Mid-training alone does not produce a good DNA assistant. For results on our five DNA benchmarks after supervised fine-tuning and reinforcement learning, where Carbon-X averages 66.7% against 40.0% for Qwen3-14B, see the Carbon-X model card.

Training

Model

• Architecture: decoder-only Transformer with 40 layers, hidden size 5120, FFN size 17408, 40 attention heads with GQA (8 KV heads), head dimension 128, SwiGLU, RMSNorm with QK-norm and untied embeddings.
• Positional embedding: RoPE with base θ = 1 × 10^6. The context window is 40,960 tokens.
• Vocabulary: 157,184 tokens: 151,669 BPE tokens and 5,515 DNA tokens, including padding to a multiple of 128.
• Precision: bfloat16.

Vocabulary extension

We add the DNA tokens to Qwen3-14B's vocabulary with randomly initialised embeddings. We then train for about 10B tokens with the transformer body frozen, so that only the new DNA embeddings learn, before unfreezing the whole model.

Mid-training

The full model is then trained for 302B tokens with a sequence length of 8,192 and a global batch size of 512 sequences (≈ 4.2M tokens per step, 72,000 steps). The learning rate follows a cosine schedule from 5e-5 to 5e-6 after 2,000 warmup steps.

Learning rate was the dominant knob. In our ablations, higher learning rates learned DNA faster but caused substantial forgetting of English, while lower ones preserved English but left DNA near chance. We chose a learning rate at which DNA improved steadily without losing English, and trained long enough to reach good DNA performance. A cosine schedule worked better than warmup–stable–decay, which keeps the learning rate high for most of training and worsened forgetting. At 4B, even the lower learning rates cost English performance; at 14B, the same recipe preserved most of it.

The data mixture is:

Component	Share
Raw DNA	55%
Eukaryotic genomes: other vertebrates 16.2%, mammals 10.3%, invertebrates 7.6%, plants 3.1%, fungi 1.1%, protozoa 0.2%	38.5%
mRNA (OpenGenome2)	8.8%
Prokaryotic genomes (OpenGenome2)	5.5%
mRNA with splice sites and promoters (OpenGenome2)	2.2%
English	20%
FineWeb-Edu	8%
DCLM	8%
FinePDFs-Edu	4%
Biology text	15%
Documents with long DNA sequences (FinePDFs, PubMed, bioRxiv, FineWeb, Biostars)	7%
Other biology documents from the same sources	7%
Synthetic descriptions of annotated RefSeq sequences	1%
Instruction data	10%
Instruction following and chat	10%

The instruction data keeps the model able to follow chat-formatted prompts, which makes it a convenient starting point for supervised fine-tuning.

Limitations

Carbon-X-Base is a research checkpoint. It has not been clinically validated, and it should not be used to make medical decisions.

• ⚠️ Genetic data is highly sensitive. Depending on how you use this model (local download, inference API or endpoints, third-party inference providers, Spaces demos or others), input and output data may be processed or handled differently by different providers or Space owners. Make sure you understand and agree with how your data is handled before using the model.
• Not post-trained for DNA questions. Carbon-X-Base can hold a conversation, but its answers about specific sequences are unreliable. For example, it can miscopy or mistranslate a sequence it is asked to explain. Use Carbon-X for question answering, or fine-tune this model for your task.
• Hallucinations. The model may invent a sequence's identity or function, and its biological claims should be verified.
• Some English forgetting. Mid-training preserved most of Qwen3-14B's English ability, but not all of it (OpenBookQA drops 2.4 points).
• Long inputs. Mid-training used 8,192-token sequences (up to ≈ 49 kbp of DNA). The model accepts up to 40,960 tokens, but we have not evaluated inputs much longer than those seen in training.
• Long sequence generation. Quality degrades when the model generates long DNA sequences.
• No GGUF support. llama.cpp and other GGUF runtimes cannot load the custom Python DNA tokenizer, so DNA would be tokenized incorrectly.

License

Apache 2.0.

Acknowledgements

Carbon-X is a joint collaboration between the research teams at Hugging Face, Zhongguancun Academy, and the Telethon Institute of Genetics and Medicine/University of Naples “Federico II”.