---
language:
- en
license: mit
tags:
- genomics
- dna
- chicken
- poultry
- caduceus
- mamba
- biology
- bioinformatics
- gallus-gallus
datasets:
- custom
pipeline_tag: feature-extraction
base_model: kuleshov-group/caduceus-ph_seqlen-131k_d_model-256_n_layer-4
---

# 🐔 PoultryCaduceus

**A Bidirectional DNA Language Model for Chicken Genome**

PoultryCaduceus is the first DNA foundation model specifically pre-trained on the chicken (*Gallus gallus*) genome, based on the [Caduceus](https://github.com/kuleshov-group/caduceus) architecture.

## Model Description

| Feature | Value |
|---------|-------|
| **Base Model** | caduceus-ph (4-layer) |
| **Pre-training Genome** | GRCg6a (galGal6) |
| **Sequence Length** | 65,536 bp |
| **Hidden Dimension** | 256 |
| **Layers** | 4 |
| **Vocab Size** | 16 |
| **Training Steps** | 10,000 |

## Usage

```python
from transformers import AutoModelForMaskedLM

# Load model
model = AutoModelForMaskedLM.from_pretrained(
    "jamie0315/PoultryCaduceus",
    subfolder="checkpoint-10000",
    trust_remote_code=True
)
```

### Get Sequence Embeddings

```python
import torch

# DNA vocabulary
DNA_VOCAB = {'A': 7, 'C': 8, 'G': 9, 'T': 10, 'N': 5, '[MASK]': 4}

# Tokenize sequence
sequence = "ATGCGATCGATCGATCG"
input_ids = torch.tensor([[DNA_VOCAB.get(c, 5) for c in sequence]])

# Get embeddings
model.eval()
with torch.no_grad():
    outputs = model(input_ids, output_hidden_states=True)
    embeddings = outputs.hidden_states[-1]  # (batch, seq_len, 256)
```

## Training Data

Pre-training data is available in `chicken_pretrain_data_GRCg6a/`:
- `train_65k.h5` - Training set (~58,000 sequences)
- `val_65k.h5` - Validation set (~1,200 sequences)

## Repository Structure

```
PoultryCaduceus/
├── checkpoint-10000/              # Model weights
│   ├── config.json
│   └── model.safetensors
└── chicken_pretrain_data_GRCg6a/  # Pre-training data
    ├── train_65k.h5
    └── val_65k.h5
```

## Links

- 📦 **GitHub**: [chengzhimin/PoultryCaduceus](https://github.com/chengzhimin/PoultryCaduceus)

## License

MIT License