DHTI Logo

  • 🚀 COPM: Converting clinical signals into outcome intelligence for DHTI.

CDS Embedding Model with Genomic Tokenizer (Minimal Variant)

Minimal CDS embedding model for quick prototyping and testing. Trained using contrastive learning with the Genomic Tokenizer on human protein-coding DNA sequences.

Model Details

  • Variant: Minimal
  • Architecture: Transformer Encoder with 2 layers, 4 attention heads
  • Hidden Size: 256 dimensions
  • Embedding Dimension: 256 dimensions (L2-normalized)
  • Vocabulary: 26 tokens (Genomic Tokenizer)
  • Max Sequence Length: 2048 tokens
  • Training Objective: Contrastive learning (instance discrimination)
  • Parameters: ~2M parameters

Model Variants

This model is part of a family of CDS embedding models:

Variant Parameters Layers Hidden Size Use Case
Minimal ~2M 2 256 Development, testing
Small ~10M 4 512 Research, moderate tasks
Regular ~50M 12 768 Advanced, high accuracy

Genomic Tokenizer

This model uses the Genomic Tokenizer which tokenizes DNA sequences at the codon level:

  • 26 unique tokens:
    • Special tokens: [CLS], [SEP], [BOS], [MASK], [PAD], [UNK]
    • 20 amino acid codons (Alanine, Cysteine, Aspartic_acid, etc.)

Each token represents a codon (3 nucleotides) that encodes a specific amino acid, enabling the model to learn at the protein-coding level rather than individual nucleotides.

Training Data

  • Dataset: GENCODE v50 human protein-coding CDS sequences
  • Genome Assembly: GRCh38
  • Transcripts: ~17,000 protein-coding transcripts
  • Preprocessing:
    • Extracted CDS from GTF annotations
    • Reconstructed sequences from GRCh38 genome
    • Quality control: start/stop codon verification, reading frame validation

Use Cases

Best for development, debugging, and small-scale experiments: ✅ Recommended:

  • Gene prioritization in rare disease diagnostics (as a research tool)
  • Functional annotation of novel variants
  • Disease gene discovery in research settings
  • Building gene-disease knowledge graphs

Clinical Validation Status

  • Research Phase: Model is intended for research and translational applications
  • Not Clinically Validated: Has not undergone clinical validation studies
  • NOT APPROVED FOR CLINICAL USE: Should not be used for patient care or diagnostics

Installation

pip install torch genomic-tokenizer pydantic

Loading the Model

import torch
import json
from pathlib import Path
from huggingface_hub import hf_hub_download
from genomic_tokenizer import GenomicTokenizer

# You need the model classes (install gtembed package or copy model.py)
# from src.gtembed.model import CDSEmbeddingModel
# from src.gtembed.config import ModelConfig

# Download model files
# Replace with your variant: minimal, small, or regular
repo_id = "beapen/gt-embed-minimal-v1"  # e.g., beapen/gt-embed-minimal-v1
config_path = hf_hub_download(repo_id=repo_id, filename="config.json", repo_type="model")
model_path = hf_hub_download(repo_id=repo_id, filename="model.pt", repo_type="model")

# Load config
with open(config_path) as f:
    config_dict = json.load(f)

# Initialize model (requires gtembed package)
# config = ModelConfig(**config_dict)
# model = CDSEmbeddingModel.from_config(config)
# model.load_state_dict(torch.load(model_path, map_location="cpu"))
# model.eval()

# Load tokenizer
tokenizer = GenomicTokenizer(model_max_length=2048)

Generating Embeddings

import torch

# Sample CDS sequence (protein-coding DNA)
cds_sequence = "ATGGCCGCCTACAAGCAGCAGTCTGGGGAAGACGGGCTACCTTGCCCCTGA"

# Tokenize
tokens = tokenizer(cds_sequence)
input_ids = torch.tensor([tokens['input_ids']])

# Generate embedding
with torch.no_grad():
    embedding = model(input_ids)

print(f"Embedding shape: {embedding.shape}")  # [1, 256]
print(f"Embedding norm: {embedding.norm().item():.4f}")  # Should be ~1.0 (L2 normalized)

Batch Processing

sequences = [
    "ATGGCCGCCTACAAGCAGCAGTCTGGGGAAGACGGGCTACCTTGCCCCTGA",
    "ATGGCCGCCTACAAGCAGCAGTCTGGGGAAGACGGGCTACCTTGCCCCTGA",
    # ... more sequences
]

# Tokenize all
tokenized = tokenizer(sequences, padding=True, return_tensors="pt")
input_ids = tokenized['input_ids']
attention_mask = tokenized['attention_mask']

# Generate embeddings
with torch.no_grad():
    embeddings = model(input_ids, attention_mask=attention_mask)

print(f"Batch embeddings shape: {embeddings.shape}")  # [batch_size, 256]

Model Card Contact

Limitations

  1. Training Data: Only includes protein-coding genes from GENCODE v50
  2. Sequence Length: Limited to 2048 tokens (~6144 nucleotides)
  3. Species: Human-only (GRCh38 assembly)
  4. Context: Does not include regulatory regions or non-coding sequences
  5. Variant: Minimal model - consider Small or Regular variants for production use

Ethical Considerations

This model is intended for research purposes in genomics and bioinformatics. Users should:

  • Verify biological interpretations with experimental data
  • Follow ethical guidelines for genetic data usage

Citations

If you use this model in your research, please cite the following:

GTEmbed Model

Eapen, B. (2026). GTEmbed: Genomic Tokenizer Embedding Pipeline. Hugging Face. https://huggingface.co/beapen/gt-embed-minimal-v1

Genomic Tokenizer

Eapen, B. (2026). Genomic Tokenizer: Codon-level Tokenization for CDS Sequences. GitHub. https://github.com/dermatologist/genomic-tokenizer

Training Data Sources

This model was trained using CDS sequences extracted from two publicly available datasets:

1. GENCODE v50 Human Gene Annotation

GENCODE provides comprehensive reference gene annotation for the human genome. For GENCODE v50 (released in 2023), please cite:

Frankish, A., Diekhans, M., Ferreira, A.M., Johnson, R., Jungreis, I., Loveland, J., Mudge, J.M., Sisu, C., Wright, J., Armstrong, J., Barnes, I., Berry, K., Bignell, A., Carbonell-Sala, S., Cerdán-Vélez, D., Cheng, Y., Davidson, C., Dyer, S.C., Eyrignoux, C., Fatima, R., Fernandez-Banet, J., Gall, A., Garcia Girón, C., Gonzalez, J.M., Greller, R., Hardy, M., Hunt, T., Ishiguro, H., James, B., Jayakumar, A., Juettemann, T., Kay, M., Kulesha, E., Mangan, R.M., Martin, F.J., Mohanan, S., Muir, P., Murphy, S., Nag, A., Overduin, B., Parker, A., Perry, M., Petty, T.L.P., Pujara, P., Raj, S., Sanchez-Castillo, D., Schmitt, B., Schuilenburg, H., Sealy, I., Shafer, R., Sharma, L., Slater, T., Smedley, D., Sobala, A., Spudich, A., Steed, E., Tang, Y., Thakuria, J., Tress, M.L., Vazquez, J., Wilder, S., Zadissa, A., Zerbino, D.R., Cunningham, F., Lander, A., Flicek, P., and Harrow, J. (2023). GENCODE: reference annotation for the human and mouse genomes in 2023. Nucleic Acids Research, 51(D1), D942-D949. https://doi.org/10.1093/nar/gkac1071

Alternative citation for latest GENCODE release (2025):

Mudge, J.M., Carbonell-Sala, S., Diekhans, M., Martinez, J.G., Hunt, T., Jungreis, I., Loveland, J.E., Arnan, C., Barnes, I., Bennett, R., Berry, K., Bignell, A., Cerdán-Vélez, D., Cochran, K., Cortés, L.T., Davidson, C., Donaldson, S., Dursun, C., Fatima, R., Hardy, M., Hebbar, P., Hollis, Z., James, B., Jiang, Y., Johnson, R., Kaur, G., Kay, M., Mangan, R.M., Maquedano, M.G., Gómez, L.M., Mathlouthi, N., Merritt, R., Ni, P., Palumbo, E., Perteghella, T., Pozo, F., Raj, S., Sisu, C., Steed, E., Sumathipala, D., Suner, M.M., Uszczynska-Ratajczak, B., Wass, E., Yang, Y., Zhang, D., Finn, R.D., Gerstein, M., Guigó, R., Hubbard, T.J.P., Kellis, M., Kundaje, A., Paten, B., Tress, M.L., Birney, E., Martin, F.J., and Frankish, A. (2025). GENCODE 2025: reference gene annotation for human and mouse. Nucleic Acids Research, 53(D1), D966-D975. https://doi.org/10.1093/nar/gkae1078

Data source:


2. GRCh38.p14 Human Reference Genome Assembly

The Genome Reference Consortium (GRC) maintains the human reference genome assembly. For GRCh38.p14 (accession: GCF_000001405.40), cite as:

Genome Reference Consortium. (2022). Human genome assembly GRCh38.p14. RefSeq assembly accession: GCF_000001405.40. https://www.ncbi.nlm.nih.gov/assembly/GCF_000001405.40

For the original GRCh38 release (2013), you may also cite:

Church, D.M., Dunn, D.M., Hodgson, A., Loveland, J.E., Smith, A.N., Bevan, P., Cunningham, F., Earhart, S., Farrell, C.M., Gildert, J., Graf, S., Kay, M., Khurana, E., Li, Y., Thakuria, J., Trevanion, S.J., Veyrieras, J.-B., Vogel, J., White, S.J., Aken, B.L., Barrell, D., Brent, S., Chen, Z., Delcher, A., Frankish, A., Gietzen, D., Gilbert, J.G., Graham, U., Hammond, S., Haussler, D., Hermjakob, H., Hume, D.A., Kasprzyk, A., Kulesha, E., Lander, A., Long, J., Melsens, S., Meidl, P., Murphy, S., Nigro, J., Park, T., Rosenfeld, J.A., Schuster, S.C., Sealy, I., Souvorov, A., Souvorova, A., Steward, C., Taudien, S., Tress, M.L., Ureta-Vidal, A., Wallen, T., Weber, R., White, O., Woodward, J.R., Zadissa, A., Zdobnov, E.M., Zerbino, D.R., Blankenberg, D., Kent, W.J., Korf, I., Lander, E.S., Hubbard, T.J.P., Flicek, P., Harrow, J., Weber, R.J., Cooper, D.M., Eichler, E.E., Felsenfeld, G., Green, E.D., Grimwood, J., Guttmacher, C., Hardison, R.C., Istrail, S., Jaffe, D.B., Myers, R.M., Pennacchio, L.A., Ponting, C.P., Rubin, E.M., Ryder, O., Venter, J.C., Waterston, R.H., Wilson, R.K., and Collins, F.S. (2014). Grading the quality of genome assemblies. Nature Biotechnology, 32, 101-104. https://doi.org/10.1038/nbt.2783

Data source:

Best practices for citing genome assemblies:

  • Always include the assembly accession number (GCF_000001405.40) for reproducibility
  • Specify the patch level (p14) when relevant
  • Reference the Genome Reference Consortium as the maintainer

License

Apache 2.0

Tags

genomics, dna-sequences, embedding, transformer, contrastive-learning, protein-coding, cds, genomic-tokenizer, bioinformatics, healthcare, minimal-model

Downloads last month
63
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support