Add Model Card

README.md

@@ -1,3 +1,173 @@
----
-license: mit
----
+---
+license: mit
+---
+
+## Model Architecture
+
+DecoderTCR is built on a Transformer-based protein language model (ESM2 family).
+
+### Core Architecture
+
+The model follows the **ESM2** architecture, a deep Transformer encoder designed for protein sequences.
+
+#### Embedding Layer
+- Token embedding dimension: *d* (e.g., 1280)
+- Learned positional embeddings
+- Vocabulary includes:
+  - 20 standard amino acids
+  - Special tokens (mask, padding, BOS/EOS, unknown)
+
+#### Transformer Stack
+- Number of layers: *L* (e.g., 33)
+- Hidden dimension: *d*
+- Number of attention heads: *h* (e.g., 20)
+- Multi-head self-attention:
+  - Full-sequence, bidirectional attention
+- Feed-forward network:
+  - Intermediate dimension ≈ 4× *d*
+  - Activation function: GELU
+- Layer normalization: Pre-LayerNorm
+- Residual connections around attention and feed-forward blocks
+
+### Continual Training Setup
+
+The model is initialized from a pretrained **ESM2 checkpoint** and further trained via continual pretraining with MLM objectives.
+
+
+### Model Scale (Example Configurations)
+
+  | Model Variant | Parameters | Layers | Hidden Dim | Attention Heads |
+| --- | --- | --- | --- | --- |
+| ESM2-650M | ~650M | 33 | 1280 | 20 |
+| ESM2-3B | ~3B | 36 | 2560 | 40 |
+
+### Model Card Authors
+
+Ben Lai
+
+### Primary Contact Email
+
+Ben Lai ben.lai@czbiohub.org
+To submit feature requests or report issues with the model, please open an issue on [the GitHub repository](https://github.com/czbiohub-chi/DecoderTCR).
+
+### System Requirements
+
+- Compute Requirements: GPU
+
+## Intended Use
+
+### Primary Use Cases
+
+The DecoderTCR models are designed for the following primary use cases:
+
+1. **TCR-pMHC Binding Prediction**: Predict the interaction between T-cell receptors (TCRs) and peptide-MHC complexes
+2. **Interaction Scoring**: Calculate interface energy scores for TCR-pMHC interactions
+3. **Sequence Analysis**: Analyze TCR sequences and their interactions with specific peptides
+4. **Immunology Research**: Support research in adaptive immunity, T-cell recognition, and antigen presentation
+
+The models are particularly useful for:
+- Identifying potential TCR-peptide binding pairs
+- Screening TCR sequences for specific antigen recognition
+- Understanding the molecular basis of T-cell recognition
+- Supporting vaccine design and immunotherapy development
+
+### Out-of-Scope or Unauthorized Use Cases
+
+Do not use the model for the following purposes:
+- Use that violates applicable laws, regulations (including trade compliance laws), or third party rights such as privacy or intellectual property rights
+- Any use that is prohibited by the [MIT license](https://github.com/czbiohub-chi/DecoderTCR/blob/main/LICENSE) and [Acceptable Use Policy](https://virtualcellmodels.cziscience.com/acceptable-use-policy).
+- Clinical diagnosis or treatment decisions without proper validation
+- Direct use in patient care without appropriate clinical validation and regulatory approval
+- Use for purposes that could cause harm to individuals or groups
+
+The models are research tools and should not be used as the sole basis for clinical or diagnostic decisions.
+
+## Training Data
+
+The models are trained with multi-component large-scale protein sequence databases. The training data consists of:
+
+- **TCR sequences**: Observerd T-cell Space(OTS) for paired $\alpha/\beta$ TCR sequences.
+- **Peptide-MHC sequences**: MHC Motif Atlas for peptide-MHC ligandomes and high confidence synthetic interactions via MixMHCpred predictions.
+- **Paired TCR-pMHC Interactions**: VDJdb for paired TCR-pMHC interaction data.
+
+
+## Continual Pre-training Strategy
+
+This model is trained using a **continual pre-training curriculum** that adapts a pretrained ESM2 backbone to new protein domains while preserving previously learned representations.
+
+### Overview
+
+Continual pre-training proceeds in **multiple stages**, each leveraging different data regimes and masking strategies:
+
+- Stage 1 emphasize **abundant marginal sequence data**, encouraging robust component-level representations.
+- Stage 2 incorporate **scarcer, structured, or interaction-rich data**, refining conditional dependencies without overwriting earlier knowledge.
+
+The architecture, tokenizer, and objective remain unchanged throughout training; only the data distribution and masking strategy evolve.
+
+### Stage 1: Component-Level Adaptation
+
+In the first stage, the model is further pretrained on large collections of unpaired or weakly structured protein sequences relevant to the target domain.
+
+- **Objective:** Masked Language Modeling (MLM)
+- **Masking:** Component- or region-aware masking schedules that upweight functionally relevant positions
+- **Purpose:**  
+  - Adapt the pretrained ESM2 representations to the target protein subspace  
+  - Learn domain-specific sequence statistics while retaining general protein knowledge
+
+This stage acts as a regularizer, anchoring learning in large-scale marginal data before introducing more complex dependencies.
+
+### Stage 2: Conditional / Interaction-Aware Refinement
+
+In subsequent stages, the model is continually trained on **structured or paired sequences** that encode higher-order dependencies (e.g., interactions between protein regions or components).
+
+- **Objective:** Masked Language Modeling (MLM)
+- **Masking:** Joint masking across interacting regions to encourage cross-context conditioning
+- **Purpose:**  
+  - Refine conditional relationships learned from limited paired data  
+  - Align representations across components without degrading Stage 1 task performance
+
+## Biases, Risks, and Limitations
+
+### Potential Biases
+
+- The model may reflect biases present in the training data, including:
+  - Overrepresentation of certain HLA alleles or peptide types
+  - Limited diversity in TCR sequences from specific populations
+  - Bias toward well-studied antigen systems
+- Certain TCR clonotypes or peptide types may be underrepresented in training data
+
+### Risks
+
+Areas of risk may include but are not limited to:
+- **Inaccurate predictions**: The model may produce incorrect binding predictions, especially for novel sequences or rare HLA-peptide combinations
+- **Overconfidence**: The model may assign high confidence to predictions that are actually uncertain
+- **Biological misinterpretation**: Users may misinterpret model outputs as definitive biological facts rather than predictions
+- **Clinical misuse**: Use in clinical settings without proper validation could lead to incorrect treatment decisions
+
+### Limitations
+
+- **Sequence length**: The model has limitations on maximum sequence length (typically ~1024 tokens)
+- **Novel sequences**: Performance may degrade on sequences very different from training data
+- **HLA diversity**: Limited training data for rare HLA alleles may affect prediction accuracy
+- **Context dependency**: The model may not capture all biological context (e.g., post-translational modifications, cellular environment)
+- **Computational requirements**: GPU is recommended for optimal performance
+
+### Caveats and Recommendations
+
+- **Review and validate outputs**: Always review and validate model predictions, especially for critical applications
+- **Experimental validation**: Model predictions should be validated experimentally before use in research or clinical contexts
+- **Uncertainty awareness**: Be aware that predictions are probabilistic and may have uncertainty
+- **Domain expertise**: Use the model in conjunction with domain expertise in immunology and T-cell biology
+- **Version tracking**: Keep track of which model version and checkpoint you are using
+
+We are committed to advancing the responsible development and use of artificial intelligence. Please follow our [Acceptable Use Policy](/acceptable-use-policy)  when engaging with our services.
+
+Should you have any security or privacy issues or questions related to the services, please reach out to our team at security@chanzuckerberg.com or privacy@chanzuckerberg.com respectively.
+
+## Acknowledgements
+
+This model builds upon:
+- **ESM2** by Meta AI (Facebook Research) for the base protein language model
+- The broader computational biology and immunology research communities
+
+Special thanks to the developers and contributors of the ESM models and the open-source tools that made this work possible.