suitch/gritlm-checkthat-2025

GritLM Fine-tuned on CheckThat! 2025 Dataset

This model is a fine-tuned version of GritLM/GritLM-7B on the CheckThat! 2025 Subtask 4B dataset for scientific claim verification and source retrieval.

Model Description

GritLM (Generative Representational Instruction Tuned Language Model) is a unified model that handles both text generation and embedding tasks. This fine-tuned version is specifically optimized for retrieving scientific paper abstracts given claims from social media (tweets).

• Base Model: GritLM/GritLM-7B
• Task: Scientific claim to paper abstract retrieval
• Training Data: CheckThat! 2025 Subtask 4B (12,853 training queries, 7,718 CORD-19 papers)
• Fine-tuning Method: Contrastive learning with hard negatives

Intended Use

This model is designed for:

• Scientific fact-checking: Retrieving source papers for scientific claims made on social media
• Academic research: Finding relevant scientific literature for tweet-length claims
• Information retrieval: Matching short informal queries to formal scientific abstracts

Training Details

Training Data

The model was fine-tuned on the CheckThat! 2025 Subtask 4B dataset:

• Queries: 12,853 scientific claims from Twitter
• Corpus: 7,718 scientific paper abstracts from CORD-19
• Hard Negatives: Mined using E5-large-v2 rankings (4 hard negatives per query)
• In-batch Negatives: 31 additional negatives per batch

Training Procedure

Data Preparation:

1. Generated initial rankings using E5-large-v2
2. Mined hard negatives from top-ranked non-relevant documents
3. Created training triplets: (query, positive abstract, hard negatives)

Training Configuration:

• Learning rate: 2e-5
• Batch size: 2 per device
• Gradient accumulation: 16 steps
• Total epochs: 3
• Max sequence length: 512
• Optimizer: AdamW
• Warmup steps: 300
• Training precision: bfloat16

Training Hyperparameters

learning_rate: 2e-5
per_device_train_batch_size: 2
gradient_accumulation_steps: 16
num_train_epochs: 3
max_seq_length: 512
warmup_steps: 300
fp16: false
bf16: true
gradient_checkpointing: true
deepspeed_stage: 2
seed: 42

Model Architecture

Based on GritLM-7B which uses:

Limitations and Bias

• Domain-specific: Optimized for scientific claims and CORD-19 abstracts; may not generalize well to other domains
• Twitter context: Trained on tweet-style queries; performance may vary with formal queries
• Corpus size: Limited to 7,718 papers; may not cover all scientific topics comprehensively
• Language: English only
• Temporal: Training data may reflect scientific understanding up to the dataset collection date

Citation

If you use this model, please cite:

@misc{gritlm-checkthat-2025,
  author = {suitch},
  title = {GritLM Fine-tuned on CheckThat! 2025 Dataset},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/suitch/gritlm-checkthat-2025}
}

Base Model Citation:

@misc{muennighoff2024generative,
    title={Generative Representational Instruction Tuning}, 
    author={Niklas Muennighoff and Hongjin Su and Liang Wang and Nan Yang and Furu Wei and Tao Yu and Amanpreet Singh and Douwe Kiela},
    year={2024},
    eprint={2402.09906},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

Dataset Citation:

@inproceedings{checkthat2025,
  title={CheckThat! at CLEF 2025: Check-Worthiness, Subjectivity, Persuasion, Roles, Authorities, and Veracity},
  booktitle={Proceedings of CLEF 2025},
  year={2025}
}

License

This model inherits the Apache 2.0 license from the base GritLM-7B model.