metadata
license: mit
language: en
tags:
- climate
- fill-mask
- minilm
base_model: microsoft/MiniLM-L12-H384-uncased
MiniClimate
A lightweight climate-domain-adapted model based on MiniLM-L12-H384-uncased (~33M params), pretrained using DAPT + TAPT on climate-related YouTube comments.
What it does
Adapts MiniLM to climate-related text using Masked Language Modelling (MLM). Can be fine-tuned for downstream tasks like:
- Stance detection
- Environmental claim detection
- Sentiment classification
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Raj7722/MiniClimate")
model = AutoModelForMaskedLM.from_pretrained("Raj7722/MiniClimate")
This is a pretrained checkpoint. Fine-tune it with a classification head for your downstream task.
Results
Avg F1 across 4 downstream tasks (after task-specific fine-tuning): 75.22% (best F1-per-parameter efficiency vs. RoBERTa, ClimateBERT, SciBERT, DistilBERT).
Author
Rajdeep Bose — NSHM Knowledge Campus, Kolkata (M.Sc. Data Science & Analytics)
License
MIT