| | --- |
| | language: |
| | - en |
| | license: apache-2.0 |
| | base_model: Qwen/Qwen3-8B |
| | datasets: |
| | - HuggingFaceTB/smollm-corpus |
| | tags: |
| | - text-generation |
| | - transformers |
| | - safetensors |
| | - qwen |
| | - climate |
| | - planetary-boundaries |
| | - domain-adaptation |
| | pipeline_tag: text-generation |
| | --- |
| | |
| | # ClimateGPT-3-8B |
| |
|
| | ClimateGPT-3-8B is an open language model domain-adapted for climate science and the **Planetary Boundaries** framework. |
| |
|
| | ## Model details |
| |
|
| | - **Base model**: `Qwen/Qwen3-8B` |
| | - **Model type**: Causal LM |
| | - **Language(s)**: English |
| | - **Context length**: 8192 tokens (SFT configuration) |
| | - **License**: Apache-2.0 |
| | - **Release artifact**: Fully merged weights (standalone model; no adapter required) |
| |
|
| | ## Intended use |
| |
|
| | - Climate and sustainability Q&A |
| | - Planetary Boundaries–focused education and analysis |
| | - Drafting and summarization of climate-related content |
| |
|
| | ## Limitations |
| |
|
| | - The model may produce incorrect or outdated information. |
| | - Training data is largely English web content; this can introduce geographic/cultural and topical biases. |
| | - The model is not a substitute for professional scientific, medical, legal, or policy advice. |
| |
|
| | ## Training |
| |
|
| | ClimateGPT-3-8B was built in multiple stages: |
| |
|
| | ### Continued pretraining (CPT) |
| |
|
| | Starting from `Qwen/Qwen3-8B`, we performed continued pretraining on climate-focused corpora primarily derived from FineWeb-Edu (SmolLM-Corpus) using climate- and Planetary Boundaries–oriented filtering. |
| |
|
| | The data selection emphasizes climate science and Planetary Boundaries terminology and includes filtering to reduce off-topic matches from ambiguous terms. |
| |
|
| | ### Supervised fine-tuning (SFT) |
| |
|
| | We performed supervised fine-tuning using a mixture of: |
| |
|
| | - Climate instruction-following data |
| | - Multi-turn conversations |
| | - Safety/refusal examples |
| | - Tool-use data |
| | - Synthetic climate / Planetary Boundaries Q&A |
| |
|
| | ## Training data |
| |
|
| | ### Public data |
| |
|
| | - **FineWeb-Edu (via `HuggingFaceTB/smollm-corpus`)** |
| | - Used for climate- and Planetary Boundaries–filtered continued pretraining. |
| | - **Dataset license**: ODC-By |
| | - Dataset page: https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus |
| |
|
| | ### Non-public / generated data |
| |
|
| | In addition to public data, the training mix includes internal and/or generated instruction data. These datasets are not redistributed with this model. |
| |
|
| | ## Evaluation |
| |
|
| | We evaluate climate-domain performance using a Planetary Boundaries evaluation suite compatible with EleutherAI’s `lm-evaluation-harness`. |
| |
|
| | A representative comparison (from this project’s Planetary Boundaries evaluation artifacts) between a ClimateGPT 8B checkpoint and the base Qwen3-8B: |
| |
|
| | | Task | Metric | ClimateGPT | Qwen3-8B | |
| | |---|---:|---:|---:| |
| | | `planetary_boundaries_mcq_large` | acc | 0.4422 | 0.3533 | |
| | | `planetary_boundaries_mcq_large` | acc_norm | 0.4278 | 0.3900 | |
| | | `planetary_boundaries_mcq_hard` | acc | 0.3467 | 0.2711 | |
| | | `planetary_boundaries_mcq_hard` | acc_norm | 0.3800 | 0.3400 | |
| | | `planetary_boundaries_qa_large` | exact_match | 0.9000 | 0.8467 | |
| | | `planetary_boundaries_qa_strict_core_nolist` | exact_match | 0.6556 | 0.4889 | |
| | |
| | ## How to use |
| | |
| | ### Transformers |
| | |
| | This repository contains a standalone model. You can load it directly with Transformers. |
| | |
| | ```python |
| | from transformers import AutoModelForCausalLM, AutoTokenizer |
| | |
| | model_id = "Erasmus-AI/climategpt-3-8b" |
| |
|
| | tokenizer = AutoTokenizer.from_pretrained(model_id) |
| | model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto") |
| | |
| | prompt = "Explain the Planetary Boundaries framework in simple terms." |
| | inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| |
|
| | out = model.generate( |
| | **inputs, |
| | max_new_tokens=512, |
| | do_sample=True, |
| | temperature=0.6, |
| | top_p=0.95, |
| | ) |
| | print(tokenizer.decode(out[0], skip_special_tokens=True)) |
| | ``` |
| | |
| | ### vLLM |
| |
|
| | This model is intended to be compatible with vLLM. |
| |
|
| | ## License |
| |
|
| | - **Model weights**: Apache-2.0 |
| | - **Base model**: `Qwen/Qwen3-8B` (Apache-2.0) |
| |
|
| | ## Attribution |
| |
|
| | If you use this model, please cite/attribute the upstream resources where appropriate: |
| |
|
| | - Base model: https://huggingface.co/Qwen/Qwen3-8B |
| | - Training data (public portion): https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus (ODC-By) |
| |
|
| | ## Citation |
| |
|
| | If you use this model in academic work, please cite: |
| |
|
| | ```bibtex |
| | @misc{climategpt3, |
| | title = {ClimateGPT-3-8B}, |
| | howpublished = {\url{https://huggingface.co/Erasmus-AI/climategpt-3-8b}}, |
| | year = {2026} |
| | } |
| | ``` |
| |
|
| | ## Contact |
| |
|
| | If you have questions, issues, or evaluation results to share, please open a discussion/issue in the repository that accompanies this release. |
| |
|
