README.md

---
license: other
license_name: climategpt-community-license
license_link: https://huggingface.co/xquantize/climategpt-7b-4bit-mlx/blob/main/LICENSE.txt
language:
- en
library_name: mlx
tags:
- mlx
- climate
- earth-science
- geoscience
- quantized
- 4-bit
- llama
base_model: eci-io/climategpt-7b
pipeline_tag: text-generation
---

# ClimateGPT-7B-4bit-mlx

This is a 4-bit [MLX](https://github.com/ml-explore/mlx) quantization of [**ClimateGPT-7B**](https://huggingface.co/eci-io/climategpt-7b) (`eci-io/climategpt-7b`), a climate-domain chat model, converted for fast local inference on Apple Silicon.

ClimateGPT is a family of models designed to synthesize interdisciplinary research on climate change. The 7B model is continued-pretrained from Llama-2-7B on a curated collection of climate documents and then instruction fine-tuned on instruction–completion pairs collected in cooperation with climate scientists. The authors report it outperforms Llama-2-70B-Chat on their climate-specific benchmarks. See the original work for full detail:

> Thulke et al., *ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change*, 2024.
> Paper: <https://arxiv.org/abs/2401.09646> · Project: <https://eci.io>

## Conversion details

- **Method:** `mlx_lm.convert` with `-q --q-bits 4`
- **Result:** 4.500 bits per weight
- **Precision of base load:** bfloat16
- **Converted on:** Apple Silicon (MLX)
- **mlx-lm version:** 0.31.3
- **Base model:** `eci-io/climategpt-7b` (continued-pretrained from Llama-2-7B)

## Usage

```bash
pip install mlx-lm
```

```python
from mlx_lm import load, generate

model, tokenizer = load("xquantize/climategpt-7b-4bit-mlx")

# ClimateGPT is an instruction-tuned CHAT model — use the chat template.
messages = [
    {"role": "user", "content": "What is the difference between weather and climate?"}
]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256, verbose=True))
```

Or from the command line:

```bash
mlx_lm.generate --model xquantize/climategpt-7b-4bit-mlx \
  --prompt "What is the difference between weather and climate?" --max-tokens 128
```

### Note on usage style

This is an **instruction-tuned chat model**, so prompt it with chat turns via the tokenizer's chat template (shown above) rather than raw text completion. It uses a ChatML-style format (`<|im_start|>` / `<|im_end|>`). The authors note the model is designed to be used with retrieval augmentation (RAG) to extend its knowledge and improve factuality and is **not** intended as a general-purpose chatbot, it is specialized for the climate domain. Context length is 4K tokens.

### Stop-token fix

The upstream `generation_config.json` sets `eos_token_id` to `2` (Llama-2's `</s>`), but this model actually ends turns with the ChatML token `<|im_end|>` (id `32006`). As shipped upstream that mismatch causes generation to run past the end of a turn and repeat. This repository sets `eos_token_id` to `[2, 32006]`, so generation stops cleanly at the end of a turn with no extra flags required.

## Performance

On an Apple Silicon Mac, this 4-bit build runs at ~34 tokens/sec with a peak memory footprint of ~4.0 GB, making a climate-specialized 7B usable on consumer hardware.

## Limitations

ClimateGPT-7B is a 2024-era Llama-2-7B derivative; its general fluency and knowledge reflect that generation, not current frontier models. As with any chat-capable LLM, it may produce biased, offensive or inaccurate responses and should not be relied on for authoritative scientific or safety-critical decisions. It is intended to be paired with retrieval augmentation. Quantization to 4 bits may introduce additional minor quality degradation relative to the full-precision model.

## License & attribution

This model is a 4-bit MLX quantization of ClimateGPT-7B and is distributed under the **ClimateGPT Non-Commercial Community License** (© EQTY R&D, Inc.; Erasmus.AI as third-party beneficiary). **Non-commercial use only.**

Per the license, this repository includes a copy of the license (`LICENSE.txt`) and the required attribution (`Notice`). By using these weights you agree to the ClimateGPT Terms of Service and Acceptable Use Policy contained in `LICENSE.txt`.

All credit for the underlying model belongs to the ClimateGPT authors (Thulke et al.; Erasmus AI, AppTek, EQTY R&D). This repository provides only an MLX-format 4-bit quantization for local inference on Apple Silicon and claims no ownership of the underlying model. The "ClimateGPT" name is used solely to describe the origin of these weights.