README.md

---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
base_model: Qwen/Qwen3-8B
thumbnail: https://huggingface.co/pengfali/GeohazardGPT/logo/GeohazardGPT_logo.png
tags:
- geohazard
- geology
- geoscience
- geotechnical-engineering
- landslide
- qwen3
- lora
- rag
datasets:
- vicgalle/alpaca-gpt4
---

<p align="center">
  <img src="./logo/GeohazardGPT_logo.png" alt="GeohazardGPT" width="420"/>
</p>

# GeohazardGPT

**GeohazardGPT** is the first large language model purpose-built for geohazard analysis and engineering practice. Built on a Qwen3-8B backbone with LoRA-based parameter-efficient fine-tuning, it is trained on a curated domain corpus of 883 million tokens spanning 12 major geological hazard categories. When combined with a retrieval-augmented generation (RAG) pipeline over authoritative engineering standards, GeohazardGPT achieves performance comparable to much larger models on both general geohazard knowledge and professional engineering examination tasks.

---

## Model Details

| Property | Value |
|---|---|
| **Base model** | Qwen3-8B |
| **Fine-tuning method** | LoRA (rank 128, α 256) |
| **Trainable parameters** | 349M |
| **Training data** | ~100K instruction–response pairs |
| **Domain corpus** | 883M tokens / 1.82M documents |
| **Hazard categories** | 12 major / 49 subcategories |
| **Context length** | 32K tokens (extendable to 128K via YaRN) |
| **Language** | English |
| **License** | Apache 2.0 |

---

## Intended Use

GeohazardGPT supports knowledge-intensive workflows in geohazard assessment and geotechnical engineering practice, including:

- **Factual QA** — precise recall of geohazard definitions, geomaterial properties, and code requirements
- **Open-ended explanation** — interpretation of hazard mechanisms, failure processes, and impact analysis
- **Engineering recommendation** — selection of stabilization measures, mitigation strategies, and monitoring plans for site-specific conditions
- **Report summarization** — structured extraction of key findings from investigation reports, case studies, and technical specifications

It is designed for use by geotechnical engineers, geohazard researchers, and practitioners who require technically accurate, domain-grounded responses. **Model outputs should complement, not replace, professional field investigation and expert judgment.**

---

## Training Data

The instruction-tuning dataset was constructed using **GeoInstruct**, a taxonomy-guided and corpus-grounded instruction generation framework. It comprises:

- **49,776** domain-specific instruction–response pairs generated from a filtered geohazard corpus
- **51,699** general instruction samples (Alpaca-GPT4) to preserve general instruction-following capability
- **~100K** total training pairs

The geohazard corpus draws from four sources:

| Source | Documents | Tokens |
|---|---|---|
| Open-access full-text papers | 1,613,089 | 788.9M |
| Licensed scientific books | 118,217 | 54.5M |
| Closed-access abstracts | 87,668 | 28.9M |
| Filtered C4 web corpus | 3,443 | 10.8M |
| **Total** | **1,822,417** | **883.1M** |

---

## RAG Integration

For standards-based engineering questions, GeohazardGPT is designed to be used with a retrieve-and-rerank RAG pipeline:

1. **Offline indexing** — technical specifications are chunked into sections/clauses and encoded with `Qwen3-Embedding` into a `ChromaDB` vector database
2. **Dense retrieval** — top-30 candidate clauses are retrieved via approximate nearest-neighbor search
3. **Cross-encoder re-ranking** — candidates are re-ranked using `Qwen3-Reranker-4B`; top-15 clauses are retained as final evidence
4. **Grounded generation** — retrieved clauses are injected into the prompt alongside the query

The RAG corpus covers national and sectoral standards in geotechnical investigation, foundation engineering, seismic design, transportation infrastructure, and hydraulic engineering.

---

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pengfali/GeohazardGPT"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto"
)

prompt = "What engineering measures should be adopted for a landslide with a tension crack at the crest and signs of local seepage?"

messages = [
    {"role": "system", "content": "You are an expert in geological disasters. This is a recommendation task."},
    {"role": "user", "content": prompt}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer([text], return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
response = tokenizer.decode(outputs[0][len(inputs.input_ids[0]):], skip_special_tokens=True)
print(response)
```

---

## Hardware Requirements

| Configuration | GPU Memory | Latency |
|---|---|---|
| GeohazardGPT (standalone) | ~10 GB | ~3.9 s/query |
| GeohazardGPT + RAG | ~26 GB | ~5.8 s/query |

Tested on NVIDIA A100 (80GB) under 4-bit deployment. The RAG configuration includes additional memory for `Qwen3-Embedding-4B` and `Qwen3-Reranker-4B`.

---


## Citation

If you use GeohazardGPT in your research, please cite:

```bibtex
@article{ge2025geohazardgpt,
  title={GeohazardGPT: Towards Large Language Models for Geohazards},
  author={Ge, Qi and Li, Pengfa and Dai, Yinhao and Li, Jin and An, Ni and Yu, Yang and Lv, Qing and Sun, Hongyue},
  journal={Under review},
  year={2025}
}
```

---

## License

This model is released under the Apache 2.0 License.

---