---
license: mit
language:
  - zh
  - en
pipeline_tag: text-generation
library_name: transformers
tags:
  - glm4_moe
  - meteorology
  - weather
  - weather-service
  - chinese
  - conversational
  - text-generation
---

# FengHe

![FengHe Overview](images/fenghe.png)

## Model Summary

**FengHe** is a meteorological service domain large language model developed by the Public Meteorological Service Center of China Meteorological Administration based on GLM-4.5-Air. It is further trained with over **50 million tokens** of high-quality meteorological service corpus and **490,000** scenario-oriented instruction-tuning examples.

FengHe is the first large language model in China specifically designed for the meteorological service domain. It is optimized for meteorological service requirement understanding, service-oriented content generation, meteorological reasoning and decision support, and meteorological service tool calling.

## Model Availability

This repository provides the complete FengHe model release, including the Model Card, model weights, configuration files, tokenizer files, training data description, and evaluation result visualizations.

The model can be loaded directly with Transformers, vLLM, or SGLang.

## Model Details

| Item                 | Description                                                                                    |
| -------------------- | ---------------------------------------------------------------------------------------------- |
| Model name           | FengHe                                                                                         |
| Model type           | Meteorological service domain large language model                                             |
| Base model           | GLM-4.5-Air                                                                                          |
| Architecture         | Mixture-of-Experts, Causal Language Model                                                      |
| Total parameters     | 106B                                                                                           |
| Activated parameters | 12B                                                                                            |
| Context length       | 128K tokens                                                                                    |
| Training precision   | BF16                                                                                           |
| License              | MIT                                                                                            |
| Languages            | Chinese, English                                                                               |
| Domain               | Meteorological services, weather forecasting, warning services, decision support, tool calling |

## Architecture

FengHe inherits the MoE architecture and hybrid reasoning capability of GLM. It supports reasoning-oriented generation for complex tasks and direct-response generation for real-time interactions.

| Parameter               | Value                |
| ----------------------- | -------------------- |
| Architecture            | `Glm4MoeForCausalLM` |
| Model type              | `glm4_moe`           |
| Number of hidden layers | 46                   |
| Hidden size             | 4096                 |
| Intermediate size       | 10944                |
| MoE intermediate size   | 1408                 |
| Attention heads         | 96                   |
| Key-value heads         | 8                    |
| Routed experts          | 128                  |
| Shared experts          | 1                    |
| Experts per token       | 8                    |
| Head dimension          | 128                  |
| Max position embeddings | 131072               |
| Vocabulary size         | 151552               |
| Activation function     | SiLU                 |
| Normalization           | RMSNorm              |
| RoPE theta              | 1000000              |
| Torch dtype             | BF16                 |

## Training Data

FengHe is trained on a high-quality meteorological service corpus built from multiple professional data sources, including:

* Meteorological books
* Meteorological industry standards
* National standards
* Warning translation texts
* Meteorological service reports
* Meteorological news and information
* Scenario-oriented meteorological question-answering instructions

Based on these sources, we constructed a high-quality meteorological service corpus of over **50 million tokens** and manually annotated **490,000** scenario-oriented meteorological instruction examples. These data improve the model's professional expression, business understanding, reasoning, decision-support capability, and tool-calling ability in the meteorological service domain.

## Capabilities

FengHe is optimized for meteorological service scenarios, with improvements in the following capabilities:

1. **Meteorological service requirement understanding**
   Understands user needs in weather forecasting, severe weather, warning services, event support, public services, and decision-support scenarios.

2. **Meteorological service content generation**
   Generates service-oriented meteorological texts for the public, industry users, and decision makers, including weather briefings, service reports, risk alerts, and warning interpretations.

3. **Meteorological reasoning and decision support**
   Performs analysis, summarization, judgment, and decision-support reasoning based on meteorological facts, forecast information, disaster risks, and service scenarios.

4. **Meteorological service tool calling**
   Supports tool calling and workflow orchestration for meteorological query, data retrieval, product generation, warning services, and related tasks.

## Evaluation

We evaluate FengHe on **MetsEval-1k**, a meteorological service evaluation benchmark containing **1,076 questions** across 4 dimensions:

* Meteorological service requirement understanding
* Meteorological service content generation
* Meteorological reasoning and decision support
* Meteorological service tool calling

![FengHe MetsEval-1k Evaluation](images/fenghe_evaluation_system.png)

The results show that FengHe achieves a higher overall score on MetsEval-1k than other mainstream general-purpose large language models, demonstrating its domain-specific advantages in meteorological services.

![FengHe MetsEval-1k Evaluation](images/fenghe_metseval_1k.png)

## Quick Start

### Transformers

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "PMSCCMA/FengHe"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {
        "role": "user",
        "content": "Please generate a public-facing risk advisory for heavy rainfall."
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.6,
    top_p=0.9,
)

response = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)

print(response)
```

### vLLM

```bash
vllm serve PMSCCMA/FengHe \
  --tensor-parallel-size 8 \
  --served-model-name fenghe
```

### SGLang

```bash
python3 -m sglang.launch_server \
  --model-path PMSCCMA/FengHe \
  --tp-size 8 \
  --host 0.0.0.0 \
  --port 30000 \
  --served-model-name fenghe
```

Call the OpenAI-compatible API:

```bash
curl -X POST "http://localhost:30000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  --data '{
    "model": "fenghe",
    "messages": [
      {
        "role": "user",
        "content": "Please generate a meteorological service advisory for transportation under typhoon impacts."
      }
    ],
    "temperature": 0.6,
    "top_p": 0.9,
    "max_tokens": 512
  }'
```

## Usage Statement

FengHe is released for meteorological services, research, industry application development, and agent system construction. Users should comply with applicable laws, regulations, industry standards, and meteorological service requirements. The model must not be used to generate fake warnings, misleading meteorological information, or any content that may pose public safety risks.

## License

This model is released under the **MIT License**. It can be used for research, commercial applications, and secondary development, subject to the terms of the MIT License.