|
|
--- |
|
|
language: |
|
|
- zh |
|
|
- en |
|
|
base_model: |
|
|
- Qwen/Qwen2.5-72B |
|
|
pipeline_tag: text-generation |
|
|
library_name: transformers |
|
|
--- |
|
|
## Introduction |
|
|
|
|
|
The Ming large language model (Ming‑LLM) is a domain‑specialized LLM for the energy sector. |
|
|
- We release both the base model and the supervised fine‑tuned (SFT) variant. |
|
|
- The Ming base model is initialized from the Qwen2.5‑72B base model and is subsequently adapted via continued pretraining on a high‑quality energy‑domain corpus. |
|
|
- The SFT variant is initialized from the Ming base model and is trained on instruction‑tuning datasets, including conversational QA, sentiment analysis, and information extraction, among others. |
|
|
- Both models demonstrate improved performance across the C‑Eval, CMMLU, MMLU, GSM8K, and IFEval benchmarks. |
|
|
|
|
|
## Model Parameters |
|
|
Base model: |
|
|
- sequence_len: 4096 |
|
|
- gradient_accumulation_steps: 128 |
|
|
- learning_rate: 1.0e-5 |
|
|
- lr_scheduler_type: cosine |
|
|
- warmup_ratio: 0 |
|
|
- num_train_epochs: 1.0 |
|
|
|
|
|
SFT: |
|
|
- sequence_len: 4096 |
|
|
- gradient_accumulation_steps: 128 |
|
|
- max learning rate: 2e-6 |
|
|
- max_grad_norm: 1.0 |
|
|
- lr_scheduler_type: cosine |
|
|
- warmup_ratio: 0.03 |
|
|
- num_train_epochs: 1.0 |
|
|
|
|
|
## Evaluation |
|
|
| Model | c-eval 5-shot | cmmlu 5-shot | mmlu 5-shot | GPQA 0-shot | BBH 0-shot | HellaSwag 10-shot | GSM8K | IFEVAL | |
|
|
|------------------------|---------------|--------------|-------------|-------------|------------|-------------------|-------|--------| |
|
|
| qwen2.5-72B-base | 89.72 | 89.75 | 84.79 | 37.88 | 85.81 | 94.93 | 89.99 | - | |
|
|
| ming1.0-base | 90.11 | 89.84 | 84.97 | 41.92 | 84.80 | 92.73 | 89.23 | - | |
|
|
| qwen2.5-72B-instruct | 87.97 | 87.26 | 84.18 | 36.87 | 83.68 | 92.65 | 89.69 | 82.81 | |
|
|
| ming1.0 | 90.08 | 89.94 | 85.12 | 37.88 | 85.24 | 94.20 | 91.43 | 78.74 | |
|
|
|
|
|
## Inference |
|
|
|
|
|
You can use Ming model with the standard HuggingFace transformers library: |
|
|
``` python |
|
|
import torch |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer |
|
|
|
|
|
dtype = torch.bfloat16 |
|
|
device_map = "auto" |
|
|
|
|
|
model_path = /model/path |
|
|
tokenizer = AutoTokenizer.from_pretrained( |
|
|
model_path, use_fast=True, trust_remote_code=True |
|
|
) |
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
model_path, torch_dtype=dtype, device_map=device_map, trust_remote_code=True |
|
|
) |
|
|
|
|
|
messages = [ |
|
|
{"role": "system", "content": "You are a helpful assistant."}, |
|
|
{"role": "user", "content": "who are you?"} |
|
|
] |
|
|
|
|
|
prompt = tokenizer.apply_chat_template( |
|
|
messages, |
|
|
tokenize=False, |
|
|
add_generation_prompt=True |
|
|
) |
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
|
|
|
|
|
with torch.no_grad(): |
|
|
output_ids = model.generate( |
|
|
**inputs, |
|
|
max_new_tokens=256, |
|
|
do_sample=True, |
|
|
temperature=0.3, |
|
|
top_p=0.9, |
|
|
repetition_penalty=1.1, |
|
|
eos_token_id=eos_token_id, |
|
|
pad_token_id=(tokenizer.pad_token_id or tokenizer.eos_token_id), |
|
|
streamer=None |
|
|
) |
|
|
gen_ids = output_ids[0, inputs["input_ids"].shape[1]:] |
|
|
text = tokenizer.decode(gen_ids, skip_special_tokens=False) |
|
|
``` |
|
|
## Bias, Risks, and Limitations |
|
|
- Like any base language model or fine-tuned model without safety filtering, these models can easily be prompted by users to generate harmful and sensitive content. |
|
|
- Such content may also be produced unintentionally, especially in cases involving bias, so we recommend that users consider the risks when applying this technology. |
|
|
- Additionally, many statements from Ming Model or any LLM are often inaccurate, so facts should be verified. |
|
|
|
|
|
## License and use |
|
|
- Ming1.0 is built with Qwen-2.5-72B. Qwen-2.5-72B is licensed under the Qwen LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved. |
|
|
- Subject to the Qwen LICENSE AGREEMENT, Ming1.0 is under MIT license. |
|
|
|
|
|
|
