|
|
--- |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- zh |
|
|
- en |
|
|
tags: |
|
|
- pytorch |
|
|
- transformers |
|
|
- causal-lm |
|
|
- qwen |
|
|
- verl |
|
|
- sft |
|
|
pipeline_tag: text-generation |
|
|
library_name: transformers |
|
|
--- |
|
|
|
|
|
# GMagoLi/test-upload |
|
|
|
|
|
这是一个基于Qwen架构的语言模型，使用VERL框架进行SFT训练。 |
|
|
|
|
|
## 模型描述 |
|
|
|
|
|
- **模型类型**: 因果语言模型 |
|
|
- **架构**: Qwen-32B |
|
|
- **训练框架**: VERL FSDP SFT Trainer |
|
|
- **语言**: 中文、英文 |
|
|
- **许可证**: Apache 2.0 |
|
|
|
|
|
## 使用方法 |
|
|
|
|
|
```python |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
import torch |
|
|
|
|
|
# 加载模型和tokenizer |
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
"GMagoLi/test-upload", |
|
|
trust_remote_code=True, |
|
|
torch_dtype=torch.bfloat16, |
|
|
device_map="auto" |
|
|
) |
|
|
tokenizer = AutoTokenizer.from_pretrained("GMagoLi/test-upload", trust_remote_code=True) |
|
|
|
|
|
# 推理示例 |
|
|
prompt = "你好，请介绍一下你自己。" |
|
|
inputs = tokenizer(prompt, return_tensors="pt") |
|
|
outputs = model.generate( |
|
|
**inputs, |
|
|
max_length=512, |
|
|
temperature=0.7, |
|
|
do_sample=True, |
|
|
pad_token_id=tokenizer.eos_token_id |
|
|
) |
|
|
response = tokenizer.decode(outputs[0], skip_special_tokens=True) |
|
|
print(response) |
|
|
``` |
|
|
|
|
|
## 训练信息 |
|
|
|
|
|
- **训练步数**: 2800 steps |
|
|
- **批大小**: 128 |
|
|
- **学习率调度**: Cosine with warmup |
|
|
- **混合精度**: bfloat16 |
|
|
- **数据集**: RepoCoder训练数据集v2.3 |
|
|
|
|
|
## 模型性能 |
|
|
|
|
|
该模型在代码生成和对话任务上表现出色，特别适合： |
|
|
- 代码生成和补全 |
|
|
- 技术问答 |
|
|
- 多轮对话 |
|
|
|
|
|
## 注意事项 |
|
|
|
|
|
- 模型较大(32B参数)，建议使用GPU推理 |
|
|
- 需要足够的显存(建议24GB+) |
|
|
- 支持量化推理以降低显存需求 |
|
|
|
|
|
## 引用 |
|
|
|
|
|
如果使用了本模型，请考虑引用： |
|
|
|
|
|
```bibtex |
|
|
@misc{qwen-repocoder-sft, |
|
|
title={Qwen RepoCoder SFT Model}, |
|
|
author={Your Name}, |
|
|
year={2025}, |
|
|
howpublished={\url{https://huggingface.co/GMagoLi/test-upload}} |
|
|
} |
|
|
``` |
|
|
|
