add model

Llama-Chinese

@@ -1 +0,0 @@
-Subproject commit 5f738c6278256f5ef89a35ba3a5c144320629dfc

Llama-Chinese/.gitattributes

@@ -0,0 +1 @@
+model/Atom-7B-Chat-pre/ filter=lfs diff=lfs merge=lfs -text

Llama-Chinese/.vscode/launch.json

@@ -0,0 +1,15 @@
+{
+    // Use IntelliSense to learn about possible attributes.
+    // Hover to view descriptions of existing attributes.
+    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "Python Debugger: Current File",
+            "type": "debugpy",
+            "request": "launch",
+            "program": "${file}",
+            "console": "integratedTerminal"
+        }
+    ]
+}

Llama-Chinese/README.md

@@ -0,0 +1,815 @@
+<p align="left">
+    <a href="README_EN.md">English</a> ｜ 中文
+</p>
+
+<h1 align="center">
+  Llama中文社区
+</h1>
+<p align="center" width="100%">
+  <img src="assets/llama.jpg" alt="Llama" style="width: 20%; display: block; margin: auto;"></a>
+</p>
+<p align="center">
+  <font face="黑体" color=orange size="6"> Llama3体验和微调已开放，最好的中文Llama大模型 </font>
+</p>
+
+<p align="center">
+🤗 <a href="https://huggingface.co/FlagAlpha" target="_blank">Hugging Face</a> • 🤖 <a href="https://www.modelscope.cn/organization/FlagAlpha/" target="_blank">ModelScope</a> • ✡️ <a href="https://wisemodel.cn/models/FlagAlpha/Atom-7B-Chat" target="_blank">WiseModel</a>
+</p> 
+
+<p align="center">
+  <a href="https://llama.family">Llama3.1 在线体验（包含Llama2）：https://llama.family</a>
+</p>
+<p align="center">
+  <a href="https://huggingface.co/FlagAlpha/Atom-7B-Chat">基于Llama的开源中文预训练大模型Atom</a>
+</p>
+
+</br></br>
+
+
+## 🗂️ 目录
+- [📌 Llama中文社区](#-llama中文社区)
+  * [🔥 社区介绍：Llama中文社区](#-社区介绍llama中文社区)
+  * [📢 最新动态](#-最新动态)
+  * [🤗 模型](#-模型)
+    + [🤗 中文预训练模型Atom-7B](#-中文预训练模型atom)
+    + [🤗 Llama3官方模型](#llama3官方模型)
+    + [🤗 Llama3中文微调模型](#llama3中文微调模型)
+    + [🤗 Llama2官方模型](#llama2官方模型)
+    + [🤗 Llama2中文微调模型](#llama2中文微调模型)
+  * [🌟 社区资源](#社区资源)
+
+
+- [📌 如何使用Llama模型?](#-如何使用llama模型)
+  - [快速上手-使用Anaconda](#快速上手-使用anaconda)
+  - [快速上手-使用Docker](#快速上手-使用docker)
+  - [快速上手-使用llama.cpp](#快速上手-使用llamacpp)
+  - [快速上手-使用gradio](#快速上手-使用gradio)
+  - [快速上手-构建API服务](#快速上手-构建api服务)
+  - [快速上手-使用ollama运行](#快速上手-使用ollama运行)
+
++ [🤖 模型预训练](#-模型预训练)
++ [💡 模型微调](#-模型微调)
+  - [Step1: 环境准备](#step1-环境准备)
+  - [Step2: 数据准备](#step2-数据准备)
+  - [Step3: 微调脚本](#step3-微调脚本)
+    * [LoRA微调](#lora微调)
+    * [全量参数微调](#全量参数微调)
+  - [Step4: 加载微调模型](#step4-加载微调模型)
+    * [LoRA微调](#lora微调-1)
+    * [全量参数微调](#全量参数微调-1)
++ [🍄 模型量化](#-模型量化)
+
++ [🚀 部署加速](#-部署加速)
+  - [TensorRT-LLM](#tensorrt-llm)
+  - [vLLM](#vllm)  
+  - [JittorLLMs](#jittorllms)
+  - [lmdeploy](#lmdeploy)
+
++ [💪 外延能力](#-外延能力)
+  - [LangChain](#langchain)
+    
+* [🥇 模型评测](#-模型评测)
+  + [Llama2和Llama3对比评测](#llama2和llama3对比评测)
+  + [Llama3模型评测](#llama3模型评测)
+  + [Llama2模型评测](#llama2模型评测)
+
+* [📖 学习中心](#-学习中心)
+    + [Llama3](#llama3)
+    + [Llama2](#llama2)
+      - [Meta官方对于Llama2的介绍](#meta官方对于llama2的介绍)
+    + [Llama相关论文](#llama相关论文)
+
+- [📌 其它](#-其它)
+  * [🎉 致谢](#-致谢)
+  * [🤔 问题反馈](#-问题反馈)
+
+## 📌 Llama中文社区
+
+### 🔥 社区介绍：llama中文社区
+
+欢迎来到Llama中文社区！我们是一个专注于Llama模型在中文方面的优化和上层建设的高级技术社区。
+**已经基于大规模中文数据，从预训练开始对Llama2模型进行中文能力的持续迭代升级【Done】**。**正在对Llama3模型进行中文能力的持续迭代升级【Doing】**
+我们热忱欢迎对大模型LLM充满热情的开发者和研究者加入我们的行列。
+
+<details>
+
+#### 为什么选择Llama中文社区？
+🚀 **高级工程师团队支持**：社区有一批专注为大家服务的NLP高级工程师，我们有着强大的技术支持和丰富的经验，为您提供专业的指导和帮助。
+
+🎯 **中文优化**：我们致力于在Llama模型的中文处理方面进行优化，探索适用于中文的最佳实践，以提升其性能和适应性【支持Llama2、Llama3】。
+
+💡 **创新交流**：我们拥有一支富有创造力和经验的社区成员团队，定期组织线上活动、技术研讨和经验分享，促进成员间的创新交流。
+
+🌐 **全球联结**：我们欢迎来自世界各地的开发者加入社区，构建一个开放、多元化的学习和交流平台。
+
+🤝 **开放共享**：我们鼓励社区成员开源分享代码和模型，推动合作共赢，共同促进中文NLP技术的发展。
+
+#### 社区活动
+🗓️ **线上讲座**：邀请行业内专家进行线上讲座，分享Llama在中文NLP领域的最新技术和应用，探讨前沿研究成果。
+
+💻 **项目展示**：成员可展示自己在Llama中文优化方面的项目成果，获得反馈和建议，促进项目协作。
+
+📚 **学习资源**：社区维护丰富的学习资料库，包括教程、文档和论文解读，为成员提供全面的学习支持。
+
+📝 **论文解读**：社区成员共同解读与Llama相关的最新研究论文，深入理解前沿算法和方法。
+
+🎉 **主题活动**：定期举办各类主题活动，包括挑战赛、黑客马拉松和技术沙龙，让社区成员在轻松愉快的氛围中交流和学习。
+
+🌟 **奖励计划**：我们设立奖励计划，对社区中积极参与、贡献优秀的成员给予荣誉和奖励，激励更多优秀人才的加入。
+
+📈 **技术咨询**：我们提供技术咨询服务，解答您在Llama开发和优化过程中遇到的问题，助您快速攻克难关。
+
+🚀 **项目合作**：鼓励成员间的项目合作，共同探索Llama在实际应用中的潜力，打造创新解决方案。
+
+
+#### 立即加入我们！
+📚 **愿景**：无论您是对Llama已有研究和应用经验的专业开发者，还是对Llama中文优化感兴趣并希望深入探索的新手，我们都热切期待您的加入。在Llama中文社区，您将有机会与行业内顶尖人才共同交流，携手推动中文NLP技术的进步，开创更加美好的技术未来！
+
+🔗 **温馨提示**：本社区为专业技术交流平台，我们热切期望志同道合的开发者和研究者加入。请遵守社区准则，共同维护积极向上的学习氛围。感谢您的理解和支持！
+
+</details>
+
+### 📢 最新动态
+
+【最新】2024年07月24日：开源最强[Llama 3.1](https://llama.meta.com/docs/overview)模型发布，包含8B、70B和405B！
+
+【最新】2024年07月16日：[社区论坛](https://forum.llamafamily.cn/)上线，有大模型问题，就找Llama中文社区！
+
+【最新】2024年05月15日：支持ollama运行Llama3-Chinese-8B-Instruct、Atom-7B-Chat，[详细使用方法](https://github.com/LlamaFamily/Llama-Chinese?tab=readme-ov-file#%E5%BF%AB%E9%80%9F%E4%B8%8A%E6%89%8B-%E4%BD%BF%E7%94%A8ollama%E8%BF%90%E8%A1%8C)。
+
+【最新】2024年04月23日：社区增加了llama3 8B中文微调模型[Llama3-Chinese-8B-Instruct](https://github.com/LlamaFamily/Llama-Chinese?tab=readme-ov-file#llama3%E4%B8%AD%E6%96%87%E5%BE%AE%E8%B0%83%E6%A8%A1%E5%9E%8B)以及对应的[免费API调用](https://llama.family/docs/chat-completion-v1)。
+ 
+【最新】2024年04月19日：社区增加了llama3 8B、llama3 70B[在线体验链接](https://llama.family/chat/#/)。
+
+【最新】2024年04月14日：社区更新了四个专家角色：心理咨询师、羊驼夸夸 、律师、医生。链接：[角色role](https://llama.family/tools/#/agent)。
+
+【最新】2024年04月10日：Atom-7B-Chat 模型回答内容相较之前更为丰富、增强了模型的指令遵循能力和回答稳定性、优化了ppo的奖励模型。下载链接[modelscope](https://modelscope.cn/models/FlagAlpha/Atom-7B-Chat)、[Huggingface](https://huggingface.co/FlagAlpha/Atom-7B-Chat)。
+
+【最新】2024年04月01日：社区上线了Llama中文[应用平台](https://llama.family/store)；同时如果你有优秀的的应用需要推广可以填写[申请表](https://atomecho.feishu.cn/share/base/form/shrcnFqpN71OmBoXDCT6y0TQgIc)。
+
+【最新】2024年03月08日：开放了免费API供大家使用，包含（Atom-1B,7B,13B 3种中文大模型）[API使用链接](https://llama.family/docs/chat-completion-v1)
+
+【最新】2024年04月14日：社区更新了四个专家角色：心理咨询师、羊驼夸夸 、律师、医生。链接：[角色role](https://llama.family/tools/#/agent)。
+
+【最新】2024年04月10日：Atom-7B-Chat 模型回答内容相较之前更为丰富、增强了模型的指令遵循能力和回答稳定性、优化了ppo的奖励模型。下载链接[modelscope](https://modelscope.cn/models/FlagAlpha/Atom-7B-Chat)、[Huggingface](https://huggingface.co/FlagAlpha/Atom-7B-Chat)。
+
+【最新】2024年04月01日：社区上线了Llama中文[应用平台](https://llama.family/store)；同时如果你有优秀的的应用需要推广可以填写[申请表](https://atomecho.feishu.cn/share/base/form/shrcnFqpN71OmBoXDCT6y0TQgIc)。
+
+【最新】2024年03月28日：[社区免费公开课](https://mp.weixin.qq.com/s/CsturoU1pOX11CqVnZgu2A)。
+
+【最新】2024年03月08日：开放了免费API供大家使用，包含（Atom-1B,7B,13B 3种中文大模型）[API使用链接](https://llama.family/docs/chat-completion-v1)
+
+【最新】2023年10月8日：新增清华大学JittorLLMs的推理加速功能[JittorLLMs](#jittorllms)！
+
+<details>
+
+- 2023年9月12日：更新预训练版本[Atom-7B](https://huggingface.co/FlagAlpha/Atom-7B)和对话版本[Atom-7B-Chat](https://huggingface.co/FlagAlpha/Atom-7B-Chat)模型参数，最新的中文预训练数据量为2.7TB token，训练进程见[llama.family](https://llama.family/)！
+
+- 2023年9月2日：新增模型[预训练代码](#-模型预训练)和[全量参数微调代码](#-模型微调)！
+  
+- 2023年8月28日：发布基于Llama2进行中文预训练的开源大模型[Atom-7B](https://huggingface.co/FlagAlpha/Atom-7B)，并将持续更新，详情参考[社区��众号文章](https://mp.weixin.qq.com/s/Bdx0JTVh1kgPn5ydYxIkEw)！
+
+- 2023年8月26日：提供[FastAPI](#fastapi接口搭建)接口搭建脚本！
+
+- 2023年8月26日：提供将Meta原始模型参数转换为兼容Hugging Face的[格式转化脚本](https://github.com/LlamaFamily/Llama-Chinese/blob/main/scripts/convert2hf/README.md)！
+
+- 2023年8月26日：新增[Code Llama](#-代码模型)模型！
+
+- 2023年8月15日：新增[PEFT加载微调模型参数](#加载微调模型)的代码示例！
+
+- 2023年8月14日：[大模型数据共享训练平台](https://llama.family)上线，没有算力也能参与大模型训练，社区每位成员贡献的数据都将决定模型能力的未来走向！
+
+- 2023年8月3日：新增FasterTransformer和vLLM的GPU[推理加速](#-推理加速)支持！
+
+- 2023年7月31日：【重磅】国内首个真正意义上的Llama2中文大模型发布！详情参见[社区公众号文章](https://mp.weixin.qq.com/s/lExUU7z_MvgJ7tzQPF8tUQ)
+
+- 2023年7月28日：通过[Docker部署](#docker部署问答接口)问答接口！
+
+- 2023年7月27日：新增[LangChain](#langchain)支持！
+
+- 2023年7月26日：新增Llama2-13B中文微调参数的[4bit量化压缩版本](#-模型量化)！
+
+- 2023年7月25日：社区微信公众号“Llama中文社区”欢迎大家关注，获取最新分享和动态！
+
+- 2023年7月24日：[FlagAlpha](https://huggingface.co/FlagAlpha)新增Llama2-13B中文微调参数！
+
+- 2023年7月24日：[llama.family](https://llama.family/)新增Llama2-70B在线体验！
+
+- 2023年7月23日：Llama2中文微调参数发布至Hugging Face仓库[FlagAlpha](https://huggingface.co/FlagAlpha)！
+
+- 2023年7月22日：Llama2在线体验链接[llama.family](https://llama.family/)上线，同时包含Meta原版和中文微调版本！
+
+- 2023年7月21日：评测了Meta原始版Llama2 Chat模型的[中文问答能力](#-模型评测)！
+
+- 2023年7月21日：新增Llama2模型的Hugging Face版本国内下载地址！
+
+- 2023年7月20日：新增[飞书知识库文档](https://chinesellama.feishu.cn/wiki/space/7257824476874768388?ccm_open_type=lark_wiki_spaceLink)，欢迎大家一起共建！
+
+- 2023年7月20日：国内Llama2最新下载地址上线！
+
+- 2023年7月19日：正式启动Llama2模型的中文预训练，关注我们获取实时动态！
+
+- 2023年7月19日：Llama2国内下载地址正在启动，敬请期待！
+
+- 2023年7月19日：开启Llama2中文社区，欢迎大家加入！
+
+</details>
+
+
+### 🤗 模型
+
+#### 🔵 中文预训练模型Atom
+
+**原子大模型Atom**由Llama中文社区和原子回声联合打造。
+
+|  类别  | 模型名称        | 🤗模型加载名称                  | 下载地址                                                     |
+| --------------- | --------------- | ------------------------------ | ------------------------------------------------------------ |
+|  预训练  | Atom-7B  | FlagAlpha/Atom-7B  | [HuggingFace](https://huggingface.co/FlagAlpha/Atom-7B) \| [ModelScope](https://modelscope.cn/models/FlagAlpha/Atom-7B) \| [WiseModel](https://wisemodel.cn/models/FlagAlpha/Atom-7B) |
+|  Chat  | Atom-7B-Chat  | FlagAlpha/Atom-7B-Chat  | [HuggingFace](https://huggingface.co/FlagAlpha/Atom-7B-Chat) \| [ModelScope](https://modelscope.cn/models/FlagAlpha/Atom-7B-Chat) \| [WiseModel](https://wisemodel.cn/models/FlagAlpha/Atom-7B-Chat)|
+
+Atom系列模型包含Atom-13B、Atom-7B和Atom-1B，基于Llama2做了中文能力的持续优化。Atom-7B和Atom-7B-Chat目前已完全开源，支持商用，可在[Hugging Face](https://huggingface.co/FlagAlpha)仓库获取模型，详情见[Atom-7B下载](#基于llama2的中文预训练模型atom)。Atom大模型针对中文做了以下优化：
+
+- 大规模的中文数据预训练
+
+原子大模型Atom在Llama2的基础上，采用大规模的中文数据进行持续预训练，包含百科、书籍、博客、新闻、公告、小说、金融数据、法律数据、医疗数据、代码数据、专业论文数据、中文自然语言处理竞赛数据集等，详见[📝 数据来源](#-数据来源)。
+
+同时对庞大的数据进行了过滤、打分、去重，筛选出超过1T token的高质量中文数据，持续不断加入训练迭代中。
+
+- 更高效的中文词表
+为了提高中文文本处理的效率，我们针对Llama2模型的词表进行了深度优化。首先，我们基于数百G的中文文本，在该模型词表的基础上扩展词库至65,000个单词。经过测试，我们的改进使得中文编码/解码速度提高了约350％。此外，我们还扩大了中文字符集的覆盖范围，包括所有emoji符号😊。这使得生成带有表情符号的文章更加高效。
+
+- 自适应上下文扩展
+Atom大模型默认支持4K上下文，利用位置插值PI和Neural Tangent Kernel （NTK）方法，经过微调可以将上下文长度扩增到32K。
+
+- 📝 中文数据
+
+我们通过以下数据来优化Llama2的中文能力:
+
+| 类型                                                       | 描述                                                         |
+| ---------------------------------------------------------- | ------------------------------------------------------------ |
+| 网络数据                                                   | 互联网上公开的网络数据，挑选出去重后的高质量中文数据，涉及到百科、书籍、博客、新闻、公告、小说等高质量长文本数据。 |
+| [Wikipedia](https://github.com/goldsmith/Wikipedia)        | 中文Wikipedia的数据                                          |
+| [悟道](https://github.com/BAAI-WuDao/Model)                | 中文悟道开源的200G数据                                       |
+| [Clue](https://github.com/CLUEbenchmark/CLUEDatasetSearch) | Clue开放的中文预训练数据，进行清洗后的高质量中文长文本数据   |
+| 竞赛数据集                                                 | 近年来中文自然语言处理多任务竞赛数据集，约150个              |
+| [MNBVC](https://github.com/esbatmop/MNBVC)                 | MNBVC 中清洗出来的部分数据集
+
+社区提供预训练版本Atom-7B和基于Atom-7B进行对话微调的模型参数供开放下载，关于模型的进展详见社区官网[llama.family](https://llama.family)。
+
+#### Llama3官方模型
+
+|  类别  | 模型名称   | 🤗模型加载名称             | 下载地址                                                     |
+|  ----------  | ---------- | ------------------------- | --------------------- |
+|  预训练  | Llama3-8B  | meta-llama/Meta-Llama-3-8B  | [HuggingFace](https://huggingface.co/meta-llama/Meta-Llama-3-8B) \| [百度网盘](https://pan.baidu.com/s/1gBZ7wEn3gC8VRok0Onh9BQ?pwd=8frq) |
+|  预训练  | Llama3-70B | meta-llama/Meta-Llama-3-70B | [HuggingFace](https://huggingface.co/meta-llama/Meta-Llama-3-7B) \| [百度网盘](https://pan.baidu.com/s/1gBZ7wEn3gC8VRok0Onh9BQ?pwd=8frq) |
+|  对话模型  | Llama3-8B-Chat  | meta-llama/Meta-Llama-3-8B-Instruct  | [HuggingFace](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) \| [百度网盘](https://pan.baidu.com/s/1gBZ7wEn3gC8VRok0Onh9BQ?pwd=8frq) |
+|  对话模型  | Llama3-70B-Chat  | meta-llama/Meta-Llama-3-70B-Instruct  | [HuggingFace](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) \| [百度网盘](https://pan.baidu.com/s/1gBZ7wEn3gC8VRok0Onh9BQ?pwd=8frq) |
+
+#### Llama3中文微调模型
+
+|  类别  | 模型名称   | 🤗模型加载名称             | 下载地址                                                     |
+|  ----------  | ---------- | ------------------------- | --------------------- |
+|  对话模型  | Llama3-Chinese-8B-Instruct  | FlagAlpha/Llama3-Chinese-8B-Instruct  | [HuggingFace](https://huggingface.co/FlagAlpha/Llama3-Chinese-8B-Instruct) \| [modelscope](https://modelscope.cn/models/FlagAlpha/Llama3-Chinese-8B-Instruct/summary) \| [wisemodel](https://wisemodel.cn/models/FlagAlpha/Llama3-Chinese-8B-Instruct/file) |
+
+
+#### Llama2官方模型
+
+<details>
+
+|  类别  | 模型名称   | 🤗模型加载名称             | 下载地址                                                     |
+|  ----------  | ---------- | ------------------------- | --------------------- |
+|  预训练  | Llama2-7B  | meta-llama/Llama-2-7b-hf  | [HuggingFace](https://huggingface.co/meta-llama/Llama-2-7b-hf) \| [迅雷网盘](https://pan.xunlei.com/s/VN_t0dUikZqOwt-5DZWHuMvqA1?pwd=66ep) |
+|  预训练  | Llama2-13B | meta-llama/Llama-2-13b-hf | [HuggingFace](https://huggingface.co/meta-llama/Llama-2-13b-hf) \| [迅雷网盘](https://pan.xunlei.com/s/VN_yT_9G8xNOz0SDWQ7Mb_GZA1?pwd=yvgf) |
+|  预训练  | Llama2-70B | meta-llama/Llama-2-70b-hf | [HuggingFace](https://huggingface.co/meta-llama/Llama-2-70b-hf) |
+|  Chat  | Llama2-7B-Chat  | meta-llama/Llama-2-7b-chat-hf  | [HuggingFace](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) \| [迅雷网盘](https://pan.xunlei.com/s/VN_oaV4BpKFgKLto4KgOhBcaA1?pwd=ufir) |
+|  Chat  | Llama2-13B-Chat | meta-llama/Llama-2-13b-chat-hf | [HuggingFace](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf) \| [迅雷网盘](https://pan.xunlei.com/s/VN_yA-9G34NGL9B79b3OQZZGA1?pwd=xqrg) |
+|  Chat  | Llama2-70B-Chat | meta-llama/Llama-2-70b-chat-hf | [HuggingFace](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) \| [迅雷网盘](https://pan.xunlei.com/s/VNa_vCGzCy3h3N7oeFXs2W1hA1?pwd=uhxh#) |
+| Code  | CodeLlama-7b    |   meta-llama/Llama-2-70b-chat-hf              | [迅雷网盘](https://pan.baidu.com/s/1cIPzdNywWLvQI7_2QanOEQ?pwd=zfwi) |
+| Code  | CodeLlama-7b-Python    |   meta-llama/Llama-2-70b-chat-hf              | [迅雷网盘](https://pan.baidu.com/s/1liY8klGoDagYbpw-g-oFag?pwd=i952) |
+| Code  | CodeLlama-7b-Instruct    |   meta-llama/Llama-2-70b-chat-hf              | [迅雷网盘](https://pan.baidu.com/s/108o9_DT2E_vfSGtOnDCQVw?pwd=zkt9) |
+| Code  | CodeLlama-13b    |   meta-llama/Llama-2-70b-chat-hf              | [迅雷网盘](https://pan.baidu.com/s/1lLaeHv0XEBv0iiZzI1dpnw?pwd=qn99) |
+| Code  | CodeLlama-13b-Python    |   meta-llama/Llama-2-70b-chat-hf              | [迅雷网盘](https://pan.baidu.com/s/1OLVfvZS_oqL3oqMKwsI87w?pwd=a78k) |
+| Code  | CodeLlama-13b-Instruct    |   meta-llama/Llama-2-70b-chat-hf              | [迅雷网盘](https://pan.baidu.com/s/1HyxJl4w8wElgkZRh2ATrXQ?pwd=seg6) |
+| Code  | CodeLlama-34b    |   meta-llama/Llama-2-70b-chat-hf              | [迅雷网盘](https://pan.baidu.com/s/1vEw0pFgIkctPUN4_5_6pIQ?pwd=q8eu) |
+
+Meta官方在2023年8月24日发布了Code Llama，基于代码数据对Llama2进行了微调，提供三个不同功能的版本：基础模型（Code Llama）、Python专用模型（Code Llama - Python）和指令跟随模型（Code Llama - Instruct），包含7B、13B、34B三种不同参数规模。不同模型能力区别如下表所示：
+
+|  模型类别          |        模型名称         | 代码续写 | 代码填充 | 指令编程 |
+|-----------------------|------------------------|------|------|------|
+| Code Llama            | CodeLlama-7b           | ✅    | ✅    | ❌    |
+|                       | CodeLlama-13b          | ✅    | ✅    | ❌    |
+|                       | CodeLlama-34b          | ✅    | ❌    | ❌    |
+| Code Llama - Python   | CodeLlama-7b-Python    | ✅    | ❌    | ❌    |
+|                       | CodeLlama-13b-Python   | ✅    | ❌    | ❌    |
+|                       | CodeLlama-34b-Python   | ✅    | ❌    | ❌    |
+| Code Llama - Instruct | CodeLlama-7b-Instruct  | ❌    | ✅    | ✅    |
+|                       | CodeLlama-13b-Instruct | ❌    | ✅    | ✅    |
+|                       | CodeLlama-34b-Instruct | ❌    | ❌    | ✅    |
+
+关于Code Llama的详细信息可以参考官方Github仓库[codellama](https://github.com/facebookresearch/codellama)。
+
+</details>
+
+#### Llama2中文微调模型
+
+我们基于中文指令数据集对Llama2-Chat模型进行了微调，使得Llama2模型有着更强的中文对话能力。LoRA参数以及与基础模型合并的参数均已上传至[Hugging Face](https://huggingface.co/FlagAlpha)，目前包含7B和13B的模型。
+
+|  类别  | 模型名称   | 🤗模型加载名称             | 基础模型版本 |    下载地址                                                     |
+|  ----------  | ---------- | ------------- |  ----------------- | ------------------- |
+|  合并参数 | Llama2-Chinese-7b-Chat | FlagAlpha/Llama2-Chinese-7b-Chat  |    meta-llama/Llama-2-7b-chat-hf       |[HuggingFace](https://huggingface.co/FlagAlpha/Llama2-Chinese-7b-Chat)  |
+|  合并参数 | Llama2-Chinese-13b-Chat | FlagAlpha/Llama2-Chinese-13b-Chat|     meta-llama/Llama-2-13b-chat-hf     |[HuggingFace](https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat) |
+|  LoRA参数 | Llama2-Chinese-7b-Chat-LoRA  | FlagAlpha/Llama2-Chinese-7b-Chat-LoRA  |     meta-llama/Llama-2-7b-chat-hf      |[HuggingFace](https://huggingface.co/FlagAlpha/Llama2-Chinese-7b-Chat-LoRA) |
+|  LoRA参数 | Llama2-Chinese-13b-Chat-LoRA | FlagAlpha/Llama2-Chinese-13b-Chat-LoRA |     meta-llama/Llama-2-13b-chat-hf     |[HuggingFace](https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat-LoRA) |
+
+
+### 社区资源
+社区资源的丰富性是社区发展的重要保障，它涵盖了各种方面，其中包括但不限于以下四个方面：算力、数据、论坛和应用。在这些方面的积极发展与充分利用，将为社区成员提供更多的机会和支持，推动整个社区向着更加繁荣的方向发展。更多的内容请看[llama.family](https://llama.family/)
+
+<details>
+
+#### 💻 算力
+- 提供低于市场价格的算力资源，可用于各类计算任务，如深度学习模型的训练、推理等。
+- 为社区成员提供专属的在线推理服务，让用户可以快速有效地对模型进行推理操作。
+- 提供一键在线微调服务，使用户可以方便地对模型进行微调，以适应不同的任务和数据。
+
+#### 📊 数据
+- 开放丰富的训练数据资源，覆盖多个领域和行业，为模型训练提供充足的数据支持。
+- 提供高质量、多样化的数据集，以满足不同用户的需求，并支持数据共享和交流，促进数据资源的充分利用。
+
+#### 💬 论坛
+- 社区论坛为社区成员提供了一个在线交流和讨论技术问题的平台。
+- 在论坛上，用户可以分享经验、提出问题、解答疑惑，促进技术交流和合作。
+- 论坛还可以定期举办线上活动、研讨会等，增进社区成员之间的联系和了解。
+
+#### 📱 应用
+- 免费提供应用推广展示位，让开发者可以将他们的应用充分展示给社区成员。
+- 提供推广的帮助，包括但不限于宣传推广、用户引导等服务，帮助应用获得更多的曝光和用户。
+- 通过社区平台，为优秀的应用提供合作机会，促进应用开发者之间的合作和交流，共同推动应用的发展和壮大。
+
+</details>
+
+## 📌 如何使用Llama模型?
+
+
+你可以选择下面的快速上手的任一种方式，开始使用 Llama 系列��型。推荐使用[中文预训练对话模型](#llama2中文预训练模型atom-7b)进行使用，对中文的效果支持更好。
+
+
+### 快速上手-使用Anaconda
+
+第 0 步：前提条件
+- 确保安装了 Python 3.10 以上版本。
+
+第 1 步：准备环境
+
+如需设置环境，安装所需要的软件包，运行下面的命令。
+```bash
+git clone https://github.com/LlamaFamily/Llama-Chinese.git
+cd Llama-Chinese
+pip install -r requirements.txt
+```
+
+第 2 步：下载模型
+
+你可以从以下来源下载Atom-7B-Chat模型。
+- [HuggingFace](https://huggingface.co/FlagAlpha)
+- [ModelScope](https://modelscope.cn/organization/FlagAlpha)
+- [WiseModel](https://wisemodel.cn/models/FlagAlpha/Atom-7B-Chat)
+
+第 3 步：进行推理
+
+使用Atom-7B-Chat模型进行推理
+创建一个名为 quick_start.py 的文件，并将以下内容复制到该文件中。
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+device_map = "cuda:0" if torch.cuda.is_available() else "auto"
+model = AutoModelForCausalLM.from_pretrained('FlagAlpha/Atom-7B-Chat',device_map=device_map,torch_dtype=torch.float16,load_in_8bit=True,trust_remote_code=True,use_flash_attention_2=True)
+model =model.eval()
+tokenizer = AutoTokenizer.from_pretrained('FlagAlpha/Atom-7B-Chat',use_fast=False)
+tokenizer.pad_token = tokenizer.eos_token
+input_ids = tokenizer(['<s>Human: 介绍一下中国\n</s><s>Assistant: '], return_tensors="pt",add_special_tokens=False).input_ids
+if torch.cuda.is_available():
+  input_ids = input_ids.to('cuda')
+generate_input = {
+    "input_ids":input_ids,
+    "max_new_tokens":512,
+    "do_sample":True,
+    "top_k":50,
+    "top_p":0.95,
+    "temperature":0.3,
+    "repetition_penalty":1.3,
+    "eos_token_id":tokenizer.eos_token_id,
+    "bos_token_id":tokenizer.bos_token_id,
+    "pad_token_id":tokenizer.pad_token_id
+}
+generate_ids  = model.generate(**generate_input)
+text = tokenizer.decode(generate_ids[0])
+print(text)
+```
+
+运行 quick_start.py 代码。
+```bash
+python quick_start.py
+```
+
+### 快速上手-使用Docker
+
+详情参见：[Docker部署](https://github.com/LlamaFamily/Llama-Chinese/blob/main/docs/chat_gradio_guide.md)
+
+第 1 步：准备docker镜像，通过docker容器启动[chat_gradio.py](../examples/chat_gradio.py)
+```bash
+git clone https://github.com/LlamaFamily/Llama-Chinese.git
+
+cd Llama-Chinese
+
+docker build -f docker/Dockerfile -t flagalpha/llama2-chinese:gradio .
+```
+
+第 2 步：通过docker-compose启动chat_gradio
+```bash
+cd Llama-Chinese/docker
+docker-compose up -d --build
+```
+
+### 快速上手-使用llama.cpp
+详情参见：[使用llama.cpp](https://github.com/LlamaFamily/Llama-Chinese/blob/main/inference-speed/CPU/ggml/README.md)
+
+### 快速上手-使用gradio
+基于gradio搭建的问答界面，实现了流式的输出，将下面代码复制到控制台运行，以下代码以Atom-7B-Chat模型为例，不同模型只需修改一下面的model_name_or_path对应的模型名称就好了😊
+```
+python examples/chat_gradio.py --model_name_or_path FlagAlpha/Atom-7B-Chat
+```
+
+### 快速上手-构建API服务
+使用FastChat构建和OpenAI一致的推理服务接口。
+
+<details>
+第 0 步：前提条件
+
+安装fastchat
+```bash
+pip3 install "fschat[model_worker,webui]"
+```
+第 1 步：启动Restful API
+
+开启三个控制台分别执行下面的三个命令
+- 首先启动controler
+```bash
+python3 -m fastchat.serve.controller \
+--host localhost \
+--port 21001
+```
+
+- 启动模型
+```bash
+CUDA_VISIBLE_DEVICES="0" python3 -m fastchat.serve.model_worker --model-path /path/Atom-7B-Chat \
+--host localhost \
+--port 21002 \
+--worker-address "http://localhost:21002" \
+--limit-worker-concurrency 5 \
+--stream-interval 2 \
+--gpus "1" \
+--load-8bit
+```
+
+- 启动RESTful API 服务
+```bash
+python3 -m fastchat.serve.openai_api_server \
+--host localhost \
+--port 21003 \
+--controller-address http://localhost:21001
+```
+
+第 2 步：测试api服务
+
+执行下面的python代码测试上面部署的api服务
+```python
+# coding=utf-8
+import json
+import time
+import urllib.request
+import sys
+import requests
+
+def test_api_server(input_text):
+    header = {'Content-Type': 'application/json'}
+
+    data = {
+          "messages": [{"role": "system", "content": ""}, {"role": "user", "content": input_text}],
+          "temperature": 0.3, 
+          "top_p" : 0.95, 
+          "max_tokens": 512, 
+          "model": "LLama2-Chinese-13B",
+          "stream" : False,
+          "n" : 1,
+          "best_of": 1, 
+          "presence_penalty": 1.2, 
+          "frequency_penalty": 0.2,           
+          "top_k": 50, 
+          "use_beam_search": False, 
+          "stop": [], 
+          "ignore_eos" :False,
+          "logprobs": None
+    }
+    response = requests.post(
+        url='http://127.0.0.1:21003/v1/chat/completions',
+        headers=header,
+        data=json.dumps(data).encode('utf-8')
+    )
+
+    result = None
+    try:
+        result = json.loads(response.content)
+        print(json.dumps(data, ensure_ascii=False, indent=2))
+        print(json.dumps(result, ensure_ascii=False, indent=2))
+
+    except Exception as e:
+        print(e)
+
+    return result
+
+if __name__ == "__main__":
+    test_api_server("如何去北京?")
+```
+
+</details>
+
+
+### 快速上手-使用ollama运行
+
+1. 首先需要安装ollama工具
+
+安装方法参考：[https://ollama.com](https://ollama.com/)
+
+2. ollama运行Llama3-Chinese-8B-Instruct、Atom-7B-Chat
+
+ollama运行基于Llama3进行中文微调的大模型[Llama3-Chinese-8B-Instruct](https://huggingface.co/FlagAlpha/Llama3-Chinese-8B-Instruct)
+
+打开命令行执行命令
+```
+ollama run llamafamily/llama3-chinese-8b-instruct
+```
+
+ollama运行基于Llama2进行中文预训练的开源大模型[Atom-7B-Chat](https://huggingface.co/FlagAlpha/Atom-7B-Chat)
+
+打开命令行执行命令
+```
+ollama run llamafamily/atom-7b-chat
+```
+
+
+## 🤖 模型预训练
+虽然Llama2的预训练数据相对于第一代LLaMA扩大了一倍，但是中文预训练数据的比例依然非常少，仅占0.13%，这也导致了原始Llama2的中文能力较弱。为了能够提升模型的中文能力，可以采用微调和预训练两种路径，其中：
+- 微调需要的算力资源少，能够快速实现一个中文Llama的雏形。但缺点也显而易见，只能激发基座模型已有的中文能力，由于Llama2的中文训练数据本身较少，所以能够激发的能力也有限，治标不治本。
+
+- 基于大规模中文语料进行预训练，成本高，不仅需要大规模高质量的中文数据，也需要大规模的算力资源。但是优点也显而易见，就是能从模型底层优化中文能力，真正达到治本的效果，从内核为大模型注入强大的中文能力。
+
+我们为社区提供了Llama模型的预训练代码，以及[中文测试语料](https://github.com/LlamaFamily/Llama-Chinese/tree/main/data)，更多数据可以参考[中文语料](#-中文数据)。具体代码和配置如下：
+- 模型预训练脚本：[train/pretrain/pretrain.sh](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/pretrain/pretrain.sh)
+- 预训练实现代码：[train/pretrain/pretrain_clm.py](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/pretrain/pretrain_clm.py)
+- [DeepSpeed](https://github.com/microsoft/DeepSpeed)加速：
+  - 对于单卡训练，可以采用ZeRO-2的方式，参数配置见 [train/pretrain/ds_config_zero2.json](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/pretrain/ds_config_zero2.json)
+  - 对于多卡训练，可以采用ZeRO-3的方式，参数配置见 [train/pretrain/ds_config_zero3.json](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/pretrain/ds_config_zero3.json)
+- 训练效果度量指标：[train/pretrain/accuracy.py](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/pretrain/accuracy.py)
+
+## 💡 模型微调
+
+本仓库中同时提供了LoRA微调和全量参数微调代码，关于LoRA的详细介绍可以参考论文“[LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)”以及微软Github仓库[LoRA](https://github.com/microsoft/LoRA)。
+
+### Step1: 环境准备
+
+根据[requirements.txt](https://github.com/LlamaFamily/Llama-Chinese/blob/main/requirements.txt)安装对应的环境依赖。
+
+### Step2: 数据准备
+在data目录下提供了一份用于模型sft的数据样例：
+- 训练数据：[data/train_sft.csv](https://github.com/LlamaFamily/Llama-Chinese/blob/main/data/train_sft.csv)
+- 验证数据：[data/dev_sft.csv](https://github.com/LlamaFamily/Llama-Chinese/blob/main/data/dev_sft.csv)
+
+每个csv文件中包含一列“text”，每一行为一个训练样例，每个训练样例按照以下格式将问题和答案组织为模型输入，您可以按照以下格式自定义训练和验证数据集：
+```
+"<s>Human: "+问题+"\n</s><s>Assistant: "+答案+"\n"</s>
+```
+例如，
+```
+<s>Human: 用一句话描述地球为什么是独一无二的。</s><s>Assistant: 因为地球是目前为止唯一已知存在生命的行星。</s>
+```
+
+### Step3: 微调脚本
+
+#### LoRA微调
+LoRA微调脚本见：[train/sft/finetune_lora.sh](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/sft/finetune_lora.sh)，关于LoRA微调的具体实现代码见[train/sft/finetune_clm_lora.py](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/sft/finetune_clm_lora.py)，单机多卡的微调可以通过修改脚本中的`--include localhost:0`来实现。
+
+#### 全量参数微调
+全量参数微调脚本见：[train/sft/finetune.sh](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/sft/finetune.sh)，关于全量参数微调的具体实现代码见[train/sft/finetune_clm.py](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/sft/finetune_clm.py)。
+
+
+### Step4: 加载微调模型
+
+#### LoRA微调
+基于LoRA微调的模型参数见：[基于Llama2的中文微调模型](#llama2中文微调模型)，LoRA参数需���和基础模型参数结合使用。
+
+通过[PEFT](https://github.com/huggingface/peft)加载预训练模型参数和微调模型参数，以下示例代码中，base_model_name_or_path为预训练模型参数保存路径，finetune_model_path为微调模型参数保存路径。
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from peft import PeftModel,PeftConfig
+# 例如: finetune_model_path='FlagAlpha/Llama2-Chinese-7b-Chat-LoRA'
+finetune_model_path=''  
+config = PeftConfig.from_pretrained(finetune_model_path)
+# 例如: base_model_name_or_path='meta-llama/Llama-2-7b-chat'
+tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path,use_fast=False)
+tokenizer.pad_token = tokenizer.eos_token
+device_map = "cuda:0" if torch.cuda.is_available() else "auto"
+model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path,device_map=device_map,torch_dtype=torch.float16,load_in_8bit=True,trust_remote_code=True,use_flash_attention_2=True)
+model = PeftModel.from_pretrained(model, finetune_model_path, device_map={"": 0})
+model =model.eval()
+input_ids = tokenizer(['<s>Human: 介绍一下北京\n</s><s>Assistant: '], return_tensors="pt",add_special_tokens=False).input_ids
+if torch.cuda.is_available():
+  input_ids = input_ids.to('cuda')
+generate_input = {
+    "input_ids":input_ids,
+    "max_new_tokens":512,
+    "do_sample":True,
+    "top_k":50,
+    "top_p":0.95,
+    "temperature":0.3,
+    "repetition_penalty":1.3,
+    "eos_token_id":tokenizer.eos_token_id,
+    "bos_token_id":tokenizer.bos_token_id,
+    "pad_token_id":tokenizer.pad_token_id
+}
+generate_ids  = model.generate(**generate_input)
+text = tokenizer.decode(generate_ids[0])
+print(text)
+```
+
+#### 全量参数微调
+对于全量参数微调的模型，调用方式同[模型调用代码示例](#模型调用代码示例)，只需要修改其中的模型名称或者保存路径即可。
+
+## 🍄 模型量化
+我们对中文微调的模型参数进行了量化，方便以更少的计算资源运行。目前已经在[Hugging Face](https://huggingface.co/FlagAlpha)上传了13B中文微调模型[FlagAlpha/Llama2-Chinese-13b-Chat](https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat)的4bit压缩版本[FlagAlpha/Llama2-Chinese-13b-Chat-4bit](https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat-4bit)，具体调用方式如下：
+
+环境准备：
+```
+pip install git+https://github.com/PanQiWei/AutoGPTQ.git
+```
+
+```python
+from transformers import AutoTokenizer
+from auto_gptq import AutoGPTQForCausalLM
+model = AutoGPTQForCausalLM.from_quantized('FlagAlpha/Llama2-Chinese-13b-Chat-4bit', device="cuda:0")
+tokenizer = AutoTokenizer.from_pretrained('FlagAlpha/Llama2-Chinese-13b-Chat-4bit',use_fast=False)
+input_ids = tokenizer(['<s>Human: 怎么登上火星\n</s><s>Assistant: '], return_tensors="pt",add_special_tokens=False).input_ids.to('cuda')        
+generate_input = {
+    "input_ids":input_ids,
+    "max_new_tokens":512,
+    "do_sample":True,
+    "top_k":50,
+    "top_p":0.95,
+    "temperature":0.3,
+    "repetition_penalty":1.3,
+    "eos_token_id":tokenizer.eos_token_id,
+    "bos_token_id":tokenizer.bos_token_id,
+    "pad_token_id":tokenizer.pad_token_id
+}
+generate_ids  = model.generate(**generate_input)
+text = tokenizer.decode(generate_ids[0])
+print(text)
+```
+
+## 🚀 部署加速
+随着大模型参数规模的不断增长，在有限的算力资源下，提升模型的推理速度逐渐变为一个重要的研究方向。常用的推理加速框架包含 lmdeploy、TensorRT-LLM、vLLM和JittorLLMs 等。
+
+### TensorRT-LLM
+[TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM/tree/main)由NVIDIA开发，高性能推理框架
+
+详细的推理文档见：[inference-speed/GPU/TensorRT-LLM_example](https://github.com/LlamaFamily/Llama-Chinese/tree/main/inference-speed/GPU/TensorRT-LLM_example)
+
+### vLLM
+[vLLM](https://github.com/vllm-project/vllm)由加州大学伯克利分校开发，核心技术是PageAttention，吞吐量比HuggingFace Transformers高出24倍。相较与FasterTrainsformer，vLLM更加的简单易用，不需要额外进行模型的转换，支持fp16推理。
+
+详细的推理文档见：[inference-speed/GPU/vllm_example](https://github.com/LlamaFamily/Llama-Chinese/blob/main/inference-speed/GPU/vllm_example/README.md)
+
+### JittorLLMs
+[JittorLLMs](https://github.com/Jittor/JittorLLMs)由非十科技领衔，与清华大学可视媒体研究中心合作研发，通过动态swap机制大幅降低硬件配置要求（减少80%）,并且Jittor框架通过零拷贝技术，大模型加载相比Pytorch开销降低40%，同时，通过元算子自动编译优化，计算性能提升20%以上。
+
+详细的推理文档见：[inference-speed/GPU/JittorLLMs](https://github.com/LlamaFamily/Llama-Chinese/blob/main/inference-speed/GPU/JittorLLMs_example/README.md)
+
+### lmdeploy
+[lmdeploy](https://github.com/InternLM/lmdeploy/) 由上海人工智能实验室开发，推理使用 C++/CUDA，对外提供 python/gRPC/http 接口和 WebUI 界���，支持 tensor parallel 分布式推理、支持 fp16/weight int4/kv cache int8 量化。
+
+详细的推理文档见：[inference-speed/GPU/lmdeploy_example](https://github.com/LlamaFamily/Llama-Chinese/tree/main/inference-speed/GPU/lmdeploy_example)
+
+## 💪 外延能力
+
+除了持续增强大模型内在的知识储备、通用理解、逻辑推理和想象能力等，未来，我们也会不断丰富大模型的外延能力，例如知识库检索、计算工具、WolframAlpha、操作软件等。
+我们首先集成了LangChain框架，可以更方便地基于Llama2开发文档检索、问答机器人和智能体应用等，关于LangChain的更多介绍参见[LangChain](https://github.com/langchain-ai/langchain)。
+
+### LangChain
+针对LangChain框架封装的Llama2 LLM类见[examples/llama2_for_langchain.py](https://github.com/LlamaFamily/Llama-Chinese/blob/main/examples/llama2_for_langchain.py)，简单的调用代码示例如下：
+```python
+from llama2_for_langchain import Llama2
+
+# 这里以调用FlagAlpha/Atom-7B-Chat为例
+llm = Llama2(model_name_or_path='FlagAlpha/Atom-7B-Chat')
+
+while True:
+    human_input = input("Human: ")
+    response = llm(human_input)
+    print(f"Llama2: {response}")
+```
+
+## 🥇 模型评测
+
+### Llama2和Llama3对比评测
+基础模型对比
+<p align="center" width="100%">
+<img src="./assets/base_eval.png" style="width: 100%; display: block; margin: auto;">
+</p>
+微调模型对比
+<p align="center" width="100%">
+<img src="./assets/tuned_eval.png" style="width: 100%; display: block; margin: auto;">
+</p>
+
+### Llama3模型评测
+<p align="center" width="100%">
+<img src="./assets/llama3_eval.png" style="width: 100%; display: block; margin: auto;">
+</p>
+
+### Llama2模型评测
+<p align="center" width="100%">
+<img src="./assets/llama_eval.jpeg" style="width: 100%; display: block; margin: auto;">
+</p>
+
+为了能够更加清晰地了解Llama2模型的中文问答能力，我们筛选了一些具有代表性的中文问题，对Llama2模型进行提问。我们测试的模型包含Meta公开的Llama2-7B-Chat和Llama2-13B-Chat两个版本，没有做任何微调和训练。测试问题筛选自[AtomBulb](https://github.com/AtomEcho/AtomBulb)，共95个测试问题，包含：通用知识、语言理解、创作能力、逻辑推理、代码编程、工作技能、使用工具、人格特征八个大的类别。
+
+测试中使用的Prompt如下，例如对于问题“列出5种可以改善睡眠质量的方法”：
+```
+[INST] 
+<<SYS>>
+You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. The answer always been translate into Chinese language.
+
+If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
+
+The answer always been translate into Chinese language.
+<</SYS>>
+
+列出5种可以改善睡眠质量的方法
+[/INST]
+```
+Llama2-7B-Chat的测试结果见[meta_eval_7B.md](assets/meta_eval_7B.md)，Llama2-13B-Chat的测试结果见[meta_eval_13B.md](assets/meta_eval_13B.md)。
+
+通过测试我们发现，Meta原始的Llama2 Chat模型对于中文问答的对齐效果一般，大部分情况下都不能给出中文回答，或者是中英文混杂的形式。因此，基于中文数据对Llama2模型进行训练和微调十分必要。
+
+
+## 📖 学习中心
+
+### 官方文档
+Meta Llama全系列模型官方文档：https://llama.meta.com/docs/get-started
+
+### Llama3
+[Llama3全套学习资料](https://chinesellama.feishu.cn/wiki/XBKPwbhWriWCfrkmJhfcrS9Rnqc?fromScene=spaceOverview)
+
+Llama3官方链接：https://llama.meta.com/llama3
+
+### Llama2
+
+#### Meta官方对于[Llama2](https://ai.meta.com/llama)的介绍
+自从Meta公司发布第一代LLaMA模型以来，羊驼模型家族繁荣发展。近期Meta发布了Llama2版本，开源可商用，在模型和效果上有了重大更新。Llama2总共公布了7B、13B和70B三种参数大小的模型。相比于LLaMA，Llama2的训练数据达到了2万亿token，上下文长度也由之前的2048升级到4096，可以理解和生成更长的文本。Llama2 Chat模型基于100万人类标记数据微调得到，在英文对话上达到了接近ChatGPT的效果。
+
+### Llama相关论文
+* [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971)
+* [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/abs/2307.09288)
+* [Code Llama: Open Foundation Models for Code](https://ai.meta.com/research/publications/code-llama-open-foundation-models-for-code/)
+
+
+## 📌 其它
+
+### 🎉 致谢
+
+感谢原子回声[AtomEcho](https://github.com/AtomEcho)团队的技术和资源支持！
+
+感谢芯格[Coremesh](https://coremesh.net)团队的技���和资源支持！
+
+感谢 [福州连天教育科技有限公司](www.3class.cc) 对Llama中文社区的贡献！
+
+感谢 @Z Potentials社区对Llama中文社区的支持！
+
+### 🤔 问题反馈
+
+如有问题，请在GitHub Issue中提交，在提交问题之前，请先查阅以往的issue是否能解决你的问题。
+
+礼貌地提出问题，构建和谐的讨论社区。
+
+加入[飞书知识库](https://chinesellama.feishu.cn/wiki/space/7257824476874768388?ccm_open_type=lark_wiki_spaceLink)，一起共建社区文档。
+
+加入微信群讨论😍😍
+
+<p align="center" width="100%">
+<img src="./assets/wechat.jpeg" alt="Wechat" style="width: 100%; display: block; margin: auto;">
+</p>
+
+<p align="center" width="100%">
+<img src="https://api.star-history.com/svg?repos=LlamaFamily/Llama-Chinese&type=Date" alt="Star" style="width: 100%; display: block; margin: auto;">
+</p>

Llama-Chinese/README_EN.md

@@ -0,0 +1,736 @@
+<p align="left">
+    English ｜ <a href="README.md">中文</a>
+</p>
+<br>
+
+<h1 align="center">
+  Llama-Chinese
+</h1>
+<p align="center" width="100%">
+  <img src="assets/llama.png" alt="Llama" style="width: 20%; display: block; margin: auto;"></a>
+</p>
+<p align="center">
+  <font face="黑体" color=orange size="6"> The Best Chinese Llama Large Language Model </font>
+</p>
+
+<p align="center">
+🤗 <a href="https://huggingface.co/FlagAlpha" target="_blank">Hugging Face</a> • 🤖 <a href="https://www.modelscope.cn/organization/FlagAlpha/" target="_blank">ModelScope</a> • ✡️ <a href="https://wisemodel.cn/models/FlagAlpha/Atom-7B-Chat" target="_blank">WiseModel</a>
+</p> 
+
+<p align="center">
+  <a href="https://llama.family">Online(Including Llama2, Llama3): llama.family</a>
+</p>
+<p align="center">
+  <a href="https://huggingface.co/FlagAlpha/Atom-7B-Chat">Open-source Chinese Pre-trained LLM Atom based on Llama2</a>
+</p>
+
+</br></br>
+
+
+## 🗂️ Content Guide
+- [📌 Chinese Llama Community](#-chinese-llama-community)
+  * [🔥 Community Introduction: Chinese Llama Community](#-community-introduction-chinese-llama-community)
+  * [📢 Community Announcements](#-community-announcements)
+  * [🤗 LLM Model](#-models-downloads)
+    + [🤗 Pre-trained Chinese Model Atom based on Llama2](#-atom-models)
+    + [🤗 Meta Official Llama2 Model](#meta-official-llama2-models)
+    + [🤗 Fine-tuned Chinese Models based on Llama2](#fine-tuned-chinese-models-based-on-llama2)
+  * [🌟 Community Source](#-community-source)
+    + [GPU Source](#-gpu-source)
+    + [Data Source](#-data-source)
+    + [Discussion](#-discussion)
+    + [Product](#-product)
+
+- [📌 How to use Llama Model?](#-how-to-use-llama-model)
+  * [Setup of Llama3](#setup-of-llama3)
+  * [Setup of Llama2](#setup-of-llama2)
+    + [Simple Setup](#simple-setup)
+      - [Simple Setup-Anaconda](#simple-setup-anaconda)
+      - [Simple Setup-Docker](#simple-setup-docker)
+      - [Simple Setup-llama.cpp](#simple-setup-llamacpp)
+      - [Simple Setup-gradio](#simple-setup-gradio)
+      - [Simple Setup-API](#simple-setup-api)
+    + [🤖 Model Pretraining](#-model-pretraining)
+    + [💡 Model Fine-tuning](#-model-fine-tuning)
+      - [Step1: Environment Setup](#step1-environment-setup)
+      - [Step2: Data Preparation](#step2-data-preparation)
+      - [Step3: Fine-tuning Scripts](#step3-fine-tuning-script)
+        - [LoRA Fine-tuning](#lora-fine-tuning)
+        - [Full-parameter Fine-tuning](#full-parameter-fine-tuning)
+      - [Step4: Load Fine-tuned Model](#step4-load-fine-tuned-model)
+        - [LoRA Fine-tuning](#lora-fine-tuning-1)
+        - [Full-parameter Fine-tuning](#full-parameter-fine-tuning-1)
+    + [🍄 Model Quantization](#-model-quantization)
+    + [🚀 Inference Acceleration](#-inference-acceleration)
+      - [TensorRT-LLM](#tensorrt-llm)
+      - [vLLM](#vllm)
+      - [JittorLLMs](#jittorllms)
+      - [lmdeploy](#lmdeploy)
+    + [💪 Extension Capabilities](#-extension-capabilities)
+      - [LangChain](#langchain)
+  * [🥇 Model Evaluation](#-model-evaluation)
+  * [📖 Learning Resources](#-learning-resources)
+    + [Llama3](#llama3)
+    + [Llama2](#llama2)
+      - [Meta Official Introduction to Llama2](#meta-official-introduction-to-llama2)
+      - [Llama-related Papers](#llama-related-papers)
+      - [Llama2 Evaluation Results](#llama2-evaluation-results)
+
+- [📌 Others](#-others)
+  * [🎉 Acknowledgments](#-acknowledgments)
+  * [🤔 Issue Feedback](#-issue-feedback)
+
+## 📌 Chinese Llama Community
+
+### 🔥 Community Introduction: Chinese Llama Community
+
+Welcome to the Chinese Llama Community! We are a technical community dedicated to optimizing and building on top of the Llama model for Chinese applications.
+**\*Based on large-scale Chinese data, we start pre-training and continuously upgrade the Llama2 model for Chinese capabilities\***.
+We warmly welcome developers and researchers passionate about LLM models to join our community.
+
+<details lang="en">
+
+#### Why Choose the Chinese Llama Community?
+🚀 **Support from a Team of Senior Engineers**: The community has a team of dedicated NLP senior engineers who provide strong technical support and rich experience to guide and assist you.
+
+🎯 **Chinese Optimization**: We focus on optimizing Llama2 for Chinese processing, exploring the best practices for Chinese to enhance its performance and adaptability.
+
+💡 **Innovative Exchange**: Our community includes a creative and experienced team of members who organize regular online events, technical discussions, and experience sharing to promote innovative exchanges.
+
+🌐 **Global Connectivity**: We welcome developers from around the world to join the community, creating an open and diverse platform for learning and communication.
+
+🤝 **Open Sharing**: We encourage community members to open-source and share code and models, promoting collaborative win-win efforts and advancing the development of Chinese NLP technology.
+
+#### Community Activities
+🗓️ **Online Lectures**: Inviting industry experts to conduct online lectures, sharing the latest technology and applications of Llama2 in the Chinese NLP field, and discussing cutting-edge research results.
+
+💻 **Project Showcase**: Members can showcase their project achievements in Llama2 Chinese optimization, receive feedback and suggestions, and promote project collaboration.
+
+📚 **Learning Resources**: The community maintains a rich library of learning materials, including tutorials, documentation, and paper interpretations, providing comprehensive learning support to members.
+
+📝 **Paper Interpretation**: Community members collectively interpret the latest research papers related to Llama2, delving into advanced algorithms and methods.
+
+🎉 **Themed Events**: Regularly organize various themed events, including challenges, hackathons, and technical salons, allowing community members to exchange and learn in a relaxed and enjoyable atmosphere.
+
+🌟 **Reward Program**: We have established a reward program to honor and reward members who actively participate and contribute outstanding work to the community, motivating more outstanding talents to join.
+
+📈 **Technical Consultation**: We provide technical consulting services to answer your questions and help you overcome challenges in the development and optimization of Llama2.
+
+🚀 **Project Collaboration**: Encourage collaboration between members on projects to explore the potential of Llama2 in practical applications and create innovative solutions.
+
+#### Join Us Now!
+📚 **Vision**: Whether you are a professional developer or researcher with experience in Llama2 or a newcomer interested in optimizing Llama2 for Chinese, we eagerly look forward to your joining. In the Chinese Llama Community, you will have the opportunity to exchange ideas with top talents in the industry, work together to advance Chinese NLP technology, and create a brighter technological future!
+
+🔗 **Friendly Reminder**: This community is a platform for professional technical exchange. We earnestly hope that like-minded developers and researchers join us. Please adhere to the community guidelines, maintain a positive learning atmosphere, and any content and advertisements unrelated to Llama2 will be removed. Thank you for your understanding and support!
+
+</details>
+
+### 📢 Community Announcements
+
+【Latest】October 8, 2023: Added the inference acceleration feature for JittorLLMs from Tsinghua University [JittorLLMs](#jittorllms)!
+
+【Latest】September 12, 2023: Updated pre-training versions [Atom-7B](https://huggingface.co/FlagAlpha/Atom-7B) and dialogue version [Atom-7B-Chat](https://huggingface.co/FlagAlpha/Atom-7B-Chat) model parameters. The latest Chinese pre-training data size is 100 billion tokens, and the training progress can be viewed at [llama.family](https://llama.family/)!
+
+【Latest】September 2, 2023: Added [pre-training code](#-model-pretraining) and [full-parameter fine-tuning code](#-model-fine-tuning)!
+
+【Latest】August 28, 2023: Released the open-source large model [Atom-7B](https://huggingface.co/FlagAlpha/Atom-7B) based on Llama2 for Chinese pre-training and will continue to be updated. Details can be found in the [community article](https://mp.weixin.qq.com/s/Bdx0JTVh1kgPn5ydYxIkEw)!
+
+【Latest】August 26, 2023: Provided [FastAPI](#fastapi-interface-setup) interface setup script!
+
+【Latest】August 26, 2023: Provided a script to convert Meta official model parameters to a format compatible with Hugging Face [Format Conversion Script](https://github.com/LlamaFamily/Llama-Chinese/blob/main/scripts/convert2hf/README.md)!
+
+【Latest】August 26, 2023: Added [Code Llama](#-code-model) model!
+
+<details lang="en">
+
+- August 15, 2023: Added [PEFT load fine-tuning model parameters](#load-fine-tuned-model) code example!
+
+- August 14, 2023: Launched the [large model data sharing training platform](https://llama.family), allowing everyone to contribute to large model training, even without computing resources. The data contributed by each community member will determine the future capabilities of the model!
+
+- August 3, 2023: Added GPU [inference acceleration](#-inference-acceleration) support for FasterTransformer and vLLM!
+
+- July 31, 2023: 【Major】The first truly meaningful Llama2 Chinese large model is released! Details can be found in the [community article](https://mp.weixin.qq.com/s/lExUU7z_MvgJ7tzQPF8tUQ)
+
+- July 28, 2023: Deployed a Q&A interface through [Docker](#docker-deployment-of-qa-interface)!
+
+- July 27, 2023: Added [LangChain](#langchain) support!
+
+- July 26, 2023: Released a [4-bit quantized compressed version](#-model-quantization) of the Llama2-13B Chinese fine-tuning parameters!
+
+- July 25, 2023: The community's WeChat public account "Llama Chinese Community" is now live. Feel free to follow for the latest updates and dynamics!
+
+- July 24, 2023: [FlagAlpha](https://huggingface.co/FlagAlpha) added Llama2-13B Chinese fine-tuned parameters!
+
+- July 24, 2023: [llama.family](https://llama.family/) added Llama2-70B online experience!
+
+- July 23, 2023: Released Llama2-13B Chinese fine-tuned parameters to the Hugging Face repository [FlagAlpha](https://huggingface.co/FlagAlpha)!
+
+- July 22, 2023: Llama2 online experience link [llama.family](https://llama.family/) is live, including both Meta original and Chinese fine-tuned versions!
+
+- July 21, 2023: Evaluated the Chinese Q&A capability of the Meta original Llama2 Chat model [Model Evaluation](#-model-evaluation)!
+
+- July 21, 2023: Added the Hugging Face version download link for Llama2 models in China!
+
+- July 20, 2023: Added [Feishu Knowledge Base Documentation](https://chinesellama.feishu.cn/wiki/space/7257824476874768388?ccm_open_type=lark_wiki_spaceLink), welcome everyone to contribute!
+
+- July 20, 2023: Chinese Llama2 latest download links are live!
+
+- July 19, 2023: Officially launched the Llama2 Chinese community, stay tuned for real-time updates!
+
+- July 19, 2023: Chinese Llama2 latest download links are in progress, stay tuned!
+
+- July 19, 2023: Launched the Llama2 Chinese community, welcome everyone to join!
+
+</details>
+
+### 🤗 Models Downloads
+
+#### 🔵 Atom Models
+
+The Atom models, created jointly by the Chinese Llama Community and AtomEcho, rank in the top ten of the Chinese Large Language Model Evaluation List C-Eval (submission on August 21).
+<p align="center" width="100%">
+<img src="./assets/ceval.jpg" alt="ceval" style="width: 100%; display: block; margin: auto;">
+</p>
+
+
+|  Category  | Model Name        | 🤗Model Loading Name                  | Download Link                                                 |
+| --------------- | --------------- | ------------------------------ | ------------------------------------------------------------ |
+|  Pretrained  | Atom-7B  | FlagAlpha/Atom-7B  | [HuggingFace](https://huggingface.co/FlagAlpha/Atom-7B) \| [ModelScope](https://modelscope.cn/models/FlagAlpha/Atom-7B) \| [WiseModel](https://wisemodel.cn/models/FlagAlpha/Atom-7B) |
+|  Chat  | Atom-7B-Chat  | FlagAlpha/Atom-7B-Chat  | [HuggingFace](https://huggingface.co/FlagAlpha/Atom-7B-Chat) \| [ModelScope](https://modelscope.cn/models/FlagAlpha/Atom-7B-Chat) \| [WiseModel](https://wisemodel.cn/models/FlagAlpha/Atom-7B-Chat) |
+
+
+The Atom series includes Atom-1B, Atom-7B and Atom-13B, with continuous optimization of Chinese language proficiency based on Llama2. Atom-7B and Atom-7B-Chat are fully open source and available for commercial use. You can obtain the models on the [Hugging Face](https://huggingface.co/FlagAlpha) repository. Details are available in [Atom-7B Download](#atom-chinese-pretrained-model-based-on-llama2).
+
+Atom models have the following optimizations for Chinese:
+
+###### Large-scale Chinese Data Pretraining
+
+Atom models are continually pretrained using a large amount of Chinese data, including encyclopedias, books, blogs, news, announcements, novels, financial data, legal data, medical data, code data, professional paper data, and Chinese natural language processing competition datasets. See [📝 Data Sources](#-data-sources) for details.
+
+The massive data is filtered, scored, and deduplicated, resulting in high-quality Chinese data exceeding 1T tokens, continuously added to the training iterations.
+
+###### More Efficient Chinese Vocabulary
+
+To improve the efficiency of Chinese text processing, we optimized the vocabulary of the Llama2 model. First, based on several hundred gigabytes of Chinese text, we expanded the word library to 65,000 words on the basis of the model's vocabulary. Our improvements increased the Chinese encoding/decoding speed by about 350% according to tests. Additionally, we expanded the coverage of the Chinese character set, including all emoji symbols 😊. This makes generating articles with emoji symbols more efficient.
+
+###### Adaptive Context Expansion
+
+Atom large models support a default context of 4K. Through position interpolation (PI) and Neural Tangent Kernel (NTK) methods, the context length can be expanded to 32K after fine-tuning.
+
+###### 📝 Chinese Data
+
+We optimized the Chinese capabilities of Llama2 using the following data:
+
+| Type                                                       | Description                                                   |
+| ---------------------------------------------------------- | ------------------------------------------------------------ |
+| Web Data                                                   | Publicly available web data on the Internet, selecting deduplicated high-quality Chinese data involving encyclopedias, books, blogs, news, announcements, novels, etc. |
+| [Wikipedia](https://github.com/goldsmith/Wikipedia)        | Chinese Wikipedia data                                        |
+| [Wudao](https://github.com/BAAI-WuDao/Model)               | 200G of Chinese Wudao open-source data                         |
+| [Clue](https://github.com/CLUEbenchmark/CLUEDatasetSearch) | High-quality Chinese long-text data cleaned from Clue's open Chinese pretraining data |
+| Competition Datasets                                       | About 150 Chinese natural language processing multi-task competition datasets in recent years |
+| [MNBVC](https://github.com/esbatmop/MNBVC)                 | Some datasets cleaned from MNBVC                              |
+
+**If you have high-quality datasets, we would greatly appreciate it if you could provide them to us! 💕💕**
+
+
+#### Meta Official Llama2 Models
+
+<details lang="en">
+
+The Llama2 pretrained models include 7B, 13B, and 70B versions. The Llama2-Chat model is fine-tuned based on the pretrained models and has enhanced conversational capabilities.
+
+|  Category  | Model Name   | 🤗Model Loading Name             | Download Link                                                 |
+|  ----------  | ---------- | ------------------------- | --------------------- |
+|  Pretrained  | Llama2-7B  | meta-llama/Llama-2-7b-hf  | [HuggingFace](https://huggingface.co/meta-llama/Llama-2-7b-hf) \| [XunLei](https://pan.xunlei.com/s/VN_t0dUikZqOwt-5DZWHuMvqA1?pwd=66ep) |
+|  Pretrained  | Llama2-13B | meta-llama/Llama-2-13b-hf | [HuggingFace](https://huggingface.co/meta-llama/Llama-2-13b-hf) \| [XunLei](https://pan.xunlei.com/s/VN_yT_9G8xNOz0SDWQ7Mb_GZA1?pwd=yvgf) |
+|  Pretrained  | Llama2-70B | meta-llama/Llama-2-70b-hf | [HuggingFace](https://huggingface.co/meta-llama/Llama-2-70b-hf) |
+|  Chat  | Llama2-7B-Chat  | meta-llama/Llama-2-7b-chat-hf  | [HuggingFace](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) \| [XunLei](https://pan.xunlei.com/s/VN_oaV4BpKFgKLto4KgOhBcaA1?pwd=ufir) |
+|  Chat  | Llama2-13B-Chat | meta-llama/Llama-2-13b-chat-hf | [HuggingFace](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf) \| [XunLei](https://pan.xunlei.com/s/VN_yA-9G34NGL9B79b3OQZZGA1?pwd=xqrg) |
+|  Chat  | Llama2-70B-Chat | meta-llama/Llama-2-70b-chat-hf | [HuggingFace](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) \| [XunLei](https://pan.xunlei.com/s/VNa_vCGzCy3h3N7oeFXs2W1hA1?pwd=uhxh#) |
+| Code  | CodeLlama-7b    |   meta-llama/Llama-2-70b-chat-hf              | [XunLei](https://pan.baidu.com/s/1cIPzdNywWLvQI7_2QanOEQ?pwd=zfwi) |
+| Code  | CodeLlama-7b-Python    |   meta-llama/Llama-2-70b-chat-hf              | [XunLei](https://pan.baidu.com/s/1liY8klGoDagYbpw-g-oFag?pwd=i952) |
+| Code  | CodeLlama-7b-Instruct    |   meta-llama/Llama-2-70b-chat-hf              | [XunLei](https://pan.baidu.com/s/108o9_DT2E_vfSGtOnDCQVw?pwd=zkt9) |
+| Code  | CodeLlama-13b    |   meta-llama/Llama-2-70b-chat-hf              | [XunLei](https://pan.baidu.com/s/1lLaeHv0XEBv0iiZzI1dpnw?pwd=qn99) |
+| Code  | CodeLlama-13b-Python    |   meta-llama/Llama-2-70b-chat-hf              | [XunLei](https://pan.baidu.com/s/1OLVfvZS_oqL3oqMKwsI87w?pwd=a78k) |
+| Code  | CodeLlama-13b-Instruct    |   meta-llama/Llama-2-70b-chat-hf              | [XunLei](https://pan.baidu.com/s/1HyxJl4w8wElgkZRh2ATrXQ?pwd=seg6) |
+| Code  | CodeLlama-34b    |   meta-llama/Llama-2-70b-chat-hf              | [XunLei](https://pan.baidu.com/s/1vEw0pFgIkctPUN4_5_6pIQ?pwd=q8eu) |
+
+Meta officially released Code Llama on August 24, 2023, which is a fine-tuned version of Llama2 based on code data. It provides three versions with different functionalities: Base Model (Code Llama), Python-specific Model (Code Llama - Python), and Instruction-following Model (Code Llama - Instruct), each available in 7B, 13B, and 34B parameter sizes. The capabilities of different models are summarized in the following table:
+
+|  Model Category         |        Model Name         | Code Completion | Code Fill | Instruction Programming |
+|-----------------------|------------------------|------|------|------|
+| Code Llama            | CodeLlama-7b           | ✅    | ✅    | ❌    |
+|                       | CodeLlama-13b          | ✅    | ✅    | ❌    |
+|                       | CodeLlama-34b          | ✅    | ❌    | ❌    |
+| Code Llama - Python   | CodeLlama-7b-Python    | ✅    | ❌    | ❌    |
+|                       | CodeLlama-13b-Python   | ✅    | ❌    | ❌    |
+|                       | CodeLlama-34b-Python   | ✅    | ❌    | ❌    |
+| Code Llama - Instruct | CodeLlama-7b-Instruct  | ❌    | ✅    | ✅    |
+|                       | CodeLlama-13b-Instruct | ❌    | ✅    | ✅    |
+|                       | CodeLlama-34b-Instruct | ❌    | ❌    | ✅    |
+
+We provide a [domestic download link for Code Llama](#-latest-downloads-of-llama2-in-china) and an online experience link at [llama.family](https://llama.family/). For detailed information on Code Llama, refer to the official GitHub repository [codellama](https://github.com/facebookresearch/codellama).
+
+</details>
+
+#### Fine-tuned Chinese Models Based on Llama2
+
+We fine-tuned the Llama2-Chat model based on a Chinese instruction dataset, enhancing its Chinese conversational abilities. LoRA parameters and merged parameters with the base model have been uploaded to [Hugging Face](https://huggingface.co/FlagAlpha) and currently include models for 7B and 13B.
+
+|  Category  | Model Name   | 🤗Model Loading Name             | Base Model Version |    Download Link                                                 |
+|  ----------  | ---------- | ------------- |  ----------------- | ------------------- |
+|  Merged Parameters | Llama2-Chinese-7b-Chat | FlagAlpha/Llama2-Chinese-7b-Chat  |    meta-llama/Llama-2-7b-chat-hf       |[HuggingFace](https://huggingface.co/FlagAlpha/Llama2-Chinese-7b-Chat)  |
+|  Merged Parameters | Llama2-Chinese-13b-Chat | FlagAlpha/Llama2-Chinese-13b-Chat|     meta-llama/Llama-2-13b-chat-hf     |[HuggingFace](https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat) |
+|  LoRA Parameters | Llama2-Chinese-7b-Chat-LoRA  | FlagAlpha/Llama2-Chinese-7b-Chat-LoRA  |     meta-llama/Llama-2-7b-chat-hf      |[HuggingFace](https://huggingface.co/FlagAlpha/Llama2-Chinese-7b-Chat-LoRA) |
+|  LoRA Parameters | Llama2-Chinese-13b-Chat-LoRA | FlagAlpha/Llama2-Chinese-13b-Chat-LoRA |     meta-llama/Llama-2-13b-chat-hf     |[HuggingFace](https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat-LoRA) |
+
+
+### 🌟 Community Source
+The abundance of community resources is a crucial guarantee for community development, covering various aspects including but not limited to the following four: computing power, data, forums, and applications. The positive development and full utilization of these aspects will provide community members with more opportunities and support, driving the entire community towards a more prosperous direction. More details in [llama.family](https://llama.family/).
+
+<details lang="en">
+#### 💻 GPU Source
+
+- Provide computing power resources at below-market prices, usable for various computing tasks such as training and inference of deep learning models.
+- Offer exclusive online inference services for community members, enabling users to quickly and effectively perform inference operations on models.
+- Provide one-click online fine-tuning services, allowing users to conveniently fine-tune models to adapt to different tasks and data.
+
+#### 📊 Data Source
+- Open up abundant training data resources covering multiple domains and industries, providing ample data support for model training.
+- Provide high-quality, diverse datasets to meet the needs of different users, supporting data sharing and exchange to facilitate the full utilization of data resources.
+
+#### 💬 Discussion
+- The community forum provides a platform for community members to engage in online discussions and exchange technical issues.
+- On the forum, users can share experiences, ask questions, and provide answers, fostering technical exchange and collaboration.
+- The forum can also host regular online events, seminars, etc., to enhance connections and understanding among community members.
+
+#### 📱 Product
+- Offer free promotion spaces for showcasing applications, allowing developers to fully present their apps to community members.
+- Provide promotional assistance, including but not limited to publicity campaigns, user guidance, etc., to help apps gain more exposure and users.
+- Through the community platform, provide collaboration opportunities for outstanding apps, promoting cooperation and communication among app developers, collectively driving the development and growth of applications.
+
+</details>
+
+## 📌 How to use Llama Model?
+
+### Setup of Llama3
+
+### Setup of Llama2
+
+### Simple Setup
+You can choose a learning path to start using the Llama series models. It is recommended to use the Chinese pre-trained dialogue model for better support of Chinese language effects.
+
+#### Simple Setup: Anaconda
+
+##### Step 0: Prerequisites
+- Make sure that Python version 3.10 or higher is installed.
+
+##### Step 1: Prepare the environment
+If you need to set up the environment and install required packages, run the following command.
+```bash
+git clone https://github.com/LlamaFamily/Llama-Chinese.git
+cd Llama-Chinese
+pip install -r requirements.txt
+```
+
+##### Step 2: Download Model
+You can download the Atom-7B-Chat model from the following source.
+- [HuggingFace](https://huggingface.co/FlagAlpha)
+- [ModelScope](https://modelscope.cn/organization/FlagAlpha)
+- [WideModel](https://wisemodel.cn/models/FlagAlpha/Atom-7B-Chat)
+
+###### Step 3：Infer
+1. Create a file named quick_start.py and copy the following content into the file.
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+device_map = "cuda:0" if torch.cuda.is_available() else "auto"
+model = AutoModelForCausalLM.from_pretrained('FlagAlpha/Atom-7B-Chat',device_map=device_map,torch_dtype=torch.float16,load_in_8bit=True,trust_remote_code=True,use_flash_attention_2=True)
+model =model.eval()
+tokenizer = AutoTokenizer.from_pretrained('FlagAlpha/Atom-7B-Chat',use_fast=False)
+tokenizer.pad_token = tokenizer.eos_token
+input_ids = tokenizer(['<s>Human: 介绍一下中国\n</s><s>Assistant: '], return_tensors="pt",add_special_tokens=False).input_ids
+if torch.cuda.is_available():
+  input_ids = input_ids.to('cuda')
+generate_input = {
+    "input_ids":input_ids,
+    "max_new_tokens":512,
+    "do_sample":True,
+    "top_k":50,
+    "top_p":0.95,
+    "temperature":0.3,
+    "repetition_penalty":1.3,
+    "eos_token_id":tokenizer.eos_token_id,
+    "bos_token_id":tokenizer.bos_token_id,
+    "pad_token_id":tokenizer.pad_token_id
+}
+generate_ids  = model.generate(**generate_input)
+text = tokenizer.decode(generate_ids[0])
+print(text)
+```
+2. run quick_start.py 
+```bash
+python quick_start.py
+```
+
+
+#### Simple Setup: Docker
+For details, refer to: [Docker Deployment](https://github.com/LlamaFamily/Llama-Chinese/blob/main/docs/chat_gradio_guide.md)
+
+Step 1: Prepare the Docker image and launch [chat_gradio.py](../examples/chat_gradio.py) through a Docker container.
+```bash
+git clone https://github.com/LlamaFamily/Llama-Chinese.git
+
+cd Llama-Chinese
+
+docker build -f docker/Dockerfile -t flagalpha/llama2-chinese:gradio .
+```
+
+Step 2: Start chat_gradio through Docker-compose.
+```bash
+cd Llama-Chinese/docker
+docker-compose up -d --build
+```
+
+#### Simple Setup: llama.cpp
+Details: [llama.cpp](https://github.com/LlamaFamily/Llama-Chinese/blob/main/inference-speed/CPU/ggml/README.md)
+
+
+##### Simple Setup: gradio
+Built on Gradio, the Q&A interface implements fluid output. Copy the following code into the console to run. The code below uses the Atom-7B model as an example, <font color="#006600">simply modify the model name in the code for different models 😊</font><br/>
+
+```
+python examples/chat_gradio.py --model_name_or_path FlagAlpha/Atom-7B-Chat
+```
+
+##### Simple Setup: API
+Use FastChat to build an inference service interface consistent with OpenAI.
+
+<details lang="en">
+
+###### step 0：Prerequisites
+Install fastchat
+```bash
+pip3 install "fschat[model_worker,webui]"
+```
+###### step 1：Run Restful API
+Open three terminals and execute the following three commands respectively.
+
+- Fist start controler
+```bash
+python3 -m fastchat.serve.controller \
+--host localhost \
+--port 21001
+```
+
+- Start Model
+```bash
+CUDA_VISIBLE_DEVICES="0" python3 -m fastchat.serve.model_worker --model-path /path/Atom-7B-Chat \
+--host localhost \
+--port 21002 \
+--worker-address "http://localhost:21002" \
+--limit-worker-concurrency 5 \
+--stream-interval 2 \
+--gpus "1" \
+--load-8bit
+```
+- Start RESTful API Service
+```bash
+python3 -m fastchat.serve.openai_api_server \
+--host localhost \
+--port 21003 \
+--controller-address http://localhost:21001
+```
+
+###### step 2：test api service
+Execute the Python code below to test the API service deployed above.
+```python
+# coding=utf-8
+import json
+import time
+import urllib.request
+import sys
+import requests
+
+def test_api_server(input_text):
+    header = {'Content-Type': 'application/json'}
+
+    data = {
+          "messages": [{"role": "system", "content": ""}, {"role": "user", "content": input_text}],
+          "temperature": 0.3, 
+          "top_p" : 0.95, 
+          "max_tokens": 512, 
+          "model": "LLama2-Chinese-13B",
+          "stream" : False,
+          "n" : 1,
+          "best_of": 1, 
+          "presence_penalty": 1.2, 
+          "frequency_penalty": 0.2,           
+          "top_k": 50, 
+          "use_beam_search": False, 
+          "stop": [], 
+          "ignore_eos" :False,
+          "logprobs": None
+    }
+    response = requests.post(
+        url='http://127.0.0.1:21003/v1/chat/completions',
+        headers=header,
+        data=json.dumps(data).encode('utf-8')
+    )
+
+    result = None
+    try:
+        result = json.loads(response.content)
+        print(json.dumps(data, ensure_ascii=False, indent=2))
+        print(json.dumps(result, ensure_ascii=False, indent=2))
+
+    except Exception as e:
+        print(e)
+
+    return result
+
+if __name__ == "__main__":
+    test_api_server("如何去北京?")
+```
+</details>
+
+
+#### 🤖 Model Pretraining
+While the pretraining data for Llama2 has doubled compared to the first generation LLaMA, the proportion of Chinese pretraining data is still very low, accounting for only 0.13%. This results in a relatively weak Chinese capability for the original Llama2. To enhance the model's Chinese capability, two approaches can be adopted: fine-tuning and pretraining.
+
+- Fine-tuning requires fewer computational resources and can quickly create a prototype of a Chinese Llama. However, its drawback is evident – it can only leverage the existing Chinese capabilities of the base model. Due to the limited amount of Chinese training data for Llama2, the potential improvement is also restricted, addressing the symptoms rather than the root cause.
+
+- Pretraining based on large-scale Chinese corpora involves high costs, requiring not only large-scale high-quality Chinese data but also substantial computational resources. However, the advantage is clear – it optimizes the Chinese capability from the model's foundational layers, achieving a fundamental improvement, injecting robust Chinese capabilities into the core of the large model.
+
+We provide the pretraining code for the Llama model to the community, along with [Chinese test data](https://github.com/LlamaFamily/Llama-Chinese/tree/main/data). More data can be found in [Chinese Data](#-chinese-data). The specific code and configurations are as follows:
+
+- Model pretraining script: [train/pretrain/pretrain.sh](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/pretrain/pretrain.sh)
+- Pretraining implementation code: [train/pretrain/pretrain_clm.py](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/pretrain/pretrain_clm.py)
+- [DeepSpeed](https://github.com/microsoft/DeepSpeed) acceleration:
+  - For single-card training, ZeRO-2 can be used. See parameters in [train/pretrain/ds_config_zero2.json](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/pretrain/ds_config_zero2.json).
+  - For multi-card training, ZeRO-3 can be used. See parameters in [train/pretrain/ds_config_zero3.json](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/pretrain/ds_config_zero3.json).
+- Training effectiveness metrics: [train/pretrain/accuracy.py](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/pretrain/accuracy.py)
+
+#### 💡 Model Fine-Tuning
+
+This repository provides both LoRA fine-tuning and full-parameter fine-tuning code. Detailed information about LoRA can be found in the paper "[LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)" and the Microsoft GitHub repository [LoRA](https://github.com/microsoft/LoRA).
+
+##### Step1: Environment Setup
+
+Install the necessary environment dependencies according to [requirements.txt](https://github.com/LlamaFamily/Llama-Chinese/blob/main/requirements.txt).
+
+##### Step2: Data Preparation
+
+In the data directory, there is a sample data for the model's SFT:
+- Training data: [data/train_sft.csv](https://github.com/LlamaFamily/Llama-Chinese/blob/main/data/train_sft.csv)
+- Validation data: [data/dev_sft.csv](https://github.com/LlamaFamily/Llama-Chinese/blob/main/data/dev_sft.csv)
+
+Each CSV file contains a "text" column, with each row representing a training example. Organize questions and answers in the model's input format, as shown below:
+```
+"<s>Human: "+question+"\n</s><s>Assistant: "+answer
+```
+For example,
+```
+<s>Human: Describe why the Earth is unique in one sentence.</s><s>Assistant: Because the Earth is currently the only known planet with existing life.</s>
+```
+
+##### Step3: Fine-tuning Scripts
+
+###### LoRA Fine-tuning
+LoRA fine-tuning script: [train/sft/finetune_lora.sh](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/sft/finetune_lora.sh). For details on LoRA fine-tuning implementation, refer to [train/sft/finetune_clm_lora.py](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/sft/finetune_clm_lora.py). Fine-tuning on a single machine with multiple cards can be achieved by modifying the "--include localhost:0" in the script.
+
+###### Full-parameter Fine-tuning
+Full-parameter fine-tuning script: [train/sft/finetune.sh](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/sft/finetune.sh). For details on full-parameter fine-tuning implementation, refer to [train/sft/finetune_clm.py](https://github.com/LlamaFamily/Llama-Chinese/blob/main/train/sft/finetune_clm.py).
+
+###### Step4: Load Fine-tuned Model
+
+###### LoRA Fine-tuning
+For LoRA fine-tuned model parameters, see [Chinese Fine-Tuned Model based on Llama2](#chinese-fine-tuned-model-based-on-llama2). LoRA parameters need to be combined with base model parameters.
+
+Use [PEFT](https://github.com/huggingface/peft) to load both pretraining and fine-tuned model parameters. In the example code below, set "base_model_name_or_path" to the pretraining model's save path and "finetune_model_path" to the fine-tuned model's save path.
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from peft import PeftModel, PeftConfig
+
+finetune_model_path = ''  # For example: 'FlagAlpha/Llama2-Chinese-7b-Chat-LoRA'
+config = PeftConfig.from_pretrained(finetune_model_path)
+tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path, use_fast=False)
+tokenizer.pad_token = tokenizer.eos_token
+device_map = "cuda:0" if torch.cuda.is_available() else "auto"
+model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, device_map=device_map, torch_dtype=torch.float16, load_in_8bit=True)
+model = PeftModel.from_pretrained(model, finetune_model_path, device_map={"": 0})
+model = model.eval()
+input_ids = tokenizer(['<s>Human: Introduce Beijing\n</s><s>Assistant: '], return_tensors="pt", add_special_tokens=False).input_ids
+if torch.cuda.is_available():
+  input_ids = input_ids.to('cuda')
+generate_input = {
+    "input_ids": input_ids,
+    "max_new_tokens": 512,
+    "do_sample": True,
+    "top_k": 50,
+    "top_p": 0.95,
+    "temperature": 0.3,
+    "repetition_penalty": 1.3,
+    "eos_token_id": tokenizer.eos_token_id,
+    "bos_token_id": tokenizer.bos_token_id,
+    "pad_token_id": tokenizer.pad_token_id
+}
+generate_ids = model.generate(**generate_input)
+text = tokenizer.decode(generate_ids[0])
+print(text)
+```
+###### Full-parameter Fine-tuning
+For full-parameter fine-tuned models, use the same calling method as in Model Calling Code Example, just modify the model name or save path accordingly.
+
+
+
+#### 🍄 Model Quantization
+We have quantized the parameters of the Chinese fine-tuned model to facilitate running with fewer computational resources. Currently, we have uploaded a 4-bit compressed version of the 13B Chinese fine-tuned model [FlagAlpha/Llama2-Chinese-13b-Chat](https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat) as [FlagAlpha/Llama2-Chinese-13b-Chat-4bit](https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat-4bit) on [Hugging Face](https://huggingface.co/FlagAlpha). The specific calling method is as follows:
+
+Environmental requirements:
+```
+pip install git+https://github.com/PanQiWei/AutoGPTQ.git
+```
+
+```python
+from transformers import AutoTokenizer
+from auto_gptq import AutoGPTQForCausalLM
+
+model = AutoGPTQForCausalLM.from_quantized('FlagAlpha/Llama2-Chinese-13b-Chat-4bit', device="cuda:0")
+tokenizer = AutoTokenizer.from_pretrained('FlagAlpha/Llama2-Chinese-13b-Chat-4bit', use_fast=False)
+input_ids = tokenizer(['<s>Human: How to land on Mars\n</s><s>Assistant: '], return_tensors="pt", add_special_tokens=False).input_ids.to('cuda')        
+generate_input = {
+    "input_ids": input_ids,
+    "max_new_tokens": 512,
+    "do_sample": True,
+    "top_k": 50,
+    "top_p": 0.95,
+    "temperature": 0.3,
+    "repetition_penalty": 1.3,
+    "eos_token_id": tokenizer.eos_token_id,
+    "bos_token_id": tokenizer.bos_token_id,
+    "pad_token_id": tokenizer.pad_token_id
+}
+generate_ids = model.generate(**generate_input)
+text = tokenizer.decode(generate_ids[0])
+print(text)
+```
+
+#### 🚀 Inference Acceleration
+As the parameter scale of large models continues to grow, improving model inference speed has become an important research direction with limited computational resources. Common inference acceleration frameworks include lmdeploy, FasterTransformer, vLLM, and JittorLLMs.
+
+##### TensorRT-LLM
+[TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM/tree/main) is developed by NVIDIA, written in C++/CUDA, and supports distributed inference. 
+
+For detailed inference documentation, visit: [inference-speed/GPU/TensorRT-LLM_example](https://github.com/LlamaFamily/Llama-Chinese/tree/main/inference-speed/GPU/TensorRT-LLM_example)
+
+##### vLLM
+[vLLM](https://github.com/vllm-project/vllm) is developed by the University of California, Berkeley, with its core technology being PageAttention. It achieves 24 times higher throughput compared to HuggingFace Transformers. Unlike FasterTransformer, vLLM is more user-friendly and does not require additional model conversion. It supports FP16 inference.
+
+For detailed inference documentation, visit: [inference-speed/GPU/vllm_example](https://github.com/LlamaFamily/Llama-Chinese/blob/main/inference-speed/GPU/vllm_example/README.md)
+
+##### JittorLLMs
+[JittorLLMs](https://github.com/Jittor/JittorLLMs) is led by Non-ten Technology in collaboration with the Visual Media Research Center at Tsinghua University. It significantly reduces hardware requirements by 80% through a dynamic swap mechanism. Jittor framework, with zero-copy technology, reduces the loading overhead of large models by 40% compared to PyTorch. Moreover, automatic compilation optimization through meta-operators enhances computational performance by over 20%.
+
+For detailed inference documentation, visit: [inference-speed/GPU/JittorLLMs](https://github.com/LlamaFamily/Llama-Chinese/blob/main/inference-speed/GPU/JittorLLMs_example/README.md)
+
+##### lmdeploy
+[lmdeploy](https://github.com/InternLM/lmdeploy/) is developed by the Shanghai AI Lab, using C++/CUDA for inference. It provides Python/gRPC/HTTP interfaces and a WebUI for inference, supporting tensor parallel distributed inference and FP16/weight int4/kv cache int8 quantization.
+
+For detailed inference documentation, visit: [inference-speed/GPU/lmdeploy_example](https://github.com/LlamaFamily/Llama-Chinese/tree/main/inference-speed/GPU/lmdeploy_example)
+
+#### 💪 Extension Capabilities
+
+In addition to continually enhancing the intrinsic qualities of large models, such as knowledge base, general understanding, logical reasoning, and imaginative capabilities, we are also actively expanding the extension capabilities of the large models. This includes features like knowledge base retrieval, computational tools, WolframAlpha integration, and software manipulation.
+
+We have initially integrated the LangChain framework to facilitate the development of applications like document retrieval, question-answering bots, and intelligent agents based on the Llama2 model. For more information on LangChain, please refer to [LangChain](https://github.com/langchain-ai/langchain).
+
+##### LangChain
+For a simplified implementation using the LangChain framework with the Llama2 LLM class, refer to [examples/llama2_for_langchain.py](https://github.com/LlamaFamily/Llama-Chinese/blob/main/examples/llama2_for_langchain.py). Here is a basic code snippet:
+
+```python
+from llama2_for_langchain import Llama2
+
+# Example using FlagAlpha/Atom-7B-Chat
+llm = Llama2(model_name_or_path='FlagAlpha/Atom-7B-Chat')
+
+while True:
+    human_input = input("Human: ")
+    response = llm(human_input)
+    print(f"Llama2: {response}")
+```
+
+
+##### 🥇 Model Evaluation
+
+###### Llama3 Evaluation
+
+###### Llama2 Evaluation
+
+<p align="center" width="100%">
+<img src="./assets/llama_eval.jpeg" style="width: 100%; display: block; margin: auto;">
+</p>
+
+To gain a clearer understanding of the Chinese question-answering capabilities of the Llama2 model, we selected a set of representative Chinese questions for testing. The tested models include Meta's publicly available versions, namely, Llama2-7B-Chat and Llama2-13B-Chat, without any fine-tuning or training. The test questions were curated from [AtomBulb](https://github.com/AtomEcho/AtomBulb), totaling 95 questions covering eight major categories: general knowledge, language understanding, creative ability, logical reasoning, code programming, work skills, tool usage, and personality traits.
+
+The prompt used during testing is as follows, for example, for the question "List 5 methods to improve sleep quality":
+
+```plaintext
+[INST] 
+<<SYS>>
+You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. The answer always been translate into Chinese language.
+
+If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
+
+The answer always been translate into Chinese language.
+<</SYS>>
+
+List 5 methods to improve sleep quality
+[/INST]
+```
+The test results for Llama2-7B-Chat can be found at[meta_eval_7B.md](assets/meta_eval_7B.md)，and for Llama2-13B-Chat at [meta_eval_13B.md](assets/meta_eval_13B.md)。
+
+Through our testing, we observed that Meta's original Llama2 Chat model generally has mediocre alignment with Chinese questions. In most cases, it fails to provide Chinese answers, or the responses are a mixture of Chinese and English. Therefore, it is crucial to train and fine-tune the Llama2 model on Chinese data. Our Chinese version of the Llama2 model is currently undergoing training and will be made available to the community in the near future.
+
+
+#### 📖 Learning Resources
+
+##### Llama3
+
+##### Llama2
+###### Meta Official Introduction to [Llama2](https://ai.meta.com/llama)
+Since the release of Meta's first-generation LLaMA model, the Llama model family has thrived. Recently, Meta released the Llama2 version, which is open-source and commercially available, with significant updates in model and performance. Llama2 has models with parameter sizes of 7B, 13B, and 70B. Compared to LLaMA, Llama2's training data has reached 20 trillion tokens, and the context length has been upgraded from the previous 2048 to 4096, allowing it to understand and generate longer text. The Llama2 Chat model, fine-tuned based on 1 million human-labeled data, achieves results close to ChatGPT in English conversations.
+
+###### Llama-related Papers
+* [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971)
+* [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/abs/2307.09288)
+* [Code Llama: Open Foundation Models for Code](https://ai.meta.com/research/publications/code-llama-open-foundation-models-for-code/)
+
+## 📌 Others
+
+## 🎉 Acknowledgments
+Special thanks to the AtomEcho team for their technical and resource support!
+
+Thanks to @xzsGenius for contributions to the Llama2 Chinese community!
+
+Thanks to the Z-Potentials community for supporting the Llama2 Chinese community!
+
+## 🤔 Issue Feedback
+If you have any issues, please submit them in the GitHub Issues. Before submitting a new issue, please check existing issues to see if your problem has already been addressed.
+
+Please be polite when raising issues and contribute to building a harmonious discussion community.
+
+Join the [Feishu Knowledge Base](https://chinesellama.feishu.cn/wiki/space/7257824476874768388?ccm_open_type=lark_wiki_spaceLink) to collaboratively build community documentation.
+
+Join the WeChat group for discussions 😍😍
+
+
+<p align="center" width="100%">
+<img src="./assets/wechat.jpeg" alt="Wechat" style="width: 100%; display: block; margin: auto;">
+</p>
+
+<p align="center" width="100%">
+<img src="https://api.star-history.com/svg?repos=LlamaFamily/Llama-Chinese&type=Date" alt="Wechat" style="width: 100%; display: block; margin: auto;">
+</p>

Llama-Chinese/docker/Dockerfile

@@ -0,0 +1,27 @@
+# 使用pytorch/pytorch:2.1.2-cuda12.1-cudnn8-devel作为基础镜像
+FROM pytorch/pytorch:2.1.2-cuda12.1-cudnn8-devel
+
+RUN apt-get update -y --allow-unauthenticated 
+RUN apt install -y git vim git-lfs
+
+#设置工作目录
+WORKDIR /root/Llama-Chinese
+
+# 从git上克隆llama-chinese仓库
+RUN git clone https://github.com/LlamaFamily/Llama-Chinese.git /root/Llama-Chinese 
+
+# tsinghua source
+RUN mkdir -p ~/.pip
+RUN echo "[global]\nindex-url = https://pypi.tuna.tsinghua.edu.cn/simple" > ~/.pip/pip.conf
+
+# 使用pip安装requirements.txt
+RUN pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn -r requirements.txt
+
+#克隆Hugging Face仓库
+RUN  git clone https://huggingface.co/FlagAlpha/Atom-7B-Chat
+
+#开启7860端口
+EXPOSE 7860
+
+#设置启动命令
+ENTRYPOINT ["python", "examples/chat_gradio.py", "--model_name_or_path", "/root/Llama-Chinese/Atom-7B-Chat/"]

Llama-Chinese/docker/Dockerfile_train

@@ -0,0 +1,9 @@
+FROM pytorch/pytorch:2.1.2-cuda12.1-cudnn8-devel  as builder
+RUN apt-get update -y --allow-unauthenticated 
+RUN apt install git tmux htop vim -y 
+RUN pip install bitsandbytes -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn
+RUN pip install transformers -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn
+RUN pip install peft -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn
+RUN pip install accelerate -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn
+RUN pip install deepspeed -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn
+RUN pip install scipy sentencepiece datasets joblib sentence_transformers cn2an evaluate tensorboard wandb -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn

Llama-Chinese/docker/docker-compose.yml

@@ -0,0 +1,16 @@
+version: '3.7'
+services:
+  app:
+    image: flagalpha/llama2-chinese:gradio # 这里替换为你实际的镜像名
+    volumes:
+      - /usr/local/nvidia:/usr/local/nvidia # 让容器访问主机的NVIDIA驱动
+    environment:
+      - NVIDIA_VISIBLE_DEVICES=all # 让容器可以访问所有的NVIDIA GPU
+    ports:
+      - 7860:7860 # 在容器和主机之间映射端口
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              capabilities: [gpu] # 使用Docker的设备请求来让容器使用GPU

Llama-Chinese/docs/chat_gradio_guide.md

@@ -0,0 +1,26 @@
+#  Docker环境执行chat_gradio.py
+
+系统需要准备的环境
+
++ docker： 24.0.2
++ docker-compose
+
+## 第一步. 准备Docker镜像
+
+通过docker镜像可以更方便的管理需要安装的环境依赖。所以这里可以直接通过docker容器启动[chat_gradio](../examples/chat_gradio.py)， 第一步准备镜像环境。
+
+```bash
+git clone https://github.com/LlamaFamily/Llama-Chinese.git
+
+cd Llama-Chinese
+
+docker build -f docker/Dockerfile  -t FlagAlpha/llama2-chinese:gradio .
+```
+
+## 第二步. 通过docker-compose启动chat_gradio
+
+
+```bash
+cd Llama-Chinese/docker
+doker-compose up -d --build
+```

Llama-Chinese/docs/inference_speed_guide.md

@@ -0,0 +1,21 @@
+# 推理部署
+
+> 训练完之后或者经过微调之后的模型或者直接从[huggingface](https://huggingface.co/FlagAlpha)下载的模型，都需要部署使用。部署也就是指的模型推理，如果直接使用原生的trainsfomers进行部署，速度会比较慢。针对推理有多种加速手段，会带来较快的推理速度。
+
+
+
+## 1. GPU推理方案
+
+### 方案一：vllm
+
+[使用说明](../inference-speed/GPU/vllm_example/README.md)
+
+### 方案二：TensorRT-LLM
+
+[使用说明](../inference-speed/GPU/TensorRT-LLM_example/README.md)
+
+
+## 2. CPU 推理方案
+
+### 方案一：ggml
+[使用说明](../inference-speed/CPU/ggml/README.md)

Llama-Chinese/examples/chat_gradio.py

@@ -0,0 +1,99 @@
+import gradio as gr
+import time
+from transformers import AutoTokenizer, AutoModelForCausalLM,TextIteratorStreamer
+from threading import Thread
+import torch,sys,os
+import json
+import pandas 
+import argparse
+
+with gr.Blocks() as demo:
+    gr.Markdown("""<h1><center>智能助手</center></h1>""")
+    chatbot = gr.Chatbot()
+    msg = gr.Textbox()
+    state = gr.State()
+    with gr.Row():
+        clear = gr.Button("新话题")
+        re_generate = gr.Button("重新回答")
+        sent_bt = gr.Button("发送")
+    with gr.Accordion("生成参数", open=False):
+        slider_temp = gr.Slider(minimum=0, maximum=1, label="temperature", value=0.3)
+        slider_top_p = gr.Slider(minimum=0.5, maximum=1, label="top_p", value=0.95)
+        slider_context_times = gr.Slider(minimum=0, maximum=5, label="上文轮次", value=0,step=2.0)
+    def user(user_message, history):
+        return "", history + [[user_message, None]]
+    def bot(history,temperature,top_p,slider_context_times):
+        if pandas.isnull(history[-1][1])==False:
+            history[-1][1] = None
+            yield history
+        slider_context_times = int(slider_context_times)
+        history_true = history[1:-1]
+        prompt = ''
+        if slider_context_times>0:
+            prompt += '\n'.join([("<s>Human: "+one_chat[0].replace('<br>','\n')+'\n</s>' if one_chat[0] else '')  +"<s>Assistant: "+one_chat[1].replace('<br>','\n')+'\n</s>'    for one_chat in history_true[-slider_context_times:] ])
+        prompt +=  "<s>Human: "+history[-1][0].replace('<br>','\n')+"\n</s><s>Assistant:"
+        input_ids = tokenizer([prompt], return_tensors="pt",add_special_tokens=False).input_ids[:,-512:].to('cuda')        
+        generate_input = {
+            "input_ids":input_ids,
+            "max_new_tokens":512,
+            "do_sample":True,
+            "top_k":50,
+            "top_p":top_p,
+            "temperature":temperature,
+            "repetition_penalty":1.3,
+            "streamer":streamer,
+            "eos_token_id":tokenizer.eos_token_id,
+            "bos_token_id":tokenizer.bos_token_id,
+            "pad_token_id":tokenizer.pad_token_id
+        }
+        thread = Thread(target=model.generate, kwargs=generate_input)
+        thread.start()
+        start_time = time.time()
+        bot_message =''
+        print('Human:',history[-1][0])
+        print('Assistant: ',end='',flush=True)
+        for new_text in streamer:
+            print(new_text,end='',flush=True)
+            if len(new_text)==0:
+                continue
+            if new_text!='</s>':
+                bot_message+=new_text
+            if 'Human:' in bot_message:
+                bot_message = bot_message.split('Human:')[0]
+            history[-1][1] = bot_message
+            yield history
+        end_time =time.time()
+        print()
+        print('生成耗时：',end_time-start_time,'文字长度：',len(bot_message),'字耗时：',(end_time-start_time)/len(bot_message))
+
+    msg.submit(user, [msg, chatbot], [msg, chatbot], queue=False).then(
+        bot, [chatbot,slider_temp,slider_top_p,slider_context_times], chatbot
+    )
+    sent_bt.click(user, [msg, chatbot], [msg, chatbot], queue=False).then(
+        bot, [chatbot,slider_temp,slider_top_p,slider_context_times], chatbot
+    )
+    re_generate.click( bot, [chatbot,slider_temp,slider_top_p,slider_context_times], chatbot )
+    clear.click(lambda: [], None, chatbot, queue=False)
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model_name_or_path", type=str, help='mode name or path')
+    parser.add_argument("--is_4bit", action='store_true', help='use 4bit model')
+    args = parser.parse_args()
+    tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path,use_fast=False)
+    tokenizer.pad_token = tokenizer.eos_token
+    if args.is_4bit==False:
+        model = AutoModelForCausalLM.from_pretrained(args.model_name_or_path,
+                                                     device_map='cuda:0' if torch.cuda.is_available() else "auto",
+                                                     torch_dtype=torch.float16,
+                                                     load_in_8bit=True,
+                                                     trust_remote_code=True,
+                                                     use_flash_attention_2=True)
+        model.eval()
+    else:
+        from auto_gptq import AutoGPTQForCausalLM
+        model = AutoGPTQForCausalLM.from_quantized(args.model_name_or_path,low_cpu_mem_usage=True, device="cuda:0", use_triton=False,inject_fused_attention=False,inject_fused_mlp=False)
+    streamer = TextIteratorStreamer(tokenizer,skip_prompt=True)
+    if torch.__version__ >= "2" and sys.platform != "win32":
+        model = torch.compile(model)
+    demo.queue().launch(share=False, debug=True,server_name="0.0.0.0")

Llama-Chinese/examples/chat_gradio_no_merge.py

@@ -0,0 +1,105 @@
+import gradio as gr
+import time
+from transformers import AutoTokenizer, AutoModelForCausalLM,TextIteratorStreamer
+from threading import Thread
+from peft import PeftModel,PeftConfig
+import torch,sys,os
+import json
+import pandas 
+import argparse
+
+with gr.Blocks() as demo:
+    gr.Markdown("""<h1><center>智能助手</center></h1>""")
+    chatbot = gr.Chatbot()
+    msg = gr.Textbox()
+    state = gr.State()
+    with gr.Row():
+        clear = gr.Button("新话题")
+        re_generate = gr.Button("重新回答")
+        sent_bt = gr.Button("发送")
+    with gr.Accordion("生成参数", open=False):
+        slider_temp = gr.Slider(minimum=0, maximum=1, label="temperature", value=0.3)
+        slider_top_p = gr.Slider(minimum=0.5, maximum=1, label="top_p", value=0.95)
+        slider_context_times = gr.Slider(minimum=0, maximum=5, label="上文轮次", value=0,step=2.0)
+    def user(user_message, history):
+        return "", history + [[user_message, None]]
+    def bot(history,temperature,top_p,slider_context_times):
+        if pandas.isnull(history[-1][1])==False:
+            history[-1][1] = None
+            yield history
+        slider_context_times = int(slider_context_times)
+        history_true = history[1:-1]
+        prompt = ''
+        if slider_context_times>0:
+            prompt += '\n'.join([("<s>Human: "+one_chat[0].replace('<br>','\n')+'\n</s>' if one_chat[0] else '')  +"<s>Assistant: "+one_chat[1].replace('<br>','\n')+'\n</s>'    for one_chat in history_true[-slider_context_times:] ])
+        prompt +=  "<s>Human: "+history[-1][0].replace('<br>','\n')+"\n</s><s>Assistant:"
+        input_ids = tokenizer([prompt], return_tensors="pt",add_special_tokens=False).input_ids[:,-512:].to('cuda')        
+        generate_input = {
+            "input_ids":input_ids,
+            "max_new_tokens":512,
+            "do_sample":True,
+            "top_k":50,
+            "top_p":top_p,
+            "temperature":temperature,
+            "repetition_penalty":1.3,
+            "streamer":streamer,
+            "eos_token_id":tokenizer.eos_token_id,
+            "bos_token_id":tokenizer.bos_token_id,
+            "pad_token_id":tokenizer.pad_token_id
+        }
+        thread = Thread(target=model.generate, kwargs=generate_input)
+        thread.start()
+        start_time = time.time()
+        bot_message =''
+        print('Human:',history[-1][0])
+        print('Assistant: ',end='',flush=True)
+        for new_text in streamer:
+            print(new_text,end='',flush=True)
+            if len(new_text)==0:
+                continue
+            if new_text!='</s>':
+                bot_message+=new_text
+            if 'Human:' in bot_message:
+                bot_message = bot_message.split('Human:')[0]
+            history[-1][1] = bot_message
+            yield history
+        end_time =time.time()
+        print()
+        print('生成耗时：',end_time-start_time,'文字长度：',len(bot_message),'字耗时：',(end_time-start_time)/len(bot_message))
+
+    msg.submit(user, [msg, chatbot], [msg, chatbot], queue=False).then(
+        bot, [chatbot,slider_temp,slider_top_p,slider_context_times], chatbot
+    )
+    sent_bt.click(user, [msg, chatbot], [msg, chatbot], queue=False).then(
+        bot, [chatbot,slider_temp,slider_top_p,slider_context_times], chatbot
+    )
+    re_generate.click( bot, [chatbot,slider_temp,slider_top_p,slider_context_times], chatbot )
+    clear.click(lambda: [], None, chatbot, queue=False)
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model_name_or_path", type=str, help='mode name or path')
+    parser.add_argument("--is_4bit", action='store_true', help='use 4bit model')
+    args = parser.parse_args()
+    # tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path,use_fast=False)
+    # tokenizer.pad_token = tokenizer.eos_token
+    if args.is_4bit==False:
+        config = PeftConfig.from_pretrained(args.model_name_or_path)
+        tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path,use_fast=False)
+        tokenizer.pad_token = tokenizer.eos_token
+        model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path,
+                                                     device_map='cuda:0' if torch.cuda.is_available() else "auto",
+                                                     torch_dtype=torch.float16,
+                                                     load_in_8bit=True,
+                                                     low_cpu_mem_usage=True,
+                                                     trust_remote_code=True,
+                                                     use_flash_attention_2=True)
+        model = PeftModel.from_pretrained(model, args.model_name_or_path, device_map={"": 0})
+        model.eval()
+    else:
+        from auto_gptq import AutoGPTQForCausalLM
+        model = AutoGPTQForCausalLM.from_quantized(args.model_name_or_path,low_cpu_mem_usage=True, device="cuda:0", use_triton=False,inject_fused_attention=False,inject_fused_mlp=False)
+    streamer = TextIteratorStreamer(tokenizer,skip_prompt=True)
+    if torch.__version__ >= "2" and sys.platform != "win32":
+        model = torch.compile(model)
+    demo.queue().launch(share=False, debug=True,server_name="0.0.0.0")

Llama-Chinese/examples/llama2_for_langchain.py

@@ -0,0 +1,52 @@
+from langchain.llms.base import LLM
+from typing import Dict, List, Any, Optional
+import torch,sys,os
+from transformers import AutoTokenizer
+
+
+class Llama2(LLM):
+    max_token: int = 2048
+    temperature: float = 0.1
+    top_p: float = 0.95
+    tokenizer: Any
+    model: Any
+    
+    def __init__(self, model_name_or_path, bit4=False):
+        super().__init__()
+        self.tokenizer = AutoTokenizer.from_pretrained(model_name_or_path,use_fast=False)
+        self.tokenizer.pad_token = self.tokenizer.eos_token
+        if bit4==False:
+            from transformers import AutoModelForCausalLM
+            device_map = "cuda:0" if torch.cuda.is_available() else "auto"
+            self.model = AutoModelForCausalLM.from_pretrained(model_name_or_path,device_map=device_map,torch_dtype=torch.float16,load_in_8bit=True,trust_remote_code=True,use_flash_attention_2=True)
+            self.model.eval()
+        else:
+            from auto_gptq import AutoGPTQForCausalLM
+            self.model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,low_cpu_mem_usage=True, device="cuda:0", use_triton=False,inject_fused_attention=False,inject_fused_mlp=False)
+            
+        if torch.__version__ >= "2" and sys.platform != "win32":
+            self.model = torch.compile(self.model)
+            
+    @property
+    def _llm_type(self) -> str:
+        return "Llama2"
+
+    def _call(self, prompt: str, stop: Optional[List[str]] = None) -> str:
+        print('prompt:',prompt)
+        input_ids = self.tokenizer(prompt, return_tensors="pt",add_special_tokens=False).input_ids.to('cuda')
+        generate_input = {
+            "input_ids":input_ids,
+            "max_new_tokens":1024,
+            "do_sample":True,
+            "top_k":50,
+            "top_p":self.top_p,
+            "temperature":self.temperature,
+            "repetition_penalty":1.2,
+            "eos_token_id":self.tokenizer.eos_token_id,
+            "bos_token_id":self.tokenizer.bos_token_id,
+            "pad_token_id":self.tokenizer.pad_token_id
+        }
+        generate_ids = self.model.generate(**generate_input)
+        generate_ids = [item[len(input_ids[0]):-1] for  item in generate_ids]
+        result_message = self.tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
+        return result_message

Llama-Chinese/finetune_test.py

@@ -0,0 +1,32 @@
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from peft import PeftModel,PeftConfig
+# 例如: finetune_model_path='train/sft/output/checkpoint-6160'
+finetune_model_path='train/sft/output/'  
+config = PeftConfig.from_pretrained(finetune_model_path)
+# 例如: base_model_name_or_path='meta-llama/Llama-2-7b-chat'
+print(config.base_model_name_or_path)
+tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path,use_fast=False)
+tokenizer.pad_token = tokenizer.eos_token
+device_map = "cuda:0" if torch.cuda.is_available() else "auto"
+model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path,device_map=device_map,torch_dtype=torch.float16,load_in_8bit=True,trust_remote_code=True,use_flash_attention_2=True)
+model = PeftModel.from_pretrained(model, finetune_model_path, device_map={"": 0})
+model =model.eval()
+input_ids = tokenizer(['<s>Human: 介绍一下北京\n</s><s>Assistant: '], return_tensors="pt",add_special_tokens=False).input_ids
+if torch.cuda.is_available():
+  input_ids = input_ids.to('cuda')
+generate_input = {
+    "input_ids":input_ids,
+    "max_new_tokens":512,
+    "do_sample":True,
+    "top_k":50,
+    "top_p":0.95,
+    "temperature":0.3,
+    "repetition_penalty":1.3,
+    "eos_token_id":tokenizer.eos_token_id,
+    "bos_token_id":tokenizer.bos_token_id,
+    "pad_token_id":tokenizer.pad_token_id
+}
+generate_ids  = model.generate(**generate_input)
+text = tokenizer.decode(generate_ids[0])
+print(text)

Llama-Chinese/important-change.txt

@@ -0,0 +1,2 @@
+root/miniconda3/envs/new-Llama/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py  Line 88
+add return True

Llama-Chinese/inference-speed/CPU/ggml/README.md

@@ -0,0 +1,66 @@
+## 使用llama.cpp量化部署
+
+以[llama.cpp工具](https://github.com/Rayrtfr/llama.cpp)为例，介绍模型量化并在本地部署的详细步骤。Windows则可能需要cmake等编译工具的安装。**本地快速部署体验推荐使用经过指令精调的[Atom-7B-Chat](https://github.com/LlamaFamily/Llama-Chinese?tab=readme-ov-file#%E5%9F%BA%E4%BA%8Ellama2%E7%9A%84%E4%B8%AD%E6%96%87%E9%A2%84%E8%AE%AD%E7%BB%83%E6%A8%A1%E5%9E%8Batom)模型，有条件的推荐使用6-bit或者8-bit模型，效果更佳。** 运行前请确保：
+
+1. 系统应有`make`（MacOS/Linux自带）或`cmake`（Windows需自行安装）编译工具
+2. 建议使用Python 3.10以上编译和运行该工具
+
+
+### Step 1: 克隆和编译llama.cpp
+
+1. （可选）如果已下载旧版仓库，建议`git pull`拉取最新代码，**并执行`make clean`进行清理**
+1. 拉取最新版适配过Atom大模型的llama.cpp仓库代码
+
+```bash
+$ git clone https://github.com/Rayrtfr/llama.cpp
+```
+
+2. 对llama.cpp项目进行编译，生成`./main`（用于推理）和`./quantize`（用于量化）二进制文件。
+
+```bash
+$ make
+```
+
+**Windows/Linux用户**如需启用GPU推理，则推荐与[BLAS（或cuBLAS如果有GPU）一起编译](https://github.com/Rayrtfr/llama.cpp#blas-build)，可以提高prompt处理速度。以下是和cuBLAS一起编译的命令，适用于NVIDIA相关GPU。参考：[llama.cpp#blas-build](https://github.com/Rayrtfr/llama.cpp#blas-build)
+
+```bash
+$ make LLAMA_CUBLAS=1
+```
+
+**macOS用户**无需额外操作，llama.cpp已对ARM NEON做优化，并且已自动启用BLAS。M系列芯片推荐使用Metal启用GPU推理，显著提升速度。只需将编译命令改为：`LLAMA_METAL=1 make`，参考[llama.cpp#metal-build](https://github.com/Rayrtfr/llama.cpp#metal-build)
+
+```bash
+$ LLAMA_METAL=1 make
+```
+
+###  Step 2: 生成量化版本模型
+
+目前llama.cpp已支持`.safetensors`文件以及huggingface格式`.bin`转换为GGUF的FP16格式。
+
+/path/Atom-7B-Chat是模型下载的目录位置。
+```bash
+$ python convert.py --outfile ./atom-7B-cpp.gguf  /path/Atom-7B-Chat
+
+$ ./quantize ./atom-7B-cpp.gguf ./ggml-atom-7B-q4_0.gguf q4_0
+```
+
+### Step 3: 加载并启动模型
+
+
+- 如果想使用GPU推理：cuBLAS/Metal编译需要指定offload层数，在`./main`中指定例如`-ngl 40`表示offload 40层模型参数到GPU
+
+
+使用以下命令启动聊天。
+```bash
+text="<s>Human: 介绍一下北京\n</s><s>Assistant:"
+./main -m \
+./ggml-atom-7B-q4_0.gguf \
+-p "${text}"  \
+--logdir ./logtxt 
+```
+如果要带聊天的上下文，上面的text需要调整成类似这样：
+```bash
+text="<s>Human: 介绍一下北京\n</s><s>Assistant:北京是一个美丽的城市</s>\n<s>Human: 再介绍一下合肥\n</s><s>Assistant:"
+```
+
+更详细的官方说明请参考：[https://github.com/ggerganov/llama.cpp/tree/master/examples/main](https://github.com/ggerganov/llama.cpp/tree/master/examples/main)

Llama-Chinese/inference-speed/GPU/FasterTransformer_example/README.md

@@ -0,0 +1,165 @@
+#  FasterTransformer &&  Triton 安装和使用
+
+FasterTransformer & Triton 加速LLama2模型推理。 目前支持fp16或者Int8推理，Int4目前还不支持。
+
+## 0. 准备环境变量
+
+```bash
+export BUILD_DICTIONARY="/workspace/build"
+export TRITON_VERSION=23.04
+```
+
+
+## 一. 镜像构建
+
+
+1. 构建镜像
+
+```bash
+cd $BUILD_DICTIONARY
+git clone https://github.com/Rayrtfr/fastertransformer_backend.git 
+
+cd $BUILD_DICTIONARY/fastertransformer_backend
+
+export TRITON_VERSION=23.04
+
+# 如何不想通过下面的命令构建，也可以直接下载我们已经构建好的镜像: docker pull xiangtao1994/atom_triton_ft:23.04
+docker build --build-arg TRITON_VERSION=${TRITON_VERSION} -t triton_ft_backend:${TRITON_VERSION} -f docker/Dockerfile .
+
+```
+TRITON_VERSION=23.04 这个镜像需的GPU的驱动版本是 Driver Version: 535.54.03，如果你的GPU的驱动不是这个版本，需要[https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/rel-22-12.html#rel-22-12](https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/rel-22-12.html#rel-22-12)
+找到cuda driver 对应版本的 triton-inference-server。
+
+
+2.启动容器
+
+```
+# 启动容器
+export TRITON_VERSION=23.04
+
+# 注意需要 BUILD_DICTIONARY 挂载到容器里面
+docker run -idt --gpus=all --net=host  --shm-size=4G --name triton_ft_backend_pure \
+  -v $BUILD_DICTIONARY:$BUILD_DICTIONARY \
+  -p18888:8888 -p18000:8000 -p18001:8001 -p18002:8002 triton_ft_backend:${TRITON_VERSION}  bash 
+
+````
+
+## 二.容器内操作
+
+下面介绍一下[Atom-7B-Chat](https://huggingface.co/FlagAlpha/Atom-7B-Chat)模型的权重转换成FasterTransformer格式。 [Llama2-Chinese-13b-Chat](https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat)也是类似的方式。
+
+1. 转换权重, 权重转换成FasterTransformer格式
+
+```
+cd $BUILD_DICTIONARY && git clone https://github.com/Rayrtfr/FasterTransformer.git
+
+cd $BUILD_DICTIONARY/FasterTransformer
+
+mkdir models && chmod -R 777 ./*
+
+python3 ./examples/cpp/llama/huggingface_llama_convert.py \
+-saved_dir=./models/llama \
+-in_file=/path/FlagAlpha/Atom-7B-Chat \
+-infer_gpu_num=1 \
+-weight_data_type=fp16 \
+-model_name=llama
+```
+
+2. 修改模型配置
+
+- 编辑config.pbtxt
+
+``` bash
+mkdir $BUILD_DICTIONARY/triton-model-store/
+
+cd $BUILD_DICTIONARY/triton-model-store/
+
+cp -r $BUILD_DICTIONARY/fastertransformer_backend/all_models/llama $BUILD_DICTIONARY/triton-model-store/
+
+# 修改 triton-model-store/llama/fastertransformer/config.pbtxt
+
+parameters {
+  key: "tensor_para_size"
+  value: {
+    string_value: "1"
+  }
+}
+
+## 修改 model_checkpoint_path 为上面转换之后的路径
+parameters {
+  key: "model_checkpoint_path"
+  value: {
+    string_value: "/workspace/build/FasterTransformer/models/llama/1-gpu/"
+  }
+}
+
+## 模型使用int8推理需要加一下面的配置
+parameters { 
+  key: "int8_mode" 
+  value: { 
+    string_value: "1"
+  } 
+}
+```
+
+
+修改 model.py
+
+```
+# 修改这两个文件
+triton-model-store/llama/preprocessing/1/model.py
+triton-model-store/llama/postprocessing/1/model.py
+
+# 检查 这个路径为tokenier对应的路径
+self.tokenizer = LlamaTokenizer.from_pretrained("/path/FlagAlpha/Atom-7B-Chat")
+```
+
+
+3. 编译 FasterTransformer Library
+
+(同一类型的模型，编译一次就行了)
+编译之前检查 FasterTransformer/examples/cpp/llama/llama_config.ini
+
+```bash
+# 单卡推理这里是1，多卡可以改成卡的数目
+tensor_para_size=1
+
+model_dir=/workspace/build/FasterTransformer/models/llama/1-gpu/
+```
+
+编译 FasterTransformer
+```bash
+cd $BUILD_DICTIONARY/FasterTransformer
+
+git submodule init && git submodule update
+pip3 install fire jax jaxlib transformers
+
+mkdir build && cd build
+cmake -DSM=86 -DCMAKE_BUILD_TYPE=Release -DBUILD_PYT=ON -DBUILD_MULTI_GPU=ON -D PYTHON_PATH=/usr/bin/python3 ..
+make -j12
+make install
+```
+
+
+## 三. 启动 triton server
+
+同样在上面的容器内操作。
+```
+CUDA_VISIBLE_DEVICES=0 /opt/tritonserver/bin/tritonserver  --model-repository=$BUILD_DICTIONARY/triton-model-store/llama/
+```
+输出
+```
+I0717 17:17:14.670037 70681 grpc_server.cc:2450] Started GRPCInferenceService at 0.0.0.0:8001
+I0717 17:17:14.670495 70681 http_server.cc:3555] Started HTTPService at 0.0.0.0:8000
+I0717 17:17:14.713000 70681 http_server.cc:185] Started Metrics Service at 0.0.0.0:8002
+```
+
+
+同样在上面的容器内操作，启动client测试（如果在容器外注意需要修改下面的url参数的端口号）
+
+```
+python3 $BUILD_DICTIONARY/fastertransformer_backend/inference_example/llama/llama_grpc_stream_client.py \
+--url 127.0.0.1:8001 \
+--hf_model_location /path/FlagAlpha/Atom-7B-Chat \
+-topp 0.95
+```

Llama-Chinese/inference-speed/GPU/JittorLLMs_example/README.md

@@ -0,0 +1,96 @@
+# JittorLLMs推理部署
+
+## 配置要求
+
+* 内存要求：至少2G，推荐32G
+* 显存：可选， 推荐16G
+* 操作系统：支持Windows，Mac，Linux全平台。
+* 磁盘空间：至少40GB空闲磁盘空间，用于下载参数和存储交换文件。
+* Python版本要求至少`3.9`。
+
+磁盘空间不够时，可以通过环境变量`JITTOR_HOME`指定缓存存放路径。
+内存或者显存不够，出现进程被杀死的情况，请参考下方，[限制内存消耗的方法](#配置要求低)。
+
+## 部署方法
+
+可以通过下述指令安装依赖。（注意：此脚本会安装Jittor版torch，推荐用户新建环境运行）
+
+```
+# 国内使用 gitlink clone
+git clone https://gitlink.org.cn/jittor/JittorLLMs.git --depth 1
+# github: git clone https://github.com/Jittor/JittorLLMs.git --depth 1
+cd JittorLLMs
+# -i 指定用jittor的源， -I 强制重装Jittor版torch
+pip install -r requirements.txt -i https://pypi.jittor.org/simple -I
+```
+
+如果出现找不到jittor版本的错误，可能是您使用的镜像还没有更新，使用如下命令更新最新版：`pip install jittor -U -i https://pypi.org/simple`
+
+部署只需一行命令即可：
+
+```
+python cli_demo.py atom7b
+```
+
+运行后会自动从服务器上下载模型文件到本地，会占用根目录下一定的硬盘空间。
+最开始运行的时候会编译一些CUDA算子，这会花费一些时间进行加载。
+
+内存或者显存不够，出现进程被杀死的情况，请参考下方，[限制内存消耗的方法](#配置要求低)。
+
+### WebDemo
+
+JittorLLM通过gradio库，允许用户在浏览器之中和大模型直接进行对话。
+
+~~~bash
+python web_demo.py atom7b
+~~~
+
+### 后端服务部署
+
+JittorLLM在api.py文件之中，提供了一个架设后端服务的示例。
+
+~~~bash
+python api.py atom7b
+~~~
+
+接着可以使用如下代码进行直接访问
+
+~~~python
+post_data = json.dumps({'prompt': 'Hello, solve 5x=13'})
+print(json.loads(requests.post("http://0.0.0.0:8000", post_data).text)['response'])
+~~~
+
+## 配置要求低
+
+针对大模型显存消耗大等痛点，Jittor团队研发了动态交换技术，Jittor框架是世界上首个支持动态图变量自动交换功能的框架，区别于以往的基于静态图交换技术，用户不需要修改任何代码，原生的动态图代码即可直接支持张量交换，张量数据可以在显存-内存-硬盘之间自动交换，降低用户开发难度。
+
+同时，Jittor大模型推理库也是目前对配置门槛要求最低的框架，只需要参数磁盘空间和2G内存，无需显卡，也可以部署大模型，下面是在不同硬件配置条件下的资源消耗与速度对比。可以发现，JittorLLMs在显存充足的情况下，性能优于同类框架，而显存不足甚至没有显卡，JittorLLMs都能以一定速度运行。
+
+节省内存方法，请安装Jittor版本大于1.3.7.8，并添加如下环境变量：
+```bash
+export JT_SAVE_MEM=1
+# 限制cpu最多使用16G
+export cpu_mem_limit=16000000000
+# 限制device内存（如gpu、tpu等）最多使用8G
+export device_mem_limit=8000000000
+# windows 用户，请使用powershell
+# $env:JT_SAVE_MEM="1"
+# $env:cpu_mem_limit="16000000000"
+# $env:device_mem_limit="8000000000"
+```
+用户可以自由设定cpu和设备内存的使用量，如果不希望对内存进行限制，可以设置为`-1`。
+```bash
+# 限制cpu最多使用16G
+export cpu_mem_limit=-1
+# 限制device内存（如gpu、tpu等）最多使用8G
+export device_mem_limit=-1
+# windows 用户，请使用powershell
+# $env:JT_SAVE_MEM="1"
+# $env:cpu_mem_limit="-1"
+# $env:device_mem_limit="-1"
+```
+
+如果想要清理磁盘交换文件，可以运行如下命令
+```bash
+python -m jittor_utils.clean_cache swap
+```

Llama-Chinese/inference-speed/GPU/TensorRT-LLM_example/README.md

@@ -0,0 +1,72 @@
+# 使用NVIDIA TensorRT-LLM部署LLama2 或者Atom
+
+[TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM/tree/main)是NVIDIA开发的高性能推理框架，您可以按照以下步骤来使用TensorRT-LLM部署LLama2模型或者Atom模型。
+
+以下部署流程参考[TensorRT-LLM/example/llama](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/llama)，需要机器Nvidia显卡驱动535版本以上
+
+## Support Matrix
+  * FP16
+  * FP8
+  * INT8 & INT4 Weight-Only
+  * SmoothQuant
+  * Groupwise quantization (AWQ/GPTQ)
+  * FP8 KV CACHE
+  * INT8 KV CACHE (+ AWQ/per-channel weight-only)
+  * Tensor Parallel
+  * STRONGLY TYPED
+
+## 1. 安装TensorRT-LLM
+#### 获取TensorRT-LLM代码：
+
+```bash
+# TensorRT-LLM 代码需要使用 git-lfs 拉取
+apt-get update && apt-get -y install git git-lfs
+
+git clone https://github.com/NVIDIA/TensorRT-LLM.git
+cd TensorRT-LLM
+
+# 本流程将使用 v0.7.0 Release 版本
+git checkout tags/v0.7.0 -b release/0.7.0
+git submodule update --init --recursive
+git lfs install
+git lfs pull
+```
+#### 构建docker镜像并安装TensorRT-LLM
+```bash
+make -C docker release_build
+```
+
+#### 运行docker镜像：
+```bash
+make -C docker release_run
+```
+
+## 2. 为LLama2模型构建TensorRT-LLM推理引擎：
+
+#### 进入build文件夹：
+```bash
+cd ./examples/llama
+```
+
+#### 从Huggingface下载Atom或者LLama2模型：
+```
+# 您可以选择具体想部署的模型下载
+git clone https://huggingface.co/FlagAlpha/Atom-7B-Chat      Atom-7B-Chat
+mv Atom-7B-Chat /origin_model
+```
+
+#### 使用build.py 构建推理引擎：
+以下是一个常见事例，更多参数参考[TensorRT-LLM/example/llama](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/llama)
+```bash
+python build.py --max_batch_size 1 --max_num_tokens 8192  --model_dir /origin_model --dtype float16  --remove_input_padding --use_inflight_batching --paged_kv_cache --use_weight_only --enable_context_fmha --use_gpt_attention_plugin float16  --use_gemm_plugin float16 --output_dir /model/tensorrt_llm/1 --world_size 1 --tp_size 1 --pp_size 1 --max_input_len 7168 --max_output_len 1024 --multi_block_mode --rotary_scaling dynamic 8.0 --rotary_base 500000
+```
+
+## 3. 使用TensorRT-LLM Python Runtime进行推理
+
+#### 使用我们提供的python代码类，启动单机单卡服务
+```bash
+python atom_inference.py \
+    /model/tensorrt_llm/1 \   # 第一个参数 build.py 的output路径
+    /origin_model \          # 第二个参数模型tokenizer的路径
+    如何成为一个更加优秀的人    # 希望问的问题
+```

Llama-Chinese/inference-speed/GPU/TensorRT-LLM_example/atom_inference.py

@@ -0,0 +1,184 @@
+import csv
+import sys
+from pathlib import Path
+
+import numpy as np
+import torch
+from utils import (DEFAULT_HF_MODEL_DIRS, DEFAULT_PROMPT_TEMPLATES,
+                   load_tokenizer, read_model_name, throttle_generator)
+
+import tensorrt_llm
+from tensorrt_llm.logger import logger
+from tensorrt_llm.runtime import PYTHON_BINDINGS, ModelRunner
+
+if PYTHON_BINDINGS:
+    from tensorrt_llm.runtime import ModelRunnerCpp
+
+class AtomTRTApi:
+    def __init__(self,engine_dir,tokenizer_dir,max_input_length=4096):
+        self.runtime_rank = tensorrt_llm.mpi_rank()
+        self.model_name = read_model_name(engine_dir)
+
+        self.tokenizer, self.pad_id, self.end_id = load_tokenizer(
+            tokenizer_dir=tokenizer_dir,
+            tokenizer_type='llama',
+        )
+        self.use_py_session=False
+        if not PYTHON_BINDINGS:
+            logger.warning(
+                "Python bindings of C++ session is unavailable, fallback to Python session."
+            )
+            self.use_py_session = True
+        runner_cls = ModelRunner if self.use_py_session else ModelRunnerCpp
+        runner_kwargs = dict(engine_dir=engine_dir,
+                            lora_dir=None,
+                            rank=self.runtime_rank,
+                            debug_mode=False,
+                            lora_ckpt_source='hf')
+        
+        if not self.use_py_session:
+            runner_kwargs.update(
+                max_batch_size=1,
+                max_input_len=max_input_length,
+                max_output_len=2048,
+                max_beam_width=1,
+                max_attention_window_size=None)
+        self.runner = runner_cls.from_dir(**runner_kwargs)
+
+
+    def ask(self,input_text,temperature=0.4,top_p=0.95,max_new_tokens=1024,repetition_penalty=1.2,system_prefix = '',merge_lambda=None,max_input_length=4096,append_next_role=True):
+        with torch.no_grad():
+            prompt = ''
+            print('max_input_length',max_input_length)
+            if type(input_text)==list:
+                for input_text_one in input_text[::-1]:
+                    if len(prompt) + len("<s>"+input_text_one['role']+": "+input_text_one['content'].strip()+"\n</s>")<max_input_length:
+                        prompt = "<s>"+input_text_one['role']+": "+input_text_one['content'].strip()+"\n</s>" + prompt
+                if append_next_role:
+                    if input_text[-1]['role']=='Human':
+                        prompt += "<s>Assistant:"
+                    else:
+                        prompt += "<s>Human:"
+            else:
+                if merge_lambda is None:
+                    if append_next_role:
+                        prompt +=  "<s>Human: "+input_text.strip()+"\n</s><s>Assistant:"
+                    else:
+                        prompt +=  "<s>Human: "+input_text.strip()+"\n</s>"
+                else:
+                    prompt +=  merge_lambda(input_text)
+            if len(system_prefix)>0:
+                prompt = '<s>System: '+system_prefix.strip()+'\n</s>'+prompt
+            print('输入模型的完整输入:',prompt)
+            input_ids = [self.tokenizer(prompt,add_special_tokens=False).input_ids]
+            print(input_ids)
+            input_ids = [
+                torch.tensor(x, dtype=torch.int32).unsqueeze(0) for x in input_ids
+            ]
+            print('输入模型的token数量',input_ids[0].shape)
+            generate_input = {
+                "batch_input_ids":input_ids,
+                "max_new_tokens":max_new_tokens,
+                "max_attention_window_size":None,
+                "do_sample":True,
+                "top_k":50,
+                "top_p":top_p,
+                "num_beams":1,
+                "length_penalty":1.0,
+                "stop_words_list":None,
+                "bad_words_list":None,
+                "streaming":False,
+                "temperature":temperature,
+                "output_sequence_lengths":True,
+                "return_dict":False,
+                "repetition_penalty":repetition_penalty,
+                "end_id":self.tokenizer.eos_token_id,
+                "bos_token_id":self.tokenizer.bos_token_id,
+                "pad_id":self.tokenizer.pad_token_id
+            }
+            generate_ids = self.runner.generate(**generate_input)
+            torch.cuda.synchronize()
+            print(generate_ids)
+            generate_ids = generate_ids.cpu().tolist()
+            generate_ids = [item[0][len(input_ids[0][0]):] for  item in generate_ids]
+            try:
+                generate_ids = [item[:item.index(self.tokenizer.eos_token_id)] for  item in generate_ids ]
+            except:
+                pass
+            print(generate_ids)
+            # output = ''.join(tokenizer.convert_ids_to_tokens(generate_ids[0]))
+            # print('生成的token长度',len(generate_ids[0]))
+            bot_message = self.tokenizer.batch_decode(generate_ids, skip_special_tokens=False, clean_up_tokenization_spaces=False)[0]
+            if 'Human:' in bot_message:
+                bot_message = bot_message.split('Human:')[0]
+            print(bot_message)
+            return bot_message.strip()
+        
+    def ask_streaming(self,input_text,temperature=0.8,top_p=0.95,max_new_tokens=1024,repetition_penalty=1.2,system_prefix = '',max_input_length=4096,append_next_role=True):
+        with torch.no_grad():
+            prompt = ''
+            print('max_input_length',max_input_length)
+            if type(input_text)==list:
+                for input_text_one in input_text[::-1]:
+                    if len(prompt) + len("<s>"+input_text_one['role']+": "+input_text_one['content'].strip()+"\n</s>")<max_input_length:
+                        prompt = "<s>"+input_text_one['role']+": "+input_text_one['content'].strip()+"\n</s>" + prompt
+                if append_next_role:
+                    if input_text[-1]['role']=='Human':
+                        prompt += "<s>Assistant:"
+                    else:
+                        prompt += "<s>Human:"
+            else:
+                if append_next_role:
+                    prompt +=  "<s>Human: "+input_text.strip()+"\n</s><s>Assistant:"
+                else:
+                    prompt +=  "<s>Human: "+input_text.strip()+"\n</s>"
+            if len(system_prefix)>0:
+                prompt = '<s>System: '+system_prefix.strip()+'\n</s>'+prompt
+            print('输入模型的完整输入:',prompt)
+            input_ids = [self.tokenizer(prompt,add_special_tokens=False).input_ids]
+            print(input_ids)
+            input_ids = [
+                torch.tensor(x, dtype=torch.int32).unsqueeze(0) for x in input_ids
+            ]
+            print('输入模型的token数量',input_ids[0].shape)
+            generate_input = {
+                "batch_input_ids":input_ids,
+                "max_new_tokens":max_new_tokens,
+                "max_attention_window_size":None,
+                "do_sample":True,
+                "top_k":50,
+                "top_p":top_p,
+                "num_beams":1,
+                "length_penalty":1.0,
+                "stop_words_list":None,
+                "bad_words_list":None,
+                "streaming":True,
+                "temperature":temperature,
+                "output_sequence_lengths":True,
+                "return_dict":True,
+                "repetition_penalty":repetition_penalty,
+                "end_id":self.tokenizer.eos_token_id,
+                "bos_token_id":self.tokenizer.bos_token_id,
+                "pad_id":self.tokenizer.pad_token_id
+            }
+            generate_ids = self.runner.generate(**generate_input)
+            torch.cuda.synchronize()
+            
+            input_token_num = len(input_ids[0][0])
+            answer_message =''
+            for curr_outputs in throttle_generator(generate_ids,2):
+                output_ids = curr_outputs['output_ids']
+                sequence_lengths = curr_outputs['sequence_lengths']
+                # print(sequence_lengths)
+                output_ids = output_ids.cpu().tolist()
+                output_ids = [item[0][input_token_num:sequence_lengths[0][0]] for  item in output_ids]
+                answer_message = self.tokenizer.batch_decode(output_ids, skip_special_tokens=False, clean_up_tokenization_spaces=False)[0]
+                if 'Human:' in answer_message:
+                    answer_message = answer_message.split('Human:')[0]                
+                yield answer_message.strip()
+            return answer_message.strip()
+            
+            
+if __name__=='__main__':
+    model = AtomTRTApi(engine_dir=sys.argv[1],tokenizer_dir=sys.argv[2])
+    model.ask('如何成为一个更优秀的人')

Llama-Chinese/inference-speed/GPU/TensorRT-LLM_example/utils.py

@@ -0,0 +1,130 @@
+# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import json
+from pathlib import Path
+from typing import Optional
+from typing import Union
+
+from transformers import AutoTokenizer, T5Tokenizer
+
+import tensorrt_llm
+
+DEFAULT_HF_MODEL_DIRS = {
+    'baichuan': 'baichuan-inc/Baichuan-13B-Chat',
+    'bloom': 'bigscience/bloom-560m',
+    'chatglm_6b': 'THUDM/chatglm-6b',
+    'chatglm2_6b': 'THUDM/chatglm2-6b',
+    'chatglm2_6b_32k': 'THUDM/chatglm2-6b-32k',
+    'chatglm3_6b': 'THUDM/chatglm3-6b',
+    'chatglm3_6b_base': 'THUDM/chatglm3-6b-base',
+    'chatglm3_6b_32k': 'THUDM/chatglm3-6b-32k',
+    'falcon': 'tiiuae/falcon-rw-1b',
+    'glm_10b': 'THUDM/glm-10b',
+    'gpt': 'gpt2-medium',
+    'gptj': 'EleutherAI/gpt-j-6b',
+    'gptneox': 'EleutherAI/gpt-neox-20b',
+    'internlm': 'internlm/internlm-chat-7b',
+    'llama': 'meta-llama/Llama-2-7b-hf',
+    'mpt': 'mosaicml/mpt-7b',
+    'phi': 'microsoft/phi-2',
+    'opt': 'facebook/opt-350m',
+    'qwen': 'Qwen/Qwen-7B',
+}
+
+DEFAULT_PROMPT_TEMPLATES = {
+    'internlm':
+    "<|User|>:{input_text}<eoh>\n<|Bot|>:",
+    'qwen':
+    "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n{input_text}<|im_end|>\n<|im_start|>assistant\n",
+}
+
+def get_engine_version(engine_dir: str) -> Union[None, str]:
+    engine_dir = Path(engine_dir)
+    config_path = engine_dir / "config.json"
+    with open(config_path, 'r') as f:
+        config = json.load(f)
+
+    if 'version' not in config:
+        return None
+
+    return config['version']
+
+def read_model_name(engine_dir: str):
+    engine_version = get_engine_version(engine_dir)
+
+    with open(Path(engine_dir) / "config.json", 'r') as f:
+        config = json.load(f)
+
+    if engine_version is None:
+        return config['builder_config']['name']
+
+    return config['pretrained_config']['architecture']
+
+
+def throttle_generator(generator, stream_interval):
+    for i, out in enumerate(generator):
+        if not i % stream_interval:
+            yield out
+
+    if i % stream_interval:
+        yield out
+
+
+def load_tokenizer(tokenizer_dir: Optional[str] = None,
+                   vocab_file: Optional[str] = None,
+                   model_name: str = 'gpt',
+                   tokenizer_type: Optional[str] = None):
+    if vocab_file is None:
+        use_fast = True
+        if tokenizer_type is not None and tokenizer_type == "llama":
+            use_fast = False
+        # Should set both padding_side and truncation_side to be 'left'
+        tokenizer = AutoTokenizer.from_pretrained(tokenizer_dir,
+                                                  legacy=False,
+                                                  padding_side='left',
+                                                  truncation_side='left',
+                                                  trust_remote_code=True,
+                                                  tokenizer_type=tokenizer_type,
+                                                  use_fast=use_fast)
+    else:
+        # For gpt-next, directly load from tokenizer.model
+        assert model_name == 'gpt'
+        tokenizer = T5Tokenizer(vocab_file=vocab_file,
+                                padding_side='left',
+                                truncation_side='left')
+
+    if model_name == 'qwen':
+        with open(Path(tokenizer_dir) / "generation_config.json") as f:
+            gen_config = json.load(f)
+        chat_format = gen_config['chat_format']
+        if chat_format == 'raw':
+            pad_id = gen_config['pad_token_id']
+            end_id = gen_config['eos_token_id']
+        elif chat_format == 'chatml':
+            pad_id = tokenizer.im_end_id
+            end_id = tokenizer.im_end_id
+        else:
+            raise Exception(f"unknown chat format: {chat_format}")
+    elif model_name == 'glm_10b':
+        pad_id = tokenizer.pad_token_id
+        end_id = tokenizer.eop_token_id
+    else:
+        if tokenizer.pad_token_id is None:
+            tokenizer.pad_token_id = tokenizer.eos_token_id
+        pad_id = tokenizer.pad_token_id
+        end_id = tokenizer.eos_token_id
+
+    return tokenizer, pad_id, end_id

Llama-Chinese/inference-speed/GPU/lmdeploy_example/README.md

@@ -0,0 +1,126 @@
+#  lmdeploy 安装和使用
+
+lmdeploy 支持 transformer 结构（例如 Atom、LLaMA、LLaMa2、InternLM、Vicuna 等），目前支持 fp16，int8 和 int4。
+
+## 一、安装
+
+安装预编译的 python 包
+```
+python3 -m pip install lmdeploy==0.2.1
+```
+
+## 二、转换huggingface模型为lmdeploy格式
+
+把模型转成 lmdeploy 推理格式，假设 huggingface 版 [Atom-7B-Chat](https://huggingface.co/FlagAlpha/Atom-7B-Chat) 模型已下载到 `/models/Atom-7B-Chat` 目录，结果会存到 当前执行命令的`workspace` 文件夹
+
+```shell
+lmdeploy convert llama2 /models/Atom-7B-Chat
+```
+lmdeploy 修改一处bug
+```
+sed -i 's/from .utils import get_logger/from transformers.utils.logging import get_logger/g' ./workspace/model_repository/preprocessing/1/tokenizer/tokenizer.py
+sed -i 's/from .utils import get_logger/from transformers.utils.logging import get_logger/g' ./workspace/model_repository/postprocessing/1/tokenizer/tokenizer.py
+```
+
+
+## 三、kv cache int8 量化
+对于最大长度是 2048 的 Atom-7B fp16 模型，服务端每创建 1 个并发，都需要大约 1030MB 显存保存 kv_cache，即便是 A100 80G，能服务的用户也非常有限。
+为了降低运行时显存，lmdeploy 实现了 kv cache PTQ 量化，同样的显存可以服务更多并发用户。
+首先计算模型参数，保存到临时目录 atom
+```shell
+mkdir atom
+lmdeploy lite calibrate \
+  /models/Atom-7B-Chat  \             # huggingface Atom 模型。也支持 llama/vicuna/internlm/baichuan 等
+  --calib-dataset 'ptb' \             # 校准数据集，支持 c4, ptb, wikitext2, pileval
+  --calib-samples 128   \             # 校准集的样本数，如果显存不够，可以适当调小
+  --device 'cuda'       \             # 单条的文本长度，如果显存不够，可以适当调小
+  --work-dir atom                     # 保存 pth 格式量化统计参数和量化后权重的文件夹
+```
+注意：可能需要安装flash_attn
+```shell
+conda install -c nvidia cuda-nvcc # 为了使用conda内的cuda环境安装 flash_attn
+pip install flash_attn
+```
+
+
+然后用 atom 目录里的参数，计算量化参数，保存到转换后参数到 `workspace/triton_models/weights` 下
+
+```shell
+lmdeploy lite kv_qparams                 \ 
+  ./atom                                 \  # 上一步计算的 atom 结果
+  ./workspace/triton_models/weights      \  # 结果保存目录
+  --num-tp 1                                # tensor parallel GPU 个数
+```
+
+修改推理配置，开启 kv cache int8。编辑 `workspace/triton_models/weights/config.ini` 
+* 把 `use_context_fmha` 改为 0，表示关闭 flashattention
+* 把 `quant_policy` 设为 4，表示打开 kv cache 量化
+
+最终执行测试即可
+```shell
+lmdeploy chat turbomind ./workspace
+```
+
+[点击这里](https://github.com/InternLM/lmdeploy/blob/main/docs/zh_cn/kv_int8.md) 查看 kv cache int8 量化实现公式、精度和显存测试报告。
+
+## 四、weight int4 量化
+
+lmdeploy 基于 [AWQ 算法](https://arxiv.org/abs/2306.00978) 实现了 weight int4 量化，性能是 FP16 的 2.4 倍以上。显存从 16G 降低到 6.3G。
+
+对于自己的模型，可以用`auto_awq`工具来优化
+```shell
+# 指定量化导出的模型路径
+WORK_DIR="./atom-7b-chta-w4"
+
+lmdeploy lite auto_awq \
+$HF_MODEL              \  # huggingface 模型位置
+--calib-dataset 'ptb'  \  # 校准数据集，支持 c4, ptb, wikitext2, pileval
+--calib-samples 128    \  # 校准集的样本数，如果显存不够，可以适当调小
+--calib-seqlen 2048    \  # 单条的文本长度，如果显存不够，可以适当调小  
+--w-bits 4             \  # 权重量化的 bit 数
+--w-group-size 128     \  # 权重量化分组统计尺寸
+--work-dir $WORK_DIR  
+```
+
+执行以下命令，启动服务：
+```shell
+# 这里的路径是上面步骤一中转换模型的layout的输出
+FasterTransformer_PATH="/path/workspace"
+
+TP=1
+# 指定需要用的显卡
+DEVICES="0"
+for ((i = 1; i < ${TP}; ++i)); do
+    DEVICES="${DEVICES},$i"
+done
+DEVICES="\"device=${DEVICES}\""
+
+# 在容器内启动服务
+docker run -idt \
+        --gpus $DEVICES \
+        -v $FasterTransformer_PATH:/workspace/models \
+        --shm-size 16g \
+        -p 33336:22 \
+        -p 33337-33400:33337-33400 \
+        --cap-add=SYS_PTRACE \
+        --cap-add=SYS_ADMIN \
+        --security-opt seccomp=unconfined \
+        --name lmdeploy \
+        --env NCCL_LAUNCH_MODE=GROUP openmmlab/lmdeploy:latest \
+        tritonserver \
+        --model-repository=/workspace/models/model_repository \
+        --allow-http=0 \
+        --allow-grpc=1 \
+        --grpc-port=33337 \
+        --log-verbose=0 \
+        --allow-metrics=1
+```
+
+客户端测试：
+```shell
+python test_api_server.py  --tritonserver_addr 127.0.0.1:33337
+```
+
+[点击这里](https://github.com/InternLM/lmdeploy/blob/main/docs/zh_cn/w4a16.md) 查看 weight int4 量化的显存和速度测试结果。
+
+额外说明，weight int4 和 kv cache int8 二者并不��突、可以同时打开，节约更多显存。

Llama-Chinese/inference-speed/GPU/lmdeploy_example/test_api_server.py

@@ -0,0 +1,73 @@
+import os
+import time
+from lmdeploy.serve.turbomind.chatbot import Chatbot
+
+def input_prompt(chat_history, system_prompt: str):
+    """Input a prompt in the consolo interface."""
+    prompt = ''
+    for input_text_one in chat_history:
+            prompt += "<s>"+input_text_one['role']+": "+input_text_one['content'].strip()+"\n</s>"
+    if chat_history[-1]['role']=='Human':
+        prompt += "<s>Assistant: "
+    else:
+        prompt += "<s>Human: "
+    prompt = prompt[-2048:]
+    if len(system_prompt)>0:
+        prompt = '<s>System: '+system_prompt.strip()+'\n</s>'+prompt
+                
+    return prompt
+
+def main(tritonserver_addr: str,
+         session_id: int = 1,
+         cap: str = 'chat',
+         stream_output: bool = True,
+         **kwargs):
+    """An example to communicate with inference server through the command line
+    interface.
+
+    Args:
+        tritonserver_addr (str): the address in format "ip:port" of
+          triton inference server
+        session_id (int): the identical id of a session
+        cap (str): the capability of a model. For example, codellama has
+            the ability among ['completion', 'infill', 'instruct', 'python']
+        stream_output (bool): indicator for streaming output or not
+        **kwargs (dict): other arguments for initializing model's chat template
+    """
+    log_level = os.environ.get('SERVICE_LOG_LEVEL', 'WARNING')
+    kwargs.update(capability=cap)
+    chatbot = Chatbot(tritonserver_addr,
+                      log_level=log_level,
+                      display=stream_output,
+                      **kwargs)
+    nth_round = 1
+    prompt = input_prompt([{"role": "Human", "content" : "心情不好怎么办"}], "")
+
+    request_id = f'{session_id}-{nth_round}'
+    begin = time.time()
+    if stream_output:
+        for status, res, n_token in chatbot.stream_infer(
+                session_id,
+                prompt,
+                request_id=request_id,
+                request_output_len=512):
+            # print("n_token:", n_token)
+            continue
+            
+    else:
+        status, res, n_token = chatbot.infer(session_id,
+                                                prompt,
+                                                request_id=request_id,
+                                                request_output_len=512)
+        print(res)
+        # print("n_token:", n_token)
+    nth_round += 1
+    end = time.time()
+    speed = n_token/(end-begin)
+    print("speed {} tokens/s".format(speed))
+    
+
+if __name__ == '__main__':
+    import fire
+
+    fire.Fire(main)

Llama-Chinese/inference-speed/GPU/vllm_example/README.md

@@ -0,0 +1,57 @@
+# vllm推理部署
+
+[vllm](https://github.com/vllm-project/vllm)同样是GPU推理的方案。相比较与FasterTrainsformer，vllm更加的简单易用。不需要额外进行模型的转换。支持fp16推理。
+
+特点：
+
++ 快速的推理速度
++ 高效的kv cache
++ 连续的batch请求推理
++ 优化cuda算子
++ 支持分布式推理
+
+## 第一步： 安装vllm
+
+```bash
+pip install vllm
+```
+
+## 第二步：启动测试server
+
+从Huggingface下载Atom或者LLama3模型：
+```
+# 您可以选择具体想部署的模型下载
+git clone https://huggingface.co/FlagAlpha/Atom-7B-Chat  Atom-7B-Chat
+
+# 或者下载Meta官方的Llama3模型：
+git clone https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct Meta-Llama-3-8B-Instruct
+```
+
+1. 单卡推理
+
+编辑single_gpus_api_server.sh里面model为上面模型的下载路径。
+
+启动测试server
+```bash
+# multi_gpus_api_server.sh 里面的CUDA_VISIBLE_DEVICES指定了要使用的GPU卡
+bash single_gpus_api_server.sh
+```
+
+2. 多卡推理
+
+13B模型，70B模型推荐多卡推理。编辑multi_gpus_api_server.sh里面model为上面的13B模型的下载路径。
+
+启动测试server
+```bash
+# multi_gpus_api_server.sh 里面的CUDA_VISIBLE_DEVICES指定了要使用的GPU卡
+# tensor-parallel-size 指定了卡的个数
+bash multi_gpus_api_server.sh
+```
+
+## 第三步：启动client测试
+
+注意下面的model_source 模型的源，可以是 llama_chinese、llama2_meta、llama3_meta 根据下载的模型不同去区分，如果下载的是[FlagAlpha](https://huggingface.co/FlagAlpha)下载的则用llama_chinese。
+
+```
+python client_test.py --model_source llama_chinese
+```

Llama-Chinese/inference-speed/GPU/vllm_example/api_server.py

@@ -0,0 +1,85 @@
+import argparse
+import json
+from typing import AsyncGenerator
+
+from fastapi import BackgroundTasks, FastAPI, Request
+from fastapi.responses import JSONResponse, Response, StreamingResponse
+import uvicorn
+
+from vllm.engine.arg_utils import AsyncEngineArgs
+from vllm.engine.async_llm_engine import AsyncLLMEngine
+from vllm.sampling_params import SamplingParams
+from vllm.utils import random_uuid
+
+TIMEOUT_KEEP_ALIVE = 5  # seconds.
+TIMEOUT_TO_PREVENT_DEADLOCK = 1  # seconds.
+app = FastAPI()
+
+
+@app.post("/generate")
+async def generate(request: Request) -> Response:
+    """Generate completion for the request.
+
+    The request should be a JSON object with the following fields:
+    - prompt: the prompt to use for the generation.
+    - stream: whether to stream the results or not.
+    - other fields: the sampling parameters (See `SamplingParams` for details).
+    """
+    request_dict = await request.json()
+    prompt = request_dict.pop("prompt")
+    stream = request_dict.pop("stream", False)
+    sampling_params = SamplingParams(**request_dict)
+    request_id = random_uuid()
+    results_generator = engine.generate(prompt, sampling_params, request_id)
+
+    # Streaming case
+    async def stream_results() -> AsyncGenerator[bytes, None]:
+        async for request_output in results_generator:
+            prompt = request_output.prompt
+            text_outputs = [
+                prompt + output.text for output in request_output.outputs
+            ]
+            ret = {"text": text_outputs}
+            yield (json.dumps(ret) + "\0").encode("utf-8")
+
+    async def abort_request() -> None:
+        await engine.abort(request_id)
+
+    if stream:
+        background_tasks = BackgroundTasks()
+        # Abort the request if the client disconnects.
+        background_tasks.add_task(abort_request)
+        return StreamingResponse(stream_results(), background=background_tasks)
+
+    # Non-streaming case
+    final_output = None
+    async for request_output in results_generator:
+        if await request.is_disconnected():
+            # Abort the request if the client disconnects.
+            await engine.abort(request_id)
+            return Response(status_code=499)
+        final_output = request_output
+
+    assert final_output is not None
+    prompt = final_output.prompt
+    text_outputs = [prompt + output.text for output in final_output.outputs]
+    ret = {"text": text_outputs}
+    return JSONResponse(ret)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--host", type=str, default="0.0.0.0")
+    parser.add_argument("--port", type=int, default=8090)
+    parser.add_argument("--trust_remote_code", type=bool, default=True)
+    parser = AsyncEngineArgs.add_cli_args(parser)
+    args = parser.parse_args()
+
+    engine_args = AsyncEngineArgs.from_cli_args(args)
+    engine = AsyncLLMEngine.from_engine_args(engine_args)
+
+    uvicorn.run(app,
+                host=args.host,
+                port=args.port,
+                log_level="debug",
+                timeout_keep_alive=TIMEOUT_KEEP_ALIVE)

Llama-Chinese/inference-speed/GPU/vllm_example/client_test.py

@@ -0,0 +1,137 @@
+# coding=utf-8
+import json
+import time
+import argparse
+
+import urllib.request
+
+import sys
+
+parser = argparse.ArgumentParser()
+parser.add_argument('--model_source', default="llama_chinese", choices =["llama_chinese", "llama2_meta", "llama3_meta"], required=False,type=str)
+args = parser.parse_args()
+
+def get_prompt_llama_chinese(
+    chat_history, system_prompt=""
+) -> str:
+    prompt = ''
+    for input_text_one in chat_history:
+            prompt += "<s>"+input_text_one['role']+": "+input_text_one['content'].strip()+"\n</s>"
+    if chat_history[-1]['role']=='Human':
+        prompt += "<s>Assistant: "
+    else:
+        prompt += "<s>Human: "
+    prompt = prompt[-2048:]
+    if len(system_prompt)>0:
+        prompt = '<s>System: '+system_prompt.strip()+'\n</s>'+prompt
+                
+    return prompt
+
+def get_prompt_llama2_meta(chat_history, system_prompt=""):
+    B_INST, E_INST = "[INST]", "[/INST]"
+    B_SYS, E_SYS = "<<SYS>>\n", "\n<</SYS>>\n\n"
+    
+    sep = " "
+    sep2 =" </s><s>"
+    stop_token_ids = [2]
+    system_template = f"[INST] <<SYS>>\n{system_prompt}\n<</SYS>>\n\n"
+    roles = ("[INST]", "[/INST]")
+    seps = [sep, sep2]
+    if system_prompt.strip() != "":
+        ret = system_template
+    else:
+        ret = "[INST] "
+    for i, chat in enumerate(chat_history):
+        message = chat["content"]
+        role = chat["role"]
+        if message:
+            if i == 0:
+                ret += message + " "
+            else:
+                if role == "Human":
+                    ret +=  "[INST]" + " " + message + seps[i % 2]
+                else:
+                    ret +=  "[/INST]" + " " + message + seps[i % 2]
+        else:
+            if role == "Human":
+                ret += "[INST]"
+            else:
+                ret += "[/INST]"
+    print("prompt:{}".format(ret))
+    return ret
+
+def get_prompt_llama3_meta(chat_history, system_prompt=""):
+    system_format='<|start_header_id|>system<|end_header_id|>\n\n{content}<|eot_id|>'
+    user_format='<|start_header_id|>user<|end_header_id|>\n\n{content}<|eot_id|>'
+    assistant_format='<|start_header_id|>assistant<|end_header_id|>\n\n{content}<|eot_id|>\n'
+    prompt_str = ''
+    # 拼接历史对话
+    for item in chat_history:
+        if item['role']=='Human':
+            prompt_str+=user_format.format(content=item['content'])
+        else:
+            prompt_str+=assistant_format.format(content=item['content'])
+    if len(system_prompt)>0:
+        prompt_str = system_format.format(content=system_prompt) + prompt_str
+    prompt_str = "<|begin_of_text|>" + prompt_str
+    return prompt_str
+
+
+def test_api_server(chat_history=[], system_prompt=""):
+    header = {'Content-Type': 'application/json'}
+
+    if args.model_source == "llama2_meta":
+        prompt = get_prompt_llama2_meta(chat_history, system_prompt)
+    elif args.model_source == "llama3_meta":
+        prompt = get_prompt_llama3_meta(chat_history, system_prompt)
+    else:
+        prompt = get_prompt_llama_chinese(chat_history, system_prompt)
+
+    data = {
+          "prompt": prompt,
+          "stream" : False,
+          "n" : 1,
+          "best_of": 1, 
+          "presence_penalty": 0.0, 
+          "frequency_penalty": 0.2, 
+          "temperature": 0.3, 
+          "top_p" : 0.95, 
+          "top_k": 50, 
+          "use_beam_search": False, 
+          "stop": [], 
+          "ignore_eos" :False, 
+          "max_tokens": 2048, 
+          "logprobs": None
+    }
+    request = urllib.request.Request(
+        url='http://127.0.0.1:8090/generate',
+        headers=header,
+        data=json.dumps(data).encode('utf-8')
+    )
+
+    result = None
+    try:
+        response = urllib.request.urlopen(request, timeout=300)
+        res = response.read().decode('utf-8')
+        result = json.loads(res)
+        print(json.dumps(data, ensure_ascii=False, indent=2))
+        print(json.dumps(result, ensure_ascii=False, indent=2))
+
+    except Exception as e:
+        print(e)
+
+    return result
+
+if __name__ == "__main__":
+    # 多伦对话测试
+    """ 多伦对话测试
+        last_question = "怎么回来呢"
+        inputs = [{"role": "Human", "content": "如何去北京"}, 
+                {"role": "Assitant", "content": "乘坐飞机或者轮船"}, 
+                {"role" : "Human", "content": last_question}]
+    """
+    # 单轮对话  
+    last_question = "怎么去北京"
+    chat_history = [ {"role" : "Human", "content": last_question}]
+    test_api_server(chat_history)
+

Llama-Chinese/inference-speed/GPU/vllm_example/multi_gpus_api_server.sh

@@ -0,0 +1,4 @@
+CUDA_VISIBLE_DEVICES=0,1 python api_server.py \
+--model "./Atom-7B-Chat" \
+--port 8090 \
+--tensor-parallel-size 2

Llama-Chinese/inference-speed/GPU/vllm_example/single_gpu_api_server.sh

@@ -0,0 +1,4 @@
+
+CUDA_VISIBLE_DEVICES=0 python api_server.py \
+--model "./Atom-7B-Chat" \
+--port 8090

Llama-Chinese/model/Atom-7B-Chat-pre/.gitattributes

@@ -0,0 +1,2 @@
+model-00001-of-00002.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00002-of-00002.safetensors filter=lfs diff=lfs merge=lfs -text

Llama-Chinese/model/Atom-7B-Chat-pre/config.json

@@ -0,0 +1,51 @@
+{
+  "_name_or_path": "FlagAlpha/Atom-7B-Chat",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "FlagAlpha/Atom-7B-Chat--configuration_atom.LlamaConfig",
+    "AutoModel": "FlagAlpha/Atom-7B-Chat--model_atom.LlamaForCausalLM",
+    "AutoModelForCausalLM": "FlagAlpha/Atom-7B-Chat--model_atom.LlamaForCausalLM",
+    "AutoModelForSequenceClassification": "FlagAlpha/Atom-7B-Chat--model_atom.LlamaForSequenceClassification"
+  },
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 11008,
+  "max_length": 4096,
+  "max_position_embeddings": 4096,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 32,
+  "pad_token_id": 2,
+  "pretraining_tp": 1,
+  "quantization_config": {
+    "_load_in_4bit": false,
+    "_load_in_8bit": true,
+    "bnb_4bit_compute_dtype": "float32",
+    "bnb_4bit_quant_storage": "uint8",
+    "bnb_4bit_quant_type": "fp4",
+    "bnb_4bit_use_double_quant": false,
+    "llm_int8_enable_fp32_cpu_offload": false,
+    "llm_int8_has_fp16_weight": false,
+    "llm_int8_skip_modules": null,
+    "llm_int8_threshold": 6.0,
+    "load_in_4bit": false,
+    "load_in_8bit": true,
+    "quant_method": "bitsandbytes"
+  },
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 500000,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.39.0",
+  "use_cache": true,
+  "vocab_size": 65000
+}

Llama-Chinese/model/Atom-7B-Chat-pre/generation_config.json

@@ -0,0 +1,8 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "max_length": 4096,
+  "pad_token_id": 2,
+  "transformers_version": "4.39.0"
+}

Llama-Chinese/model/Atom-7B-Chat-pre/model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37de609a2beb1671bd5f8b921144769b332e8e1c5d6630b6c10dab985615458a
+size 4988872000

Llama-Chinese/model/Atom-7B-Chat-pre/model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7843f6c0984f493e2d18e58c15791ba55e8c70678f059e83358439b47fba9e6f
+size 2558121080

Llama-Chinese/model/Atom-7B-Chat-pre/model.safetensors.index.json

@@ -0,0 +1,522 @@
+{
+  "metadata": {
+    "total_size": 7546937344
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00002-of-00002.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.28.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.28.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.28.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.28.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.28.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.28.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.28.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.28.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.28.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.29.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.29.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.30.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.30.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.mlp.down_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.mlp.gate_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.31.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.mlp.up_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.31.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.k_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.o_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.q_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.v_proj.SCB": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.down_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.gate_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.up_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.o_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.v_proj.SCB": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.norm.weight": "model-00002-of-00002.safetensors"
+  }
+}

Llama-Chinese/output/train.log

@@ -0,0 +1 @@
+[2024-08-14 09:27:03,273] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)

Llama-Chinese/quick_start.py

@@ -0,0 +1,26 @@
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+device_map = "cuda:0" if torch.cuda.is_available() else "auto"
+model = AutoModelForCausalLM.from_pretrained('FlagAlpha/Atom-7B-Chat',device_map=device_map,torch_dtype=torch.float16,load_in_8bit=True,trust_remote_code=True,use_flash_attention_2=True)
+model =model.eval()
+tokenizer = AutoTokenizer.from_pretrained('FlagAlpha/Atom-7B-Chat',use_fast=False)
+tokenizer.pad_token = tokenizer.eos_token
+input_ids = tokenizer(['<s>Human: 介绍一下中国\n</s><s>Assistant: '], return_tensors="pt",add_special_tokens=False).input_ids
+if torch.cuda.is_available():
+  input_ids = input_ids.to('cuda')
+generate_input = {
+    "input_ids":input_ids,
+    "max_new_tokens":512,
+    "do_sample":True,
+    "top_k":50,
+    "top_p":0.95,
+    "temperature":0.3,
+    "repetition_penalty":1.3,
+    "eos_token_id":tokenizer.eos_token_id,
+    "bos_token_id":tokenizer.bos_token_id,
+    "pad_token_id":tokenizer.pad_token_id
+}
+generate_ids  = model.generate(**generate_input)
+text = tokenizer.decode(generate_ids[0])
+print(text)
+model.save_pretrained('model/Atom-7B-Chat-pre')