Instructions to use Linly-AI/Chinese-LLaMA-2-7B-hf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Linly-AI/Chinese-LLaMA-2-7B-hf with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Linly-AI/Chinese-LLaMA-2-7B-hf")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Linly-AI/Chinese-LLaMA-2-7B-hf")
model = AutoModelForCausalLM.from_pretrained("Linly-AI/Chinese-LLaMA-2-7B-hf")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Linly-AI/Chinese-LLaMA-2-7B-hf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Linly-AI/Chinese-LLaMA-2-7B-hf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Linly-AI/Chinese-LLaMA-2-7B-hf",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Linly-AI/Chinese-LLaMA-2-7B-hf

SGLang

How to use Linly-AI/Chinese-LLaMA-2-7B-hf with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Linly-AI/Chinese-LLaMA-2-7B-hf" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Linly-AI/Chinese-LLaMA-2-7B-hf",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Linly-AI/Chinese-LLaMA-2-7B-hf" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Linly-AI/Chinese-LLaMA-2-7B-hf",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Linly-AI/Chinese-LLaMA-2-7B-hf with Docker Model Runner:
```
docker model run hf.co/Linly-AI/Chinese-LLaMA-2-7B-hf
```

请教扩词表问题

by tomom - opened Jul 24, 2023

Discussion

tomom

Jul 24, 2023

技术博客中关于词表扩增提到：LLaMA-2 沿用了 LLaMA-1 的词表，因此依然面临缺乏中文词的问题。在 Linly-LLaMA-2 中，我们直接扩充了 8076 个常用汉字和标点符号，在模型 embedding 和 target 层使用这些汉字在原始词表中对应 tokens 位置的均值作为初始化。
请问这部分这样直接添加汉字，能提升中文编码效率吗？对模型增量训练稳定性影响大吗？

yuhaofeng-shiba

Linly org Jul 24, 2023

编码效率肯定是可以提升的，之前一个没见过的字可能会被tokenize成3个id，现在会被tokenize成1个，所以训练和推理都会快很多，而且生成的内容也会长一些。
目前从效果来看，增量训练还算稳定。

tomom

Jul 25, 2023

感谢回复~
另外，技术博客中提到的：在模型 embedding 和 target 层使用这些汉字在原始词表中对应 tokens 位置的均值作为初始化。
新添加的汉字其实在原始 tokenizer 可以被编码？新添加的 8076 全是单个汉字，没有词组？

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment