Instructions to use fangloveskari/ORCA_LLaMA_70B_QLoRA with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use fangloveskari/ORCA_LLaMA_70B_QLoRA with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="fangloveskari/ORCA_LLaMA_70B_QLoRA")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("fangloveskari/ORCA_LLaMA_70B_QLoRA")
model = AutoModelForCausalLM.from_pretrained("fangloveskari/ORCA_LLaMA_70B_QLoRA", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use fangloveskari/ORCA_LLaMA_70B_QLoRA with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "fangloveskari/ORCA_LLaMA_70B_QLoRA"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fangloveskari/ORCA_LLaMA_70B_QLoRA",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/fangloveskari/ORCA_LLaMA_70B_QLoRA

SGLang

How to use fangloveskari/ORCA_LLaMA_70B_QLoRA with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "fangloveskari/ORCA_LLaMA_70B_QLoRA" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fangloveskari/ORCA_LLaMA_70B_QLoRA",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "fangloveskari/ORCA_LLaMA_70B_QLoRA" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fangloveskari/ORCA_LLaMA_70B_QLoRA",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use fangloveskari/ORCA_LLaMA_70B_QLoRA with Docker Model Runner:
```
docker model run hf.co/fangloveskari/ORCA_LLaMA_70B_QLoRA
```

May I ask how the your method of merge adapters to base model?

by HCZhang - opened Sep 4, 2023

Discussion

HCZhang

Sep 4, 2023

Sorry to bother you.
This question may not be directly related to your model, but I've been looking around and yet to find a solution.

I've fine-tuned a model using QLORA, and I can't merge the adapters (checkpoint) back to the base model.
I've tried the script provided by The Block, but there were some errors showing layer sizes don't match.

And your model seems works fine, so I wonder how did you merge the model.
Thank you.

Also, your model is really impressive

fangloveskari

Owner Sep 4, 2023

•

edited Sep 4, 2023

Hi,
I tried two ways for fusion:

as I trained the model with https://github.com/hiyouga/LLaMA-Efficient-Tuning, I directly use https://github.com/hiyouga/LLaMA-Efficient-Tuning/blob/main/src/export_model.py to do the fusion job.
or you can use https://github.com/jondurbin/qlora/blob/main/qmerge.py to do it.

I tried the two methods, generally, the second will get better ARC(+0.15) and Truthful_QA(+0.3) scores but the other two(MMLU(-0.2) and HelloSwag(-0.2)) seems to degenerate.

The version for leadboard is generated by the first fusion method.

HCZhang

Sep 4, 2023

Thank you so much, would check the two methods you mentioned.

fangloveskari changed discussion status to closed Sep 4, 2023

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment