Instructions to use kazcfz/Llama-2-7b-chat-MedFAQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use kazcfz/Llama-2-7b-chat-MedFAQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="kazcfz/Llama-2-7b-chat-MedFAQ")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("kazcfz/Llama-2-7b-chat-MedFAQ")
model = AutoModelForCausalLM.from_pretrained("kazcfz/Llama-2-7b-chat-MedFAQ", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use kazcfz/Llama-2-7b-chat-MedFAQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kazcfz/Llama-2-7b-chat-MedFAQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kazcfz/Llama-2-7b-chat-MedFAQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/kazcfz/Llama-2-7b-chat-MedFAQ

SGLang

How to use kazcfz/Llama-2-7b-chat-MedFAQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "kazcfz/Llama-2-7b-chat-MedFAQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kazcfz/Llama-2-7b-chat-MedFAQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "kazcfz/Llama-2-7b-chat-MedFAQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kazcfz/Llama-2-7b-chat-MedFAQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use kazcfz/Llama-2-7b-chat-MedFAQ with Docker Model Runner:
```
docker model run hf.co/kazcfz/Llama-2-7b-chat-MedFAQ
```

YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

Llama-2-7b-chat-MedFAQ

NousResearch/Llama-2-7b-chat-hf fine-tuned on kazcfz/MedFAQ-llama2-2k

QLoRA parameters

Parameter	Value
lora_r	64
lora_alpha	16
lora_dropout	0.1

bitsandbytes parameters

Parameter	Value
use_4bit	True
bnb_4bit_compute_dtype	"float16"
bnb_4bit_quant_type	"nf4"
use_nested_quant	False

TrainingArguments parameters

Parameter	Value
num_train_epochs	1
fp16	False
bf16	False
per_device_train_batch_size	4
gradient_accumulation_steps	1
gradient_checkpointing	True
max_grad_norm	0.3
learning_rate	2e-4
weight_decay	0.001
optim	"paged_adamw_32bit"
lr_scheduler_type	"cosine"
max_steps	-1
warmup_ratio	0.03
group_by_length	True
save_steps	0
logging_steps	25

SFT parameters

Parameter	Value
max_seq_length	None
packing	False

Prompt template: Llama-2-Chat

<s>[INST] {prompt} [/INST]

Sample (actual) output (Max length: 200)

Base Model

Query: <s>[INST] Am I at risk for kidney disease? [/INST]
Response: I cannot provide medical advice or diagnose any health conditions, including kidney disease. everybody's health is unique, and there are many factors that can increase or decrease the risk of developing kidney disease. However, I can provide some general information on the risk factors for kidney disease and suggest ways to reduce your risk.

Risk factors for kidney disease include:

1. Family history: If you have a family history of kidney disease, you are more likely to develop the condition.
2. Diabetes: Diabetes is a common cause of kidney disease. If you have diabetes, you are at a higher risk of developing kidney disease.
3. High blood pressure: High blood pressure can damage the kidneys and increase the risk of kidney disease.
4. Obesity: Being overweight or obese can increase

Fine-tuned

Query: <s>[INST] Am I at risk for kidney disease? [/INST]
Response: If you have diabetes, you are at risk for kidney disease. If you have high blood pressure, you are also at risk. If you have a family history of kidney disease, you are at risk. If you are African American, Hispanic, or Native American, you are at higher risk. If you are overweight or obese, you are at higher risk. If you have a history of smoking, you are at higher risk. If you have a history of kidney disease, you are at higher risk. Source: NATIONAL KIDNEY FOUNDATION. Link: https://www.kidney.org/help/faq/dialysis-faq. Field: Nephrology. Disease: Kidney Disease. Odom: 601.200.1647.
Response: If you have diabetes, you are at risk for kidney disease. If you have high blood pressure, you are also at risk. If you have a family history of kidney disease, you are at risk. If you are African American, Hispanic, or Native American, you are at higher risk. If you are older than 60, you are at higher risk. Source: NATIONAL KIDNEY FOUNDATION. Link: https://www.kidney.org/help/faq/dialysis-faq. Field: Nephrology. Disease: Kidney Disease. Odom: 60125106.  Source: OLD COURTHOUSE. Link: https://www.kidney.org/help/faq/dialysis-faq.
Response: If you have diabetes, you are at risk for kidney disease. If you have high blood pressure, you are also at risk. If you have a family history of kidney disease, you are at risk. If you are African American, Hispanic, or Native American, you are at higher risk. If you are older than 60, you are at higher risk. Source: NATIONAL KIDNEY FOUNDATION. Link: https://www.kidney.org/help/faq. Field: Nephrology. Disease: Kidney Disease. Odom: 60125106. Complexity: Intermediate. Currency: USD. Type: General Guide. Source Puth: 1. Field Puth: Nephrology. Disease Puth: Kidney

Reload model + Merge LoRA weights?

Query: <s>[INST] Am I at risk for kidney disease? [/INST]
Response: If you have diabetes, you are at risk for developing kidney disease.

If you have high blood pressure, you are at risk for developing kidney disease.

If you have a family history of kidney disease, you are at risk for developing kidney disease.

If you are overweight or obese, you are at risk for developing kidney disease.

If you are African American, Hispanic/Latino, or Native American, you are at higher risk for developing kidney disease.

If you have a history of kidney disease in your family, you are at higher risk for developing kidney disease.

If you have a history of heart disease, you are at higher risk for developing kidney disease.

If you have a history of stroke, you are at higher risk for developing kidney disease.

If you

Downloads last month: 8

Safetensors

Model size

7B params

Tensor type

F32

F16

Collection including kazcfz/Llama-2-7b-chat-MedFAQ

MedFAQ

Collection

LLMs fine-tuned on kazcfz/Llama-2-7b-chat-FAQ • 2 items • Updated Jan 13, 2024