Instructions to use kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1")
model = AutoModelForCausalLM.from_pretrained("kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1

SGLang

How to use kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Studio

How to use kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1 to start chatting

Load model with FastModel

pip install unsloth
from unsloth import FastModel
model, tokenizer = FastModel.from_pretrained(
    model_name="kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1",
    max_seq_length=2048,
)

Docker Model Runner
How to use kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1 with Docker Model Runner:
```
docker model run hf.co/kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Uploaded model

Developed by: kevinkawchak
License: llama3
Finetuned from model : meta-llama/Meta-Llama-3-8B-Instruct
Finetuned using dataset : yahma/alpaca-cleaned

Cover Image. META LLAMA 3 COMMUNITY LICENSE AGREEMENT. Built with Meta Llama 3.

The Meta-Llama-3-8B-Instruct Large Language Model was fine-tuned using the alpaca-cleaned dataset and varying LoRA values. LoRA experiments were run to affect model size and performance, with the top model being uploaded to Hugging Face as a 'Text generation' model: kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1. (a)

Open-source Llama 3 8 Billion parameter model was released April 18, 2024, and it's selection was based on several highly functional existing Llama 2 Medical applications, an increase in Llama 3 downloads by developers, and Meta's growing infrastructure to support Llama improvements. (b-e)

Low-Rank Adaptation, also known as LoRA, makes fine-tuning LLMs easier by reducing the number of trainable parameters to produce lightweight and efficient models. LoRA was utilized by modifying matrix rank 'r' and alpha values. Run times were similar between experiments, however loss tended to favor lower values of rank, which corresponded to smaller model sizes. The most effective 'R1' model mentioned above was trained on a rank of 1 and alpha of 5, which performed better by loss than a rank of up to 5. Further testing of the model uploaded to Hugging Face is relevant to determine what benefits the finetuning has over the base Llama 3 8B Instruct model. (f)

A recent Unsloth Llama 3 Colab Notebook, a Llama 3 discussion, and clarifications from Hugging Chat and ChatGPT allowed for the completion of the finetuned of the model using an NVIDIA 40GB A100. The remainder of the notebook experiments are avaialable on GitHub. (g-i)

The yahma/alpaca-cleaned dataset used is an update to resolve issues regarding the original 2023 Stanford alpaca instruction dataset. Text generations answering several questions were of high quality and comparable to other leading GenAI platforms. Answers to the following questions are available on GitHub notebooks such as 'What is a famous university in San Francisco bay area?', and 'What are the DNA bases?' (j) A look ahead at datasets for Drug Discovery Generative AI will be covered on Thursday 04/25/2024.

a) https://huggingface.co/kevinkawchak/Meta-Llama-3-8B-Instruct-LoRA-R1
b) https://ollama.com/library/medllama2
c) https://github.com/AIAnytime/Llama2-Medical-Chatbot
d) https://github.com/entbappy/End-to-end-Medical-Chatbot-using-Llama2
e) https://engineering.fb.com/2024/03/12/data-center-engineering/building-metas-genai-infrastructure/
f) https://snorkel.ai/lora-low-rank-adaptation-for-llms/
g) https://colab.research.google.com/drive/1mPw6P52cERr93w3CMBiJjocdTnyPiKTX#scrollTo=6bZsfBuZDeCL
h) https://www.youtube.com/watch?v=aQmoog_s8HE&t=1s
i) https://github.com/kevinkawchak/Medical-Quantum-Machine-Learning/tree/main/Code/Hugging%20Face/Llama-3-8B-Instruct
j) https://huggingface.co/datasets/yahma/alpaca-cleaned