Instructions to use QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF", dtype="auto")

llama-cpp-python

How to use QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF",
	filename="Open-Insurance-LLM-Llama3-8B.Q2_K.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF:Q4_K_M

SGLang

How to use QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF with Ollama:
```
ollama run hf.co/QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF:Q4_K_M
```

Unsloth Studio new

How to use QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF to start chatting

Docker Model Runner
How to use QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF with Docker Model Runner:
```
docker model run hf.co/QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF:Q4_K_M
```

Lemonade

How to use QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Open-Insurance-LLM-Llama3-8B-GGUF-Q4_K_M

List all available models

lemonade list

QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF

This is quantized version of Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B created using llama.cpp

Original Model Card

Open-Insurance-LLM-Llama3-8B

This model is a domain-specific language model based on Nvidia Llama 3 ChatQA, fine-tuned for insurance-related queries and conversations. It leverages the architecture of Llama 3 and is specifically trained to handle insurance domain tasks.

Model Details

Model Type: Instruction-tuned Language Model
Base Model: nvidia/Llama3-ChatQA-1.5-8B
Finetuned Model: Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B
Quantized Model: Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B-GGUF
Model Architecture: Llama
Parameters: 8.05 billion
Developer: Raj Maharajwala
License: llama3
Language: English

Quantized Model

Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B-GGUF: https://huggingface.co/Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B-GGUF

Training Data

The model has been fine-tuned on the InsuranceQA dataset using LoRA (8 bit), which contains insurance-specific question-answer pairs and domain knowledge. trainable params: 20.97M || all params: 8.05B || trainable %: 0.26%

LoraConfig(
  r=8,
  lora_alpha=32,
  lora_dropout=0.05,
  bias="none",
  task_type="CAUSAL_LM",
  target_modules=['up_proj', 'down_proj', 'gate_proj', 'k_proj', 'q_proj', 'v_proj', 'o_proj']
)

Model Architecture

The model uses the Llama 3 architecture with the following key components:

8B parameter configuration
Enhanced attention mechanisms from Llama 3
ChatQA 1.5 instruction-tuning framework
Insurance domain-specific adaptations

Files in Repository

Model Files:
- model-00001-of-00004.safetensors (4.98 GB)
- model-00002-of-00004.safetensors (5 GB)
- model-00003-of-00004.safetensors (4.92 GB)
- model-00004-of-00004.safetensors (1.17 GB)
- model.safetensors.index.json (24 kB)
Tokenizer Files:
- tokenizer.json (17.2 MB)
- tokenizer_config.json (51.3 kB)
- special_tokens_map.json (335 Bytes)
Configuration Files:
- config.json (738 Bytes)
- generation_config.json (143 Bytes)

Use Cases

This model is specifically designed for:

Insurance policy understanding and explanation
Claims processing assistance
Coverage analysis
Insurance terminology clarification
Policy comparison and recommendations
Risk assessment queries
Insurance compliance questions

Limitations

The model's knowledge is limited to its training data cutoff
Should not be used as a replacement for professional insurance advice
May occasionally generate plausible-sounding but incorrect information

Bias and Ethics

This model should be used with awareness that:

It may reflect biases present in insurance industry training data
Output should be verified by insurance professionals for critical decisions
It should not be used as the sole basis for insurance decisions
The model's responses should be treated as informational, not as legal or professional advice

Citation and Attribution

If you use this model in your research or applications, please cite:

@misc{maharajwala2024openinsurance,
  author = {Raj Maharajwala},
  title = {Open-Insurance-LLM-Llama3-8B},
  year = {2024},
  publisher = {HuggingFace},
  url = {https://huggingface.co/Raj-Maharajwala/Open-Insurance-LLM-Llama3-8B}
}

Downloads last month: 174

GGUF

Model size

8B params

Architecture

llama

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Model tree for QuantFactory/Open-Insurance-LLM-Llama3-8B-GGUF

Base model

nvidia/Llama3-ChatQA-1.5-8B

Quantized

(18)

this model