Instructions to use Babblu2821/tinyllama-medqa-qlora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Babblu2821/tinyllama-medqa-qlora with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
model = PeftModel.from_pretrained(base_model, "Babblu2821/tinyllama-medqa-qlora")

Transformers

How to use Babblu2821/tinyllama-medqa-qlora with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Babblu2821/tinyllama-medqa-qlora")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Babblu2821/tinyllama-medqa-qlora", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Babblu2821/tinyllama-medqa-qlora with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Babblu2821/tinyllama-medqa-qlora"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Babblu2821/tinyllama-medqa-qlora",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Babblu2821/tinyllama-medqa-qlora

SGLang

How to use Babblu2821/tinyllama-medqa-qlora with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Babblu2821/tinyllama-medqa-qlora" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Babblu2821/tinyllama-medqa-qlora",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Babblu2821/tinyllama-medqa-qlora" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Babblu2821/tinyllama-medqa-qlora",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Babblu2821/tinyllama-medqa-qlora with Docker Model Runner:
```
docker model run hf.co/Babblu2821/tinyllama-medqa-qlora
```

Babblu2821 commited on 4 days ago

Commit

2d4d0bf

verified ·

1 Parent(s): 87b077a

docs: replace LLM-judge factual ratings with the human pass

Browse files

Files changed (1) hide show

README.md +10 -7

README.md CHANGED Viewed

@@ -32,7 +32,7 @@ Code, method and full results: **https://github.com/fayazhussain2821/llm-finetun
 > This is a **methodology demonstration**, not a medical tool. Its factual reliability
 > has been measured, and while it is the best arm in this project, it is still poor: on
 > a blinded review of 20 held-out questions it **contradicted the reference answer or
-> invented an entity in 40% of them**, scoring **2.70 out of 5** for factual soundness.
 >
 > It produces fluent, confident, well-formed text that is frequently wrong. Fluency is
 > exactly what makes this dangerous.
@@ -113,16 +113,19 @@ No degeneration — the model does not loop.
 | | mean ↑ | contradicts reference ↓ |
 |---|---|---|
-| base (control) | 2.30 | 50% |
-| **this model** | **2.70** | **40%** |
 Fine-tuning produced **no detectable improvement in factual accuracy** (paired 95% CI
-−0.10 to +0.90, spanning zero), despite the automatic metrics reporting large gains.
 What *is* detectable is that the untouched base model already beats the fully
-fine-tuned GPT-2 arm by +1.10 (95% CI +0.55 to +1.65).
-These ratings were produced by an LLM judge, not a clinician, and are labelled as such
-in the source repository. They are a strong prior, not a clinical assessment.
 ## Usage

 > This is a **methodology demonstration**, not a medical tool. Its factual reliability
 > has been measured, and while it is the best arm in this project, it is still poor: on
 > a blinded review of 20 held-out questions it **contradicted the reference answer or
+> invented an entity in 35% of them**, scoring **3.25 out of 5** for factual soundness.
 >
 > It produces fluent, confident, well-formed text that is frequently wrong. Fluency is
 > exactly what makes this dangerous.
 | | mean ↑ | contradicts reference ↓ |
 |---|---|---|
+| base (control) | 2.90 | 50% |
+| **this model** | **3.25** | **35%** |
 Fine-tuning produced **no detectable improvement in factual accuracy** (paired 95% CI
+−0.20 to +0.90, spanning zero), despite the automatic metrics reporting large gains.
 What *is* detectable is that the untouched base model already beats the fully
+fine-tuned GPT-2 arm by +1.10 (95% CI +0.35 to +1.85).
+These ratings are a human pass, rated blind to which model produced each answer, by
+the repository's author — one non-expert rater, not a clinician and not adjudicated by
+a second. An earlier LLM-judge pass over the same sheet scored every arm lower (2.70
+mean, 40% contradiction for this model) but reached every identical verdict; both are
+published in the source repository.
 ## Usage