epigenetics-slm

A Llama 3.2 1B Instruct model fine-tuned via QLoRA to generate five-category epigenetic health assessments from wearable/biomarker data, grounded in a Bio-RAG evidence retrieval pipeline.

Given a patient's biomarkers (HbA1c, NLR, circadian rest-activity metrics, sleep architecture, CosinorAge acceleration) and retrieved evidence chunks, the model produces a structured report with five sections: AGING, STRESS, METABOLISM, INFLAMMATION, SLEEP — each citing the evidence it was given.

Files in this repo

File	Description
slm.q4_k_m.gguf	Quantized model (q4_k_m), ~771MB, for CPU inference via llama-cpp-python
slm_lora/	Full LoRA adapter + tokenizer + all training checkpoints (100–1491 steps), with optimizer/scheduler state for resuming training
chroma_db/	Populated Bio-RAG vector store (47 chunks, all-MiniLM-L6-v2 embeddings) — required alongside the model for grounded generation

Training details

• Base model: unsloth/llama-3.2-1b-instruct-unsloth-bnb-4bit
• Method: QLoRA via Unsloth + TRL SFTTrainer
• Trainable params: 11.2M / 1.25B (0.90%)
• Dataset: 4,415 examples derived from NHANES 2011-2012 biomarkers + Bio-RAG evidence (3,973 train / 442 eval)
• Epochs: 3 (1,491 steps), effective batch size 8
• Hardware: RTX 4070 Laptop GPU (8GB VRAM), WSL2 Ubuntu 22.04
• Precision: bf16 training, fp32 LoRA adapter weights

Eval results (442-row held-out split)

Metric	Score
Category coverage (all 5 headers present)	99.8%
Classification match rate	98.5%
ROUGE-L	0.822

These metrics check structural adherence (all five sections present) and classification-label accuracy against the training data's target responses. They do not measure citation faithfulness — see Limitations.

Usage

Requires the model to be prompted via the exact training-time template (see slm_prompt.py in the source repo for the canonical build_prompt()/build_inference_prompt() functions — byte-identical prompt formatting between training and inference is required for output quality).

from llama_cpp import Llama

llm = Llama(model_path="slm.q4_k_m.gguf", n_ctx=4096, verbose=False)

# prompt must be built via build_prompt() + build_inference_prompt()
# from the source repo — see link above
out = llm(prompt, max_tokens=512, temperature=0.2, stop=["<|eot_id|>"])
print(out["choices"][0]["text"])

For full end-to-end usage (XGBoost baseline + Bio-RAG retrieval + this SLM), see scripts/demo.py in the source repo.

Limitations

• Requires Bio-RAG evidence to ground citations. When run without retrieved evidence chunks (empty RAG context), the model fabricates plausible-sounding but nonexistent citations (invented author names, journals, and DOIs). Always pass real retrieved evidence chunks.
• Citation precision, not just presence. Even with real evidence available, the model sometimes reuses the same citation pair across multiple report sections rather than mapping each specific claim to its most relevant source chunk. This is an attribution-precision issue, not fabrication.
• 1B parameter model. Domain-specific acronym/definition accuracy (e.g. circadian rest-activity metrics) is not guaranteed to be robust; spot-check outputs before use in any downstream decision-making context.
• Not a medical device. This is a research/prototype system trained on NHANES survey data. Outputs should not be used for clinical decision-making.

Source code

Training, export, and evaluation pipeline: github.com/rtsh13/epigenetics-slm (branch feat/week7-slm-finetuning)