Questo modello non e' un dispositivo medico e non deve essere usato su persone reali. E' un artefatto di ricerca addestrato su dati interamente sintetici, i cui valori di sensibilita' e specificita' non sono mai stati validati clinicamente. Ogni metrica riportata misura la fedelta' a un simulatore, non la validita' clinica.
This model is not a medical device and must not be used on real people. It is a research artifact trained on entirely synthetic data whose sensitivity and specificity values have never been clinically validated. Every metric below measures fidelity to a simulator, not clinical validity.
knee-physio-ft (LoRA adapter)
This model is not a medical device under Regulation (EU) 2017/745 and is not intended for any clinical use. It is a research and teaching artifact. It must not be used to form a diagnosis, to determine triage or referral, to plan treatment, or as decision support of any kind concerning a real person.
A LoRA adapter on Qwen3-4B-Instruct-2507 trained on entirely synthetic data to
produce, from the findings of a knee physiotherapy assessment written in Italian, a JSON
object containing red flags, ranked clinical hypotheses with confidences, suggested
additional tests, and rehabilitation goals.
Intended purpose
To demonstrate a reproducible fine-tuning and evaluation methodology, and to serve as a teaching example of how to measure a fine-tune honestly: with a baseline, with known ceilings, and with guards against silent evaluation failure.
Out of scope
- Any clinical, diagnostic, or triage use concerning real patients
- Body regions other than the knee
- Paediatric populations
- Languages other than Italian
- Any setting where a person might act on the output
Training data
Procedurally generated synthetic dataset: 4,000 examples (3,200 train / 400 validation / 400 test), seed 42. No real patient data was used at any stage. Train/test input overlap was verified to be zero.
A condition is sampled from indicative prior prevalences; clinical signs are then sampled conditionally using approximate sensitivity and specificity values; the training target is the posterior computed by a Naive Bayes model over those signs.
Known limits of the data
- The prevalences and sensitivity/specificity values have not been clinically validated. They are approximations created for this exercise. The model reproduces the generator's assumptions, not real epidemiology.
- Naive Bayes assumes conditional independence between clinical tests, which is clinically false. Confidences are systematically overstated when several correlated tests agree — expect a leading hypothesis at 0.99 with alternatives at exactly 0.0.
- Evidence lists include negative findings in a somewhat unnatural compressed phrasing.
Training
LoRA r=16, alpha=32, dropout 0.0, applied to all seven attention and MLP projections. QLoRA 4-bit base. 2 epochs / 400 steps at effective batch 16, lr 2e-4 cosine, 5% warmup. Loss computed on the response span only. ~5h40m on a Colab T4, 8.46 GB peak VRAM.
Training loss 0.0297 and validation loss 0.0298 at step 400, tracking each other monotonically throughout — no overfitting observed at 2 epochs.
Evaluation
200 held-out test cases, greedy decoding, identical prompt for both conditions.
| Metric | Base | Fine-tuned | Ceiling |
|---|---|---|---|
| Valid JSON | 97.0% | 100% | 100% |
| Schema valid | 96.5% | 100% | 100% |
| Leading hypothesis vs. generator target | 0.0% | 90.5% | 100% |
| Leading hypothesis vs. sampled truth | 0.0% | 60.0% | ~67% |
| Red-flag recall | 100% | 100% (15/15) | 100% |
| Red-flag precision | 9.2% | 100% | 100% |
| Referral action correct | 23.3% | 100% | 100% |
| Confidence MAE | 17.1% | 6.8% | 0% |
| Evidence grounded in input | 91.0% | 83.9% | 83.8% |
Full definitions, denominators, and the derivation of both ceilings are in evaluation.md.
Reading these numbers
- 0.0% baseline on hypothesis accuracy is a vocabulary mismatch, not a reasoning failure. The base model emits well-formed JSON with sensible content but invents free-text condition names instead of using the project's eleven enum labels.
- 60.0% against a ~67% ceiling is the headline. The ceiling exists because the generator's own signs point away from the sampled condition about a third of the time. Staying below it is evidence of generalisation rather than memorisation.
- 83.9% grounding is convergence, not regression. It sits exactly on the 83.8% ceiling measured against the gold targets.
- Red-flag metrics rest on 15 positives. The 95% CI on recall is roughly 78–100%.
Evaluation is conducted against the same generator that produced the training data. It does not measure clinical validity. No claim about real-world performance is made or implied.
Bias and fairness
Not assessed. The generator does not model demographic variation beyond age, and no subgroup analysis was performed. Any deployment consideration would require this work first — which is one of several reasons there should be no deployment.
Environmental cost
A single ~5.7-hour training run on one T4. Roughly 0.4 kWh of GPU energy, plus the earlier failed runs documented in the postmortem.
Reproduction
Code, generator, and executed notebook: https://github.com/rogiskhan/knee-physio-ft
Full run manifest: results/run_manifest.json
Licence
Apache 2.0, inherited from the base model. The NOTICE restating the non-clinical intended purpose must be retained in redistributions.
Prompt format
The model expects the system prompt used in training, and a user turn containing the assessment findings in Italian. It returns a single JSON object. Anything else — an English prompt, a free-form question, a different body region — is out of distribution and the output should not be trusted even as a demonstration.
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("rogiskhan/knee-physio-qwen3-4b")
model = AutoModelForCausalLM.from_pretrained("rogiskhan/knee-physio-qwen3-4b", device_map="auto")
messages = [
{"role": "system", "content": SYSTEM_PROMPT}, # see the repo's generator
{"role": "user", "content": "VALUTAZIONE FISIOTERAPICA - ginocchio sx ..."},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
enc = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**enc, max_new_tokens=700, do_sample=False)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))
add_special_tokens=False is not optional: apply_chat_template already emits them, and
adding them twice degrades the output badly. See the repository postmortem.
- Downloads last month
- 29
Model tree for rogiskhan/knee-physio-qwen3-4b
Base model
Qwen/Qwen3-4B-Instruct-2507