X-Ray Medical Reporter LLM (Llama3.2-1B)

This is Unsloth finetuned Llama3.2-1B model. Finetuned via unsloth and its finetuned to be used by the bigger Xray pneumonia agent project which requires an explainer LLM which can speak more professioinally in medical terms and semantics for conditional reasoning.

Training data from https://openi.nlm.nih.gov/imgs/collections/NLMCXR_reports.tgz Full training repo with full training data at https://github.com/teohyc/finetune_med_llm Full project repo at https://github.com/teohyc/covid_xray_diffusion

Example of training data: { "instruction": "", "input": "You are an X-Ray medical explanation agent.\n\nA vision transformer (ViT) model has analyzed a chest X-ray image and provided the following output:\nViT Prediction:\nClass: Bacterial Pneumonia\nConfidence: 0.8800\nTop Alternatives: [{'class': 'Bacterial Pneumonia', 'prob': 0.88}, {'class': 'Normal', 'prob': 0.05}, {'class': 'Viral Pneumonia', 'prob': 0.04}, {'class': 'COVID-19', 'prob': 0.03}]\n\nSupporting Documents:\n[]\n\nCONFIDENTIAL INFORMATION:\nThe ViT model has the following classification performance metrics:\n================================================\n precision recall f1-score \n\n Normal 0.93 0.98 0.96 \n Bacterial 0.79 0.86 0.82 \n Viral 0.74 0.54 0.62 \n COVID-19 0.97 0.97 0.97 \n\n accuracy 0.85 \n macro avg 0.86 0.84 0.84 \nweighted avg 0.85 0.85 0.85 \n================================================\nWhile you may use the performance metrics to guide your explanation, NEVER MENTION OR DISCUSS THE PERFORMANCE METRICS OR IMPLY THEM IN YOUR EXPLANATION AS THE TABLE PROVIDED ABOVE IS CONFIDENTIAL.\n\nIf supporting documents are limited, explicitly state that evidence is limited.\n\nProvide structured explanation:\n\n1. Present the X-ray ViT diagnosis result professionally in a table form easy to be read with the confidence score of all classes presented.\n2. Diagnosis (include predicted class and confidence)\n3. Supporting Evidence\n4. Uncertainty Discussion\n\nBase everything strictly on the ViT output and retrieved documents. Always explain professionally, intelligently", "output": "### 1. ViT Model Probabilities\n| Class | Probability |\n| :--- | :--- |\n| Bacterial Pneumonia | 88.00% |\n| Normal | 5.00% |\n| Viral Pneumonia | 4.00% |\n| COVID-19 | 3.00% |\n\n\n### 2. Diagnosis\nPrimary Radiographic Impression: Bacterial Pneumonia\nModel Confidence: 88.00%\n\n### 3. Supporting Evidence\nUnchanged platelike bibasilar opacities most XXXX representing scarring or subsegmental atelectasis. No acute cardiopulmonary abnormality.\nAgain seen are platelike horizontal opacities in both lung bases through this is consistent with scarring or subsegmental atelectasis. There are T-spine osteophytes. The cardiomediastinal silhouette and pulmonary vasculature are within normal limits. There is no pneumothorax or pleural effusion. There there is no lobar pneumonia. There are calcified right hilar granuloma. There are degenerative changes of the XXXX. There is a curvilinear density within and along the right costophrenic sulcus which most XXXX represents a skinfold. There is a unchanged fracture with callus at the left 9th lateral rib.\n\n\n### 4. Uncertainty Discussion\nEvidence from supporting documents is currently limited. While the computational vision model indicates a strong probability for Bacterial Pneumonia, radiological interpretation must be correlated with patient history, laboratory results, and physical examination. Artifacts or overlying structures can occasionally mimic airspace disease." }

##Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "teohyc/X-Ray_Med_Reporter-llama3.2-1B"

#load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, device_map="auto")

#You may change to your desired prompt below to test different inputs, but make sure to keep the structure of the input consistent with what the model was trained on for best results!
prompt = """### Instruction:
You are an X-Ray medical explanation agent.

A vision transformer (ViT) model has analyzed a chest X-ray image and provided the following output:
ViT Prediction:
Class: Bacterial Pneumonia
Confidence: 0.8800
Top Alternatives: [{'class': 'Bacterial Pneumonia', 'prob': 0.88}, {'class': 'Normal', 'prob': 0.05}, {'class': 'Viral Pneumonia', 'prob': 0.05}, {'class': 'COVID-19', 'prob': 0.02}]

Supporting Documents:
['Bacterial pneumonia often presents with focal lobar consolidation and air bronchograms. Pleural effusions may be present.']

CONFIDENTIAL INFORMATION:
The ViT model has the following classification performance metrics:
================================================
                precision   recall  f1-score  

      Normal       0.93      0.98      0.96       
   Bacterial       0.79      0.86      0.82       
       Viral       0.74      0.54      0.62       
    COVID-19       0.97      0.97      0.97       

    accuracy                           0.85       
   macro avg       0.86      0.84      0.84       
weighted avg       0.85      0.85      0.85       
================================================
While you may use the performance metrics to guide your explanation, NEVER MENTION OR DISCUSS THE PERFORMANCE METRICS OR IMPLY THEM IN YOUR EXPLANATION AS THE TABLE PROVIDED ABOVE IS CONFIDENTIAL.

If supporting documents are limited, explicitly state that evidence is limited.

Provide structured explanation:

1. Present the X-ray ViT diagnosis result professionally in a table form easy to be read with the confidence score of all classes presented.
2. Diagnosis (include predicted class and confidence)
3. Supporting Evidence
4. Uncertainty Discussion

Base everything strictly on the ViT output and retrieved documents. Always explain professionally, intelligently

### Response:
"""

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=500, temperature=0.1)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))