Upload folder using huggingface_hub

README.md

@@ -1,189 +1,62 @@
 ---
-language: en
-license: mit
-tags:
-  - medical
-  - icd-10
-  - clinical-nlp
-  - mimic-iv
-  - llama
-  - lora
-  - generative
-  - sft
-  - text-generation
 base_model: aaditya/Llama3-OpenBioLLM-8B
-datasets:
-  - physionet/mimic-iv
+library_name: peft
+model_name: openbiollm_sft_sfttrainer
+tags:
+- base_model:adapter:aaditya/Llama3-OpenBioLLM-8B
+- lora
+- sft
+- transformers
+- trl
+licence: license
 pipeline_tag: text-generation
 ---
 
-# OpenBioLLM-G — Generative ICD-10 Coder
-
-Trained as part of the Master's thesis:
-**"Enhancing Automated ICD-10 Medical Coding with Large Language Models"**
-California State University, Sacramento
-Author: Namirah Imtieaz Shaik
-Advisor: Dr. Haiquan Chen
-
----
-
-## Model Description
-
-OpenBioLLM-G is a generative ICD-10 medical coding model built on top of
-Llama3-OpenBioLLM-8B. Unlike the discriminative version (OpenBioLLM-D),
-it generates the ICD-10 code autoregressively as a text string by completing
-a structured clinical prompt.
-
-**Architecture:**
-- Backbone: aaditya/Llama3-OpenBioLLM-8B (full causal LM with generation head)
-- LoRA: r=16, alpha=32, task_type=CAUSAL_LM
-- Training method: Supervised Fine-Tuning (SFT) with TRL SFTTrainer
-- Quantization: QLoRA 4-bit NF4 during training to fit 8B model on single GPU
-
-**Key Design Choices:**
-- Tail-only supervision: loss computed only on the last 16 ICD completion tokens,
-  not on the system prompt or clinical note
-- Head+tail note cropping: 40% from note start, 60% from note end
-- Constrained decoding at inference: prefix trie forces valid ICD codes only
-
-**Prompt Format:**
-
-[SYSTEM]
-You are a medical coding assistant. Given a clinical note, you output
-the most accurate ICD-10 code. Respond with only the ICD-10 code,
-no extra text.
-[/SYSTEM]
-[USER]
-<discharge note cropped to fit 512 token budget>
-[/USER]
-[ICD]
-ICD-10 code: <model generates here>
-
-**Training Details:**
-- Dataset: MIMIC-IV discharge summaries
-- Train / Val / Test: 16,540 / 2,068 / 2,068 examples
-- Optimizer: AdamW with cosine LR schedule and 5% warmup
-- Epochs: 3 with early stopping on macro F1
-
----
-
-## Results
-
-| Metric | Score |
-|---|---|
-| Micro F1 | 0.7896 |
-| ROC-AUC (Weighted OVR) | 0.8897 |
+# Model Card for openbiollm_sft_sfttrainer
 
-**Best performing generative model in the thesis.**
-Outperforms Meditron-G (0.7591) and BioMistral-G (0.7093) on Micro F1.
+This model is a fine-tuned version of [aaditya/Llama3-OpenBioLLM-8B](https://huggingface.co/aaditya/Llama3-OpenBioLLM-8B).
+It has been trained using [TRL](https://github.com/huggingface/trl).
 
-Note: ROC-AUC for generative models is computed from hard one-hot predictions
-rather than real softmax class probabilities, making it less directly comparable
-to the discriminative model ROC-AUC scores.
-
----
-
-## How to Load and Use
+## Quick start
 
 ```python
-from transformers import AutoTokenizer, AutoModelForCausalLM
-from peft import PeftModel
-import torch
-
-# Step 1 - Load base model
-base_model = AutoModelForCausalLM.from_pretrained(
-    "aaditya/Llama3-OpenBioLLM-8B",
-    torch_dtype=torch.float16,
-    device_map="auto",
-)
-
-# Step 2 - Attach LoRA adapter
-model = PeftModel.from_pretrained(base_model, "Namirah07/OpenBioLLM-G-ICD10")
-model.eval()
-
-# Step 3 - Load tokenizer
-tokenizer = AutoTokenizer.from_pretrained("Namirah07/OpenBioLLM-G-ICD10")
-if tokenizer.pad_token is None:
-    tokenizer.pad_token = tokenizer.eos_token
-
-# Step 4 - Build prompt and generate
-note = "Patient admitted with chest pain and shortness of breath..."
-
-prompt = (
-    "[SYSTEM]\n"
-    "You are a medical coding assistant. Given a clinical note, you output "
-    "the most accurate ICD-10 code. Respond with only the ICD-10 code, "
-    "no extra text.\n"
-    "[/SYSTEM]\n"
-    "[USER]\n"
-    + note +
-    "\n[/USER]\n"
-    "[ICD]\n"
-    "ICD-10 code: "
-)
-
-inputs = tokenizer(
-    prompt,
-    return_tensors="pt",
-    truncation=True,
-    max_length=512,
-).to(model.device)
-
-with torch.no_grad():
-    output = model.generate(
-        **inputs,
-        max_new_tokens=8,
-        do_sample=False,
-        pad_token_id=tokenizer.pad_token_id,
-        eos_token_id=tokenizer.eos_token_id,
-    )
-
-input_len = inputs["input_ids"].shape[1]
-generated = tokenizer.decode(
-    output[0, input_len:],
-    skip_special_tokens=True
-).strip()
+from transformers import pipeline
 
-print("Predicted ICD-10 code:", generated)
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="None", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
 ```
 
-For constrained decoding (forces only valid ICD codes from a vocabulary)
-see the full evaluation script in the GitHub repository below.
+## Training procedure
 
----
+ 
 
-## Dataset
 
-MIMIC-IV clinical discharge summaries (Johnson et al., 2023).
-Access requires a PhysioNet credentialed account and data use agreement.
-https://physionet.org/content/mimiciv/
+This model was trained with SFT.
 
----
+### Framework versions
 
-## GitHub Repository
+- PEFT 0.18.1
+- TRL: 0.26.2
+- Transformers: 4.57.5
+- Pytorch: 2.6.0+cu124
+- Datasets: 4.4.2
+- Tokenizers: 0.22.1
 
-Full training code, evaluation scripts, constrained decoding implementation,
-and the Gradio explainability demo:
-https://github.com/Namirah07/Enhancing-Automated-ICD-Medical-Coding-with-Large-Language-Models
+## Citations
 
----
 
-## Citation
 
+Cite TRL as:
+    
 ```bibtex
-@mastersthesis{shaik2025icd10,
-  author  = {Namirah Imtieaz Shaik},
-  title   = {Enhancing Automated ICD-10 Medical Coding with Large Language Models},
-  school  = {California State University, Sacramento},
-  year    = {2025},
-  advisor = {Dr. Haiquan Chen}
+@misc{vonwerra2022trl,
+	title        = {{TRL: Transformer Reinforcement Learning}},
+	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
+	year         = 2020,
+	journal      = {GitHub repository},
+	publisher    = {GitHub},
+	howpublished = {\url{https://github.com/huggingface/trl}}
 }
-```
-
----
-
-## License
-
-MIT License.
-The base model (Llama3-OpenBioLLM-8B) is subject to its own license on HuggingFace.
-The MIMIC-IV dataset requires a PhysioNet data use agreement.
+```