|
|
--- |
|
|
library_name: transformers |
|
|
tags: |
|
|
- llama |
|
|
- lora-merged |
|
|
- math-tutor |
|
|
license: llama3.1 |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- Sashank-810/LFT_Final_FineTuned_Increased_Metrics |
|
|
--- |
|
|
|
|
|
# LFT + IDC Math Tutor (LoRA-merged) |
|
|
|
|
|
Summary: A math-tutor student model with an integrated IDC critic adapter merged into the base Llama-3.1-8B-Instruct (LoRA weights merged into base). Intended for math tutoring and doubt clarification. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- Base: meta-llama/Llama-3.1-8B-Instruct |
|
|
- Finetuned for: math tutoring + IDC-style critique/fix |
|
|
- Precision: FP16/BF16 compatible |
|
|
- Hardware: Single-GPU inference recommended |
|
|
|
|
|
## Intended Use |
|
|
|
|
|
- Educational tutoring, step-by-step math help, critique-and-fix of student answers. |
|
|
|
|
|
## Out-of-Scope |
|
|
|
|
|
- Safety-sensitive, legal, medical, or any harmful/abusive use. |
|
|
|
|
|
## How to Use (Transformers) |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
|
|
name = "Sashank-810/IDC_Global_Merged" |
|
|
tok = AutoTokenizer.from_pretrained(name) |
|
|
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype="auto", device_map="auto") |
|
|
prompt = "Explain the derivative of sin(x)." |
|
|
out = model.generate(--tok(prompt, return_tensors="pt").to(model.device), max_new_tokens=128) |
|
|
print(tok.decode(out[0], skip_special_tokens=True)) |
|
|
``` |
|
|
|
|
|
## How to Use (vLLM) |
|
|
|
|
|
```bash |
|
|
python -m vllm.entrypoints.api_server \ |
|
|
--model Sashank-810/IDC_Global_Merged \ |
|
|
--dtype auto \ |
|
|
--tensor-parallel-size 1 |
|
|
``` |
|
|
|
|
|
## License & Responsible Use |
|
|
|
|
|
- Use responsibly for education; avoid harmful or malicious outputs. |
|
|
|
|
|
--- |
|
|
|
|
|
# 📊 Evaluation Results (Llama 3.1-8B-Instruct Base vs Fine‑Tuned) |
|
|
|
|
|
## ✅ Structured Evaluation Summary |
|
|
|
|
|
--Total Questions:-- 2617 |
|
|
|
|
|
### Base Model Performance |
|
|
|
|
|
- --Correct:-- 625 |
|
|
- --Accuracy:-- 23.88% |
|
|
|
|
|
### Fine‑Tuned Model Performance |
|
|
|
|
|
- --Correct:-- 916 |
|
|
- --Accuracy:-- 35.00% |
|
|
|
|
|
### 🎯 Improvement |
|
|
|
|
|
- --Accuracy Gain:-- +11.12 percentage points |
|
|
- --Improved Answers:-- 483 |
|
|
- --Regressed Answers:-- 192 |
|
|
|
|
|
--- |
|
|
|
|
|
# 📝 Text Generation Metrics |
|
|
|
|
|
## Base Model |
|
|
|
|
|
--BLEU:-- 38.24 |
|
|
--ROUGE-1:-- 0.2947 |
|
|
--ROUGE-2:-- 0.0934 |
|
|
--ROUGE-L:-- 0.2936 |
|
|
--METEOR:-- 0.1633 |
|
|
|
|
|
<details> |
|
|
<summary>Full Base Model Metrics</summary> |
|
|
|
|
|
```json |
|
|
{ |
|
|
"bleu": { |
|
|
"score": 38.24172039700722, |
|
|
"counts": [2214, 1378, 1110, 875], |
|
|
"totals": [3765, 2033, 1740, 1462], |
|
|
"precisions": [58.80, 67.78, 63.79, 59.85], |
|
|
"bp": 0.612276654279684, |
|
|
"sys_len": 3765, |
|
|
"ref_len": 5612 |
|
|
}, |
|
|
"rouge": { |
|
|
"rouge1": 0.29469964396406867, |
|
|
"rouge2": 0.09342261992242887, |
|
|
"rougeL": 0.2935582970928785, |
|
|
"rougeLsum": 0.2940696059343364 |
|
|
}, |
|
|
"meteor": { |
|
|
"meteor": 0.16327044830765994 |
|
|
} |
|
|
} |
|
|
``` |
|
|
|
|
|
</details> |
|
|
|
|
|
--- |
|
|
|
|
|
## Fine‑Tuned Model |
|
|
|
|
|
--BLEU:-- 59.31 |
|
|
--ROUGE-1:-- 0.4423 |
|
|
--ROUGE-2:-- 0.1247 |
|
|
--ROUGE-L:-- 0.4424 |
|
|
--METEOR:-- 0.2478 |
|
|
|
|
|
<details> |
|
|
<summary>Full Fine‑Tuned Metrics</summary> |
|
|
|
|
|
```json |
|
|
{ |
|
|
"bleu": { |
|
|
"score": 59.31334282676538, |
|
|
"counts": [3324, 2048, 1600, 1201], |
|
|
"totals": [5734, 3124, 2659, 2219], |
|
|
"precisions": [57.97, 65.55, 60.17, 54.12], |
|
|
"bp": 1.0, |
|
|
"sys_len": 5734, |
|
|
"ref_len": 5612 |
|
|
}, |
|
|
"rouge": { |
|
|
"rouge1": 0.4423208144549374, |
|
|
"rouge2": 0.1247048391679649, |
|
|
"rougeL": 0.4424399985443162, |
|
|
"rougeLsum": 0.4414589284956114 |
|
|
}, |
|
|
"meteor": { |
|
|
"meteor": 0.24778242330127054 |
|
|
} |
|
|
} |
|
|
``` |
