Update model after training on balanced dataset (95.6K examples, 3 epochs)

README.md

@@ -1,7 +1,7 @@
 ---
 language: en
 license: llama3.1
-library_name: peft
+library_name: transformers
 tags:
 - text-classification
 - energy
@@ -13,13 +13,10 @@ tags:
 - energy-documents
 pipeline_tag: text-classification
 widget:
-- text: Solar energy has become increasingly cost-competitive with fossil fuels in
-    recent years. The price of photovoltaic panels has dropped significantly, making
-    renewable energy more accessible.
-  example_title: Energy Document
-- text: The committee discussed the implementation of new operational guidelines.
-    Training sessions will be conducted for all staff members next month.
-  example_title: Non-Energy Document
+- text: "Solar energy has become increasingly cost-competitive with fossil fuels in recent years. The price of photovoltaic panels has dropped significantly, making renewable energy more accessible."
+  example_title: "Energy Document"
+- text: "The committee discussed the implementation of new operational guidelines. Training sessions will be conducted for all staff members next month."
+  example_title: "Non-Energy Document"
 datasets:
 - custom
 metrics:
@@ -39,51 +36,88 @@ model-index:
       type: custom
     metrics:
     - type: accuracy
-      value: 0.9806
+      value: 0.9839
       name: Test Accuracy
       verified: true
     - type: f1
-      value: 0.9807
+      value: 0.9841
       name: Test F1 Score
       verified: true
     - type: precision
-      value: 0.9747
+      value: 0.9717
       name: Test Precision
       verified: true
     - type: recall
-      value: 0.9869
+      value: 0.9969
       name: Test Recall
       verified: true
     - type: roc_auc
-      value: 0.9935
+      value: 0.9976
       name: ROC-AUC
       verified: true
 ---
 
 # 🔋 Llama-3.1-8B Energy Document Classifier
 
-A fine-tuned **Llama-3.1-8B** model for binary classification of energy-related documents, achieving **98.06% accuracy** on test data.
+A fine-tuned **Llama-3.1-8B** model for binary classification of energy-related documents, achieving **98.39% accuracy** on test data.
 
-This model uses **LoRA (Low-Rank Adaptation)** for parameter-efficient fine-tuning, making it lightweight and fast while maintaining high performance.
+This model uses **LoRA (Low-Rank Adaptation)** for parameter-efficient fine-tuning, trained on **95,602 documents** (perfectly balanced: 47,801 energy + 47,801 non-energy).
 
 ## 📊 Model Performance
 
+### Test Set Results (9,562 documents)
+
 | Metric | Score |
 |--------|-------|
-| **Test Accuracy** | 98.06% |
-| **F1 Score** | 98.07% |
-| **Precision** | 97.47% |
-| **Recall** | 98.69% |
-| **ROC-AUC** | 99.35% |
+| **Test Accuracy** | 98.39% |
+| **F1 Score** | 98.41% |
+| **Precision** | 97.17% |
+| **Recall** | 99.69% |
+| **ROC-AUC** | 99.76% |
+
+### Validation Set Results (9,560 documents)
 
-### Confusion Matrix (Test Set - 3,818 samples)
+| Metric | Score |
+|--------|-------|
+| **Val Accuracy** | 98.55% |
+| **Val F1 Score** | 98.56% |
+| **Val Precision** | 97.54% |
+| **Val Recall** | 99.60% |
+| **Val ROC-AUC** | 99.76% |
+
+### Confusion Matrix (Test Set - 9,562 documents)
 
 |  | Predicted Non-Energy | Predicted Energy |
 |--|---------------------|------------------|
-| **Actual Non-Energy** | 1,860 (97.43%) | 49 (2.57%) |
-| **Actual Energy** | 25 (1.31%) | 1,884 (98.69%) |
-
-**Only 74 misclassifications out of 3,818 documents!**
+| **Actual Non-Energy** | 4,642 (97.09%) | 139 (2.91%) |
+| **Actual Energy** | 15 (0.31%) | 4,766 (99.69%) |
+
+**Only 154 misclassifications out of 9,562 documents (1.61% error rate)!**
+
+### Training Details
+
+- **Base Model**: meta-llama/Llama-3.1-8B
+- **Training Method**: LoRA (r=16, alpha=32, dropout=0.05)
+- **Target Modules**: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
+- **Trainable Parameters**: 45M out of 8B (0.56%)
+- **Total Dataset**: 95,602 documents (perfectly balanced)
+  - Train: 76,480 (38,240 energy + 38,240 non-energy)
+  - Val: 9,560 (4,780 energy + 4,780 non-energy)
+  - Test: 9,562 (4,781 energy + 4,781 non-energy)
+- **Energy Data Sources**:
+  - EnergyAI/finepdfs_energy (40,989 docs)
+  - EnergyAI/wikipedia_energy (5,459 docs)
+  - EnergyAI/eartharxiv_engrxiv_energy (27 docs)
+  - EnergyAI/scored_chunks_from_SPE_pipeline (1,326 docs)
+- **Training Time**: ~2 hours on 4× A100 80GB GPUs
+- **Convergence**: Early stopping at step 1,100 (< 1 epoch!)
+- **Effective Batch Size**: 64 (per_device=4, gradient_accum=4, 4 GPUs)
+- **Learning Rate**: 2e-5 with cosine schedule and 10% warmup
+- **Precision**: bfloat16 mixed precision
+
+### Data Curation
+
+Energy-labeled documents were sourced from four HuggingFace datasets (see above). Non-energy documents were sampled from a base document pipeline, with deduplication to ensure no overlap with energy documents (validated by both document ID and MD5 hash matching).
 
 ## 🎯 Use Cases
 
@@ -549,6 +583,3 @@ For questions or issues:
 ---
 
 **Happy Classifying! 🔋⚡**
-### Framework versions
-
-- PEFT 0.12.0