Add model card with exact and within-1 confusion matrices and per-class metrics for non-fine-tuned LLaMA evaluation

README.md

@@ -0,0 +1,63 @@
+
+# Non-Fine-Tuned LLaMA-3-8B CEFR Evaluation
+
+This repository contains the evaluation results of the base `unsloth/llama-3-8b-instruct-bnb-4bit` model for CEFR-level sentence generation, without fine-tuning, as part of an ablation study. The model is evaluated using a fine-tuned classifier from `Mr-FineTuner/Skripsi_validator_best_model`.
+
+- **Base Model**: unsloth/llama-3-8b-instruct-bnb-4bit
+- **Evaluation Details**:
+  - Dataset: Rebalanced test dataset (`test_merged_output.txt`), which was also used to train and evaluate the classifier, potentially introducing bias.
+  - No fine-tuning performed; base model used directly.
+  - Classifier: MLP classifier trained on `train_merged_output.txt`, `dev_merged_output.txt`, and `test_merged_output.txt` for CEFR level prediction.
+- **Evaluation Metrics (Exact Matches)**:
+  - CEFR Classifier Accuracy: 0.150
+  - Precision (Macro): 0.194
+  - Recall (Macro): 0.150
+  - F1-Score (Macro): 0.140
+- **Evaluation Metrics (Within ±1 Level)**:
+  - CEFR Classifier Accuracy: 0.750
+  - Precision (Macro): 0.826
+  - Recall (Macro): 0.750
+  - F1-Score (Macro): 0.741
+- **Other Metrics**:
+  - Perplexity: 86.022
+  - Diversity (Unique Sentences): 0.967
+  - Inference Time (ms): 4952.351
+  - Model Size (GB): 8.0
+  - Robustness (F1): 0.133
+- **Confusion Matrix (Exact Matches)**:
+  - CSV: [confusion_matrix_exact.csv](confusion_matrix_exact.csv)
+  - Image: [confusion_matrix_exact.png](confusion_matrix_exact.png)
+- **Confusion Matrix (Within ±1 Level)**:
+  - CSV: [confusion_matrix_within1.csv](confusion_matrix_within1.csv)
+  - Image: [confusion_matrix_within1.png](confusion_matrix_within1.png)
+- **Per-Class Confusion Metrics (Exact Matches)**:
+  - A1: TP=0, FP=0, FN=10, TN=50
+  - A2: TP=1, FP=11, FN=9, TN=39
+  - B1: TP=3, FP=19, FN=7, TN=31
+  - B2: TP=2, FP=16, FN=8, TN=34
+  - C1: TP=2, FP=4, FN=8, TN=46
+  - C2: TP=1, FP=1, FN=9, TN=49
+- **Per-Class Confusion Metrics (Within ±1 Level)**:
+  - A1: TP=4, FP=0, FN=6, TN=50
+  - A2: TP=8, FP=2, FN=2, TN=48
+  - B1: TP=10, FP=6, FN=0, TN=44
+  - B2: TP=8, FP=7, FN=2, TN=43
+  - C1: TP=10, FP=0, FN=0, TN=50
+  - C2: TP=5, FP=0, FN=5, TN=50
+- **Note on Bias**:
+  - The test dataset used for evaluation (`test_merged_output.txt`) was part of the training and evaluation data for the classifier (`Mr-FineTuner/Skripsi_validator_best_model`). This may lead to inflated performance metrics due to the classifier's familiarity with the dataset. For a more robust evaluation, a new dataset not used in classifier training is recommended.
+- **Usage**:
+  ```python
+  from transformers import AutoModelForCausalLM, AutoTokenizer
+
+  model = AutoModelForCausalLM.from_pretrained("unsloth/llama-3-8b-instruct-bnb-4bit")
+  tokenizer = AutoTokenizer.from_pretrained("unsloth/llama-3-8b-instruct-bnb-4bit")
+
+  # Example inference
+  prompt = "[INST] Generate a CEFR B1 level sentence. [/INST]"
+  inputs = tokenizer(prompt, return_tensors="pt")
+  outputs = model.generate(**inputs, max_length=50)
+  print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+  ```
+
+Uploaded using `huggingface_hub`.