Automated MNLP evaluation report (2026-06-11)
#1
by zechen-nlp - opened
- EVAL_REPORT.md +57 -0
EVAL_REPORT.md
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Automated MNLP evaluation report
|
| 2 |
+
|
| 3 |
+
- **Model repo:** [`cs-552-2026-TopHaylin/general_knowledge_model`](https://huggingface.co/cs-552-2026-TopHaylin/general_knowledge_model)
|
| 4 |
+
- **Owner(s):** group **TopHaylin**
|
| 5 |
+
- **Generated at:** 2026-06-11T06:23:10+00:00 (UTC)
|
| 6 |
+
- **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
|
| 7 |
+
|
| 8 |
+
_This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
|
| 9 |
+
|
| 10 |
+
## Evaluated checkpoint
|
| 11 |
+
|
| 12 |
+
- **Commit:** [`980ff33`](https://huggingface.co/cs-552-2026-TopHaylin/general_knowledge_model/commit/980ff33ee27b611de66ca5122b1f70810443ab11)
|
| 13 |
+
- **Message:** Upload D2 LoRA128 Redux+MMLU-Pro 70k pass1-best checkpoint
|
| 14 |
+
- **Committed:** 2026-06-09T14:37:32+00:00
|
| 15 |
+
|
| 16 |
+
## Summary
|
| 17 |
+
|
| 18 |
+
| Benchmark | Accuracy | Status |
|
| 19 |
+
|---|---:|---|
|
| 20 |
+
| Math | — | not run |
|
| 21 |
+
| Knowledge | 0.5000 | ok |
|
| 22 |
+
| Multilingual | — | not run |
|
| 23 |
+
| Safety | — | not run |
|
| 24 |
+
|
| 25 |
+
## Sample completions
|
| 26 |
+
|
| 27 |
+
_Prompts are intentionally omitted to avoid revealing benchmark contents. For multi-completion problems, only one completion is shown per sample._
|
| 28 |
+
|
| 29 |
+
### Knowledge
|
| 30 |
+
|
| 31 |
+
**Correct** (1 shown)
|
| 32 |
+
|
| 33 |
+
- **reference**: `C`
|
| 34 |
+
- **overall** (1/1 completions correct)
|
| 35 |
+
- **extracted** (✓): `C`
|
| 36 |
+
- **completion**:
|
| 37 |
+
|
| 38 |
+
```text
|
| 39 |
+
I need to select the option that is best supported by the question. Option C, 'An overflow error occurred.', directly matches the required fact or relation. The other choices are less consistent with the question.
|
| 40 |
+
</think>
|
| 41 |
+
|
| 42 |
+
\boxed{C}
|
| 43 |
+
```
|
| 44 |
+
|
| 45 |
+
**Incorrect** (1 shown)
|
| 46 |
+
|
| 47 |
+
- **reference**: `B`
|
| 48 |
+
- **overall** (0/1 completions correct)
|
| 49 |
+
- **extracted** (✗): `D`
|
| 50 |
+
- **completion**:
|
| 51 |
+
|
| 52 |
+
```text
|
| 53 |
+
I need to select the option that is best supported by the question. Option D, '(2S,3S,4S)-2-benzyl-4-hydroxy-2-methyl-3-phenylcyclohexan-1-one', directly matches the required fact or relation. The other choices are less consistent with the question.
|
| 54 |
+
</think>
|
| 55 |
+
|
| 56 |
+
\boxed{D}
|
| 57 |
+
```
|