Automated MNLP evaluation report (2026-06-11)

EVAL_REPORT.md

@@ -0,0 +1,92 @@
+# Automated MNLP evaluation report
+
+- **Model repo:** [`cs-552-2026-painlp/math_model`](https://huggingface.co/cs-552-2026-painlp/math_model)
+- **Owner(s):** group **painlp**
+- **Generated at:** 2026-06-11T06:23:10+00:00 (UTC)
+- **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
+
+_This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
+
+## Evaluated checkpoint
+
+- **Commit:** [`97762f0`](https://huggingface.co/cs-552-2026-painlp/math_model/commit/97762f02e2ae010a19fe54c26600f0fc92c4d9af)
+- **Message:** upload: generation_config
+- **Committed:** 2026-05-24T16:28:57+00:00
+
+## Summary
+
+| Benchmark | Accuracy | Status |
+|---|---:|---|
+| Math | 0.3100 | ok |
+| Knowledge | — | not run |
+| Multilingual | — | not run |
+| Safety | — | not run |
+
+## Sample completions
+
+_Prompts are intentionally omitted to avoid revealing benchmark contents. For multi-completion problems, only one completion is shown per sample._
+
+### Math
+
+**Correct** (1 shown)
+
+- **reference**: `4`
+- **overall** (8/8 completions correct)
+- **extracted** (✓): `4`
+- **completion**:
+
+    ```text
+    <think>
+    To solve for $x$ in the equation $10^x - 10 = 9990$, we can start by isolating the term $10^x$.
+    
+    1. Add 10 to both sides of the equation:
+    \[ 10^x = 9990 + 10 \Rightarrow 10^x = 10000 \]
+    
+    2. Recognize that $10000 = 10^4$, so we have:
+    \[ 10^x = 10^4 \]
+    
+    3. Since the bases are the same, we can equate the exponents:
+    \[ x = 4 \]
+    
+    Thus, the solution is:
+    \[
+    </think>
+    
+    \boxed{4} \]
+    ```
+
+**Incorrect** (1 shown)
+
+- **reference**: `2 - \frac{\pi}{2}`
+- **overall** (0/8 completions correct)
+- **extracted** (✗): `<no answer>`
+- **completion**:
+
+    ```text
+    <think>
+    Okay, so Derek starts with a debt of π MIT, which is approximately 3.1416 MIT. Each day, he tosses a fair coin: heads reduces his debt by 1 MIT, tails by 2 MITs. If his debt ever becomes nonpositive, he's debt-free. Otherwise, after paying, his debt is doubled. We need to find the probability that Derek eventually becomes debt-free. Hmm, interesting problem. Let me try to break this down.
+    
+    ... [truncated 823 chars] ...
+    
+    P(0) = 1/2 * 1/2 + 1/2 * 1 = 3/4. But earlier, we thought P(2) = 1/2. This is a contradiction.
+    
+    Wait, this suggests that our earlier calculation was wrong. Let's re-examine.
+    
+    When x = 2, paying 1 MIT leads to 1 MIT, which is positive, so P(2) = 1/2 * P(1) + 1/2 * P(0). P(1) = 1/2, P(0) = 1. So P(2) = 1/2 * 1/2 + 1/2 * 1 = 3/4. But earlier, we thought P(2) = 1/2. This is a mistake! Earlier, I incorrectly stated that P(2) = 1/2, but actually, it should be 3/4. This changes everything!
+    
+    Let's recalculate the earlier terms with this correction.
+    
+    x = 1: P(1) = 1/2
+    
+    x = 2: P(2) = 1/2 * P(1) + 1/2 * P(0) = 1/2 * 1/2 + 1/2 * 1 = 3/4
+    
+    x = 3: P(3) = 1/2 * P(2) + 1/2 * P(2(x - 2)) = 1/2 * 3/4 + 1/2 * P(2) = 3/8 + 1/2 * 3/4 = 3/8 + 3/8 = 6/8 = 3/4
+    
+    x = 4: P(4) = 1/2 * P(3) + 1/2 * P(2(x - 2)) = 1/2 * 3/4 + 1/2 * P(2) = 3/8 + 1/2 * 3/4 = 3/8 + 3/8 = 6/8 = 3/4
+    
+    x = 5: P(5) = 1/2 * P(4) + 1/2 * P(2(x - 2)) = 1/2 * 3/4 + 1/2 * P(2) = 3/8 + 3/8 = 6/8 = 3/4
+    
+    Wait, this can't be right. All P(x) from x = 2 onwards are 3/4? That doesn't make sense. Let's check:
+    
+    Wait, no, x = 5: P(5) = 1/2 * P(4) + 1/2 * P(2(5 - 2)) = 1/2 * 3/4 + 1/2 * P(6) = 3/8 + 1/2 * P(6). But P(6) = 1/2 * P(4) + 1/2 * P(4) = 1/2 *
+    ```