Automated MNLP evaluation report (2026-06-11)

EVAL_REPORT.md

@@ -0,0 +1,170 @@
+# Automated MNLP evaluation report
+
+- **Model repo:** [`cs-552-2026-group1/math_model`](https://huggingface.co/cs-552-2026-group1/math_model)
+- **Owner(s):** group **group1**
+- **Generated at:** 2026-06-11T06:23:10+00:00 (UTC)
+- **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
+
+_This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
+
+## Evaluated checkpoint
+
+- **Commit:** [`04a59ca`](https://huggingface.co/cs-552-2026-group1/math_model/commit/04a59caae3c7206c40e83107be3ae79322ed41cd)
+- **Message:** Submit model to HF
+- **Committed:** 2026-06-10T21:15:26+00:00
+
+## Summary
+
+| Benchmark | Accuracy | Status |
+|---|---:|---|
+| Math | 0.3200 | ok |
+| Knowledge | — | not run |
+| Multilingual | — | not run |
+| Safety | — | not run |
+
+## Sample completions
+
+_Prompts are intentionally omitted to avoid revealing benchmark contents. For multi-completion problems, only one completion is shown per sample._
+
+### Math
+
+**Correct** (1 shown)
+
+- **reference**: `\frac{639}{40}`
+- **overall** (8/8 completions correct)
+- **extracted** (✓): `\frac{639}{40}`
+- **completion**:
+
+    ```text
+    <think>
+    1. **Identify the given information:**
+       - On the map, a length of $3$ inches represents $10$ miles in the real world.
+       - The actual distance between two buildings is $53.25$ miles.
+    
+    2. **Set up the proportion:**
+       - Let $x$ be the distance between the buildings on the map in inches.
+       - The proportion can be set up as:
+         \[
+         \frac{3 \text{ inches}}{10 \text{ miles}} = \frac{x \text{ inches}}{53.25 \text{ miles}}
+         \]
+    
+    3. **Cross-multiply to solve for $x$:**
+       \[
+       3 \times 53.25 = 10 \times x
+       \]
+       \[
+       159.75 = 10x
+       \]
+    
+    4. **Solve for $x$:**
+       \[
+       x = \frac{159.75}{10}
+       \]
+       \[
+       x = 15.975
+       \]
+    
+    5. **Convert the decimal to a fraction:**
+       - The decimal $15.975$ can be written as:
+         \[
+         15.975 = 15 + 0.975
+         \]
+       - Convert $0.975$ to a fraction:
+         \[
+         0.975 = \frac{975}{1000}
+         \]
+       - Simplify $\frac{975}{1000}$ by dividing both the numerator and the denominator by their greatest common divisor, which is $25$:
+         \[
+         \frac{975}{1000} = \frac{39}{40}
+         \]
+       - Therefore:
+         \[
+         15.975 = 15 + \frac{39}{40} = \frac{15 \times 40}{40} + \frac{39}{40} = \frac{600 + 39}{40} = \frac{639}{40}
+         \]
+    
+    6. **Conclusion:**
+       - The distance between the buildings on the map, written as a fraction, is $\frac{639}{40}$ inches.
+    
+    The final answer is $\boxed{\frac{639}{40}}$.
+    </think>
+    
+    \boxed{\frac{639}{40}}
+    ```
+
+**Incorrect** (1 shown)
+
+- **reference**: `2 - \frac{\pi}{2}`
+- **overall** (0/8 completions correct)
+- **extracted** (✗): `\frac{2}{3}`
+- **completion**:
+
+    ```text
+    <think>
+    1. **Define the problem and initial conditions:**
+       - Derek starts with a debt of $\pi$ MIT.
+       - Each day, he tosses a fair coin:
+         - Heads: Reduces debt by 1 MIT.
+         - Tails: Reduces debt by 2 MITs.
+       - If at any point his debt is nonpositive, he is considered debt-free.
+       - If his debt is positive, it is doubled after the payment.
+    
+    2. **Define the probability of debt-free status
+    
+    ... [truncated 822 chars] ...
+    
+    \frac{3}{2}}{2} = -\frac{1}{2}
+         \]
+       - The general solution to the recurrence relation is:
+         \[
+         p_n = A \cdot 1^n + B \cdot \left(-\frac{1}{2}\right)^n = A + B \left(-\frac{1}{2}\right)^n
+         \]
+    
+    6. **Determine the constants $A$ and $B$:**
+       - Using the initial conditions $p_0 = 1$ and $p_1 = \frac{1}{2}$:
+         \[
+         p_0 = A + B = 1
+         \]
+         \[
+         p_1 = A + B \left(-\frac{1}{2}\right) = \frac{1}{2}
+         \]
+       - Solving these equations:
+         \[
+         A + B = 1
+         \]
+         \[
+         A - \frac{B}{2} = \frac{1}{2}
+         \]
+         \[
+         2A - B = 1
+         \]
+         \[
+         2A - B = 1
+         \]
+         \[
+         2A - (1 - A) = 1
+         \]
+         \[
+         3A = 2
+         \]
+         \[
+         A = \frac{2}{3}
+         \]
+         \[
+         B = 1 - A = 1 - \frac{2}{3} = \frac{1}{3}
+         \]
+    
+    7. **Find the probability for $n = \pi$:**
+       - Substitute $n = \pi$ into the general solution:
+         \[
+         p_\pi = \frac{2}{3} + \frac{1}{3} \left(-\frac{1}{2}\right)^\pi
+         \]
+       - Since $\left(-\frac{1}{2}\right)^\pi$ is a very small number (approaching 0 as $\pi$ increases), we can approximate:
+         \[
+         p_\pi \approx \frac{2}{3}
+         \]
+    
+    The final answer is $\boxed{\frac{2}{3}}$.
+    </think>
+    
+    \boxed{\frac{2}{3}}
+    ```