Automated MNLP evaluation report (2026-06-11)

EVAL_REPORT.md

@@ -0,0 +1,186 @@
+# Automated MNLP evaluation report
+
+- **Model repo:** [`cs-552-2026-bilko/math_model`](https://huggingface.co/cs-552-2026-bilko/math_model)
+- **Owner(s):** group **bilko**
+- **Generated at:** 2026-06-11T06:23:10+00:00 (UTC)
+- **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
+
+_This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
+
+## Evaluated checkpoint
+
+- **Commit:** [`0236551`](https://huggingface.co/cs-552-2026-bilko/math_model/commit/0236551766f1f73213ce94ec93e99168322135ff)
+- **Message:** SFT stage 1: sft/new_tries/sft_enhanced_June5/sft_stage1_merged model
+- **Committed:** 2026-06-06T19:06:57+00:00
+
+## Summary
+
+| Benchmark | Accuracy | Status |
+|---|---:|---|
+| Math | 0.5200 | ok |
+| Knowledge | — | not run |
+| Multilingual | — | not run |
+| Safety | — | not run |
+
+## Sample completions
+
+_Prompts are intentionally omitted to avoid revealing benchmark contents. For multi-completion problems, only one completion is shown per sample._
+
+### Math
+
+**Correct** (1 shown)
+
+- **reference**: `99774`
+- **overall** (1/8 completions correct)
+- **extracted** (✓): `99774`
+- **completion**:
+
+    ```text
+    <think>
+    Okay, let's tackle this cryptarithm problem. So, we need to find a 5-digit number EARTH where HATER plus HEART equals EARTH. All the letters E, A, R, T, H are digits, and E and H can't be zero. Also, they don't have to be distinct. Hmm, cryptarithms can be tricky because of the carry-over rules. Let me start by writing down the equation:
+    
+    HATER + HEART = EARTH
+    
+    First, let me align the numb
+    
+    ... [truncated 823 chars] ...
+    
+    arry}_5 = 0
+    \]
+    
+    ### Checking Validity:
+    Using the values \(H = 4\), \(E = 9\), and solving the remaining equations:
+    
+    1. **Units Place**:
+    \[
+    R + T = 4 + 10 \cdot \text{carry}_1
+    \]
+    If \(\text{carry}_1 = 0\):
+    \[
+    R + T = 4
+    \]
+    If \(\text{carry}_1 = 1\):
+    \[
+    R + T = 14
+    \]
+    
+    2. **Tens Place**:
+    \[
+    E + R + \text{carry}_1 = T + 10 \cdot \text{carry}_2
+    \]
+    If \(E = 9\) and \(\text{carry}_1 = 0\):
+    \[
+    9 + R = T + 10 \cdot \text{carry}_2
+    \]
+    If \(\text{carry}_2 = 0\):
+    \[
+    9 + R = T
+    \]
+    If \(\text{carry}_2 = 1\):
+    \[
+    9 + R = T + 10
+    \]
+    
+    3. **Hundreds Place**:
+    \[
+    T + A + \text{carry}_2 = R + 10 \cdot \text{carry}_3
+    \]
+    With \(T + A + \text{carry}_2 = 14\):
+    \[
+    14 + A = R + 10 \cdot 1
+    \]
+    \[
+    A + R = 4
+    \]
+    
+    4. **Thousands Place**:
+    \[
+    A + E + \text{carry}_3 = A + 10 \cdot \text{carry}_4
+    \]
+    With \(E = 9\) and \(\text{carry}_3 = 1\):
+    \[
+    A + 9 + 1 = A + 10
+    \]
+    \[
+    A = 8
+    \]
+    
+    5. **Ten-Thousands Place**:
+    \[
+    H + H + \text{carry}_4 = E + 10 \cdot \text{carry}_5
+    \]
+    With \(H = 4\) and \(E = 9\):
+    \[
+    4 + 4 + 1 = 9 + 10 \cdot 0
+    \]
+    \[
+    9 = 9
+    \]
+    
+    ### Final Solution:
+    From the above, we have:
+    \[
+    H = 4, \quad E = 9, \quad A = 8, \quad R = 7, \quad T = 7
+    \]
+    Thus, the unique 5-digit integer \(\underline{EARTH}\) is:
+    \[
+    \boxed{99774}
+    \]
+    ```
+
+**Incorrect** (1 shown)
+
+- **reference**: `7200`
+- **overall** (0/8 completions correct)
+- **extracted** (✗): `301950`
+- **completion**:
+
+    ```text
+    <think>
+    Okay, so I need to figure out how many polynomials P with integer coefficients satisfy three conditions: degree at most 3, all coefficients between 0 and 2012, for each x from 1 to 100, P(x) + P(-x) is divisible by 2013, and the residues modulo 2013 of P(0), P(1), ..., P(2012) are all distinct. Hmm, that's a mouthful. Let me break it down step by step.
+    
+    First, let's note that 2013 is a com
+    
+    ... [truncated 823 chars] ...
+    
+    ubic polynomial modulo 3 must be a permutation polynomial. The number of permutation polynomials modulo a prime \( p \) is given by \( \frac{p(p-1)}{2} \). For \( p = 3 \):
+    \[
+    \frac{3 \times 2}{2} = 3
+    \]
+    Thus, there are 3 permutation cubic polynomials modulo 3.
+    
+    ### Step 2: Modulo 11 Analysis
+    Similarly, the number of permutation polynomials modulo 11 is:
+    \[
+    \frac{11 \times 10}{2} = 55
+    \]
+    Thus, there are 55 permutation cubic polynomials modulo 11.
+    
+    ### Step 3: Modulo 61 Analysis
+    For \( p = 61 \), the number of permutation polynomials is:
+    \[
+    \frac{61 \times 60}{2} = 1830
+    \]
+    Thus, there are 1830 permutation cubic polynomials modulo 61.
+    
+    ### Step 4: Combining Results Using Chinese Remainder Theorem
+    Since the polynomial \( P \) must be a permutation polynomial modulo \( 2013 \), the total number of such polynomials is the product of the number of permutation polynomials modulo each prime factor:
+    \[
+    3 \times 55 \times 1830
+    \]
+    
+    ### Step 5: Calculating the Product
+    First, calculate \( 3 \times 55 \):
+    \[
+    3 \times 55 = 165
+    \]
+    Next, multiply the result by 1830:
+    \[
+    165 \times 1830 = 301950
+    \]
+    
+    Thus, the number of polynomials \( P \) that satisfy all the given conditions is:
+    \[
+    \boxed{301950}
+    \]
+    ```