Automated MNLP evaluation report (2026-06-11)

EVAL_REPORT.md

@@ -0,0 +1,100 @@
+# Automated MNLP evaluation report
+
+- **Model repo:** [`cs-552-2026-flab/general_knowledge_model`](https://huggingface.co/cs-552-2026-flab/general_knowledge_model)
+- **Owner(s):** group **flab**
+- **Generated at:** 2026-06-11T06:23:10+00:00 (UTC)
+- **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
+
+_This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
+
+## Evaluated checkpoint
+
+- **Commit:** [`b087c9c`](https://huggingface.co/cs-552-2026-flab/general_knowledge_model/commit/b087c9c239486a7969ce19f6c2eae209785e1763)
+- **Message:** Switch chat_template to the conservative group prompt (Knowledge CI 0.45 -> 0.51; identical weights to group_model)
+- **Committed:** 2026-06-10T18:02:30+00:00
+
+## Summary
+
+| Benchmark | Accuracy | Status |
+|---|---:|---|
+| Math | — | not run |
+| Knowledge | 0.4600 | ok |
+| Multilingual | — | not run |
+| Safety | — | not run |
+
+## Sample completions
+
+_Prompts are intentionally omitted to avoid revealing benchmark contents. For multi-completion problems, only one completion is shown per sample._
+
+### Knowledge
+
+**Correct** (1 shown)
+
+- **reference**: `I`
+- **overall** (1/1 completions correct)
+- **extracted** (✓): `I`
+- **completion**:
+
+    ```text
+    <think>
+    Okay, let's tackle this problem step by step. So, we have a generator with an initial emf of 125 volts at 1,200 rpm. We need to find the new emf in two scenarios: first, when the field flux is lowered by 10% while keeping the speed the same, and second, when the speed is decreased to 1,100 rpm while keeping the flux constant.
+    
+    First, I remember that the emf (ε) generated in a generator is
+    
+    ... [truncated 823 chars] ...
+    
+    Alternatively, maybe the formula is ε ∝ N * φ. So when flux is constant, ε ∝ N. So if N is decreased by 10%, then ε is decreased by 10%. So original ε is 125. 125 * 0.9 = 112.5 volts for part (a). Then for part (b), when speed is decreased to 1100 rpm, which is 11/12 of 1200 rpm. So 125 * (11/12) ≈ 114.58 volts. So that's 114.58 volts, which is approximately 114.6 volts. So option I is correct. Therefore, the answer is I.
+    </think>
+    
+    The generated emf in a generator is proportional to both the speed and the flux. The formula is $ \varepsilon = k \cdot N \cdot \phi $, where $ k $ is a constant.
+    
+    **Part (a):**  
+    - Original emf: $ \varepsilon = 125 \, \text{V} $  
+    - Speed remains the same: $ N = 1200 \, \text{rpm} $  
+    - Flux is lowered by 10%: $ \phi' = 0.9 \cdot \phi $  
+    
+    $$
+    \varepsilon' = 125 \cdot \frac{1200}{1200} \cdot \frac{0.9}{1.0} = 125 \cdot 0.9 = 112.5 \, \text{V}
+    $$
+    
+    **Part (b):**  
+    - Speed is decreased to $ 1100 \, \text{rpm} $  
+    - Flux remains constant: $ \phi = \phi' $  
+    
+    $$
+    \varepsilon' = 125 \cdot \frac{1100}{1200} \cdot \frac{1.0}{1.0} = 125 \cdot \frac{11}{12} \approx 114.58 \, \text{V}
+    $$
+    
+    The closest option is **I** (a) 112.5 volts, (b) 114.6 volts.
+    
+    $$
+    \boxed{I}
+    $$
+    ```
+
+**Incorrect** (1 shown)
+
+- **reference**: `A`
+- **overall** (0/1 completions correct)
+- **extracted** (✗): `C`
+- **completion**:
+
+    ```text
+    <think>
+    Okay, let's try to figure out this problem. So, the question is about determining how many hours a person will work given certain conditions. Let me break it down step by step.
+    
+    First, the problem states that a year has 8,000 hours, but actually it's 8,760. However, we're told to use 8,000 for calculations. The person's potential market wage is $5 per hour. The uncle leaves them an annual
+    
+    ... [truncated 536 chars] ...
+    
+    $1,000. The rest, which is 75% of 4,000, is $3,000, which they use for work. 
+    
+    But wait, the question is about how many hours they work. So the work income is $3,000. Since the wage is $5 per hour, the number of hours worked would be 3,000 divided by 5. Let me calculate that. 3,000 divided by 5 is 600. So that would be 600 hours. But let me check again to make sure I didn't make a mistake.
+    
+    Wait, the problem says they allocate 75% of full income to leisure. So full income is $4,000. 75% of that is $3,000, which is spent on leisure, and 25% is spent on work. So yes, 25% of 4,000 is 1,000, and 75% is 3,000. Then 3,000 divided by 5 is 600 hours. So the answer should be 600, which is option C. 
+    
+    But let me check the choices again. The options are from A to J. Option C is 600. So that's the answer. But wait, the problem mentions that the year has 8,000 hours. But if they work 600 hours, that's 600/8000 = 7.5% of the year. But the question is about how many hours they will work, not how many hours they spend in a year. So even though the actual year has more hours, the problem says to use 8,000 for calculation. So the answer is 600. Therefore, the correct choice is C.
+    </think>
+    
+    \boxed{C}
+    ```