Automated MNLP evaluation report (2026-06-01)

EVAL_REPORT.md

@@ -0,0 +1,95 @@
+# Automated MNLP evaluation report
+
+- **Model repo:** [`cs-552-2026-baseline/general_knowledge_model`](https://huggingface.co/cs-552-2026-baseline/general_knowledge_model)
+- **Owner(s):** group **baseline**
+- **Generated at:** 2026-06-01T06:13:57+00:00 (UTC)
+- **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
+
+_This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
+
+## Summary
+
+| Benchmark | Accuracy | Status |
+|---|---:|---|
+| Math | — | not run |
+| Knowledge | 0.2700 | ok |
+| Multilingual | — | not run |
+| Safety | — | not run |
+
+## Sample completions
+
+_Prompts are intentionally omitted to avoid revealing benchmark contents. For multi-completion problems, only one completion is shown per sample._
+
+### Knowledge
+
+**Correct** (1 shown)
+
+- **reference**: `I`
+- **overall** (1/1 completions correct)
+- **extracted** (✓): `I`
+- **completion**:
+
+    ```text
+    <think>
+    Okay, let's try to figure out this credit term question. The term given is (7/10) EOM. I need to understand what this means. 
+    
+    First, I remember that credit terms like 7/10 EOM are commonly used in business. The format is usually something like (discount percentage)/days allowed, followed by the EOM. So, the 7/10 part would mean a 7% discount if paid within 10 days of the end of the month.
+    
+    ... [truncated 823 chars] ...
+    
+    r the end of the month"  
+      ✗ Incorrect. The discount is 7%, not 10%.
+    
+    - **F.** "A customer may deduct 7% of the amount of the invoice if payment is made on the tenth day of the month"  
+      ✗ Incorrect. The discount is not available on the 10th day of the month, but 10 days after the end of the month.
+    
+    - **G.** "A customer may deduct 10% of the amount of the invoice if payment is made no later than the seventh day of the month"  
+      ✗ Incorrect. The discount is 7%, not 10%.
+    
+    - **H.** "A customer must pay the full amount of the invoice within 7 days of the end of the month"  
+      ✗ Incorrect. This is not a credit term. The full payment is not required unless the discount is not taken.
+    
+    - **I.** "A customer may deduct 7% of the amount of the invoice if payment is made no later than the tenth day after the end of the month"  
+      ✅ **Correct.** This directly matches the meaning of the term **(7/10) EOM**, where the **7% discount is available within 10 days of the end of the month**.
+    
+    - **J.** "A customer may deduct 10% of the amount of the invoice if payment is made within 7 days of receiving the invoice"  
+      ✗ Incorrect. The discount is 7%, not 10%.
+    
+    ---
+    
+    ### Final Answer:
+    
+    $$
+    \boxed{I}
+    $$
+    ```
+
+**Incorrect** (1 shown)
+
+- **reference**: `A`
+- **overall** (0/1 completions correct)
+- **extracted** (✗): `<no answer>`
+- **completion**:
+
+    ```text
+    <think>
+    Okay, let's try to figure out this Cope rearrangement problem. So, the question is about determining the major products of two reactions: one is a Cope rearrangement involving a 1,5-diene and some reagents, and the other is a heat-induced rearrangement of a compound. The choices are A, B, C, D, and I need to pick the correct one.
+    
+    First, let me recall what a Cope rearrangement is. From wha
+    
+    ... [truncated 823 chars] ...
+    
+    ent, and the reaction is with BuLi and H+, maybe the reaction is a Grignard reaction. For example, if the substituent is a 1,2-diene, then the base (BuLi) might deprotonate the carbon adjacent to the double bond, forming a Grignard reagent, which then reacts with H+ to form a carbonyl. But I'm not sure.
+    
+    Alternatively, maybe the reaction is a hydroboration-oxidation, but that would require a different set of reagents.
+    
+    Alternatively, maybe the reaction is a Cope rearrangement. But the starting material is a 1,2-diene. Wait, the Cope rearrangement typically involves a 1,5-diene, so maybe the starting material is a 1,5-diene, but the structure given is different. 
+    
+    Alternatively, maybe the reaction is a cyclopropane formation. Wait, but the question says that the Cope rearrangement involves a 1,5-diene. 
+    
+    Alternatively, maybe the reaction is a Diels-Alder reaction, but that's not mentioned here.
+    
+    Alternatively, maybe the reaction is a nucleophilic substitution where the base deprotonates a carbon, leading to a carbonyl group. For example, if the substituent is a 1,2-diene, then the deprotonation might occur at the carbon adjacent to the double bond, leading to a carbonyl group. 
+    
+    But
+    ```