Automated MNLP evaluation report (2026-05-26)

EVAL_REPORT.md

@@ -0,0 +1,60 @@
+# Automated MNLP evaluation report
+
+- **Model repo:** [`cs-552-2026-mvte/general_knowledge_model`](https://huggingface.co/cs-552-2026-mvte/general_knowledge_model)
+- **Owner(s):** group **mvte**
+- **Generated at:** 2026-05-26T12:29:10+00:00 (UTC)
+- **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
+
+_This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
+
+## Summary
+
+| Benchmark | Accuracy | Status |
+|---|---:|---|
+| Math | — | not run |
+| Knowledge | 0.3300 | ok |
+| Multilingual | — | not run |
+| Safety | — | not run |
+
+## Sample completions
+
+_Prompts are intentionally omitted to avoid revealing benchmark contents. For multi-completion problems, only one completion is shown per sample._
+
+### Knowledge
+
+**Correct** (1 shown)
+
+- **reference**: `C`
+- **overall** (1/1 completions correct)
+- **extracted** (✓): `C`
+- **completion**:
+
+    ```text
+    To determine why the program outputs 0 when adding 5 and 3 using 3-bit integers, we need to consider how integers are represented in binary and how addition works in such a system.
+    
+    - **5 in binary is 101** (3 bits).
+    - **3 in binary is 011** (3 bits).
+    - Adding these:  
+      $$
+      101 \text{ (5)} + 011 \text{ (3)} = 1000 \text{ (8)}
+      $$
+      However, since we're only using 3 bits, the result would be **000**, which is 0.
+    
+    This is due to **overflow** — when the result of an operation exceeds the number of bits available to represent it, the value wraps around to 0.
+    
+    ### Correct Answer:
+    \boxed{C}
+    ```
+
+**Incorrect** (1 shown)
+
+- **reference**: `G`
+- **overall** (0/1 completions correct)
+- **extracted** (✗): `J`
+- **completion**:
+
+    ```text
+    In the Thomas and Chess model of temperament, the child's behavior is described based on two dimensions: **easy** and **difficult**. The child in question is **passive** (quiet, does not initiate activities) and **fearful** (holds tightly to her mother's hand, shows anxiety).
+    
+    The correct answer is: \boxed{J}
+    ```