Automated MNLP evaluation report (2026-06-11)

EVAL_REPORT.md

@@ -0,0 +1,128 @@
+# Automated MNLP evaluation report
+
+- **Model repo:** [`cs-552-2026-camykaz/general_knowledge_model`](https://huggingface.co/cs-552-2026-camykaz/general_knowledge_model)
+- **Owner(s):** group **camykaz**
+- **Generated at:** 2026-06-11T06:23:10+00:00 (UTC)
+- **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
+
+_This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
+
+## Evaluated checkpoint
+
+- **Commit:** [`92dd482`](https://huggingface.co/cs-552-2026-camykaz/general_knowledge_model/commit/92dd482c600515e8ec2e881dc18688767ead0103)
+- **Message:** Restore GK v12b safetensors
+- **Committed:** 2026-06-05T09:10:53+00:00
+
+## Summary
+
+| Benchmark | Accuracy | Status |
+|---|---:|---|
+| Math | — | not run |
+| Knowledge | 0.4500 | ok |
+| Multilingual | — | not run |
+| Safety | — | not run |
+
+## Sample completions
+
+_Prompts are intentionally omitted to avoid revealing benchmark contents. For multi-completion problems, only one completion is shown per sample._
+
+### Knowledge
+
+**Correct** (1 shown)
+
+- **reference**: `D`
+- **overall** (1/1 completions correct)
+- **extracted** (✓): `D`
+- **completion**:
+
+    ```text
+    <think>
+    Okay, let's tackle this problem step by step. So, we have a quantum mechanical system where a particle of mass m is moving in a harmonic potential V(r) = (1/2)mω²r². This is a standard harmonic oscillator potential, right? And we need to find the energy of the third excited state and the number of linearly independent eigenfunctions for that energy.
+    
+    First, I remember that the harmonic osc
+    
+    ... [truncated 821 chars] ...
+    
+    second excited state $ n = 2 $, and the third excited state $ n = 3 $. So:
+    
+    $$
+    E_{\text{third excited}} = \hbar \omega \left(3 + \frac{3}{2}\right) = \frac{9}{2} \hbar \omega
+    $$
+    
+    ---
+    
+    ### Number of Linearly Independent Eigenfunctions
+    
+    The number of linearly independent eigenfunctions for a given energy level corresponds to the number of distinct ways to distribute the quanta $ n = 3 $ among the three dimensions $ x, y, z $. This is equivalent to the number of integer solutions to the equation:
+    
+    $$
+    n_x + n_y + n_z = 3
+    $$
+    
+    where $ n_x, n_y, n_z \geq 0 $ are integers. The number of such solutions is given by the stars and bars formula:
+    
+    $$
+    \binom{3 + 3 - 1}{3 - 1} = \binom{5}{2} = 10
+    $$
+    
+    However, in quantum mechanics, the states are not counted in this way due to the symmetry of the system. Instead, the number of **distinct eigenstates** with the same energy (i.e., same $ n $) is equal to the number of **distinct permutations** of the quanta, which is:
+    
+    $$
+    \text{Number of states} = \text{number of integer solutions with } n_x + n_y + n_z = 3 = 10
+    $$
+    
+    Hence, the number of linearly independent eigenfunctions for the third excited state is **10**.
+    
+    ---
+    
+    ### Final Answer
+    
+    $$
+    \boxed{D}
+    $$
+    ```
+
+**Incorrect** (1 shown)
+
+- **reference**: `C`
+- **overall** (0/1 completions correct)
+- **extracted** (✗): `B`
+- **completion**:
+
+    ```text
+    <think>
+    Okay, let's try to figure this out. The problem is about identifying which two aromatic compounds are in a mixture that has an equimolar 1:1 mix. The NMR data gives us some information about the chemical shifts. 
+    
+    First, the ¹H NMR spectrum shows two singlet signals near 6.7 ppm with a 1:1 ratio. Then there are three singlet signals near 2.2 ppm in a 2:1:1 ratio. The compounds given are fo
+    
+    ... [truncated 823 chars] ...
+    
+    e** has four methyl groups on positions 1, 2, 3, 5.
+    - **1,4-diethylbenzene** has two ethyl groups on positions 1 and 4.
+    - The **2.2 ppm signals** would come from the **methyl and ethyl groups** of the substituents.
+    - The **2:1:1 ratio** could be due to the distribution of substituents between the two compounds. This is plausible and matches the NMR pattern.
+    
+    #### **Option C: 1,2,3,4-tetramethylbenzene and 1,2,3,5-tetramethylbenzene**
+    - Both have four methyl groups, but the **2:1:1 ratio** is harder to explain unless one compound has two methyl groups and the other one ethyl group, which is not consistent with the given compounds.
+    
+    #### **Option D: 1,2,4,5-tetramethylbenzene and 1,2,3,4-tetramethylbenzene**
+    - Similar to Option A, but again the **2:1:1 ratio** is not clearly explained.
+    
+    ---
+    
+    ### **Conclusion**
+    
+    The **most plausible** scenario is that the **two compounds** are:
+    
+    - **1,2,3,5-tetramethylbenzene** (with four methyl groups on positions 1, 2, 3, 5)
+    - **1,4-diethylbenzene** (with two ethyl groups on positions 1 and 4)
+    
+    This results in a **2:1:1 ratio** of substituents at 2.2 ppm and the **1:1 ratio** of aromatic protons at 6.7 ppm.
+    
+    ---
+    
+    ### **Final Answer**
+    $$
+    \boxed{B}
+    $$
+    ```