Automated MNLP evaluation report (2026-06-11)

EVAL_REPORT.md

@@ -0,0 +1,110 @@
+# Automated MNLP evaluation report
+
+- **Model repo:** [`cs-552-2026-the-transformers/general_knowledge_model`](https://huggingface.co/cs-552-2026-the-transformers/general_knowledge_model)
+- **Owner(s):** group **the-transformers**
+- **Generated at:** 2026-06-11T06:23:10+00:00 (UTC)
+- **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
+
+_This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
+
+## Evaluated checkpoint
+
+- **Commit:** [`a4ce155`](https://huggingface.co/cs-552-2026-the-transformers/general_knowledge_model/commit/a4ce155f8f52f41e8bf577b2d750a53186729715)
+- **Message:** Upload general knowledge SFT checkpoint
+- **Committed:** 2026-06-08T18:41:16+00:00
+
+## Summary
+
+| Benchmark | Accuracy | Status |
+|---|---:|---|
+| Math | — | not run |
+| Knowledge | 0.4200 | ok |
+| Multilingual | — | not run |
+| Safety | — | not run |
+
+## Sample completions
+
+_Prompts are intentionally omitted to avoid revealing benchmark contents. For multi-completion problems, only one completion is shown per sample._
+
+### Knowledge
+
+**Correct** (1 shown)
+
+- **reference**: `B`
+- **overall** (1/1 completions correct)
+- **extracted** (✓): `B`
+- **completion**:
+
+    ```text
+    <think>
+    Okay, let's try to figure this out. The question is about determining the correct 1H NMR data for a specific compound. Let me start by recalling what each part of the NMR data tells us.
+    
+    First, the compound is a di-substituted 6-membered aromatic ring with 8 carbon atoms. That means the aromatic ring is a benzene ring, but since it's di-substituted, there are two substituents. The ring has
+    
+    ... [truncated 822 chars] ...
+    
+    th aromatic protons.
+    
+    - **Option B**:  
+      - Aromatic protons at **7.8 (2H, d)** and **7.6 (2H, d)** — again, consistent with **two aromatic protons**.  
+      - The **s at 2.3 ppm** is consistent with **two CH3 groups** (each at 2.3 ppm).  
+      - The **carbonyl group** is at **1700–1750 ppm**, which is typical for a carbonyl group.
+      - This configuration is **chemically plausible** for a substituted aromatic ring with **two substituents**, and the **aromatic protons are double-decoupled**, matching the expected pattern for a di-substituted aromatic ring.
+    
+    - **Option C**:  
+      - Aromatic protons at **4.8 (2H, d)** and **4.6 (2H, d)** — **too low** for aromatic protons (typically around 6.5–8.5 ppm), so inconsistent with aromatic structure.
+    
+    - **Option D**:  
+      - Similar to Option C, aromatic protons are **too low** for aromatic protons and **not consistent** with the presence of a carbonyl group.
+    
+    ---
+    
+    ### Conclusion:
+    
+    Option **B** provides the most chemically consistent and plausible NMR data for a **di-substituted aromatic ring** with a **carbonyl group** and an **aromatic-halogen bond**. It matches all expected chemical shifts and splitting patterns for such a compound.
+    
+    $$
+    \boxed{B}
+    $$
+    ```
+
+**Incorrect** (1 shown)
+
+- **reference**: `I`
+- **overall** (0/1 completions correct)
+- **extracted** (✗): `F`
+- **completion**:
+
+    ```text
+    <think>
+    Okay, let's see. The question is about executory contracts and which statement is least correct. First, I need to remember the definitions related to executory contracts. 
+    
+    An executory contract is one where one party has already performed, and the other hasn't yet begun. So, for example, if Party A has delivered a product to Party B, but Party B hasn't accepted it yet. That would be an ex
+    
+    ... [truncated 822 chars] ...
+    
+    yet begun performance.*  
+      ✅ Correct. This matches the definition of an executory contract.
+    
+    - **D.** *A wholly executory contract is where only promises have been exchanged and there has been no performance by either party.*  
+      ✅ Correct. This is the definition of a wholly executory contract.
+    
+    - **E.** *A partially executed contract means that at least one party has begun performance.*  
+      ✅ Correct. This is consistent with the definition of a partially executed contract.
+    
+    - **F.** *An executory contract is when both parties have yet to begin performance.*  
+      ❌ **Incorrect.** An executory contract is not one where **both parties have yet to begin** performance. It is one where **one has begun**, and the **other has not**. Therefore, this statement is **false**.
+    
+    - **G.** *A fully executed contract is when both parties have completed their performance.*  
+      ✅ Correct. This is the definition of a fully executed contract.
+    
+    ---
+    
+    ### Conclusion:
+    
+    Among all the options, **F** is the **least correct** because it incorrectly defines an executory contract. It suggests that both parties have not yet begun performance, which is **not** the case for an executory contract.
+    
+    $$
+    \boxed{F}
+    $$
+    ```