Automated MNLP evaluation report (2026-05-13)

#2
Files changed (1) hide show
  1. EVAL_REPORT.md +19 -7
EVAL_REPORT.md CHANGED
@@ -2,7 +2,7 @@
2
 
3
  - **Model repo:** [`cs-552-2026-catma/general_knowledge_model`](https://huggingface.co/cs-552-2026-catma/general_knowledge_model)
4
  - **Owner(s):** group **catma**
5
- - **Generated at:** 2026-05-13T04:17:52+00:00 (UTC)
6
  - **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
7
 
8
  _This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
@@ -24,22 +24,34 @@ _Prompts are intentionally omitted to avoid revealing benchmark contents. For mu
24
 
25
  **Correct** (1 shown)
26
 
27
- - **reference**: `C`
28
  - **overall** (1/1 completions correct)
29
- - **extracted** (✓): `C`
30
  - **completion**:
31
 
32
  ```text
33
- \boxed{C}
34
  ```
35
 
36
  **Incorrect** (1 shown)
37
 
38
- - **reference**: `D`
39
  - **overall** (0/1 completions correct)
40
- - **extracted** (✗): `C`
41
  - **completion**:
42
 
43
  ```text
44
- \boxed{C}
 
 
 
 
 
 
 
 
 
 
 
 
45
  ```
 
2
 
3
  - **Model repo:** [`cs-552-2026-catma/general_knowledge_model`](https://huggingface.co/cs-552-2026-catma/general_knowledge_model)
4
  - **Owner(s):** group **catma**
5
+ - **Generated at:** 2026-05-13T23:30:17+00:00 (UTC)
6
  - **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
7
 
8
  _This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
 
24
 
25
  **Correct** (1 shown)
26
 
27
+ - **reference**: `G`
28
  - **overall** (1/1 completions correct)
29
+ - **extracted** (✓): `G`
30
  - **completion**:
31
 
32
  ```text
33
+ \boxed{G}
34
  ```
35
 
36
  **Incorrect** (1 shown)
37
 
38
+ - **reference**: `C`
39
  - **overall** (0/1 completions correct)
40
+ - **extracted** (✗): `<no answer>`
41
  - **completion**:
42
 
43
  ```text
44
+ C. 1,2,4
45
+
46
+ **Explanation:**
47
+
48
+ In the SMEFT (Standard Model Extension with Fermion Mass Terms), the fundamental symmetries that must be obeyed by every operator are:
49
+
50
+ - **Lorentz symmetry**: This is the symmetry of spacetime, ensuring that the laws of physics are the same in all inertial frames.
51
+ - **Poincaré symmetry**: This includes Lorentz symmetry and translations, ensuring that the laws of physics are invariant under changes in space and time.
52
+ - **CPT symmetry**: This is a combination of charge conjugation (C), parity (P), and time reversal (T), and it is a fundamental symmetry of the Standard Model.
53
+
54
+ **CP symmetry** is not required to be obeyed by every operator in SMEFT, as it is not a fundamental symmetry of the Standard Model and is not required in the SMEFT framework. Instead, SMEFT allows for the breaking of CP symmetry in certain operators, but it does not require CP symmetry to be preserved in all operators.
55
+
56
+ Thus, the correct answer is **C. 1,2,4**.
57
  ```