Automated MNLP evaluation report (2026-06-11)

EVAL_REPORT.md

@@ -0,0 +1,83 @@
+# Automated MNLP evaluation report
+
+- **Model repo:** [`cs-552-2026-ChatMODS/multilingual_model`](https://huggingface.co/cs-552-2026-ChatMODS/multilingual_model)
+- **Owner(s):** group **ChatMODS**
+- **Generated at:** 2026-06-11T06:23:10+00:00 (UTC)
+- **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
+
+_This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
+
+## Evaluated checkpoint
+
+- **Commit:** [`e43b15a`](https://huggingface.co/cs-552-2026-ChatMODS/multilingual_model/commit/e43b15a585b22ae5131179087948f443861ac6a0)
+- **Message:** SFT-synth epoch3: base + synthetic Wikipedia QCMs (10124 traces) - testing CI
+- **Committed:** 2026-06-08T16:48:35+00:00
+
+## Summary
+
+| Benchmark | Accuracy | Status |
+|---|---:|---|
+| Math | — | not run |
+| Knowledge | — | not run |
+| Multilingual | 0.6400 | ok |
+| Safety | — | not run |
+
+## Sample completions
+
+_Prompts are intentionally omitted to avoid revealing benchmark contents. For multi-completion problems, only one completion is shown per sample._
+
+### Multilingual
+
+**Correct** (1 shown)
+
+- **reference**: `A`
+- **overall** (1/1 completions correct)
+- **extracted** (✓): `A`
+- **completion**:
+
+    ```text
+    <think>Okay, let's tackle this question. The passage is from William Lloyd Garrison in 1831, arguing that free people of color in the U.S. should be citizens entitled to all rights, and that current laws treating them as foreigners are unconstitutional. He's challenging the legality of their treatment, which implies he's advocating for their full citizenship.
+    
+    The question asks which Supreme Court decision later contradicted Garrison's argument. Let's look at the options:
+    
+    A. Dred Scott case (1857) ruled that African Americans couldn't be citizens and had no standing to sue in federal court. This directly contradicts the idea that free people of color are citizens. So this might be the answer.
+    
+    B. Brown v. Board of Education (1954) ended segregation, but it's about racial equality and education, not citizenship. So maybe not directly contradicting Garrison's argument about citizenship.
+    
+    C. Plessy v. Ferguson (1896) established "separate but equal," which upheld racial segregation. This is about equality but not citizenship. So not directly contradicting the argument for citizenship.
+    
+    D. Ex parte Milligan (1866) was about the trial of a Confederate general during the Civil War, not directly related to citizenship or equality.
+    
+    So the key point is that Garrison's argument was about citizenship and legal rights. Dred Scott directly contradicted the idea that African Americans were citizens. The other cases either dealt with segregation or different issues. Therefore, the answer should be A.</think>
+    
+    \boxed{A}
+    ```
+
+**Incorrect** (1 shown)
+
+- **reference**: `C`
+- **overall** (0/1 completions correct)
+- **extracted** (✗): `B`
+- **completion**:
+
+    ```text
+    <think>Okay, let's tackle this question. The user is asking about the Pleistocene epoch and a specific period when sea levels were very low, causing a landmass to emerge in Siberia's eastern part. The options are about different landmasses and time periods.
+    
+    First, I need to recall geological terms related to Pleistocene sea levels. The key here is the sea level being much lower, which would expos
+    
+    ... [truncated 55 chars] ...
+    
+    part is likely referring to the region now known as Beringia, which was a land bridge during the Pleistocene. 
+    
+    Looking at the options, option B mentions Beringia and the "Beringian Maximum." Wait, the term "Beringian Maximum" sounds familiar. I think that's the period when Beringia was the land bridge, and sea levels were low, allowing people to migrate between Asia and North America. 
+    
+    Option A mentions Sundaland, which I associate with the lower part of the Indian subcontinent, not Siberia. Sundaland is more related to the Indian subcontinent's lower regions, so that's probably not the right answer. 
+    
+    Option C refers to Laurentia and the Late Glacial Maximum. Laurentia is a continent, but the Late Glacial Maximum is a time period, not a landmass. Also, the question is about a landmass emerging, not a climate period. 
+    
+    Option D mentions Vaalispia, which I don't recall being a significant landmass in Siberia. Vaalispia is more related to the African part of the continent during the Pleistocene, but not Siberia. 
+    
+    So, the correct answer should be B: Beringia and Beringian Maximum. The other options don't fit the geographical context or the time period described.</think>
+    
+    \boxed{B}
+    ```