Automated MNLP evaluation report (2026-06-11)

EVAL_REPORT.md

@@ -0,0 +1,80 @@
+# Automated MNLP evaluation report
+
+- **Model repo:** [`cs-552-2026-barn/multilingual_model`](https://huggingface.co/cs-552-2026-barn/multilingual_model)
+- **Owner(s):** group **barn**
+- **Generated at:** 2026-06-11T06:23:10+00:00 (UTC)
+- **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
+
+_This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
+
+## Evaluated checkpoint
+
+- **Commit:** [`bd9c025`](https://huggingface.co/cs-552-2026-barn/multilingual_model/commit/bd9c0254ede4d7cd6a730edcd1f9337d3e2b23a8)
+- **Message:** same model, tmp =0.3
+- **Committed:** 2026-06-09T18:41:56+00:00
+
+## Summary
+
+| Benchmark | Accuracy | Status |
+|---|---:|---|
+| Math | — | not run |
+| Knowledge | — | not run |
+| Multilingual | 0.4400 | ok |
+| Safety | — | not run |
+
+## Sample completions
+
+_Prompts are intentionally omitted to avoid revealing benchmark contents. For multi-completion problems, only one completion is shown per sample._
+
+### Multilingual
+
+**Correct** (1 shown)
+
+- **reference**: `A`
+- **overall** (1/1 completions correct)
+- **extracted** (✓): `A`
+- **completion**:
+
+    ```text
+    <think>
+    Okay, let's try to figure out this question. The user is asking which achievement inspired the stories about Yuanzhang's travels from the 600s and 1500s. The options are A to D.
+    
+    First, I need to recall the information given. The passage talks about Yuanzhang, a Buddhist monk who was chosen from monks. He started eating vegetarian food and followed strict rules. His family background inclu
+    
+    ... [truncated 823 chars] ...
+    
+    g to spread the teachings. However, the options don't directly mention traveling to India. But maybe the answer is A, even though it's not explicitly stated. Alternatively, maybe the answer is D, but that's about Christianity, which isn't mentioned. 
+    
+    Wait, the passage says he wanted to print Buddhist scriptures and free the Bodhisattva of the Deer King to go with him. The Bodhisattva of the Deer King is a Buddhist figure, so this is about spreading Buddhism. The stories of his travels would be inspired by the desire to spread Buddhist teachings, which would involve studying and finding texts in India, but the passage doesn't explicitly say he went there. However, the question is about what achievements inspired the stories. The stories might be inspired by the idea of spreading Buddhism, which would involve studying texts in India. So maybe A is the answer. 
+    
+    Alternatively, the answer could be D, but that's about Christianity, which isn't mentioned. So I think A is the correct answer here. The key is that the stories are about his travels and the desire to spread Buddhism, which would involve studying and finding texts in India. So the answer is A.
+    </think>
+    
+    **Ответ:**  
+    \boxed{A}
+    ```
+
+**Incorrect** (1 shown)
+
+- **reference**: `A`
+- **overall** (0/1 completions correct)
+- **extracted** (✗): `C`
+- **completion**:
+
+    ```text
+    <think>
+    Okay, let's try to figure out this question. So, the scenario is about a train car ad showing comfortable seats and a lot of space, but when a customer sits down, they find lack of space and stiff seats. The question is asking which type of gap this is, with options A to D.
+    
+    First, I need to recall what each of these gaps refers to. From what I remember, there are different types of gaps i
+    
+    ... [truncated 822 chars] ...
+    
+    ad). That's Gap 4. But Gap 4 is between the service provided and the external communication. However, in this case, the ad is the external communication, and the service provided is different. So if the ad is the external communication, and the actual service is different, then it's Gap 4. But I'm confused because the ad is part of the service specification. So maybe the service specification is the ad, and the actual service is different. So that's Gap 3.
+    
+    Alternatively, maybe the problem is that the customer's expectations (based on the ad) are not met. So that's Gap 1. But the ad is the external communication, so the customer's expectations are based on the ad. But the actual service is different. So if the ad is the service specification, then the actual service is different, which is Gap 3. So the answer would be C.
+    
+    But I need to check the definitions again. Gap 1 is between customer expectations and the service provided. Gap 3 is between the service specifications (what's supposed to be provided) and the actual service. So if the ad is the service specification, and the actual service is different, then it's Gap 3. Therefore, the answer is C.
+    </think>
+    
+    **Answer:** \boxed{C}
+    ```