Automated MNLP evaluation report (2026-06-11)

.gitattributes

@@ -33,4 +33,3 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
-tokenizer.json filter=lfs diff=lfs merge=lfs -text

EVAL_REPORT.md

@@ -0,0 +1,96 @@
+# Automated MNLP evaluation report
+
+- **Model repo:** [`cs-552-2026-flab/math_model`](https://huggingface.co/cs-552-2026-flab/math_model)
+- **Owner(s):** group **flab**
+- **Generated at:** 2026-06-11T06:23:10+00:00 (UTC)
+- **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
+
+_This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
+
+## Evaluated checkpoint
+
+- **Commit:** [`19cb971`](https://huggingface.co/cs-552-2026-flab/math_model/commit/19cb971ecbfdfb455c8f90bb6b83cd7921a40c68)
+- **Message:** g2rpo-guided-ds4-8g-ckpt200-tuned-sampling-T0.6-topp1.0-topk-1
+- **Committed:** 2026-06-09T09:22:40+00:00
+
+## Summary
+
+| Benchmark | Accuracy | Status |
+|---|---:|---|
+| Math | 0.3800 | ok |
+| Knowledge | — | not run |
+| Multilingual | — | not run |
+| Safety | — | not run |
+
+## Sample completions
+
+_Prompts are intentionally omitted to avoid revealing benchmark contents. For multi-completion problems, only one completion is shown per sample._
+
+### Math
+
+**Correct** (1 shown)
+
+- **reference**: `194`
+- **overall** (5/8 completions correct)
+- **extracted** (✓): `194`
+- **completion**:
+
+    ```text
+    <think>
+    Okay, so I need to figure out the minimum number of cells to mark in a 194x194 grid so that there's exactly one tiling with dominoes where each domino has at most one marked cell. Hmm. Let me start by understanding the problem better.
+    
+    First, domino tiling of a grid. Each domino is either 1x2 or 2x1. The problem is about placing dominoes such that each domino has at most one marked cell. S
+    
+    ... [truncated 822 chars] ...
+    
+    s) to ensure each domino can contain at most one marked cell.
+       - The minimal number of marked cells required to force a unique tiling is related to the structure of the grid and the constraints of domino tiling.
+    
+    3. **Smaller Grid Analysis**: For smaller grids like \( 2 \times 2 \) or \( 4 \times 4 \), the minimal number of marked cells needed to force a unique tiling is equal to the number of rows or columns. This suggests a pattern where marking one cell per row (or column) can force a unique tiling.
+    
+    4. **Generalization to \( 194 \times 194 \)**:
+       - By marking one cell in each row such that no two marked cells are in the same column, we ensure that each marked cell is part of a vertical domino.
+       - This forces vertical dominoes in each column, which are unique due to the marking constraints.
+       - The horizontal dominoes can be placed between the marked cells and unmarked cells, but the unique vertical domino placements ensure a unique tiling.
+    
+    5. **Conclusion**: The minimal number of marked cells required is equal to the number of rows (or columns) in the grid. For a \( 194 \times 194 \) grid, this number is 194.
+    
+    Thus, the least positive integer \( k \) is \(\boxed{194}\).
+    ```
+
+**Incorrect** (1 shown)
+
+- **reference**: `$g(x)=c, g(x)=\lceil x \rceil, g(x)=\lfloor x \rfloor$`
+- **overall** (0/8 completions correct)
+- **extracted** (✗): `h(x) = c`
+- **completion**:
+
+    ```text
+    <think>
+    Okay, let's try to tackle this problem. So, we need to find all functions \( h: \mathbb{Q} \to \mathbb{Z} \) such that for every \( x \in \mathbb{Q} \), every integer \( a \), and every positive integer \( b \), the equation \( h(x) = h\left( \frac{h(bx - a) + a}{b} \right) \) holds. Hmm, that's a mouthful. Let me break it down.
+    
+    First, let's parse the equation. For any rational number \(
+    
+    ... [truncated 823 chars] ...
+    
+    d properties of the function \( h \).
+    
+    1. **Constant Functions**: 
+       - Suppose \( h(x) = c \) for some constant \( c \in \mathbb{Z} \). Substituting this into the equation, we get:
+         \[
+         h(x) = h\left( \frac{c + a}{b} \right) = c,
+         \]
+         which is always true. Thus, constant functions are solutions.
+    
+    2. **Non-Constant Functions**:
+       - If \( h \) were non-constant, there would exist \( x_1 \) and \( x_2 \) such that \( h(x_1) \neq h(x_2) \). Using the given functional equation, we can derive that \( h(x) \) must be constant for all \( x \). Specifically, setting \( x = 0 \) and analyzing the resulting equation shows that \( h \) must be constant everywhere.
+    
+    3. **Key Insight**:
+       - By setting \( x = 0 \) and considering the equation for all \( a \) and \( b \), we find that \( h(0) \) must equal \( h\left( \frac{h(-a) + a}{b} \right) \). This implies that \( h \) must be constant on the set of all rational numbers of the form \( \frac{c}{b} \), which is dense in \( \mathbb{Q} \). Therefore, \( h \) must be constant everywhere.
+    
+    Thus, the only solutions are constant functions. The final answer is:
+    
+    \[
+    \boxed{h(x) = c} \text{ where } c \text{ is an integer constant.}
+    \]
+    ```

README.md

@@ -1,57 +1,3 @@
----
-library_name: transformers
-model_name: '20260529_093919'
-tags:
-- generated_from_trainer
-- sft
-- trl
-licence: license
----
-
-# Model Card for 20260529_093919
-
-This model is a fine-tuned version of [None](https://huggingface.co/None).
-It has been trained using [TRL](https://github.com/huggingface/trl).
-
-## Quick start
-
-```python
-from transformers import pipeline
-
-question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
-generator = pipeline("text-generation", model="None", device="cuda")
-output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
-print(output["generated_text"])
-```
-
-## Training procedure
-
-[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/cs-552-2026-flab/flab-cs552/runs/cwg4aje3) 
-
-
-
-This model was trained with SFT.
-
-### Framework versions
-
-- TRL: 1.3.0
-- Transformers: 5.7.0
-- Pytorch: 2.10.0+cu128
-- Datasets: 4.8.5
-- Tokenizers: 0.22.2
-
-## Citations
-
-
-
-Cite TRL as:
-    
-```bibtex
-@software{vonwerra2020trl,
-  title   = {{TRL: Transformers Reinforcement Learning}},
-  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
-  license = {Apache-2.0},
-  url     = {https://github.com/huggingface/trl},
-  year    = {2020}
-}
-```
+---
+license: apache-2.0
+---

chat_template.jinja

@@ -1,91 +0,0 @@
-{%- set enable_thinking = true %}
-{%- set flab_boxed_instruction = "You are an expert mathematician. Reason carefully and rigorously: recall the relevant definitions and theorems, try small cases or sanity checks, and verify each step before continuing. When you reach the answer, state it on its own line as \\boxed{...}. The grader reads ONLY the contents of the LAST \\boxed{} in your response, so your final answer must appear there, complete and simplified." %}
-{%- if flab_boxed_instruction %}{{ '<|im_start|>system\n' + flab_boxed_instruction + '<|im_end|>\n' }}{%- endif %}{%- if tools %}
-    {{- '<|im_start|>system\n' }}
-    {%- if messages[0].role == 'system' %}
-        {{- messages[0].content + '\n\n' }}
-    {%- endif %}
-    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
-    {%- for tool in tools %}
-        {{- "\n" }}
-        {{- tool | tojson }}
-    {%- endfor %}
-    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
-{%- else %}
-    {%- if messages[0].role == 'system' %}
-        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
-    {%- endif %}
-{%- endif %}
-{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
-{%- for message in messages[::-1] %}
-    {%- set index = (messages|length - 1) - loop.index0 %}
-    {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
-        {%- set ns.multi_step_tool = false %}
-        {%- set ns.last_query_index = index %}
-    {%- endif %}
-{%- endfor %}
-{%- for message in messages %}
-    {%- if message.content is string %}
-        {%- set content = message.content %}
-    {%- else %}
-        {%- set content = '' %}
-    {%- endif %}
-    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
-        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
-    {%- elif message.role == "assistant" %}
-        {%- set reasoning_content = '' %}
-        {%- if message.reasoning_content is string %}
-            {%- set reasoning_content = message.reasoning_content %}
-        {%- else %}
-            {%- if '</think>' in content %}
-                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
-                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
-            {%- endif %}
-        {%- endif %}
-        {%- if loop.index0 > ns.last_query_index %}
-            {%- if loop.last or (not loop.last and reasoning_content) %}
-                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
-            {%- else %}
-                {{- '<|im_start|>' + message.role + '\n' + content }}
-            {%- endif %}
-        {%- else %}
-            {{- '<|im_start|>' + message.role + '\n' + content }}
-        {%- endif %}
-        {%- if message.tool_calls %}
-            {%- for tool_call in message.tool_calls %}
-                {%- if (loop.first and content) or (not loop.first) %}
-                    {{- '\n' }}
-                {%- endif %}
-                {%- if tool_call.function %}
-                    {%- set tool_call = tool_call.function %}
-                {%- endif %}
-                {{- '<tool_call>\n{"name": "' }}
-                {{- tool_call.name }}
-                {{- '", "arguments": ' }}
-                {%- if tool_call.arguments is string %}
-                    {{- tool_call.arguments }}
-                {%- else %}
-                    {{- tool_call.arguments | tojson }}
-                {%- endif %}
-                {{- '}\n</tool_call>' }}
-            {%- endfor %}
-        {%- endif %}
-        {{- '<|im_end|>\n' }}
-    {%- elif message.role == "tool" %}
-        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
-            {{- '<|im_start|>user' }}
-        {%- endif %}
-        {{- '\n<tool_response>\n' }}
-        {{- content }}
-        {{- '\n</tool_response>' }}
-        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
-            {{- '<|im_end|>\n' }}
-        {%- endif %}
-    {%- endif %}
-{%- endfor %}
-{%- if add_generation_prompt %}
-    {{- '<|im_start|>assistant\n' }}
-    {%- if enable_thinking is defined and enable_thinking is false %}
-        {{- '<think>\n\n</think>\n\n' }}
-    {%- endif %}
-{%- endif %}

completions/completions_00001.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:f6be364352a9089981f9fec6118939e8afd83700f4b47680d338f55a7d4bd567
-size 185867

completions/completions_00002.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:bc32dac308471db150ee75a3cd566549e265a50f758dbe8bb635eafbc7163887
-size 99403

completions/completions_00003.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:e4b3590d7f4f7bd3df54915fda0a993fb6722c3e471b66cf0d4b1600189dea86
-size 92470

completions/completions_00004.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:2812ea7b083398d4d5f62fbc34932a60a931efc408a19193f4363d2368115677
-size 87440

completions/completions_00005.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:3063a9f56c99ca95decd174d718eafaf1a2be6742a902fc4062d3ffb59d7c7f0
-size 130955

completions/completions_00006.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:1bb6e8d16cebfa160563a98ea2dc927c3fa8893b26cf8bf772f3de85ba35b726
-size 81139

completions/completions_00007.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:64ea5fe94ae52d2092b0c44ba5a7311eaf2d321748a22f4dd3fdfdd3cb485c40
-size 202224

completions/completions_00008.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:06b40db60bddb173d482b4d4c8b020af2b1c381ad3b5498d16959b9069b78cc2
-size 91417

completions/completions_00009.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:51f6ad7b624d3fa3e0a52cfdcb12b810c1ff1016e02385f8bc4d1b3196b217a1
-size 169824

completions/completions_00010.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:ae8c3a0b29e9a10de7b1e1c0bdc5e0a8b35bc2f169c6f90b1713ddbd313d521b
-size 96383

completions/completions_00011.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:27bed05dc4f8cd49ec2a3ce896e320f54fb2867fae6ba450ee2a93c92f9688f3
-size 81976

completions/completions_00012.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:007eaf5fc077fc2fd9b3fe7569db1252b76e84fd9a2d9e26bc0e6d56bb4d4c31
-size 133298

completions/completions_00013.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:099bc0a7fb333def5703c77300cdb2ccd9a2f4e4d8819d5bff3f9a988cfe950c
-size 93140

completions/completions_00014.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:053cd354371a476a3212829bf13553297a7380351521f82257b83c2dc64c8ac0
-size 119536

completions/completions_00015.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:3f14fd480b3bf89ea5b8990e7fabb2962b0e59a8cc745c97076d80df018a59e2
-size 104520

completions/completions_00016.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:578ea0eff571612832bf0ec78cf11e39b92fa549e42dcdf72ecc6612972ddff5
-size 79169

completions/completions_00017.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:213d1ecb25833627c135b61d7718fd2de2aba8ea55aa5e472ab06513aa1d5931
-size 62960

completions/completions_00018.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:85dc5381314434c9b9bd186ff0b4e14711b265a4e8466314eb9963b54511fec3
-size 157319

completions/completions_00019.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:5ed10b00be237d5830ec9d04c9bb28be637b66221c7c5b70f662195e22390be1
-size 71138

completions/completions_00020.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:dcef2b31c8d9467fec7908dd7a1f09e36d7bc5a84888dcb56217dee34c176dbf
-size 197317

completions/completions_00021.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:b6819ebb85fe16437d0037a536abf329bfe7e79868843d46f52fd596b240d0d4
-size 141682

completions/completions_00022.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:d721e852b2bb2bf01ec5b513f8c74be38157cf11dfdefdacb34e136298f57188
-size 123491

completions/completions_00023.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:ba8f3348664c65a4d7dbdf96325fcb803720fa50f718d2bfa319437960ca1d39
-size 122864

completions/completions_00024.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:c8fbdddffc9acb1411b22d900cb49be468286c82798d62ae8fc583a4c0ec1ba5
-size 112258

completions/completions_00025.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:ef1f52a5539eec3bbddc01559fe9ab40b747c7b9730e6594c10f71cf84f011f0
-size 61162

completions/completions_00026.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:53dbd50ebefd29a5f8f2732aec8941f6a8fe779e4b097f9b9e56676d24ddbbd2
-size 73280

completions/completions_00027.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:02dc5b03893df9a80d0810a373bb4bcdc1ccc510e46f5dacd75292cffed4cd4e
-size 106272

completions/completions_00028.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:fc3678aabc72de61722fc02d1cc19d3918734a2f7ab853205abff62f7cfe5d94
-size 99352

completions/completions_00029.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:9bee4d54aea341c68e0bba4f7c688e829fc6e915f503e92bb406a39ccec3f747
-size 109502

completions/completions_00030.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:96e9738421aa2d46cfb87f53770f561f5f5a928465bfe634a07b9e97eecac825
-size 114211

completions/completions_00031.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:01f050ffbdf5fa54031101896e00ce0b34793a0752850fc258359afae7e483f1
-size 59562

completions/completions_00032.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:e3cbeac19f5d10166d8a803cd2b6d2e61a6a9397d4b24bb25bb00796cccce5e4
-size 128485

completions/completions_00033.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:aaa73bcd6ff87464b78d5ca1dfda1d5017551caa6ff653ec1469fa14590eebcb
-size 107557

completions/completions_00034.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:ca55821f7f8eed98312c895cadcc31c3429ce84dfa451d571d83f3f58ed3d313
-size 109354

completions/completions_00035.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:8bc0f0d214395ac605a871ed69f08296ba177f0387de786c8c1fa1c1583f9f25
-size 70236

completions/completions_00036.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:150e0f1eeb4934eb427050e2cc7b53b8828e143f39e40ce993babd24dca38c12
-size 55458

completions/completions_00037.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:234f7633bba9674a0324729326dc4e7fcabe881d4b6fb73af1fb28962dbac26b
-size 86243

completions/completions_00038.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:7d59c3e43abd1e866ce9f1348804bd86eda62fe38e09f3f05cc66eb4e6d5420b
-size 155854

completions/completions_00039.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:e467ca004028d8ff54f72fdb73be2e0e8f7c6578084564ac9c5e3fe4c2c735a9
-size 176351

completions/completions_00040.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:89fe088636c01f9cb4de1fddaa75042bc86c81005e2f7ac4b257b68f85d14250
-size 67041

completions/completions_00041.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:69964811b7b7d42efd0b5e54abb1d25a1cfbe227ceab33148acc647d2a13c1ba
-size 72313

completions/completions_00042.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:6b2f32d584b699cd2a55298eca086a0c08a3ca0f5478d289b907f16bf7ef1566
-size 125863

completions/completions_00043.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:5697004c83ca584c0248bbfbed9e2cf5b7948010cc21191c456231c8dd8d5308
-size 84350

completions/completions_00044.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:d4782dbfa6749fa3fedcf9077956a66f5aa4133b2b8af825cc7e10b3ae12482a
-size 78064

completions/completions_00045.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:7209b37bada45c1f64f73116bc742ee8f5ab408b1c8671f389220314da0e5617
-size 143317

completions/completions_00046.parquet

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:102f34cab49ff28fd5b67953c76985fc1e13bf24c11ebcf7dbdd3f559d32d892
-size 64131