| --- |
| license: mit |
| datasets: |
| - DaniilOr/SemEval-2026-Task13 |
| metrics: |
| - accuracy |
| - f1 |
| - precision |
| - recall |
| base_model: |
| - microsoft/unixcoder-base |
| library_name: transformers |
| tags: |
| - detection |
| - AI-generated |
| - transformers |
| - bert |
| --- |
| ## 🔍 Task Overview |
|
|
| The rise of generative models has made it increasingly difficult to distinguish machine-generated code from human-written code — especially across different programming languages, domains, and generation techniques. |
|
|
| **SemEval-2026 Task 13** challenges participants to build systems that can **detect machine-generated code** under diverse conditions by evaluating generalization to unseen languages, generator families, and code application scenarios. |
|
|
| The task consists of **three subtasks**: |
|
|
| --- |
|
|
| ### Subtask A: Binary Machine-Generated Code Detection |
|
|
| **Goal:** |
| Given a code snippet, predict whether it is: |
|
|
| - **(i)** Fully **human-written**, or |
| - **(ii)** Fully **machine-generated** |
|
|
| **Training Languages:** `C++`, `Python`, `Java` |
| **Training Domain:** `Algorithmic` (e.g., Leetcode-style problems) |
|
|
| **Evaluation Settings:** |
|
|
| | Setting | Language | Domain | |
| |--------------------------------------|-------------------------|------------------------| |
| | (i) Seen Languages & Seen Domains | C++, Python, Java | Algorithmic | |
| | (ii) Unseen Languages & Seen Domains | Go, PHP, C#, C, JS | Algorithmic | |
| | (iii) Seen Languages & Unseen Domains| C++, Python, Java | Research, Production | |
| | (iv) Unseen Languages & Domains | Go, PHP, C#, C, JS | Research, Production | |
|
|
| **Dataset Size**: |
| - Train - 500K samples (238K Human-Written | 262K Machine-Generated) |
| - Validation - 100K samples |
|
|
| **Data Format** |
| Each dataset contains the following fields: |
| - `code`: The code snippet |
| - `label`: The binary label (0 for human-written, 1 for machine-generated) |
| - `language`: The programming language of the snippet |
|
|
| Label mappings are provided in `task_A/label_to_id.json` and `task_A/id_to_label.json`. |
|
|
| **Evaluation Metric** |
| The primary evaluation metric for Subtask A is **Macro F1-score**. This metric ensures balanced performance across both classes. |
|
|
| **Submission Format** |
| Participants must submit a `.csv` file with the following columns: |
| - `id`: Unique identifier for each code snippet |
| - `label`: Predicted label (0 or 1) |
|
|
| A sample submission file is available in the `task_A/` folder. |
|
|
| **Baseline Models** |
| Baseline implementations for Subtask A are provided in the `baselines/` directory. These include starter code and pre-trained checkpoints for models such as GraphCodeBERT and UniXcoder. |
|
|
| **Restrictions** |
| - **No external training data**: Use only the provided datasets. |
| - **No specialized AI-generated code detectors**: General-purpose code models (e.g., CodeBERT, StarCoder) are allowed. |
|
|
