| ``` |
| license: mit |
| metrics: |
| - accuracy |
| - f1 |
| - precision |
| - recall |
| base_model: |
| - microsoft/unixcoder-base |
| library_name: transformers |
| tags: |
| - detection |
| - AI-generated |
| - transformers |
| - bert |
| ``` |
|
|
| ## Task Overview |
|
|
| The rapid advancement of generative models has made it increasingly challenging to distinguish machine-generated code from human-written code, particularly across different programming languages, domains, and generation techniques. |
|
|
| SemEval-2026 Task 13 focuses on developing systems capable of detecting machine-generated code under diverse conditions. The evaluation emphasizes generalization to unseen programming languages, generator families, and application scenarios. |
|
|
| The task is divided into three subtasks. |
|
|
| --- |
|
|
| ### Subtask A: Binary Machine-Generated Code Detection |
|
|
| **Goal:** |
| Given a code snippet, determine whether it is: |
|
|
| - Fully human-written, or |
| - Fully machine-generated |
|
|
| **Training Languages:** C++, Python, Java |
| **Training Domain:** Algorithmic (e.g., LeetCode-style problems) |
|
|
| **Evaluation Settings:** |
|
|
| | Setting | Language | Domain | |
| |--------------------------------------|-------------------------|----------------------| |
| | (i) Seen Languages & Seen Domains | C++, Python, Java | Algorithmic | |
| | (ii) Unseen Languages & Seen Domains | Go, PHP, C#, C, JS | Algorithmic | |
| | (iii) Seen Languages & Unseen Domains| C++, Python, Java | Research, Production | |
| | (iv) Unseen Languages & Domains | Go, PHP, C#, C, JS | Research, Production | |
|
|
| **Dataset Size:** |
| - Train: 500,000 samples (238,000 human-written, 262,000 machine-generated) |
| - Validation: 100,000 samples |
|
|
| **Data Format:** |
| Each dataset includes the following fields: |
| - `code`: The code snippet |
| - `label`: Binary label (0 for human-written, 1 for machine-generated) |
| - `language`: Programming language of the snippet |
|
|
| Label mappings are provided in `task_A/label_to_id.json` and `task_A/id_to_label.json`. |
|
|
| **Evaluation Metric:** |
| The primary metric for Subtask A is Macro F1-score, ensuring balanced performance across both classes. |
|
|
| **Submission Format:** |
| Participants must submit a `.csv` file containing: |
| - `id`: Unique identifier for each code snippet |
| - `label`: Predicted label (0 or 1) |
|
|
| A sample submission file is available in the `task_A/` directory. |
|
|
| **Baseline Models:** |
| Baseline implementations are provided in the `baselines/` directory, including starter code and pre-trained checkpoints for models such as GraphCodeBERT and UniXcoder. |
|
|
| **Restrictions:** |
| - No external training data may be used; only the provided datasets are allowed. |
| - Specialized AI-generated code detectors are not permitted. General-purpose code models (e.g., CodeBERT, StarCoder) are allowed. |
|
|
