README.md

---
license: mit
library_name: transformers
pipeline_tag: text-classification
tags:
  - code
  - clone-detection
  - graphcodebert
  - code-similarity
base_model: microsoft/graphcodebert-base
datasets:
  - PoolC/1-fold-clone-detection-600k-5fold
language:
  - code
metrics:
  - accuracy
  - precision
  - recall
  - f1
model-index:
  - name: graphcodebert-code-clone-detection
    results:
      - task:
          type: text-classification
          name: Binary code clone detection
        dataset:
          type: PoolC/1-fold-clone-detection-600k-5fold
          name: PoolC/1-fold-clone-detection-600k-5fold
          split: test (group-disjoint half of the `val` fold)
        metrics:
          - type: f1
            value: 0.8805
          - type: accuracy
            value: 0.8747
          - type: precision
            value: 0.841
          - type: recall
            value: 0.924
---

# graphcodebert-code-clone-detection

Binary code-clone detection. Full fine-tune of
[`microsoft/graphcodebert-base`](https://huggingface.co/microsoft/graphcodebert-base) on
[`PoolC/1-fold-clone-detection-600k-5fold`](https://huggingface.co/datasets/PoolC/1-fold-clone-detection-600k-5fold),
using GraphCodeBERT's data-flow-aware pairwise architecture.

Output labels: `0 = not clone`, `1 = clone`.

## Architecture

Not a generic sequence-pair classifier. The two snippets are encoded
**separately** by one shared GraphCodeBERT encoder, each with its own
graph-guided masked attention, and the two `<s>` vectors are concatenated for
classification:

```
Linear(2 x 768 -> 768) -> tanh -> Linear(768 -> 2)
```

Per-snippet input layout (length 640):

| segment | length | content | `position_idx` |
|---|---|---|---|
| code tokens | 512 | `<s>` + BPE code tokens + `</s>` | `2 .. n+1` |
| data-flow nodes | 128 | one slot per DFG variable node (`<unk>` id) | `0` |
| padding | remainder | `<pad>` | `1` |

A data-flow node's input embedding is the **average of the embeddings of the
code tokens it was identified from**. Graph-guided attention allows: code to
code; `<s>`/`</s>` to everything; node to the code tokens it comes from (and
back); node to adjacent nodes.

## Preprocessing

The dataset contains **Python** snippets, so data flow is extracted with the
`tree-sitter-python` grammar via a port of GraphCodeBERT's `DFG_python`
extractor (comment/docstring stripping -> AST -> variable states ->
`comesFrom` / `computedFrom` edges).

| | |
|---|---|
| `code_length` | 512 |
| `data_flow_length` | 128 |
| total sequence length | 640 |
| distinct snippets featurised | 44,950 |
| mean data-flow nodes / snippet | 44.22 |
| snippets with empty data flow | 263 |
| total data-flow edges | 2,481,388 |
| extraction status counts | `{"ok": 44930, "comment_strip_failed": 13, "dfg_failed": 7}` |

No example was dropped: a snippet whose data flow could not be extracted is
kept with an empty graph and counted above.

## Data splits

The repository provides one of 5 predefined folds as `train` + `val`; those groups are disjoint and are kept as-is. `val` is partitioned further into validation/test along problem-group boundaries.

`similar` equals `(code1_group == code2_group)` for every row, so the group
columns are a perfect label proxy and are never used as features.

| split | source | pairs | positives | negatives | groups |
|---|---|---:|---:|---:|---:|
| train | `train` fold | 50,000 | 25,000 | 25,000 | 240 |
| validation | half of `val` by group | 20,000 | 10,000 | 10,000 | 29 |
| test | other half of `val` by group | 20,000 | 10,000 | 10,000 | 30 |

Train/validation/test share **no problem group and no code snippet**; this is
asserted at runtime before training starts. 337,398 pairs of the
held-out fold were dropped because their two snippets fell on opposite sides of
the validation/test group boundary.

Class weighting: Measured majority-class share 0.5000 is within the 0.6 threshold, so weighted cross entropy is NOT used.

## Training

| | |
|---|---|
| optimizer | adamw_torch |
| learning rate | 2e-05 |
| scheduler | linear with 0.1 warmup ratio (938 steps) |
| epochs | 3.0 |
| per-device batch size | 16 |
| gradient accumulation | 1 |
| effective batch size | 16 |
| weight decay | 0.01 |
| gradient clipping | 1.0 |
| mixed precision | fp16 |
| gradient checkpointing | False |
| seed | 42 |
| trainable parameters | 125,236,994 |
| training time | 1.923 h |
| GPU | NVIDIA GeForce RTX 5060 Ti (15.9 GB) |
| torch / transformers | 2.11.0+cu128 / 5.17.0 |

Checkpoint selection: best validation **F1** (`load_best_model_at_end=True`,
`metric_for_best_model="f1"`). Best validation F1 = **0.8672**.
The test split was scored once, after selection.

## Results

| split | accuracy | precision | recall | F1 | TP | TN | FP | FN |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| validation | 0.8557 | 0.8032 | 0.9422 | 0.8672 | 9,422 | 7,692 | 2,308 | 578 |
| test | 0.8747 | 0.8410 | 0.9240 | 0.8805 | 9,240 | 8,253 | 1,747 | 760 |

Test confusion matrix (`[[TN, FP], [FN, TP]]`): `[[8253, 1747], [760, 9240]]`

## Usage

This checkpoint uses a **custom pairwise head and a graph-guided attention
mask**, so `AutoModelForSequenceClassification` will not reproduce these
results. Use the repository's own model class and preprocessing:

```python
import torch
from transformers import AutoTokenizer
from modeling import load_model                      # from this project
from preprocess import build_snippet_features, CloneCollator
from config import Config

cfg = Config()
tokenizer = AutoTokenizer.from_pretrained("thealper2/graphcodebert-code-clone-detection")
model = load_model("thealper2/graphcodebert-code-clone-detection").eval()

features = build_snippet_features(cfg, [code_a, code_b], tokenizer, num_proc=1)
collator = CloneCollator(features)
batch = collator([(0, 1, 0)])                        # (snippet_a, snippet_b, dummy label)
with torch.no_grad():
    logits = model(**{k: v for k, v in batch.items() if k != "labels"}).logits
label = int(logits.argmax(-1))                       # 0 = not clone, 1 = clone
```

## Limitations

- Trained on competitive-programming Python solutions grouped by problem;
  "clone" therefore means *solves the same problem*, which is closer to
  semantic (Type-4) similarity than to syntactic copy-paste detection.
- Data flow is extracted with the Python grammar only. Other languages need the
  matching `tree_sitter_<lang>` grammar and `DFG_<lang>` function.
- Snippets longer than 512 BPE tokens are truncated; 885 of
  44,950 distinct snippets hit that limit.
- Both directions of a pair are not explicitly symmetrised; the head sees
  `concat(<s>_1, <s>_2)` in the given order.