File size: 7,119 Bytes
9636971 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 |
---
language:
- en
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
license:
- mit
datasets:
- NLBSE/nlbse26-code-comment-classification
widget:
- text: dataright np^sin 2 np^pi 224 t | Audio
- text: robust way to ask the database for its current transaction state. | AtomicTests
- text: the string marking the beginning of a print statement. | Environment
- text: handled otherwise by a particular method. | StringMethods
- text: table. | PlotAccessor
metrics:
- accuracy
pipeline_tag: text-classification
library_name: setfit
inference: false
base_model: sentence-transformers/paraphrase-MiniLM-L6-v2
model-index:
- name: SetFit with sentence-transformers/paraphrase-MiniLM-L6-v2
results:
- task:
type: text-classification
name: Text Classification
dataset:
name: NLBSE Code Comment Classification Dataset (Python)
type: NLBSE/nlbse26-code-comment-classification
split: test
metrics:
- type: accuracy
value: 0.4482758620689655
name: Accuracy
---
# SetFit Model for Python Code Comment Classification
## Model Details
- **Model Type:** SetFit (Sentence Transformer Fine-tuning)
- **Base Model:** [sentence-transformers/paraphrase-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/paraphrase-MiniLM-L6-v2)
- **Language:** Python (Comments in English)
- **License:** MIT
- **Developed by:** TheClouds
- **Model Date:** November 17, 2025
- **Model Version:** 1.0
- **Maximum Sequence Length:** 128 tokens
- **Contact:** For questions or comments about this model, please contact us via GitHub or email.
### Description
This model is a SetFit model trained on the **Python** subset of the **NLBSE Code Comment Classification Dataset**. It is designed to classify code comments into categories that describe the semantic purpose of the comment (e.g., Summary, Usage, Parameters).
The model uses a multi-label classification approach, where a single comment can belong to multiple categories.
## Intended Use
This model has been created for the Code Comment Classification task, and trained specifically on code comments extracted from Python projects. As such, it is useful for research and development in code comment classification of projects made in Python, or software documentation analysis tasks.
### Out-of-Scope Use Cases
General text classification outside the domain of software engineering (e.g., social media sentiment analysis) is out of scope.
## Factors
- **Programming Language:** The model is specifically trained on Python code comments (including inline comments `#` and docstrings `"""`).
- **Comment Types:** The model has been evaluated on the following categories specific to software documentation:
1. `Summary`
2. `Usage`
3. `Parameters`
4. `Expand`
5. `DevelopmentNotes`
## Metrics
- **Model Performance Measures:** The primary metrics used for evaluation are **Precision**, **Recall**, **F1-Score**, and **Accuracy**.
- **Decision Thresholds:** A probability threshold of **0.5** was used for classification.
- **Global Performance:** The model achieves an overall Accuracy of **0.4483** on the test set.
## Evaluation Data
- **Dataset:** NLBSE Code Comment Classification Dataset (Python test split).
- **Motivation:** This dataset was chosen because it is the established benchmark for the NLBSE (Natural Language-Based Software Engineering) workshop.
- **Size** 290 rows.
- **Preprocessing:** Comments were extracted from real-world open-source Python projects, split into sentences, and manually classified.
## Training Data
- **Dataset:** NLBSE Code Comment Classification Dataset (Python train split).
- **Dataset Stats:**
| Training set | Min | Median | Max |
|:-------------|:----|:--------|:----|
| Word count | 3 | 15.5217 | 299 |
## Quantitative Analyses
The following table shows the performance breakdown per category on the Python test set:
| Language | Category | Precision | Recall | F1-Score |
|---|---|---|---|---|
| python | **Summary** | 0.6897 | 0.6557 | 0.6723 |
| python | **Usage** | 0.6667 | 0.6813 | 0.6739 |
| python | **Parameters** | 0.6882 | 0.7529 | 0.7191 |
| python | **Expand** | 0.4533 | 0.6667 | 0.5397 |
| python | **DevelopmentNotes** | 0.2192 | 0.5000 | 0.3048 |
## Ethical Considerations
- **Biases:** The dataset is drawn from open-source software projects. The comments reflect the writing styles and norms of the open-source Python community.
- **Content:** Comments are user-generated content and may contain informal language or jargon.
## Caveats and Recommendations
- **Performance Variation:** The model performs well on structural comments like `Parameters` (F1 0.72) but struggles significantly with `DevelopmentNotes` (F1 0.30). Users should exercise caution when relying on the model for identifying development notes or rationale.
- **Context:** The model relies on text-only comment sentences. Surrounding code context is not included.
## How to Use
First install the SetFit library:
```bash
pip install setfit
```
Then you can load this model and run inference:
```python
from setfit import SetFitModel
# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("se4ai2526-uniba/setfit-python")
# Run inference
preds = model(["# yields the next value | generator.py"])
print(preds)
```
## Training Details
### Training Hyperparameters
- batch_size: (32, 32)
- num_epochs: (2, 2)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 5
- body_learning_rate: (2e-05, 1e-05)
- head_learning_rate: 0.01
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- l2_weight: 0.01
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: False
### Training Results
| Metric | Value |
|:-------|:------|
| **Accuracy** | 0.4482758620689655 |
| **Embedding Loss** | 0.177 |
| **Training Loss** | 0.215 |
| **Training Runtime** | 137.40 s |
| **Training Samples/Sec** | 198.189 |
| **Training Steps/Sec** | 6.213 |
### Framework Versions
- Python: 3.11.9
- SetFit: 1.1.2
- Sentence Transformers: 5.1.2
- Transformers: 4.57.1
- PyTorch: 2.7.1
- Datasets: 3.6.0
- Tokenizers: 0.22.1
## Citation
If you use this model in academic work or derived systems, please cite:
> TheClouds Team. "NLBSE'26 Code Comment Classification – Python Model." 2025.
BibTeX:
```bibtex
@misc{theclouds_nlbse26_code_comment_classification_python,
title = {NLBSE'26 Code Comment Classification: Python Model},
author = {TheClouds Team},
year = {2025},
note = {Model available on Hugging Face},
howpublished = {\url{To be published}}
}
```
Contact:
For questions, feedback, or collaboration requests related to this model, please contact:
> Giacomo Signorile: g.signorile14@studenti.uniba.it
> Davide Pio Posa: d.posa3@studenti.uniba.it
> Marco Lillo: m.lillo21@studenti.uniba.it
> Rebecca Margiotta: m.margiotta5@studenti.uniba.it
> Adriano Gentile: a.gentile97@studenti.uniba.com
Issue tracker: https://github.com/se4ai2526-uniba/TheClouds
``` |