Hugging Face's logo

Spaces:
seai2526-uniba-TheClouds
/
Code-Comment-Classification-Api
Running

App Files Community
Code-Comment-Classification-Api / models /model_cards /java /setfit /README.md
Sky-Blue-da-ba-dee's picture
Sky-Blue-da-ba-dee
fixed a typo in the project name
9636971
preview code
|
raw
history blame contribute delete
7.97 kB
metadata
language:
  - en
tags:
  - text-classification
  - code-comment-classification
  - setfit
  - java
  - software-engineering
  - multi-label
  - sentence-transformers
  - generated_from_setfit_trainer
license: mit
datasets:
  - NLBSE/nlbse26-code-comment-classification
metrics:
  - f1
  - precision
  - recall
pipeline_tag: text-classification
library_name: setfit
inference: false
widget:
  - text: '@link FSNamesystem#readLock() | FSPermissionChecker.java'
  - text: previous^checkpoint li | TestSaveNamespace.java
  - text: // the file doesn't have anything | TaskLog.java
  - text: " @param file the file the include directives point to\n\t * @param depth depth to which includes are followed, should be one of\n\t * {@link #DEPTH_ZERO} or {@link #DEPTH_INFINITE}\n\t * @return an array of include relations\n\t * @throws CoreException | IIndex.java"
  - text: // quotes are removed | ScannerUtility.java
base_model: sentence-transformers/paraphrase-MiniLM-L6-v2
model-index:
  - name: SetFit with sentence-transformers/paraphrase-MiniLM-L6-v2
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: NLBSE Code Comment Classification Dataset (Java)
          type: NLBSE/nlbse26-code-comment-classification
          split: test
        metrics:
          - type: accuracy
            value: 0.7435
            name: Accuracy

SetFit Model for Java Code Comment Classification

Model Details

  • Model Type: SetFit (Sentence Transformer Fine-tuning)
  • Base Model: sentence-transformers/paraphrase-MiniLM-L6-v2
  • Language: Java (Comments in English)
  • License: MIT
  • Developed by: TheClouds
  • Model Date: November 4, 2025
  • Model Version: 1.0
  • Contact: For additional information contact team TheClouds on github

Description

This model is a SetFit model trained on the Java subset of the NLBSE Code Comment Classification Dataset. It is designed to classify code comments into one or more of 7 categories that describe the semantic purpose of the comment.

The model uses a multi-label classification approach, where a single comment can belong to multiple categories.

Intended Use

This model has been created for the Code Comment Classification task, and trained specifically on code comments extracted from Java projects. As such, it is useful for research and development in code comment classification of projects made in Java or other object-oriented languages, or software documentation analysis tasks that perform supervised multi-label classification.

Out-of-Scope Use Cases

General text classification outside the domain of software engineering.

Factors

  • Programming Language: The model is specifically trained on Java code comments.
  • Comment Types: The model recognizes the following 7 categories specific to Java documentation:
    1. summary
    2. Ownership
    3. Expand
    4. usage
    5. Pointer
    6. deprecation
    7. rational

Metrics

  • Model Performance Measures: The primary metrics used for evaluation are Precision, Recall, F1-Score and Accuracy.
  • Decision Threshold: A probability threshold of 0.5 was used for classification.
  • Performance: The model achieves an overall Accuracy of 0.7669 on the test set.

Dataset Summary

The NLBSE Code Comment Classification Dataset is a collection of code comment sentences accompanied by multi-label category annotations.

  • Java Labels (7): summary, Ownership, Expand, usage, Pointer, deprecation, rational.

Each entry corresponds to a comment sentence extracted from real projects.

Motivation

The usage of this specific dataset was a requirement of the NLBSE 2025 code comment classification challenge.

Training Data

  • Dataset: NLBSE Code Comment Classification Dataset (Java train split).
  • Size: 5,390 rows.
  • Label Distribution: The dataset contains 7 categories with varying frequencies. Common categories include "summary" and "usage".

Evaluation Data

  • Dataset: NLBSE Code Comment Classification Dataset (Java test split).
  • Size: 1,200 rows.
  • Preprocessing: Comments were extracted from real-world open-source Java projects, split into sentences, and manually classified.

Quantitative Analyses

lan cat precision recall f1
java summary 0.871224 0.886731 0.878909
java Ownership 1.000000 1.000000 1.000000
java Expand 0.330097 0.430380 0.373626
java usage 0.883803 0.850847 0.867012
java Pointer 0.775641 0.968000 0.861210
java deprecation 0.875000 0.700000 0.777778
java rational 0.311688 0.413793 0.355556

Ethical Considerations

  • Biases: The dataset is drawn from open-source software projects. The comments reflect the writing styles and norms of the open-source community, which may not be representative of all software development environments (e.g., proprietary software).
  • Content: Comments are user-generated content and may contain informal language or jargon specific to the projects they were extracted from.

Caveats and Recommendations

  • Language Specificity: The label set is specific to Java.
  • Context: The model relies on text-only comment sentences. Surrounding code context is not included, which may limit the model's ability to resolve ambiguous comments.
  • Class Imbalance: Some categories (e.g., deprecation, Ownership) may be underrepresented compared to summary or usage.

How to Use

First install the SetFit library:

pip install setfit

Then you can load this model and run inference:

from setfit import SetFitModel

# Download from the 🤗 Huggingface Hub
model = SetFitModel.from_pretrained("se4ai2526-uniba/setfit-java") # Replace with actual model ID if different

# Run inference
preds = model(["// quotes are removed | ScannerUtility.java"])
print(preds)

Training Details

Training Hyperparameters

  • batch_size: (32, 32)
  • num_epochs: (2, 2)
  • max_steps: -1
  • sampling_strategy: oversampling
  • num_iterations: 5
  • body_learning_rate: (2e-05, 1e-05)
  • head_learning_rate: 0.01
  • loss: CosineSimilarityLoss
  • distance_metric: cosine_distance
  • margin: 0.25
  • end_to_end: False
  • use_amp: False
  • warmup_proportion: 0.1
  • l2_weight: 0.01
  • seed: 42
  • eval_max_steps: -1
  • load_best_model_at_end: False
  • probability_threshold: 0.5

Training Results

Metric Value
Accuracy 0.7669
Embedding Loss 0.0239
Training Loss 0.0587
Training Runtime 1515.40 s
Training Samples/Sec 71.189
Training Steps/Sec 2.225

Framework Versions

  • Python: 3.11.9
  • SetFit: 1.1.2
  • Sentence Transformers: 5.1.2
  • Transformers: 4.57.1
  • PyTorch: 2.7.1
  • Datasets: 3.6.0
  • Tokenizers: 0.22.1

Citation

If you use this model in academic work or derived systems, please cite:

TheClouds Team. "NLBSE'26 Code Comment Classification – Java Model." 2025.

BibTeX:

@misc{theclouds_nlbse26_code_comment_classification_java,
  title        = {NLBSE'26 Code Comment Classification: Java Model},
  author       = {TheClouds Team},
  year         = {2025},
  note         = {Model available on Hugging Face},
  howpublished = {\url{To be published}}
}

Contact:

For questions, feedback, or collaboration requests related to this model, please contact:

Giacomo Signorile: g.signorile14@studenti.uniba.it Davide Pio Posa: d.posa3@studenti.uniba.it Marco Lillo: m.lillo21@studenti.uniba.it Rebecca Margiotta: m.margiotta5@studenti.uniba.it Adriano Gentile: a.gentile97@studenti.uniba.com

Issue tracker: https://github.com/se4ai2526-uniba/TheClouds

Acknowledgements

This model was created for research in the context of NLBSE (Natural Language-Based Software Engineering).