models/model_cards/pharo/setfit/README.md

metadata

language:
  - en
tags:
  - setfit
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
license:
  - mit
datasets:
  - NLBSE/nlbse26-code-comment-classification
metrics:
  - f1
  - precision
  - recall
  - accuracy
pipeline_tag: text-classification
library_name: setfit
inference: false
base_model: sentence-transformers/paraphrase-MiniLM-L6-v2
model-index:
  - name: SetFit with sentence-transformers/paraphrase-MiniLM-L6-v2
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: NLBSE Code Comment Classification Dataset (Pharo)
          type: NLBSE/nlbse26-code-comment-classification
          split: test
        metrics:
          - type: accuracy
            value: 0.5673
            name: Accuracy

SetFit Model for Pharo Code Comment Classification

Model Details

• Model Type: SetFit (Sentence Transformer Fine-tuning)
• Base Model: sentence-transformers/paraphrase-MiniLM-L6-v2
• Language: Pharo (Comments in English)
• License: MIT
• Developed by: TheClouds
• Model Date: November 17, 2025
• Model Version: 1.0
• Maximum Sequence Length: 128 tokens
• Contact: For questions or comments about this model, please contact us via GitHub or email.

Description

This model is a SetFit model trained on the Pharo subset of the NLBSE Code Comment Classification Dataset. It is designed to classify code comments into one or more of 6 categories that describe the semantic purpose of the comment.

The model uses a multi-label classification approach, where a single comment can belong to multiple categories.

Intended Use

This model has been created for the Code Comment Classification task, and trained specifically on code comments extracted from Pharo projects. As such, it is useful for research and development in code comment classification of projects made in Pharo, or software documentation analysis tasks.

Out-of-Scope Use Cases

General text classification outside the domain of software engineering (e.g., social media sentiment analysis) is out of scope.

Factors

• Programming Language: The model is specifically trained on Pharo code comments.
• Comment Types: The model recognizes the following 6 categories specific to Pharo documentation:
  1. Keyimplementationpoints
  2. Example
  3. Responsibilities
  4. Intent
  5. Keymessages
  6. Collaborators

Metrics

• Model Performance Measures: The primary metrics used for evaluation are Precision, Recall, and F1-Score.
• Performance: The model achieves an average F1-Score of 0.4628 on the test set.

Evaluation Data

• Dataset: NLBSE Code Comment Classification Dataset (Pharo test split).
• Size: 208 rows.
• Preprocessing: Comments were extracted from real-world open-source Pharo projects, split into sentences, and manually classified.

Training Data

• Dataset: NLBSE Code Comment Classification Dataset (Pharo train split).
• Size: 900 rows.
• Label Distribution: The dataset contains 6 categories with varying frequencies.

Dataset Summary

The NLBSE Code Comment Classification Dataset is a collection of code comment sentences accompanied by multi-label category annotations.

• Pharo Labels (6): collaborators, example, intent, keyimplementationpoints, keymessages, responsibilities.

Each entry corresponds to a comment sentence extracted from real projects.

Quantitative Analyses

The following table shows the performance breakdown per category on the Pharo test set:

lan	cat	precision	recall	f1
pharo	Keyimplementationpoints	0.562500	0.642857	0.600000
pharo	Example	0.886364	0.876404	0.881356
pharo	Responsibilities	0.632653	0.738095	0.681319
pharo	Intent	0.720000	0.857143	0.782609
pharo	Keymessages	0.478261	0.733333	0.578947
pharo	Collaborators	0.103448	0.428571	0.166667

Ethical Considerations

• Biases: The dataset is drawn from open-source software projects. The comments reflect the writing styles and norms of the open-source community, which may not be representative of all software development environments (e.g., proprietary software).
• Content: Comments are user-generated content and may contain informal language or jargon specific to the projects they were extracted from.

Caveats and Recommendations

• Performance Variation: The model performs well on example comments (F1 0.881) and intent comments (F1 0.782) but struggles significantly with all the other categories. Users should exercise caution when relying on the model for identifying development notes or rationale.
• Context: The model relies on text-only comment sentences. Surrounding code context is not included.

How to Use

First install the SetFit library:

pip install setfit

Then you can load this model and run inference:

from setfit import SetFitModel

# Download from the 🤗 Huggingface Hub
model = SetFitModel.from_pretrained("se4ai2526-uniba/setfit-pharo") # Replace with actual model ID if different

# Run inference
preds = model(["each phase knows about its start time and send a corresponding event once the phase is completed. | BlSpaceFramePhase"])
print(preds)

Training Details

Training Hyperparameters

• batch_size: (32, 32)
• body_learning_rate: (2e-05, 1e-05)
• distance_metric: cosine_distance
• end_to_end: False
• eval_delay: False
• eval_max_steps: -1
• eval_steps: None
• eval_strategy: IntervalStrategy.NO
• evaluation_strategy: None
• greater_is_better: False
• head_learning_rate: 0.01
• l2_weight: 0.01
• load_best_model_at_end: False
• loss: CosineSimilarityLoss
• margin: 0.25
• max_length: None
• max_steps: -1
• metric_for_best_model: embedding_loss
• num_epochs: (2, 2)
• num_iterations: 5
• samples_per_label: 2
• sampling_strategy: oversampling
• save_steps: 500
• save_strategy: steps
• save_total_limit: 1
• seed: 42
• use_amp: False
• warmup_proportion: 0.1

Training Results

Metric	Value
Accuracy	0.5673
Embedding Loss	0.105
Training Loss	0.1566
Training Runtime	161.2121 s
Training Samples/Sec	111.654
Training Steps/Sec	3.498

Framework Versions

• Python: 3.11.9
• SetFit: 1.1.2
• Sentence Transformers: 5.1.2
• Transformers: 4.57.1
• PyTorch: 2.7.1
• Datasets: 3.6.0
• Tokenizers: 0.22.1

Citation

If you use this model in academic work or derived systems, please cite:

TheClouds Team. "NLBSE'26 Code Comment Classification – Pharo Model." 2025.

BibTeX:

@misc{theclouds_nlbse26_code_comment_classification_pharo,
  title        = {NLBSE'26 Code Comment Classification: Pharo Model},
  author       = {TheClouds Team},
  year         = {2025},
  note         = {Model available on Hugging Face},
  howpublished = {\url{To be published}}
}

Contact:

For questions, feedback, or collaboration requests related to this model, please contact:

Giacomo Signorile: g.signorile14@studenti.uniba.it Davide Pio Posa: d.posa3@studenti.uniba.it Marco Lillo: m.lillo21@studenti.uniba.it Rebecca Margiotta: m.margiotta5@studenti.uniba.it Adriano Gentile: a.gentile97@studenti.uniba.com

Issue tracker: https://github.com/se4ai2526-uniba/TheClouds