Create README.md

README.md

@@ -0,0 +1,141 @@
+# Experiment 2 – Latin Square 2: CCT5 & COME on MCMD-NL (Redesigned)
+
+This repository contains the artifacts for **Latin Square 2 of Experiment 2**, which corresponds to the **redesigned and reimplemented experiment** evaluated on the **MCMD-NL dataset** using the DNN-based commit message generation baselines **CCT5** and **COME**. 
+The models have been retrained for each language on the MCMD-NL dataset and then evaluated utilizing the BLEU, METEOR, ROUGE-L, and CIDEr metrics.
+
+***
+
+## Models
+
+### CCT5
+CCT5 is a code-change-oriented pre-trained model built on top of the **T5 architecture**, initialized from **CodeT5** weights and further pre-trained on **CodeChangeNet** (~40GB, 1.5M diff/commit pairs). Released at ESEC/FSE 2023.
+
+- Architecture: Encoder-decoder Transformer (`T5-base` → `CodeT5` → `CCT5`)
+- Pre-training corpus: CodeChangeNet (code diffs paired with commit messages)
+- For MCMD-NL: **new checkpoint trained by fine-tuning the pre-trained CCT5 model on the MCMD-NL training set**, then evaluated on the MCMD-NL test set
+
+### COME
+COME (Commit Message Generation with Modification Embedding) is a hybrid DNN system built on top of CodeT5 with three core components:
+- **Modification embedding**: converts code changes into numerical vectors capturing code evolution
+- **Fine-tuned CodeT5**: generates candidate commit messages from the embedded representation
+- **SVM-based decision algorithm**: selects between the generated and retrieved candidate messages
+
+Released at ISSTA 2023. Does not include additional large-scale pre-training beyond CodeT5.
+
+- For MCMD-NL: **new checkpoint trained by fine-tuning the pre-trained COME model on the MCMD-NL training set**, then evaluated on the MCMD-NL test set
+
+***
+
+## Dataset
+
+**MCMD-NL** – Part of MCMD-New; commits from repositories with programming languages **not present** in the original MCMD dataset.
+
+| Property | Details |
+|----------|---------|
+| Languages | PHP, R, TypeScript, Swift, Objective-C |
+| Repositories | 329 new repositories (not in MCMD) |
+| Total commits | 135,699 |
+| Date range | January 1st, 2022 onwards |
+| Split | 80% train / 10% validation / 10% test |
+| Authors | Wu et al. (2025) |
+
+MCMD-NL was constructed to test model generalization to **entirely new programming languages**, requiring full fine-tuning from the pre-trained model checkpoints rather than reuse of existing MCMD-trained weights.
+
+***
+
+## Repository Structure
+
+Each run folder corresponds to a **programming language** evaluated in this Latin Square. Both CCT5 and COME were fine-tuned on MCMD-NL and evaluated independently for each language.
+
+```
+experiment2_ls2/
+├── run_php/
+│   ├── checkpoint/          # CCT5 and COME checkpoints fine-tuned on MCMD-NL (PHP)
+│   ├── predictions/         # Generated commit messages on MCMD-NL PHP test set
+│   └── metrics/             # BLEU, METEOR, ROUGE-L, CIDEr scores
+├── run_r/
+│   ├── checkpoint/
+│   ├── predictions/
+│   └── metrics/
+├── run_typescript/
+│   ├── checkpoint/
+│   ├── predictions/
+│   └── metrics/
+├── run_swift/
+│   ├── checkpoint/
+│   ├── predictions/
+│   └── metrics/
+└── run_objectivec/
+    ├── checkpoint/
+    ├── predictions/
+    └── metrics/
+```
+
+### `checkpoint/`
+Contains the model checkpoint files produced after fine-tuning CCT5 and COME on the MCMD-NL training set for the corresponding language. These are **newly trained checkpoints**, not reused from prior work. The best checkpoint selected during validation is stored here.
+
+### `predictions/`
+Contains the generated commit messages produced by each model on the MCMD-NL test set for the corresponding language. Files are stored as `.txt` with one prediction per line, aligned to the reference messages in the test set.
+
+### `metrics/`
+Contains the evaluation metric scores computed by comparing the predictions against the MCMD-NL test set reference messages. Each file records BLEU, METEOR, ROUGE-L, and CIDEr scores per model and language under the redesigned evaluation protocol.
+
+***
+
+## Evaluation Metrics
+
+| Metric | Description |
+|--------|-------------|
+| **BLEU** | Bilingual Evaluation Understudy — measures n-gram precision between generated and reference messages |
+| **METEOR** | Metric for Evaluation of Translation with Explicit Ordering — extends BLEU with recall, stemming, and synonym matching via WordNet |
+| **ROUGE-L** | Recall-Oriented Understudy for Gisting Evaluation (LCS variant) — measures longest common subsequence overlap |
+| **CIDEr** | Consensus-based Image Description Evaluation — TF-IDF-weighted n-gram similarity against reference messages |
+
+### Reported Results (Original Paper – Wu et al., 2025)
+
+| Language | Model | BLEU | METEOR | ROUGE-L | CIDEr |
+|----------|-------|------|--------|---------|-------|
+| PHP | CCT5 | 31.96 | 27.31 | 37.99 | 2.26 |
+| PHP | COME | 34.68 | 30.51 | 40.27 | 2.59 |
+| R | CCT5 | 33.02 | 28.92 | 37.17 | 2.19 |
+| R | COME | 35.56 | 31.99 | 38.06 | 2.66 |
+| TypeScript | CCT5 | 32.33 | 27.92 | 43.62 | 2.24 |
+| TypeScript | COME | 35.72 | 30.97 | 47.38 | 2.61 |
+| Swift | CCT5 | 29.29 | 24.58 | 37.09 | 1.98 |
+| Swift | COME | 31.72 | 27.54 | 39.32 | 2.36 |
+| Objective-C | CCT5 | 28.57 | 24.62 | 31.63 | 1.68 |
+| Objective-C | COME | 33.43 | 29.44 | 38.32 | 2.17 |
+| **Average** | **CCT5** | **31.02** | **26.67** | **37.50** | **2.06** |
+| **Average** | **COME** | **34.22** | **30.09** | **40.67** | **2.47** |
+
+These values serve as the reference for comparison with the results produced under the redesigned protocol.
+
+***
+
+## Methodological Differences from Experiment 1
+
+This experiment was redesigned to address the validity and reproducibility concerns identified during the Experiment 1 reproduction phase:
+
+- **Explicit random seed documentation** for all fine-tuning runs
+- **Fully documented fine-tuning procedure**: hyperparameters, batch size, learning rate, number of epochs, and hardware specifications
+- **Best checkpoint selection criteria** explicitly defined using the validation set
+- **Controlled evaluation procedure** with clearly specified evaluation script versions
+- **Full documentation of execution conditions** (hardware, software versions, environment)
+- **Explicit treatment of validity threats** including language-specific variability and training randomness
+
+***
+
+## Important Notes
+
+- The fine-tuning procedure for MCMD-NL is **not a reuse of existing checkpoints** — both models were trained from their pre-trained weights on the MCMD-NL training partition.
+- The original paper does not clarify whether a single multilingual checkpoint or separate per-language checkpoints were trained for MCMD-NL; this ambiguity is addressed and documented in the thesis.
+- MCMD-NL scores are generally **higher than MCMD scores** across all metrics, likely due to the different commit style distributions across the new languages.
+
+***
+
+## References
+
+- Wu et al. (2025). *An Empirical Study on Commit Message Generation with Large Language Models via In-Context Learning.* arXiv:2502.18904.
+- Lin et al. (2023). *CCT5: A Code-Change-Oriented Pre-Trained Model.* ESEC/FSE 2023.
+- He et al. (2023). *COME: Commit Message Generation with Modification Embedding.* ISSTA 2023.
+- Vegas & Elbaum (2023). *Pitfalls in Experiments with DNN4SE.* ESEC/FSE 2023.