Update README.md
Browse files
README.md
CHANGED
|
@@ -30,148 +30,74 @@ tags:
|
|
| 30 |
---
|
| 31 |
|
| 32 |
|
| 33 |
-
# **CodeModernBERT-Owl**
|
| 34 |
|
| 35 |
-
|
| 36 |
|
| 37 |
-
|
| 38 |
-
**CodeModernBERT-Owl** is a **pretrained model** designed from scratch for **code search and code understanding tasks**.
|
| 39 |
|
| 40 |
-
|
| 41 |
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
✅ **Multi-language support**: **Python, PHP, Java, JavaScript, Go, Ruby, and Rust**
|
| 48 |
|
| 49 |
---
|
| 50 |
|
| 51 |
## **📊 モデルパラメータ / Model Parameters**
|
| 52 |
-
| パラメータ / Parameter | 値 / Value |
|
| 53 |
-
|-------------------------|------------|
|
| 54 |
-
| **vocab_size** | 50,004 |
|
| 55 |
-
| **hidden_size** | 768 |
|
| 56 |
-
| **num_hidden_layers** | 12 |
|
| 57 |
-
| **num_attention_heads**| 12 |
|
| 58 |
-
| **intermediate_size** | 3,072 |
|
| 59 |
-
| **max_position_embeddings** | 2,048 |
|
| 60 |
-
| **type_vocab_size** | 2 |
|
| 61 |
-
| **hidden_dropout_prob**| 0.1 |
|
| 62 |
-
| **attention_probs_dropout_prob** | 0.1 |
|
| 63 |
-
| **local_attention_window** | 128 |
|
| 64 |
-
| **rope_theta** | 160,000 |
|
| 65 |
-
| **local_attention_rope_theta** | 10,000 |
|
| 66 |
-
|
| 67 |
-
---
|
| 68 |
-
|
| 69 |
-
## **💻 モデルの使用方法 / How to Use**
|
| 70 |
-
This model can be easily loaded using the **Hugging Face Transformers** library.
|
| 71 |
-
|
| 72 |
-
⚠️ **Requires transformers >= 4.48.0**
|
| 73 |
-
|
| 74 |
-
🔗 **[Colab Demo (Replace with "CodeModernBERT-Owl")](https://github.com/Shun0212/CodeBERTPretrained/blob/main/UseMyCodeMorph_ModernBERT.ipynb)**
|
| 75 |
-
|
| 76 |
-
### **モデルのロード / Load the Model**
|
| 77 |
-
```python
|
| 78 |
-
from transformers import AutoModelForMaskedLM, AutoTokenizer
|
| 79 |
-
|
| 80 |
-
tokenizer = AutoTokenizer.from_pretrained("Shuu12121/CodeModernBERT-Owl")
|
| 81 |
-
model = AutoModelForMaskedLM.from_pretrained("Shuu12121/CodeModernBERT-Owl")
|
| 82 |
-
```
|
| 83 |
-
|
| 84 |
-
### **コード埋め込みの取得 / Get Code Embeddings**
|
| 85 |
-
```python
|
| 86 |
-
import torch
|
| 87 |
-
|
| 88 |
-
def get_embedding(text, model, tokenizer, device="cuda"):
|
| 89 |
-
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
|
| 90 |
-
if "token_type_ids" in inputs:
|
| 91 |
-
inputs.pop("token_type_ids")
|
| 92 |
-
inputs = {k: v.to(device) for k, v in inputs.items()}
|
| 93 |
-
outputs = model.model(**inputs)
|
| 94 |
-
embedding = outputs.last_hidden_state[:, 0, :]
|
| 95 |
-
return embedding
|
| 96 |
-
|
| 97 |
-
embedding = get_embedding("def my_function(): pass", model, tokenizer)
|
| 98 |
-
print(embedding.shape)
|
| 99 |
-
```
|
| 100 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 101 |
|
| 102 |
---
|
| 103 |
|
| 104 |
-
# **
|
| 105 |
-
|
| 106 |
-
### **データセット / Dataset**
|
| 107 |
-
📌 **Tested on code_x_glue_ct_code_to_text with a candidate pool size of 100.**
|
| 108 |
-
📌 **Rust-specific evaluations were conducted using Shuu12121/rust-codesearch-dataset-open.**
|
| 109 |
|
| 110 |
-
-
|
| 111 |
-
|
| 112 |
-
## **📈 主要な評価指標の比較（同一シード値）/ Key Evaluation Metrics (Same Seed)**
|
| 113 |
-
| 言語 / Language | **CodeModernBERT-Owl** | **CodeHawks-ModernBERT** | **Salesforce CodeT5+** | **Microsoft CodeBERT** | **GraphCodeBERT** |
|
| 114 |
-
|-----------|-----------------|----------------------|-----------------|------------------|------------------|
|
| 115 |
-
| **Python** | **0.8793** | 0.8551 | 0.8266 | 0.5243 | 0.5493 |
|
| 116 |
-
| **Java** | **0.8880** | 0.7971 | **0.8867** | 0.3134 | 0.5879 |
|
| 117 |
-
| **JavaScript** | **0.8423** | 0.7634 | 0.7628 | 0.2694 | 0.5051 |
|
| 118 |
-
| **PHP** | **0.9129** | 0.8578 | **0.9027** | 0.2642 | 0.6225 |
|
| 119 |
-
| **Ruby** | **0.8038** | 0.7469 | **0.7568** | 0.3318 | 0.5876 |
|
| 120 |
-
| **Go** | **0.9386** | 0.9043 | 0.8117 | 0.3262 | 0.4243 |
|
| 121 |
|
| 122 |
-
✅ **Achieves the highest accuracy in all target languages.**
|
| 123 |
-
✅ **Significantly improved Java accuracy using additional fine-tuned GitHub data.**
|
| 124 |
-
✅ **Outperforms previous models, especially in PHP and Go.**
|
| 125 |
|
| 126 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 127 |
|
| 128 |
-
|
| 129 |
-
| 指標 / Metric | **CodeModernBERT-Owl** |
|
| 130 |
-
|--------------|----------------|
|
| 131 |
-
| **MRR** | 0.7940 |
|
| 132 |
-
| **MAP** | 0.7940 |
|
| 133 |
-
| **R-Precision** | 0.7173 |
|
| 134 |
-
|
| 135 |
-
### **📌 K別評価指標 / Evaluation Metrics by K**
|
| 136 |
-
| K | **Recall@K** | **Precision@K** | **NDCG@K** | **F1@K** | **Success Rate@K** | **Query Coverage@K** |
|
| 137 |
-
|----|-------------|---------------|------------|--------|-----------------|-----------------|
|
| 138 |
-
| **1** | 0.7173 | 0.7173 | 0.7173 | 0.7173 | 0.7173 | 0.7173 |
|
| 139 |
-
| **5** | 0.8913 | 0.7852 | 0.8118 | 0.8132 | 0.8913 | 0.8913 |
|
| 140 |
-
| **10** | 0.9333 | 0.7908 | 0.8254 | 0.8230 | 0.9333 | 0.9333 |
|
| 141 |
-
| **50** | 0.9887 | 0.7938 | 0.8383 | 0.8288 | 0.9887 | 0.9887 |
|
| 142 |
-
| **100** | 1.0000 | 0.7940 | 0.8401 | 0.8291 | 1.0000 | 1.0000 |
|
| 143 |
|
| 144 |
---
|
| 145 |
|
| 146 |
-
## **🔁 別のおすすめモデル / Recommended Alternative Models**
|
| 147 |
-
|
| 148 |
-
### 1. **CodeSearch-ModernBERT-Owl🦉** (https://huggingface.co/Shuu12121/CodeSearch-ModernBERT-Owl)
|
| 149 |
-
If you need a model that is **more specialized for code search**, this model is highly recommended.
|
| 150 |
-
コードサーチに**特化したモデルが必要な場合**はこちらがおすすめです。
|
| 151 |
-
|
| 152 |
-
### 2. **CodeModernBERT-Snake🐍** (https://huggingface.co/Shuu12121/CodeModernBERT-Snake)
|
| 153 |
-
If you need a pretrained model that supports **longer sequences or a smaller model size**, this model is ideal.
|
| 154 |
-
**シーケンス長が長い**、または**モデルサイズが小さい**事前学習済みモデルが必要な場合はこちらをおすすめします。
|
| 155 |
-
- **Maximum Sequence Length:** 8192 tokens
|
| 156 |
-
- **Smaller Model Size:** ~75M parameters
|
| 157 |
-
|
| 158 |
-
### 3. **CodeSearch-ModernBERT-Snake🐍** (https://huggingface.co/Shuu12121/CodeSearch-ModernBERT-Snake)
|
| 159 |
-
For those looking for a model that combines **long sequence length and code search specialization**, this model is the best choice.
|
| 160 |
-
**コードサーチに特化しつつ長いシーケンスを処理できるモデル**が欲しい場合にはこちらがおすすめです。
|
| 161 |
-
- **Maximum Sequence Length:** 8192 tokens
|
| 162 |
-
- **High Code Search Performance**
|
| 163 |
-
|
| 164 |
-
|
| 165 |
## **📝 結論 / Conclusion**
|
| 166 |
-
|
| 167 |
-
|
| 168 |
-
|
|
|
|
|
|
|
| 169 |
|
| 170 |
---
|
| 171 |
|
| 172 |
## **📜 ライセンス / License**
|
| 173 |
-
|
|
|
|
| 174 |
|
| 175 |
## **📧 連絡先 / Contact**
|
| 176 |
-
|
| 177 |
-
|
|
|
|
|
|
| 30 |
---
|
| 31 |
|
| 32 |
|
| 33 |
+
# **🦉 CodeModernBERT-Owl v1.0: 高精度なコード検索 & コード理解モデル**
|
| 34 |
|
| 35 |
+
**CodeModernBERT-Owl v1.0** is a **pretrained model** designed from scratch for **code search and code understanding tasks**.
|
| 36 |
|
| 37 |
+
This model **now supports Rust** and **improves search accuracy** in Python, PHP, Java, JavaScript, Go, and Ruby.
|
|
|
|
| 38 |
|
| 39 |
+
## **🛠️ 主な特徴 / Key Features**
|
| 40 |
|
| 41 |
+
- **Supports long sequences up to 8192 tokens** (training used up to 2048)
|
| 42 |
+
- **Optimized for code search, code understanding, and code clone detection**
|
| 43 |
+
- **Achieves top-tier performance across multiple languages**
|
| 44 |
+
- **Multi-language support**: Python, PHP, Java, JavaScript, Go, Ruby, and Rust
|
| 45 |
+
- **Mean pooling performs significantly better than CLS token on this model**
|
|
|
|
| 46 |
|
| 47 |
---
|
| 48 |
|
| 49 |
## **📊 モデルパラメータ / Model Parameters**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
|
| 51 |
+
| パラメータ / Parameter | 値 / Value |
|
| 52 |
+
| ------------------------------- | ------------------------- |
|
| 53 |
+
| vocab\_size | 50,004 |
|
| 54 |
+
| hidden\_size | 768 |
|
| 55 |
+
| num\_hidden\_layers | 12 |
|
| 56 |
+
| num\_attention\_heads | 12 |
|
| 57 |
+
| intermediate\_size | 3,072 |
|
| 58 |
+
| max\_position\_embeddings | 8,192 (trained with 2048) |
|
| 59 |
+
| type\_vocab\_size | 2 |
|
| 60 |
+
| hidden\_dropout\_prob | 0.1 |
|
| 61 |
+
| attention\_probs\_dropout\_prob | 0.1 |
|
| 62 |
+
| local\_attention\_window | 128 |
|
| 63 |
+
| rope\_theta | 160,000 |
|
| 64 |
+
| local\_attention\_rope\_theta | 10,000 |
|
| 65 |
|
| 66 |
---
|
| 67 |
|
| 68 |
+
## **📊 言語別 MRR 比較 (Mean Pooling)**
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
|
| 70 |
+
- 実験は CodeSearchNet の test split を使用して実施しました。
|
| 71 |
+
- 候補プールサイズは 100 に固定し、言語ごとの性能を測定しました。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 72 |
|
|
|
|
|
|
|
|
|
|
| 73 |
|
| 74 |
+
| 言語 / Language | **CodeModernBERT-Owl-1.0** | CodeT5+ | GraphCodeBERT | CodeBERTa-small | CodeBERT |
|
| 75 |
+
| ------------- | ---------------------- | ------- | ------------- | --------------- | -------- |
|
| 76 |
+
| Python | **0.8936** | 0.8048 | 0.3496 | 0.6123 | 0.0927 |
|
| 77 |
+
| Java | **0.8479** | 0.7853 | 0.3299 | 0.4738 | 0.0816 |
|
| 78 |
+
| JavaScript | **0.7711** | 0.7111 | 0.2581 | 0.3593 | 0.0692 |
|
| 79 |
+
| PHP | **0.8056** | 0.7893 | 0.2507 | 0.4533 | 0.0623 |
|
| 80 |
+
| Ruby | **0.7993** | 0.7201 | 0.3186 | 0.4418 | 0.0762 |
|
| 81 |
+
| Go | **0.8426** | 0.7577 | 0.4453 | 0.5338 | 0.0856 |
|
| 82 |
|
| 83 |
+
✅ CodeModernBERT-Owl-1.0 (Mean Pooling) achieves the best MRR across all evaluated languages.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 84 |
|
| 85 |
---
|
| 86 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 87 |
## **📝 結論 / Conclusion**
|
| 88 |
+
|
| 89 |
+
- **Top performance in all languages**
|
| 90 |
+
- **Rust support successfully added through dataset augmentation**
|
| 91 |
+
- **Mean pooling is significantly more effective than CLS embedding**
|
| 92 |
+
- **Further performance improvements possible with better datasets**
|
| 93 |
|
| 94 |
---
|
| 95 |
|
| 96 |
## **📜 ライセンス / License**
|
| 97 |
+
|
| 98 |
+
📄 Apache-2.0
|
| 99 |
|
| 100 |
## **📧 連絡先 / Contact**
|
| 101 |
+
|
| 102 |
+
📩 For any questions, please contact:
|
| 103 |
+
📧 [shun0212114@outlook.jp](mailto:shun0212114@outlook.jp)
|