| | --- |
| | tags: |
| | - sentence-transformers |
| | - sentence-similarity |
| | - feature-extraction |
| | base_model: Shuu12121/CodeModernBERT-Owl |
| | pipeline_tag: sentence-similarity |
| | library_name: sentence-transformers |
| | metrics: |
| | - code_eval |
| | model-index: |
| | - name: SentenceTransformer based on Shuu12121/CodeModernBERT-Owl |
| | results: |
| | - task: |
| | type: semantic-similarity |
| | name: Semantic Similarity |
| | dataset: |
| | name: code docstring dev |
| | type: code-docstring-dev |
| | metrics: |
| | - type: pearson_cosine |
| | value: null |
| | name: Pearson Cosine |
| | - type: spearman_cosine |
| | value: null |
| | name: Spearman Cosine |
| | license: apache-2.0 |
| | datasets: |
| | - code-search-net/code_search_net |
| | - Shuu12121/java-codesearch-dataset-open |
| | - Shuu12121/rust-codesearch-dataset-open |
| | - google/code_x_glue_ct_code_to_text |
| | language: |
| | - en |
| | --- |
| | |
| |
|
| |
|
| |
|
| |
|
| | # SentenceTransformer based on Shuu12121/CodeModernBERT-Owl🦉 |
| |
|
| |
|
| |
|
| | This model is a **sentence-transformers** model fine-tuned from **[Shuu12121/CodeModernBERT-Owl](https://huggingface.co/Shuu12121/CodeModernBERT-Owl)**, which is a **ModernBERT model specifically designed for code, pre-trained from scratch by me**. |
| | **It is specifically designed for code search and efficiently calculates semantic similarity between code snippets and documentation.** |
| | One of the key features of this model is its **maximum sequence length of 2048 tokens**, which allows it to handle moderately long code snippets and documentation. |
| | Despite being a relatively small model with about **150 million parameters**, it demonstrates remarkable performance in code search tasks. |
| |
|
| | --- |
| |
|
| | このモデルは、**私が一から事前学習を行ったコード特化のModernBERTモデルである [Shuu12121/CodeModernBERT-Owl](https://huggingface.co/Shuu12121/CodeModernBERT-Owl)** をベースにファインチューニングされた **[sentence-transformers](https://www.SBERT.net)** モデルです。 |
| | **特にコードサーチに特化しており、コード片やドキュメントから効果的に意味的類似性を計算できる** ように設計されています。 |
| | 本モデルの特徴として、**最大シーケンス長が2048トークン**に対応しており、**中程度の長さのコード片やドキュメントにも対応可能**です。 |
| | **150M程度と比較的小さいモデル**ながらも、コード検索タスクにおいて高い性能を発揮します。 |
| |
|
| |
|
| | --- |
| |
|
| | ### Model Evaluation / モデル評価 |
| |
|
| | #### CoIR Evaluation Results / CoIRにおける評価結果 |
| |
|
| | Despite being a relatively small model with around **150M parameters**, this model achieved an impressive **76.89** on the **CodeSearchNet** benchmark, demonstrating its high performance in code search tasks. |
| | Since this model is specialized for code search, it does not support other tasks, and thus evaluation scores for other tasks are not provided. |
| | In the CodeSearchNet task, this model outperforms many well-known models, as shown in the comparison table below. |
| |
|
| | このモデルは、**150M程度と比較的小さいモデル**ながら、**コードサーチタスクにおける評価指標である CodeSearchNet で 76.89** を達成しました。 |
| | 他のタスクには対応していないため、評価値は提供されていません。 |
| | CodeSearchNetタスクにおける評価値としては、他の有名なモデルと比較しても高いパフォーマンスを示しています。 |
| |
|
| | | Model Name | CodeSearchNet Score | |
| | |-----------------------------------------------|----------------------| |
| | | **Shuu12121/CodeModernBERT-Owl** | **76.89** | |
| | | Salesforce/SFR-Embedding-Code-2B_R | 73.5 | |
| | | CodeSage-large-v2 | 94.26 | |
| | | Salesforce/SFR-Embedding-Code-400M_R | 72.53 | |
| | | CodeSage-large | 90.58 | |
| | | Voyage-Code-002 | 81.79 | |
| | | E5-Mistral | 54.25 | |
| | | E5-Base-v2 | 67.99 | |
| | | OpenAI-Ada-002 | 74.21 | |
| | | BGE-Base-en-v1.5 | 69.6 | |
| | | BGE-M3 | 43.23 | |
| | | UniXcoder | 60.2 | |
| | | GTE-Base-en-v1.5 | 43.35 | |
| | | Contriever | 34.72 | |
| |
|
| | --- |
| |
|
| | ### Model Details / モデル詳細 |
| |
|
| | - **Model Type / モデルタイプ:** Sentence Transformer |
| | - **Base Model / ベースモデル:** [Shuu12121/CodeModernBERT-Owl](https://huggingface.co/Shuu12121/CodeModernBERT-Owl) |
| | - **Maximum Sequence Length / 最大シーケンス長:** 2048 tokens |
| | - **Output Dimensions / 出力次元:** 768 dimensions |
| | - **Similarity Function / 類似度関数:** Cosine Similarity |
| | - **License / ライセンス:** Apache-2.0 |
| |
|
| | --- |
| |
|
| | ### Usage / 使用方法 |
| |
|
| | #### Installation / インストール |
| |
|
| | To install Sentence Transformers, run the following command: |
| | Sentence Transformers をインストールするには、以下のコマンドを実行します。 |
| |
|
| | ```bash |
| | pip install -U sentence-transformers |
| | ``` |
| |
|
| | #### Model Loading and Inference / モデルのロードと推論 |
| |
|
| | ```python |
| | from sentence_transformers import SentenceTransformer |
| | |
| | # Load the model / モデルをダウンロードしてロード |
| | model = SentenceTransformer("Shuu12121/CodeSearch-ModernBERT-Owl") |
| | |
| | # Example sentences for inference / 推論用の文リスト |
| | sentences = [ |
| | 'Encrypts the zip file', |
| | 'def freeze_encrypt(dest_dir, zip_filename, config, opt):\n \n pgp_keys = grok_keys(config)\n icefile_prefix = "aomi-%s" % \\\n os.path.basename(os.path.dirname(opt.secretfile))\n if opt.icefile_prefix:\n icefile_prefix = opt.icefile_prefix\n\n timestamp = time.strftime("%H%M%S-%m-%d-%Y",\n datetime.datetime.now().timetuple())\n ice_file = "%s/%s-%s.ice" % (dest_dir, icefile_prefix, timestamp)\n if not encrypt(zip_filename, ice_file, pgp_keys):\n raise aomi.exceptions.GPG("Unable to encrypt zipfile")\n\n return ice_file', |
| | 'def transform(self, sents):\n \n\n def convert(tokens):\n return torch.tensor([self.vocab.stoi[t] for t in tokens], dtype=torch.long)\n\n if self.vocab is None:\n raise Exception(\n "Must run .fit() for .fit_transform() before " "calling .transform()."\n )\n\n seqs = sorted([convert(s) for s in sents], key=lambda x: -len(x))\n X = torch.LongTensor(pad_sequence(seqs, batch_first=True))\n return X', |
| | ] |
| | |
| | # Generate embeddings / 埋め込みベクトルの生成 |
| | embeddings = model.encode(sentences) |
| | print(embeddings.shape) # Output: [3, 768] |
| | |
| | # Calculate similarity scores / 類似度スコアの計算 |
| | similarities = model.similarity(embeddings, embeddings) |
| | print(similarities.shape) # Output: [3, 3] |
| | ``` |
| |
|
| | --- |
| |
|
| | ### Library Versions / ライブラリバージョン |
| |
|
| | - Python: 3.11.11 |
| | - Sentence Transformers: 3.4.1 |
| | - Transformers: 4.50.0 |
| | - PyTorch: 2.6.0+cu124 |
| | - Accelerate: 1.5.2 |
| | - Datasets: 3.4.1 |
| | - Tokenizers: 0.21.1 |
| |
|
| | --- |
| |
|
| | ### Citation / 引用情報 |
| |
|
| | #### Sentence Transformers |
| | ```bibtex |
| | @inproceedings{reimers-2019-sentence-bert, |
| | title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", |
| | author = "Reimers, Nils and Gurevych, Iryna", |
| | booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", |
| | month = "11", |
| | year = "2019", |
| | publisher = "Association for Computational Linguistics", |
| | url = "https://arxiv.org/abs/1908.10084", |
| | } |
| | ``` |
| |
|
| | #### MultipleNegativesRankingLoss |
| | ```bibtex |
| | @misc{henderson2017efficient, |
| | title={Efficient Natural Language Response Suggestion for Smart Reply}, |
| | author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, |
| | year={2017}, |
| | eprint={1705.00652}, |
| | archivePrefix={arXiv}, |
| | primaryClass={cs.CL} |
| | } |
| | ``` |
