benjamintli/modernbert-code-v2

SentenceTransformer based on answerdotai/ModernBERT-base

This is a sentence-transformers model finetuned from answerdotai/ModernBERT-base on the code-retrieval-combined-v2 dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: answerdotai/ModernBERT-base
• Maximum Sequence Length: 1024 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • code-retrieval-combined-v2

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False, 'architecture': 'OptimizedModule'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("modernbert-code-v2")
# Run inference
queries = [
    "If MultiTenantMiddleware is used, filter queryset by request.site_id",
]
documents = [
    "def get_queryset(self):\n        '''\n        If MultiTenantMiddleware is used, filter queryset by request.site_id\n        '''\n        queryset = super(PageList, self).get_queryset()\n        if hasattr(self.request, 'site_id'):\n            queryset = queryset.filter(site_id=self.request.site_id)\n        return queryset",
    'def reduce_ticks(ax, which, maxticks=3):\n    """Given a pyplot axis, resamples its `which`-axis ticks such that are at most\n    `maxticks` left.\n\n    Parameters\n    ----------\n    ax : axis\n        The axis to adjust.\n    which : {\'x\' | \'y\'}\n        Which axis to adjust.\n    maxticks : {3, int}\n        Maximum number of ticks to use.\n\n    Returns\n    -------\n    array\n        An array of the selected ticks.\n    """\n    ticks = getattr(ax, \'get_{}ticks\'.format(which))()\n    if len(ticks) > maxticks:\n        # make sure the left/right value is not at the edge\n        minax, maxax = getattr(ax, \'get_{}lim\'.format(which))()\n        dw = abs(maxax-minax)/10.\n        start_idx, end_idx = 0, len(ticks)\n        if ticks[0] < minax + dw:\n            start_idx += 1\n        if ticks[-1] > maxax - dw:\n            end_idx -= 1\n        # get reduction factor\n        fac = int(len(ticks) / maxticks)\n        ticks = ticks[start_idx:end_idx:fac]\n    return ticks',
    'function (isPublic, name, data, ttl, published_at, coreid) {\n        var rawFn = function (msg) {\n            try {\n                msg.setMaxAge(parseInt((ttl && (ttl >= 0)) ? ttl : 60));\n                if (published_at) {\n                    msg.setTimestamp(moment(published_at).toDate());\n                }\n            }\n            catch (ex) {\n                logger.error("onCoreHeard - " + ex);\n            }\n            return msg;\n        };\n\n        var msgName = (isPublic) ? "PublicEvent" : "PrivateEvent";\n        var userID = (this.userID || "").toLowerCase() + "/";\n        name = (name) ? name.toString() : name;\n        if (name && name.indexOf && (name.indexOf(userID) == 0)) {\n            name = name.substring(userID.length);\n        }\n\n        data = (data) ? data.toString() : data;\n        this.sendNONTypeMessage(msgName, { event_name: name, _raw: rawFn }, data);\n    }',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.9183, -0.0231, -0.0561]])

Evaluation

Metrics

Information Retrieval

• Dataset: eval
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.873
cosine_accuracy@3	0.9367
cosine_accuracy@5	0.9543
cosine_accuracy@10	0.973
cosine_precision@1	0.873
cosine_precision@3	0.3122
cosine_precision@5	0.1909
cosine_precision@10	0.0973
cosine_recall@1	0.873
cosine_recall@3	0.9367
cosine_recall@5	0.9543
cosine_recall@10	0.973
cosine_ndcg@10	0.9241
cosine_mrr@10	0.9083
cosine_map@100	0.9094

Training Details

Training Dataset

code-retrieval-combined-v2

• Dataset: code-retrieval-combined-v2 at 2b971a6
• Size: 283,621 training samples
• Columns: query and positive
• Approximate statistics based on the first 1000 samples:

  	query	positive
  type	string	string
  details	min: 5 tokens mean: 44.94 tokens max: 856 tokens	min: 30 tokens mean: 181.2 tokens max: 1024 tokens

• Samples:

  query	positive
  Start the asyncio event loop and runs the application.	def main(): """Start the asyncio event loop and runs the application.""" # Helper method so that the coroutine exits cleanly if an exception # happens (which would leave resources dangling) async def _run_application(loop): try: return await cli_handler(loop) except KeyboardInterrupt: pass # User pressed Ctrl+C, just ignore it except SystemExit: pass # sys.exit() was used - do nothing except: # pylint: disable=bare-except # noqa import traceback traceback.print_exc(file=sys.stderr) sys.stderr.writelines( '\n>>> An error occurred, full stack trace above\n') return 1 try: loop = asyncio.get_event_loop() return loop.run_until_complete(_run_application(loop)) except KeyboardInterrupt: pass return 1
  Initialize the pool manager with the number of pools, the entry sizes for each pool, and the maximum depth of the free pool. @param bufferEntrySizes the memory sizes of each entry in the pools @param bufferEntryDepths the maximum number of entries in the free pool	public void initialize(int[] bufferEntrySizes, int[] bufferEntryDepths) { if (TraceComponent.isAnyTracingEnabled() && tc.isEntryEnabled()) { Tr.entry(tc, "initialize"); } // order both lists from smallest to largest, based only on Entry Sizes int len = bufferEntrySizes.length; int[] bSizes = new int[len]; int[] bDepths = new int[len]; int sizeCompare; int depth; int sizeSort; int j; for (int i = 0; i < len; i++) { sizeCompare = bufferEntrySizes[i]; depth = bufferEntryDepths[i]; // go backwards, for speed, since first Array List is // probably already ordered small to large for (j = i - 1; j >= 0; j--) { sizeSort = bSizes[j]; if (sizeCompare > sizeSort) { // add the bigger one after the smaller one bSizes[j + 1] = sizeCompare; bDepths[j ...
  // List lists all of the documents in an index. The documents are returned in // increasing ID order.	func (x *Index) List(c context.Context, opts *ListOptions) *Iterator { t := &Iterator{ c: c, index: x, count: -1, listInclusive: true, more: moreList, limit: -1, } if opts != nil { t.listStartID = opts.StartID if opts.Limit > 0 { t.limit = opts.Limit } t.idsOnly = opts.IDsOnly } return t }

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 128,
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

code-retrieval-combined-v2

• Dataset: code-retrieval-combined-v2 at 2b971a6
• Size: 31,516 evaluation samples
• Columns: query and positive
• Approximate statistics based on the first 1000 samples:

  	query	positive
  type	string	string
  details	min: 5 tokens mean: 42.73 tokens max: 834 tokens	min: 30 tokens mean: 180.42 tokens max: 1024 tokens

• Samples:

  query	positive
  This gets the version of OpenALPR :return: Version information	def get_version(self): """ This gets the version of OpenALPR :return: Version information """ ptr = self._get_version_func(self.alpr_pointer) version_number = ctypes.cast(ptr, ctypes.c_char_p).value version_number = _convert_from_charp(version_number) self._free_json_mem_func(ctypes.c_void_p(ptr)) return version_number
  Remove all unnecessary comments from a lexer or parser file	public String stripUnnecessaryComments(String javaContent, AntlrOptions options) { if (!options.isOptimizeCodeQuality()) { return javaContent; } javaContent = stripMachineDependentPaths(javaContent); if (options.isStripAllComments()) { javaContent = stripAllComments(javaContent); } return javaContent; }
  Serialize reply to array or JSON. @param {Object} packet @param {String} packet.method "get", "search", "post", "put", "delete", "sub", "unsub". @param {String} packet.resource @param {String} packet.id @param {*} packet.body @param {Number} [packet.status] @param {Number|String} [packet.date] @param {Object} [packet.headers] @param {Boolean} [json] true to generate JSON instead of array. @returns {Array|String|null}	function reply(packet, json) { return _create(packet, packet.status || 500, (METHODS[packet.method] || '') + packet.resource, json); }

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 128,
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 1024
• per_device_eval_batch_size: 1024
• learning_rate: 8e-05
• num_train_epochs: 1
• warmup_steps: 0.05
• bf16: True
• dataloader_num_workers: 4
• load_best_model_at_end: True
• push_to_hub: True
• hub_model_id: modernbert-code-v2
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 1024
• per_device_eval_batch_size: 1024
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 8e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.05
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 4
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: True
• resume_from_checkpoint: None
• hub_model_id: modernbert-code-v2
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	eval_cosine_ndcg@10
0.0722	20	3.9983	1.3745	0.7545
0.1444	40	1.0297	0.7864	0.8493
0.2166	60	0.6830	0.5917	0.8833
0.2888	80	0.5476	0.5128	0.8973
0.3610	100	0.4891	0.4641	0.9028
0.4332	120	0.4436	0.4370	0.9098
0.5054	140	0.4304	0.4151	0.9154
0.5776	160	0.4101	0.3948	0.9161
0.6498	180	0.3910	0.3829	0.9190
0.7220	200	0.3794	0.3729	0.9188
0.7942	220	0.3668	0.3650	0.9207
0.8664	240	0.3683	0.3573	0.9230
0.9386	260	0.359	0.3534	0.9241

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.3.0
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}