SentenceTransformer based on answerdotai/ModernBERT-base

This is a sentence-transformers model finetuned from answerdotai/ModernBERT-base on the code-retrieval-combined-v2 dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: answerdotai/ModernBERT-base
Maximum Sequence Length: 1024 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Training Dataset:
- code-retrieval-combined-v2

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False, 'architecture': 'OptimizedModule'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("modernbert-code-v2")
# Run inference
queries = [
    "If MultiTenantMiddleware is used, filter queryset by request.site_id",
]
documents = [
    "def get_queryset(self):\n        '''\n        If MultiTenantMiddleware is used, filter queryset by request.site_id\n        '''\n        queryset = super(PageList, self).get_queryset()\n        if hasattr(self.request, 'site_id'):\n            queryset = queryset.filter(site_id=self.request.site_id)\n        return queryset",
    'def reduce_ticks(ax, which, maxticks=3):\n    """Given a pyplot axis, resamples its `which`-axis ticks such that are at most\n    `maxticks` left.\n\n    Parameters\n    ----------\n    ax : axis\n        The axis to adjust.\n    which : {\'x\' | \'y\'}\n        Which axis to adjust.\n    maxticks : {3, int}\n        Maximum number of ticks to use.\n\n    Returns\n    -------\n    array\n        An array of the selected ticks.\n    """\n    ticks = getattr(ax, \'get_{}ticks\'.format(which))()\n    if len(ticks) > maxticks:\n        # make sure the left/right value is not at the edge\n        minax, maxax = getattr(ax, \'get_{}lim\'.format(which))()\n        dw = abs(maxax-minax)/10.\n        start_idx, end_idx = 0, len(ticks)\n        if ticks[0] < minax + dw:\n            start_idx += 1\n        if ticks[-1] > maxax - dw:\n            end_idx -= 1\n        # get reduction factor\n        fac = int(len(ticks) / maxticks)\n        ticks = ticks[start_idx:end_idx:fac]\n    return ticks',
    'function (isPublic, name, data, ttl, published_at, coreid) {\n        var rawFn = function (msg) {\n            try {\n                msg.setMaxAge(parseInt((ttl && (ttl >= 0)) ? ttl : 60));\n                if (published_at) {\n                    msg.setTimestamp(moment(published_at).toDate());\n                }\n            }\n            catch (ex) {\n                logger.error("onCoreHeard - " + ex);\n            }\n            return msg;\n        };\n\n        var msgName = (isPublic) ? "PublicEvent" : "PrivateEvent";\n        var userID = (this.userID || "").toLowerCase() + "/";\n        name = (name) ? name.toString() : name;\n        if (name && name.indexOf && (name.indexOf(userID) == 0)) {\n            name = name.substring(userID.length);\n        }\n\n        data = (data) ? data.toString() : data;\n        this.sendNONTypeMessage(msgName, { event_name: name, _raw: rawFn }, data);\n    }',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.9183, -0.0231, -0.0561]])

Evaluation

Metrics

Information Retrieval

Dataset: eval
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.873
cosine_accuracy@3	0.9367
cosine_accuracy@5	0.9543
cosine_accuracy@10	0.973
cosine_precision@1	0.873
cosine_precision@3	0.3122
cosine_precision@5	0.1909
cosine_precision@10	0.0973
cosine_recall@1	0.873
cosine_recall@3	0.9367
cosine_recall@5	0.9543
cosine_recall@10	0.973
cosine_ndcg@10	0.9241
cosine_mrr@10	0.9083
cosine_map@100	0.9094

Training Details

Training Dataset

code-retrieval-combined-v2

Dataset: code-retrieval-combined-v2 at 2b971a6
Size: 283,621 training samples
Columns: query and positive
Approximate statistics based on the first 1000 samples:
query positive
type string string
details
min: 5 tokens
mean: 44.94 tokens
max: 856 tokens

min: 30 tokens
mean: 181.2 tokens
max: 1024 tokens

	query	positive
type	string	string
details	min: 5 tokens mean: 44.94 tokens max: 856 tokens	min: 30 tokens mean: 181.2 tokens max: 1024 tokens

Samples:

query	positive
`Start the asyncio event loop and runs the application.`	def main(): """Start the asyncio event loop and runs the application.""" # Helper method so that the coroutine exits cleanly if an exception # happens (which would leave resources dangling) async def _run_application(loop): try: return await cli_handler(loop) except KeyboardInterrupt: pass # User pressed Ctrl+C, just ignore it except SystemExit: pass # sys.exit() was used - do nothing except: # pylint: disable=bare-except # noqa import traceback traceback.print_exc(file=sys.stderr) sys.stderr.writelines( '\n>>> An error occurred, full stack trace above\n') return 1 try: loop = asyncio.get_event_loop() return loop.run_until_complete(_run_application(loop)) except KeyboardInterrupt: pass return 1
`Initialize the pool manager with the number of pools, the entry sizes for each pool, and the maximum depth of the free pool. @param bufferEntrySizes the memory sizes of each entry in the pools @param bufferEntryDepths the maximum number of entries in the free pool`	public void initialize(int[] bufferEntrySizes, int[] bufferEntryDepths) { if (TraceComponent.isAnyTracingEnabled() && tc.isEntryEnabled()) { Tr.entry(tc, "initialize"); } // order both lists from smallest to largest, based only on Entry Sizes int len = bufferEntrySizes.length; int[] bSizes = new int[len]; int[] bDepths = new int[len]; int sizeCompare; int depth; int sizeSort; int j; for (int i = 0; i < len; i++) { sizeCompare = bufferEntrySizes[i]; depth = bufferEntryDepths[i]; // go backwards, for speed, since first Array List is // probably already ordered small to large for (j = i - 1; j >= 0; j--) { sizeSort = bSizes[j]; if (sizeCompare > sizeSort) { // add the bigger one after the smaller one bSizes[j + 1] = sizeCompare; bDepths[j ...
`// List lists all of the documents in an index. The documents are returned in // increasing ID order.`	`func (x Index) List(c context.Context, opts ListOptions) *Iterator { t := &Iterator{ c: c, index: x, count: -1, listInclusive: true, more: moreList, limit: -1, } if opts != nil { t.listStartID = opts.StartID if opts.Limit > 0 { t.limit = opts.Limit } t.idsOnly = opts.IDsOnly } return t }`

Loss: CachedMultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "mini_batch_size": 128,
    "gather_across_devices": false,
    "directions": [
        "query_to_doc"
    ],
    "partition_mode": "joint",
    "hardness_mode": null,
    "hardness_strength": 0.0
}

Evaluation Dataset

code-retrieval-combined-v2

Dataset: code-retrieval-combined-v2 at 2b971a6
Size: 31,516 evaluation samples
Columns: query and positive
Approximate statistics based on the first 1000 samples:
query positive
type string string
details
min: 5 tokens
mean: 42.73 tokens
max: 834 tokens

min: 30 tokens
mean: 180.42 tokens
max: 1024 tokens

	query	positive
type	string	string
details	min: 5 tokens mean: 42.73 tokens max: 834 tokens	min: 30 tokens mean: 180.42 tokens max: 1024 tokens

Samples:

query	positive
`This gets the version of OpenALPR :return: Version information`	`def get_version(self): """ This gets the version of OpenALPR :return: Version information """ ptr = self._get_version_func(self.alpr_pointer) version_number = ctypes.cast(ptr, ctypes.c_char_p).value version_number = _convert_from_charp(version_number) self._free_json_mem_func(ctypes.c_void_p(ptr)) return version_number`
`Remove all unnecessary comments from a lexer or parser file`	`public String stripUnnecessaryComments(String javaContent, AntlrOptions options) { if (!options.isOptimizeCodeQuality()) { return javaContent; } javaContent = stripMachineDependentPaths(javaContent); if (options.isStripAllComments()) { javaContent = stripAllComments(javaContent); } return javaContent; }`
`Serialize reply to array or JSON. @param {Object} packet @param {String} packet.method "get", "search", "post", "put", "delete", "sub", "unsub". @param {String} packet.resource @param {String} packet.id @param {*} packet.body @param {Number} [packet.status] @param {Number\|String} [packet.date] @param {Object} [packet.headers] @param {Boolean} [json] true to generate JSON instead of array. @returns {Array\|String\|null}`	`function reply(packet, json) { return _create(packet, packet.status \|\| 500, (METHODS[packet.method] \|\| '') + packet.resource, json); }`

Loss: CachedMultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "mini_batch_size": 128,
    "gather_across_devices": false,
    "directions": [
        "query_to_doc"
    ],
    "partition_mode": "joint",
    "hardness_mode": null,
    "hardness_strength": 0.0
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 1024
per_device_eval_batch_size: 1024
learning_rate: 8e-05
num_train_epochs: 1
warmup_steps: 0.05
bf16: True
dataloader_num_workers: 4
load_best_model_at_end: True
push_to_hub: True
hub_model_id: modernbert-code-v2
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 1024
per_device_eval_batch_size: 1024
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 8e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 1
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_ratio: None
warmup_steps: 0.05
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
enable_jit_checkpoint: False
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
use_cpu: False
seed: 42
data_seed: None
bf16: True
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: -1
ddp_backend: None
debug: []
dataloader_drop_last: False
dataloader_num_workers: 4
dataloader_prefetch_factor: None
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: True
ignore_data_skip: False
fsdp: []
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
group_by_length: False
length_column_name: length
project: huggingface
trackio_space_id: trackio
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
push_to_hub: True
resume_from_checkpoint: None
hub_model_id: modernbert-code-v2
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_for_metrics: []
eval_do_concat_batches: True
auto_find_batch_size: False
full_determinism: False
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_num_input_tokens_seen: no
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: True
use_cache: False
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	eval_cosine_ndcg@10
0.0722	20	3.9983	1.3745	0.7545
0.1444	40	1.0297	0.7864	0.8493
0.2166	60	0.6830	0.5917	0.8833
0.2888	80	0.5476	0.5128	0.8973
0.3610	100	0.4891	0.4641	0.9028
0.4332	120	0.4436	0.4370	0.9098
0.5054	140	0.4304	0.4151	0.9154
0.5776	160	0.4101	0.3948	0.9161
0.6498	180	0.3910	0.3829	0.9190
0.7220	200	0.3794	0.3729	0.9188
0.7942	220	0.3668	0.3650	0.9207
0.8664	240	0.3683	0.3573	0.9230
0.9386	260	0.359	0.3534	0.9241

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.12.12
Sentence Transformers: 5.3.0
Transformers: 5.0.0
PyTorch: 2.10.0+cu128
Accelerate: 1.13.0
Datasets: 4.0.0
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}