aaa961/modernbert-embed-base-legal-matryoshka-2

ModernBERT Embed base Legal Matryoshka

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("aaa961/modernbert-embed-base-legal-matryoshka-2")
# Run inference
sentences = [
    'What motion did the court grant?',
    'failure to state claims upon which relief can be granted.  The court \ngranted the motion.  \n \n5 \nII. \nThe Court Did Not Err by Dismissing the Case \n¶ 10 \nAl-Hamim contends that the court erred by granting the \nlandlords’ motion to dismiss.  Specifically, he argues that the court \nerred by determining that the landlords did not breach the warranty',
    'advance the development of artificial intelligence . . . to comprehensively address the national \nsecurity and defense needs of the United States.”  Id. § 1051(b)(1).  The Commission must report \nits findings and recommendations to the President and Congress.  Id. § 1051(c)(1). \nThe Commission was originally set to end this October, but Congress recently extended',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Metric	Value
cosine_accuracy@1	0.5487
cosine_accuracy@3	0.6028
cosine_accuracy@5	0.6878
cosine_accuracy@10	0.7728
cosine_precision@1	0.5487
cosine_precision@3	0.5209
cosine_precision@5	0.3963
cosine_precision@10	0.2326
cosine_recall@1	0.1981
cosine_recall@3	0.5197
cosine_recall@5	0.6441
cosine_recall@10	0.7573
cosine_ndcg@10	0.6574
cosine_mrr@10	0.5991
cosine_map@100	0.6391

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 512
  }
  

Metric	Value
cosine_accuracy@1	0.5518
cosine_accuracy@3	0.592
cosine_accuracy@5	0.6832
cosine_accuracy@10	0.7666
cosine_precision@1	0.5518
cosine_precision@3	0.5188
cosine_precision@5	0.3913
cosine_precision@10	0.2315
cosine_recall@1	0.198
cosine_recall@3	0.5165
cosine_recall@5	0.6382
cosine_recall@10	0.7552
cosine_ndcg@10	0.6553
cosine_mrr@10	0.5981
cosine_map@100	0.6365

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 256
  }
  

Metric	Value
cosine_accuracy@1	0.5085
cosine_accuracy@3	0.558
cosine_accuracy@5	0.6522
cosine_accuracy@10	0.7218
cosine_precision@1	0.5085
cosine_precision@3	0.4838
cosine_precision@5	0.3716
cosine_precision@10	0.2168
cosine_recall@1	0.1826
cosine_recall@3	0.4821
cosine_recall@5	0.6047
cosine_recall@10	0.707
cosine_ndcg@10	0.6125
cosine_mrr@10	0.5575
cosine_map@100	0.6001

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 128
  }
  

Metric	Value
cosine_accuracy@1	0.4451
cosine_accuracy@3	0.4884
cosine_accuracy@5	0.5781
cosine_accuracy@10	0.6538
cosine_precision@1	0.4451
cosine_precision@3	0.4261
cosine_precision@5	0.3317
cosine_precision@10	0.1981
cosine_recall@1	0.1582
cosine_recall@3	0.4205
cosine_recall@5	0.5384
cosine_recall@10	0.644
cosine_ndcg@10	0.5485
cosine_mrr@10	0.4924
cosine_map@100	0.5357

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 64
  }
  

Metric	Value
cosine_accuracy@1	0.3385
cosine_accuracy@3	0.374
cosine_accuracy@5	0.456
cosine_accuracy@10	0.527
cosine_precision@1	0.3385
cosine_precision@3	0.3158
cosine_precision@5	0.2491
cosine_precision@10	0.1584
cosine_recall@1	0.1274
cosine_recall@3	0.3238
cosine_recall@5	0.4125
cosine_recall@10	0.5138
cosine_ndcg@10	0.4303
cosine_mrr@10	0.3789
cosine_map@100	0.4232

Training Details

Training Dataset

json

• Dataset: json
• Size: 5,822 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 8 tokens mean: 16.57 tokens max: 41 tokens	min: 15 tokens mean: 97.04 tokens max: 156 tokens

• Samples:

  anchor	positive
  Under what solicitations do all task orders qualify according to the Defendant?	orders. See SHS MJAR at 36–37; VCH MJAR at 36–37 (same). For the reasons discussed below, this Court concludes that the correct interpretation of “feature” as used in Section 3306(c)(3) lies between Plaintiffs’ and Defendant’s positions. As Defendant argues all task orders contemplated under the Polaris Solicitations qualify as
  What type of project is related to the cost-reimbursement category?	2156–57, 2647–48. Further, offerors can earn additional points for Primary Relevant Experience by submitting (1) projects completed for various government customers; (2) cost-reimbursement 12 projects; (3) task order awards on multiple-award contracts; (4) projects outside the contiguous United States; (5) projects related to cybersecurity experience; and (6) projects demonstrating a
  Who drafted the one-sentence order that lacked stated reasons?	its discretion, a reviewing court looks to the trial court’s “stated justification for refusing to modify” the order. Skolnick, 191 Ill. 2d at 226. ¶ 35 In the case at bar, the one-sentence April 25 order did not provide any reasons at all. The losing party drafted the order without any stated reasons, although a lack of stated reasons may

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• fp16: True
• tf32: False
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
0.8791	10	78.6994	-	-	-	-	-
1.0	12	-	0.5978	0.5973	0.5636	0.4959	0.3714
1.7033	20	35.3464	-	-	-	-	-
2.0	24	-	0.6518	0.6459	0.6049	0.5403	0.4242
2.5275	30	27.0527	-	-	-	-	-
3.0	36	-	0.6577	0.6541	0.6116	0.5467	0.4295
3.3516	40	25.149	-	-	-	-	-
3.7033	44	-	0.6574	0.6553	0.6125	0.5485	0.4303

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.10
• Sentence Transformers: 4.1.0
• Transformers: 4.51.3
• PyTorch: 2.7.0+cu128
• Accelerate: 1.6.0
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}