test2 / README.md

Add new SentenceTransformer model

6e65de2 verified over 1 year ago

35.7 kB

language:
  - en
license: apache-2.0
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:3692
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
base_model: nomic-ai/modernbert-embed-base
widget:
  - source_sentence: >-
      | :-- | :--: | :--: || After the coming into force date | 1.00 | 1.00 || 6
      April 2014 - coming into force date | 1.30 | 1.30 || 1 Oct 2010 - 5 April
      2014 | 1.40 | 1.40 || 6 April 2006 - 30 Sept 2010 | 1.67 | 1.67 || Pre 6
      April 2006 | 1.67 | 1.67 |
    sentences:
      - >-
        What are the applicable building regulation change factors for different
        time periods starting from pre-6 April 2006 to the current date?
      - >-
        When are protected lobbies or corridors required for staircases in
        buildings with multiple storeys above ground level?
      - >-
        What are the standards for repairing, reconstructing, and altering
        existing drains and sewers?
  - source_sentence: >-
      The door has a minimum clear opening width of 850 mm , when measured in
      accordance with Diagram 3.2.g.Where there are double doors, the main (or
      leading) leaf provides the required minimum clear opening width.A minimum
      200 mm nib is provided to the following edge of the door and the extra
      width created by the nib is maintained for a minimum of 1500 mm beyond
      it.[IMAGE]h.The door is located reasonably centrally within the thickness
      of the wall while ensuring that the depth of the reveal on the leading
      face of the door (usually the inside) is a maximum of 200 mm .i.The
      threshold is an accessible threshold.j.Where there is a lobby or porch,
      the doors are a minimum of 1500 mm apart and there is a minimum of 1500 mm
      between door swings.k.Door entry controls, where provided, are mounted
      900-1000mm above finished ground level a minimum of 300 mm away from any
      external return corner.l.A fused spur, suitable for the fitting of a
      powered door opener, is provided on the hinge side of the door.# Other
      external doors3.23 All other external doors - including doors to and from
      a private garden, balcony, terrace, garage, carport, conservatory or
      storage area that is integral with, or connected, the dwelling comply with
      provisions $f$. to $k$. of paragraph 3.22 and should have a minimum 300 m
      nib to the leading edge of the door with the extra width created by this
      nib extending for a minimum 1800 mm beyond it.## Circulation areas,
      internal doorways and storage## Hall and door widths3.24 To facilitate
      wheelchair movement into and between rooms, internal halls and doors
      should comply with all of the following (see Diagram 3.4).
    sentences:
      - >-
        What are the minimum clear opening width and placement requirements for
        doors according to the UK building regulations?
      - >-
        What are the material class and limits for rooms and circulation spaces
        in buildings, excluding protected stairways?
      - >-
        What is the purpose of the Building and Engineering Services
        Association's DW/144 Specification for Sheet Metal Ductwork as
        referenced in the UK building regulations?
  - source_sentence: >-
      a.Every individual dwelling complies with all of the following
      conditions.i.The dwelling primary energy rate must not exceed the
      individual dwelling's target primary energy rate.ii.The dwelling emission
      rate must not exceed the individual dwelling's target emission
      rate.iii.The dwelling fabric energy efficiency rate must not exceed the
      individual dwelling's target fabric energy efficiency rate.ORb.All of the
      following are met.i.
    sentences:
      - >-
        What are the requirements for a parking space in a dwelling to
        accommodate a wheelchair user?
      - >-
        What are the fire separation requirements for a garage attached to a
        dwellinghouse?
      - >-
        What criteria must a dwelling meet to comply with target energy and
        emission rates in UK building regulations?
  - source_sentence: >-
      ISBN 071760 4136.The Workplace (Health, Safety and Welfare) Regulations
      1992 apply to the common parts of flats and similar buildings if people
      such as cleaners and caretakers are employed to work in these common
      parts.Where the requirements of the Building Regulations that are covered
      by this Part do not apply to dwellings, the provisions may still be
      required in the situations described above in order to satisfy the
      Workplace Regulations.# USE OF GUIDANCE# The RequirementsThis Approved
      Document, which took effect on 1 July 2003, deals with the Requirements of
      Part E of Schedule 1 to the Building Regulations 2010.## Requirement##
      Protection against sound from other parts of the building and adjoining
      buildings
    sentences:
      - >-
        What are the fire resistance requirements for constructions separating
        firefighting shafts from the rest of the building or from their own
        components within the shaft?
      - >-
        What is required to provide sprinkler protection to open-plan areas in a
        building?
      - >-
        What are the sound protection requirements for the common parts of flats
        and similar buildings employed with workers such as cleaners and
        caretakers?
  - source_sentence: >-
      Appendix A:Key terms ..... 11Appendix B:Standards referred to ..... 12#
      Approved Document P:Electrical safety Dwellings## Summary0.1 This approved
      document gives guidance on how to comply with Part P of the Building
      Regulations.It contains the following sections:Section 1:Technical
      requirements for electrical work in dwellingsSection 2:The types of
      building and electrical installation within the scope of Part P, and the
      types of electrical work that are notifiableSection 3:The different
      procedures that may be followed to show that electrical work complies with
      Part PAppendix A:Key terms
    sentences:
      - >-
        What conditions classify a district heat network as 'under construction'
        based on the building regulations defined on 15 June 2022?
      - >-
        What are the requirements for providing reasonable access for
        maintaining ventilation systems in buildings?
      - >-
        What guidance does Approved Document P provide for complying with Part P
        of the Building Regulations regarding electrical work in dwellings?
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
model-index:
  - name: ModernBERT Embed base Legal Matryoshka
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 768
          type: dim_768
        metrics:
          - type: cosine_accuracy@1
            value: 0.656934306569343
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8442822384428224
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8905109489051095
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9367396593673966
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.656934306569343
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.28142741281427414
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.1781021897810219
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09367396593673966
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.656934306569343
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8442822384428224
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8905109489051095
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9367396593673966
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8031293036354475
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7597178774186073
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7623797955453413
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 512
          type: dim_512
        metrics:
          - type: cosine_accuracy@1
            value: 0.656934306569343
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8442822384428224
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8880778588807786
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9440389294403893
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.656934306569343
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.28142741281427414
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.17761557177615572
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09440389294403892
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.656934306569343
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8442822384428224
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8880778588807786
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9440389294403893
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8037305107001143
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7585679527285365
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7605016315461136
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 256
          type: dim_256
        metrics:
          - type: cosine_accuracy@1
            value: 0.6472019464720195
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8369829683698297
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8880778588807786
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9440389294403893
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.6472019464720195
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.27899432278994324
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.1776155717761557
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09440389294403892
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.6472019464720195
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8369829683698297
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8880778588807786
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9440389294403893
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.7994650257091209
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7527816398254354
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7549760733760151
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 128
          type: dim_128
        metrics:
          - type: cosine_accuracy@1
            value: 0.610705596107056
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8126520681265207
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.851581508515815
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9294403892944039
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.610705596107056
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.27088402270884027
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.170316301703163
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09294403892944038
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.610705596107056
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8126520681265207
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.851581508515815
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9294403892944039
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.7725417704200617
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7222550496273127
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7253503796965786
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 64
          type: dim_64
        metrics:
          - type: cosine_accuracy@1
            value: 0.5450121654501217
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.7493917274939172
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.7931873479318735
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.8637469586374696
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.5450121654501217
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.2497972424979724
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.15863746958637467
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.08637469586374696
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.5450121654501217
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.7493917274939172
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.7931873479318735
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.8637469586374696
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.7081090686112841
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.657939404472251
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.663604050590536
            name: Cosine Map@100

ModernBERT Embed base Legal Matryoshka

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: nomic-ai/modernbert-embed-base
Maximum Sequence Length: 8192 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("calvin2258000/test2")
# Run inference
sentences = [
    'Appendix A:Key terms ..... 11Appendix B:Standards referred to ..... 12# Approved Document P:Electrical safety Dwellings## Summary0.1 This approved document gives guidance on how to comply with Part P of the Building Regulations.It contains the following sections:Section 1:Technical requirements for electrical work in dwellingsSection 2:The types of building and electrical installation within the scope of Part P, and the types of electrical work that are notifiableSection 3:The different procedures that may be followed to show that electrical work complies with Part PAppendix A:Key terms',
    'What guidance does Approved Document P provide for complying with Part P of the Building Regulations regarding electrical work in dwellings?',
    "What conditions classify a district heat network as 'under construction' based on the building regulations defined on 15 June 2022?",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Datasets: dim_768, dim_512, dim_256, dim_128 and dim_64
Evaluated with InformationRetrievalEvaluator

Metric	dim_768	dim_512	dim_256	dim_128	dim_64
cosine_accuracy@1	0.6569	0.6569	0.6472	0.6107	0.545
cosine_accuracy@3	0.8443	0.8443	0.837	0.8127	0.7494
cosine_accuracy@5	0.8905	0.8881	0.8881	0.8516	0.7932
cosine_accuracy@10	0.9367	0.944	0.944	0.9294	0.8637
cosine_precision@1	0.6569	0.6569	0.6472	0.6107	0.545
cosine_precision@3	0.2814	0.2814	0.279	0.2709	0.2498
cosine_precision@5	0.1781	0.1776	0.1776	0.1703	0.1586
cosine_precision@10	0.0937	0.0944	0.0944	0.0929	0.0864
cosine_recall@1	0.6569	0.6569	0.6472	0.6107	0.545
cosine_recall@3	0.8443	0.8443	0.837	0.8127	0.7494
cosine_recall@5	0.8905	0.8881	0.8881	0.8516	0.7932
cosine_recall@10	0.9367	0.944	0.944	0.9294	0.8637
cosine_ndcg@10	0.8031	0.8037	0.7995	0.7725	0.7081
cosine_mrr@10	0.7597	0.7586	0.7528	0.7223	0.6579
cosine_map@100	0.7624	0.7605	0.755	0.7254	0.6636

Training Details

Training Dataset

Unnamed Dataset

Size: 3,692 training samples
Columns: positive and anchor
Approximate statistics based on the first 1000 samples:
positive anchor
type string string
details
min: 3 tokens
mean: 176.61 tokens
max: 742 tokens

min: 12 tokens
mean: 23.56 tokens
max: 51 tokens

	positive	anchor
type	string	string
details	min: 3 tokens mean: 176.61 tokens max: 742 tokens	min: 12 tokens mean: 23.56 tokens max: 51 tokens

Samples:

positive	anchor
# Section 3:Subsoil drainage3.1 The provisions which follow assume that the site of the building is not subject to general flooding (see paragraph 0.8 ) or, if it is, that appropriate steps are being taken.3.2 Where the water table can rise to within 0.25 m of the lowest floor of the building, or where surface water could enter or adversely affect the building, either the ground to be covered by the building should be drained by gravity, or other effective means of safeguarding the building should be taken.3.3 If an active subsoil drain is cut during excavation and if it passes under the building it should be:a. re-laid in pipes with sealed joints and have access points outside the building; orb. re-routed around the building; orc. re-run to another outfall (see Diagram 3).3.4 Where there is a risk that groundwater beneath or around the building could adversely affect the stability and properties of the ground, consideration should be given to site drainage or other protection (see Sec...	`What measures should be taken if a building site has a water table that can rise to within 0.25 meters of the lowest floor?`
`Easily accessibleEither:- a window or doorway, any part of which is within 2 m vertically of an accessible level surface such as the ground or basement level, or an access balcony, or- a window within 2 m vertically of a flat or sloping roof (with a pitch of less than $30^{\circ}$ ) that is within 3.5 m of ground level.Coupled assemblyA doorset and window that are supplied as separate self-contained frames and fixed together on site.`	`What criteria determine if a window or doorway is considered easily accessible in a building?`
Fuels such as bituminous coal, untreated wood or compressed paper are not smokeless or low-volatiles fuels.3.These appliances are known as 'exempted fireplaces'.2.7 For fireplaces with openings larger than $500 \mathrm{mm} \times 550 \mathrm{mm}$ or fireplaces exposed on two or more sides (such as a fireplace under a canopy or open on both sides of a central chimney breast) a way of showing compliance would be to provide a flue with a cross-sectional area equal to 15 per cent of the total face area of the fireplace opening(s) (see Appendix B).However, specialist advice should be sought when proposing to construct flues having an area of:a. more than 15 per cent of the total face area of the fireplace openings; or	`What is the required flue cross-sectional area for fireplaces with openings larger than 500 mm x 550 mm or exposed on multiple sides?`

Loss: MatryoshkaLoss with these parameters:

{
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        768,
        512,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: epoch
per_device_train_batch_size: 32
per_device_eval_batch_size: 16
gradient_accumulation_steps: 16
learning_rate: 2e-05
num_train_epochs: 10
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: True
tf32: True
load_best_model_at_end: True
optim: adamw_torch_fused
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: epoch
prediction_loss_only: True
per_device_train_batch_size: 32
per_device_eval_batch_size: 16
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 16
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 10
max_steps: -1
lr_scheduler_type: cosine
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: True
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: True
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
tp_size: 0
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
1.0	8	-	0.7456	0.7485	0.7467	0.7117	0.6394
1.2759	10	33.8924	-	-	-	-	-
2.0	16	-	0.7781	0.7773	0.7785	0.7466	0.6837
2.5517	20	10.7256	-	-	-	-	-
3.0	24	-	0.7935	0.7870	0.7888	0.7535	0.7016
3.8276	30	5.5408	-	-	-	-	-
4.0	32	-	0.8000	0.7962	0.7969	0.7585	0.7082
5.0	40	3.4556	0.8017	0.8011	0.7992	0.7644	0.7082
6.0	48	-	0.8037	0.8021	0.7974	0.7692	0.7082
6.2759	50	2.9963	-	-	-	-	-
7.0	56	-	0.8025	0.8013	0.7987	0.7719	0.7072
7.5517	60	3.1681	-	-	-	-	-
8.0	64	-	0.8035	0.8024	0.7996	0.7723	0.7077
8.8276	70	2.5551	0.8031	0.8037	0.7995	0.7725	0.7081

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.12.2
Sentence Transformers: 3.4.1
Transformers: 4.50.0
PyTorch: 2.6.0+cu124
Accelerate: 1.5.2
Datasets: 3.4.1
Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}