schaitanya/modernbert-embed-base-transcript-documents-v1

Fine-tuned with Transcripts + Documents v1

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 1024 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("schaitanya/modernbert-embed-base-transcript-documents-v1")
# Run inference
sentences = [
    'What is the recommendation for a water birth if a woman has COVID-19?',
    '- Specific medications and antibody treatment.\nIf you are very unwell, your healthcare team may advise that your baby needs to be born early to help with your own treatment and recovery. How and when this may happen will depend on your individual situation.\nIf I have COVID-19, will this affect where I give birth and my choice of pain relief in labour?\nIf you have symptoms and have tested positive for COVID-19 at the time of birth:\nIt is recommended that you give birth in a consultant led maternity unit where you and your baby can be monitored more closely during labour.\nIt is safe for you to have a vaginal birth, and if you and your baby are both well you do not need to have a planned caesarean birth. Your birth choices should be respected and followed as closely as possible.\nA caesarean birth may be recommended if you or your baby are unwell or there are other complications. However, your chance of needing an emergency caesarean birth may be higher than usual.\nAll the usual options for pain relief for labour and birth are available to you, however a water birth is not recommended. This is because it is harder to monitor and give you any treatments needed.\nIf I have COVID-19, will this affect care of my baby after birth?\nIf your baby is well and does not require care in the neonatal unit, you will stay together after you have given birth. Skin-to-skin contact is encouraged.\nHow you feed your baby is dependent on your own circumstances and preferences, and your choices will be supported. Breastfeeding may help pass protection from infections (including COVID-19) to your baby. There is no strong evidence to show that COVID-19 can be passed on in breast milk.',
    "(0:00 - 4:09)\nSo what is the color of the cake? Pink or blue? What is the color of the nursery? Pink or blue? And what about the baby's clothes? Are they frilly skirts or the soccer shirts? Have you ever wondered what is it which decides the sex of the baby inside? How that little pea-shaped embryo grows into that little baby girl or a baby boy? Since ages it was the mother who was held responsible for the sex of the baby. But now we know it is the father who decides whether it is going to be a pretty little daughter or a handsome baby boy. So hello everyone, this is Dr. Anjali Kumar once again bringing you greetings from Maitri.\n\nMaitri is a space where we talk anything and everything about women's health. So today we are starting our pregnancy series season 2 with this very question which every parent wants to know. So the baby inherits its genes from both the parents.\n\nThe genes are present in the DNA and the DNA is present in the chromosomes and the chromosomes are present in the nucleus of the cell. Every human cell has 23 pairs of chromosomes. So total 46 chromosomes.\n\nEach pair inherited from one parent. 22 of these pairs are called autosomes. They look the same in both males and females.\n\nThe 23rd pair, the sex chromosomes, that's the special one. It differs between males and females. Females have two copies of X chromosomes which makes it XX, while males have one X and one Y which makes it XY.\n\nSo at the time of fertilization, the father's sperm and the mother's egg each contributes one sex chromosomes to the baby. The mother can contribute only X since it has two copies of X chromosomes only, while the father can contribute either X or Y chromosomes. So the baby's biological or the genetic sex which is male or female is determined by the chromosome which the father contributes.\n\nSo if the father contributes his Y chromosome, it will be a male baby which is XY, while if he contributes the X chromosome, it will be a female baby XX. Baby's sex is determined at the time of fertilization or the conception when the sperm fertilizes the egg. Now this typically happens around day 14 to maybe day 17 in women who have regular cycles.\n\nNow this is the time when you don't even know that you are pregnant. You might not be even expecting a pregnancy. This is the time when the baby's sex is decided.\n\nAfter that, nobody and nothing can ever change the genetic sex of the baby. No medicine, no food, no kada, no jariputi, no exercise can change the sex of the baby afterwards.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.5191
cosine_accuracy@3	0.7441
cosine_accuracy@5	0.8
cosine_accuracy@10	0.8912
cosine_precision@1	0.5191
cosine_precision@3	0.248
cosine_precision@5	0.16
cosine_precision@10	0.0891
cosine_recall@1	0.5191
cosine_recall@3	0.7441
cosine_recall@5	0.8
cosine_recall@10	0.8912
cosine_ndcg@10	0.7051
cosine_mrr@10	0.6457
cosine_map@100	0.6503

Training Details

Training Dataset

Unnamed Dataset

• Size: 6,116 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 7 tokens mean: 15.08 tokens max: 33 tokens	min: 38 tokens mean: 389.33 tokens max: 683 tokens

• Samples:

  anchor	positive
  What lifestyle changes are recommended before pregnancy?	(0:02 - 0:50) Are you excited to be a father? How do you think you can help your wife or partner in this journey? Would you want to help your wife or partner during labour? Do you know how does the delivery occurs? Have you read something about the delivery in the baby care? So a lot has been written about women and pregnancy, but we do not talk much about the fathers. Is their role only up to providing the sperm to fertilize the egg? Is it all about the moms? So hello everyone, this is Dr. Anjali Kumar, once again bringing greetings from Maitri. Maitri is a space where we talk anything and everything about women's health. (0:50 - 1:06) But this time in this episode, we will talk about the fathers. We were not sure when to plan our family. She wanted a baby early and I wanted to wait for a few years. (1:07 - 1:28) Plan and talk when you want to plan the pregnancy. Plan well the career, the finances, visit a doctor together for the pre-conceptional checks, tests and the contraceptive ...
  Does the absence of symptoms indicate an absence of infection?	(0:00 - 0:21) Very important point. Some people with STDs may not actually have any symptoms. Now this means that the person is a carrier of infection but she is absolutely capable of transmitting the infection to the other person. So remember absence of symptoms does not mean absence of infection.
  When does pre-eclampsia usually occur during pregnancy?	What is pre-eclampsia? Pre-eclampsia is a condition that usually happens after 20 weeks of pregnancy. The exact cause of pre-eclampsia is not understood. It is usually a combination of: raised blood pressure (hypertension) protein in your urine (proteinuria). Sometimes pre-eclampsia can affect your liver, kidneys and blood clotting without protein in your urine. Pre-eclampsia is common, affecting between 1–5 in 100 women during pregnancy. It is usually mild but in a small number of cases, it can develop into a more serious illness. Around one in 200 women develop severe pre-eclampsia, which can be life-threatening for both you and your baby. How will I know if I have pre-eclampsia? Often you will have no symptoms and pre-eclampsia may be diagnosed for the first time at your routine antenatal appointments or during labour when you have your blood pressure checked and your urine tested. If you do develop symptoms they usually happen towards the end of your pregnancy but can also happen f...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• log_level: debug
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: debug
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: True
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	cosine_ndcg@10
0.4178	10	5.9152	-
0.8355	20	2.7824	-
0.9608	23	-	0.6781
1.2924	30	1.9575	-
1.7102	40	1.5202	-
1.9608	46	-	0.6943
2.1671	50	1.4008	-
2.5849	60	1.1741	-
2.9608	69	-	0.7031
3.0418	70	1.0995	-
3.4595	80	1.0416	-
3.8773	90	1.1648	-
3.9608	92	-	0.7051

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.4.1
• Transformers: 4.49.0
• PyTorch: 2.6.0+cu124
• Accelerate: 1.3.0
• Datasets: 3.4.1
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}