ChenyuEcho/hospital_emaillevel_LoRA_newtrainmethod

SentenceTransformer based on Qwen/Qwen3-Embedding-0.6B

This is a sentence-transformers model finetuned from Qwen/Qwen3-Embedding-0.6B. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Qwen/Qwen3-Embedding-0.6B
• Maximum Sequence Length: 32768 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 32768, 'do_lower_case': False, 'architecture': 'Qwen3Model'})
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    "What is the current status of reviewing the attached document referenced by Jennifer Morrison for Margaret L. Hendricks?",
]
documents = [
    'Subject: See Attached Document\nDate: 2025-11-12T10:37:00\nFrom: Jennifer Morrison\nParticipants: Margaret L. Hendricks\n\nBody:\nHello Margaret,\n\nI am reaching out to provide you with the attached file as previously referenced in our correspondence. Please take a moment to review the attached document at your earliest convenience, as it contains important information relevant to your current inquiry. If you have any questions or require clarification regarding the content, do not hesitate to reach out to me by reply email or phone. I am committed to supporting you throughout this process and ensuring you have all necessary documentation for your review.\n\nKind regards,\nJennifer Morrison',
    'Subject: Confirmation of Attendance for Upcoming Meeting\nDate: 2025-12-08T12:46:00\nFrom: Barbara J. Young\nParticipants: Carol A. Campbell\n\nBody:\nHi Carol,\n\nThank you for reaching out regarding your attendance at the upcoming meeting. I can confirm that your slot is secured and you are listed on our schedule for Thursday at 10:00 AM. If you have any specific topics or materials you intend to present, please forward those to me so I can ensure the agenda accommodates your contributions. Feel free to notify me if there are any changes to your availability.\n\nRegards,\nBarbara\n\n--\nBarbara J. Young | Executive Office',
    "Subject: Re: Urgent: Suction Equipment Malfunction Impacting Medication Administration\nDate: 2025-10-20T15:51:00\nFrom: Karen M. Phillips\nParticipants: Taylor A. Richardson\n\nBody:\nHi Taylor,\n\nThank you for raising this concern so promptly. I recommend we temporarily review all patient regimens in the east wing, especially those involving oral and respiratory medications, to identify any high-risk cases where delays could impact drug absorption or safety. In the interim, please ensure that nursing staff document any instances where medication administration is delayed due to equipment issues. I will coordinate with pharmacy and clinical teams to develop interim precautions and will follow up once we've completed the assessment.\n\nBest regards,\nKaren",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 1024] [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.6719,  0.0349, -0.0535]], dtype=torch.bfloat16)

Evaluation

Metrics

Information Retrieval

• Dataset: val_full_corpus
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8289
cosine_accuracy@3	0.9086
cosine_accuracy@5	0.9402
cosine_accuracy@10	0.9718
cosine_precision@1	0.8289
cosine_precision@3	0.3029
cosine_precision@5	0.188
cosine_precision@10	0.0972
cosine_recall@1	0.8289
cosine_recall@3	0.9086
cosine_recall@5	0.9402
cosine_recall@10	0.9718
cosine_ndcg@10	0.9003
cosine_mrr@10	0.8774
cosine_map@100	0.8793

Training Details

Training Dataset

Unnamed Dataset

• Size: 2,384 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 10 tokens mean: 26.82 tokens max: 73 tokens	min: 109 tokens mean: 177.36 tokens max: 470 tokens

• Samples:

  sentence_0	sentence_1
  Who is the selected EHR vendor for the upgrade?	Subject: Upcoming Electronic Health Record System Upgrade: Strategic Overview & Impact Date: 2025-09-08T15:43:00 From: Catherine R. Miller Participants: All Hospital Staff Body: Dear St. Catherine's Regional Hospital Team, I am pleased to announce that our hospital will be implementing a major upgrade to our Electronic Health Record (EHR) system, scheduled for Q4 this fiscal year. This strategic investment, reflective of our ongoing commitment to operational excellence, efficiency, and compliance, has been carefully budgeted at $1.2M. While this represents a significant outlay, our projected long-term savings on administrative costs and revenue cycle optimization are estimated to exceed $500K annually. We evaluated several vendors to ensure our chosen solution will maximize data security, interoperability, and regulatory compliance. The transition is expected to have a direct positive effect on both patient outcomes and our financial performance. Training sessions and resource guide...
  What is the plan to develop a comprehensive documentation checklist to address anesthesia supply claim rejections caused by documentation gaps?	Subject: Re: Claim Rejection Analysis: Supply Documentation Concern Date: 2026-01-28T17:43:00 From: Maya S. Patel Participants: Carol A. Campbell Body: Hi Carol, Thank you for highlighting the ongoing issues with anesthesia supply claim rejections tied to documentation gaps. I agree this is a significant concern that requires immediate attention to prevent further disruptions with vendor reconciliations and ensure claims compliance. I’m available later this week to review our current documentation process in detail and collaborate on a comprehensive checklist. Please let me know your availability so we can set up a meeting. Best regards, Maya
  What decisions, owners, and timelines exist for implementing a prioritization protocol for lab draws due to limited isolation room capacity, including proposed criteria, scheduled collection intervals, and cross-functional actions with lab and nursing teams?	Subject: Re: Isolation Room Capacity and Lab Processing Constraints Date: 2025-12-29T17:47:00 From: David S. Wilson Participants: Linda R. Taylor Body: Hi Linda, Thank you for reaching out and bringing this concern to my attention. I completely understand the challenges that our limited isolation room capacity is causing, especially as it relates to delays with critical specimen processing. I agree that implementing a prioritization protocol for lab draws, coupled with scheduled collection intervals, could help mitigate some of these bottlenecks. Let's set up a meeting with the lab and nursing teams to review current workflows and identify actionable strategies to streamline urgent cases. In the meantime, please let me know if you have specific cases that require immediate intervention, and I'll coordinate with the necessary departments to ensure we address those promptly. Looking forward to collaborating on this important issue. Best regards, David

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	val_full_corpus_cosine_ndcg@10
1.0	149	0.8884
2.0	298	0.8970
3.0	447	0.9003

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.2.3
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}