lizchu414/mpnet-base-all-pittsburgh-squad

MPNet base trained on synthetic Pittsburgh data

This is a sentence-transformers model finetuned from sentence-transformers/all-mpnet-base-v2. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-mpnet-base-v2
• Maximum Sequence Length: 384 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("lizchu414/mpnet-base-all-pittsburgh-squad")
# Run inference
sentences = [
    '"What cultural celebration will be honored at the 2024 Greater Pittsburgh Lunar New Year Gala, and what is the significance of this event in the community?"',
    'Event Name: 2024 Greater Pittsburgh Lunar New Year Gala\nCategories: Arts + Culture, Community, Holidays, Nightlife\nDates: Feb 3, 2024 - Feb 3, 2024   | 4:00 pm - 9:00 pm\nLocation: PNC Theater, 350 Forbes Avenue, Pittsburgh, PA 15222',
    "This page informs City of Pittsburgh residents about the city's Snow Angels program. This page is also where volunteers can sign up, and recipients can submit a request.\nCity Collection Equity Audit\nThe City of Pittsburgh is conducting an audit to identify inequity and bias in the City’s collection of public art and memorials.\nDavis Avenue Bridge\nDesign and construction for the new Davis Avenue Bridge between Brighton Heights and Riverview Park.\nSouth Side Park Public Art\nA new public art project is being planned in South Side Park. This is being done in coordination with the park’s Phase 1 renovations and funded by the Percent For Art.\nProjects that are no longer accepting feedback, but are now in the construction or development phase.\nPHAD Projects\nCurrent Projects – find out about ongoing projects underway throughout the city and learn how to apply for new projects each year.\nEmerald View Phase I Trails & Trailheads",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: pittsburgh
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7375
cosine_accuracy@3	0.9038
cosine_accuracy@5	0.9369
cosine_accuracy@10	0.9628
cosine_precision@1	0.7375
cosine_precision@3	0.3013
cosine_precision@5	0.1874
cosine_precision@10	0.0963
cosine_recall@1	0.7375
cosine_recall@3	0.9038
cosine_recall@5	0.9369
cosine_recall@10	0.9628
cosine_ndcg@10	0.859
cosine_mrr@10	0.8248
cosine_map@100	0.8263
dot_accuracy@1	0.7375
dot_accuracy@3	0.9038
dot_accuracy@5	0.9369
dot_accuracy@10	0.9628
dot_precision@1	0.7375
dot_precision@3	0.3013
dot_precision@5	0.1874
dot_precision@10	0.0963
dot_recall@1	0.7375
dot_recall@3	0.9038
dot_recall@5	0.9369
dot_recall@10	0.9628
dot_ndcg@10	0.859
dot_mrr@10	0.8248
dot_map@100	0.8263

Training Details

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_eval_batch_size: 2
• eval_accumulation_steps: 1
• learning_rate: 2e-05
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 2
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: 1
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	pittsburgh_dot_map@100
0	0	-	-	0.5984
0.8	100	0.587	0.1954	0.7780
1.592	200	0.1828	0.1805	0.8020
2.384	300	0.2224	0.1605	0.8263

Framework Versions

• Python: 3.12.7
• Sentence Transformers: 3.2.0
• Transformers: 4.45.2
• PyTorch: 2.2.2+cu121
• Accelerate: 1.0.1
• Datasets: 3.0.1
• Tokenizers: 0.20.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}