Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Paper • 1908.10084 • Published • 19
How to use kokitabayashi/e5-mistral_FinModernBERT_query with sentence-transformers:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("kokitabayashi/e5-mistral_FinModernBERT_query")
sentences = [
"What were the primary drivers for the increase in Employer Services' revenues as reported in fiscal 2016?",
"As a Class 4 insurer, Arch Re Bermuda is required to submit an opinion of its approved loss reserve specialist with its statutory financial return in respect of its loss and loss expense provisions. The loss reserve specialist, who will normally be a qualified casualty actuary, must be approved by the BMA.",
"Despite the challenges of the current operating environment, in 2007 we grew our sales from market share gains by approximately 6% and reduced our selling, general and administrative expenses by 14.6% compared to our sales volume decline of 24.6%, when compared to last year.",
"Employer Services' revenues, as reported, increase d 5% in fiscal 2016 , as compared to fiscal 2015 , despite a negative impact of one percentage point from foreign currency translation. Revenues increase d due to new business started from new business bookings, the impact of price increases, and an increase in the number of employees on our clients’ payrolls as our U.S. pays per control increased 2.5% in fiscal 2016 as compared to fiscal 2015 ."
]
embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [4, 4]This is a sentence-transformers model finetuned from intfloat/e5-mistral-7b-instruct on the fin_modern_bert-pairs-sec-synthetic-v1 dataset. It maps sentences & paragraphs to a 4096-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.
SentenceTransformer(
(0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'MistralModel'})
(1): Pooling({'embedding_dimension': 4096, 'pooling_mode': 'lasttoken', 'include_prompt': True})
(2): Normalize({})
)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("kokitabayashi/e5-mistral_FinModernBERT_query")
# Run inference
queries = [
'What were the total capital expenditures for capacity expansion projects and the amount of cash returned to shareholders in fiscal 2015?',
]
documents = [
'In fiscal 2015, our return on average invested capital improved modestly, despite near record levels of capital spending of $120 million in our capacity expansion projects... Also during fiscal 2015, we returned $718 million in cash to our shareholders through dividend payments of $256 million and share repurchases of $462 million, while maintaining investment-grade credit ratings.',
'BNO will be subject to credit risk with respect to counterparties to OTC contracts entered into by BNO or held by special purpose or structured vehicles. Unlike in futures contracts, the counterparty to these contracts is generally a single bank or other financial institution, rather than a clearing organization backed by a group of financial institutions. As a result, there will be greater counterparty credit risk in these transactions.',
'The decrease in core unit volume in 2005 reflects the following factors: \x95 Loss of market share in our North American roll materials business following our implementation of selling price increases to offset higher raw material costs \x95 The impact of an extra week in the 2004 fiscal year \x95 Accelerated purchases by Office and Consumer Products customers in advance of our 2005 selling price increases, which contributed to higher growth in 2004',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 4096] [3, 4096]
# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[1.0000, 1.0000, 0.9999]])
finance-evalInformationRetrievalEvaluator| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.0002 |
| cosine_accuracy@3 | 0.0002 |
| cosine_accuracy@5 | 0.0004 |
| cosine_accuracy@10 | 0.0006 |
| cosine_precision@1 | 0.0002 |
| cosine_precision@3 | 0.0001 |
| cosine_precision@5 | 0.0001 |
| cosine_precision@10 | 0.0001 |
| cosine_recall@1 | 0.0002 |
| cosine_recall@3 | 0.0002 |
| cosine_recall@5 | 0.0004 |
| cosine_recall@10 | 0.0006 |
| cosine_ndcg@10 | 0.0004 |
| cosine_mrr@10 | 0.0003 |
| cosine_map@100 | 0.0007 |
query and positive| query | positive | |
|---|---|---|
| type | string | string |
| modality | text | text |
| details |
|
|
| query | positive |
|---|---|
What were the primary reasons for the decline in gross margins in 2014 compared to 2013? |
Gross margins declined 1 percentage point in 2014 compared to 2013 on slightly higher revenue. Higher inventory charges and an increase in three year warranty costs were the primary reasons for the lower gross margins. |
What is the specific mechanism BNO uses to ensure that losses do not accumulate in its futures accounts? |
Futures Contracts are marked to market at the end of each trading day and the margin required with respect to such contracts is adjusted accordingly. This process of marking-to-market is designed to prevent losses from accumulating in any futures account. |
What were the total net charges recognized for the PTI (Process to Improve) initiative in 2009, and what were the primary drivers of the accelerated depreciation and asset impairment costs? |
Net charges for the PTI were $31 million in 2009. Most of the accelerated depreciation, asset impairment charges and other shutdown costs primarily relate to the rationalization of the manufacturing network in the BioPharmaceuticals segment. |
MultipleNegativesRankingLoss with these parameters:{
"scale": 20.0,
"similarity_fct": "cos_sim",
"gather_across_devices": false,
"directions": [
"query_to_doc"
],
"partition_mode": "joint",
"hardness_mode": null,
"hardness_strength": 0.0
}
query and positive| query | positive | |
|---|---|---|
| type | string | string |
| modality | text | text |
| details |
|
|
| query | positive |
|---|---|
What specific factors contributed to the $88 million increase in Ameren Missouri's electric margins in 2012 compared to 2011? |
Ameren Missouri's electric margins increased by $88 million, or 4%, in 2012 compared with 2011. The following items had a favorable impact on Ameren Missouri's electric margins: Higher electric base rates, effective July 2011 as a result of the 2011 MoPSC electric rate order, which increased revenues by $102 million, partially offset by an increase in net energy costs of $25 million. Reduced purchased power expense as a result of a FERC-ordered refund received from Entergy in 2012 relating to a power purchase agreement that expired in 2009, which increased margins by $24 million. The absence in 2012 of a reduction in revenues recorded in 2011 resulting from the MoPSC's April 2011 FAC prudence review order. The recovery of labor and benefit costs for crews assisting with Hurricane Sandy power restoration, which increased revenues by $7 million and was fully offset by a related increase in operations and maintenance costs, with no overall impact on net income. Higher transmission service... |
What was the net cash provided from operating activities for the fiscal year ended December 31, 2009? |
Net cash provided from operating activities: 358,525 (in thousands) |
What are the maturity dates and interest rates for the notes listed in the filing? |
4n. Form of 5.875% Notes due 2036... 4o. Form of 4.375% Notes due 2016... 4p. Form of 4.625% Notes due 2021... 4q. Form of 5.45% Notes due 2018... 4r. Form of 6.125% Notes due 2038. |
MultipleNegativesRankingLoss with these parameters:{
"scale": 20.0,
"similarity_fct": "cos_sim",
"gather_across_devices": false,
"directions": [
"query_to_doc"
],
"partition_mode": "joint",
"hardness_mode": null,
"hardness_strength": 0.0
}
per_device_train_batch_size: 2num_train_epochs: 1warmup_steps: 0.1gradient_accumulation_steps: 32bf16: Trueper_device_eval_batch_size: 2dataloader_num_workers: 2batch_sampler: no_duplicatesper_device_train_batch_size: 2num_train_epochs: 1max_steps: -1learning_rate: 5e-05lr_scheduler_type: linearlr_scheduler_kwargs: Nonewarmup_steps: 0.1optim: adamw_torch_fusedoptim_args: Noneweight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08optim_target_modules: Nonegradient_accumulation_steps: 32average_tokens_across_devices: Truemax_grad_norm: 1.0label_smoothing_factor: 0.0bf16: Truefp16: Falsebf16_full_eval: Falsefp16_full_eval: Falsetf32: Nonegradient_checkpointing: Falsegradient_checkpointing_kwargs: Nonetorch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Noneuse_liger_kernel: Falseliger_kernel_config: Noneuse_cache: Falseneftune_noise_alpha: Nonetorch_empty_cache_steps: Noneauto_find_batch_size: Falselog_on_each_node: Truelogging_nan_inf_filter: Trueinclude_num_input_tokens_seen: nolog_level: passivelog_level_replica: warningdisable_tqdm: Falseproject: huggingfacetrackio_space_id: Nonetrackio_bucket_id: Nonetrackio_static_space_id: Noneper_device_eval_batch_size: 2prediction_loss_only: Trueeval_on_start: Falseeval_do_concat_batches: Trueeval_use_gather_object: Falseeval_accumulation_steps: Noneinclude_for_metrics: []batch_eval_metrics: Falsesave_only_model: Falsesave_on_each_node: Falseenable_jit_checkpoint: Falsepush_to_hub: Falsehub_private_repo: Nonehub_model_id: Nonehub_strategy: every_savehub_always_push: Falsehub_revision: Noneload_best_model_at_end: Falseignore_data_skip: Falserestore_callback_states_from_checkpoint: Falsefull_determinism: Falseseed: 42data_seed: Noneuse_cpu: Falseaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}parallelism_config: Nonedataloader_drop_last: Falsedataloader_num_workers: 2dataloader_pin_memory: Truedataloader_persistent_workers: Falsedataloader_prefetch_factor: Nonedataloader_multiprocessing_context: Nonedataloader_in_order: Trueremove_unused_columns: Truelabel_names: Nonetrain_sampling_strategy: randomlength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falseddp_static_graph: Noneddp_backend: Noneddp_timeout: 1800fsdp: Nonefsdp_config: Nonedeepspeed: Nonedebug: []skip_memory_metrics: Truedo_predict: Falseresume_from_checkpoint: Nonelocal_rank: -1prompts: Nonebatch_sampler: no_duplicatesmulti_dataset_batch_sampler: proportionalrouter_mapping: {}learning_rate_mapping: {}warmup_ratio: None| Epoch | Step | Training Loss | Validation Loss | finance-eval_cosine_ndcg@10 |
|---|---|---|---|---|
| -1 | -1 | - | - | 0.5788 |
| 0.1431 | 100 | 0.6689 | 0.6931 | 0.0007 |
| 0.2863 | 200 | 0.6931 | 0.6931 | 0.0006 |
| 0.4294 | 300 | 0.6931 | 0.6931 | 0.0007 |
| 0.5726 | 400 | 0.6931 | 0.6931 | 0.0007 |
| 0.7157 | 500 | 0.6931 | 0.6931 | 0.0008 |
| 0.8589 | 600 | 0.6931 | 0.6931 | 0.0006 |
| 1.0 | 699 | - | 0.6931 | 0.0004 |
| -1 | -1 | - | - | 0.0004 |
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
@misc{oord2019representationlearningcontrastivepredictive,
title={Representation Learning with Contrastive Predictive Coding},
author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
year={2019},
eprint={1807.03748},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/1807.03748},
}
Base model
intfloat/e5-mistral-7b-instruct