e5-mistral-7b-instruct base trained on BatuhanECB/FinModernBERT-pairs-sec-synthetic-v1

This is a sentence-transformers model finetuned from intfloat/e5-mistral-7b-instruct on the fin_modern_bert-pairs-sec-synthetic-v1 dataset. It maps sentences & paragraphs to a 4096-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: intfloat/e5-mistral-7b-instruct
  • Maximum Sequence Length: 32768 tokens
  • Output Dimensionality: 4096 dimensions
  • Similarity Function: Cosine Similarity
  • Supported Modality: Text
  • Training Dataset:
    • fin_modern_bert-pairs-sec-synthetic-v1
  • Language: en
  • License: apache-2.0

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'MistralModel'})
  (1): Pooling({'embedding_dimension': 4096, 'pooling_mode': 'lasttoken', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("kokitabayashi/e5-mistral_FinModernBERT_query")
# Run inference
queries = [
    'What were the total capital expenditures for capacity expansion projects and the amount of cash returned to shareholders in fiscal 2015?',
]
documents = [
    'In fiscal 2015, our return on average invested capital improved modestly, despite near record levels of capital spending of $120 million in our capacity expansion projects... Also during fiscal 2015, we returned $718 million in cash to our shareholders through dividend payments of $256 million and share repurchases of $462 million, while maintaining investment-grade credit ratings.',
    'BNO will be subject to credit risk with respect to counterparties to OTC contracts entered into by BNO or held by special purpose or structured vehicles. Unlike in futures contracts, the counterparty to these contracts is generally a single bank or other financial institution, rather than a clearing organization backed by a group of financial institutions. As a result, there will be greater counterparty credit risk in these transactions.',
    'The decrease in core unit volume in 2005 reflects the following factors: \x95 Loss of market share in our North American roll materials business following our implementation of selling price increases to offset higher raw material costs \x95 The impact of an extra week in the 2004 fiscal year \x95 Accelerated purchases by Office and Consumer Products customers in advance of our 2005 selling price increases, which contributed to higher growth in 2004',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 4096] [3, 4096]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[1.0000, 1.0000, 0.9999]])

Evaluation

Metrics

Information Retrieval

Metric Value
cosine_accuracy@1 0.0002
cosine_accuracy@3 0.0002
cosine_accuracy@5 0.0004
cosine_accuracy@10 0.0006
cosine_precision@1 0.0002
cosine_precision@3 0.0001
cosine_precision@5 0.0001
cosine_precision@10 0.0001
cosine_recall@1 0.0002
cosine_recall@3 0.0002
cosine_recall@5 0.0004
cosine_recall@10 0.0006
cosine_ndcg@10 0.0004
cosine_mrr@10 0.0003
cosine_map@100 0.0007

Training Details

Training Dataset

fin_modern_bert-pairs-sec-synthetic-v1

  • Dataset: fin_modern_bert-pairs-sec-synthetic-v1
  • Size: 44,709 training samples
  • Columns: query and positive
  • Approximate statistics based on the first 100 samples:
    query positive
    type string string
    modality text text
    details
    • min: 15 tokens
    • mean: 30.88 tokens
    • max: 50 tokens
    • min: 14 tokens
    • mean: 79.35 tokens
    • max: 277 tokens
  • Samples:
    query positive
    What were the primary reasons for the decline in gross margins in 2014 compared to 2013? Gross margins declined 1 percentage point in 2014 compared to 2013 on slightly higher revenue. Higher inventory charges and an increase in three year warranty costs were the primary reasons for the lower gross margins.
    What is the specific mechanism BNO uses to ensure that losses do not accumulate in its futures accounts? Futures Contracts are marked to market at the end of each trading day and the margin required with respect to such contracts is adjusted accordingly. This process of marking-to-market is designed to prevent losses from accumulating in any futures account.
    What were the total net charges recognized for the PTI (Process to Improve) initiative in 2009, and what were the primary drivers of the accelerated depreciation and asset impairment costs? Net charges for the PTI were $31 million in 2009. Most of the accelerated depreciation, asset impairment charges and other shutdown costs primarily relate to the rationalization of the manufacturing network in the BioPharmaceuticals segment.
  • Loss: MultipleNegativesRankingLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim",
        "gather_across_devices": false,
        "directions": [
            "query_to_doc"
        ],
        "partition_mode": "joint",
        "hardness_mode": null,
        "hardness_strength": 0.0
    }
    

Evaluation Dataset

fin_modern_bert-pairs-sec-synthetic-v1

  • Dataset: fin_modern_bert-pairs-sec-synthetic-v1
  • Size: 9,580 evaluation samples
  • Columns: query and positive
  • Approximate statistics based on the first 100 samples:
    query positive
    type string string
    modality text text
    details
    • min: 18 tokens
    • mean: 31.56 tokens
    • max: 48 tokens
    • min: 14 tokens
    • mean: 70.35 tokens
    • max: 301 tokens
  • Samples:
    query positive
    What specific factors contributed to the $88 million increase in Ameren Missouri's electric margins in 2012 compared to 2011? Ameren Missouri's electric margins increased by $88 million, or 4%, in 2012 compared with 2011. The following items had a favorable impact on Ameren Missouri's electric margins: Higher electric base rates, effective July 2011 as a result of the 2011 MoPSC electric rate order, which increased revenues by $102 million, partially offset by an increase in net energy costs of $25 million. Reduced purchased power expense as a result of a FERC-ordered refund received from Entergy in 2012 relating to a power purchase agreement that expired in 2009, which increased margins by $24 million. The absence in 2012 of a reduction in revenues recorded in 2011 resulting from the MoPSC's April 2011 FAC prudence review order. The recovery of labor and benefit costs for crews assisting with Hurricane Sandy power restoration, which increased revenues by $7 million and was fully offset by a related increase in operations and maintenance costs, with no overall impact on net income. Higher transmission service...
    What was the net cash provided from operating activities for the fiscal year ended December 31, 2009? Net cash provided from operating activities: 358,525 (in thousands)
    What are the maturity dates and interest rates for the notes listed in the filing? 4n. Form of 5.875% Notes due 2036... 4o. Form of 4.375% Notes due 2016... 4p. Form of 4.625% Notes due 2021... 4q. Form of 5.45% Notes due 2018... 4r. Form of 6.125% Notes due 2038.
  • Loss: MultipleNegativesRankingLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim",
        "gather_across_devices": false,
        "directions": [
            "query_to_doc"
        ],
        "partition_mode": "joint",
        "hardness_mode": null,
        "hardness_strength": 0.0
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 2
  • num_train_epochs: 1
  • warmup_steps: 0.1
  • gradient_accumulation_steps: 32
  • bf16: True
  • per_device_eval_batch_size: 2
  • dataloader_num_workers: 2
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • per_device_train_batch_size: 2
  • num_train_epochs: 1
  • max_steps: -1
  • learning_rate: 5e-05
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: None
  • warmup_steps: 0.1
  • optim: adamw_torch_fused
  • optim_args: None
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • optim_target_modules: None
  • gradient_accumulation_steps: 32
  • average_tokens_across_devices: True
  • max_grad_norm: 1.0
  • label_smoothing_factor: 0.0
  • bf16: True
  • fp16: False
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • use_liger_kernel: False
  • liger_kernel_config: None
  • use_cache: False
  • neftune_noise_alpha: None
  • torch_empty_cache_steps: None
  • auto_find_batch_size: False
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • include_num_input_tokens_seen: no
  • log_level: passive
  • log_level_replica: warning
  • disable_tqdm: False
  • project: huggingface
  • trackio_space_id: None
  • trackio_bucket_id: None
  • trackio_static_space_id: None
  • per_device_eval_batch_size: 2
  • prediction_loss_only: True
  • eval_on_start: False
  • eval_do_concat_batches: True
  • eval_use_gather_object: False
  • eval_accumulation_steps: None
  • include_for_metrics: []
  • batch_eval_metrics: False
  • save_only_model: False
  • save_on_each_node: False
  • enable_jit_checkpoint: False
  • push_to_hub: False
  • hub_private_repo: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_always_push: False
  • hub_revision: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • restore_callback_states_from_checkpoint: False
  • full_determinism: False
  • seed: 42
  • data_seed: None
  • use_cpu: False
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • dataloader_drop_last: False
  • dataloader_num_workers: 2
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • dataloader_prefetch_factor: None
  • dataloader_multiprocessing_context: None
  • dataloader_in_order: True
  • remove_unused_columns: True
  • label_names: None
  • train_sampling_strategy: random
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • ddp_static_graph: None
  • ddp_backend: None
  • ddp_timeout: 1800
  • fsdp: None
  • fsdp_config: None
  • deepspeed: None
  • debug: []
  • skip_memory_metrics: True
  • do_predict: False
  • resume_from_checkpoint: None
  • local_rank: -1
  • prompts: None
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}
  • warmup_ratio: None

Training Logs

Epoch Step Training Loss Validation Loss finance-eval_cosine_ndcg@10
-1 -1 - - 0.5788
0.1431 100 0.6689 0.6931 0.0007
0.2863 200 0.6931 0.6931 0.0006
0.4294 300 0.6931 0.6931 0.0007
0.5726 400 0.6931 0.6931 0.0007
0.7157 500 0.6931 0.6931 0.0008
0.8589 600 0.6931 0.6931 0.0006
1.0 699 - 0.6931 0.0004
-1 -1 - - 0.0004

Training Time

  • Training: 3.2 hours

Framework Versions

  • Python: 3.13.16
  • Sentence Transformers: 5.7.0
  • Transformers: 5.22.0
  • PyTorch: 2.11.0+cu130
  • Accelerate: 1.15.0
  • Datasets: 4.8.5
  • Tokenizers: 0.23.3

Additional Resources

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}
Downloads last month
60
Safetensors
Model size
7B params
Tensor type
F16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for kokitabayashi/e5-mistral_FinModernBERT_query

Finetuned
(7)
this model

Papers for kokitabayashi/e5-mistral_FinModernBERT_query

Evaluation results