BrooksLi/bge-base-financial-matryoshka

BGE base Financial Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5 on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("BrooksLi/bge-base-financial-matryoshka")
# Run inference
sentences = [
    'The Cybersecurity Strategy outlines the key priorities for our cybersecurity program and the methods by which our Information Security department seeks to accomplish those goals, including assessing risk, prioritizing testing, identifying remedial actions and validating improvements.',
    'What are the key priorities outlined in the cybersecurity strategy described?',
    'What was the percentage increase in Intelligent Edge net revenue in fiscal 2023 compared to fiscal 2022?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.8907, 0.1987],
#         [0.8907, 1.0000, 0.1670],
#         [0.1987, 0.1670, 1.0000]])

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Metric	Value
cosine_accuracy@1	0.72
cosine_accuracy@3	0.8257
cosine_accuracy@5	0.87
cosine_accuracy@10	0.92
cosine_precision@1	0.72
cosine_precision@3	0.2752
cosine_precision@5	0.174
cosine_precision@10	0.092
cosine_recall@1	0.72
cosine_recall@3	0.8257
cosine_recall@5	0.87
cosine_recall@10	0.92
cosine_ndcg@10	0.8181
cosine_mrr@10	0.7858
cosine_map@100	0.7881

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 512
  }
  

Metric	Value
cosine_accuracy@1	0.7186
cosine_accuracy@3	0.8257
cosine_accuracy@5	0.8629
cosine_accuracy@10	0.92
cosine_precision@1	0.7186
cosine_precision@3	0.2752
cosine_precision@5	0.1726
cosine_precision@10	0.092
cosine_recall@1	0.7186
cosine_recall@3	0.8257
cosine_recall@5	0.8629
cosine_recall@10	0.92
cosine_ndcg@10	0.8164
cosine_mrr@10	0.7836
cosine_map@100	0.7861

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 256
  }
  

Metric	Value
cosine_accuracy@1	0.7143
cosine_accuracy@3	0.8286
cosine_accuracy@5	0.8671
cosine_accuracy@10	0.9043
cosine_precision@1	0.7143
cosine_precision@3	0.2762
cosine_precision@5	0.1734
cosine_precision@10	0.0904
cosine_recall@1	0.7143
cosine_recall@3	0.8286
cosine_recall@5	0.8671
cosine_recall@10	0.9043
cosine_ndcg@10	0.81
cosine_mrr@10	0.7797
cosine_map@100	0.7832

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 128
  }
  

Metric	Value
cosine_accuracy@1	0.6986
cosine_accuracy@3	0.81
cosine_accuracy@5	0.86
cosine_accuracy@10	0.9043
cosine_precision@1	0.6986
cosine_precision@3	0.27
cosine_precision@5	0.172
cosine_precision@10	0.0904
cosine_recall@1	0.6986
cosine_recall@3	0.81
cosine_recall@5	0.86
cosine_recall@10	0.9043
cosine_ndcg@10	0.7991
cosine_mrr@10	0.7656
cosine_map@100	0.7689

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 64
  }
  

Metric	Value
cosine_accuracy@1	0.6686
cosine_accuracy@3	0.7829
cosine_accuracy@5	0.8286
cosine_accuracy@10	0.8814
cosine_precision@1	0.6686
cosine_precision@3	0.261
cosine_precision@5	0.1657
cosine_precision@10	0.0881
cosine_recall@1	0.6686
cosine_recall@3	0.7829
cosine_recall@5	0.8286
cosine_recall@10	0.8814
cosine_ndcg@10	0.7739
cosine_mrr@10	0.7396
cosine_map@100	0.7435

Training Details

Training Dataset

json

• Dataset: json
• Size: 6,300 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 8 tokens mean: 45.85 tokens max: 301 tokens	min: 9 tokens mean: 20.65 tokens max: 40 tokens

• Samples:

  positive	anchor
  In the U.S., REMICADE sales decreased by 19.3% from 2022 to 2023.	What was the percentage change in sales for REMICADE in the U.S. from 2022 to 2023?
  AbbVie is a global, diversified research-based biopharmaceutical company with a comprehensive product portfolio that has leadership positions across immunology, oncology, aesthetics, neuroscience, and eye care.	What types of products does AbbVie's portfolio include?
  Prior to January 1, 2023, intersegment adjusted operating income eliminations occurred when members of the Health Services segment's clients enrolled in Maintenance Choice® elected to pick up maintenance prescriptions at one of the Company’s retail pharmacies instead of receiving them through the mail, with both segments recording the income separately. From January 1, 2023, this income is reported only in the Pharmacy & Consumer Wellness segment, making intersegment elimination unnecessary.	What policy change regarding adjusted operating income reporting took effect on January 1, 2023?

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
0.4061	10	1.0474	-	-	-	-	-
0.8122	20	0.5157	-	-	-	-	-
1.0	25	-	0.8080	0.8088	0.8016	0.7924	0.7586
1.2030	30	0.2927	-	-	-	-	-
1.6091	40	0.2701	-	-	-	-	-
2.0	50	0.2486	0.8153	0.8129	0.8100	0.7974	0.7694
2.4061	60	0.2336	-	-	-	-	-
2.8122	70	0.2097	-	-	-	-	-
3.0	75	-	0.8169	0.8144	0.8088	0.7987	0.7741
3.2030	80	0.1821	-	-	-	-	-
3.6091	90	0.1976	-	-	-	-	-
4.0	100	0.1703	0.8181	0.8164	0.81	0.7991	0.7739

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 5.0.0
• Transformers: 4.55.0
• PyTorch: 2.6.0+cu124
• Accelerate: 1.9.0
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}