BjarneNPO/finetune_21_08_2025_18_14_50

SentenceTransformer based on Snowflake/snowflake-arctic-embed-m-v2.0

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-m-v2.0 on the train dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Snowflake/snowflake-arctic-embed-m-v2.0
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • train

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'GteModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("BjarneNPO/finetune_21_08_2025_18_14_50")
# Run inference
queries = [
    "fragt wie der Stand zu dem aktuellen Problem ist",
]
documents = [
    'In Klärung mit der Kollegin - Das Problem liegt leider an deren Betreiber. Die sind aber informiert und arbeiten bereits daran',
    'findet diese in der Übersicht der Gruppen.',
    'Userin muss sich an die Bistums IT wenden.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.2935, 0.0077, 0.0270]])

Evaluation

Metrics

Information Retrieval

• Dataset: Snowflake/snowflake-arctic-embed-m-v2.0
• Evaluated with scripts.InformationRetrievalEvaluatorCustom.InformationRetrievalEvaluatorCustom with these parameters:

  {
      "query_prompt_name": "query",
      "corpus_prompt_name": "document"
  }
  

Metric	Value
cosine_accuracy@1	0.2628
cosine_accuracy@3	0.5401
cosine_accuracy@5	0.6131
cosine_accuracy@10	0.7007
cosine_precision@1	0.2628
cosine_precision@3	0.2798
cosine_precision@5	0.238
cosine_precision@10	0.2387
cosine_recall@1	0.0078
cosine_recall@3	0.0353
cosine_recall@5	0.0577
cosine_recall@10	0.0937
cosine_ndcg@10	0.2537
cosine_mrr@10	0.4159
cosine_map@100	0.1106

Training Details

Training Dataset

train

• Dataset: train
• Size: 19,964 training samples
• Columns: query and answer
• Approximate statistics based on the first 1000 samples:

  	query	answer
  type	string	string
  details	min: 4 tokens mean: 27.77 tokens max: 615 tokens	min: 3 tokens mean: 22.87 tokens max: 151 tokens

• Samples:

  query	answer
  Wie kann man die Jahresurlaubsübersicht exportieren?	über das 3 Punkte Menü rechts oben. Mitarbeiter auswählen und exportieren
  1. Vertragsabschlüsse werden nicht übertragen
  2. Kinder kommen nicht von nach
  3. Absage kann bei Portalstatus nicht erstellt werden.	Ticket
  Userin gebeten sich an den Support zu wenden, da der Fehler liegt.
  Wird im Anmeldeportal nicht gefunden.	Die Schnittstelle war noch nicht aktiviert und Profil ebenfalls nicht.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• gradient_accumulation_steps: 4
• learning_rate: 2e-05
• num_train_epochs: 10
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 4
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: True
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Snowflake/snowflake-arctic-embed-m-v2.0_cosine_ndcg@10
0.1282	10	3.4817	-
0.2564	20	3.3293	-
0.3846	30	3.2454	-
0.5128	40	2.9855	-
0.6410	50	2.836	-
0.7692	60	2.6829	-
0.8974	70	2.5137	-
1.0	78	-	0.2860
1.0256	80	2.2992	-
1.1538	90	2.2564	-
1.2821	100	2.1378	-
1.4103	110	2.1207	-
1.5385	120	2.0066	-
1.6667	130	1.955	-
1.7949	140	1.8552	-
1.9231	150	1.8479	-
2.0	156	-	0.2599
2.0513	160	1.6706	-
2.1795	170	1.7236	-
2.3077	180	1.7102	-
2.4359	190	1.6504	-
2.5641	200	1.6336	-
2.6923	210	1.5329	-
2.8205	220	1.5751	-
2.9487	230	1.4674	-
3.0	234	-	0.2617
3.0769	240	1.3776	-
3.2051	250	1.4624	-
3.3333	260	1.4278	-
3.4615	270	1.4219	-
3.5897	280	1.3454	-
3.7179	290	1.3035	-
3.8462	300	1.3244	-
3.9744	310	1.2764	-
4.0	312	-	0.2474
4.1026	320	1.1612	-
4.2308	330	1.2641	-
4.3590	340	1.2748	-
4.4872	350	1.1997	-
4.6154	360	1.1697	-
4.7436	370	1.1381	-
4.8718	380	1.1687	-
5.0	390	1.0895	0.2536
5.1282	400	1.0488	-
5.2564	410	1.123	-
5.3846	420	1.1352	-
5.5128	430	1.0717	-
5.6410	440	1.0292	-
5.7692	450	1.0195	-
5.8974	460	1.0495	-
6.0	468	-	0.2523
6.0256	470	0.9438	-
6.1538	480	1.0088	-
6.2821	490	0.9978	-
6.4103	500	1.0165	-
6.5385	510	0.9807	-
6.6667	520	0.9563	-
6.7949	530	0.971	-
6.9231	540	0.9676	-
7.0	546	-	0.2563
7.0513	550	0.8548	-
7.1795	560	0.9309	-
7.3077	570	0.9712	-
7.4359	580	0.9342	-
7.5641	590	0.9417	-
7.6923	600	0.8594	-
7.8205	610	0.9432	-
7.9487	620	0.8845	-
8.0	624	-	0.2570
8.0769	630	0.8528	-
8.2051	640	0.9071	-
8.3333	650	0.9064	-
8.4615	660	0.9166	-
8.5897	670	0.8569	-
8.7179	680	0.8539	-
8.8462	690	0.8965	-
8.9744	700	0.8716	-
9.0	702	-	0.2562
9.1026	710	0.7946	-
9.2308	720	0.8884	-
9.3590	730	0.9222	-
9.4872	740	0.898	-
9.6154	750	0.8687	-
9.7436	760	0.8556	-
9.8718	770	0.8921	-
10.0	780	0.8502	0.2537

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.11
• Sentence Transformers: 5.1.0
• Transformers: 4.55.2
• PyTorch: 2.8.0+cu129
• Accelerate: 1.10.0
• Datasets: 3.6.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}