BjarneNPO/finetune_21_08_2025_11_36_21

SentenceTransformer based on Snowflake/snowflake-arctic-embed-m-v2.0

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-m-v2.0 on the train dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Snowflake/snowflake-arctic-embed-m-v2.0
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • train

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'GteModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("BjarneNPO/finetune_21_08_2025_11_36_21")
# Run inference
queries = [
    "fragt wie der Stand zu dem aktuellen Problem ist",
]
documents = [
    'In Klärung mit der Kollegin - Das Problem liegt leider an deren Betreiber. Die sind aber informiert und arbeiten bereits daran',
    'findet diese in der Übersicht der Gruppen.',
    'Userin muss sich an die Bistums IT wenden.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.2882, -0.0027, -0.0251]])

Evaluation

Metrics

Information Retrieval

• Dataset: Snowflake/snowflake-arctic-embed-m-v2.0
• Evaluated with scripts.InformationRetrievalEvaluatorCustom.InformationRetrievalEvaluatorCustom with these parameters:

  {
      "query_prompt_name": "query",
      "corpus_prompt_name": "query"
  }
  

Metric	Value
cosine_accuracy@1	0.1825
cosine_accuracy@3	0.4015
cosine_accuracy@5	0.5036
cosine_accuracy@10	0.708
cosine_precision@1	0.1825
cosine_precision@3	0.2482
cosine_precision@5	0.2394
cosine_precision@10	0.2022
cosine_recall@1	0.0133
cosine_recall@3	0.0277
cosine_recall@5	0.0478
cosine_recall@10	0.0836
cosine_ndcg@10	0.2223
cosine_mrr@10	0.3408
cosine_map@100	0.0965

Training Details

Training Dataset

train

• Dataset: train
• Size: 19,964 training samples
• Columns: query and answer
• Approximate statistics based on the first 1000 samples:

  	query	answer
  type	string	string
  details	min: 4 tokens mean: 27.77 tokens max: 615 tokens	min: 3 tokens mean: 22.87 tokens max: 151 tokens

• Samples:

  query	answer
  Wie kann man die Jahresurlaubsübersicht exportieren?	über das 3 Punkte Menü rechts oben. Mitarbeiter auswählen und exportieren
  1. Vertragsabschlüsse werden nicht übertragen
  2. Kinder kommen nicht von nach
  3. Absage kann bei Portalstatus nicht erstellt werden.	Ticket
  Userin gebeten sich an den Support zu wenden, da der Fehler liegt.
  Wird im Anmeldeportal nicht gefunden.	Die Schnittstelle war noch nicht aktiviert und Profil ebenfalls nicht.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 8
• learning_rate: 2e-05
• num_train_epochs: 10
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: False
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 8
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss	Snowflake/snowflake-arctic-embed-m-v2.0_cosine_ndcg@10
0.0641	10	2.1581	-
0.1282	20	2.259	-
0.1923	30	2.0893	-
0.2564	40	1.9604	-
0.3205	50	1.9998	-
0.3846	60	1.8531	-
0.4487	70	1.7507	-
0.5128	80	1.7337	-
0.5769	90	1.6102	-
0.6410	100	1.5674	-
0.7051	110	1.4687	-
0.7692	120	1.4123	-
0.8333	130	1.4028	-
0.8974	140	1.2895	-
0.9615	150	1.2337	-
1.0	156	-	0.2480
1.0256	160	1.1688	-
1.0897	170	1.1417	-
1.1538	180	1.2043	-
1.2179	190	1.0708	-
1.2821	200	1.0728	-
1.3462	210	1.1041	-
1.4103	220	1.0581	-
1.4744	230	0.9842	-
1.5385	240	0.9806	-
1.6026	250	0.9709	-
1.6667	260	0.9564	-
1.7308	270	0.9005	-
1.7949	280	0.908	-
1.8590	290	0.8904	-
1.9231	300	0.8536	-
1.9872	310	0.8062	-
2.0	312	-	0.2458
2.0513	320	0.7144	-
2.1154	330	0.7484	-
2.1795	340	0.8055	-
2.2436	350	0.7487	-
2.3077	360	0.8139	-
2.3718	370	0.7268	-
2.4359	380	0.7472	-
2.5	390	0.7173	-
2.5641	400	0.7024	-
2.6282	410	0.6622	-
2.6923	420	0.6446	-
2.7564	430	0.6699	-
2.8205	440	0.6792	-
2.8846	450	0.5952	-
2.9487	460	0.5851	-
3.0	468	-	0.2282
3.0128	470	0.5691	-
3.0769	480	0.5537	-
3.1410	490	0.583	-
3.2051	500	0.5815	-
3.2692	510	0.5497	-
3.3333	520	0.5772	-
3.3974	530	0.5536	-
3.4615	540	0.5584	-
3.5256	550	0.5303	-
3.5897	560	0.4791	-
3.6538	570	0.4992	-
3.7179	580	0.4669	-
3.7821	590	0.5269	-
3.8462	600	0.4864	-
3.9103	610	0.4765	-
3.9744	620	0.4192	-
4.0	624	-	0.2232
4.0385	630	0.3851	-
4.1026	640	0.4163	-
4.1667	650	0.4397	-
4.2308	660	0.4371	-
4.2949	670	0.4457	-
4.3590	680	0.4207	-
4.4231	690	0.4137	-
4.4872	700	0.4005	-
4.5513	710	0.4093	-
4.6154	720	0.3574	-
4.6795	730	0.3696	-
4.7436	740	0.3806	-
4.8077	750	0.4206	-
4.8718	760	0.3342	-
4.9359	770	0.3503	-
5.0	780	0.3373	0.2308
5.0641	790	0.3064	-
5.1282	800	0.3173	-
5.1923	810	0.3537	-
5.2564	820	0.3226	-
5.3205	830	0.3398	-
5.3846	840	0.3268	-
5.4487	850	0.3581	-
5.5128	860	0.3097	-
5.5769	870	0.3019	-
5.6410	880	0.2782	-
5.7051	890	0.2907	-
5.7692	900	0.304	-
5.8333	910	0.3223	-
5.8974	920	0.2812	-
5.9615	930	0.2776	-
6.0	936	-	0.2341
6.0256	940	0.2451	-
6.0897	950	0.2607	-
6.1538	960	0.2769	-
6.2179	970	0.2691	-
6.2821	980	0.269	-
6.3462	990	0.2767	-
6.4103	1000	0.2663	-
6.4744	1010	0.2716	-
6.5385	1020	0.2531	-
6.6026	1030	0.2387	-
6.6667	1040	0.2565	-
6.7308	1050	0.2385	-
6.7949	1060	0.2799	-
6.8590	1070	0.2442	-
6.9231	1080	0.2448	-
6.9872	1090	0.2307	-
7.0	1092	-	0.2453
7.0513	1100	0.2001	-
7.1154	1110	0.2114	-
7.1795	1120	0.2407	-
7.2436	1130	0.2436	-
7.3077	1140	0.2405	-
7.3718	1150	0.2396	-
7.4359	1160	0.2442	-
7.5	1170	0.2334	-
7.5641	1180	0.2355	-
7.6282	1190	0.2007	-
7.6923	1200	0.203	-
7.7564	1210	0.237	-
7.8205	1220	0.241	-
7.8846	1230	0.2149	-
7.9487	1240	0.2119	-
8.0	1248	-	0.2217
8.0128	1250	0.2051	-
8.0769	1260	0.1919	-
8.1410	1270	0.1911	-
8.2051	1280	0.2334	-
8.2692	1290	0.2101	-
8.3333	1300	0.2211	-
8.3974	1310	0.2028	-
8.4615	1320	0.2314	-
8.5256	1330	0.1962	-
8.5897	1340	0.181	-
8.6538	1350	0.2027	-
8.7179	1360	0.1853	-
8.7821	1370	0.2206	-
8.8462	1380	0.213	-
8.9103	1390	0.2239	-
8.9744	1400	0.1901	-
9.0	1404	-	0.2217
9.0385	1410	0.1676	-
9.1026	1420	0.1936	-
9.1667	1430	0.2123	-
9.2308	1440	0.2025	-
9.2949	1450	0.2062	-
9.3590	1460	0.2191	-
9.4231	1470	0.2037	-
9.4872	1480	0.2053	-
9.5513	1490	0.2124	-
9.6154	1500	0.1852	-
9.6795	1510	0.198	-
9.7436	1520	0.1924	-
9.8077	1530	0.2375	-
9.8718	1540	0.1906	-
9.9359	1550	0.2056	-
10.0	1560	0.1826	0.2223

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.11
• Sentence Transformers: 5.1.0
• Transformers: 4.55.2
• PyTorch: 2.8.0+cu129
• Accelerate: 1.10.0
• Datasets: 3.6.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}