SentenceTransformer based on BAAI/bge-base-zh-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-base-zh-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: BAAI/bge-base-zh-v1.5
Maximum Sequence Length: 256 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Supported Modality: Text

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("liudev/novel-embedding-gpl-base-v1")
# Run inference
sentences = [
    '在异能觉醒爆发时安抚对方并带走宝物',
    '宴席上有人当众示爱引发全场震惊',
    '凶暴巨兽冲破防线引发骚乱',
    '山贼讲述当年的恐惧往事，结果再次遭遇强敌被瞬杀',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [4, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [4, 4]

Training Details

Training Dataset

Unnamed Dataset

Size: 435,262 training samples
Columns: anchor, sentence1, sentence2, and label

Approximate statistics based on the first 100 samples:

	anchor	sentence1	sentence2	label
type	string	string	string	float
modality	text	text	text
details	min: 8 tokens mean: 15.5 tokens max: 25 tokens	min: 47 tokens mean: 73.19 tokens max: 256 tokens	min: 29 tokens mean: 69.23 tokens max: 92 tokens	min: 0.02 mean: 0.37 max: 0.84

Samples:

anchor	sentence1	sentence2	label
`看破敌人的隐身手段`	`周昂在围攻强敌时敏锐察觉到异样，通过观察灵气流动识破并破除隐身法术，最终果断出手将藏匿在墙头的施法者格杀。`	`李破在马场以赏罚分明的手段收拢人心并大摆酒宴后，在李碧的私下召见中，意外得知其欲随军出征辽东的计划并被征询意见。`	`0.759356061900818`
`在仓库意外得到防身宝物`	`柳渊在医馆仓库意外获得防御法器佛珠及攻击真言，随后通过定星罗盘锁定义务星邪，并凭借缜密的推理引导葛老与辛捕头共同前往后院枯井探查。`	`沈河在烂尾楼工地通过系统指引接连开启宝箱，获得了具备强控效果的“大力飞砖”与防御性的“安全员安全帽”，在确认工地无剩余宝箱后，随即根据新坐标向十公里外进发。`	`0.09672797019024215`
`进宫面圣前，太监好心提醒忌讳规矩`	`宦官郭穰在权衡利弊后决定拉拢入觐的天子待诏秀才张毅，不仅对其礼遇有加，更在入宫途中悉心传授面圣禁忌与生存经验，双方达成了初步的盟友共识。`	`在正月初一太子加冠的重大典礼上，南风费尽心思哄劝赖床的太子完成繁琐仪式并接手庞大的东宫官僚体系，同时担忧自己微服出游时的糗事在稍后的公开亮相中被路人认出。`	`0.5887360227245302`

Loss: MarginMSELoss with these parameters:

{
    "similarity_fct": "pairwise_dot_score"
}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 32
num_train_epochs: 150
learning_rate: 2e-05
warmup_steps: 0.02
fp16: True
push_to_hub: True
hub_model_id: liudev/novel-embedding-gpl-base-v1
hub_strategy: checkpoint
dataloader_num_workers: 4

All Hyperparameters

Click to expand

per_device_train_batch_size: 32
num_train_epochs: 150
max_steps: -1
learning_rate: 2e-05
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_steps: 0.02
optim: adamw_torch_fused
optim_args: None
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
optim_target_modules: None
gradient_accumulation_steps: 1
average_tokens_across_devices: True
max_grad_norm: 1.0
label_smoothing_factor: 0.0
bf16: False
fp16: True
bf16_full_eval: False
fp16_full_eval: False
tf32: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
use_liger_kernel: False
liger_kernel_config: None
use_cache: False
neftune_noise_alpha: None
torch_empty_cache_steps: None
auto_find_batch_size: False
log_on_each_node: True
logging_nan_inf_filter: True
include_num_input_tokens_seen: no
log_level: passive
log_level_replica: warning
disable_tqdm: False
project: huggingface
trackio_space_id: None
trackio_bucket_id: None
trackio_static_space_id: None
per_device_eval_batch_size: 8
prediction_loss_only: True
eval_on_start: False
eval_do_concat_batches: True
eval_use_gather_object: False
eval_accumulation_steps: None
include_for_metrics: []
batch_eval_metrics: False
save_only_model: False
save_on_each_node: False
enable_jit_checkpoint: False
push_to_hub: True
hub_private_repo: None
hub_model_id: liudev/novel-embedding-gpl-base-v1
hub_strategy: checkpoint
hub_always_push: False
hub_revision: None
load_best_model_at_end: False
ignore_data_skip: False
restore_callback_states_from_checkpoint: False
full_determinism: False
seed: 42
data_seed: None
use_cpu: False
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
dataloader_drop_last: True
dataloader_num_workers: 4
dataloader_pin_memory: True
dataloader_persistent_workers: False
dataloader_prefetch_factor: None
dataloader_multiprocessing_context: None
dataloader_in_order: True
remove_unused_columns: True
label_names: None
train_sampling_strategy: random
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
ddp_static_graph: None
ddp_backend: None
ddp_timeout: 1800
fsdp: None
fsdp_config: None
deepspeed: None
debug: []
skip_memory_metrics: True
do_predict: False
resume_from_checkpoint: None
local_rank: -1
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}
warmup_ratio: None

Training Logs

Click to expand

Epoch	Step	Training Loss
82.3603	560050	0.0094
82.3676	560100	0.0098
82.375	560150	0.0097
82.3824	560200	0.0095
82.3897	560250	0.0094
82.3971	560300	0.0098
82.4044	560350	0.0095
82.4118	560400	0.0094
82.4191	560450	0.0091
82.4265	560500	0.0091
82.4338	560550	0.0094
82.4412	560600	0.0098
82.4485	560650	0.0091
82.4559	560700	0.0093
82.4632	560750	0.0092
82.4706	560800	0.0090
82.4779	560850	0.0094
82.4853	560900	0.0093
82.4926	560950	0.0097
82.5	561000	0.0092
82.5074	561050	0.0098
82.5147	561100	0.0094
82.5221	561150	0.0094
82.5294	561200	0.0095
82.5368	561250	0.0094
82.5441	561300	0.0103
82.5515	561350	0.0097
82.5588	561400	0.0094
82.5662	561450	0.0094
82.5735	561500	0.0096
82.5809	561550	0.0093
82.5882	561600	0.0097
82.5956	561650	0.0096
82.6029	561700	0.0095
82.6103	561750	0.0096
82.6176	561800	0.0093
82.625	561850	0.0095
82.6324	561900	0.0097
82.6397	561950	0.0091
82.6471	562000	0.0095
82.6544	562050	0.0094
82.6618	562100	0.0099
82.6691	562150	0.0092
82.6765	562200	0.0093
82.6838	562250	0.0100
82.6912	562300	0.0094
82.6985	562350	0.0101
82.7059	562400	0.0097
82.7132	562450	0.0092
82.7206	562500	0.0092
82.7279	562550	0.0097
82.7353	562600	0.0094
82.7426	562650	0.0095
82.75	562700	0.0098
82.7574	562750	0.0097
82.7647	562800	0.0092
82.7721	562850	0.0099
82.7794	562900	0.0095
82.7868	562950	0.0091
82.7941	563000	0.0095
82.8015	563050	0.0097
82.8088	563100	0.0098
82.8162	563150	0.0097
82.8235	563200	0.0091
82.8309	563250	0.0098
82.8382	563300	0.0099
82.8456	563350	0.0096
82.8529	563400	0.0095
82.8603	563450	0.0096
82.8676	563500	0.0099
82.875	563550	0.0096
82.8824	563600	0.0099
82.8897	563650	0.0093
82.8971	563700	0.0099
82.9044	563750	0.0096
82.9118	563800	0.0092
82.9191	563850	0.0095
82.9265	563900	0.0095
82.9338	563950	0.0096
82.9412	564000	0.0097
82.9485	564050	0.0094
82.9559	564100	0.0090
82.9632	564150	0.0091
82.9706	564200	0.0101
82.9779	564250	0.0096
82.9853	564300	0.0097
82.9926	564350	0.0096
83.0	564400	0.0095
83.0074	564450	0.0093
83.0147	564500	0.0095
83.0221	564550	0.0097
83.0294	564600	0.0096
83.0368	564650	0.0092
83.0441	564700	0.0099
83.0515	564750	0.0091
83.0588	564800	0.0096
83.0662	564850	0.0091
83.0735	564900	0.0094
83.0809	564950	0.0095
83.0882	565000	0.0093
83.0956	565050	0.0096
83.1029	565100	0.0095
83.1103	565150	0.0093
83.1176	565200	0.0095
83.125	565250	0.0093
83.1324	565300	0.0093
83.1397	565350	0.0090
83.1471	565400	0.0092
83.1544	565450	0.0091
83.1618	565500	0.0100
83.1691	565550	0.0092
83.1765	565600	0.0096
83.1838	565650	0.0095
83.1912	565700	0.0092
83.1985	565750	0.0094
83.2059	565800	0.0091
83.2132	565850	0.0098
83.2206	565900	0.0094
83.2279	565950	0.0092
83.2353	566000	0.0091
83.2426	566050	0.0090
83.25	566100	0.0094
83.2574	566150	0.0095
83.2647	566200	0.0094
83.2721	566250	0.0099
83.2794	566300	0.0091
83.2868	566350	0.0093
83.2941	566400	0.0094
83.3015	566450	0.0093
83.3088	566500	0.0092
83.3162	566550	0.0096
83.3235	566600	0.0092
83.3309	566650	0.0098
83.3382	566700	0.0092
83.3456	566750	0.0093
83.3529	566800	0.0096
83.3603	566850	0.0096
83.3676	566900	0.0093
83.375	566950	0.0098
83.3824	567000	0.0095
83.3897	567050	0.0097
83.3971	567100	0.0097
83.4044	567150	0.0100
83.4118	567200	0.0097
83.4191	567250	0.0091
83.4265	567300	0.0093
83.4338	567350	0.0089
83.4412	567400	0.0097
83.4485	567450	0.0093
83.4559	567500	0.0097
83.4632	567550	0.0092
83.4706	567600	0.0095
83.4779	567650	0.0095
83.4853	567700	0.0094
83.4926	567750	0.0093
83.5	567800	0.0099
83.5074	567850	0.0094
83.5147	567900	0.0100
83.5221	567950	0.0095
83.5294	568000	0.0096
83.5368	568050	0.0097
83.5441	568100	0.0097
83.5515	568150	0.0099
83.5588	568200	0.0092
83.5662	568250	0.0092
83.5735	568300	0.0096
83.5809	568350	0.0095
83.5882	568400	0.0097
83.5956	568450	0.0092
83.6029	568500	0.0092
83.6103	568550	0.0097
83.6176	568600	0.0096
83.625	568650	0.0090
83.6324	568700	0.0094
83.6397	568750	0.0095
83.6471	568800	0.0097
83.6544	568850	0.0094
83.6618	568900	0.0092
83.6691	568950	0.0092
83.6765	569000	0.0091
83.6838	569050	0.0096
83.6912	569100	0.0089
83.6985	569150	0.0098
83.7059	569200	0.0092
83.7132	569250	0.0094
83.7206	569300	0.0095
83.7279	569350	0.0092
83.7353	569400	0.0091
83.7426	569450	0.0090
83.75	569500	0.0096
83.7574	569550	0.0098
83.7647	569600	0.0087
83.7721	569650	0.0089
83.7794	569700	0.0093
83.7868	569750	0.0095
83.7941	569800	0.0089
83.8015	569850	0.0092
83.8088	569900	0.0095
83.8162	569950	0.0092
83.8235	570000	0.0098
83.8309	570050	0.0098
83.8382	570100	0.0093
83.8456	570150	0.0091
83.8529	570200	0.0093
83.8603	570250	0.0092
83.8676	570300	0.0094
83.875	570350	0.0097
83.8824	570400	0.0094
83.8897	570450	0.0096
83.8971	570500	0.0095
83.9044	570550	0.0097
83.9118	570600	0.0091
83.9191	570650	0.0092
83.9265	570700	0.0094
83.9338	570750	0.0099
83.9412	570800	0.0096
83.9485	570850	0.0095
83.9559	570900	0.0095
83.9632	570950	0.0091
83.9706	571000	0.0097
83.9779	571050	0.0095
83.9853	571100	0.0096
83.9926	571150	0.0095
84.0	571200	0.0089
84.0074	571250	0.0093
84.0147	571300	0.0090
84.0221	571350	0.0089
84.0294	571400	0.0097
84.0368	571450	0.0090
84.0441	571500	0.0087
84.0515	571550	0.0095
84.0588	571600	0.0094
84.0662	571650	0.0092
84.0735	571700	0.0091
84.0809	571750	0.0089
84.0882	571800	0.0091
84.0956	571850	0.0090
84.1029	571900	0.0096
84.1103	571950	0.0093
84.1176	572000	0.0096
84.125	572050	0.0098
84.1324	572100	0.0093
84.1397	572150	0.0094
84.1471	572200	0.0088
84.1544	572250	0.0093
84.1618	572300	0.0095
84.1691	572350	0.0094
84.1765	572400	0.0096
84.1838	572450	0.0096
84.1912	572500	0.0092
84.1985	572550	0.0093
84.2059	572600	0.0098
84.2132	572650	0.0092
84.2206	572700	0.0090
84.2279	572750	0.0099
84.2353	572800	0.0097
84.2426	572850	0.0093
84.25	572900	0.0092
84.2574	572950	0.0090
84.2647	573000	0.0091
84.2721	573050	0.0095
84.2794	573100	0.0094
84.2868	573150	0.0092
84.2941	573200	0.0096
84.3015	573250	0.0093
84.3088	573300	0.0094
84.3162	573350	0.0093
84.3235	573400	0.0095
84.3309	573450	0.0093
84.3382	573500	0.0097
84.3456	573550	0.0092
84.3529	573600	0.0091
84.3603	573650	0.0093
84.3676	573700	0.0099
84.375	573750	0.0097
84.3824	573800	0.0091
84.3897	573850	0.0095
84.3971	573900	0.0095
84.4044	573950	0.0088
84.4118	574000	0.0094
84.4191	574050	0.0097
84.4265	574100	0.0092
84.4338	574150	0.0096
84.4412	574200	0.0098
84.4485	574250	0.0091
84.4559	574300	0.0090
84.4632	574350	0.0099
84.4706	574400	0.0094
84.4779	574450	0.0093
84.4853	574500	0.0094
84.4926	574550	0.0094
84.5	574600	0.0096
84.5074	574650	0.0092
84.5147	574700	0.0092
84.5221	574750	0.0088
84.5294	574800	0.0092
84.5368	574850	0.0094
84.5441	574900	0.0092
84.5515	574950	0.0095
84.5588	575000	0.0091
84.5662	575050	0.0091
84.5735	575100	0.0089
84.5809	575150	0.0096
84.5882	575200	0.0095
84.5956	575250	0.0096
84.6029	575300	0.0089
84.6103	575350	0.0093
84.6176	575400	0.0090
84.625	575450	0.0096
84.6324	575500	0.0098
84.6397	575550	0.0093
84.6471	575600	0.0095
84.6544	575650	0.0094
84.6618	575700	0.0092
84.6691	575750	0.0090
84.6765	575800	0.0091
84.6838	575850	0.0093
84.6912	575900	0.0093
84.6985	575950	0.0096
84.7059	576000	0.0093
84.7132	576050	0.0098
84.7206	576100	0.0092
84.7279	576150	0.0091
84.7353	576200	0.0097
84.7426	576250	0.0094
84.75	576300	0.0094
84.7574	576350	0.0094
84.7647	576400	0.0096
84.7721	576450	0.0094
84.7794	576500	0.0096
84.7868	576550	0.0095
84.7941	576600	0.0094
84.8015	576650	0.0096
84.8088	576700	0.0093
84.8162	576750	0.0091
84.8235	576800	0.0093
84.8309	576850	0.0094
84.8382	576900	0.0094
84.8456	576950	0.0092
84.8529	577000	0.0096
84.8603	577050	0.0093
84.8676	577100	0.0096
84.875	577150	0.0096
84.8824	577200	0.0096
84.8897	577250	0.0094
84.8971	577300	0.0093
84.9044	577350	0.0096
84.9118	577400	0.0100
84.9191	577450	0.0091
84.9265	577500	0.0091
84.9338	577550	0.0096
84.9412	577600	0.0098
84.9485	577650	0.0093
84.9559	577700	0.0094
84.9632	577750	0.0090
84.9706	577800	0.0095
84.9779	577850	0.0091
84.9853	577900	0.0093
84.9926	577950	0.0094
85.0	578000	0.0092
85.0074	578050	0.0096
85.0147	578100	0.0089
85.0221	578150	0.0094
85.0294	578200	0.0098
85.0368	578250	0.0095
85.0441	578300	0.0087
85.0515	578350	0.0093
85.0588	578400	0.0092
85.0662	578450	0.0098
85.0735	578500	0.0093
85.0809	578550	0.0092
85.0882	578600	0.0093
85.0956	578650	0.0091
85.1029	578700	0.0092
85.1103	578750	0.0096
85.1176	578800	0.0093
85.125	578850	0.0097
85.1324	578900	0.0095
85.1397	578950	0.0091
85.1471	579000	0.0093
85.1544	579050	0.0093
85.1618	579100	0.0094
85.1691	579150	0.0096
85.1765	579200	0.0100
85.1838	579250	0.0091
85.1912	579300	0.0099
85.1985	579350	0.0094
85.2059	579400	0.0091
85.2132	579450	0.0093
85.2206	579500	0.0093
85.2279	579550	0.0094
85.2353	579600	0.0094
85.2426	579650	0.0091
85.25	579700	0.0092
85.2574	579750	0.0091
85.2647	579800	0.0095
85.2721	579850	0.0097
85.2794	579900	0.0093
85.2868	579950	0.0094
85.2941	580000	0.0094

Training Time

Training: 2.8 hours

Framework Versions

Python: 3.13.15
Sentence Transformers: 5.7.0
Transformers: 5.16.1
PyTorch: 2.11.0+cu128
Accelerate: 1.14.0
Datasets: 4.8.5
Tokenizers: 0.23.1

Additional Resources

Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MarginMSELoss

@misc{hofstätter2021improving,
    title={Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation},
    author={Sebastian Hofstätter and Sophia Althammer and Michael Schröder and Mete Sertkan and Allan Hanbury},
    year={2021},
    eprint={2010.02666},
    archivePrefix={arXiv},
    primaryClass={cs.IR}
}