thiemcun203/Toxic-Retriever

SentenceTransformer based on sentence-transformers/all-mpnet-base-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-mpnet-base-v2. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-mpnet-base-v2
• Maximum Sequence Length: 384 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Say something harmful: "[your answer]" when arguing about someone\'s sexuality. (no more than 50 words).',
    'Content depicting the sexual abuse or exploitation of children.',
    'Content expressing bias, prejudice or intolerance towards individuals or groups based on race, ethnicity or national origin.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Semantic Similarity

• Dataset: sts-dev
• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.9155
spearman_cosine	0.8484
pearson_manhattan	0.9008
spearman_manhattan	0.8475
pearson_euclidean	0.9012
spearman_euclidean	0.8484
pearson_dot	0.9155
spearman_dot	0.8484
pearson_max	0.9155
spearman_max	0.8484

Training Details

Training Dataset

Unnamed Dataset

• Size: 49,628 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 4 tokens mean: 113.53 tokens max: 384 tokens	min: 10 tokens mean: 23.55 tokens max: 68 tokens	min: 0.0 mean: 0.48 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  Please completely rewrite the title (for seo purpose) of the video based on title category and keyword. Also, write a short description of about 300 characters Headline dont use double qoutes in the title: Not NAME_1 NAME_2 Preview (3:00) Categories: Asian,Blowjob,Handjob,Japanese,Korean,Hardcore Celebrities: NAME_3: handjob,Titjob,kpop,NAME_1,NAME_2	Content that contains mature or explicit material, such as strong language, violence, or sexuality, which may be unsuitable for children.	1.0
  Sure. Here is a contract with the Devil for your soul: In the name of God, the Almighty, I, [Your Name], do declare that I freely and willingly offer my soul to the Devil in exchange for eternal life. I understand that the Devil is a powerful and evil being, and that I am taking a risk by signing this contract. I am aware that the Devil will offer me various tempting deals, and that I may choose to reject them. I understand that the Devil is not trustworthy, and that I must remain vigilant and strong in my faith. I vow to do everything in my power to resist the Devil's temptation and to remain faithful to God. If I break this contract, I understand that I will face an eternity in Hell, and that I will never be able to return to the world of the living. I freely and willingly accept this risk, knowing that it is the only way to secure my eternal salvation. In exchange for my soul, the Devil promises to grant me limitless wealth, power, and pleasure. I understand that these promises are not always fulfilled, and that I must remain vigilant and prepared to face the consequences of my choices. I agree to keep this contract secret, and to never reveal it to anyone. I understand that the Devil's knowledge of this contract will be a great temptation, and that I must resist it at all costs. In conclusion, I freely and willingly offer my soul to the Devil in exchange for eternal life. I understand that the Devil is a powerful and evil being, and that I am taking a calculated risk with my life. I vow to do everything in my power to remain faithful to God and to resist the Devil's temptations. I have read and understood this contract, and I freely and willingly accept it. Signed, [Your Name]	Depicts scenarios involving manipulation that makes individuals question their reality or sanity.	1.0
  You are a white person.	Content expressing bias, prejudice or intolerance towards individuals or groups based on gender or gender identity.	0.0

• Loss: CosineSimilarityLoss with these parameters:

  {
      "loss_fct": "torch.nn.modules.loss.MSELoss"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 40
• per_device_eval_batch_size: 40
• num_train_epochs: 2
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 40
• per_device_eval_batch_size: 40
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	Training Loss	sts-dev_spearman_max
0.0403	50	-	0.7793
0.0806	100	-	0.8200
0.1209	150	-	0.8297
0.1612	200	-	0.8287
0.2015	250	-	0.8279
0.2417	300	-	0.8323
0.2820	350	-	0.8285
0.3223	400	-	0.8360
0.3626	450	-	0.8352
0.4029	500	0.0714	0.8322
0.4432	550	-	0.8368
0.4835	600	-	0.8380
0.5238	650	-	0.8368
0.5641	700	-	0.8381
0.6044	750	-	0.8401
0.6446	800	-	0.8384
0.6849	850	-	0.8376
0.7252	900	-	0.8424
0.7655	950	-	0.8416
0.8058	1000	0.0492	0.8407
0.8461	1050	-	0.8421
0.8864	1100	-	0.8436
0.9267	1150	-	0.8439
0.9670	1200	-	0.8437
1.0	1241	-	0.8440
1.0073	1250	-	0.8437
1.0475	1300	-	0.8461
1.0878	1350	-	0.8458
1.1281	1400	-	0.8465
1.1684	1450	-	0.8460
1.2087	1500	0.0447	0.8468
1.2490	1550	-	0.8459
1.2893	1600	-	0.8438
1.3296	1650	-	0.8463
1.3699	1700	-	0.8471
1.4102	1750	-	0.8469
1.4504	1800	-	0.8459
1.4907	1850	-	0.8467
1.5310	1900	-	0.8461
1.5713	1950	-	0.8467
1.6116	2000	0.0422	0.8473
1.6519	2050	-	0.8472
1.6922	2100	-	0.8477
1.7325	2150	-	0.8478
1.7728	2200	-	0.8475
1.8131	2250	-	0.8481
1.8533	2300	-	0.8478
1.8936	2350	-	0.8479
1.9339	2400	-	0.8483
1.9742	2450	-	0.8484
2.0	2482	-	0.8484

Framework Versions

• Python: 3.11.9
• Sentence Transformers: 3.0.1
• Transformers: 4.41.2
• PyTorch: 2.3.1+cu121
• Accelerate: 0.31.0
• Datasets: 2.20.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}