models/ksaa_v2/README.md

metadata

tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:90678
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
base_model: Omartificial-Intelligence-Space/Arabic-Triplet-Matryoshka-V2
widget:
  - source_sentence: 'ماهي الكلمه التي تعني: حزن الشخص وجرت دمعته.'
    sentences:
      - استذكاء
      - مستعبر
      - تشخيصه
  - source_sentence: 'ماهي الكلمه التي تعني: المره من تناول طعام يسير؛ لتهدئه الجوع مؤقتا.'
    sentences:
      - صاحن
      - ادعج
      - تسكيته
  - source_sentence: 'ماهي الكلمه التي تعني: اعتياد التقشف وشظف العيش.'
    sentences:
      - اخشيشان
      - هزيم
      - استذهال
  - source_sentence: 'ماهي الكلمه التي تعني: تعب مرهق منهك القوى.'
    sentences:
      - تلفان
      - نقزه
      - عامل
  - source_sentence: 'ماهي الكلمه التي تعني: بال قديم، عديم القيمه.'
    sentences:
      - ادرن
      - خثير
      - هريء
pipeline_tag: sentence-similarity
library_name: sentence-transformers

SentenceTransformer based on Omartificial-Intelligence-Space/Arabic-Triplet-Matryoshka-V2

This is a sentence-transformers model finetuned from Omartificial-Intelligence-Space/Arabic-Triplet-Matryoshka-V2. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Omartificial-Intelligence-Space/Arabic-Triplet-Matryoshka-V2
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'ماهي الكلمه التي تعني: بال قديم، عديم القيمه.',
    'هريء',
    'ادرن',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.4617, 0.1454],
#         [0.4617, 1.0000, 0.0522],
#         [0.1454, 0.0522, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 90,678 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 10 tokens mean: 14.58 tokens max: 27 tokens	min: 3 tokens mean: 3.75 tokens max: 5 tokens

• Samples:

  anchor	positive
  ماهي الكلمه التي تعني: وفقا للشيء.	تبعا لـ
  ماهي الكلمه التي تعني: مركب لنقل الناس او البضائع في البحر او النهر او الفضاء الخارجي .	سفين
  ماهي الكلمه التي تعني: المهزوم.	هزيم

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768
      ],
      "matryoshka_weights": [
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 128
• num_train_epochs: 5
• warmup_steps: 0.1
• gradient_accumulation_steps: 2
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 128
• num_train_epochs: 5
• max_steps: -1
• learning_rate: 5e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 2
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
0.2821	100	1.8383
0.5642	200	1.3010
0.8463	300	1.1525
1.1269	400	0.9740
1.4090	500	0.8594
1.6911	600	0.8258
1.9732	700	0.8039
2.2539	800	0.6164
2.5360	900	0.6076
2.8181	1000	0.6035
3.0987	1100	0.5412
3.3808	1200	0.4620
3.6629	1300	0.4595
3.9450	1400	0.4667
4.2257	1500	0.4030
4.5078	1600	0.3940
4.7898	1700	0.3759

Training Time

• Training: 10.7 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.5.1
• Transformers: 5.9.0
• PyTorch: 2.11.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}