dopaminemaximizer/rag_ubiza

SentenceTransformer based on Qwen/Qwen3-Embedding-0.6B

This is a sentence-transformers model finetuned from Qwen/Qwen3-Embedding-0.6B. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Qwen/Qwen3-Embedding-0.6B
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'Qwen3Model'})
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    "Instruct: Given a question about a video transcript, retrieve the most relevant transcript passage.\nQuery: How do you maintain an open throat and a lifted soft palate in opera as you ascend in pitch?",
]
documents = [
    "Eeeeee! Too narrow in the throat. Too open, I can't make an E. Eeeeee! And then again. ZE FERE LTE LOSINGIARI DE VOLLO ZE FERE FE-E-E-I In the middle. ZE FERE LTE LOSINGIARI And then later it has E GYDITE. So... Eeeeee! Eeeeee! Eeeeee! Eeeeee! Eeeeee! Eeeeee! Eeeeee! Now I'm at an F sharp. Now comes the caveat. As you are actually getting higher, those closed vowels, now they do slightly open in the mouth.",
    "As artists, we use lines a lot. We use them as placeholders in the early stages of our drawings to measure, indicate perspective, contour, and gesture. And we use them in the final stages of our drawings when we're shading tone or as a powerful visual element that can guide the eye. So bad line quality can quickly ruin a drawing and give off a really bad impression of your skill. You want to avoid building habits of poor line quality. The longer you practice bad lines, the stronger those habits will become and the harder it will be to break out of those bad habits. So I want to get you on the path to develop the good habits. And if you liked this video, make sure to check out the full drawing basics course at proko.com slash drawing. It's designed to give you the foundation you need to draw from observation or from imagination. Number one, avoid the common mistakes. Two very common mistakes are short scratchy lines and chaotic searching lines.",
    "This whole area asserts Ottoman wealth and by the way guys I wore this blue shirt for the Blue Mosque today. Thinking about it now we probably would recommend you do a tour for all of this. To dive deeper into the history there's so much to learn. Day four. Since we did so much the day before we decided to do some mild sightseeing and lunch in the quaint area of Balant. It's wonderfully colourful and reminds us a lot of Bo Carp in Cape Town. Oh my word, it's so cute already. For that evening we booked a Bosphorus boat cruise through Get Your Guide once again. You can't come to Istanbul without cruising down the Bosphorus Strait. Tonight we're on a yacht to eat delicious Turkish food and watch live Turkish performances. I'm so excited and by the way guys look at the view behind me. Oh this is gonna be great. Honestly though it wasn't quite our cup of tea. It was just a little bit too touristy for us. But you might like it. The views were undoubtedly spectacular though. And that was basically the end of our time in Istanbul. There's so much to do in tea in this city. Like you could just spend the whole day counting cats if you wanted to. There's almost half a million strays here. But here's a list of other things that look great too.",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 1024] [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.4922, 0.2266, 0.1797]], dtype=torch.bfloat16)

Evaluation

Metrics

Information Retrieval

• Dataset: val_ir
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.5238
cosine_accuracy@3	0.746
cosine_accuracy@5	0.836
cosine_accuracy@10	0.9683
cosine_precision@1	0.5238
cosine_precision@3	0.3404
cosine_precision@5	0.2772
cosine_precision@10	0.1857
cosine_recall@1	0.2904
cosine_recall@3	0.4897
cosine_recall@5	0.6495
cosine_recall@10	0.8344
cosine_ndcg@10	0.6498
cosine_mrr@10	0.6582
cosine_map@100	0.5579

Training Details

Training Dataset

Unnamed Dataset

• Size: 3,548 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 29 tokens mean: 39.33 tokens max: 84 tokens	min: 20 tokens mean: 223.09 tokens max: 380 tokens

• Samples:

  anchor	positive
  Instruct: Given a question about a video transcript, retrieve the most relevant transcript passage. Query: How to use Hawaii’s local transportation?	rap for sometimes being a bit too crowded, but sunset, the vibe is great. There's often live music. Everybody's just enjoying watching that sun go down. So it's a great place to start your vacation. You can even hop in one of the catamarans that has beach loading and go out sailing and watch the sunset that way. That's a great way to start. There you go. Gosh, we're good. Oh my gosh. You're lucky you found this channel. Day number two. And guess what? You are up early, my friend, because you have jet lag. Real bad. Real bad. Especially if you have kids with you. Wow. Good luck. So we are going to take advantage of that today and you are going to head up to the North Shore to go snorkeling at Kualima Cove or just enjoy a nice beach day up there. Kualima Cove is at Turtle Bay Resort, which is like five-star luxury resort that's been remodeled, but all beaches in Hawaii are public. And this is a great place to go snorkeling. It's well protected. And yeah, there's public parking. So enjoy ...
  Instruct: Given a question about a video transcript, retrieve the most relevant transcript passage. Query: How to use Hawaii’s local transportation?	So are you going to Oahu, Hawaii and you're either too cheap or broke to get a rental car for your time there? There's no judgment in that question because that was my mentality and you definitely can have an amazing time in Oahu, Hawaii even for like a week or so without any rental car whatsoever. I was there for a month and mostly got by without a rental car so I'm definitely very qualified to give you an amazing seven-day one-week itinerary for Oahu, Hawaii with no rental car. That being said let's start off with day number zero or arrival day. So unless you're a very strong swimmer or taking a very fast speedboat you'll be arriving at the Daniel K. Inouye International Airport. Probably a little bit jet lagged. It's definitely a little bit tired so let's just get to your hotel and check in. I do recommend staying in Waikiki which people might kind of roll their eyes or scoff at because it's very touristic. It's like a little bit of a concrete jungle in the middle of Hawaii built on...
  Instruct: Given a question about a video transcript, retrieve the most relevant transcript passage. Query: How to use Hawaii’s local transportation?	do you want to walk there and burn out your legs a little bit? It's up to you, depends on how much energy you have to spare. Either way the hike isn't so difficult. It costs about five dollars to enter, walking that is, which you'll be doing because you don't have a car of course, and I believe nowadays you have to either reserve a time slot or something like that. It's also very busy so the earlier you go the better. The hike itself is really cool, it's not too difficult like I said, you go through a little bit of a tunnel and at the end you will get some amazing views of Waikiki, Waikiki Beach, and some other parts of the island which you're probably super excited to explore which you'll definitely do on other days besides today. So once you finish hiking Diamond Head, return back down the way you came, it's an out and back trail, and head to the nearby neighborhood called Kaimuki. Now Kaimuki is a really cool and trendy sort of hip and sort of like a college town feel neighborhood w...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• num_train_epochs: 10
• learning_rate: 0.0002
• warmup_steps: 0.1
• gradient_accumulation_steps: 4
• eval_strategy: steps
• per_device_eval_batch_size: 32
• load_best_model_at_end: True
• remove_unused_columns: False

All Hyperparameters

Click to expand

• per_device_train_batch_size: 8
• num_train_epochs: 10
• max_steps: -1
• learning_rate: 0.0002
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 4
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: trackio
• eval_strategy: steps
• per_device_eval_batch_size: 32
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: False
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	val_ir_cosine_ndcg@10
0.3604	20	0.9475	-
0.7207	40	0.8095	-
1.0721	60	0.5994	-
1.4324	80	0.4689	-
1.7928	100	0.5015	0.6131
2.1441	120	0.4094	-
2.5045	140	0.3802	-
2.8649	160	0.3823	-
3.2162	180	0.3307	-
3.5766	200	0.3340	0.6115
3.9369	220	0.3041	-
4.2883	240	0.2975	-
4.6486	260	0.3046	-
5.0	280	0.2837	-
5.3604	300	0.2595	0.6423
5.7207	320	0.2558	-
6.0721	340	0.2673	-
6.4324	360	0.2468	-
6.7928	380	0.2649	-
7.1441	400	0.2285	0.6564
7.5045	420	0.2203	-
7.8649	440	0.2180	-
8.2162	460	0.1992	-
8.5766	480	0.1888	-
8.9369	500	0.1936	0.6498
9.2883	520	0.2144	-
9.6486	540	0.2128	-
10.0	560	0.2116	-

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.3.0
• Transformers: 5.2.0
• PyTorch: 2.9.0+cu126
• Accelerate: 1.12.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}