swardiantara/bert-tiny-yelp-k3-fixed-euclidean

SentenceTransformer based on google/bert_uncased_L-2_H-128_A-2

This is a sentence-transformers model finetuned from google/bert_uncased_L-2_H-128_A-2. It maps sentences & paragraphs to a 128-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/bert_uncased_L-2_H-128_A-2
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 128, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("swardiantara/bert-tiny-yelp-k3-fixed-euclidean")
# Run inference
sentences = [
    "This place is absolute garbage...  Half of the tees are not available, including all the grass tees.  It is cash only, and they sell the last bucket at 8, despite having lights.  And if you finish even a minute after 8, don't plan on getting a drink.  The vending machines are sold out (of course) and they sell drinks inside, but close the drawers at 8 on the dot.  There are weeds grown all over the place.  I noticed some sort of batting cage, but it looks like those are out of order as well.  Someone should buy this place and turn it into what it should be.",
    '\\"Absolute Perfect Dining!\\"\\n\\nBoth my husband and I ate at this restaurant on the fourth of July and were amazed at the views and all the firework displays all over Las Vegas. What a spectacular view this restaurant provided. The food, ambiance and service were superb. If you want to try something fun for dessert, try the three scoops of sorbet (raspberry, mango and orange) with mint and crystalized sugar. What a treat! Make sure you share this dessert as it is very large. In fact, everything they serve is large so make sure you have an appetite. We enjoyed ourselves and will definitely return. \\n\\n\\nVisited July 2013',
    "This is an older retail store.  I normally have a loving relationship with Eat 'n Park, but this one just didn't hit home for me.  I came during the Sunday Brunch Buffet; my friend partook, I abstained after seeing the fare.  I ordered two eggs over-easy with a side of toast.  Getting beyond the murky, dated feel of the store, it was actually fairly clean on the interior and in the restrooms.  \\n\\nI received my food and the eggs were cooked perfectly.  Maybe not a big deal for some, but you would be surprised at how often over-easy comes out over-hard or, worse, uncooked.  I think the real reason I enjoy Eat 'n Park most of the time is the normal salad bar items.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 128]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9956, 0.9968],
#         [0.9956, 1.0000, 0.9930],
#         [0.9968, 0.9930, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 7,605,030 training samples
• Columns: text_a, text_b, and label
• Approximate statistics based on the first 100 samples:

  	text_a	text_b	label
  type	string	string	list
  modality	text	text
  details	min: 62 tokens mean: 104.38 tokens max: 128 tokens	min: 52 tokens mean: 115.28 tokens max: 128 tokens	size: 2 elements

• Samples:

  text_a	text_b	label
  dr. goldberg offers everything i look for in a general practitioner. he's nice and easy to talk to without being patronizing; he's always on time in seeing his patients; he's affiliated with a top-notch hospital (nyu) which my parents have explained to me is very important in case something happens and you need surgery; and you can get referrals to see specialists without having to see him first. really, what more do you need? i'm sitting here trying to think of any complaints i have about him, but i'm really drawing a blank.	I might be in love. This place may have just blown my last crush away! I had driven by this rather nondescript place a million times thinking, "I really need to stop in there and check it out". I also had a fellow, respected cocktail-connoisseur tell me I had to go here for the amazing cocktails and ambiance...that was over a year ago. So I finally mosied in on a Thursday night and was IMPRESSED. Shady's has great tunes coming out of the jukebox, great bartenders that made fantastic beverages, and quite a friendly cool crowd. For all the beer people they also have a good variety of beers. Oh and did I mention it is INEXPENSIVE! I thought they screwed something up when the bill arrived because it was so FREAKING cheap. I went back Friday and Saturday just to confirm: Shady's is indeed AWESOME.	[1.0, 0.0]
  dr. goldberg offers everything i look for in a general practitioner. he's nice and easy to talk to without being patronizing; he's always on time in seeing his patients; he's affiliated with a top-notch hospital (nyu) which my parents have explained to me is very important in case something happens and you need surgery; and you can get referrals to see specialists without having to see him first. really, what more do you need? i'm sitting here trying to think of any complaints i have about him, but i'm really drawing a blank.	No, it's not worth it! Bellagio overcharged me and their accounting department never got back to me (I asked them for an itemized receipt 2 weeks ago). When I called them, I was (twice!) put on this music-less "dead" hold for 30-40 minutes. They're so full of themselves, they don't provide good customer service. Their wi-fi needed fixing when I stayed there. And by the way, the woman at the buffet was super rude to me. Finally, the famous waterworks never happened while I was standing outside in the cold waiting (though i asked the staff and they said they're up and running). No, don't waste your money! I won't and won't let my family and friends do that either.	[0.0, 1.0]
  dr. goldberg offers everything i look for in a general practitioner. he's nice and easy to talk to without being patronizing; he's always on time in seeing his patients; he's affiliated with a top-notch hospital (nyu) which my parents have explained to me is very important in case something happens and you need surgery; and you can get referrals to see specialists without having to see him first. really, what more do you need? i'm sitting here trying to think of any complaints i have about him, but i'm really drawing a blank.	Lidia, you should be ashamed! This spot wouldn't have lasted a year in NYC. Citizens of Pittsburgh, don't be fooled. Lidia's is average at best. I ate here when it opened,a and again 4 years later. Nothing has changed. We did have some great bar nibbles, quality cocktails, and a good bottle of wine off of a pretty good list, but the food and service lacked. For starters we were served ice cold meatballs and limp soggy salads. The took away everything and only brought back fresh salads (mind you, we were one of about 10 tables being served that evening!). The entrees were dull and lacked seasoning. The servers were confused (how many times does it take to ask fro fresh pepper?) and our waiter seemed to hand off our table to another server. It's tough to find anything good here. Just another 'poser'.	[0.0, 1.0]

• Loss: main.OrdinalProxyContrastiveLoss

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 1024
• learning_rate: 1e-05
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• per_device_train_batch_size: 1024
• num_train_epochs: 3
• max_steps: -1
• learning_rate: 1e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0
• optim: adamw_torch
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
0.0673	500	0.2328
0.1346	1000	0.0745
0.2020	1500	0.0679
0.2693	2000	0.0652
0.3366	2500	0.0630
0.4039	3000	0.0613
0.4713	3500	0.0604
0.5386	4000	0.0589
0.6059	4500	0.0588
0.6732	5000	0.0578
0.7405	5500	0.0573
0.8079	6000	0.0570
0.8752	6500	0.0568
0.9425	7000	0.0561
1.0	7427	-
1.0098	7500	0.0564
1.0772	8000	0.0556
1.1445	8500	0.0561
1.2118	9000	0.0557
1.2791	9500	0.0551
1.3464	10000	0.0547
1.4138	10500	0.0549
1.4811	11000	0.0549
1.5484	11500	0.0548
1.6157	12000	0.0546
1.6830	12500	0.0540
1.7504	13000	0.0546
1.8177	13500	0.0541
1.8850	14000	0.0542
1.9523	14500	0.0540
2.0	14854	-
2.0197	15000	0.0537
2.0870	15500	0.0541
2.1543	16000	0.0543
2.2216	16500	0.0537
2.2889	17000	0.0540
2.3563	17500	0.0540
2.4236	18000	0.0540
2.4909	18500	0.0535
2.5582	19000	0.0536
2.6256	19500	0.0533
2.6929	20000	0.0535
2.7602	20500	0.0536
2.8275	21000	0.0532
2.8948	21500	0.0534
2.9622	22000	0.0539
3.0	22281	-

• The bold row denotes the saved checkpoint.

Training Time

• Training: 1.7 hours
• Evaluation: 34.7 seconds
• Total: 1.7 hours

Framework Versions

• Python: 3.12.4
• Sentence Transformers: 5.5.1
• Transformers: 5.11.0
• PyTorch: 2.5.1+cu121
• Accelerate: 1.13.0
• Datasets: 2.21.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}