thomfoolery/AIE5-MidTerm-finetuned-embeddings

SentenceTransformer based on Snowflake/snowflake-arctic-embed-l

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Snowflake/snowflake-arctic-embed-l
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("thomfoolery/AIE5-MidTerm-finetuned-embeddings")
# Run inference
sentences = [
    'How can you access step and property information on the custom configuration page?',
    "Implementing a custom configuration page\n\nTo build a custom configuration page, you'll use Shopify App Bridge to render a page from your app page in Shopify Flow.\n\nFrom the context of the custom configuration page, you can then access step and property information that you can use to display the appropriate information.\n\nYou can also add additional buttons to the App Bridge title bar, or trigger a redirect to the previous page.\n\nUse Shopify App Bridge to render your app page\n\nNote: The specifics of the Custom Configuration Page integration varies between Shopify App Bridge versions. Make sure you implement the integration specific to your Shopify App Bridge version.\n\nTo render your custom configuration page, you need to integrate Shopify App Bridge on the route that you want to render. To learn about setting up Shopify App Bridge, refer to one of the following pages:\n\nGetting started with Shopify App Bridge\n\nGetting started with App Bridge React\n\nAccess action information",
    "To give merchants a more seamless action configuration experience, and to allow them to manage resources that are external to Shopify Flow, you can embed a page from your app in the Shopify Flow editor.\n\nIn your Shopify Flow action configuration, merchants see a preview with an image and text that's fetched from your custom configuration page preview URL. Merchants can click the button to access the custom configuration page.\n\nYour custom configuration page is then displayed in a frame in the Shopify admin.\n\nIn this tutorial, you'll learn how to render a custom configuration page in Shopify Flow, customize the page frame, and access data relevant to your action in the custom configuration page context.\n\nRequirements\n\nYou've created a Partner account.\n\nYou've created an app.\n\nResources\n\nTo implement this feature, you'll use the following:\n\nShopify App Bridge\n\nApp Bridge components\n\nApp Bridge actions specific to the custom configuration page\n\nImplementing a custom configuration page",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.5417
cosine_accuracy@3	1.0
cosine_accuracy@5	1.0
cosine_accuracy@10	1.0
cosine_precision@1	0.5417
cosine_precision@3	0.3333
cosine_precision@5	0.2
cosine_precision@10	0.1
cosine_recall@1	0.5417
cosine_recall@3	1.0
cosine_recall@5	1.0
cosine_recall@10	1.0
cosine_ndcg@10	0.8145
cosine_mrr@10	0.75
cosine_map@100	0.75

Training Details

Training Dataset

Unnamed Dataset

• Size: 164 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 164 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 13 tokens mean: 19.51 tokens max: 32 tokens	min: 77 tokens mean: 198.13 tokens max: 309 tokens

• Samples:

  sentence_0	sentence_1
  What is the purpose of templates in Shopify Flow?	How templates work A template in Shopify Flow is an example workflow that can be copied into a merchant's shop. Templates help merchants automate a specific use case with minimal or no additional configuration. Flow's template library offers hundreds of templates with features to filter, browse, and search. You can create a template for Shopify Flow that showcases your custom triggers and actions and help merchants do more. Next steps Follow our step by step guide on creating a Flow template. Learn more about how localize your template, the approval process, and more in the reference resource
  How can merchants utilize the template library in Shopify Flow?	How templates work A template in Shopify Flow is an example workflow that can be copied into a merchant's shop. Templates help merchants automate a specific use case with minimal or no additional configuration. Flow's template library offers hundreds of templates with features to filter, browse, and search. You can create a template for Shopify Flow that showcases your custom triggers and actions and help merchants do more. Next steps Follow our step by step guide on creating a Flow template. Learn more about how localize your template, the approval process, and more in the reference resource
  What is required for the test web server to successfully send information to Shopify Flow?	Requirements Make sure that you have the following: A test web server that you can use to send information to Shopify Flow. You can use an existing web server. This web server needs to be able to send POST requests to Shopify's GraphQL Admin API. A test app that works with the test web server and can send HTTP requests. A development store that has Shopify Flow and the test app installed. Your application has access to the read_customers scope. The trigger you will build using this tutorial will be using a customer reference which requires that scope. Step 1: Create a Flow trigger extension To give your Flow action a meaningful name, use the following guidelines: Use an object acted on + past tense verb format. For example, Auction bid placed. Use sentence case. Don't use punctuation. Separate words using spaces. Using Shopify CLI The following steps show how to create a trigger that sends bid information to Shopify Flow when a bid is placed on an auction.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• num_train_epochs: 10
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	cosine_ndcg@10
1.0	17	0.8353
2.0	34	0.8299
2.9412	50	0.8507
3.0	51	0.8507
4.0	68	0.8353
5.0	85	0.8452
5.8824	100	0.8606
6.0	102	0.8606
7.0	119	0.8299
8.0	136	0.8145
8.8235	150	0.8145
9.0	153	0.8145
10.0	170	0.8145

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.4.1
• Transformers: 4.48.3
• PyTorch: 2.5.1+cu124
• Accelerate: 1.3.0
• Datasets: 3.3.2
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}