Hugging Face's logo

AntoineGourru
/
FT_phi_cos

Sentence Similarity
sentence-transformers
Safetensors
phi3
feature-extraction
dense
Generated from Trainer
dataset_size:5749
loss:CosineSimilarityLoss
custom_code
Eval Results (legacy)
Model card Files
xet
Community
FT_phi_cos / README.md
AntoineGourru's picture
AntoineGourru
Add new SentenceTransformer model
1246bf6 verified
preview code
|
raw
history blame contribute delete
13.8 kB
metadata 
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - dense
  - generated_from_trainer
  - dataset_size:5749
  - loss:CosineSimilarityLoss
base_model: microsoft/Phi-3-mini-4k-instruct
widget:
  - source_sentence: A man is playing the guitar.
    sentences:
      - A man plays an acoustic guitar.
      - Ryanair chief hails report findings
      - 2 British soldiers killed in Afghan insider attack
  - source_sentence: You're "a bunch of cells."
    sentences:
      - I'm "a bunch of cells."
      - Four arrested as Bangladesh building toll rises to 352
      - Five cows grazing on a patch of grass between two roadways.
  - source_sentence: The skateboarder gets to the top of the ramp.
    sentences:
      - >-
        Kroger's Ralphs chain and Albertsons immediately locked out their
        grocery workers in a show of solidarity.
      - The man took a piece of pepperoni pizza out of the box.
      - The skateboarder rides the pipe wall at a skater park.
  - source_sentence: Iran dissidents 'killed in Iraq'
    sentences:
      - A baby is crawling across the floor.
      - >-
        The Mets took Lastings Milledge, an outfielder from Florida, with the
        12th pick.
      - France loses influence in Europe
  - source_sentence: N Korea warns of retaliation for South Korea drill
    sentences:
      - North warns of retaliation for Seoul's naval drill plan
      - Bangladeshi Islamists rally to demand action against atheist bloggers
      - India's Anti-Graft Party Forms Government in Delhi
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - pearson_cosine
  - spearman_cosine
model-index:
  - name: SentenceTransformer based on microsoft/Phi-3-mini-4k-instruct
    results:
      - task:
          type: semantic-similarity
          name: Semantic Similarity
        dataset:
          name: stsb dev
          type: stsb-dev
        metrics:
          - type: pearson_cosine
            value: 0.8730412567769437
            name: Pearson Cosine
          - type: spearman_cosine
            value: 0.8731618839441975
            name: Spearman Cosine

SentenceTransformer based on microsoft/Phi-3-mini-4k-instruct

This is a sentence-transformers model finetuned from microsoft/Phi-3-mini-4k-instruct. It maps sentences & paragraphs to a 3072-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: microsoft/Phi-3-mini-4k-instruct
  • Maximum Sequence Length: 128 tokens
  • Output Dimensionality: 3072 dimensions
  • Similarity Function: Cosine Similarity

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False, 'architecture': 'Phi3Model'})
  (1): Pooling({'word_embedding_dimension': 3072, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("AntoineGourru/FT_phi_cos")
# Run inference
sentences = [
    'N Korea warns of retaliation for South Korea drill',
    "North warns of retaliation for Seoul's naval drill plan",
    'Bangladeshi Islamists rally to demand action against atheist bloggers',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 3072]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.7161, 0.1516],
#         [0.7161, 1.0000, 0.1610],
#         [0.1516, 0.1610, 1.0000]])

Evaluation

Metrics

Semantic Similarity

Metric Value
pearson_cosine 0.873
spearman_cosine 0.8732

Training Details

Training Dataset

Unnamed Dataset

  • Size: 5,749 training samples
  • Columns: sentence_0, sentence_1, and label
  • Approximate statistics based on the first 1000 samples:
    sentence_0 sentence_1 label
    type string string float
    details
    • min: 4 tokens
    • mean: 15.28 tokens
    • max: 87 tokens
    • min: 5 tokens
    • mean: 15.04 tokens
    • max: 68 tokens
    • min: 0.0
    • mean: 0.53
    • max: 1.0
  • Samples:
    sentence_0 sentence_1 label
    The results were released at Tuesday's meeting in Seattle of the American Thoracic Society and will be published in Thursday's New England Journal of Medicine. The study results were released at a meeting in Seattle of the American Thoracic Society and also will be published in tomorrow's issue of The New England Journal of Medicine. 0.8727999687194824
    Put a Little Love in your Heart We are all vessels filled with many wonders. Landon And So This is Christmas We are all vessels filled with many wonders. 0.5599999904632569
    Wall Street analysts had expected 22 cents a share, according to Thomson First Call. The results were 3 cents a share lower than the forecast of analysts surveyed by Thomson First Call. 0.4400000095367432
  • Loss: CosineSimilarityLoss with these parameters:
    {
    "loss_fct": "torch.nn.modules.loss.MSELoss"
}

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 4
  • per_device_eval_batch_size: 4
  • num_train_epochs: 1
  • multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand
  • do_predict: False
  • eval_strategy: no
  • prediction_loss_only: True
  • per_device_train_batch_size: 4
  • per_device_eval_batch_size: 4
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 5e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1
  • num_train_epochs: 1
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: None
  • warmup_ratio: None
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • enable_jit_checkpoint: False
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • use_cpu: False
  • seed: 42
  • data_seed: None
  • bf16: False
  • fp16: False
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: -1
  • ddp_backend: None
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch_fused
  • optim_args: None
  • group_by_length: False
  • length_column_name: length
  • project: huggingface
  • trackio_space_id: trackio
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: None
  • hub_always_push: False
  • hub_revision: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • auto_find_batch_size: False
  • full_determinism: False
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • include_num_input_tokens_seen: no
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • liger_kernel_config: None
  • eval_use_gather_object: False
  • average_tokens_across_devices: True
  • use_cache: False
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: round_robin
  • router_mapping: {}
  • learning_rate_mapping: {}

Training Logs

Epoch Step Training Loss stsb-dev_spearman_cosine
0.3477 500 0.0533 -
0.6954 1000 0.0263 -
1.0 1438 - 0.8732

Framework Versions

  • Python: 3.12.12
  • Sentence Transformers: 5.2.3
  • Transformers: 5.0.0
  • PyTorch: 2.10.0+cu128
  • Accelerate: 1.12.0
  • Datasets: 4.0.0
  • Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}