kaamd/ll

SentenceTransformer

This is a sentence-transformers model trained on the fiqa dataset. It maps sentences & paragraphs to a 4096-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Maximum Sequence Length: 32768 tokens
• Output Dimensionality: 4096 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • fiqa

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 32768, 'do_lower_case': False, 'architecture': 'MistralModel'})
  (1): Pooling({'word_embedding_dimension': 4096, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True})
  (2): SparseEmbedding(
    (sparsifyer): ZeroNeuron(in_features=4096, out_features=4096)
  )
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: I am under 18 years old, in the US, my parents have terrible credit, how can I take out a loan?',
    "Depending on the state this might not be possible. Loans are considered contracts, and various states regulate how minors may enter into them. For example, in the state of Oregon, a minor may NOT enter into a contract without their parent being on the contract as well. So you are forced to wait until you turn 18. At that time you won't have a credit history, and to lenders that often is worse than having bad credit. I can't help with the car (other than to recommend you buy a junker for $500-$1,000 and just live with it for now), but you could certainly get a secured credit card or line of credit from your local bank. The way they are arranged is, you make a deposit of an amount of your choosing (generally at least $200 for credit cards, and $1,000 for lines of credit), and receive a revolving line with a limit of that same amount. As you use and pay on this loan, it will be reported in your credit history. If you start that now, by the time you turn 18 you will have much better options for purchasing vehicles.",
    "In the equity markets, the P/E is usually somewhere around 15.  The P/E can be viewed as the inverse of the rate of a perpetuity. Since the average is 15, and the E/P of that would be 6.7%, r should be 6.7% on average. If your business is growing, the growth rate can be incorporated like so: As you can see, a high g would make the price negative, in essence the seller should actually pay someone to take the business, but in reality, r is determined from the p and an estimated g. For a business of any growth rate, it's best to compare the multiple to the market, so for the average business in the market with your business's growth rate and industry, that P/E would be best applied to your company's income.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 4096]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6277, 0.2807],
#         [0.6277, 1.0000, 0.2775],
#         [0.2807, 0.2775, 1.0000]])

Training Details

Training Dataset

fiqa

• Dataset: fiqa
• Size: 14,166 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 24 tokens mean: 36.38 tokens max: 62 tokens	min: 2 tokens mean: 223.61 tokens max: 1683 tokens

• Samples:

  anchor	positive
  Instruct: Given a web search query, retrieve relevant passages that answer the query Query: What is considered a business expense on a business trip?	The IRS Guidance pertaining to the subject. In general the best I can say is your business expense may be deductible. But it depends on the circumstances and what it is you want to deduct. Travel Taxpayers who travel away from home on business may deduct related expenses, including the cost of reaching their destination, the cost of lodging and meals and other ordinary and necessary expenses. Taxpayers are considered “traveling away from home” if their duties require them to be away from home substantially longer than an ordinary day’s work and they need to sleep or rest to meet the demands of their work. The actual cost of meals and incidental expenses may be deducted or the taxpayer may use a standard meal allowance and reduced record keeping requirements. Regardless of the method used, meal deductions are generally limited to 50 percent as stated earlier. Only actual costs for lodging may be claimed as an expense and receipts must be kept for documentation. ...
  Instruct: Given a web search query, retrieve relevant passages that answer the query Query: Business Expense - Car Insurance Deductible For Accident That Occurred During a Business Trip	As a general rule, you must choose between a mileage deduction or an actual expenses deduction. The idea is that the mileage deduction is supposed to cover all costs of using the car. Exceptions include parking fees and tolls, which can be deducted separately under either method. You explicitly cannot deduct insurance costs if you claim a mileage deduction. Separately, you probably won't be able to deduct the deductible for your car as a casualty loss. You first subtract $100 from the deductible and then divide it by your Adjusted Gross Income (AGI) from your tax return. If your deductible is over 10% of your AGI, you can deduct it. Note that even with a $1500 deductible, you won't be able to deduct anything if you made more than $14,000 for the year. For most people, the insurance deductible just isn't large enough relative to income to be tax deductible. Source
  Instruct: Given a web search query, retrieve relevant passages that answer the query Query: Starting a new online business	Most US states have rules that go something like this: You will almost certainly have to pay some registration fees, as noted above. Depending on how you organize, you may or may not need to file a separate tax return for the business. (If you're sole proprietor for tax purposes, then you file on Schedule C on your personal Form 1040.) Whether or not you pay taxes depends on whether you have net income. It's possible that some losses might also be deductible. (Note that you may have to file a return even if you don't have net income - Filing and needing to pay are not the same since your return may indicate no tax due.) In addition, at the state level, you may have to pay additional fees or taxes beyond income tax depending on what you sell and how you sell it. (Sales tax, for example, might come into play as might franchise taxes.) You'll need to check your own state law for that. As always, it could be wise to get professional tax and accounting advice that's tailored to your si...

• Loss: zero-neuron.src.embedding.sparse_loss.SparseLoss

Training Hyperparameters

Non-Default Hyperparameters

• per_device_eval_batch_size: 16
• learning_rate: 0.0005
• num_train_epochs: 2
• lr_scheduler_type: cosine
• warmup_ratio: 0.01
• save_safetensors: False
• bf16: True
• remove_unused_columns: False
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 0.0005
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.01
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: False
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0056	10	1.1694
0.0113	20	1.2809
0.0169	30	1.2538
0.0226	40	1.2237
0.0282	50	1.1809
0.0339	60	1.1602
0.0395	70	1.2008
0.0452	80	1.1064
0.0508	90	1.0857
0.0565	100	1.0553
0.0621	110	1.0513
0.0678	120	0.9236
0.0734	130	0.8998
0.0791	140	0.8509
0.0847	150	0.7769
0.0903	160	0.7268
0.0960	170	0.7287
0.1016	180	0.6437
0.1073	190	0.6653
0.1129	200	0.5888
0.1186	210	0.6097
0.1242	220	0.6939
0.1299	230	0.5969
0.1355	240	0.5333
0.1412	250	0.5143
0.1468	260	0.6152
0.1525	270	0.4779
0.1581	280	0.5182
0.1637	290	0.5724
0.1694	300	0.5073
0.1750	310	0.4924
0.1807	320	0.5219
0.1863	330	0.5621
0.1920	340	0.4535
0.1976	350	0.4818
0.2033	360	0.4773
0.2089	370	0.4948
0.2146	380	0.4277
0.2202	390	0.5043
0.2259	400	0.5746
0.2315	410	0.4762
0.2372	420	0.4432
0.2428	430	0.4771
0.2484	440	0.5298
0.2541	450	0.4352
0.2597	460	0.5714
0.2654	470	0.508
0.2710	480	0.5215
0.2767	490	0.5096
0.2823	500	0.4598
0.2880	510	0.5843
0.2936	520	0.5581
0.2993	530	0.4686
0.3049	540	0.4956
0.3106	550	0.4209
0.3162	560	0.4181
0.3219	570	0.4847
0.3275	580	0.5193
0.3331	590	0.4235
0.3388	600	0.4626
0.3444	610	0.4309
0.3501	620	0.451
0.3557	630	0.4742
0.3614	640	0.4892
0.3670	650	0.4478
0.3727	660	0.4461
0.3783	670	0.5197
0.3840	680	0.4692
0.3896	690	0.4272
0.3953	700	0.4196
0.4009	710	0.4737
0.4065	720	0.4015
0.4122	730	0.4786
0.4178	740	0.3968
0.4235	750	0.4499
0.4291	760	0.478
0.4348	770	0.4003
0.4404	780	0.4679
0.4461	790	0.4129
0.4517	800	0.452
0.4574	810	0.4238
0.4630	820	0.4761
0.4687	830	0.4324
0.4743	840	0.4535
0.4800	850	0.4914
0.4856	860	0.5368
0.4912	870	0.4106
0.4969	880	0.419
0.5025	890	0.3884
0.5082	900	0.4833
0.5138	910	0.4295
0.5195	920	0.3673
0.5251	930	0.4245
0.5308	940	0.4636
0.5364	950	0.3897
0.5421	960	0.4342
0.5477	970	0.442
0.5534	980	0.4443
0.5590	990	0.3737
0.5647	1000	0.441
0.5703	1010	0.4247
0.5759	1020	0.4583
0.5816	1030	0.4077
0.5872	1040	0.5236
0.5929	1050	0.4307
0.5985	1060	0.5054
0.6042	1070	0.4787
0.6098	1080	0.4521
0.6155	1090	0.4011
0.6211	1100	0.3864
0.6268	1110	0.4191
0.6324	1120	0.436
0.6381	1130	0.4469
0.6437	1140	0.4416
0.6494	1150	0.4475
0.6550	1160	0.3857
0.6606	1170	0.3571
0.6663	1180	0.441
0.6719	1190	0.4144
0.6776	1200	0.4108
0.6832	1210	0.4051
0.6889	1220	0.489
0.6945	1230	0.3881
0.7002	1240	0.4971
0.7058	1250	0.415
0.7115	1260	0.4048
0.7171	1270	0.3805
0.7228	1280	0.3869
0.7284	1290	0.3804
0.7340	1300	0.4141
0.7397	1310	0.4223
0.7453	1320	0.3836
0.7510	1330	0.4012
0.7566	1340	0.4725
0.7623	1350	0.3946
0.7679	1360	0.4424
0.7736	1370	0.4256
0.7792	1380	0.4381
0.7849	1390	0.3634
0.7905	1400	0.3568
0.7962	1410	0.4158
0.8018	1420	0.3982
0.8075	1430	0.4734
0.8131	1440	0.3787
0.8187	1450	0.4492
0.8244	1460	0.4504
0.8300	1470	0.4125
0.8357	1480	0.4059
0.8413	1490	0.419
0.8470	1500	0.4269
0.8526	1510	0.5586
0.8583	1520	0.4664
0.8639	1530	0.5185
0.8696	1540	0.422
0.8752	1550	0.5141
0.8809	1560	0.4576
0.8865	1570	0.372
0.8922	1580	0.4194
0.8978	1590	0.4074
0.9034	1600	0.3894
0.9091	1610	0.4172
0.9147	1620	0.4274
0.9204	1630	0.4013
0.9260	1640	0.4072
0.9317	1650	0.3616
0.9373	1660	0.3485
0.9430	1670	0.4478
0.9486	1680	0.4543
0.9543	1690	0.4229
0.9599	1700	0.4186
0.9656	1710	0.378
0.9712	1720	0.3753
0.9768	1730	0.4575
0.9825	1740	0.4291
0.9881	1750	0.4288
0.9938	1760	0.3678
0.9994	1770	0.5298
1.0051	1780	0.3926
1.0107	1790	0.3799
1.0164	1800	0.4288
1.0220	1810	0.4323
1.0277	1820	0.4371
1.0333	1830	0.4652
1.0390	1840	0.3565
1.0446	1850	0.4567
1.0503	1860	0.3947
1.0559	1870	0.3868
1.0615	1880	0.4143
1.0672	1890	0.482
1.0728	1900	0.3763
1.0785	1910	0.3795
1.0841	1920	0.4413
1.0898	1930	0.4761
1.0954	1940	0.3907
1.1011	1950	0.4066
1.1067	1960	0.3905
1.1124	1970	0.3944
1.1180	1980	0.4022
1.1237	1990	0.398
1.1293	2000	0.3473
1.1350	2010	0.4357
1.1406	2020	0.3823
1.1462	2030	0.3628
1.1519	2040	0.403
1.1575	2050	0.3965
1.1632	2060	0.3837
1.1688	2070	0.5012
1.1745	2080	0.3959
1.1801	2090	0.3661
1.1858	2100	0.4603
1.1914	2110	0.4607
1.1971	2120	0.4241
1.2027	2130	0.5183
1.2084	2140	0.3533
1.2140	2150	0.3877
1.2196	2160	0.4298
1.2253	2170	0.4228
1.2309	2180	0.4131
1.2366	2190	0.4034
1.2422	2200	0.3834
1.2479	2210	0.4183
1.2535	2220	0.5475
1.2592	2230	0.4755
1.2648	2240	0.4478
1.2705	2250	0.3763
1.2761	2260	0.4493
1.2818	2270	0.4001
1.2874	2280	0.3765
1.2931	2290	0.3379
1.2987	2300	0.337
1.3043	2310	0.4143
1.3100	2320	0.4794
1.3156	2330	0.4004
1.3213	2340	0.3674
1.3269	2350	0.3963
1.3326	2360	0.3896
1.3382	2370	0.5062
1.3439	2380	0.4114
1.3495	2390	0.3955
1.3552	2400	0.4682
1.3608	2410	0.3551
1.3665	2420	0.3536
1.3721	2430	0.3784
1.3778	2440	0.3456
1.3834	2450	0.4273
1.3890	2460	0.4005
1.3947	2470	0.3957
1.4003	2480	0.3371
1.4060	2490	0.3451
1.4116	2500	0.4735
1.4173	2510	0.4013
1.4229	2520	0.3751
1.4286	2530	0.365
1.4342	2540	0.3548
1.4399	2550	0.4227
1.4455	2560	0.3626
1.4512	2570	0.404
1.4568	2580	0.4055
1.4625	2590	0.4513
1.4681	2600	0.4147
1.4737	2610	0.3623
1.4794	2620	0.404
1.4850	2630	0.4
1.4907	2640	0.3854
1.4963	2650	0.4082
1.5020	2660	0.3502
1.5076	2670	0.4022
1.5133	2680	0.4479
1.5189	2690	0.3456
1.5246	2700	0.3992
1.5302	2710	0.4143
1.5359	2720	0.3925
1.5415	2730	0.366
1.5471	2740	0.4254
1.5528	2750	0.4337
1.5584	2760	0.471
1.5641	2770	0.4201
1.5697	2780	0.4357
1.5754	2790	0.4289
1.5810	2800	0.4287
1.5867	2810	0.4349
1.5923	2820	0.4551
1.5980	2830	0.3562
1.6036	2840	0.4925
1.6093	2850	0.4104
1.6149	2860	0.4691
1.6206	2870	0.383
1.6262	2880	0.3612
1.6318	2890	0.4584
1.6375	2900	0.3828
1.6431	2910	0.3784
1.6488	2920	0.4148
1.6544	2930	0.4535
1.6601	2940	0.3523
1.6657	2950	0.3501
1.6714	2960	0.3703
1.6770	2970	0.388
1.6827	2980	0.3846
1.6883	2990	0.4212
1.6940	3000	0.4192
1.6996	3010	0.4265
1.7053	3020	0.4385
1.7109	3030	0.4197
1.7165	3040	0.3488
1.7222	3050	0.3666
1.7278	3060	0.3909
1.7335	3070	0.5085
1.7391	3080	0.3495
1.7448	3090	0.4198
1.7504	3100	0.3647
1.7561	3110	0.3873
1.7617	3120	0.4038
1.7674	3130	0.4471
1.7730	3140	0.4078
1.7787	3150	0.3823
1.7843	3160	0.4852
1.7899	3170	0.3891
1.7956	3180	0.4334
1.8012	3190	0.3836
1.8069	3200	0.4
1.8125	3210	0.4126
1.8182	3220	0.3767
1.8238	3230	0.4085
1.8295	3240	0.3919
1.8351	3250	0.358
1.8408	3260	0.3709
1.8464	3270	0.4131
1.8521	3280	0.4082
1.8577	3290	0.4547
1.8634	3300	0.4317
1.8690	3310	0.3981
1.8746	3320	0.4585
1.8803	3330	0.3698
1.8859	3340	0.3662
1.8916	3350	0.3955
1.8972	3360	0.4387
1.9029	3370	0.5001
1.9085	3380	0.3708
1.9142	3390	0.4448
1.9198	3400	0.3632
1.9255	3410	0.4589
1.9311	3420	0.4085
1.9368	3430	0.3993
1.9424	3440	0.4598
1.9481	3450	0.4019
1.9537	3460	0.4179
1.9593	3470	0.3804
1.9650	3480	0.4229
1.9706	3490	0.3933
1.9763	3500	0.4217
1.9819	3510	0.4182
1.9876	3520	0.5265
1.9932	3530	0.403
1.9989	3540	0.3758

Framework Versions

• Python: 3.10.16
• Sentence Transformers: 5.1.1
• Transformers: 4.55.4
• PyTorch: 2.6.0+cu124
• Accelerate: 1.8.0.dev0
• Datasets: 3.2.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}