seongil-dn/further_trainset_large_v2

SentenceTransformer

This is a sentence-transformers model trained. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Maximum Sequence Length: 1024 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'When were bluebonnets named the state flower of Texas?',
    'Bluebonnet is a name given to any number of blue-flowered species of the genus "Lupinus" predominantly found in southwestern United States and is collectively the state flower of Texas. The shape of the petals on the flower resembles the bonnet worn by pioneer women to shield them from the sun.\nSpecies often called bluebonnets include:On March 7, 1901, "Lupinus subcarnosus" became the only species of bluebonnet recognized as the state flower of Texas; however, "Lupinus texensis" emerged as the favorite of most Texans. So, in 1971, the Texas Legislature made any similar species of "Lupinus" that could be found in Texas the state flower.',
    'The second major festival hosted in Ennis is the Bluebonnet Trails Festival, celebrating the state flower of Texas and the vibrant bloom of wildflowers in the surrounding countryside. The event attracts tens of thousands of tourists each year to events including sightseeing excursions and a festival in downtown. The festival is held on the third weekend of April, and the Bluebonnet Trails are hosted for the entire month. First hosted along the Kachina Prairie Park\'s historic mile-long trail system in 1938, the Bluebonnet Trails have since expanded into a route map of several dozen miles along rural farm roads throughout the surrounding countryside east and northeast of the city. The routes for these sightseeing excursions have been officially hosted and mapped out by the Ennis Garden Club since 1951. To commemorate the popularity of the Bluebonnet Trails Festival and the efforts made to celebrate and preserve the state flower of Texas, Ennis was designated by the 1997 Texas State Legislature as the "Official Bluebonnet City of Texas" and home to the "Official Bluebonnet Trail of Texas."',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 105,064 training samples
• Columns: anchor, positive, negative, negative_2, negative_3, and negative_4
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative	negative_2	negative_3	negative_4
  type	string	string	string	string	string	string
  details	min: 6 tokens mean: 11.81 tokens max: 26 tokens	min: 17 tokens mean: 169.73 tokens max: 986 tokens	min: 12 tokens mean: 181.7 tokens max: 759 tokens	min: 22 tokens mean: 184.76 tokens max: 817 tokens	min: 14 tokens mean: 186.03 tokens max: 859 tokens	min: 12 tokens mean: 179.54 tokens max: 759 tokens

• Samples:

  anchor	positive	negative	negative_2	negative_3	negative_4
  When was quantum field theory developed?	The third thread in the development of quantum field theory was the need to handle the statistics of many-particle systems consistently and with ease. In 1927, Pascual Jordan tried to extend the canonical quantization of fields to the many-body wave functions of identical particles using a formalism which is known as statistical transformation theory; this procedure is now sometimes called second quantization. In 1928, Jordan and Eugene Wigner found that the quantum field describing electrons, or other fermions, had to be expanded using anti-commuting creation and annihilation operators due to the Pauli exclusion principle (see Jordan–Wigner transformation). This thread of development was incorporated into many-body theory and strongly influenced condensed matter physics and nuclear physics.	The application of the new quantum theory to electromagnetism resulted in quantum field theory, which was developed starting around 1930. Quantum field theory has driven the development of more sophisticated formulations of quantum mechanics, of which the ones presented here are simple special cases.	Two classic text-books from the 1960s, James D. Bjorken, Sidney David Drell, "Relativistic Quantum Mechanics" (1964) and J. J. Sakurai, "Advanced Quantum Mechanics" (1967), thoroughly developed the Feynman graph expansion techniques using physically intuitive and practical methods following from the correspondence principle, without worrying about the technicalities involved in deriving the Feynman rules from the superstructure of quantum field theory itself. Although both Feynman's heuristic and pictorial style of dealing with the infinities, as well as the formal methods of Tomonaga and Schwinger, worked extremely well, and gave spectacularly accurate answers, the true analytical nature of the question of "renormalizability", that is, whether ANY theory formulated as a "quantum field theory" would give finite answers, was not worked-out until much later, when the urgency of trying to formulate finite theories for the strong and electro-weak (and gravitational interactions) demanded i...	It was evident from the beginning that a proper quantum treatment of the electromagnetic field had to somehow incorporate Einstein's relativity theory, which had grown out of the study of classical electromagnetism. This need to put together relativity and quantum mechanics was the second major motivation in the development of quantum field theory. Pascual Jordan and Wolfgang Pauli showed in 1928 that quantum fields could be made to behave in the way predicted by special relativity during coordinate transformations (specifically, they showed that the field commutators were Lorentz invariant). A further boost for quantum field theory came with the discovery of the Dirac equation, which was originally formulated and interpreted as a single-particle equation analogous to the Schrödinger equation, but unlike the Schrödinger equation, the Dirac equation satisfies both the Lorentz invariance, that is, the requirements of special relativity, and the rules of quantum mechanics. The Dirac equa...	Through the works of Born, Heisenberg, and Pascual Jordan in 1925-1926, a quantum theory of the free electromagnetic field (one with no interactions with matter) was developed via canonical quantization by treating the electromagnetic field as a set of quantum harmonic oscillators. With the exclusion of interactions, however, such a theory was yet incapable of making quantitative predictions about the real world.
  Was there a year 0?	Cassini gave the following reasons for using a year 0: Fred Espanak of NASA lists 50 phases of the moon within year 0, showing that it is a full year, not an instant in time. Jean Meeus gives the following explanation: Although he used the usual French terms "avant J.-C." (before Jesus Christ) and "après J.-C." (after Jesus Christ) to label years elsewhere in his book, the Byzantine historian Venance Grumel used negative years (identified by a minus sign, −) to label BC years and unsigned positive years to label AD years in a table. He did so possibly to save space and put no year 0 between them.	Games Def Interceptions Fumbles Sacks & Tackles Year Age Tm Pos No. G GS Int Yds TD Lng PD FF Fmb FR Yds TD Sk Tkl Ast Sfty AV 2004 23 NWE ss 42 13 2 0 0 0 0 2 1 0 1 0 0 15 8 2 2005 24 IND 36 16 0 1 0 1 0 0 8 2 1 2006 25 IND ss 36 10 1 0 0 0 0 2 11 0 1 Career 39 3 0 0 0 0 4 2 0 2 0 0 34 10 4 2 yrs IND 26 1 0 0 0 0 2 1 0 1 0 0 19 2 2 1 yr NWE 13 2 0 0 0 0 2 1 0 1 0 0 15 8 2 After pleading guilty in January 2008 to drug charges in Virginia Beach, VA stemming from a March 2007 incident, Reid was initially sentenced to two years in prison for possessing marijuana with the intent to distribute but had the sentence suspended with the agreement he would stay out of trouble for two years. His license was also suspended for six months and ordered to attend drug treatment and counseling.	This enzyme belongs to the family of oxidoreductases, specifically those acting on paired donors, with O2 as oxidant and incorporation or reduction of oxygen. The oxygen incorporated need not be derived from O2 with 2-oxoglutarate as one donor, and incorporation of one atom o oxygen into each donor. The systematic name of this enzyme class is N6,N6,N6-trimethyl-L-lysine,2-oxoglutarate:oxygen oxidoreductase (3-hydroxylating). Other names in common use include trimethyllysine alpha-ketoglutarate dioxygenase, TML-alpha-ketoglutarate dioxygenase, TML hydroxylase, 6-N,6-N,6-N-trimethyl-L-lysine,2-oxoglutarate:oxygen oxidoreductase, and (3-hydroxylating). This enzyme participates in lysine degradation and L-carnitine biosynthesis and requires the presence of iron and ascorbate.	ㅜ is one of the Korean hangul. The Unicode for ㅜ is U+315C.	ㅌ is one of the Korean hangul. The Unicode for ㅌ is U+314C.
  When is the dialectical method used?	The Dialect Test was created by A.J. Ellis in February 1879, and was used in the fieldwork for his work "On Early English Pronunciation". It stands as one of the earliest methods of identifying vowel sounds and features of speech. The aim was to capture the main vowel sounds of an individual dialect by listening to the reading of a short passage. All the categories of West Saxon words and vowels were included in the test so that comparisons could be made with the historic West Saxon speech as well as with various other dialects.	Karl Popper has attacked the dialectic repeatedly. In 1937, he wrote and delivered a paper entitled "What Is Dialectic?" in which he attacked the dialectical method for its willingness "to put up with contradictions". Popper concluded the essay with these words: "The whole development of dialectic should be a warning against the dangers inherent in philosophical system-building. It should remind us that philosophy should not be made a basis for any sort of scientific system and that philosophers should be much more modest in their claims. One task which they can fulfill quite usefully is the study of the critical methods of science" (Ibid., p. 335).	He was one of the first to apply Labovian methods in Britain with his research in 1970-1 on the speech of Bradford, Halifax and Huddersfield. He concluded that the speech detailed in most of dialectology (e.g. A. J. Ellis, the Survey of English Dialects) had virtually disappeared, having found only one speaker out of his sample of 106 speakers who regularly used dialect. However, he found that differences in speech persisted as an indicator of social class, age and gender. This PhD dissertation was later adapted into a book, "Dialect and Accent in Industrial West Yorkshire". The work was criticised by Graham Shorrocks on the grounds that the sociolinguistic methods used were inappropriate for recording the traditional vernacular and that there was an inadequate basis for comparison with earlier dialect studies in West Yorkshire.	The Institute also attempted to reformulate dialectics as a concrete method. The use of such a dialectical method can be traced back to the philosophy of Hegel, who conceived dialectic as the tendency of a notion to pass over into its own negation as the result of conflict between its inherent contradictory aspects. In opposition to previous modes of thought, which viewed things in abstraction, each by itself and as though endowed with fixed properties, Hegelian dialectic has the ability to consider ideas according to their movement and change in time, as well as according to their interrelations and interactions.	For Marx, dialectics is not a formula for generating predetermined outcomes but is a method for the empirical study of social processes in terms of interrelations, development, and transformation. In his introduction to the Penguin edition of Marx's "Capital", Ernest Mandel writes, "When the dialectical method is applied to the study of economic problems, economic phenomena are not viewed separately from each other, by bits and pieces, but in their inner connection as an integrated totality, structured around, and by, a basic predominant mode of production."

• Loss: CachedGISTEmbedLoss with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
    (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.01}
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 1024
• learning_rate: 3e-05
• weight_decay: 0.01
• num_train_epochs: 6
• warmup_ratio: 0.05
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 1024
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 3e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 6
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.05
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss
0.04	1	0.1495
0.08	2	0.1625
0.12	3	0.1623
0.16	4	0.1883
0.2	5	0.1559
0.24	6	0.15
0.28	7	0.1513
0.32	8	0.1646
0.36	9	0.1608
0.4	10	0.1774
0.44	11	0.1545
0.48	12	0.1646
0.52	13	0.1901
0.56	14	0.1791
0.6	15	0.1621
0.64	16	0.1539
0.68	17	0.1623
0.72	18	0.1802
0.76	19	0.1613
0.8	20	0.172
0.84	21	0.1481
0.88	22	0.1783
0.92	23	0.1636
0.96	24	0.1753
1.0	25	0.1763
1.04	26	0.1302
1.08	27	0.1172
1.12	28	0.1207
1.16	29	0.1259
1.2	30	0.1241
1.24	31	0.1208
1.28	32	0.1155
1.32	33	0.1168
1.3600	34	0.0985
1.4	35	0.119
1.44	36	0.1063
1.48	37	0.1277
1.52	38	0.1071
1.56	39	0.1234
1.6	40	0.106
1.6400	41	0.109
1.6800	42	0.1149
1.72	43	0.1068
1.76	44	0.1035
1.8	45	0.1221
1.8400	46	0.1007
1.88	47	0.1001
1.92	48	0.1105
1.96	49	0.1144
2.0	50	0.099
2.04	51	0.0873
2.08	52	0.0845
2.12	53	0.0848
2.16	54	0.0815
2.2	55	0.073
2.24	56	0.0915
2.2800	57	0.0833
2.32	58	0.0808
2.36	59	0.0837
2.4	60	0.0889
2.44	61	0.0829
2.48	62	0.0887
2.52	63	0.0898
2.56	64	0.0679
2.6	65	0.0835
2.64	66	0.0736
2.68	67	0.0813
2.7200	68	0.0832
2.76	69	0.0785
2.8	70	0.076
2.84	71	0.0833
2.88	72	0.0891
2.92	73	0.0709
2.96	74	0.0825
3.0	75	0.0695
3.04	76	0.0553
3.08	77	0.0612
3.12	78	0.0663
3.16	79	0.0663
3.2	80	0.0585
3.24	81	0.0647
3.2800	82	0.0631
3.32	83	0.0676
3.36	84	0.0708
3.4	85	0.0599
3.44	86	0.0629
3.48	87	0.0618
3.52	88	0.0529
3.56	89	0.0572
3.6	90	0.0641
3.64	91	0.0636
3.68	92	0.0538
3.7200	93	0.061
3.76	94	0.0541
3.8	95	0.0671
3.84	96	0.0589
3.88	97	0.0575
3.92	98	0.0639
3.96	99	0.059
4.0	100	0.0593
4.04	101	0.0481
4.08	102	0.0496
4.12	103	0.0519
4.16	104	0.0536
4.2	105	0.0481
4.24	106	0.0521
4.28	107	0.0551
4.32	108	0.0495
4.36	109	0.0524
4.4	110	0.0463
4.44	111	0.0572
4.48	112	0.0419
4.52	113	0.0524
4.5600	114	0.053
4.6	115	0.0503
4.64	116	0.0522
4.68	117	0.0388
4.72	118	0.0436
4.76	119	0.0527
4.8	120	0.0454
4.84	121	0.0503
4.88	122	0.053
4.92	123	0.0566
4.96	124	0.0534
5.0	125	0.0455
5.04	126	0.0471
5.08	127	0.0446
5.12	128	0.0469
5.16	129	0.0495
5.2	130	0.0412
5.24	131	0.0482
5.28	132	0.0425
5.32	133	0.0389
5.36	134	0.0468
5.4	135	0.046
5.44	136	0.0438
5.48	137	0.0465
5.52	138	0.0418
5.5600	139	0.0453
5.6	140	0.0463
5.64	141	0.0439
5.68	142	0.0447
5.72	143	0.0464
5.76	144	0.0413
5.8	145	0.0388
5.84	146	0.0468
5.88	147	0.0416
5.92	148	0.0441
5.96	149	0.0446
6.0	150	0.0469

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.4.1
• Transformers: 4.49.0
• PyTorch: 2.5.1+cu124
• Accelerate: 1.4.0
• Datasets: 2.21.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}