Chandar/sv-subject-based-bert-base-uncased

SentenceTransformer based on google-bert/bert-base-uncased

This is a sentence-transformers model finetuned from google-bert/bert-base-uncased. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google-bert/bert-base-uncased
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Chandar/sv-subject-based-bert-base-uncased")
# Run inference
sentences = [
    'mason-wasp\tis\tsolitary.\nThus\tthe\tprocess\tof\torganic\tevolution\tis\tfar\tfrom\tbeing\tfully\tunderstood.\tWe\tcan\nonly\tsuppose\tthat\tas\tthere\tare\tdevised\tby\thuman\tbeings\tmany\tpuzzles\tapparently\nunanswerable\ttill\tthe\tanswer\tis\tgiven,\tand\tmany\tnecromantic\ttricks\twhich\tseem\nimpossible\ttill\tthe\tmode\tof\tperformance\tis\tshown;\tso\tthere\tare\tapparently\nincomprehensible\tresults\twhich\tare\treally\tachieved\tby\tnatural\tprocesses.\tOr,\notherwise,\twe\tmust\tconclude\tthat\tsince\tLife\titself\tproves\tto\tbe\tin\tits\tultimate\nnature\tinconceivable,\tthere\tis\tprobably\tan\tinconceivable\telement\tin\tits\tultimate\nworkings.\nEND\tOF\tVOL.\tI.',
    '11.4 The Flea on Schr ¨odinger’s Cat 455\nwhere M = Nm is the total mass of the system, for simplicity we assumed V to be\nanalytic (it will even be taken to be polynomial), and we abbreviated\nfk(ρ)=\n(\n− 1√\nN′\nN\n∑\nl=1\nρl\n)k\n+\nN\n∑\nn=1\n(\n√\nN′ρn − 1√\nN′\nN\n∑\nl=1\nρl\n)k\n. (11.22)\nNote that f1(ρ)= 0, so that to lowest order (i.e. k = 2) we have\nhAE (Q,ρ)=\n(\n1\n2 N\nN\n∑\nn=1\nρ2\nn −\nN\n∑\nk̸=l\nρkρl\n)\nV ′′(Q)+ ··· (11.23)\nWe pass to the corresponding quantum-mechanical Hamiltonians in the usual way,\nand couple a two-level quantum system to the apparatus through the Hamiltonian\nhSA = μ·σ3 ⊗P, (11.24)\nwhere the object observable s = σ3, acting on HS = C2, is to be measured. The idea\nis that hA is the Hamiltonian of a pointer that registers outcomes by localization on\nthe real line, hE is the (free) Hamiltonian of the “environment”, realized as the in-\nternal degrees of the freedom of the total apparatus that are not used in recording\nthe outcome of the measurement, and hAE describes the pointer-environment inter-\naction. The classical description of the apparatus then involves two approximations:\n• Ignoring all degrees of freedom except those of A, which classically are (P,Q);\n• Taking the classical limit of hA, here realized as N →∞ (in lieu of ¯h →0).\nThe measurement ofs is now expected to unfold according to the following scenario:\n1. The apparatus is initially in a metastable state (this is a very common assump-\ntion), whose wave-function is e.g. a Gaussian centered at the origin.\n2. If the object state is “spin up”, i.e., ψS =( 1,0), then it kicks the pointer to the\nright, where it comes to a standstill at the bottom of the double well. If spin is\ndown, likewise to the left. IfψS =( 1,1)/\n√\n2, the pointer moves to a superposition\nof these, which is close to the ground state of V displayed in Figure 11.2.\n3. In the last case, the Flea mechanism of §10.2 comes into play: tiny asymmetric\nperturbations irrelevant for small N localize the ground state as N →∞.\n4. Mere localization of the ground state of the perturbed (apparatus) Hamiltonian in\nthe classical regime is not enough: there should be a dynamical transition from\nthe ground state of the original (unperturbed) Hamiltonian (which has become\nmetastable upon perturbation) to the ground state of the perturbed one. This dy-\nnamical mechanism in question should also recover the Born rule.\nThus the classical description of the apparatus is at the same time the root of the\nmeasurement problem and the key to its solution: it creates the problem because at\nfirst sight a Schr¨odinger Cat state has the wrong classical limit (namely a mixture),\nbut it also solves it, because precisely in the classical limit Cat states are destabilized\neven by the tiniest (asymmetric) perturbations and collapse to the “right” states.',
    'tration,p.115).Thebaby\'sparentshavealreadyattainedtheirfullgrowth.Theamountoffoodthatapersonneedstomakeupfortheheatradiatedfromthesurfaceofthebodyvariesw^iththesizeandalsow^iththeshapeofthebody(seeillustrationopposite).Thesmallerachild,themoresurfacehehasinproportiontohisbodyweight,andhencehelosesrelativelymoreheat.Byactualmeasurement,aone-year-oldchildneedsapproximatelytwiceasmuchenergyperpoundofbody-weightasdoesanadult.Energyneedsareindirectlyrelatedtosex.Girlsandwomenhaveathickerlayeroffattytissuebeneaththeskinthanboysandmen.Thisfatpreventsrapidradiationofheatfromthebody.Itisinterestingtorecallthatmostlong-distanceswimmingrecordsareheldbywomenratherthanbymen.Exposurealsoaffectsthebody\'slossofheat.Thebodylosesheatfasterinacold,dry,windyclimatethaninawarm,moistclimate.Cloth-ingandshelterare,ofcourse,factorsinthelossofheat.Circulationoftheblood,breathing,andotherprocessesarecontinuallygoingonwhenthebodyisatrest."Warm-blooded"animalsmaintainaconstanttemperature.Theheatcontinuallyradiatingfromthesurfaceisconstantlybeingreplaced.Muscularmovementsarecontinuallytakingplaceinthedigestiveorgans,andenergyisusedinvariousotherwayswithinthebody.From40to50percentofthebodyismadeupofmus-culartissue.Thebulkofthistissueisattachedtotheskeletonandisusedinstandingaswellasinlocomotionandothervoluntaryactions.Atalltimes,evenwhenthesemusclesarerelaxed,energyisusedinkeepingthemsomewhatonthestretch.AbovetheBaseLineTheamountofenergythatthebodyuses,evenwhileitis"doingnothing",isconstantlyinfluencedbytwosetsoffactors.Digestingfoodinvolvesameasurableamountofenergy.Thusthebodyusesabout6percentmoreenergysoonafterameal,whenthedigestiveorgansaremostactive,thanjustbeforeameal,whendigestionispracti-callyatastandstill.Whenyouaresittingandreading,orwhenyouarestandingquietly,yourbodyusesaboutone-and-a-thirdtimesasmuchenergyasitdoeswhilesleeping.Walkingatamoderatepaceusesabouttwo-and-a-halftimesasmuch;runningusesaboutseventimesandstair-climbingaboutfifteentimesasmuch.UnitofEnergyTomeasuretheenergyexpendedbythelivingbody,weuseaunitdevelopedbyengineers.ThisistheCalorie(Cal),and,likethemorefamiliarfoot-pound(ft-lb)usedinmeasuringwork,itiscomposedoftwofactors.Wemeasureworkasifitalwaysconsistedofsomequantityofmatter(pounds)movingacertaindistance(feet).Inasimilarwaywemeasureheatasaquantityofmatter,forexample,1kilogram(kg)ofwater,beingheatedacertain"distance"(1degreeonthecentigradescale).116',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Binary Classification

• Evaluated with BinaryClassificationEvaluator

Metric	Value
cosine_accuracy	0.7932
cosine_accuracy_threshold	0.8014
cosine_f1	0.6447
cosine_f1_threshold	0.7684
cosine_precision	0.6414
cosine_recall	0.648
cosine_ap	0.7217
cosine_mcc	0.4692

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,124,250 training samples
• Columns: sentence1, sentence2, and label
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	label
  type	string	string	int
  details	min: 127 tokens mean: 127.0 tokens max: 127 tokens	min: 22 tokens mean: 421.72 tokens max: 512 tokens	0: ~50.10% 1: ~49.90%

• Samples:

  sentence1	sentence2	label
  mason-wasp is solitary. Thus the process of organic evolution is far from being fully understood. We can only suppose that as there are devised by human beings many puzzles apparently unanswerable till the answer is given, and many necromantic tricks which seem impossible till the mode of performance is shown; so there are apparently incomprehensible results which are really achieved by natural processes. Or, otherwise, we must conclude that since Life itself proves to be in its ultimate nature inconceivable, there is probably an inconceivable element in its ultimate workings. END OF VOL. I.	difficulties which my own hypothesis avoids? If, as I have argued, the germ- plasm consists of substantially similar units (having only those minute differences expressive of individual and ancestral differences of structure), none of the complicated requirements which Dr. Romanes emphasizes exist; and the alleged inconceivability disappears. Here I must end: not intending to say more, unless for some very urgent reason; and leaving others to carry on the discussion. I have, indeed, been led to suspend for a short time my proper work, only by consciousness of the transcendent importance of the question at issue. As I have before contended, a right answer to the question whether acquired characters are or are not inherited, underlies right beliefs, not only in Biology and Psychology, but also in Education, Ethics, and Politics. III. As a species of literature, controversy is characterised by a terrible fertility. Each proposition becomes the parent of half a dozen; so that a few replie...	1
  mason-wasp is solitary. Thus the process of organic evolution is far from being fully understood. We can only suppose that as there are devised by human beings many puzzles apparently unanswerable till the answer is given, and many necromantic tricks which seem impossible till the mode of performance is shown; so there are apparently incomprehensible results which are really achieved by natural processes. Or, otherwise, we must conclude that since Life itself proves to be in its ultimate nature inconceivable, there is probably an inconceivable element in its ultimate workings. END OF VOL. I.	RNA Editing in Trypanosomes The trypanosomes are a group of protozoa that include the pathogenTrypanosoma brucei, which causes sleeping sickness in humans (Figure 15.12). Trypanosomes, and virtually all other eukaryotes, have organelles called mitochondria that supply the cell with chemical energy. Mitochondria are organelles that express their own DNA and are believed to be the remnants of a symbiotic relationship between a eukaryote and an engulfed prokaryote. The mitochondrial DNA of trypanosomes exhibit an interesting exception to The Central Dogma: their pre-mRNAs do not have the correct information to specify a functional protein. Usually, this is because the mRNA is missing several U nucleotides. The cell performs an additional RNA processing step calledRNA editing to remedy this. Figure 15.12 Trypanosoma brucei is the causative agent of sleeping sickness in humans. The mRNAs of this pathogen must be modified by the addition of nucleotides before protein synthesis can occur. (cr...	1
  mason-wasp is solitary. Thus the process of organic evolution is far from being fully understood. We can only suppose that as there are devised by human beings many puzzles apparently unanswerable till the answer is given, and many necromantic tricks which seem impossible till the mode of performance is shown; so there are apparently incomprehensible results which are really achieved by natural processes. Or, otherwise, we must conclude that since Life itself proves to be in its ultimate nature inconceivable, there is probably an inconceivable element in its ultimate workings. END OF VOL. I.	plane subtends with the axis an angle of 89° 59′, we have an ellipse which no human eye, even when aided by an accurate pair of compasses, can distinguish from a circle. Decreasing the angle minute by minute, this closed curve becomes perceptibly eccentric, then manifestly so, and by and by acquires so immensely elongated a form so as to bear no recognizable resemblance to a circle. By continuing this process the ellipse changes insensibly into a parabola. On still further diminishing the angle, the parabola becomes an hyperbola. And finally, if the cone be made gradually more obtuse, the hyperbola passes into a straight line as the angle of the cone approaches 180°. Here then we have five different species of line—circle, ellipse, parabola, hyperbola, and straight line—each having its peculiar properties and its separate equation, and the first and last of which are quite opposite in nature, connected together as members of one series, all producible by a single process of insensible...	1

• Loss: CoSENTLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "pairwise_cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• per_device_eval_batch_size: 32
• learning_rate: 2e-05
• weight_decay: 0.01
• max_steps: 2000

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3.0
• max_steps: 2000
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	cosine_ap
-1	-1	-	0.7217
0.0071	500	0.4089	-
0.0142	1000	0.007	-
0.0213	1500	0.0	-
0.0285	2000	0.0	-

Framework Versions

• Python: 3.12.9
• Sentence Transformers: 4.1.0
• Transformers: 4.53.0
• PyTorch: 2.7.1
• Accelerate: 1.8.1
• Datasets: 3.6.0
• Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CoSENTLoss

@online{kexuefm-8847,
    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
    author={Su Jianlin},
    year={2022},
    month={Jan},
    url={https://kexue.fm/archives/8847},
}