Wakeleyhuh/chowder-embed-specter

chowder-embed (SPECTER fine-tuned on the chowder corpus)

This is a sentence-transformers model finetuned from sentence-transformers/allenai-specter on the parquet dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/allenai-specter
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • parquet
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'cls', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Wakeleyhuh/chowder-embed-specter")
# Run inference
queries = [
    'ERBB3 blockade sensitizes hepatocellular carcinoma to regorafenib after first-line tyrosine kinase inhibitor resistance by inhibiting HIF1A-ABCB1 signaling',
]
documents = [
    'BACKGROUND/AIMS: Regorafenib is recommended by guidelines and trials as a sequential second-line therapy following progression on first-line sorafenib or lenvatinib in hepatocellular carcinoma (HCC). However, efficacy is limited, highlighting the urgent need to screen suitable patients and develop sensitization strategies. METHODS: Acquired sorafenib- or lenvatinib-resistant (SR or LR) HCC cell lines and organoids were established. Genome-wide CRISPR library screen was performed in SR or LR cell strains to identify synthetic lethal targets of regorafenib. RNA-seq and FITC-regorafenib efflux assay were used to elucidate ERBB3-driven downstream signaling. Preclinical mouse models of cell line- and patient-derived xenografts and clinical cohorts of HCC patients were employed to validate the efficacy of ERBB3-guided patient stratification. RESULTS: Screening with CRISPR library, we showed that inhibition of ERBB3 was synthetic lethal with regorafenib in SR or LR cell strains and organoids. Mechanistically, SR or LR triggered feedback activation of ERBB3 signaling and mediated regorafenib efflux via ERBB3-HIF1A-ABCB1 cascade pathway, limiting sensitivity to regorafenib. Moreover, ERBB3-low tumors following SR or LR exhibited significant sensitivity to regorafenib, suggesting its potential as a predictive biomarker to screen optimal candidates for sequential therapy. Seribantumab, an ERBB3-targeting monoclonal antibody, inhibited ERBB3-HIF1A-ABCB1 cascade, and its combination with regorafenib exerted marked synergistic anti-tumor effects on ERBB3-high tumors resistant to sorafenib or lenvatinib both in vitro and in vivo. CONCLUSIONS: This study revealed that ERBB3 was a key resistance factor driving limited efficacy to sequential regorafenib, but also an effective therapeutic target whose inhibition enhanced regorafenib sensitivity after SR or LR.',
    'We improve the theoretical predictions for rare $Z$-boson decays, $Z\\to V\\ell^+\\ell^-$ ($\\ell=e$ or $\\mu$), where $V$ denotes a heavy vector quarkonium including $J/\\Psi$, $\\Psi (2S)$, and $\\Upsilon (nS)$ with $n=1,2,3$. These processes are thought to be dominated by the electromagnetic fragmentation transition, i.e., $Z\\to \\gamma^*\\ell^+\\ell^-$ followed by $\\gamma^*\\rightarrow V$. The present study includes all of the relevant tree-level Feynman diagrams, which contribute to these decays in the standard model. Our analysis shows that, for the charmonium final states, the fragmentation transition almost saturates the whole contribution and the other diagrams can be neglected; while for the bottomonium final states, the inclusion of other diagrams can increase their branching fractions by $4\\%\\sim 9\\%$. Further investigation of the differential distributions, especially the angular distributions, indicates that forward-backward asymmetries for final leptons in these processes would be zero in the standard model. Therefore, in future experimental facilities with large number of $Z$-boson events accumulated, studies of these rare $Z$ decays may help both to test the standard model and to probe its interesting extensions.',
    'Historically, veterinary studies screening for breed, age and sex predisposition to disease have relied on collating small-scale studies of clinical datasets. The availability of larger datasets through groups such as the Small Animal Veterinary Surveillance Network (SAVSNET) promise access to information regarding a wide range of clinical presentations at scale, however, methodological limitations surrounding the extraction of specific disease information or screening for disease predispositions result in a substantial reduction in the number of animals studied. These studies often address very focused hypotheses - only leveraging a small fraction of the intrinsic value of the data at any one time. Here, we implemented an unsupervised machine learning methodology, creating a representation of a large volume of clinical notes collected by SAVSNET from veterinary practices across the UK. We utilise BERTopic, a topic-modelling tool based on Bidirectional Encoder Representations using Transformers (BERT) architecture, and show it is able to surface known phenotypes, such as breed predispositions to hypoadrenocorticism, diabetes mellitus and mitral valve disease, as well as potential novel patterns of disease phenotypes. This scalable and granular modelling technique facilitates the rapid interrogation of large clinical datasets, enabling the identification of a broad range of phenotypes within the population and the early detection of temporal changes indicative of emerging infectious or environmental diseases. Supplementary Information: The online version contains supplementary material available at 10.1186/s40537-026-01365-0.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.8640, 0.0814, 0.0283]])

Evaluation

Metrics

Information Retrieval

• Datasets: NanoSCIDOCS, NanoSciFact, NanoNFCorpus and chowder-val
• Evaluated with InformationRetrievalEvaluator

Metric	NanoSCIDOCS	NanoSciFact	NanoNFCorpus	chowder-val
cosine_accuracy@1	0.38	0.6	0.3	0.952
cosine_accuracy@3	0.62	0.68	0.36	0.9805
cosine_accuracy@5	0.72	0.72	0.46	0.989
cosine_accuracy@10	0.86	0.8	0.54	0.9955
cosine_precision@1	0.38	0.6	0.3	0.952
cosine_precision@3	0.3133	0.2467	0.2533	0.3268
cosine_precision@5	0.256	0.16	0.248	0.1978
cosine_precision@10	0.176	0.09	0.222	0.0996
cosine_recall@1	0.0807	0.565	0.0125	0.952
cosine_recall@3	0.1947	0.67	0.0309	0.9805
cosine_recall@5	0.2627	0.715	0.0474	0.989
cosine_recall@10	0.3617	0.79	0.1022	0.9955
cosine_ndcg@10	0.3456	0.6831	0.245	0.9743
cosine_mrr@10	0.5344	0.6557	0.3606	0.9674
cosine_map@100	0.2648	0.6497	0.0919	0.9676

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "scidocs",
          "scifact",
          "nfcorpus"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en"
  }
  

Metric	Value
cosine_accuracy@1	0.4267
cosine_accuracy@3	0.5533
cosine_accuracy@5	0.6333
cosine_accuracy@10	0.7333
cosine_precision@1	0.4267
cosine_precision@3	0.2711
cosine_precision@5	0.2213
cosine_precision@10	0.1627
cosine_recall@1	0.2194
cosine_recall@3	0.2985
cosine_recall@5	0.3417
cosine_recall@10	0.4179
cosine_ndcg@10	0.4246
cosine_mrr@10	0.5169
cosine_map@100	0.3354

Training Details

Training Dataset

parquet

• Dataset: parquet
• Size: 70,764 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 8 tokens mean: 18.96 tokens max: 51 tokens	min: 54 tokens mean: 276.39 tokens max: 512 tokens

• Samples:

  anchor	positive
  Los-mamba: A low-rank recursive mamba framework for mitigating stationary bias in trajectory prediction	Trajectory prediction plays a key role in autonomous driving and intelligent transportation systems. Mamba performs well in modeling long sequences but struggles with short-term static or local motion features. In this paper, we propose the Lo w-rank Recur s ive Mamba framework ( Los-Mamba ). It addresses prediction bias caused by inadequate modeling of local static features in trajectory prediction tasks. Firstly, Los-Mamba uses hybrid Mamba branches to capture both global motion trends and local features. Secondly, it reduces trajectory feature dimensionality through low-rank approximation, simplifying computational complexity. Thirdly, Los-Mamba also includes a spatio-temporal constraint mechanism. It uses position coding and a multilayer perceptron (MLP) to interpolate static and dynamic trajectories, improving the representation of local features. Experiments show that Los-Mamba outperforms baseline models on datasets like ETH and Hotel. Its average displacement error (ADE) and fi...
  The Universal Language of CSI:Unifying Wireless Sensing Across Devices and Environments	WiFi sensing based on Channel State Information (CSI) promises ubiquitous, device-free perception, yet current research remains trapped in a Tower of Babel - fragmented into isolated silos where models are tailored to specific hardware dialects, fixed environments, and narrow tasks. The primary bottleneck is the Heterogeneity Gap: the disparity in signal dimensions, sampling rates, and semantic labels that prevents cross-system understanding. To bridge this gap, we propose a foundation-model framework that treats CSI not merely as raw signals but as a structured language with a learnable universal grammar. We first curate and standardize a large collection of heterogeneous real-world CSI datasets, establishing a unified infrastructure that allows incompatible signal formats to be treated as a single corpus. Second, we introduce a modular architecture that acts as a universal translator where lightweight dataset-specific adapters tokenize diverse signal inputs into a shared latent vocab...
  Long-term follow-up of the transplant-eligible cohort of the EMN12/HOVON-129 study for primary plasma cell leukemia patients	Primary plasma cell leukemia (pPCL) is a rare and aggressive plasma cell disorder [ 1 , 2 ]. Compared to multiple myeloma (MM), pPCL is characterized by a higher frequency of poor-risk features, such as high-risk cytogenetic abnormalities (HRCAs) and elevated LDH [ 1 , 2 ]. Although MM treatment has improved, the survival of pPCL patients remains poor [ 1 , 2 ]. We therefore initiated the phase 2 EMN12/HOVON-129 study for patients with newly diagnosed pPCL, which showed that carfilzomib and lenalidomide-based therapy in conjunction with autologous stem-cell-transplantation (ASCT) provides efficient disease control [ 3 ]. Here, we present the long-term follow-up of patients aged 18–65 years included in the EMN12/HOVON-129 study. We also aimed to evaluate if patients with HRCAs, elevated LDH, or other high-risk features have a worse clinical outcome. Inclusion criteria were newly diagnosed pPCL (defined as >2 × 10 9 /L circulating plasma cells or >20% plasma cells in peripheral blood) an...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

parquet

• Dataset: parquet
• Size: 2,000 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 7 tokens mean: 19.21 tokens max: 37 tokens	min: 68 tokens mean: 278.03 tokens max: 512 tokens

• Samples:

  anchor	positive
  Boolean Algebra -- Driven Sepsis Diagnosis	Sepsis remains a diagnostic challenge due to its heterogeneous molecular signatures and complex immune responses. In this study, we develop a logical data analysis framework based on Boolean polynomial rings. This method constructs an ideal $\mathcal{I}$ of selection criteria that isolate empty subsets of previously analyzed patient data. This approach enables the derivation of interpretable classification rules based on biomarker profiles. We demonstrate that logical data analysis identifies distinct logical patterns for positive and negative sepsis classification. For instance, elevated levels of GLP-1 and MyD88 are associated with septic states in our dataset, whereas high TRAIL and low MyD88 concentrations may suggest a non-septic condition. Importantly, a new way to integrate expert knowledge to filter out potential overfitting or dataset-specific artifacts is shown. Our findings highlight the utility of logics in generating transparent, biologically plausible rules for a data-bas...
  Notes on the bootstrap of four-point conformal integrals	We set up a bootstrap workflow to study four-point conformal integrals in position space, using leading singularities, single-valued multiple polylogarithmic ans"atze and boundary data from expansion by regions. These four-point conformal integrals are general in the sense that they are generated by the four-point projections of all possible $f$-graphs, including all non-planar $f$-graph sectors. For three-loop cases, fourteen of the fifteen inequivalent integrand basis can be directly calculated by \texttt{HyperlogProcedures} and the last one is fixed by Gram identity. Then we concentrate on how far the bootstrap workflow can go for four-loop cases, though it works for three-loop cases as well. We show that integrals with several leading singularities can be made tractable by decomposing them into pieces with simpler cut structure. Some four-loop integrals which can not be calculated or very hard to be calculated by other methods for now are obtained in this way. We also provide a pa...
  Clean2FX: Label-conditioned modeling for clean-to-effect guitar audio transformations	We present Clean2FX, a study and demo of label-conditioned clean-to-effect transformation for electric guitar audio. Given a clean guitar input and a target effect label, the task is to synthesize the corresponding effected signal while preserving the musical content. Training and evaluation pairs are constructed from EGFxSet real, single tone recordings by assembling matched clean/effected chords, melodies, and mixed timelines. This allows for controlled comparison across effects. We evaluate four neural approaches under a common spectrogram-based transformation setting: two variational autoencoders and two U-Net models that differ in whether they operate on linear or log-magnitude representations. Performance is measured using linear-magnitude spectrogram MSE and Fr'echet Audio Distance. The U-Net models outperform the variational autoencoder variants. Per-effect results show that distortion effects are most readily improved, whereas delay and reverb effects exhibit weaker FAD gains...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• learning_rate: 2e-05
• warmup_steps: 0.1
• weight_decay: 0.01
• fp16: True
• per_device_eval_batch_size: 16
• load_best_model_at_end: True
• seed: 12
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 16
• num_train_epochs: 3
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 16
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 12
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

Training Logs

Epoch	Step	Training Loss	Validation Loss	NanoSCIDOCS_cosine_ndcg@10	NanoSciFact_cosine_ndcg@10	NanoNFCorpus_cosine_ndcg@10	NanoBEIR_mean_cosine_ndcg@10	chowder-val_cosine_ndcg@10
-1	-1	-	-	0.3211	0.6251	0.1796	0.3753	0.8622
0.0002	1	0.1790	-	-	-	-	-	-
0.0301	133	0.1970	-	-	-	-	-	-
0.0601	266	0.0910	-	-	-	-	-	-
0.0902	399	0.0571	-	-	-	-	-	-
0.1203	532	0.0445	-	-	-	-	-	-
0.1504	665	0.0331	-	-	-	-	-	-
0.1804	798	0.0311	-	-	-	-	-	-
0.2105	931	0.0304	-	-	-	-	-	-
0.2406	1064	0.0234	-	-	-	-	-	-
0.2706	1197	0.0229	-	-	-	-	-	-
0.3000	1327	-	0.0102	0.3527	0.6570	0.2448	0.4182	-
0.3007	1330	0.0225	-	-	-	-	-	-
0.3308	1463	0.0199	-	-	-	-	-	-
0.3608	1596	0.0220	-	-	-	-	-	-
0.3909	1729	0.0143	-	-	-	-	-	-
0.4210	1862	0.0186	-	-	-	-	-	-
0.4511	1995	0.0136	-	-	-	-	-	-
0.4811	2128	0.0191	-	-	-	-	-	-
0.5112	2261	0.0151	-	-	-	-	-	-
0.5413	2394	0.0220	-	-	-	-	-	-
0.5713	2527	0.0170	-	-	-	-	-	-
0.6	2654	-	0.0069	0.3455	0.6831	0.245	0.4245	-
0.6014	2660	0.0150	-	-	-	-	-	-
0.6315	2793	0.0174	-	-	-	-	-	-
0.6615	2926	0.0123	-	-	-	-	-	-
0.6916	3059	0.0178	-	-	-	-	-	-
0.7217	3192	0.0182	-	-	-	-	-	-
0.7518	3325	0.0151	-	-	-	-	-	-
0.7818	3458	0.0143	-	-	-	-	-	-
0.8119	3591	0.0124	-	-	-	-	-	-
0.8420	3724	0.0165	-	-	-	-	-	-
0.8720	3857	0.0134	-	-	-	-	-	-
0.9001	3981	-	0.0047	0.3545	0.6515	0.2247	0.4102	-
0.9021	3990	0.0119	-	-	-	-	-	-
0.9322	4123	0.0156	-	-	-	-	-	-
0.9622	4256	0.0127	-	-	-	-	-	-
0.9923	4389	0.0154	-	-	-	-	-	-
1.0224	4522	0.0091	-	-	-	-	-	-
1.0525	4655	0.0034	-	-	-	-	-	-
1.0825	4788	0.0065	-	-	-	-	-	-
1.1126	4921	0.0072	-	-	-	-	-	-
1.1427	5054	0.0041	-	-	-	-	-	-
1.1727	5187	0.0065	-	-	-	-	-	-
1.2001	5308	-	0.0032	0.3672	0.6294	0.2201	0.4055	-
1.2028	5320	0.0055	-	-	-	-	-	-
1.2329	5453	0.0038	-	-	-	-	-	-
1.2629	5586	0.0043	-	-	-	-	-	-
1.2930	5719	0.0040	-	-	-	-	-	-
1.3231	5852	0.0038	-	-	-	-	-	-
1.3532	5985	0.0069	-	-	-	-	-	-
1.3832	6118	0.0063	-	-	-	-	-	-
1.4133	6251	0.0052	-	-	-	-	-	-
1.4434	6384	0.0068	-	-	-	-	-	-
1.4734	6517	0.0047	-	-	-	-	-	-
1.5001	6635	-	0.0032	0.3589	0.6404	0.1966	0.3986	-
-1	-1	-	-	0.3456	0.6831	0.2450	0.4246	0.9743

• The bold row denotes the saved checkpoint.

Training Time

• Training: 44.8 minutes
• Evaluation: 3.9 minutes
• Total: 48.7 minutes

Framework Versions

• Python: 3.14.7
• Sentence Transformers: 5.7.0
• Transformers: 5.15.0
• PyTorch: 2.13.0+cu130
• Accelerate: 1.14.0
• Datasets: 5.0.1
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}