File size: 16,742 Bytes
4fd4743 9da3822 6991a45 989191e 6991a45 989191e 6991a45 4fd4743 989191e 4fd4743 f39747b 989191e 4fd4743 989191e f39747b 989191e 4fd4743 f39747b 4fd4743 989191e 4fd4743 989191e 4fd4743 989191e f39747b 989191e 4fd4743 989191e 4fd4743 f39747b 4fd4743 989191e 4fd4743 989191e 4fd4743 989191e f39747b 989191e 4fd4743 989191e 4fd4743 6991a45 989191e 4fd4743 989191e 4fd4743 6991a45 f39747b e610f37 6991a45 4fd4743 f39747b 4fd4743 f39747b 4fd4743 989191e 4fd4743 989191e 4fd4743 f39747b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 |
---
license: mit
language:
- en
widget:
- source_sentence: 'Ventral humeral ridge: or not'
sentences:
- >-
If metasternum ossified, shape: long, narrow and tapering markedly
anteriorly to posteriorly, length up to 3.5 times maximum width
- >-
Astragalus, dorsolateral margin:: overlaps the anterior and posterior
portions of the calcaneum equally
- 'Ulna size: does not apply'
- source_sentence: >-
Form of distal portion of anteroventral process of ectopterygoid: varyingly
falcate
sentences:
- 'Middle and distal radials in dorsal and anal fins: absent'
- >-
Degree of development of primitively medial portion of fourth upper
pharyngeal tooth-plate: fourth upper pharyngeal tooth-plate covers ventral,
posterior, dorsal and sometimes anterior surfaces of fourth
infrapharyngobranchial
- 'Shape of pharyngeal apophysis (basioccipital): forked anteriorly'
- source_sentence: >-
Form of distal portion of anteroventral process of ectopterygoid: varyingly
falcate
sentences:
- 'parhypural: present'
- 'Epural: heavy'
- 'First infraorbital: short'
- source_sentence: >-
Form of distal portion of anteroventral process of ectopterygoid: varyingly
falcate
sentences:
- 'Dentary and angular: touch'
- 'Urohyal and first basibranchial: firmly attached'
- 'Supraneural 3-4 (nonadditive): absent'
- source_sentence: >-
Form of distal portion of anteroventral process of ectopterygoid: varyingly
falcate
sentences:
- 'Ventral diverging lamellae of mesethmoid: lamellae reduced or absent'
- 'Ventral ridge of the coracoid with a posterior process: absent'
- 'carpals: fully or partially ossified'
pipeline_tag: sentence-similarity
library_name: sentence-transformers
base_model: sentence-transformers/all-mpnet-base-v2
metrics:
- pearson_cosine
- spearman_cosine
model-index:
- name: SentenceTransformer based on sentence-transformers/all-mpnet-base-v2
results:
- task:
type: semantic-similarity
name: Semantic Similarity
dataset:
name: pheno dev
type: pheno-dev
metrics:
- type: pearson_cosine
value: 0.6082332469417436
name: Pearson Cosine
- type: spearman_cosine
value: 0.6250387873495056
name: Spearman Cosine
- task:
type: semantic-similarity
name: Semantic Similarity
dataset:
name: pheno test
type: pheno-test
metrics:
- type: pearson_cosine
value: 0.6822053314599665
name: Pearson Cosine
- type: spearman_cosine
value: 0.705688010939619
name: Spearman Cosine
tags:
- ontology
- nlp
- biology
- animals
- fish
- embedding
- trait
- sentence-transformers
- sentence-similarity
- feature-extraction
- loss:CoSENTLoss
datasets:
- imageomics/char-sim-data
model_name: Trait2Vec
model_description: "Language model for embedding organismal trait descriptions. Built using Sentence-Transformer architecture and trained with trait descriptions from Imageomics/char-sim-data."
---
# Model Card for Trait2Vec
Trait2Vec is a language model to embed organismal trait descriptions in a way that preserves the structure induced by a semantic similarity metric (e.g. SimGIC). The model was trained on the [Character Similarity Dataset](https://huggingface.co/datasets/imageomics/char-sim-data). It is fine-tuned from [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2).
Through qualitative data exploration we observe the cosine similarity between embeddings of raw trait description is proportional to the semantic similarity of their corresponding ontological representations.
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** Juan Garcia, Soumyashree Kar, Jim Balhoff, Hilmar Lapp
- **Model type:** Sentence Transformer
- **Language(s) (NLP):** English
- **License:** MIT
- **Fine-tuned from model:** [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)
### Model Sources
- **Repository:** [Imageomics/char-sim](https://github.com/Imageomics/char-sim/)
## Uses
Trait2Vec has been qualitatively evaluated in the ability to embed raw trait descriptions in a way that preserves the structure of an ontology. Accordingly, we expect it to produce an alternative computational representation of the traits of an organism.
### Direct Use
It can be used to embed the textual trait descriptions associated with an organism.
## Bias, Risks, and Limitations
This model is finetuned from [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2), therefore it inherits its corresponding biases and risks. The training dataset([Character Similarity Dataset](https://huggingface.co/datasets/imageomics/char-sim-data)) introduces the biases of the single similarity metric and ontology. This means the embedding inherits that metric’s inductive biases, coverage gaps, and evolving definitions. Biological conclusions may differ under alternative metrics (e.g., Resnik, Jaccard) or other phenotype ontologies.
### Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
First install the Sentence Transformers library:
```bash
pip install -U sentence-transformers
```
Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("imageomics/trait2vec")
# Run inference
sentences = [
'Form of distal portion of anteroventral process of ectopterygoid: varyingly falcate',
'Ventral ridge of the coracoid with a posterior process: absent',
'carpals: fully or partially ossified',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 256]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```
## Training Details
### Training Data
This model was trained on the [Character Similarity Dataset](https://huggingface.co/datasets/imageomics/char-sim-data).
* Size: 438,516 training samples
* Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
* Approximate statistics based on the first 1000 samples:
| | sentence1 | sentence2 | score |
|:--------|:-----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:---------------------------------------------------------------|
| type | string | string | float |
| details | <ul><li>min: 9 tokens</li><li>mean: 42.84 tokens</li><li>max: 164 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 22.8 tokens</li><li>max: 164 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.1</li><li>max: 0.61</li></ul> |
* Samples:
| sentence1 | sentence2 | score |
|:-----------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------|
| <code>Gill raker shape between ceratobranchial 1 and ceratobranchials 2--4: Homomorphic</code> | <code>Extent of development of inferior lamella of lateral ethmoid: inferior lamella absent</code> | <code>0.014706667500582846</code> |
| <code>Gill raker shape between ceratobranchial 1 and ceratobranchials 2--4: Homomorphic</code> | <code>Shape of anal-fin pterygiophore tips: tips of pterygiophores shaped like an arrow-head; axial series of pterygiophores providing the ventral margin of the anal-fin base a scalloped appearance</code> | <code>0.030538703023734296</code> |
| <code>Gill raker shape between ceratobranchial 1 and ceratobranchials 2--4: Homomorphic</code> | <code>Suprapreopercle: present</code> | <code>0.3385057414877959</code> |
* Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
```json
{
"scale": 20.0,
"similarity_fct": "pairwise_cos_sim"
}
```
#### Training Hyperparameters
#### Non-Default Hyperparameters
- `eval_strategy`: steps
- `per_device_train_batch_size`: 64
- `per_device_eval_batch_size`: 64
- `learning_rate`: 2e-05
- `num_train_epochs`: 10
- `warmup_ratio`: 1e-06
- **Training regime:** fp32
## Evaluation
We tested Trait2Vec on a hold-out split of 20\% of the [Character Similarity Dataset](https://huggingface.co/datasets/imageomics/char-sim-data/). No descriptor overlap was ensured.
### Testing Data, Factors & Metrics
#### Testing Data
* Size: 111,628 evaluation samples
* Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
* Approximate statistics based on the first 1000 samples:
| | sentence1 | sentence2 | score |
|:--------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:---------------------------------------------------------------|
| type | string | string | float |
| details | <ul><li>min: 9 tokens</li><li>mean: 17.19 tokens</li><li>max: 24 tokens</li></ul> | <ul><li>min: 3 tokens</li><li>mean: 21.97 tokens</li><li>max: 143 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.1</li><li>max: 0.86</li></ul> |
* Samples:
| sentence1 | sentence2 | score |
|:-------------------------------------------|:------------------------------------------------------------------------------------------------------------------|:----------------------------------|
| <code>Ventral humeral ridge: or not</code> | <code>Metacarpals, Metacarpal I, presence: absent</code> | <code>0.05558851078197206</code> |
| <code>Ventral humeral ridge: or not</code> | <code>Metapterygoid–quadrate fenestra: absent</code> | <code>0.004860625129173212</code> |
| <code>Ventral humeral ridge: or not</code> | <code>Dorsal and ventral borders of the maxillary articular process: straight or slightly curved ventrally</code> | <code>0.10380567059620477</code> |
* Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
```json
{
"scale": 20.0,
"similarity_fct": "pairwise_cos_sim"
}
```
#### Metrics
**Semantic Similarity:**
* Datasets: `pheno-dev` and `pheno-test`
* Evaluated with [<code>EmbeddingSimilarityEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)
### Results
| Metric | Validation set | Test set |
|:--------------------|:----------|:-----------|
| pearson_cosine | 0.6082 | 0.6822 |
| **spearman_cosine** | **0.625** | **0.7057** |
#### Summary
Trait2Vec embeds organismal trait descriptors in a way that preserves some of the ranking structure induced by the similarity metric of the ontology.
## Environmental Impact
Experiments were conducted using a private infrastructure, which has a carbon efficiency of 0.432 kgCO$_2$eq/kWh. A cumulative of 20 hours of computation was performed on hardware of type A100 PCIe 40/80GB (TDP of 250W).
Total emissions are estimated to be 2.16 kgCO$_2$eq of which 0 percents were directly offset.
Estimations were conducted using the [MachineLearning Impact calculator](https://mlco2.github.io/impact#compute) presented in:
```bibtex
@article{lacoste2019quantifying,
title={Quantifying the Carbon Emissions of Machine Learning},
author={Lacoste, Alexandre and Luccioni, Alexandra and Schmidt, Victor and Dandres, Thomas},
journal={arXiv preprint arXiv:1910.09700},
year={2019}
}
```
### Model Architecture and Objective
```
SentenceTransformer(
(0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: MPNetModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Dense({'in_features': 768, 'out_features': 256, 'bias': True, 'activation_function': 'torch.nn.modules.activation.Tanh'})
)
```
* Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
```json
{
"scale": 20.0,
"similarity_fct": "pairwise_cos_sim"
}
```
#### Software
- Python: 3.10.16
- Sentence Transformers: 3.3.1
- Transformers: 4.48.1
- PyTorch: 2.5.1.post303
- Accelerate: 1.3.0
- Datasets: 2.14.4
- Tokenizers: 0.21.0
## Citation
If you use this model in your research, please cite both it and the source model & method from which it was fine-tuned:
### Model
```bibtex
@software{trait2vec2025,
author = {Juan Garcia and Soumyashree Kar and Jim Balhoff and Hilmar Lapp},
doi = {10.57967/hf/6892},
title = {Trait2Vec (Revision f39747b)},
version = {1.0.0},
year = {2025},
url = {https://huggingface.co/imageomics/trait2vec}
}
```
### Source Model & Method
#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
pages = "3982-3992",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/D19-1410/",
doi = "10.18653/v1/D19-1410"
}
```
#### CoSENTLoss
```bibtex
@online{kexuefm-8847,
title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
author={Su Jianlin},
year={2022},
month={Jan},
url={https://kexue.fm/archives/8847},
}
```
## Acknowledgements
This work was supported by the [Imageomics Institute](https://imageomics.org), which is funded by the US National Science Foundation's Harnessing the Data Revolution (HDR) program under [Award #2118240](https://www.nsf.gov/awardsearch/showAward?AWD_ID=2118240) (Imageomics: A New Frontier of Biological Information Powered by Knowledge-Guided Machine Learning). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
## Model Card Authors
Juan Garcia
## Model Card Contact
Please open a [Discussion on the Community Tab](https://huggingface.co/imageomics/trait2vec/discussions) with any questions on the model. |