| | --- |
| | datasets: |
| | - relbert/conceptnet_high_confidence |
| | model-index: |
| | - name: relbert/roberta-large-conceptnet-average-prompt-e-nce |
| | results: |
| | - task: |
| | name: Relation Mapping |
| | type: sorting-task |
| | dataset: |
| | name: Relation Mapping |
| | args: relbert/relation_mapping |
| | type: relation-mapping |
| | metrics: |
| | - name: Accuracy |
| | type: accuracy |
| | value: 0.8862103174603174 |
| | - task: |
| | name: Analogy Questions (SAT full) |
| | type: multiple-choice-qa |
| | dataset: |
| | name: SAT full |
| | args: relbert/analogy_questions |
| | type: analogy-questions |
| | metrics: |
| | - name: Accuracy |
| | type: accuracy |
| | value: 0.5 |
| | - task: |
| | name: Analogy Questions (SAT) |
| | type: multiple-choice-qa |
| | dataset: |
| | name: SAT |
| | args: relbert/analogy_questions |
| | type: analogy-questions |
| | metrics: |
| | - name: Accuracy |
| | type: accuracy |
| | value: 0.49258160237388726 |
| | - task: |
| | name: Analogy Questions (BATS) |
| | type: multiple-choice-qa |
| | dataset: |
| | name: BATS |
| | args: relbert/analogy_questions |
| | type: analogy-questions |
| | metrics: |
| | - name: Accuracy |
| | type: accuracy |
| | value: 0.7443023902167871 |
| | - task: |
| | name: Analogy Questions (Google) |
| | type: multiple-choice-qa |
| | dataset: |
| | name: Google |
| | args: relbert/analogy_questions |
| | type: analogy-questions |
| | metrics: |
| | - name: Accuracy |
| | type: accuracy |
| | value: 0.886 |
| | - task: |
| | name: Analogy Questions (U2) |
| | type: multiple-choice-qa |
| | dataset: |
| | name: U2 |
| | args: relbert/analogy_questions |
| | type: analogy-questions |
| | metrics: |
| | - name: Accuracy |
| | type: accuracy |
| | value: 0.5526315789473685 |
| | - task: |
| | name: Analogy Questions (U4) |
| | type: multiple-choice-qa |
| | dataset: |
| | name: U4 |
| | args: relbert/analogy_questions |
| | type: analogy-questions |
| | metrics: |
| | - name: Accuracy |
| | type: accuracy |
| | value: 0.5439814814814815 |
| | - task: |
| | name: Lexical Relation Classification (BLESS) |
| | type: classification |
| | dataset: |
| | name: BLESS |
| | args: relbert/lexical_relation_classification |
| | type: relation-classification |
| | metrics: |
| | - name: F1 |
| | type: f1 |
| | value: 0.9085430164230828 |
| | - name: F1 (macro) |
| | type: f1_macro |
| | value: 0.9007282568605484 |
| | - task: |
| | name: Lexical Relation Classification (CogALexV) |
| | type: classification |
| | dataset: |
| | name: CogALexV |
| | args: relbert/lexical_relation_classification |
| | type: relation-classification |
| | metrics: |
| | - name: F1 |
| | type: f1 |
| | value: 0.8380281690140845 |
| | - name: F1 (macro) |
| | type: f1_macro |
| | value: 0.656362704638303 |
| | - task: |
| | name: Lexical Relation Classification (EVALution) |
| | type: classification |
| | dataset: |
| | name: BLESS |
| | args: relbert/lexical_relation_classification |
| | type: relation-classification |
| | metrics: |
| | - name: F1 |
| | type: f1 |
| | value: 0.6657638136511376 |
| | - name: F1 (macro) |
| | type: f1_macro |
| | value: 0.6498144246049421 |
| | - task: |
| | name: Lexical Relation Classification (K&H+N) |
| | type: classification |
| | dataset: |
| | name: K&H+N |
| | args: relbert/lexical_relation_classification |
| | type: relation-classification |
| | metrics: |
| | - name: F1 |
| | type: f1 |
| | value: 0.9565277874382695 |
| | - name: F1 (macro) |
| | type: f1_macro |
| | value: 0.8746667490411619 |
| | - task: |
| | name: Lexical Relation Classification (ROOT09) |
| | type: classification |
| | dataset: |
| | name: ROOT09 |
| | args: relbert/lexical_relation_classification |
| | type: relation-classification |
| | metrics: |
| | - name: F1 |
| | type: f1 |
| | value: 0.8896897524287057 |
| | - name: F1 (macro) |
| | type: f1_macro |
| | value: 0.8862724322889753 |
| |
|
| | --- |
| | # relbert/roberta-large-conceptnet-average-prompt-e-nce |
| |
|
| | RelBERT fine-tuned from [roberta-large](https://huggingface.co/roberta-large) on |
| | [relbert/conceptnet_high_confidence](https://huggingface.co/datasets/relbert/conceptnet_high_confidence). |
| | Fine-tuning is done via [RelBERT](https://github.com/asahi417/relbert) library (see the repository for more detail). |
| | It achieves the following results on the relation understanding tasks: |
| | - Analogy Question ([dataset](https://huggingface.co/datasets/relbert/analogy_questions), [full result](https://huggingface.co/relbert/roberta-large-conceptnet-average-prompt-e-nce/raw/main/analogy.json)): |
| | - Accuracy on SAT (full): 0.5 |
| | - Accuracy on SAT: 0.49258160237388726 |
| | - Accuracy on BATS: 0.7443023902167871 |
| | - Accuracy on U2: 0.5526315789473685 |
| | - Accuracy on U4: 0.5439814814814815 |
| | - Accuracy on Google: 0.886 |
| | - Lexical Relation Classification ([dataset](https://huggingface.co/datasets/relbert/lexical_relation_classification), [full result](https://huggingface.co/relbert/roberta-large-conceptnet-average-prompt-e-nce/raw/main/classification.json)): |
| | - Micro F1 score on BLESS: 0.9085430164230828 |
| | - Micro F1 score on CogALexV: 0.8380281690140845 |
| | - Micro F1 score on EVALution: 0.6657638136511376 |
| | - Micro F1 score on K&H+N: 0.9565277874382695 |
| | - Micro F1 score on ROOT09: 0.8896897524287057 |
| | - Relation Mapping ([dataset](https://huggingface.co/datasets/relbert/relation_mapping), [full result](https://huggingface.co/relbert/roberta-large-conceptnet-average-prompt-e-nce/raw/main/relation_mapping.json)): |
| | - Accuracy on Relation Mapping: 0.8862103174603174 |
| |
|
| |
|
| | ### Usage |
| | This model can be used through the [relbert library](https://github.com/asahi417/relbert). Install the library via pip |
| | ```shell |
| | pip install relbert |
| | ``` |
| | and activate model as below. |
| | ```python |
| | from relbert import RelBERT |
| | model = RelBERT("relbert/roberta-large-conceptnet-average-prompt-e-nce") |
| | vector = model.get_embedding(['Tokyo', 'Japan']) # shape of (1024, ) |
| | ``` |
| |
|
| | ### Training hyperparameters |
| |
|
| | The following hyperparameters were used during training: |
| | - model: roberta-large |
| | - max_length: 64 |
| | - mode: average |
| | - data: relbert/conceptnet_high_confidence |
| | - template_mode: manual |
| | - template: I wasn’t aware of this relationship, but I just read in the encyclopedia that <obj> is <subj>’s <mask> |
| | - loss_function: nce_logout |
| | - temperature_nce_constant: 0.05 |
| | - temperature_nce_rank: {'min': 0.01, 'max': 0.05, 'type': 'linear'} |
| | - epoch: 85 |
| | - batch: 128 |
| | - lr: 5e-06 |
| | - lr_decay: False |
| | - lr_warmup: 1 |
| | - weight_decay: 0 |
| | - random_seed: 0 |
| | - exclude_relation: None |
| | - n_sample: 640 |
| | - gradient_accumulation: 8 |
| | |
| | The full configuration can be found at [fine-tuning parameter file](https://huggingface.co/relbert/roberta-large-conceptnet-average-prompt-e-nce/raw/main/trainer_config.json). |
| | |
| | ### Reference |
| | If you use any resource from RelBERT, please consider to cite our [paper](https://aclanthology.org/2021.eacl-demos.7/). |
| | |
| | ``` |
| | |
| | @inproceedings{ushio-etal-2021-distilling-relation-embeddings, |
| | title = "{D}istilling {R}elation {E}mbeddings from {P}re-trained {L}anguage {M}odels", |
| | author = "Ushio, Asahi and |
| | Schockaert, Steven and |
| | Camacho-Collados, Jose", |
| | booktitle = "EMNLP 2021", |
| | year = "2021", |
| | address = "Online", |
| | publisher = "Association for Computational Linguistics", |
| | } |
| | |
| | ``` |
| | |
