encodingai/electra-base-discriminator-im-multilabel-V3

This is an updated version of https://huggingface.co/encodingai/electra-base-discriminator-im-multilabel, which follows the initial model reported in Pang & Ring (2020) and found at implicitmotives.com. The classifier identifies the presence of implicit motive imagery in sentences, namely the three felt needs for Power, Achievement, and Affiliation.

The current classifier is finetuned from ELECTRA-base and achieves > 0.90 ICC on the Winter (1994) training data (see the OSF repo for the benchmark dataset, or follow this direct link). Development of this classifier is ongoing, and the current version has been trained on a larger and more diverse dataset, which means it generalizes better to unseen data.

This model is being made available to other researchers for inference via a Huggingface api. The current license allows for free use without modification for non-commercial purposes. If you would like to use this model commercially, get in touch with us for access to our most recent model.

Predictions on Winter manual dataset
-----
Intra-class Correlation Coefficient:
| Pow (Label_0): | 0.90 |
| Ach (Label_1): | 0.94 |
| Aff (Label_2): | 0.89 |
| mean: | 0.91 |

Pearson correlations:
| Pow (Label_0): 0.800 |
| Ach (Label_1): 0.86 |
| Aff (Label_2): 0.79 |
| mean: 0.82 |

API Inference guide

The inference api requires a Huggingface token. The sample code below illustrates how it can be used to classify individual sentences.

import json
import requests
api_key = "<HF Token>"
headers = {"Authorization": f"Bearer {api_key}"}
api_url = "https://utuu5vtk4i2h1vpk.us-east-1.aws.endpoints.huggingface.cloud"

# This is a sentence from the Winter manual that is dual-scored for both Pow and Aff
prompt = """The recollection of skating on the Charles, and the time she had
            pushed me through the ice, brought a laugh to the conversation; but
            it quickly faded in the murky waters of the river that could no
            longer freeze over."""

# Since this is a multilabel classifier, we want to return scores for the top 3 labels
data = {"inputs": prompt, "parameters": {"top_k": 3}}

response = requests.request("POST", api_url, headers=headers, json=data)

# Print the labels and scores (arranged in order of likelihood)
scores = {x['label']: x['score'] for x in response.json()}
print(scores)

# {'Aff': 0.9999667406082153, 'Pow': 0.999929666519165, 'Ach': 0.0000024892888177}

Local Inference guide

This model can also be loaded for local inference using the transformers library, as with any open Huggingface model. When the sample python code first runs, it will download the model to disk and use a cached version for subsequent calls.

from transformers import pipeline

# This is the current model, previous models are labeled V2 etc.
model = "encodingai/electra-base-discriminator-im-multilabel-V3"

# This is a sentence from the Winter manual that is dual-scored for both Pow and Aff
sentence = """The recollection of skating on the Charles, and the time she had
            pushed me through the ice, brought a laugh to the conversation; but
            it quickly faded in the murky waters of the river that could no
            longer freeze over."""

# Instantiate a text classifier using a standard transformers pipeline
classifier = pipeline("text-classification", model=model)

# Since this is a multilabel classifier, we want to return scores for the top 3 labels,
# which we store in a dict for the scored sentence
scores = {x['label']: x['score'] for x in classifier(sentence, top_k=3)}
# Print the labels and scores (arranged in order of likelihood)
print(scores)

# {'Aff': 0.9999632835388184, 'Pow': 0.9999274015426636, 'Ach': 2.489295866325847e-06}

# Or get the rounded scores for each motive
rounded = {x['label']: int(round(x['score'])) for x in classifier(sentence, top_k=3)}
print(rounded)

# {'Aff': 1, 'Pow': 1, 'Ach': 0}

References

McClelland, D. C. (1965). Toward a theory of motive acquisition. American Psychologist, 20,321-333.

Pang, J. S., & Ring, H. (2020). Automated Coding of Implicit Motives: A Machine-Learning Approach. Motivation and Emotion, 44(4), 549-566. DOI: 10.1007/s11031-020-09832-8.

Winter, D.G. (1994). Manual for scoring motive imagery in running text. Unpublished Instrument. Ann Arbor: University of Michigan.