|
|
--- |
|
|
library_name: transformers |
|
|
datasets: |
|
|
- cardiffnlp/tweet_eval |
|
|
language: |
|
|
- en |
|
|
metrics: |
|
|
- f1 |
|
|
base_model: |
|
|
- OuteAI/Lite-Oute-1-300M-Instruct |
|
|
pipeline_tag: text-classification |
|
|
--- |
|
|
|
|
|
# Model Card for Lora-adopted Lite-Oute-1-300M-Instruct |
|
|
|
|
|
The model was trained with LoRA adapter to classify the sentiment of twitter messages into 'positive', 'negative', and 'neutral'. It was trained on cardiffnlp/tweet_eval dataset. |
|
|
LoRA-adopted layers include k_proj and v_proj weight matrices for all attention layers. |
|
|
|
|
|
|
|
|
## Model Details |
|
|
|
|
|
The system prompt for the model is as follows: |
|
|
|
|
|
>You are a helpful assistant that classifies the sentiment of a message. Classify the sentiment of the given message as exactly one word: 'negative', 'neutral', or 'positive'. Be brief, respond with exactly one word. |
|
|
|
|
|
Inputs for the model should be provided in the following format: |
|
|
|
|
|
>Message: "[text of the message]" |
|
|
> |
|
|
The model is trained to output labels in the following format: |
|
|
|
|
|
>The sentiment of the message is [label]. |
|
|
|
|
|
where [label] is either 'positive', 'negative' or 'neutral'. |
|
|
|
|
|
Labels can be extracted from the model's outputs with the following function: |
|
|
|
|
|
~~~python |
|
|
import re |
|
|
def postprocess_sentiment(output_text: str) -> str: |
|
|
""" |
|
|
Extracts the sentiment classification ("positive" or "negative") from the model's output text. |
|
|
|
|
|
Process: |
|
|
1. Splits the output at the first occurrence of the keyword "assistant" and processes the text after it. |
|
|
2. Uses a regular expression to search for the first occurrence of the words "positive" or "negative" (ignoring case). |
|
|
3. Returns the found sentiment in lowercase. If no match is found, returns an empty string. |
|
|
|
|
|
Parameters: |
|
|
output_text (str): The complete text output from the model, including conversation headers. |
|
|
|
|
|
Returns: |
|
|
str: The sentiment classification or empty string |
|
|
""" |
|
|
|
|
|
parts = output_text.split("assistant", 1) |
|
|
text_to_process = parts[0] if len(parts) > 1 else output_text |
|
|
text_to_process = text_to_process.lower() |
|
|
match = re.search(rf"\b({'|'.join(IDX2NAME.values())})\b", text_to_process, re.IGNORECASE) |
|
|
return match.group(1).lower() if match else "" |
|
|
~~~ |
|
|
|
|
|
## Training Details |
|
|
|
|
|
Only k_proj and v_proj layers were adopted. LoRA layers are of rank=8 and use scaling factor alpha=16. |
|
|
Model was trained for 1 epoch with learning rate=5e-4 and batch_size=16. Final loss (CrossEntropy) was 0.0673. |
|
|
|
|
|
## Evaluation |
|
|
|
|
|
Confusion matrix calculated on the test set is presented below: |
|
|
|
|
|
 |
|
|
|
|
|
It corresponds to macro f1-score of 0.52. |
|
|
|
|
|
## Examples of outputs: |
|
|
|
|
|
Input (correct label is 'positive'): |
|
|
>Message: "I think I may be finally in with the in crowd #mannequinchallenge #grads2014 @user" |
|
|
|
|
|
Output: |
|
|
>"The sentiment of the message is positive" |
|
|
|
|
|
|
|
|
Input (correct label is 'neutral'): |
|
|
>Message: "@user @user That's coming, but I think the victims are going to be Medicaid recipients." |
|
|
|
|
|
Output: |
|
|
>"The sentiment of the message is neutral" |
|
|
|
|
|
Input (correct label is 'negative'): |
|
|
>Message: "@user Wow,first Hugo Chavez and now Fidel Castro. Danny Glover, Michael Moore, Oliver Stone, and Sean Penn are running out of heroes." |
|
|
|
|
|
Output: |
|
|
>"The sentiment of the message is positive" |
|
|
|
