SAGESSE-EPFL/Mistral-7b-subject-extraction

Mistral-7B-Instruct-v0.3 Debate's Subject Extraction Model

Model Description

This model is a fine-tuned version of Mistral-7B-Instruct-v0.3, specifically designed for the task of extracting and rephrasing the main subject of a debate, from Reddit Submissions. The subject of a debate is the main topic that is being discussed, it can be a question, a statement, or a opinion that is being analyzed and discussed. Given a Reddit Submission created to start a discussion, the model should extract the subject of the debate from the submission, focusing on the more general topic / subject that is being discussed and the opinion of the author if it's present. This core subject should be rephrased in a clear and concise way, removing irrelevant information. This model is part of SAGESSE, a pipeline for processing data from Reddit threads to create argument maps.

Training and Fine-Tuning

• Base Model: Mistral-7B-Instruct-v0.3
• Training Data:
  • Claims Extraction Dataset: 381 annotated submissions from political domain.
  • Annotation Source: Annotated using GPT-4-turbo with few-shot prompting.
• Fine-Tuning Approach:
  • Technique: Parameter Efficient Fine-Tuning (PEFT) using Low-Rank Adaptation (LoRA).
  • Training Configuration:
    • Epochs: 2
    • Learning Rate: 2e-4
    • Batch Size: 1
    • Gradient Accumulation Steps: 8
    • Hardware: Single Nvidia A100 GPU with 40GB memory
    • LoRA Rank: 16
    • LoRA Alpha: 16
• Libraries Used: HuggingFace’s Transformers library

Performance

• Evaluation Metric: BERTScore (Precision, Recall, F1)
• Performance:
  • Precision: 0.947
  • Recall: 0.959
  • F1 Score: 0.953
• Efficiency:
  • Average Inference Time: 0.11 seconds per submission

Usage

This model can be used to extract and rephrase the core debate's subject from Reddit submissions. It is particularly useful for creating argument maps and understanding debates in online forums.

Input Format

The model was used with the following prompt template:

### Instruction:
You are a helpful assistant for debate's subject extraction from Reddit submissions. The subject of a debate is the main topic that is being discussed,
it can be a question, a statement, or a opinion that is being analyzed and discussed. You will get a Reddit submission created to start a discussion
and you have to extract the subject of the debate from the submission. For long submissions, focus on the more general topic / subject that is being
discussed and the opinion of the author if it's present. Rephrase the core subject of the debate in a clear and concise. Make sure to remove from the
extracted subject any irrelevant information, like greetings or expressions like 'I think' or 'I believe that'.
You should put the output in a JSON object with the key 'subject' referring to a string containing the core subject.
If the input submission doesn't contain a clear subject or it's not starting a debate, you should set the 'subject' to null.

### Submission:
<submission>

### Response:

Output Format

The model outputs a JSON object containing the extracted claims:

{
  "subject": "Extracted debate's subject"
}

Limitations

• Context Dependency: The model performs best when provided with sufficient context to resolve implicit subjects.
• Model Hallucination: There may be instances where the model rephrases the core subject incorrectly, particularly with sarcastic or highly nuanced submissions.
• Dataset Specificity: The model is trained on political discussions from Reddit and may not generalize well to other domains.

Citation

If you use this model in your research, please cite the following paper:

TBD

Contact

For questions or issues, please contact Matteo Santelmo at matteo.santelmo@epfl.ch.