|
|
--- |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- answerdotai/ModernBERT-base |
|
|
tags: |
|
|
- modernbert |
|
|
- summarization |
|
|
- regression |
|
|
- qna |
|
|
--- |
|
|
# Model Card for ModernBERT-base-cos |
|
|
|
|
|
ModernBERT-base-cos is a ModernBERT-based sequence classification model specifically fine-tuned to assess the quality of summaries in a QnA context. This model is designed to evaluate how well a generated summary captures essential information needed for question-answering tasks as part of research on the "chain of summaries" approach. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
### Model Description |
|
|
|
|
|
This model evaluates the quality and completeness of summaries by providing a quality score. It helps determine whether a summary adequately captures the information needed for downstream QnA tasks, making it useful for: |
|
|
|
|
|
- Researchers working on summarization evaluation |
|
|
- QnA pipeline optimization |
|
|
- Educational applications requiring assessment of student-generated summaries |
|
|
- Content creation platforms where summary quality is important |
|
|
|
|
|
- **Developed by:** [More Information Needed] |
|
|
- **Funded by [optional]:** [More Information Needed] |
|
|
- **Shared by [optional]:** [More Information Needed] |
|
|
- **Model type:** [More Information Needed] |
|
|
- **Language(s) (NLP):** [More Information Needed] |
|
|
- **License:** [More Information Needed] |
|
|
- **Finetuned from model [optional]:** [More Information Needed] |
|
|
|
|
|
### Model Sources [optional] |
|
|
|
|
|
- **Repository:** [More Information Needed] |
|
|
- **Paper [optional]:** [More Information Needed] |
|
|
- **Demo [optional]:** [More Information Needed] |
|
|
|
|
|
|
|
|
|
|
|
## How to Get Started with the Model |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForSequenceClassification |
|
|
import torch |
|
|
|
|
|
# Load model and tokenizer |
|
|
model_id = "williambrach/ModernBERT-base-cos" |
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
|
model = AutoModelForSequenceClassification.from_pretrained(model_id).to("cuda") |
|
|
|
|
|
def summary_score( |
|
|
tokenizer, |
|
|
summaries: list[str], |
|
|
device: str = "cuda", |
|
|
return_tensor: bool = True, |
|
|
): |
|
|
inputs = tokenizer( |
|
|
summaries, return_tensors="pt", padding=True, truncation=True |
|
|
).to(device) |
|
|
with torch.no_grad(): |
|
|
outputs = model(**inputs) |
|
|
logits = torch.sigmoid(outputs["logits"]) |
|
|
if return_tensor: |
|
|
logits = logits |
|
|
else: |
|
|
logits = logits.cpu().numpy().tolist() |
|
|
return logits |
|
|
|
|
|
# Example |
|
|
texts = [ |
|
|
"test", |
|
|
"Michael Jackson was a famous singer and dancer.", |
|
|
"Michael Jackson was a famous singer.", |
|
|
"Michael Jackson was a famous dancer.", |
|
|
] |
|
|
scores = summary_score(tokenizer, texts, return_tensor=False) |
|
|
print(scores) |
|
|
``` |
|
|
|
|
|
|
|
|
#### Hardware |
|
|
|
|
|
[More Information Needed] |
|
|
|
|
|
#### Software |
|
|
|
|
|
[More Information Needed] |
|
|
|
|
|
## Citation [optional] |
|
|
|
|
|
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> |
|
|
|
|
|
**BibTeX:** |
|
|
|
|
|
[More Information Needed] |
|
|
|
|
|
|
