|
|
--- |
|
|
library_name: transformers |
|
|
tags: |
|
|
- multimodal |
|
|
- video-understanding |
|
|
- sports |
|
|
- commentary-generation |
|
|
- llama3 |
|
|
- soccer |
|
|
language: |
|
|
- en |
|
|
datasets: |
|
|
- MatchTime |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# Matchcommentary: Automatic Soccer Game Commentary Generation |
|
|
|
|
|
## Model Description |
|
|
|
|
|
Matchcommentary is a multimodal model designed for automatic soccer game commentary generation. It combines video feature understanding with large language models to generate fluent and contextually appropriate soccer commentary. |
|
|
|
|
|
## Architecture |
|
|
|
|
|
The model consists of: |
|
|
- **Vision Encoder**: Q-Former architecture for processing video features |
|
|
- **Language Model**: LLaMA-3-8B-Instruct for text generation |
|
|
- **Feature Fusion**: Cross-attention mechanism between visual and textual information |
|
|
- **Domain Adaptation**: Soccer-specific vocabulary constraints |
|
|
|
|
|
## Intended Use |
|
|
|
|
|
### Primary Use Cases |
|
|
- Automatic soccer game commentary generation |
|
|
- Sports video understanding and description |
|
|
- Multimodal video-to-text generation |
|
|
|
|
|
### Limitations |
|
|
- Trained specifically on soccer/football content |
|
|
- Requires pre-extracted video features |
|
|
- Performance may vary on different video qualities or angles |
|
|
|
|
|
## Training Data |
|
|
|
|
|
The model was trained on the MatchTime dataset, which contains: |
|
|
- Soccer game videos with corresponding commentary |
|
|
- Multiple leagues and seasons |
|
|
- Temporal alignment between visual events and commentary |
|
|
|
|
|
## Performance |
|
|
|
|
|
The model achieves state-of-the-art performance on the MatchTime benchmark, with the best validation CIDEr score among tested configurations. |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from models.matchvoice_model import matchvoice_model |
|
|
import torch |
|
|
|
|
|
# Load model |
|
|
model = matchvoice_model( |
|
|
llm_ckpt="meta-llama/Meta-Llama-3-8B-Instruct", |
|
|
tokenizer_ckpt="meta-llama/Meta-Llama-3-8B-Instruct", |
|
|
num_video_query_token=32, |
|
|
num_features=512, |
|
|
device="cuda:0", |
|
|
inference=True |
|
|
) |
|
|
|
|
|
# Load checkpoint |
|
|
checkpoint = torch.load("model_save_best_val_CIDEr.pth") |
|
|
model.load_state_dict(checkpoint) |
|
|
model.eval() |
|
|
|
|
|
# Generate commentary |
|
|
with torch.no_grad(): |
|
|
commentary = model(video_samples) |
|
|
``` |
