Longformer QASPER Extractive Document QA
This model card is the upload template for the checkpoint produced by
notebooks/complete_longformer_training_evaluation_pipeline.ipynb.
Base model
valhalla/longformer-base-4096-finetuned-squadv1
Project fine-tuning
The checkpoint is further fine-tuned by this project on a documented subset of QASPER v0.3 containing only answerable questions with one contiguous extractive span that can be located in the reconstructed paper text. Free-form, yes/no, unanswerable, unresolved, and multi-span annotations are excluded because this model predicts one contiguous span.
Evaluation
The model was evaluated on 200 contiguous-extractive examples from the QASPER validation set.
| Model | Exact Match | Token F1 | Evidence Recovery | Evidence Token Recall |
|---|---|---|---|---|
| BERT truncated to 512 tokens | 1.50% | 7.37% | 26.00% | 41.34% |
| Base Longformer with sliding windows | 6.00% | 16.16% | 30.00% | 45.88% |
| QASPER-fine-tuned Longformer | 12.50% | 26.66% | 49.00% | 60.14% |
Training configuration
- Training examples: 803
- Validation examples: 419
- Evaluation examples: 200
- Maximum training length: 3,072 tokens
- Sliding-window stride: 384 tokens
- Training epochs: 2
- Learning rate: 1e-5
- Precision: BF16
- GPU: NVIDIA GeForce RTX 5090
- Fine-tuned by this project: Yes
Performance interpretation
The QASPER-fine-tuned Longformer outperformed both the truncated BERT baseline and the original SQuAD-fine-tuned Longformer checkpoint across Exact Match, token-level F1, evidence recovery, and evidence-token recall.
The model remains imperfect. Exact Match is a strict metric, and QASPER questions frequently contain long, technical, and semantically complex answers. The reported values should not be interpreted as production-level reliability.
- Downloads last month
- 54