nawazishpatana/structure-extraction-t2-paper-split

Claim Information Extractor for Brain Tumor Research

Model: Flan-T5-base (fine-tuned for structured extraction)

Task: Extract structured claim information from research paper sentences.

Overview

This model extracts structured information from claim sentences in brain tumor research papers. It identifies key fields: model, task, dataset, metric, value, comparison, and domain.

The train/validation/test split is paper-based: all sentences from the same DOI/paper are kept in one split to avoid leakage.

Performance

• Validation Overall Exact Match: 0.5447
• Test Overall Exact Match: 0.4793
• Validation Macro Presence F1: 0.5402
• Test Macro Presence F1: 0.5055

Field-level Performance

Field	Exact Match	Non-Null Accuracy	Null Accuracy
model	0.7233	0.0863	1.0000
task	0.6318	0.1616	0.9885
dataset	0.7778	0.0286	1.0000
metric	0.6797	0.3250	0.9537
value	0.8824	0.4091	0.9946
comparison	0.8519	0.2556	0.9973
domain	0.6405	0.3962	0.9742

Dataset

• Training: 4,040 claim sentences from 1,233 papers
• Validation: 481 claim sentences from 155 papers
• Test: 459 claim sentences from 155 papers
• Total: 4,980 claim sentences from 1,543 papers
• Split method: paper/DOI-level split, not sentence-level split

Slot Fields

The model extracts the following structured fields:

• model: Name of the machine learning or deep learning model
• task: Type of task (e.g., classification, segmentation)
• dataset: Dataset or benchmark used
• metric: Evaluation metric (e.g., accuracy, Dice score)
• value: Numeric value or performance result
• comparison: Comparative statements (e.g., "outperforms ResNet")
• domain: Application domain (e.g., medical imaging)

Quick Start

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import json

model_name = "nawazishpatana/claim-extractor-brain-tumor"  # Example
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# Example input
prompt = '''Extract structured claim information as JSON with keys model, task, dataset, metric, value, comparison, domain. Use null for missing values.
Title: Dilated SE-DenseNet for Brain Tumor Segmentation
Year: 2024
Claim Sentence: Our model achieved 95% Dice score on BraTS dataset.'''

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
prediction = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(prediction)
# Output: {"model": "Dilated SE-DenseNet", "dataset": "BraTS", "metric": "Dice", "value": "0.95", ...}

Training Details

• Base Model: google/flan-t5-base
• Optimizer: AdamW
• Learning Rate: 2e-05
• Per-device Batch Size: 2
• Gradient Accumulation Steps: 1
• Effective Batch Size: 2
• Epochs: 5 (with early stopping)
• Max Input Length: 256
• Max Output Length: 128

Evaluation Metrics

• Exact Match (EM): Percentage of predictions that exactly match ground truth
• Non-Null Accuracy: Accuracy on fields where ground truth is not null
• Null Accuracy: Accuracy in predicting null when ground truth is null

Post-Processing

The model includes automatic post-processing:

1. Fuzzy matching for model names
2. Dataset name normalization
3. Metric standardization
4. Numeric value parsing

Citation

If you use this model, please cite:

@misc{claim-extractor-brain-tumor,
  title={Claim Information Extractor for Brain Tumor Research},
  author={Your Name},
  year={2025},
  howpublished={\url{https://huggingface.co/username/claim-extractor-brain-tumor}}
}