BFTree/rwgbench-citation-frame-classifier

RWGBench Citation Frame Classifier

This model is the citation-frame classifier used by RWGBench to compute Citation Frame Alignment (CFA). CFA evaluates whether a generated related work section uses citations with a rhetorical frame distribution similar to the author-written reference section.

The classifier is a DeBERTa-v3-large sequence-classification model fine-tuned for multi-label citation-function prediction. It predicts seven MultiCite-derived labels:

Label	Intended citation function
motivation	motivates the target problem or research direction
background	provides general background or prior findings
uses	describes methods, data, tools, or results used by the target work
extends	indicates that the target work builds on or extends prior work
similarities	marks similarity or close relationship to prior work
differences	marks contrast, limitation, gap, or distinction
future_work	discusses future directions

Intended Use

The model is intended for automatic evaluation and analysis of related work generation systems. In RWGBench, citation contexts are extracted from generated and reference related work sections, classified with this model, aggregated into frame-count profiles, and compared with macro count F1.

It is not intended as a general-purpose judge of paper quality or citation correctness. It should be used together with citation-selection and citation-support metrics.

Files

File	Description
config.json, model.safetensors	Transformers sequence-classification checkpoint
tokenizer.json, tokenizer_config.json, special_tokens_map.json, spm.model, added_tokens.json	Tokenizer files
thresholds.json	Per-label development thresholds used for multi-label decoding
metrics.json	Development and held-out MultiCite evaluation metrics
label_map.json	Label mapping used during training
rwgb_classifier_config.json	RWGBench-specific usage metadata

Evaluation

Held-out MultiCite test performance:

Metric	Score
Macro F1	0.5585
Micro F1	0.6751
Samples F1	0.6933
Weighted F1	0.6750
Subset accuracy	0.5119

Per-label held-out F1:

Label	F1
motivation	0.3523
background	0.7756
uses	0.7372
extends	0.4450
similarities	0.5508
differences	0.6274
future_work	0.4211

The thresholds in thresholds.json are applied after sigmoid probabilities. If no label passes its threshold, RWGBench assigns the highest-probability label so that every citation context contributes at least one frame.

Usage With RWGBench

Download the model into the RWGBench code repository:

models/citation_frame_multicite/
  config.json
  model.safetensors
  tokenizer.json
  thresholds.json
  ...

run:

python src/evaluation/evaluate_generated_results.py \
  --input results/your_results.json \
  --output results/your_eval.json \
  --gold data/gold100_papers.json \
  --corpus data/corpus.json \
  --citation-frame-model-path models/citation_frame_multicite

RWGBench automatically marks numbered citations with <cite>...</cite> before classification, matching the input format used during training.