AbeerMostafa/Novelty_Reviewer

Novelty Reviewer

Novelty Reviewer assesses the novelty of a research paper the way expert reviewers do. Given the full text of a paper, it returns a novelty score on a four-point scale and a written assessment of the paper's original contributions and their overlap with prior work.

The model is Llama-3.1-8B-Instruct, fully fine-tuned on the novelty judgments of 25,951 ICLR and NeurIPS submissions from the Novelty Reviewer Dataset.

Developed by	Abeer Mostafa, Thi Huyen Nguyen, Zahra Ahmadi
Model type	Decoder-only language model, 8B parameters
Base model	meta-llama/Llama-3.1-8B-Instruct (full fine-tuning)
Language	English
License	Llama 3.1 Community License
Paper	Are We Truly Innovating? A Qualitative and Quantitative Study of Originality in AI Research Papers
Code	github.com/AbeerMostafa/Novelty-Reviewer
Dataset	AbeerMostafa/Novelty-Reviewer-Dataset

Novelty scale

Score	Label	Definition
−1	Not novel	The work is derivative and largely replicates existing approaches.
0	Limited novelty	The work makes minor variations of known methods.
1	Moderately novel	The work is original but overlaps notably with prior work or mainly extends existing ideas.
2	Highly novel	The work introduces fundamentally new ideas, methods, or problem formulations.

The scale follows the Expert-Consensus Novelty Alignment (ECNA) labels of the training data, which summarize the stance of each paper's reviewers.

How to use

The model expects the full paper as markdown (or plain text) inside its training prompt. The companion package wraps the prompt, the decoding settings, PDF conversion and output parsing:

pip install "novelty-reviewer[model] @ git+https://github.com/AbeerMostafa/Novelty-Reviewer"

from novelty_reviewer import NoveltyReviewer
from novelty_reviewer.pdf import pdf_to_markdown

reviewer = NoveltyReviewer.from_pretrained("AbeerMostafa/Novelty_Reviewer")
prediction = reviewer.predict(pdf_to_markdown("paper.pdf"))

print(prediction.score, prediction.label)   # e.g. 1 Moderately novel
print(prediction.review)

With Transformers only

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AbeerMostafa/Novelty_Reviewer"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

prompt = NOVELTY_REVIEW_PROMPT.format(paper_text=paper_markdown)   # prompt below
input_ids = tokenizer.apply_chat_template([{"role": "user", "content": prompt}], add_generation_prompt=True,
                                          return_tensors="pt").to(model.device)
output = model.generate(input_ids, max_new_tokens=300, do_sample=True, temperature=0.1, top_p=0.5,
                        pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(output[0][input_ids.shape[1]:], skip_special_tokens=True))

Prompt used for training and evaluation (use it verbatim; {paper_text} is the paper)

NOVELTY_REVIEW_PROMPT = """
You are an expert reviewer for AI conferences. 
Your task is to evaluate the NOVELTY of research papers according to reviewer guidelines. Be factual, concise, and balanced.
You should carefully read the paper, judge whether the ideas are novel or not based on the technical content and the previous related work,
and provide a concise justification based only on the content of the paper.

Return two things: Novelty Score and Short Human-like Novelty Review.

Review the following paper for novelty:

{paper_text}

Provide your evaluation in the following format:
Novelty Score: 
-1 = Not novel (work is incremental, derivative, or replicating existing approaches with minimal innovation.)
0 = Limited novelty (work is somewhat standard, showing minor variations or applications of known methods without substantial conceptual or technical innovation.)
1 = Moderately novel (some originality but overlap with prior work, or extension/combination of existing ideas.)
2 = Highly novel (fundamentally new ideas, approaches, problem formulations, or insights that significantly advance the field.)

Short Novelty Review: A paragraph of ~ 250 words containing your summary and reasoning about paper novelty/originality.
"""

Decoding. All reported results use temperature 0.1, top-p 0.5 and 300 new tokens. The repository's generation_config.json defaults (temperature 0.6, top-p 0.9) are inherited from the base model and give more varied outputs.

Output format. The model writes Novelty Score: <score>, followed by a Score Justification and a Detailed Assessment. The score line may appear twice, because the training targets prefixed the assessment with its score. novelty_reviewer.parse_novelty_output returns the score and the assessment text.

Hardware. The weights take about 16 GB in bf16; a GPU with at least 24 GB is recommended for full papers.

Intended uses

Direct use. Decision support for reviewers, area chairs and authors: a first, reviewer-aligned estimate of a paper's novelty and a starting point for checking related work. Research on automated novelty assessment and review generation.

Downstream use. Within the retrieval-augmented framework of the code repository, the model receives summaries of related papers retrieved from Semantic Scholar and grounds its assessment in explicit comparisons with them.

Out-of-scope use. Accept/reject decisions without human review; evaluating researchers; papers outside machine learning without validation; detecting plagiarism without human verification.

Training

Data. The 25,951 training submissions of the Novelty Reviewer Dataset (ICLR 2022–2025, NeurIPS 2022–2024). The input is the full paper converted to markdown; the target is the ECNA score and paper-level assessment derived from the paper's reviews. The 6,488 test submissions are held out.

Procedure. Full-parameter supervised fine-tuning with axolotl. The loss is computed on the target only.

Hyperparameter	Value
Epochs / steps	3 / 2,433
Effective batch size	32 (1 per GPU × 32 GPUs)
Learning rate	2 × 10⁻⁵, cosine schedule, 50 warm-up steps
Optimizer	AdamW (β = 0.9, 0.999; ε = 10⁻⁸), no weight decay
Sequence length	32,120 tokens, no packing
Precision	bf16 with TF32
Efficiency	DeepSpeed ZeRO-3, gradient checkpointing, Flash Attention, Liger kernels
Hardware	32 × NVIDIA A100 80 GB
Software	axolotl 0.12.2, Transformers 4.55.2, PyTorch 2.6.0

axolotl configuration

base_model: meta-llama/Llama-3.1-8B-Instruct

#load_in_4bit: true

#adapter: lora
#lora_r: 16
#lora_alpha: 32
#lora_dropout: 0.05
#lora_target_modules:
#  - q_proj
#  - v_proj
#  - k_proj
#  - o_proj


plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: true

strict: false

chat_template: llama3
datasets:
  - path: data/train_tokenized.parquet  # scripts/training/prepare_training_data.py
    type:
    ds_type: parquet

dataset_prepared_path:
val_set_size: 0.00
output_dir: ./checkpoints/novelty_reviewer

dataset_processes: 16

sequence_len: 32120
sample_packing: false
pad_to_sequence_len: true

gradient_accumulation_steps: 1
micro_batch_size: 1
num_epochs: 3
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 2e-5

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
flash_attention: true

warmup_steps: 50
evals_per_epoch: 0
eval_table_size:
saves_per_epoch: 2
save_only_model: true
debug:
deepspeed: scripts/training/configs/deepspeed_zero3_bf16.json
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  pad_token: <|finetune_right_pad_id|>
  eos_token: <|eot_id|>

Evaluation

Novelty score prediction on the 500-paper evaluation subset of the test split. Precision, recall and F1 are macro-averaged over the four classes. All models received the full paper and used the same decoding.

Model	Accuracy	Precision	Recall	F1
Qwen2.5-14B-Instruct-1M	0.052	0.262	0.252	0.026
Llama-3.1-8B-Instruct	0.148	0.162	0.280	0.089
Paper Reviewer (Qwen3-8B)	0.328	0.172	0.289	0.161
GPT-OSS-20B	0.533	0.205	0.295	0.227
Novelty Reviewer	0.620	0.378	0.319	0.323

The predicted score distribution of Novelty Reviewer follows the reviewers' distribution, whereas general-purpose LLMs mostly assign high novelty. The paper additionally reports free-text quality (NLI, LLM-as-a-judge, BERTScore, AlignScore), generalization to an unseen venue (MIDL 2026) and an idea-level plagiarism case study. The evaluation scripts are in the code repository.

Limitations and risks

• The model learns reviewer opinion, including its subjectivity and inconsistencies; its scores are not ground-truth novelty.
• Training labels were generated by an LLM from the reviews and validated on a sample.
• Without retrieval, the model judges novelty from the paper and its training knowledge only, and cannot know work published after its training data.
• Training covers English machine learning papers from two venues; other fields need validation.
• Absolute performance on the four-class task is moderate, with the extreme classes being hardest.

Use the model to support, not replace, human judgment.

License

The model is released under the Llama 3.1 Community License. Built with Llama.

Citation

@misc{mostafa2026novelty,
  title         = {Are We Truly Innovating? A Qualitative and Quantitative Study of Originality in AI Research Papers},
  author        = {Mostafa, Abeer and Nguyen, Thi Huyen and Ahmadi, Zahra},
  year          = {2026},
  eprint        = {2602.06054},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2602.06054}
}