docs/parser/test/mvp_eval_output/323.json

{
  "id": "323",
  "source_file": "docs/test/docling_output/323.pdf.json",
  "llm_provider": "openrouter",
  "llm_model": "openai/gpt-oss-120b:free",
  "paper_title": "A Neural Local Coherence Model",
  "input_chars": 33602,
  "contributions": [
    "Proposes a convolutional neural network architecture that operates on the entity grid representation to capture long-range entity transitions",
    "Introduces distributed (embedding) representations for grammatical roles and entity-specific features to improve generalization over discrete models",
    "Presents an end-to-end pairwise ranking training method that learns task‑specific high‑level features automatically",
    "Demonstrates state‑of‑the‑art performance on three coherence assessment tasks (discrimination, insertion, summary rating) with significant absolute improvements over previous models"
  ],
  "research_topic": "This paper addresses the problem of improving text coherence assessment by developing a neural network model that operates on entity grid representations. It tackles limitations of traditional entity grid approaches, such as discrete feature representations and task‑agnostic feature extraction, by using distributed embeddings and end‑to‑end training to capture long‑range entity transitions and learn task‑specific features.",
  "related_count": 0,
  "related_summaries": [],
  "elapsed_seconds": 31.1,
  "review": {
    "paper_summary": "The paper proposes a convolutional neural network that operates directly on the entity‑grid representation of a document, replacing the traditional discrete feature extraction with distributed embeddings for grammatical roles and entity‑specific attributes. By training the network with a pairwise ranking loss, the model learns task‑specific high‑level features end‑to‑end. Experiments on three standard coherence evaluation tasks (discrimination, insertion, and summary rating) show consistent improvements over previous entity‑grid baselines, with absolute gains of roughly 4% to 4.5%. The authors also release their code to facilitate reproducibility.",
    "strengths": [
      "Introduces a novel neural architecture that leverages CNNs over entity grids, addressing the limitation of discrete feature representations.",
      "End‑to‑end training with a pairwise ranking loss allows the model to automatically learn task‑specific features, simplifying the feature engineering pipeline.",
      "Comprehensive experimental evaluation across three established coherence tasks demonstrates consistent and statistically significant improvements over strong baselines."
    ],
    "weaknesses": [
      "The description of the CNN architecture (e.g., filter sizes, depth, handling of variable‑length grids) lacks sufficient detail to fully assess design choices.",
      "Ablation studies are missing; it is unclear how much each component (embeddings, convolutional layers, ranking loss) contributes to the reported gains.",
      "The paper does not discuss computational efficiency or scalability, which are important when processing long documents with large grids."
    ],
    "comments_suggestions": "- Provide a more thorough architectural diagram and hyper‑parameter settings (filter widths, stride, padding, embedding dimensions) to aid reproducibility.\n- Include ablation experiments that isolate the impact of embedding the grammatical roles versus using the CNN, and compare the ranking loss to a standard cross‑entropy objective.\n- Report training and inference time compared to the traditional entity‑grid SVM baseline to give readers a sense of practical overhead.\n- Clarify how out‑of‑vocabulary entities and coreference errors are handled, as these could affect performance on noisy data.\n- Minor typographical issues: \"loosing\" → \"losing\"; inconsistent spacing around punctuation; some sentences are incomplete (e.g., the paragraph ending with \"needs to compute R k transition probabilities from a g\").",
    "ethics_concerns": "None identified.",
    "soundness": 3,
    "excitement": 3,
    "reproducibility": 4,
    "confidence": 4,
    "overall_assessment": 3.0,
    "overall_assessment_label": "Accept — Findings"
  }
}