LIMITATIONS.md

Limitations, safeguards and next research steps

• Synthetic-only benchmarks: accuracy and macro F1 are not estimates of real-world scholarly-document performance.
• Domain shift: the controlled IID set gives substantially higher performance than the fixed novel-pattern challenge. The gap is visible in the app. A prior different Python classifier only reached 35.7% on a 28-sentence PubMed development smoke sample, demonstrating a transfer concern but not a definitive external evaluation for this app.
• Dataset size: the challenge has only 54 rows; small subgroup samples make comparisons fragile.
• Leakage audit: exact text and ID overlap checks cannot rule out shared templates or semantically similar training text.
• Tokenizer constraints: accent normalization and word/character n-grams do not understand discourse context or domain meaning.
• Uncertainty: bootstrap resampling from a fixed synthetic set does not capture selection bias or future deployment shift.
• Scoring: Naive Bayes outputs are not calibrated; cosine scores are not probabilities.
• Fairness: language/domain slice breakdowns are descriptive diagnostics, not equity certification or bias-free evidence.
• Risk: no individual-level consequential use (grading, admissions, hiring, publication acceptance, misconduct allegations).
• External data: a future phase could seek permitted, independent, human-labeled corpora; preserve source licensing and consent as needed. New external scores must come from real executed experiments, never estimated or fabricated.