Spaces:
Running
Running
| # Limitations, safeguards and next research steps | |
| - **Synthetic-only benchmarks:** accuracy and macro F1 are not estimates of real-world scholarly-document performance. | |
| - **Domain shift:** the controlled IID set gives substantially higher performance than the fixed novel-pattern challenge. The gap is visible in the app. A prior *different* Python classifier only reached 35.7% on a 28-sentence PubMed development smoke sample, demonstrating a transfer concern but not a definitive external evaluation for this app. | |
| - **Dataset size:** the challenge has only 54 rows; small subgroup samples make comparisons fragile. | |
| - **Leakage audit:** exact text and ID overlap checks cannot rule out shared templates or semantically similar training text. | |
| - **Tokenizer constraints:** accent normalization and word/character n-grams do not understand discourse context or domain meaning. | |
| - **Uncertainty:** bootstrap resampling from a fixed synthetic set does not capture selection bias or future deployment shift. | |
| - **Scoring:** Naive Bayes outputs are not calibrated; cosine scores are not probabilities. | |
| - **Fairness:** language/domain slice breakdowns are descriptive diagnostics, not equity certification or bias-free evidence. | |
| - **Risk:** no individual-level consequential use (grading, admissions, hiring, publication acceptance, misconduct allegations). | |
| - **External data:** a future phase could seek permitted, independent, human-labeled corpora; preserve source licensing and consent as needed. New external scores must come from real executed experiments, never estimated or fabricated. | |