README.md

---
title: ICSAC Editorial System
emoji: 📄
colorFrom: indigo
colorTo: gray
sdk: static
pinned: false
license: mit
short_description: Open-source AI editorial pipeline for ICSAC submissions
---

# ICSAC Editorial System

This is the open-source AI editorial pipeline that accepts and reviews every
submission to the [Institute for Complexity Science and Advanced Computing](https://icsacinstitute.org)
and its journal [*Persistence*](https://icsacinstitute.org/journal) ([`/submit`](https://icsacinstitute.org/submit)).

If you sent a paper to ICSAC — this repository is exactly what received and
read it. The submission handler, the rubrics, the prompts, the citation checks,
the redaction layer, the audit pass that scores the panel itself. With one
defensive exception (the prompt-injection signal in the RQC audit is not
published, to avoid teaching adversarial submissions what to avoid — see
[`rubrics/review_quality_control.md`](rubrics/review_quality_control.md) for
the rationale), nothing about how your work is evaluated is hidden.

## Why this is public

Independent and heterodox researchers have legitimate cause to be skeptical of
black-box editorial AI. The reasonable response is not to hide the system
behind a "trust us." The reasonable response is to publish it.

You can read the rubrics ICSAC reviewers apply ([`rubrics/`](rubrics/)). You can
read the templates that draft the acceptance and revise-and-resubmit letters
([`templates/`](templates/), [`intake/templates/`](intake/templates/)). You can
read the redacted reviews of every accepted paper at
[icsacinstitute.org/accepted](https://icsacinstitute.org/accepted) — produced
by the exact redaction layer in this repo. You can run the pipeline yourself.

## What this repo does NOT include

This repo is the AI reviewer half of the institute. **Intentionally not public**:
the manuscript archive, author records, member directory, internal moderation
tooling, the website that fronts all of this, and the operational infrastructure
that runs it. Those remain private to protect the privacy and security of
authors, members, the team, affiliates, and site visitors. What this repository
exposes is *how* your manuscript was evaluated and *what* was recorded — not
the people, payment, identity, or infrastructure layers around that
evaluation.

## What it does

The pipeline has two halves: a long-running **submission front-end** that
accepts new manuscripts, and a **batch review pipeline** that runs on a timer
and drains the queue. Both live in this repo.

For each submission:

1. **Intake** ([`intake/intake_server.py`](intake/intake_server.py)) — the
   FastAPI handler behind `icsacinstitute.org/submit` accepts a DOI / arXiv
   reference OR a direct PDF upload. The upstream form host verifies the
   submitter's ORCID via OAuth and signs each request with an HMAC; this
   handler re-validates and persists the submission record.
2. **Manuscript fetch** ([`submission_intake.py`](submission_intake.py)) —
   the worker either downloads the manuscript from the resolver (DOI route)
   or uses the uploaded PDF as the archive of record. Text and references are
   extracted.
3. **Citation verification** ([`citation_verify.py`](citation_verify.py),
   [`citation_misattribution.py`](citation_misattribution.py)) — every cited
   work is checked against arXiv, Crossref, and Semantic Scholar. Fabricated
   and misattributed citations are flagged.
4. **Five-reviewer panel** ([`review.py`](review.py)) — a panel of independent
   model instances reviews the manuscript against ICSAC's rubrics (scope,
   methodology, calibration, tone, AI provenance signal). Each reviewer scores
   blind to the others. The panel tolerates one slot failure per pass via
   self-heal (MIN_REVIEWERS=4), so a published review may reflect four valid
   outputs if a slot errored.
5. **Review quality control** ([`review_quality_control.py`](review_quality_control.py)) —
   a separate auditor reviews the panel itself. Low-confidence dimensions,
   missing injection indicators, or systemic drift trigger curator alerts.
6. **Redaction** ([`redaction.py`](redaction.py)) — internal reasoning, vendor
   names, and operational metadata are stripped before any review is shared
   with the author or published.
7. **Decision** ([`action.py`](action.py)) — the panel recommends one of three
   outcomes:
   - **Accept** — published to `icsacinstitute.org/accepted/<id>` with a
     redacted copy of the panel's review.
   - **Revise and resubmit** — author receives the panel's feedback and may
     resubmit a revised version.
   - **Scope reject** — reserved for submissions that fall outside the
     institute's remit (pseudoscience, non-engageable). Not a standard
     editorial outcome.

   Every panel run produces a *recommendation*, not a verdict. The
   recommendation, the full panel report, and the RQC audit are sent to
   the curator via the operator's configured notification channel. The
   curator confirms, modifies, or overrides before any author email is
   drafted, any publications-registry entry is written, or any Zenodo
   deposit is staged. The panel never auto-delivers a decision.

## What it is not

This is the system one institute uses to evaluate its own submissions. It is
not a general-purpose academic peer review platform. It is not a service you
can submit to without going through `icsacinstitute.org/submit`. It is not a
replacement for human editorial judgment — every panel outcome routes to a
human curator for the final verdict, and any author who disagrees with that
verdict can appeal to a human editor.

It is also opinionated. The rubrics reflect the institute's editorial scope:
complexity science, information theory, persistence dynamics, and adjacent
methodology. A submission outside that scope will be flagged as out-of-scope
regardless of its merit on its own terms.

## Repository layout

The editorial-side modules live at the top level. The submission front-end is
a Python subpackage at [`intake/`](intake/).

### Editorial pipeline (top level)

| Path | Purpose |
|------|---------|
| `editorial_workflow.py` | Top-level workflow orchestration |
| `submission_intake.py` | Manuscript fetching and text extraction |
| `citation_verify.py` | Cross-reference citation validation |
| `citation_misattribution.py` | Catches "real DOI, wrong paper" errors |
| `review.py` | The five-reviewer panel |
| `review_quality_control.py` | Auditor that scores the panel |
| `redaction.py` | Redaction layer for public review output |
| `action.py` | Decision dispatch (accept / R&R / scope reject) |
| `repository_deposit.py` | Stages accepted papers to a repository backend (Zenodo) |
| `email_send.py`, `email_render.py` | Author correspondence primitives |
| `publications.py`, `publish_watcher.py` | Post-acceptance publication |
| `watch.py` | Polls Zenodo for curator state transitions and posts branded comments on accept/decline |
| `notify.py` | Curator-channel notification helpers (channel-agnostic dispatcher; forkable) |
| `directory.py` | Community directory data layer |
| `stats.py` | Aggregate statistics over historical reviews |
| `rubrics/` | The editorial rubrics applied to every submission |
| `templates/` | Editorial-side correspondence templates (Zenodo-comment posts, etc.) |
| `editorial-batch.{service,timer}`, `editorial-review.{service,timer}`, `submission-watcher.{service,timer}` | systemd units for the batch workflow, community-request poller, and intake/decision watcher |

### Submission front-end ([`intake/`](intake/))

| Path | Purpose |
|------|---------|
| `intake/intake_server.py` | FastAPI app: `POST /api/submit`, `GET /api/submission/{id}/state`, `GET /healthz` |
| `intake/submission_worker.py` | Drains the queue, resolves deferred DOIs, dispatches into the review pipeline |
| `intake/apply_decision.py` | Applies the curator's verdict: publications registration, deposit staging, author email |
| `intake/notify_author.py` | Author email rendering; attaches redacted panel report + RQC as PDFs |
| `intake/rehydrate.py` | Refetch a stubbed DOI submission's bytes from the resolver, verify SHA |
| `intake/templates/` | Author-facing email templates |
| `intake/scripts/` | Smoke tests for the test-mode pipeline |
| `intake/README.md` | Subpackage-scoped documentation |

## Running it yourself

The pipeline is designed to run on a single host. The submission server runs
as a long-lived FastAPI service; the editorial workflow runs from systemd
timers (twice daily by default) and drains the queue when it ticks.

Dependencies:

- A submission repository (we use Zenodo — needs an API token with read + deposit scopes)
- An LLM provider (OpenRouter, Anthropic, or compatible)
- An SMTP account for author correspondence
- A registry destination (the institute's website repository, in our case) for
  post-acceptance publication of accepted papers and redacted reviews

```bash
python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt

cp config.example.py config.py     # edit for SMTP, Zenodo, LLM provider
export INTAKE_HMAC_SECRET=$(openssl rand -hex 32)   # for the intake HMAC gate

# Submission server (long-running):
uvicorn intake.intake_server:app --host 127.0.0.1 --port 8443 --log-level warning

# Submission worker (drains the queue when markers appear):
python -m intake.submission_worker

# Editorial batch tick (run from systemd timer in production):
python editorial_workflow.py batch-tick
```

Set secrets in `/etc/icsac/editorial.env` (loaded by systemd `EnvironmentFile=`).
For manual runs, source the same file. See [`intake/README.md`](intake/README.md)
for the full list of intake-side environment variables.

## License

MIT. See [`LICENSE`](LICENSE).

The rubrics and review templates are MIT-licensed code artifacts — feel free
to fork, adapt, and use as the basis for your own institute's review system.

## A note to authors

If your paper was reviewed by this system and you disagree with the panel's
recommendation: write to `help@icsacinstitute.org`. A human editor reads every
appeal. The panel is not the last word — it is a thorough first pass that the
curator turns into the verdict, and that the editor overrides on appeal.

If your paper was accepted: the redacted review is published at
`icsacinstitute.org/accepted/<your-record-id>` alongside the work itself.

If you want to know exactly which prompts the panel saw, which rubrics it
applied, which citations it verified, and which form fields the submission
handler collected before forming its recommendation: read the source. That is
why this repository exists.

## Support this work

ICSAC operates on bootstrap funding and private donations. Open peer review for
independent and heterodox researchers is the kind of infrastructure no one is
paid to build. If you want to help keep it running:

**[manifund.org/projects/icsac-ai-assisted-peer-review-for-independent-researchers](https://manifund.org/projects/icsac-ai-assisted-peer-review-for-independent-researchers)**