FrontierLab-Meta-Hackathon-2026 / docs /PROCESS_METRICS.md
Jaycee766's picture
Deploy FrontierLab v0.35 source f7b67b6a3b09
a0e9e53 verified
|
Raw History Blame Contribute Delete
2.71 kB

Process metrics

FrontierLab exposes deterministic workflow metrics for audit and demo use. These metrics describe process state, not scientific accuracy.

Metrics

SCR — stage completion rate

SCR is the number of completed FrontierLab milestones divided by five:

  1. research question defined;
  2. evidence stage reached a terminal audit/review state;
  3. at least one hypothesis exists;
  4. a method evaluation exists;
  5. a research plan exists.

A 100% SCR means the five workflow milestones exist. It does not mean the science is correct.

Human-intervention event stream

New runs record four consequential researcher events in Run.human_events:

  • CLAIM_REVIEW_SAVED;
  • METHOD_EVALUATION_SAVED;
  • PLANNING_APPROVED;
  • PLAN_STEP_APPROVED.

Each event carries a UTC timestamp, event kind, bound artifact ID, actor label when available, note and identity_verified=false. The record therefore provides attributable process evidence without pretending that a typed reviewer name is authenticated identity.

Process metrics prefer this recorded event stream. Older runs that predate the event schema remain readable and fall back to the conservative artifact-derived observable count.

The UI continues to call the value Observed human actions rather than a fully verified HIC, because the event stream covers the four instrumented consequential actions rather than every possible researcher interaction.

AERR — autonomous error recovery rate

AERR is reported only for the verified NVIDIA NIM research backend.

  • If no runtime error occurred, AERR is N/A because the denominator is zero.
  • If runtime errors occurred, the current no-retry policy means autonomous recoveries are zero, so AERR is 0% for that nonzero-error denominator.
  • Other optional backends return NOT_APPLICABLE rather than inheriting NIM assumptions.

Efficiency fields

For NIM receipts, the process-metrics layer sums:

  • attempted calls;
  • failed calls;
  • prompt tokens;
  • completion tokens;
  • total tokens;
  • recorded model-call elapsed seconds.

Export

frontierlab.reporting.export_json(run) adds a top-level process_metrics object to the provenance JSON.

The Streamlit interface exposes the same values in Process metrics · audit only. The complete provenance JSON also contains the raw human_events array through Run.as_dict().

Scientific boundary

The metrics record workflow activity. They do not measure:

  • claim entailment accuracy;
  • hypothesis quality;
  • wet-lab validity;
  • clinical correctness;
  • model generalization.

Every process-metrics export therefore records scientific_accuracy_measured = false and scientific_validity_verified = false.