FrontierLab-Meta-Hackathon-2026 / docs /HACKATHON_FINAL_SPRINT.md
Jaycee766's picture
Deploy FrontierLab v0.40 source b8cd97cdbdad
4ab6f02 verified
|
Raw History Blame Contribute Delete
14.5 kB

FrontierLab — Meta Hackathon Final Sprint

Updated 2026-11-03 (Asia/Taipei) · Planning memo, NOT proof of completed inference. Formal submission completed on Cerebral Valley on 2026-11-02.

Verified inference update — October 28, 2026

FrontierLab now has several dated genuine Meta Muse Glimmer / NVIDIA NIM records rather than only the original synthetic smoke test:

  • the original synthetic one-shot returned final text with verified model identity;
  • a published-excerpt strict-JSON check completed;
  • a 12-case joint-context batch completed;
  • the NIM-backed core executed two real Evidence Extraction Agent calls on frozen published excerpts and stopped at the mandatory human-review gate;
  • both core claims were subsequently reviewed by Jaycee871 as SUPPORTED with written rationales;
  • a source-bound continuation runner now revalidates that completed review and permits exactly two further calls: Adversarial Skeptic Agent and Hypothesis Agent.

The reviewed continuation was executed successfully in GitHub Actions run 40574846326. Two citation-linked hypotheses were preserved from the exact live artifact. M2 is selected, its canonical Method Evaluation still reconstructs exactly, and a zero-call Stage-5 readiness contract now carries all open reproducibility checks and planning obligations forward. Planning remains deliberately unapproved and ungenerated.

Official submission boundary

  • Official organizer schedule: https://dev.meta.ai/events/global-hackathon
  • Project submissions due 2026-11-04 21:00 ET = 2026-11-05 10:00 Asia/Taipei (US Eastern Standard Time applies after the November 1 DST change). Organizer may update dates.
  • Track: Indie Developer; project theme: Push the Frontier.
  • The organizer explicitly invites work with Meta Muse family, Meta open-weights and related tooling. A truly executed Meta inference capability must be demonstrable.
  • Detailed Final Submission form / exact required demo-video duration must be rechecked in participant portal; a 3-minute video is our internal production target, not asserted as a new official mandatory limit.

Verified vs. unverified, as of this plan

Deliverable Status Evidence / limitation
Python research evidence/provenance pipeline SOFTWARE VERIFIED v0.40 main; 407 tests passed after merge on run 41408148161. Test count is software coverage, not model accuracy.
Manuscript-to-abstract source matching SOFTWARE VERIFIED Valid DOI and verbatim excerpt checks are not semantic entailment
Human review gate, version-bound annotations SOFTWARE + RECORDED REVIEW VERIFIED Two real core claims have explicit researcher SUPPORTED judgments; these are not independent expert gold labels
BioNeMo Boltz-2 request and response checks OFFLINE PREFLIGHT ONLY NO hosted Boltz-2 model computation, no validated binding function
HF public Streamlit Space v0.40 DEPLOYED, IN SYNC WITH MAIN Redeployed by the deploy workflow from the latest main merge (most recent 2026-11-03) after 407 tests; the public Space API then reported runtime RUNNING, and the served deployment.json records the exact source SHA. Earlier deploys: 6f0f7d2 → 94bc1eb, 21f9092 → 714c31d, a0a6a51 → 8f3db35, 1012ce5 → 6041356 (2026-11-02), then a docs-only hand edit f339cbf and e6a81fa (PR #42) → ec5d057 (run 41596184744, 2026-11-03 15:25). Any later merge needs one more deploy for exact source parity. NIM remains owner-disabled there.
Authentic Meta Muse Glimmer response MULTIPLE RECORDED NIM RUNS Synthetic smoke, strict JSON, 12-case batch and two-call core extraction recorded; no scientific accuracy claim
Human-reviewed live continuation EXECUTED AND RECORDED Run 40574846326 completed the bounded skeptic → hypothesis continuation; artifact provenance validated and exact outputs preserved
Researcher method selection COMPLETED M2 selected; M1 retained as comparator; canonical evaluation bound to exact live hypotheses and human rationale stored separately
M2 planning readiness COMPLETED, NOT AUTHORIZED Zero-call packet validates current context and carries data/code/environment gaps plus SPR/avidity/search-bias/replication obligations. planning_approved=false; no plan generated.
Research Operating Interface IMPLEMENTED / MEASURED Recorded M2 core: 4/4 source-traceable artifacts (100%) and 2/4 fully EXTRACTED (50%); provenance metric, not accuracy.
Maintainability canary + advisory repair proposal EXECUTED / SCHEDULED Main v0.40 canary run 41408148311 passed 407 tests and reported HEALTHY, with an advisory DRIFT_REVIEW_REQUIRED / HEALTHY_DRIFT because the requirements.txt hash differs from the accepted Nov 1 baseline (accepting a new baseline is left to the owner). Failures remain quarantined; repair proposals are advisory only and cannot edit/commit/merge/deploy.
Gate-ablation comparison PARTIALLY EXECUTED Recorded 12-case post-generation replay is implemented: 8 decisive / 4 abstaining outputs, 0 new abstentions from the source/quote post gate, 8 human reviews pending. A true same-model/same-cases two-call prompt-level workflow is prepared but not yet executed. No accuracy claim.
Three-minute competition video RECORDED AND PUBLIC YouTube, public, 3:03; follows DEMO_VIDEO_SCRIPT.md
Formal submission through Cerebral Valley SUBMITTED 2026-11-02 Indie Developer Track, Push the Frontier, team FrontierLab (solo); the event page shows "View Submission". Deadline 2026-11-04 21:00 ET.

Priority 1: explicit Stage 5 decision — highest priority

  1. The M2 readiness packet is now implemented and machine-checks the current reviewed context before Stage 5.
  2. The packet carries all unresolved checks forward: design-input provenance, executable/alternative ADAPT implementation, environment/dependencies and protonation/modeling settings.
  3. It also requires matched-pH controls, Fab-level validation, monomeric-Fab SPR where feasible, avidity/stoichiometry caveats, structure/search-bias disclosure, replication and falsification.
  4. Do not invoke the Planning Agent until the researcher explicitly approves Stage 5 planning. Readiness is not approval.
  5. If approval is given later, the generated plan must remain a cited conceptual draft and then pass the existing evidence-graph plan-step gate.

Priority 2: controlled gate ablation and claim-error benchmark

  • Start with 12 preregistered cases: 4 supported, 4 contradicted, 4 insufficient, using legally reusable snippets. Cover exact quote but opposite interpretation, source-ID mismatch, stale version, claims that cannot be decided from abstract, and tool failure.
  • Expert/researcher labels are separate from agent predictions. Blind dual review is desirable before publishing any gold standard.
  • The zero-call post-generation replay is now implemented and visible in the app. It must be described as a replay because the original prompt already requested evidence quotes.
  • The manual two-call prompt-level workflow now holds the model and case/source payload fixed while changing only the task instruction: (A) label-only, (B) quote-grounded + deterministic source/quote gate; (C) remains human review.
  • Keep human intervention explicit rather than calculating fake fully autonomous accuracy.
  • Record per-case outcome, citations, reject/abstain, latency, token usage, false positives and false negatives. A reviewer correction is not automatically evidence the model corrected itself.
  • Do not present 12 offline fictional rule tests as model benchmark outcomes.

Optional frontier extension: Muse Spark Open frontier backend

  • Provider-neutral software adapter is implemented for Meta Muse Spark Open, including text/image/audio/video request contracts and provenance controls.
  • Do not make Spark a submission blocker: the verified Glimmer path remains the stable demo baseline.
  • If hosted Spark access is obtained, use one bounded multimodal evidence case tied to M2 (for example, a structure or assay figure) and preserve the provider/model receipt.
  • Liquid d1-2B remains a lightweight evidence-verification option; Spark is the Meta-native frontier reasoning layer.
  • One live Muse Spark Open text extraction is recorded through Hugging Face Inference Providers (Baseten), not through this adapter: docs/runs/hf_muse_live_extraction_20261101.json (single paper, 2 claims with verified quotes, researcher review pending). No multimodal Spark inference is claimed.

Optional edge extension: Muse2-230M edge worker

  • Keep Muse2-230M strictly in the preprocessing layer: metadata normalization, log-to-JSON, schema formatting and routing.
  • Do not use it for evidence support/contradiction labels, scientific entailment or clinical decisions.
  • No model download or live Muse2 inference is required for the competition submission; the stable Meta reasoning path remains primary.

Optional baseline extension: Llama 3.1 8B baseline

  • Use Llama 3.1 8B only as a medium-size local baseline / worker, not as a replacement for the verified Muse Glimmer path.
  • A future controlled benchmark can compare Muse2-230M, Llama 3.1 8B and Muse Glimmer on the same structured evidence task, with Muse Spark reserved for frontier multimodal reasoning.
  • Report JSON validity, abstention, citation validity, quotation validity, researcher intervention, latency and cost/size proxies separately.
  • Do not turn the competition demo into a model leaderboard; the core story remains evidence-grounded, human-gated scientific research.

Optional audit extension: DetectorScope output-audit lens

  • Use DetectorScope only to explain detector features in model-generated scientific prose; it is outside the scientific truth path.
  • Never use its label or p_ai for academic-integrity, authorship, publication, hiring or person-level decisions.
  • A future controlled model-scale study may compare recurring interpretable linguistic features across Muse2-230M, Llama 3.1 8B, Muse Glimmer and Muse Spark outputs.
  • Keep this extension non-blocking for submission; the five-stage evidence/human-review pipeline remains the core demo.

Priority 3: judge-operable demo

  • Use DEMO_RUNBOOK.md as the canonical ~3-minute recording story. Keep the narrative on the scientific problem, genuine recorded Meta inference, governance effect, human decision and provenance—not the number of optional models.
  • Failure-path demo must remain honest if the model unexpectedly produces a correct response. Include a recorded genuine model trace and a replay only when transparently labeled 'recorded real run'.
  • Public HF CPU Basic is the frontend. It does not have a loaded 30B model; hosting Meta inference needs an approved external provider/backend with secrets kept off GitHub and appropriate cost controls.
  • Package a stable backup video that matches the application; do not call prewritten fake responses live model output.

Current official schedule check — October 30, 2026

The current Meta and Cerebral Valley event pages both still list November 4, 2026 at 9:00 PM ET as the project submission deadline. The official Meta page describes the Indie Developer Track as being judged on creativity, technical ambition and whether the project works; the Push the Frontier theme explicitly includes accelerating scientific experiments.

Official references:

The participant portal may contain additional submission-form fields that are not visible on the public event pages, so the logged-in final form needed a separate check (done: submitted on the portal on 2026-11-02).

Priority 4: release and submission

  • Follow RELEASE_CHECKLIST.md: manually deploy current main to the existing HF Space, then verify exact source SHA, current central version, RUNNING runtime, health endpoint and rendered marker.
  • Use SUBMISSION_COPY.md as the source of truth for public one-line/short/long descriptions and limitations; edit only when shipped behavior changes.
  • Confirm actual current requirements within the logged-in Cerebral Valley participant portal; complete all mandatory fields, video, code and links. (Done: submitted on 2026-11-02.)
  • Target internal completion by 2026-11-04 in Taipei to leave room for portal and upload failures; formal deadline remains 2026-11-05 10:00 Taiwan time.

Features deferred until after project submission

  • Additional talking-only agents or elaborate agent orchestration not tied to measured results.
  • Full BioNeMo hosted science execution, wet-lab binder optimization, or unlicensed private paper upload.
  • Claims about AlphaZero-style human concept transfer without a proper participant study.
  • Publication-scale biomedical benchmark or claims of superhuman AI scientific discovery.

GO / NO-GO: Do not claim a Meta-powered live end-to-end scientific product unless there is authentic dated Meta model output, a working endpoint, and a reproducible recorded trace. A transparent partial prototype is preferred over a false success claim.

Post-submission Information Management research direction

FrontierLab now documents a project-origin Information Management extension around maintainability, provenance and governance.

This is not a claim that Paper2Agent or another cited paper already validated those three pillars. It is a FrontierLab / Pack Kwan Low research direction motivated by the move from passive research artefacts to operable agents.

Post-submission priorities:

  • extend the already executed maintenance canary into controlled dependency/API/environment fault-injection and bounded-repair experiments;
  • extend provenance from paper/model/human records to code/data/environment artefacts;
  • study institutional tool/data authorization and responsibility boundaries;
  • evaluate a Research Operating Interface that manages Paper / Code / Data, MCP-style tools/resources/prompts, Evidence Trace, Governance and Maintainability;
  • study FrontierLab's proposed field list for an Agent Availability reporting statement (a section first anticipated by the Paper2Agent authors) as a complement to Data/Code Availability.

These items are research agenda, not current hackathon deliverables.