Spaces:
Sleeping
Sleeping
Commit History
-
Add local H100 scientist eval tooling
a29a83d
-
Add 12 scoring & environment improvements with full test coverage
- LLM judge integration via OpenRouter with config toggle and deterministic fallback - Communication bonus scoring (action diversity, responsiveness, substantiveness) - Per-domain scoring weight profiles (math, ML, finance) - Adaptive reward shaping (later rounds carry heavier penalties) - Richer fidelity matching with bigram overlap and substring fuzzy matching - Cross-dimension feasibility tradeoffs in Lab Manager policy - Mid-episode judge feedback hints at negotiation midpoint - Scenario consistency validation (budget, equipment, reagent checks) - Cached substitution index per episode in environment - Docker HEALTHCHECK directive for container orchestration - .dockerignore security hardening for .env files - Fix circular import in judge_policy via lazy import - 25 new tests (integration + LLM judge mock tests), 514 total passing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>2b98daa
-
Add model-driven local runtime and dynamic demo flow
4ea916a -
Finalize demo flow and training assets
b878f5b -
Merge Kush frontend integration, close API 16/UI 10/UI 11
- Merge multi-stage Docker build, SPA serving, and frontend components - API 16: Docker now builds frontend + API in single container - UI 10: Frontend styled with new components (ProtocolEditor, LiveScoreGauges, etc) - UI 11: Server mounts frontend/dist with SPA catch-all for React Router - Fix root endpoint test for SPA-mode serving - Max at 97.56% (40/41, only DOC 08 remaining blocked on TRN 10)abb29f8 -
Add API 19 /web fallback route, merge Kush frontend, close UI 07
- API 19: Self-contained /web endpoint with interactive episode controls - Merge origin/master with Kush's frontend api.ts, types, components - UI 07: Closed as completed by Kush (REST+WS client helpers) - 3 new /web fallback tests, 474 total tests passingb001a03 -
Add ENV 09 disk persistence, OBS 07/09, TST 11 audit tests, close 10 Max tasks
- ENV 09: Wire write_episode_log + log_episode_reward into REST/WS terminal handlers - OBS 09: Add invalid_action_count/rate fields to EpisodeLog model - OBS 07: Create scripts/run_episode.py for local smoke testing - TST 11: Add 17 judge audit contract tests in test_audit_contract.py - Close API 05/11/18, OBS 01/03, TST 06 as already implemented11faa95 -
Add MOD 08 schema tests, V2 training stack, and close MOD 08/JDG 07/API 01/OBS 02
- MOD 08: 70 comprehensive unit tests for all Pydantic models (Kian lane 100% done) - V2 training: GRPO, SFT, evaluation, metrics, plots, CLI, corpus, datasets modules - TRN 01/02/14: Notebook skeleton, install cell, model selection - JDG 07/API 01/OBS 02: Verified and closed in trackers (already implemented) - 409 tests pass, 92/152 tasks complete (60.53%)685783a -
Add JDG 07: reward breakdown logging to CSV and JSONL per episode
Expand append_reward_csv with V2 columns (parsimony, efficiency_bonus, communication_bonus, penalty_total, verdict). Add append_reward_jsonl for rich per-episode logs preserving penalties dict and bounded-tool metrics. Add log_episode_reward convenience wrapper writing both formats. 21 tests pass (10 new).82805bf -
Add hybrid Oracle layer and update architecture docs
ec2e890 -
Recover env judge training stack and sync project tracking
13ae015 -
Recover env server client stack and deployment tracking
8b157ab -
Add deterministic judge scoring engine (JDG 01-03)
JDG 01 (rigor): structural completeness + success criteria + required element coverage. Weighted sub-scores with token matching. JDG 02 (feasibility): continuous signal from FeasibilityCheckResult with partial credit for budget, equipment, reagents, and staff. Does not rescore — wraps AGT 05 to prevent drift. JDG 03 (fidelity): substitution-aware adherence to hidden reference spec. Direct match = 1.0, allowed substitution = 0.7, miss = 0.0. Shared text helpers extracted to replicalab/utils/text.py. 18 new tests covering all 3 scorers across all domains and difficulties. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>e50dca9 -
Add AGT 04/05/07 implementations, server integration, and doc updates
- AGT 04: build_baseline_scientist_action with domain inference - AGT 05: check_feasibility with per-dimension checks - AGT 07: compose_lab_manager_response with action type selection - Server: real Lab Manager pipeline replaces hardcoded stub responses - Tests: AGT 04/05/07 test coverage - Docs: task completion tracking and model selection notes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>7c2246c -
Add deterministic alternative suggestion logic for Lab Manager (AGT 06)
New suggest_alternative() builds a concrete revised Protocol by applying fixes in deterministic order: 1. Substitute unavailable equipment (first stable alternative) 2. Substitute unavailable reagents/resources 3. Clamp duration to time limit 4. Reduce sample size for budget/staff pressure (iterative halving) 5. Re-run check_feasibility on the revised protocol Returns AlternativeSuggestion with: - revised_protocol: the modified Protocol - applied_changes: list of SuggestionChange with field/original/revised/reason/tradeoff - remaining_failures: dimension names still failing after fixes - improved: bool indicating whether revision reduced failure count - pre_check / post_check: feasibility results before and after Returns None if protocol is already feasible. No LLM calls, fully deterministic, surfaces tradeoff strings from the scenario pack. 8 tests covering feasible no-op, equipment/reagent substitution, duration clamping, budget/sample reduction, determinism, post-check improvemen…87a89f0 -
Add parse-and-retry loop for Scientist agent with telemetry (AGT 03)
New call_scientist_with_retry() wraps the observation formatter and output parser in a retry loop with injected model backend: - Error-specific correction prompts (no_json, invalid_json, invalid_action) that restate the output contract each time - RetryMetadata exposes attempt_count, retry_count, last_error_code and last_error_message for training metrics (TRN 15, OBS 09) - ScientistCallResult bundles the validated action + metadata - No silent auto-fix of fields; final failure raises the last ScientistOutputParseError - generate_fn is fully injected (works with mocks, local models, hosted inference) 7 tests covering first-try success, malformed-then-valid, invalid-action-then-valid, exhausted retries, correction message content, and metadata serialization. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>19ab848 -
Add per-turn observation formatter for Scientist agent (AGT 02)
New format_scientist_observation(ScientistObservation) -> str that renders round status, paper summary, conversation history, current protocol, and action instruction in a fixed section order. Pure function, deterministic, no LLM calls. Completes the Scientist prompt pipeline: system prompt (AGT 01) + observation prompt (AGT 02) -> LLM -> parser (MOD 09) 5 tests covering empty/populated state, section order stability, null action_type entries, and scenario-derived observations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>528bd7d -
Add deterministic protocol validation against scenario constraints (MOD 05)
New replicalab/utils/validation.py with validate_protocol() that checks: - Obvious impossibilities (zero sample size, zero duration) - Duration vs lab time limit - Equipment vocabulary (available, booked, unknown) - Reagent vocabulary (in stock, out of stock, unknown) - Required-element coverage via token matching - Allowed substitution awareness for booked/out-of-stock items Returns structured ValidationResult with error/warning issues. Never crashes, no LLM calls, no budget heuristics. 13 tests covering all check categories and JSON round-trip. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>e239665 -
Type EpisodeState and EpisodeLog with Protocol, ConversationEntry, RewardBreakdown (MOD 04)
Replace loose dict fields with typed Pydantic models: - EpisodeState.current_protocol: Optional[Protocol] - EpisodeState.conversation_history: list[ConversationEntry] - EpisodeLog.transcript: list[ConversationEntry] - EpisodeLog.reward_breakdown: Optional[RewardBreakdown] Stub server now explicitly constructs Protocol and ConversationEntry objects instead of raw dicts. Fix WS handler info serialization. Add 8 tests covering typed construction, JSON round-trip, and nesting. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>2f4ed4a -
Add MOD 11 typed StepInfo/RewardBreakdown and import completed foundation modules
MOD 11: Replace untyped StepResult.info dict with StepInfo model (extra="allow") and RewardBreakdown model with constrained score fields. Stub server now explicitly constructs typed objects. Also imports previously completed work: config.py (MOD 12), seed.py (SCN 01), normalized scenario layer with three domain adapters (SCN 02-09), scientist policy parser, and all tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>5f8c92c -
Add future improvements doc, import FND 09 and MOD 01-03 completions
- Add docs/future_improvements.md with domain-agnostic normalized scenario layer design, impact analysis, and 6 planned work items - Import completed FND 09 (openenv.yaml), MOD 01 (ScientistAction validators), MOD 02 (LabManagerAction validators), MOD 03 (typed ConversationEntry and Protocol models) - Update task tracking docs, completion tracker, and Kian task lists - Add model validation tests and uv.lock Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>db07024 -
Complete FND 01 and FND 10, update task division with status tracking
- FND 01: Add repo scaffold with all top-level folders and subfolders - FND 10: Add replicalab/outputs/ with logs, replays, plots subdirs - Add status and completed-by columns to all epic task tables - Add status legend to epic backlog section - Add Section 4.1 training compute availability (H100) - Mark FND 01 and FND 10 as completed8a624de