Spaces:
Sleeping
Sleeping
Commit History
-
Clarify documentation: anomaly signal explainer, GRPO scope notes
User-authored documentation improvements that resolve the structural confusion flagged in review. README.md + New section 'The anomaly signal (the cleanest inference channel)' with concrete worked example showing how per-step anomalies encode the hidden profile. Explains why the env can train meta-RL inference even on a small model. + 'Continuous Personality Space' table now has a 'concrete intuition' column for each parameter (turns abstract distributions into readable behavioral descriptions). docs/training.md + 'Scope of this doc' banner: clarifies this guide describes the pure-GRPO-from-scratch stack (the historical Iter 5 path), NOT the canonical Algorithm Distillation recipe that produced the headline numbers. Explicit pointer to README's AD section. + Reward stack section expanded from 3 to 4 layers (now correctly documents belief_accuracy alongside format_valid, action_legal, env_reward). training/RhythmEnv_GRPO_Training.ipynb …361aed7 -
Fix HF Space README rendering + Dockerfile encoding
Two fixes: 1. README not visible in HF Space's playground UI ('No README available' shown to visitors). OpenEnv's gradio_ui.py reads README from /app/README.md by default, but our Dockerfile copies the repo to /app/env/, so the README lookup failed. Fix: set ENV_README_PATH=/app/env/README.md in the Dockerfile, which the OpenEnv web_interface.py reads as a fallback path. 2. Root Dockerfile was UTF-16 LE with BOM (FF FE prefix) which is correct enough for Docker but fragile and inconsistent with all other text files in the repo (UTF-8). Re-encoded as UTF-8/ASCII. Functional content unchanged from server/Dockerfile counterpart.39e98db
-
Tighten README: resolve GRPO contradiction, drop duplicate baseline table, remove internal mentor docs
Three structural fixes after a structural review of the submission docs: 1. GRPO contradiction (the worst issue): the README's headline section said GRPO refine 'lifted OOD by +0.023' while the Training section later said 'Stage 3 — Skipped in our headline result'. Both were simultaneously visible to a careful reader. Rewrote Stage 3 to say what actually happened: GRPO refine ran on top of SFT, lifted OOD generalization by +0.023 and discrete-3 by +0.013, no in-dist regression. Added the actual submission command. 2. Duplicate 'Baseline Scores (v2 grader)' table near line 255 had: - 'TBD' for the OOD teacher (which was 0.621 in the headline) - '(see results.md)' for the distilled student (which was already in the headline table) Pure noise; deleted. Headline table is the canonical reference. 3. Removed two docs that were internal references, not for judges: - docs/WhatMAkesSubmissionStandOut.md (mentor's stand-out criteria pasted in as a check…0503beb
-
Add SFT v3 + GRPO refine results to README + results.md
GRPO refine on top of SFT v3 (200 steps, lr 1e-5, beta 0.1) lifted: - OOD final_score: 0.536 -> 0.559 (+0.023, +4% relative) - discrete-3 final_score: 0.507 -> 0.520 (+0.013) - in-dist final_score: no change (within noise) belief_MAE essentially unchanged across all conditions; the SFT-prime distillation already extracted near-maximum inference quality from the teacher. Refined model uploaded to InosLihka/rhythm-env-meta-trained-sft-grpo-v1 with eval_results_v2.json containing full per-episode breakdown. New artifact: plots/sft_grpo_comparison.png (34KB) — heuristic / SFT v3 / SFT+GRPO side-by-side bar chart across all 3 conditions, embedded in README Helper scripts (added to repo for reproducibility): scripts/verify_rubric_equivalence.py — replays eval episodes locally under the new Rubric-based grader, verifies numerical equivalence against saved eval JSON. Confirmed 105/105 episodes match within float-precision tolerance after the Rubric refac…666b4ce
-
Refactor grader to use openenv.core.rubrics.WeightedSum + Rubric subclasses
Closes the acknowledged conformance gap. The functional behavior is preserved exactly (52/52 tests pass, including 2 new tests verifying the grader literally uses WeightedSum with 6 named child rubrics). Architecture: server/rubrics.py — 6 Rubric subclasses, one per scored axis: CrashFreeRubric, ProgressRubric, ConnectionRubric, AdaptationRubric, EfficiencyRubric, BeliefAccuracyRubric Each holds a reference to the env in __init__; forward(action, obs) ignores the per-step args (RFC 004 pattern for trajectory-summary scoring) and reads aggregated env state. make_grade_rubric(env) returns a WeightedSum composing all 6 with weights summing to 1.0 (0.15 + 0.20 + 0.10 + 0.25 + 0.10 + 0.20). RhythmEnvironment._grade_episode now lazy-builds and delegates to the WeightedSum on done=True. Also updated: - server/rhythm_environment.py: cached _grade_rubric field on the env - tests/test_rhythm_env.py: 2 tests verifying WeightedSum is used - docs/iterations.md: rep…f0ca22d -
Post-deadline: full eval results + bigger plots via Git LFS
Eval retry (HF Job 69edf4ddd70108f37ace0165) completed and uploaded eval_results_v2.json. Pulling those numbers in: in-distribution: Heuristic 0.463 -> Distilled 0.574 (+0.111) OOD generalization: Heuristic 0.455 -> Distilled 0.536 (+0.081) discrete-3: Heuristic 0.455 -> Distilled 0.507 (+0.052) belief_MAE in-dist: 0.213 (matches teacher 0.196) belief_MAE OOD: 0.265 Student wins all 3 conditions; in-dist belief_MAE matches the teacher within 0.02 (distillation transferred the inference skill cleanly). Changes: - README: replace single-condition headline table with full 3-condition table; add v3 baseline-vs-trained bar chart inline; add reward components + belief_accuracy plots inline - docs/results.md: fill in student column for all 3 conditions; clean interpretation paragraph - plots/sft_v3_baseline_vs_trained.png: new bar chart from v3 eval data - plots/grpo_iter2_reward_curve.png, reward_components.png, belief_accuracy.png: now stored via Git LFS so …d64efa6 -
README: embed reward curve and belief-accuracy curve plots
4dd50e0 -
README: drop iter2 plots, keep only SFT v3 loss curve (current pipeline)
8227b63 -
README: surface headline result table at top so judges don't need to click through
6226884 -
results.md: tighten language, present results without internal-process noise
d51061f -
Fill results.md with student numbers from partial v3 eval
The continuous-in-dist + OOD conditions were still running at the deadline (each MODEL condition takes ~28 min on a100 due to max_new_tokens=256 generation cost over 140 inference calls per condition). The discrete-3-profiles condition completed cleanly: HEURISTIC: 0.455 RANDOM: 0.426 MODEL: 0.527 (+0.072 vs heuristic, +0.101 vs random) This is a real win: the SFT-distilled Qwen 3B beats the heuristic baseline on a held-out eval condition, demonstrating the AD pipeline worked end-to-end. Teacher numbers (in-dist + OOD) remain as locked-in ceilings from the offline reeval.9d0f270 -
Acknowledge OpenEnv Rubric system conformance gap
We use a custom multi-component grader in _grade_episode that's functionally equivalent to openenv.core.rubrics.WeightedSum (composable weighted scoring, same independent components, same explicit weights) but doesn't use the literal Rubric class. The Rubric API is per-step (action, observation), while our grader reads aggregated episode-end state — a clean refactor would use TrajectoryRubric. Decision: don't refactor in the last 30 min before deadline. Risk of breaking the live HF Space (which IS the submission) outweighs the literal-conformance benefit. Documented as a v2 cleanup task in the code docstring + iterations.md.dc5658d -
Embed training plots inline in README with captions
Per mentor guidance ('Reviewers spend seconds on each plot'), don't make judges click through — show the plots in the README itself. Three plots embedded with one-line captions: - SFT v3 loss (the headline training run) - GRPO iter2 loss (proof we ran RL too, before pivoting to AD) - Baseline vs trained comparisonefe2271 -
Add plots/ folder: SFT v3 loss + GRPO iter2 reward curves
Per submission requirement: 'Evidence that you actually trained; at minimum, loss and reward plots from a real run.' Plots captured from two distinct training phases: - SFT v3: clean loss convergence (2.77 -> 0.083 over 525 steps) on the full 180-episode teacher dataset - GRPO iter 2: real 400-step run showing loss, reward curve, all 4 reward components, belief_accuracy trajectory, and baseline-vs-trained eval bars README links to plots/ from the submission-links section.f2401bf -
Move blog to root as BLOG.md (per Meta mentor guidance)
Mentor tip: 'Please push Blog.MD into your HF Space' — a separate markdown file at the root, distinct from the README, makes it obvious to judges where the writeup lives. Updated README link accordingly. Also removed training/WhatMakesAGoodSubmission.md (internal hackathon notes, not for judges).eccca42 -
Prune internal/stale docs; sharpen README submission links
Removed (internal-only or stale, not for judges): - docs/SubmissionFormDump.md (form notes) - docs/handoff_for_next_session.md (chat-handoff doc) - docs/iter4_partial_analysis.txt (mid-iteration analysis) - eval_baselines_meta.json (v1 grader baselines, replaced by v2) - eval_results.json (v1 single-task eval, superseded) - training_log.json (v1 GRPO log, superseded by per-iter HF Hub model repos) README: - Renamed 'Links' section to 'Submission links (for judges)' with explicit callouts for HF Space URL, training notebook, blog post — the 3 URLs judges need from the form1ba0d0e -
Fix prompt truncation in inference_eval.py: max_seq_length 768 -> 2048
Our user prompts (system + observation with 7-step history + anomalies) are 900-1200 tokens. The model was being loaded with max_seq_length=768, silently truncating prompts on the left. The model was seeing the END of the prompt (last few history entries) but missing the system prompt or older context. This explains why the earlier 'v1 buggy eval' had muted final_score even though belief_MAE was good — the model was generating from incomplete context. The HF Job logs surfaced this clearly: 'Unsloth: Input IDs of shape torch.Size([1, 986]) with length 986 > the model's max sequence length of 768. We shall truncate it ourselves.' Bumping to 2048 leaves room for ~1200 prompt + 256 generation + slack.1217c1d -
Add SKIP_EVAL flag to sft_on_hf.py for faster training-only runs
ff20f02 -
Fix max_new_tokens for CoT format + add eval-only HF Jobs script
inference_eval.py: - max_new_tokens 20 -> 256. The original 20 was set when the format was a single-line "S M W ACTION_NAME" answer (~7 tokens). After the SFT switch to CoT format (<reasoning>...</reasoning>\nS M W ACTION_NAME) the answer line lives at the END of ~100-200 token output, so 20 was truncating mid-reasoning and the parser was falling back to action names found in the partial reasoning text. This explains why SFT v1's belief_MAE was great (0.117 in-dist) but final_score was muted — inference was working but action choices were noisy. scripts/eval_on_hf.py: - Eval-only HF Jobs orchestrator. Pulls a trained model, runs inference_eval.py with the fix, uploads new eval_results_v2.json. Useful for re-evaluating SFT v1 without re-training. Story docs updated: - results.md: teacher numbers locked (in-dist 0.611, OOD 0.621) - README.md: AD recipe section, v2 grader formula, 3 reference profiles - blog_post.md: pivot story (grader fix + algorithm distillation) - …b9c9b8f -
Algorithm Distillation: grader v2 with belief_accuracy + SFT pipeline
Grader changes: - Add belief_accuracy term (weight 0.20) to _grade_episode - env.record_belief() for callers to register the agent's emitted belief - Redistribute weights: crash 0.20→0.15, progress 0.25→0.20, conn 0.15→0.10, adapt 0.30→0.25 - Heuristic + random baselines still emit no belief, score 0 on this component - Validates: gpt-5.4 teacher beats heuristic 0.617 vs 0.449 on 30/30 episodes Distillation pipeline: - scripts/generate_teacher_trajectories.py: Azure OpenAI teacher rollouts - scripts/reeval_teacher_trajectories.py: offline re-score under v2 grader - scripts/upload_teacher_data.py: push trajectories to HF Hub dataset - training/sft_prime.py: TRL SFTTrainer fine-tune of Qwen 2.5-3B on teacher data - scripts/sft_on_hf.py: HF Jobs orchestrator for SFT prime stage Parser + prompt: - extract_action_and_belief now handles both single-line and CoT-prefixed formats (takes LAST <digit> <digit> <digit> <ACTION> match) - SYSTEM_PROMPT in dataset.py unified with teacher's CoT-as…ece0bbe -
handoff: add iter 5 results + whack-a-mole pattern doc
iter 5 finished but landed at 0.33 (vs heuristic 0.58). Third distinct mode collapse mode: 86% SLEEP + constant (0.5, 0.5, 0.5) belief in all 14 eval episodes. Belief MAE looks great (0.110) but it's a measurement artifact — neutral belief accidentally matches average true profile under the tighter profile cap. Pattern across iterations: iter 1: EXERCISE single-action collapse iter 2: MEDITATE-EXERCISE 2-cycle collapse iter 5: SLEEP-spam + neutral-belief collapse Each fix closes one loophole and opens another. The unaddressed structural issue (per-step GRPO vs per-episode grader) is the only path forward — multi-step plan generation from external agent's analysis remains the single highest-leverage missing fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>839a758 -
docs: handoff bundle for new chat session + iter 4 partial analysis
- docs/handoff_for_next_session.md: comprehensive context dump for resuming in a new Claude session. Covers project state, all 5 iterations, both rounds of fixes (round 1 = 7 fixes for iter 2, round 2 = 7 fixes for iter 4+), what's broken, what's not, pending iter 6+ candidates, key files, hardware notes. - docs/logdump.txt: HF Jobs UI export from iter 4 cancelled run (235 steps, 3272 lines, the only data we have for the round-2 fixes in action). - docs/iter4_partial_analysis.txt: parsed trajectory + trends + final-step metrics. Shows belief_accuracy FLAT at -0.10 — the unsolved core issue. - scripts/analyze_logdump.py: parser used to produce the analysis. Honest meta-note: this project was hit-and-trial. Each iter exposed bugs we should have caught upfront. Future iterations should compute reward variance across IDENTICAL completions before training, analyze belief target distribution analytically, and run 50-step micro-smoke-trains before committing to 200+ step runs. C…7bb9278 -
docs: add explicit sim-to-real mapping diagram (vision alignment)
The architecture doc was purely technical. Added a new section 10 'Sim -> Real mapping' that shows side-by-side how each simulated signal maps to its production analog: - meters <-> wearable HRV, calendar, screen time, voice sentiment - profile <-> the real user's actual personality (also hidden) - belief <-> agent's internal user model (visible in POC only) - action <-> recommendation to user - reward <-> meter trend + accept/ignore taps - episode <-> rolling weekly window - OOD eval <-> cold start with new user Plus an explicit list of design choices that preserve the passive-only constraint (no ASK action, no profile feature at eval, belief is internal). The KEY connection: continuous-OOD eval = production cold-start scenario. If the agent personalizes to unseen profiles in sim, it has acquired the skill the real product needs when meeting a new user. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>63216a8 -
docs: detailed architecture diagrams with concrete values
10 visual diagrams (mostly ASCII trees and box layouts) walking through: 1. Three-layer system separation (agent / orchestration / env) 2. Episode timeline (28 steps = 7 days x 4 slots) 3. State tree (hidden profile + visible meters + history with anomalies) 4. What the agent sees (concrete prompt + completion example) 5. Reward stack with worked example: '3 7 5 DEEP_WORK' for seed=42 step 5 - Layer-by-layer reward computation with actual numbers - Final reward = +1.49 (sum of 4 weighted layers) 6. GRPO update step (8 completions -> advantages -> backprop) - Shows why mode collapse happens when reward_std=0 7. Dataset structure (replay metadata, NOT supervised labels) 8. Final episode grader (5 components weighted 0.20/0.25/0.15/0.30/0.10) 9. Three eval conditions (discrete-3, in-dist, OOD) 10. Spend & timing per iter Every diagram uses concrete values from a real example (sampled_42 profile, specific belief vector, real meter values, real reward calculations). Co-Authored-…4c69214 -
client: surface ALL observation fields (was dropping deltas, anomalies, last_action, step_history)
Pre-fix, an external user calling the deployed HF Space via this client got a strictly worse view than the server returned: per-meter deltas, anomalies, last_action, and the full step_history were silently dropped during JSON parsing. This violated the 'client/server separation done right' criterion in the hackathon rubric (training/WhatMakesAGoodSubmission.md): the client was pretending fields didn't exist. Fix: _parse_result now reconstructs the full RhythmObservation including all 5 *_delta fields, last_action, and step_history with full StepRecord fidelity (including the new vitality/cognition/progress/serenity/connection anomaly fields added in iter 4 commit). Also adds the hackathon submission criteria doc to the repo for reference. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>105973d -
iter4: fix the 'constant belief = free reward' bug + 6 other deep issues
Investigation prompted by external code review. Validated 5/5 critical findings against actual code; applied fixes for the 7 most impactful. [1] Anomaly signals NOW REACH THE LLM (Issue 1, was CRITICAL) Env was computing per-meter anomalies (actual_delta - expected_under_neutral) and stashing them in reward_breakdown — but format_observation_prompt only read meters and step_history. StepRecord didn't include anomalies. So the cleanest profile-inference signal in the env was COMPUTED BUT NEVER SHOWN to the agent. The agent has been doing meta-learning blindfolded. Fix: extend StepRecord with 5 anomaly fields, populate in env.step, surface in prompt and inference.py. The agent now sees how each person's response deviates from the average baseline — direct profile fingerprint. [2] Belief baseline subtraction (Issue 4, was CRITICAL — biggest impact) OLD: reward = (1 - mae) - 0.5 Constant '5 5 5' emission scored +0.336 raw per step (× 3.0 weight = …bb2a9c7 -
tooling: scripts/analyze_iter.py + docs/results.md template
scripts/analyze_iter.py — one-shot iteration analyzer: - Pulls trained model + eval JSON + log_history from HF Hub for any iteration - Prints summary table (3 conditions x 3 strategies) - Action distribution histogram - Belief diversity check - Training trajectory snapshots - Pass/fail verdict against success criteria docs/results.md — placeholder for the headline results page. Will be filled in once iter 3 (or final long run) completes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>d6d9e31 -
docs: iteration journal with hypothesis/result/root-cause/fix per iter
Captures the training journey honestly: - Iter 0 (pre-existing): single-task training regressed - Refactor: meta-RL conversion (continuous profiles + belief + adaptation) - Iter 1: mode collapse to single action (constant rewards = no GRPO gradient) - Iter 2: mode collapse to 2-cycle (proxy/goal misalignment exposed) - Iter 3: in flight with 5 architectural fixes + belief-first format Includes 'Why we missed it' sections for each failure - useful for both future maintenance and hackathon storytelling. Honest post-mortems are better submission material than polished success-only writeups. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>e12fc69 -
iter3: align reward with grader + belief-first format + exploration shaping
ITER 2 FAILURE: agent escaped single-action collapse only to find a 2-cycle loop (MEDITATE+EXERCISE alternation). final_score 0.22, worse than random. Belief learning (+0.36) works but doesn't influence action choice — belief was emitted as a string AFTER the action, so action couldn't condition on it. Five runtime-level fixes for iter 3: [1] Per-step reward grader-alignment (server/rhythm_environment.py:_compute_reward) Add profile-INDEPENDENT bias: +0.5*progress_delta + 0.4*connection_delta. Profile-weighted reward still drives belief inference (varies by profile), but agent now ALWAYS gets penalized for ignoring progress and connection. Closes the proxy/goal misalignment that let M+E alternation win. [2] Belief-first output format (training/reward_functions.py, dataset.py, inference.py) Was: 'ACTION_NAME S M W' (belief was post-hoc afterthought, didn't condition action) Now: 'S M W ACTION_NAME' (causal LM generates belief tokens FIRST, then action cond…64d24b3
-
iter2: fix mode collapse + 3 deeper bugs from code review
ITER 1 FAILURE: mode collapse to 'EXERCISE 5 5 5' at step 100. Root cause: format_valid (+1.0) and action_legal (+0.5) saturated for any valid output, contributing zero to GRPO advantage. Once 4/4 completions per prompt were identical, reward_std=0 and the policy froze. 7 fixes applied (4 from initial diagnosis + 3 from deep review subagent): [1] sampling temperature 1.0 -> 1.5 Forces diverse rollouts per prompt, breaks mode collapse mechanism [2] reward_weights [0.3, 0.3, 1.0, 1.0] -> [0.05, 0.05, 1.5, 3.0] Aggressively suppress saturated layers, amplify variable signals. Belief weight bumped to 3.0 (CRITICAL-1) so emitting belief is clearly rewarded vs the no-belief penalty. [3] action_legal: drop +0.5 for valid, return 0 instead Removes another constant-reward source. Layer is now pure penalty for malformed outputs. [4] explicit repetition penalty in env_reward (-0.3 if action seen 3+ in row) Direct training signal against mode collapse. Env already …e21a960 -
tune: GRPO hyperparameter fixes from ML reviewer
Root cause of likely under-training in v1: 4-layer reward stack with format/legal saturating at +1 each was drowning the env_reward and belief_accuracy meta-RL signals. Plus several TRL/DeepSeek best-practice deviations. Changes: - beta: 0.1 -> 0.04 (TRL/DeepSeek default; was anchoring policy too hard) - num_generations: 4 -> 8 (halves advantage-estimate variance in continuous-profile setting; critical for GRPO signal) - max_completion_length: 20 -> 32 (was silently truncating belief digits for actions like FAMILY_TIME and BINGE_WATCH after BPE tokenization) - reward_weights=[0.3, 0.3, 1.0, 1.0] in GRPOConfig: scales format_valid + action_legal down so env_reward and belief_accuracy dominate the gradient. Wrapped in try/except for TRL versions that don't support reward_weights. FAST_MODE preset also bumped NUM_GENERATIONS from 2 -> 4. Two completions per group is too low to estimate group advantage at all. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>dc0186f -
feat: FAST_MODE preset for 10-15 min iteration cycles
Adds knobs for fast hyperparameter sweeps: - FAST_MODE=1 -> 200 steps, 80 episodes, 800 samples, 2 generations, 2 eval episodes - All knobs individually overridable via env vars - MODEL_REPO_SUFFIX lets each run upload to a unique repo for comparison On A100-large: FAST_MODE finishes in ~10-12 min for ~$0.70 per iteration. Use to debug training stability and tune beta/lr before committing to a full 1500-step run. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1a865f8 -
feat: HF Jobs training script + plot generator
Adds: - scripts/train_on_hf.py: PEP 723 self-contained job script. Submit with: hf jobs uv run --flavor a10g-large --secrets HF_TOKEN scripts/train_on_hf.py Clones repo, trains, evals, generates plots, uploads to HF Hub. - scripts/plot_from_log.py: regenerates all 5 plots from saved log_history.json. Usable both inside the HF Job AND locally to re-plot without re-training. - training/train.py: now saves trainer.state.log_history to log_history.json so plots can be generated offline. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>73c7ea0 -
fix: notebook plot cell syntax error (newline in string literal)
The \n in ax.set_title("...\n...") was converted to a literal newline by JSON serialization, breaking the Python syntax inside the cell. Replaced with a single-line title. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>7340206 -
notebook: add belief-accuracy + reward-components plots
The original plot cell only generated training_loss.png and reward_curve.png. For meta-RL submission, the most important plot is missing: how the agent's belief_accuracy reward evolved over training. That curve directly proves the meta-learning thesis (agent learns to model the user from observation). Added two plots: - plots/reward_components.png: all 4 reward functions overlaid over training (format_valid, action_legal, env_reward, belief_accuracy) so you can see which signals were gradient-providing - plots/belief_accuracy.png: focused belief reward with rolling mean and neutral-baseline reference line Plot generation is defensive: it discovers all log_history keys and tries multiple TRL key conventions (rewards/X/mean, rewards/X, X) since these vary by TRL version. Prints "Available log keys" so the user can debug if any series isn't found. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>b5ac530 -
merge hf/main: meta-RL refactor supersedes prior commits
The HF-side commits 9ed122d (env: enrich observation) and 52e33e8 (Run 3 training results) have been functionally superseded by the meta-RL refactor in 6f180e1. Merging with -s ours preserves their history without altering files.786249b -
env: meta-RL refactor (continuous profiles, action+belief, adaptation grader)
Transforms rhythm_env from a 3-profile single-task RL benchmark into a meta-RL environment where the agent must infer the user's hidden personality from observations alone. Original 500-step training regressed vs heuristic on all 3 profiles; this refactor addresses the four root causes: 1. server/rhythm_environment.py - Continuous profile sampling: sample_profile(seed) draws 14 personality parameters from bounded distributions per episode (memorization impossible, agent must learn the *skill* of inference) - Discrete 3-profile mode preserved via profile_mode='discrete' or profile=<name> for legacy comparison - profile_to_belief_vector(): reduces a profile to 3-dim ground truth [social, morning, work] for belief-accuracy reward - get_belief_target() / get_profile_hint(): privileged accessors for training-side reward computation and curriculum hints - Grader rewrite: dropped meter_balance (was rewarding random behavior), added adaptation_scor…ecbe0d8 -
env: enrich observation with history, anomalies, and discovery bonus
Five environment-level changes to make personality discovery learnable: 1. step_history (last 7 steps) added to RhythmObservation — agent now has the raw action/reward/delta trajectory needed to detect profile anomalies across steps, not just the current snapshot 2. Per-meter anomaly signals in reward_breakdown — each step computes actual_delta minus expected_delta (neutral-profile baseline after time-of-day + vitality factor), giving the agent a direct fingerprint of the hidden modifier (e.g. +0.06 vitality_anomaly on DEEP_WORK = workaholic) 3. First-class delta fields on RhythmObservation (vitality_delta, etc.) and last_action — no longer buried in reward_breakdown dict 4. Discovery bonus (15%) added to _grade_episode — rewards profile-adapted strategy in steps 14–27 (second half of week); introvert avoids social, extrovert embraces it, workaholic front-loads work. Without this, the grader rewarded generic meter management and ignored personality infere…9ed122d -
env: enrich observation with history, anomalies, and discovery bonus
Five environment-level changes to make personality discovery learnable: 1. step_history (last 7 steps) added to RhythmObservation — agent now has the raw action/reward/delta trajectory needed to detect profile anomalies across steps, not just the current snapshot 2. Per-meter anomaly signals in reward_breakdown — each step computes actual_delta minus expected_delta (neutral-profile baseline after time-of-day + vitality factor), giving the agent a direct fingerprint of the hidden modifier (e.g. +0.06 vitality_anomaly on DEEP_WORK = workaholic) 3. First-class delta fields on RhythmObservation (vitality_delta, etc.) and last_action — no longer buried in reward_breakdown dict 4. Discovery bonus (15%) added to _grade_episode — rewards profile-adapted strategy in steps 14–27 (second half of week); introvert avoids social, extrovert embraces it, workaholic front-loads work. Without this, the grader rewarded generic meter management and ignored personality infere…0a15ab5 -
Add Run 3 training results: README update + training log (no plots)
- README: trained scores filled in (Introvert 0.617, Extrovert 0.725, Workaholic 0.539) - README: added analysis of what agent learned vs. missing (workaholic DEEP_WORK discovery) - training_log.json: per-step GRPO metrics from 500-step run 3 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>52e33e8 -
Add Run 3 training results: plots, training log, README update
- plots/: training_loss.png, reward_curve.png, baseline_vs_trained.png from 500-step GRPO run - training_log.json: per-step metrics (loss, KL, rewards) for full run 3 - README: trained scores filled in (Introvert 0.617, Extrovert 0.725, Workaholic 0.539) - README: added analysis of what agent learned vs. what still needs training (workaholic DEEP_WORK modifier discovery gap) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>c67f463 -
docs: fix README accuracy + add training results structure
- Fix REWARD_SCALE 10 → 15 (matches code) - Fix critical penalty -0.5 → -0.30 (matches code) - Fix blog post link to correct path - Uncomment plot references (ready for when Colab finishes) - Add product vision paragraph + sim-to-real link - Add validation section (openenv validate passes, HF Space 200) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>92808b9 -
docs: add sim-to-real deployment architecture reference
Captures the real-world product pipeline: wearable proxies for meters, calendar-to-action-type mapping, Accept/Ignore as reward signal. Also covers why the frozen model adapts in-context and why RL pretraining enables fast personalization (MAML analogy). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>24adee5 -
fix: correct GRPO training hyperparameters to prevent KL explosion
beta=0.01 weakened the KL penalty (opposite of intended), causing policy to diverge at step 18 and collapse to learn×28. Fix: beta=0.1 (stronger constraint), lr=5e-5 (more conservative), max_grad_norm=0.5 (clipping). Also fix train.py: max_completion_length was 368 (prompt-pad remainder) instead of 16, which would allow verbose drift in standalone runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>fb112e4 -
restore: validate-submission.sh to scripts/
Was accidentally deleted from docs/round1/scripts/ during docs cleanup. Moved to scripts/ at repo root (more appropriate location). inference.py at root was never deleted and remains correct for Life Simulator. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>8a56903 -
docs: reorganize — 25 files → 4 focused docs
Deleted all brainstorming (Plan_v2), Round 1 specs, stale Round 2 docs, and research references that no longer serve the submission. Final structure: docs/blog_post.md — public narrative (product vision + training story) docs/entity_definitions.md — technical reference (state/actions/profiles/reward) docs/environment_design.md — design rationale (why these choices, training arc) docs/training.md — training guide (GRPO config, pipeline, results) docs/references/judging_criteria.md — external hackathon criteria Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>1a25a1a -
refactor: rewrite blog around product vision; fix UI for Gradio 6
Blog now opens with the concrete product scenario (watch/calendar/sleep tracker inputs, Accept-or-Ignore as rewards) and frames RhythmEnv as the training curriculum for the inference skill the product needs. UI: replace ASCII meter bars with HTML progress bars, add week calendar grid and live matplotlib meter trajectory chart, merge configure+play into one tab. Fix Gradio 6 incompatibilities (theme kwarg, show_copy_button). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>5fbafee -
fix: rename kl_coef to beta (correct param name in TRL GRPOConfig)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>2c6ee11 -
docs: expand blog with purpose, sim-to-real framing, lightweight model goal
Adds sections explaining why we target a small/lightweight model (always-on, private, cheap vs frontier API cost), why simulation is valid as a curriculum (sim-to-real is standard RL practice), and what the trained behavioral inference skill looks like in a real product. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>26b1e6a -
fix: reduce kl_coef to prevent training instability
KL divergence exploded to 10731 at step 205 in the first run, causing the policy to drift and generate verbose 368-token outputs for the rest of training. Two fixes: kl_coef=0.01 (from default 0.1) and max_completion_length=16 (action names are ≤15 chars, no need for more). Also moves blog_post.md into docs/ for cleaner repo layout. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>0bdfeaa