Update model card: candid Kaggle status and troubleshooting narrative

README.md

@@ -39,63 +39,76 @@ model-index:
 
 # Tiny Recursive Models — ARC-AGI-2 (8× H200 Resume, Step 119 432)
 
-**What’s new (Nov 2025).** This refresh publishes the best-performing checkpoint from the CoreWeave resume campaign—`trm_arc2_8gpu_resume_step115815_plus100k_v2` at global step **119 432**. The job resumed from TinyRecursiveModels commit `e7b68717` with the full resume guard stack (`trm-common-script` + `trm-pyshim`) and legacy ARC identifier mapping. This is the same checkpoint we attempted to ship to Kaggle; the submission stalled at 0.83 % pass@1 because every task duplicated attempts, so we are documenting the shortfall here instead of claiming leaderboard progress.
+Summary
+- This is an 8× H200 resume snapshot at global step 119 432 from run `trm_arc2_8gpu_resume_step115815_plus100k_v2` (TinyRecursiveModels commit `e7b68717`).
+- It packages one consolidated `model.ckpt` plus the exact configs used on CoreWeave. Integrity hash: `2bc8bb3a5a85cd73e169a6fd285f9138427db894bd157edc20e92a58ed8ee33e`.
+- Evaluation quality is currently poor (ARC public evaluation pass@1 ≈ 0.83%) due to duplicate candidate generation after the resume. Until we address sampler issues, this artifact is provided for transparency and reproduction, not for leaderboard use.
 
-**Why the name mentions 119 434.** Internal tracking labelled this snapshot “step 119 434”, but the persisted shard on the CoreWeave PVC is `step_119432`. The W&B records for the run confirm that resume guard initialized at the expected `115 815` step and advanced to the 119k block; no 119 434 shard survived the routine pruning. When downstream tooling expects the 119 434 identifier, point it at this artifact and note the two-step discrepancy.
+Why the “119 434” label sometimes appears
+- Internal tracking briefly referred to a “step 119 434” snapshot. The durable shard that survived PVC pruning is `step_119432`. W&B logs confirm the resume guard initialized at step 115 815 and advanced into the 119k block; no distinct 119 434 shard remains.
 
-## Checkpoint Snapshot
-- **Run name**: `trm_arc2_8gpu_resume_step115815_plus100k_v2`
-- **Global step**: 119 432 (3 617 optimizer updates after the 115 815 resume point)
-- **Architecture**: Tiny Recursive Model ACT V1 (`L_layers=2`, `H_cycles=3`, `L_cycles=4`, hidden size 512, 8 heads, RoPE, bfloat16 activations)
-- **Optimizer**: Adam-atan2 (`beta1=0.9`, `beta2=0.95`, `weight_decay=0.1`, EMA 0.999, global batch size 768)
-- **Dataset builder**: Legacy identifier order (`dataset/build_arc_dataset_legacy.py`) targeting `arc2concept-aug-1000`
-- **Resume provenance**:
-  - `RESUME_CHECKPOINT_PATH` → `/workspace/TinyRecursiveModels/checkpoints/Arc2concept-aug-1000-ACT-torch/trm_arc2_8gpu_resume_plus100k/step_115815`
-  - `RESUME_EXPECTED_STEP` → `115815`
-  - `[resume] initializing train_state.step to 115815` appears in pod logs before training continues
-- **PVC retention**: Latest PVC shards now extend to `step_662428`; earlier 119k shards were pruned after packaging this export.
+What’s included
+- `model.ckpt` — Consolidated PyTorch checkpoint (weights, optimizer, EMA) containing tensors from `step_119432/*`.
+- `COMMANDS.txt`, `COMMANDS_resumed.txt` — Exact `torchrun` invocations (8× H200) and resume flags.
+- `ENVIRONMENT.txt`, `all_config.yaml` — Hydra-resolved configs captured on the pod.
+- `MANIFEST.txt` — Packaging metadata (step, source path, timestamp, sha256).
+- `TRM_COMMIT.txt` — Upstream TinyRecursiveModels SHA (`e7b68717f0a6c4cbb4ce6fbef787b14f42083bd9`).
+- `dataset-metadata.json` — Kaggle manifest retained for parity (legacy identifier mapping), plus cached W&B CSV/summary.
 
-## Files Included
-| Path | Description |
-| --- | --- |
-| `model.ckpt` | Consolidated PyTorch checkpoint (optimizer, EMA, and weights) containing `step_119432/*` tensors. SHA-256: `2bc8bb3a5a85cd73e169a6fd285f9138427db894bd157edc20e92a58ed8ee33e`. |
-| `COMMANDS.txt` / `COMMANDS_resumed.txt` | Torch distributed launch (8 × H200) showing the resume flags and dataset path. |
-| `ENVIRONMENT.txt` | Hydra-resolved configuration captured on CoreWeave after overlays. |
-| `MANIFEST.txt` | Packaging metadata (checkpoint step, source path, timestamp, sha256). |
-| `TRM_COMMIT.txt` | Upstream TinyRecursiveModels Git SHA (`e7b68717f0a6c4cbb4ce6fbef787b14f42083bd9`). |
-| `all_config.yaml` | Structured config snapshot exported alongside the checkpoint. |
-| `dataset-metadata.json` | Kaggle dataset manifest (kept for parity with previous releases). |
+Kaggle status (candid)
+- We did not achieve a working Kaggle submission from this checkpoint. Despite multiple attempts, both internal evaluator pods and Kaggle notebooks produced duplicate attempts for every ARC task, yielding ≈0.83% pass@1 with no diversity across samples.
+- A separate, community-maintained Kaggle implementation exists and may be discoverable on Kaggle by searching for TRM ARC-AGI-2 inference. That work is independent; this release does not mirror their kernel or guarantees.
+- Please treat this model card as a transparent record of what did and did not work in our pipeline rather than a polished Kaggle solution.
 
-## Evaluation Status
-- **Validation (CoreWeave pod evaluator, legacy mapping)**: `pass@1 = 0.83 %`, identical scores for pass@2/5/10/100 because samples were duplicates. Mean token accuracy ≈ 70.1 %, `train/lm_loss` ≈ 0.134 at resume, `all/lm_loss` ≈ 1.56.
-- **Kaggle inference notebook (test split)**: Also produced 259/259 duplicate attempts, yielding 0.83 % pass@1 and no leaderboard improvement. The issue remains unresolved; do not submit this checkpoint to Kaggle until the sampler divergence is fixed.
-- **Copy-mode diagnostics** (`scripts/debug_eval_cpu.py` in legacy mode): 0/120 grid matches (consistent with earlier baselines).
+Why our Kaggle pipeline is broken right now
+- Unique-candidate guard disabled: A production overlay path bypassed the uniqueness filter, allowing identical candidates to pass through even when `ARC_SAMPLING_COUNT>1`.
+- Sampler settings not propagating: Environment variables like `ARC_SAMPLING_COUNT`, `ARC_SAMPLING_MODE=sample`, and temperature sometimes failed to reach the evaluator process under our overlays.
+- Multinomial fallback may be degenerate: With the above misconfigurations, sampling often collapsed to identical outputs across K attempts.
+- Limited attempt-level telemetry: Our evaluator builds aggregated only summary stats; we lacked per-attempt logits/strings on GPU, and Kaggle kernels restricted richer logging. As a result we could not visualize candidate selection or diagnose collapse directly inside the kernels.
+- Identifier alignment: This checkpoint uses the legacy ARC identifier mapping (training-time choice). Kaggle evaluation notebooks increasingly assume the sorted mapping. Without remapping or dedicated compatibility code, comparisons are brittle.
 
-The metrics bundled here are sufficient to reproduce our internal dashboards without requiring live W&B access. If you have Weights & Biases credentials, the run is listed under `trm_arc2_8gpu_resume_step115815_plus100k_v2` in project `trm-arc2`; the first logged step after resume exceeds 115 815, confirming the guard executed.
+What we tried (and why it wasn’t enough)
+- Relaunched controlled eval pods with explicit sampling vars (`ARC_SAMPLING_COUNT>1`, `ARC_SAMPLING_MODE=sample`) and verified resume guard messages in logs — duplicates persisted.
+- Instrumented a CPU debugger (`scripts/debug_eval_cpu.py --samples 8 --log-attempts`) and ran on GPU-backed pods where possible — candidate diversity remained near zero.
+- Raised temperatures and experimented with multinomial/top-k settings — diversity did not improve under the broken overlay path.
+- Prepared a Kaggle dataset and notebooks for this checkpoint — kernels executed end-to-end but emitted duplicate attempts and stalled at ≈0.83% pass@1.
 
-## Inference & Reproduction
+Guidance if you still want to evaluate
+- This artifact is best used to replicate our internal numbers or to help debug sampler behavior — not for immediate leaderboard submissions.
+- If you build your own evaluator:
+  - Re-enable a strict unique-candidate guard before scoring.
+  - Ensure sampling environment variables propagate into the inference process.
+  - Capture per-attempt outputs and logits for diversity diagnostics.
+  - Be explicit about identifier mapping (legacy vs sorted) and ensure your dataset builder matches the checkpoint’s training-time choice.
+
+Reproduction snippets
 ```python
 from huggingface_hub import hf_hub_download
 import torch
 
-ckpt_path = hf_hub_download("seconds0/trm-arc2-8gpu", "model.ckpt")
+ckpt_path = hf_hub_download("seconds-0/trm-arc2-8gpu", "model.ckpt")
 state = torch.load(ckpt_path, map_location="cpu")
 print(state["hyperparameters"]["arch"]["hidden_size"])  # 512
 ```
 
-To recreate the CoreWeave launch:
-```bash
+```
+# CoreWeave resume (reference only)
 kubectl apply -f infra/kubernetes/trm-train-8gpu-resume.yaml
-# Ensure ConfigMaps trm-common-script, trm-pyshim-cm, and trm-eval-overlay are applied first.
+# Requires configmaps: trm-common-script, trm-pyshim-cm, trm-eval-overlay-cm
 ```
-Before submitting jobs, verify:
-1. `RESUME_CHECKPOINT_PATH` points to the 115 815 shard.
-2. `[resume] initializing train_state.step to 115815` appears once training boots.
-3. The first W&B point is ≥115 815 with `train/lm_loss` ≈ 0.13.
 
-## Known Gaps & Next Steps
-1. **Sampler instability** – Deduplicate sampler outputs before retrying Kaggle submissions.
-2. **Identifier remapping** – Remains legacy-only; switching to sorted identifiers requires remapping or finetuning.
-3. **W&B rehydration** – Set `WANDB_API_KEY` locally if you need fresh metrics; the release ships cached configs only.
+Resume guard signals to expect
+- `RESUME_CHECKPOINT_PATH=/workspace/TinyRecursiveModels/checkpoints/Arc2concept-aug-1000-ACT-torch/trm_arc2_8gpu_resume_plus100k/step_115815`
+- `RESUME_EXPECTED_STEP=115815`
+- Pod logs contain: `[resume] initializing train_state.step to 115815` before training proceeds.
+
+Known issues and next steps
+- Sampler instability and duplicate attempts — must restore uniqueness guards, verify env propagation, and re-check multinomial sampling.
+- Identifier mapping mismatch — this release remains “legacy”; remapping or finetuning would be needed for sorted-mapping evaluations.
+- Limited visibility — add candidate-level telemetry and visualizations before making claims about ARC performance.
+
+Ethics, license, and intended use
+- MIT license. Research and educational use encouraged. Do not claim leaderboard results from this artifact without independently validating evaluation protocol and candidate diversity.
 
-Please cite the Tiny Recursive Models paper and ARC Prize 2025 when using this checkpoint. Contributions, bug reports, and sampler fixes are welcome via the repository issues.
+Acknowledgements
+- Built on the Tiny Recursive Models codebase and evaluated against ARC Prize 2025 materials. Thanks to community contributors who have shared working Kaggle kernels; while we do not link to third-party work here, those resources can be found on Kaggle.