Polish model card prose: professional Kaggle status, limitations, guidance

README.md

@@ -39,49 +39,46 @@ model-index:
 
 # Tiny Recursive Models — ARC-AGI-2 (8× H200 Resume, Step 119 432)
 
-Summary
-- This is an 8× H200 resume snapshot at global step 119 432 from run `trm_arc2_8gpu_resume_step115815_plus100k_v2` (TinyRecursiveModels commit `e7b68717`).
-- It packages one consolidated `model.ckpt` plus the exact configs used on CoreWeave. Integrity hash: `2bc8bb3a5a85cd73e169a6fd285f9138427db894bd157edc20e92a58ed8ee33e`.
-- Evaluation quality is currently poor (ARC public evaluation pass@1 ≈ 0.83%) due to duplicate candidate generation after the resume. Until we address sampler issues, this artifact is provided for transparency and reproduction, not for leaderboard use.
+Overview
+- 8× H200 resume snapshot at global step 119 432 from run `trm_arc2_8gpu_resume_step115815_plus100k_v2` (TinyRecursiveModels commit `e7b68717`).
+- Consolidated `model.ckpt` with accompanying configuration and provenance files. Integrity hash: `2bc8bb3a5a85cd73e169a6fd285f9138427db894bd157edc20e92a58ed8ee33e`.
+- Current evaluation on the ARC public evaluation split is approximately 0.83% pass@1, attributable to duplicate candidate generation in our evaluation pipeline after the resume. This release is intended for reproducibility and analysis rather than leaderboard submissions.
 
-Why the “119 434” label sometimes appears
-- Internal tracking briefly referred to a “step 119 434” snapshot. The durable shard that survived PVC pruning is `step_119432`. W&B logs confirm the resume guard initialized at step 115 815 and advanced into the 119k block; no distinct 119 434 shard remains.
+About “119 434” vs “119 432”
+- Internal tracking referenced “step 119 434”; the persistent shard is `step_119432`. W&B logs confirm the resume guard initialized at step 115 815 and advanced into the 119k block. No distinct 119 434 shard remains.
 
-What’s included
-- `model.ckpt` — Consolidated PyTorch checkpoint (weights, optimizer, EMA) containing tensors from `step_119432/*`.
-- `COMMANDS.txt`, `COMMANDS_resumed.txt` — Exact `torchrun` invocations (8× H200) and resume flags.
-- `ENVIRONMENT.txt`, `all_config.yaml` — Hydra-resolved configs captured on the pod.
+Contents
+- `model.ckpt` — Consolidated PyTorch checkpoint (weights, optimizer, EMA) reflecting `step_119432/*`.
+- `COMMANDS.txt`, `COMMANDS_resumed.txt` — Exact `torchrun` invocations (8× H200) and resume parameters.
+- `ENVIRONMENT.txt`, `all_config.yaml` — Hydra-resolved configurations captured on the training pod.
 - `MANIFEST.txt` — Packaging metadata (step, source path, timestamp, sha256).
-- `TRM_COMMIT.txt` — Upstream TinyRecursiveModels SHA (`e7b68717f0a6c4cbb4ce6fbef787b14f42083bd9`).
-- `dataset-metadata.json` — Kaggle manifest retained for parity (legacy identifier mapping), plus cached W&B CSV/summary.
+- `TRM_COMMIT.txt` — Upstream TinyRecursiveModels commit (`e7b68717f0a6c4cbb4ce6fbef787b14f42083bd9`).
+- `dataset-metadata.json` — Kaggle packaging manifest (legacy identifier mapping). W&B CSV/summary are included for convenience.
 
-Kaggle status (candid)
-- We did not achieve a working Kaggle submission from this checkpoint. Despite multiple attempts, both internal evaluator pods and Kaggle notebooks produced duplicate attempts for every ARC task, yielding ≈0.83% pass@1 with no diversity across samples.
-- A separate, community-maintained Kaggle implementation exists and may be discoverable on Kaggle by searching for TRM ARC-AGI-2 inference. That work is independent; this release does not mirror their kernel or guarantees.
-- Please treat this model card as a transparent record of what did and did not work in our pipeline rather than a polished Kaggle solution.
+Kaggle Status
+- This checkpoint is not accompanied by a validated Kaggle submission from our pipeline. In our evaluation path, inference produced duplicate candidates across attempts, yielding ≈0.83% pass@1 on the public evaluation split.
+- A third‑party Kaggle implementation demonstrating TRM inference is available on Kaggle; interested users can locate it by searching the platform. That implementation is independent of this release.
 
-Why our Kaggle pipeline is broken right now
-- Unique-candidate guard disabled: A production overlay path bypassed the uniqueness filter, allowing identical candidates to pass through even when `ARC_SAMPLING_COUNT>1`.
-- Sampler settings not propagating: Environment variables like `ARC_SAMPLING_COUNT`, `ARC_SAMPLING_MODE=sample`, and temperature sometimes failed to reach the evaluator process under our overlays.
-- Multinomial fallback may be degenerate: With the above misconfigurations, sampling often collapsed to identical outputs across K attempts.
-- Limited attempt-level telemetry: Our evaluator builds aggregated only summary stats; we lacked per-attempt logits/strings on GPU, and Kaggle kernels restricted richer logging. As a result we could not visualize candidate selection or diagnose collapse directly inside the kernels.
-- Identifier alignment: This checkpoint uses the legacy ARC identifier mapping (training-time choice). Kaggle evaluation notebooks increasingly assume the sorted mapping. Without remapping or dedicated compatibility code, comparisons are brittle.
+Current Kaggle Limitations
+- Unique‑candidate enforcement: A production overlay inadvertently bypassed the uniqueness filter, allowing duplicates even with `ARC_SAMPLING_COUNT > 1`.
+- Sampler configuration propagation: Variables such as `ARC_SAMPLING_COUNT`, `ARC_SAMPLING_MODE=sample`, and temperature did not reliably reach the evaluator under our overlays.
+- Sampling degeneracy: Under misconfiguration, multinomial sampling collapsed to identical outputs across attempts.
+- Limited attempt‑level telemetry: Evaluators emitted aggregate metrics only; per‑attempt logits and strings were not retained due to GPU runtime constraints and Kaggle logging limits. This prevents visualization of candidate selection and diversity.
+- Identifier mapping: The checkpoint was trained with the legacy identifier mapping, whereas some evaluators assume the sorted mapping. Without remapping or a compatibility layer, comparisons can be brittle.
 
-What we tried (and why it wasn’t enough)
-- Relaunched controlled eval pods with explicit sampling vars (`ARC_SAMPLING_COUNT>1`, `ARC_SAMPLING_MODE=sample`) and verified resume guard messages in logs — duplicates persisted.
-- Instrumented a CPU debugger (`scripts/debug_eval_cpu.py --samples 8 --log-attempts`) and ran on GPU-backed pods where possible — candidate diversity remained near zero.
-- Raised temperatures and experimented with multinomial/top-k settings — diversity did not improve under the broken overlay path.
-- Prepared a Kaggle dataset and notebooks for this checkpoint — kernels executed end-to-end but emitted duplicate attempts and stalled at ≈0.83% pass@1.
+Attempted Mitigations
+- Relaunched controlled evaluator pods with explicit sampling parameters and verified resume‑guard logs; duplicates persisted.
+- Instrumented CPU/GPU debug evaluators (`scripts/debug_eval_cpu.py --samples 8 --log-attempts`); candidate diversity remained near zero.
+- Adjusted temperature and top‑k settings; no material improvement under the broken overlay path.
+- Prepared Kaggle datasets/notebooks and executed end‑to‑end; duplicate attempts persisted and no leaderboard submission was made.
 
-Guidance if you still want to evaluate
-- This artifact is best used to replicate our internal numbers or to help debug sampler behavior — not for immediate leaderboard submissions.
-- If you build your own evaluator:
-  - Re-enable a strict unique-candidate guard before scoring.
-  - Ensure sampling environment variables propagate into the inference process.
-  - Capture per-attempt outputs and logits for diversity diagnostics.
-  - Be explicit about identifier mapping (legacy vs sorted) and ensure your dataset builder matches the checkpoint’s training-time choice.
+Evaluation Guidance
+- Enforce a strict unique‑candidate guard prior to scoring.
+- Validate that sampling environment variables propagate into the inference process.
+- Capture per‑attempt outputs and, where possible, logits to diagnose diversity.
+- Match the identifier mapping to the checkpoint (legacy vs sorted) and use the corresponding dataset builder.
 
-Reproduction snippets
+Reproduction
 ```python
 from huggingface_hub import hf_hub_download
 import torch
@@ -92,23 +89,23 @@ print(state["hyperparameters"]["arch"]["hidden_size"])  # 512
 ```
 
 ```
-# CoreWeave resume (reference only)
+# CoreWeave resume (reference)
 kubectl apply -f infra/kubernetes/trm-train-8gpu-resume.yaml
 # Requires configmaps: trm-common-script, trm-pyshim-cm, trm-eval-overlay-cm
 ```
 
-Resume guard signals to expect
+Resume Guard Signals
 - `RESUME_CHECKPOINT_PATH=/workspace/TinyRecursiveModels/checkpoints/Arc2concept-aug-1000-ACT-torch/trm_arc2_8gpu_resume_plus100k/step_115815`
 - `RESUME_EXPECTED_STEP=115815`
 - Pod logs contain: `[resume] initializing train_state.step to 115815` before training proceeds.
 
-Known issues and next steps
-- Sampler instability and duplicate attempts — must restore uniqueness guards, verify env propagation, and re-check multinomial sampling.
-- Identifier mapping mismatch — this release remains “legacy”; remapping or finetuning would be needed for sorted-mapping evaluations.
-- Limited visibility — add candidate-level telemetry and visualizations before making claims about ARC performance.
+Known Issues and Next Steps
+- Duplicate candidate generation: restore uniqueness enforcement, validate environment propagation, and re‑verify multinomial sampling.
+- Identifier mapping mismatch: this release remains “legacy”; sorted‑mapping evaluations require remapping or fine‑tuning.
+- Observability: add candidate‑level telemetry and visualizations to support reliable evaluation claims.
 
-Ethics, license, and intended use
-- MIT license. Research and educational use encouraged. Do not claim leaderboard results from this artifact without independently validating evaluation protocol and candidate diversity.
+Ethics, License, and Intended Use
+- MIT license. Intended for research and educational use. Users should independently validate evaluation protocols and candidate diversity prior to reporting results.
 
 Acknowledgements
-- Built on the Tiny Recursive Models codebase and evaluated against ARC Prize 2025 materials. Thanks to community contributors who have shared working Kaggle kernels; while we do not link to third-party work here, those resources can be found on Kaggle.
+- Built on the Tiny Recursive Models codebase and evaluated against ARC Prize 2025 materials. A community‑maintained Kaggle implementation demonstrates a functioning TRM inference pipeline and can be located via search on Kaggle.