add: ADR-003 Future-as-Label demo — detailed implementation plan with research validation

docs/ADR-003-future-as-label.md

@@ -0,0 +1,445 @@
+# ADR-003: Future-as-Label Demo on Olist
+
+**Status:** Proposed  
+**Date:** 2026-05-02  
+**Author:** Rafael Ferraz  
+**Context:** V4.2 concluded. Next experiment: implement the Future-as-Label paradigm (arXiv 2601.06336) as a proof-of-concept on the Olist e-commerce dataset to demonstrate that GRPO with a proper scoring rule reward produces calibrated probability estimates.
+
+---
+
+## 1. Context
+
+### What is Future-as-Label?
+
+The **Foresight Learning** paradigm (Turtel et al., 2601.06336) trains LLMs to produce calibrated probability estimates by:
+
+1. **Temporal masking:** At prediction time `t`, the model only sees information timestamped ≤ `t` (even when training offline on resolved events)
+2. **Terminal reward via proper scoring rule:** GRPO reward is assigned only after binary outcome resolution — no intermediate rewards
+3. **Calibration pressure:** A proper scoring rule penalizes overconfidence more than appropriate uncertainty, creating an intrinsic pressure toward calibrated outputs that SFT cannot replicate
+
+**Paper results (Qwen3-32B, 160 GRPO steps):**
+- Brier score: 0.2472 → **0.1793** (−27%)
+- ECE: 0.2175 → **0.1042** (halved)
+- Outperforms Qwen3-235B base (7× larger) on calibration
+
+### Why this matters for Tucano2-Commerce
+
+The V4.2 project showed GRPO can teach format-following (extraction +29.5%) but struggles with reasoning tasks (SQL −15.6%). Probability calibration is fundamentally different:
+- The model doesn't need to "know" new facts — it needs to express uncertainty correctly
+- The reward signal is dense (every sample gets a continuous score based on the probability)
+- Brier/log score rewards are mathematically guaranteed to be optimized by the true probability
+
+This tests whether GRPO on a 0.5B model can learn **calibration** — a capability orthogonal to format-following.
+
+### Infrastructure continuity
+
+| Component | V4.2 | FAL Demo | Change? |
+|---|---|---|---|
+| Model | Tucano2-qwen-0.5B-Instruct | Same | No |
+| Framework | Unsloth + TRL 0.24.0 | Same | No |
+| Hardware | L4 (24GB) | Same | No |
+| Data source | 1,480 custom prompts | Olist HF dataset | New |
+| Reward | 4 task-specific functions | Single Brier/log score | Simpler |
+| Output format | JSON/text/SQL/notification | Single probability decimal | Simpler |
+
+---
+
+## 2. Decisions
+
+### Decision 1: Use Brier Score reward (not log score)
+
+**Paper uses:** Log score: `y·log(p) + (1−y)·log(1−p)`  
+**We use:** Brier score: `1 − (p − y)²`
+
+**Rationale:**
+- Both are proper scoring rules — uniquely minimized by the true probability
+- Brier score is bounded [0, 1] — no risk of −∞ gradient explosions near p=0 or p=1
+- Log score requires careful clamping (paper uses [0.001, 0.999]) and can still produce extreme gradients
+- For a 0.5B model that may output poorly calibrated probabilities early in training, bounded rewards are safer
+- V4.2 validated that bounded reward functions work well with our GRPO setup
+
+**Consequence:** Slightly weaker calibration pressure at extremes (Brier penalizes p=0.99, y=0 as 0.98², log score penalizes as −4.6). Acceptable for a PoC where the goal is "does it improve at all."
+
+**Alternative considered:** Use log score with aggressive clamping. Rejected because it adds a failure mode (gradient explosion) with minimal upside for a demo.
+
+### Decision 2: Three prediction tasks (repurchase, review, delivery)
+
+**Rationale:**
+- Tests whether calibration transfers across different prediction domains
+- Ensures the model learns general probability estimation, not just one base rate
+- Each task has different characteristics:
+  - **Repurchase (Task A):** Low base rate (~15-20%), model must learn to be appropriately pessimistic
+  - **Review (Task B):** High base rate (~77%), model must learn to be appropriately optimistic
+  - **Delivery (Task C):** Medium-high base rate (~85%), model must distinguish nuanced factors
+- Stratified split ensures eval has balanced representation
+
+**Consequence:** Smaller per-task sample size in eval (~67 samples each). Sufficient for Brier/ECE computation but may not reach statistical significance on Wilcoxon per-task.
+
+### Decision 3: Temporal split (not random split)
+
+**Rationale:**
+- Core requirement of Future-as-Label: no temporal leakage
+- Train on orders before 2018-06-01, eval on 2018-06-01 to 2018-08-31
+- This simulates real deployment: model trained on historical data, evaluated on "future" data it hasn't seen
+- Random split would allow the model to memorize patterns from concurrent orders
+
+**Consequence:** Distribution shift between train and eval (seasonality, seller changes). This is REALISTIC — calibration should be robust to temporal shift. If it isn't, that's a valid negative result.
+
+### Decision 4: Short completion length (32-64 tokens)
+
+**Rationale:**
+- Output is just "PROB: 0.73" (≈10 tokens including reasoning)
+- With MAX_COMPLETION_LENGTH=32, each generation is fast → step time drops dramatically
+- V4.2 ran at 33s/step with 512-token completions; at 32 tokens expect ~5-8s/step
+- 800 steps × 8s = ~1.8 hours (vs. 22h for V4.2)
+- No thinking/reasoning required — just a number
+
+**Consequence:** Model can't "show its work." It outputs only the probability. This is intentional — the paper shows that even without chain-of-thought, proper scoring rule rewards improve calibration. We're testing the minimal case.
+
+**Risk:** If the model needs reasoning tokens to produce meaningful probabilities, 32 tokens won't be enough. Mitigation: if parse rate is <50% at 32 tokens, increase to 64 or 128.
+
+### Decision 5: NUM_GENERATIONS=16 (not paper's K=4)
+
+**Rationale:**
+- Paper uses K=4 trajectories per event on 32B model
+- Our 0.5B model + short completions has massive VRAM headroom
+- V4.2 validated G=16 on this exact hardware and model
+- More rollouts = more reward variance per prompt = stronger GRPO signal
+- At 32-token completions, 16 rollouts use negligible memory
+
+**Consequence:** 4× more samples per prompt than the paper. May accelerate learning or may waste compute if all 16 produce the same probability (zero variance). Monitored via `frac_reward_zero_std`.
+
+### Decision 6: Add format bonus (+0.1) for parseable output
+
+**Rationale:**
+- The #1 risk with a 0.5B model is it never produces a parseable probability
+- If parse_rate = 0%, reward is always 0.0, GRPO has no signal
+- Adding `+0.1` bonus when ANY parseable probability is found gives a baseline gradient toward format compliance
+- This is the same "staged reward" approach from V4.2 (format reward converges first, then task reward improves)
+- V4.2's extraction task showed this works: model learned JSON format first, then field values
+
+**Consequence:** The reward is no longer a pure proper scoring rule. It becomes `0.1*(parseable) + 0.9*(1 - (p-y)²)`. The format bonus dominates early in training (when most outputs are unparseable), then Brier dominates once format is learned. The calibration incentive is preserved because the Brier component still constitutes 90% of the achievable reward.
+
+### Decision 7: Use the HF pre-processed Olist dataset
+
+**Source:** `miminmoons/olist-ecommerce-for-delivery-and-review-prediction` (~100K rows)
+
+**Rationale:**
+- Pre-joined columns: `delivery_delay_hours`, `time_to_ship_hours`, `purchase_count`, `avg_review_score` — saves data engineering
+- Has `customer_unique_id` for repurchase computation
+- Has `order_purchase_timestamp` for temporal split
+- Has `review_score` for binary label
+
+**Limitation:** No raw `order_estimated_delivery_date` column. `delivery_delay_hours` is pre-computed (negative = early). For Task C (delivery_ontime), the binary label becomes: `int(delivery_delay_hours <= 0)`.
+
+**Consequence:** Task C context cannot include the estimated delivery date explicitly — instead we include the order metadata that predicts delivery success (weight, dimensions [not available], seller state, customer state, shipping time). This is actually a harder prediction task (no "cheat" feature), which makes it more interesting.
+
+### Decision 8: LR=5e-6 for first run, with note to test 1e-6
+
+**Rationale:**
+- V4.2 validated 5e-6 on this exact model and framework
+- Paper doesn't specify LR (uses TRL defaults, likely 1e-6)
+- Higher LR extracts signal faster from small datasets (V4.2 lesson)
+- 800 steps is modest — want to see signal quickly
+
+**Risk:** V4.2 showed entropy collapse at ~1.5 epochs with 5e-6. With ~1500 training prompts and batch_size=2, one epoch = 750 steps. At 800 steps we're at ~1.07 epochs — right at the edge. May need to reduce to 600 steps or use 1e-6 with more steps.
+
+---
+
+## 3. Consequences: Expected Results
+
+### Optimistic scenario (model learns calibration)
+
+| Metric | Base | Tuned | Improvement |
+|---|---|---|---|
+| Brier score | ~0.20-0.25 | ~0.18-0.22 | −10-15% |
+| ECE | ~0.15-0.25 | ~0.10-0.15 | −30-40% |
+| Parse rate | ~30-60% | ~85-95% | Format learned |
+| Wilcoxon p | — | <0.05 | Significant |
+
+**Why optimistic is plausible:**
+- The paper shows even 40 GRPO steps improve calibration on 32B
+- Our V4.2 showed GRPO improves format-following on 0.5B (extraction +29.5%)
+- Probability estimation is simpler than JSON schema compliance
+- Dense reward signal (every sample gets meaningful reward) vs. sparse in V4.2
+
+### Realistic scenario (partial learning)
+
+| Metric | Base | Tuned | Notes |
+|---|---|---|---|
+| Brier score | ~0.22 | ~0.20 | Modest improvement |
+| ECE | ~0.20 | ~0.15 | Some calibration gain |
+| Parse rate | ~40% | ~80% | Format learned, calibration partial |
+| Wilcoxon p | — | ~0.05-0.10 | Borderline significance |
+
+**Most likely outcome:** Model first learns to output parseable probabilities (format), then slowly moves toward base rates. May not achieve fine-grained calibration (distinguishing 0.7 from 0.8) but should distinguish "likely" from "unlikely" events.
+
+### Pessimistic scenario (model fails)
+
+| Metric | Base | Tuned | Notes |
+|---|---|---|---|
+| Brier score | ~0.22 | ~0.24 | Worse (overconfident) |
+| ECE | ~0.20 | ~0.25 | Degraded |
+| Parse rate | <30% | <50% | Never learns format |
+
+**When this happens:**
+- Parse rate stays <50% despite format bonus → increase MAX_COMPLETION_LENGTH to 128 or add few-shot examples
+- Model outputs 0.50 for everything → degenerate solution (safe under Brier), no calibration
+- All rollouts produce same probability → `frac_reward_zero_std=1.0`, zero GRPO signal
+
+### What we learn regardless of outcome
+
+| Outcome | Scientific conclusion |
+|---|---|
+| Calibration improves | GRPO + proper scoring rule works at 0.5B scale |
+| Format learned but no calibration | 0.5B can't reason about probabilities, but can format-follow |
+| Nothing works | 0.5B is below minimum viable scale for probabilistic reasoning with GRPO |
+| Task A improves but B/C don't | Calibration learning depends on base rate / task structure |
+
+---
+
+## 4. Implementation Plan
+
+### Phase 1: Data Pipeline (`olist_fal_dataset.py`)
+
+```python
+# Pseudocode — full implementation needed
+from datasets import load_dataset
+import pandas as pd
+from datetime import timedelta
+
+TRAIN_CUTOFF = "2018-06-01"
+EVAL_END = "2018-08-31"
+
+ds = load_dataset("miminmoons/olist-ecommerce-for-delivery-and-review-prediction", split="train")
+df = ds.to_pandas()
+df["order_purchase_timestamp"] = pd.to_datetime(df["order_purchase_timestamp"])
+
+# ── Task A: Repurchase within 90 days ────────────────────────────────────
+# For each order, check if same customer_unique_id has another order within 90 days
+# Context: customer history up to this order (categories, values, scores)
+# Label: 1 if repurchase within 90 days, 0 otherwise
+
+# ── Task B: Positive review (score ≥ 4) ──────────────────────────────────
+# Context: order metadata at fulfillment (category, freight, delivery delta, seller)
+# Label: int(review_score >= 4)
+# NOTE: Do NOT include review text or score in context
+
+# ── Task C: On-time delivery ─────────────────────────────────────────────
+# Context: order-placed metadata (seller/customer state, product info, shipping time)
+# Label: int(delivery_delay_hours <= 0)  # negative = early
+# NOTE: Do NOT include delivery_delay_hours in context (that's the answer)
+
+# ── Temporal split ────────────────────────────────────────────────────────
+train_df = df[df["order_purchase_timestamp"] < TRAIN_CUTOFF]
+eval_df = df[(df["order_purchase_timestamp"] >= TRAIN_CUTOFF) & 
+             (df["order_purchase_timestamp"] <= EVAL_END)]
+
+# ── Output: ~500 per task in train, ~67 per task in eval ─────────────────
+# Save as JSONL with schema: {"task", "prompt", "outcome", "split"}
+```
+
+**Key implementation details:**
+
+1. **Task A repurchase computation:** Group by `customer_unique_id`, sort by timestamp, for each order check if next order from same customer is within 90 days. Orders with no subsequent order in the dataset get label 0 (conservative — some may be censored).
+
+2. **Context construction:** Each task gets a natural language paragraph describing the relevant features. NO inclusion of the target variable or any post-outcome information.
+
+3. **Temporal leakage prevention:**
+   - Task A: Only include customer history BEFORE the current order
+   - Task B: Include delivery delta (occurs before review) but NOT the review score
+   - Task C: Include order-placed info only (NOT delivery outcome or time_to_ship)
+
+4. **Sampling strategy:** Stratify by task AND outcome to ensure balanced base rates in eval. Training can use natural distribution.
+
+### Phase 2: Training Notebook (`fal_grpo_training.ipynb`)
+
+Fork from V4.2 with these minimal changes:
+
+```python
+# ── Reward function ──────────────────────────────────────────────────────
+def reward_fal(completions, outcome, **kwargs):
+    """
+    Brier score reward with format bonus.
+    Range: [0.0, 1.0]. Higher = better calibration.
+    """
+    rewards = []
+    for completion, y in zip(completions, outcome):
+        if isinstance(completion, list):
+            text = completion[-1]["content"] if completion else ""
+        else:
+            text = str(completion)
+        
+        # Parse probability
+        match = re.search(r'PROB:\s*([01]?\.\d+|[01])', text)
+        if match is None:
+            # Fallback: last decimal in [0,1] range
+            matches = re.findall(r'\b(0\.\d{1,3}|1\.00?|0)\b', text)
+            p_raw = float(matches[-1]) if matches else None
+        else:
+            p_raw = float(match.group(1))
+        
+        if p_raw is None:
+            rewards.append(0.0)  # No parse → no reward
+        else:
+            p = max(0.001, min(0.999, p_raw))
+            y_val = float(y)
+            brier = 1.0 - (p - y_val) ** 2  # Range [0, 1]
+            format_bonus = 0.1  # Parseable output bonus
+            rewards.append(min(1.0, format_bonus + 0.9 * brier))
+    
+    return rewards
+
+# ── Config changes from V4.2 ─────────────────────────────────────────────
+MAX_COMPLETION_LENGTH  = 32    # "PROB: 0.73" = ~10 tokens
+MAX_STEPS              = 800
+EVAL_STEPS             = 50
+EARLY_STOPPING_PATIENCE = 10
+NUM_GENERATIONS        = 16
+LEARNING_RATE          = 5e-6
+TEMPERATURE            = 1.0
+BETA                   = 0.0
+SCALE_REWARDS          = False
+
+# ── System prompt ────────────────────────────────────────────────────────
+SYSTEM_FAL = (
+    "You are a Brazilian e-commerce analyst. "
+    "Given order information, estimate the probability of the stated outcome. "
+    "Answer with ONLY: PROB: followed by a number between 0.00 and 1.00. "
+    "Example: PROB: 0.73"
+)
+
+# ── Dataset loading ──────────────────────────────────────────────────────
+# Load fal_train.jsonl, format as {"prompt": [...messages...], "outcome": 0/1}
+```
+
+### Phase 3: Evaluation (`evaluate_fal.py`)
+
+```python
+# 1. Generate predictions from both base and tuned models
+# 2. Parse PROB values (fallback to 0.5 if unparseable)
+# 3. Compute per-task and overall metrics:
+
+def compute_metrics(predictions, outcomes):
+    """
+    predictions: list of floats [0, 1]
+    outcomes: list of ints {0, 1}
+    """
+    import numpy as np
+    
+    preds = np.array(predictions)
+    outs = np.array(outcomes, dtype=float)
+    
+    # Brier score (lower = better)
+    brier = np.mean((preds - outs) ** 2)
+    
+    # ECE with 10 bins
+    n_bins = 10
+    bin_boundaries = np.linspace(0, 1, n_bins + 1)
+    ece = 0.0
+    for i in range(n_bins):
+        mask = (preds >= bin_boundaries[i]) & (preds < bin_boundaries[i+1])
+        if mask.sum() > 0:
+            bin_acc = outs[mask].mean()
+            bin_conf = preds[mask].mean()
+            ece += mask.sum() * abs(bin_acc - bin_conf)
+    ece /= len(preds)
+    
+    # Base rate Brier (always predict marginal probability)
+    base_rate = outs.mean()
+    brier_baseline = np.mean((base_rate - outs) ** 2)
+    
+    return {
+        "brier": float(brier),
+        "ece": float(ece),
+        "brier_baseline": float(brier_baseline),
+        "base_rate": float(base_rate),
+        "brier_skill": float(1 - brier / brier_baseline),  # >0 means better than baseline
+    }
+
+# 4. Plot calibration curve
+# 5. Wilcoxon signed-rank test on per-sample Brier scores
+```
+
+### Phase 4: Deliverables
+
+```
+fal_demo/
+├── olist_fal_dataset.py        # Data pipeline script
+├── fal_train.jsonl             # ~1500 training samples (500/task)
+├── fal_eval.jsonl              # ~200 eval samples (67/task, stratified)
+├── fal_grpo_training.ipynb     # Training notebook
+├── evaluate_fal.py             # Evaluation + plotting
+├── calibration_curve.png       # Visual result
+└── fal_results.json            # All metrics + Wilcoxon p-value
+```
+
+---
+
+## 5. Risk Mitigation
+
+| Risk | Probability | Impact | Mitigation |
+|---|---|---|---|
+| Parse rate < 50% | Medium | Training fails (no signal) | Add few-shot examples to system prompt; increase MAX_COMPLETION_LENGTH to 64-128 |
+| All rollouts same probability | Medium | GRPO zero-signal | Add noise to prompt (randomize feature order); check `frac_reward_zero_std` |
+| Model always outputs 0.5 | Low-Medium | Degenerate equilibrium | Log score would break this (infinite penalty for 0.5 when y=0/1), but Brier doesn't. Accept partial result. |
+| Entropy collapse at step 600+ | Medium | Same as V4.2 | Early stopping patience=10 (500 steps). V4.2 peaked at 1.5 epochs = ~750 steps on similar-sized data. |
+| Data leakage in Task A | Low | Inflated results | Rigorous temporal check: only count repurchases AFTER the order date, never include future context |
+| Censoring in Task A (no repurchase observed but data ends) | Medium | Noisy labels | Exclude orders within 90 days of dataset end (2018-08-31 - 90d = 2018-06-02). Only label orders where full 90-day window is observable. |
+
+---
+
+## 6. Timeline Estimate
+
+| Phase | Effort | Duration |
+|---|---|---|
+| Data pipeline (Task 1) | 2-3 hours | Half day |
+| Training notebook setup (Task 3) | 1-2 hours | Half day |
+| Training run (800 steps) | ~2 hours | Automatic |
+| Evaluation + plotting (Task 4) | 1-2 hours | Half day |
+| **Total** | **~6-8 hours active work** | **1-2 days** |
+
+---
+
+## 7. Success Criteria
+
+**Minimum viable result (ship as demo):**
+- Parse rate ≥ 70% on both base and tuned models
+- Tuned Brier score < Base Brier score (any improvement)
+- Calibration curve shows tuned model closer to diagonal than base
+
+**Strong result (publishable finding):**
+- Parse rate ≥ 90% on tuned model
+- Brier improvement ≥ 10% (e.g., 0.22 → 0.20)
+- ECE improvement ≥ 20%
+- Wilcoxon p < 0.05
+- Calibration curve visually demonstrates improved calibration across all bins
+
+**Negative result (still informative):**
+- If model never learns format: "0.5B is below minimum viable scale for probability estimation"
+- If format learned but no calibration: "Format-following ≠ reasoning at this scale"
+- If only one task improves: "Calibration learning is task-structure dependent"
+
+---
+
+## 8. References
+
+| Paper | Key finding | Relevance |
+|---|---|---|
+| Future-as-Label (2601.06336) | GRPO + log score → 27% Brier improvement on 32B | The method we're implementing |
+| Dr. GRPO (2503.20783) | β=0, scale_rewards=False for rule-based rewards | Config choices validated |
+| V4.2 Final Report | GRPO works for format-learning at 0.5B, peaks at ~1.5 epochs | Expectations and step budget |
+| Brier (1950) | Proper scoring rule: uniquely minimized by true probability | Theoretical guarantee |
+| Gneiting & Raftery (2007) | Proper scoring rules and calibration | Why this reward works |
+
+---
+
+## 9. Relationship to V4.2
+
+This is **not V4.3** — it's a separate experiment testing a different capability (calibration vs. format-following). It shares infrastructure but has a completely different:
+- Dataset (Olist prediction tasks vs. commerce 4-task)
+- Reward function (single Brier score vs. 4 task-specific rewards + GDPO + IWU)
+- Output format (decimal probability vs. JSON/text/SQL/notification)
+- Research question ("can GRPO teach calibration at 0.5B?" vs. "how much does GRPO improve format-following?")
+
+If successful, this demonstrates that the Tucano2-Commerce GRPO methodology generalizes beyond format-following to probabilistic reasoning — a significantly stronger claim for the project's overall contribution.