Upload folder using huggingface_hub

Blog.md

@@ -0,0 +1,58 @@
+# 🏬 ShopManagerEng: Training LLMs to Run a Business
+
+Welcome to **ShopManagerEng**, a Reinforcement Learning (RL) environment designed to test and train Large Language Models (LLMs) on complex, multi-step business operations. 
+
+While LLMs have become incredibly adept at chatting and writing code, evaluating their ability to make strategic, long-term decisions in a dynamic environment remains a challenge. ShopManagerEng tackles this by putting the AI in the shoes of a jewelry shop manager. Dont get into name 'Eng',just typos mistake of 'Env'.
+
+This blog post breaks down how the environment works, its architecture, and how it can be used to train smarter, more strategic agents.
+
+---
+
+## 🎮 The Environment: A 3-Phase Business Simulation
+
+ShopManagerEng is not a simple Q&A benchmark. It is a continuous simulation where the agent must manage inventory, respond to market fluctuations, and negotiate with customers. Every "episode" (a full game loop) consists of three distinct phases:
+
+### Phase 1: 📈 The Market (Supply)
+The agent starts with a limited budget and must decide when to buy raw gold. 
+*   **The Catch:** Gold prices fluctuate. The agent can observe price trends and decide to "wait" for a better price or "buy" immediately. 
+*   **Real-World Noise:** The environment supports a "real" market mode that pulls live gold prices from Yahoo Finance, testing the agent against real-world economic volatility.
+
+### Phase 2: 🏭 The Warehouse (Production)
+Once gold is acquired, the agent must decide what to craft: Rings, Necklaces, or Bracelets.
+*   **The Catch:** The agent must balance raw material costs, labor costs, and market demand. It receives a "demand forecast" (which includes noise) and must make production choices that maximize potential profit without running out of cash or gold.
+
+### Phase 3: 🤝 The Showroom (Sales)
+The final phase tests the agent's negotiation skills.
+*   **The Catch:** Customers make initial offers. The agent can accept, reject, or counter-offer over a maximum of 5 rounds. Push too hard, and the customer walks away (resulting in zero profit). Settle too early, and the agent leaves money on the table.
+
+The agent's final score is a cumulative reward based on how well it navigated all three phases.
+
+---
+
+## 🏗️ Architecture Under the Hood
+
+ShopManagerEng is built on a robust Client-Server architecture, making it highly scalable for Reinforcement Learning tasks.
+
+1.  **The Server (`/server/`):** The core simulation engine runs as a standalone web application (often deployed via Docker to a Hugging Face Space). It tracks the hidden "true" state of the world, handles state transitions, fetches real market data, and manages an SQLite database to log inventory and invoices.
+2.  **The Client (`client.py` & `models.py`):** Provides a clean, typed Python interface. Agents use this client to send actions (e.g., `{"market_action": "buy", "gold_qty": 2.0}`) and receive structured observations in return.
+3.  **The AI Player (`inference.py`):** A script demonstrating how to plug an LLM (like Llama 3) into the environment. It dynamically builds text prompts explaining the current state (e.g., *"Gold is $2000/oz and trending down. You have $1000. Buy or wait?"*) and parses the LLM's text output back into valid game actions.
+4.  **The Training Suite (`/training/`):** The real power of this project lies in its RL training capabilities. Using algorithms like GRPO (Group Relative Policy Optimization), the framework can run massive batches of episodes, evaluate the agent using custom reward functions, and iteratively update the model's weights to forge a master negotiator and supply-chain manager.
+
+### 🛠️ Technical Stack & Data Flow
+
+*   **FastAPI & OpenEnv:** The server is powered by FastAPI, leveraging the `openenv` framework to standardize interactions. This ensures the environment can be easily hosted (e.g., on Hugging Face Spaces) and queried by agents remotely.
+*   **Pydantic Models:** All actions and observations are strongly typed using Pydantic (`models.py`). This guarantees that agents send properly formatted JSON payloads (like `target_price_usd` or `inventory_urgent` flags) and receive structured state data.
+*   **State Persistence:** A built-in SQLite store (`sqlite_store.py`) tracks the complex state across episodes, maintaining ledgers for cash, gold (in troy ounces and grams), and specific inventory items (rings, necklaces, bracelets).
+*   **yfinance Integration:** For the "real" market mode, the environment dynamically fetches live GC=F (Gold Futures) ticker data via `market_data.py`, injecting true market volatility into the simulation rather than relying purely on synthetic random walks.
+*   **GRPO Training:** The training pipeline (`train_jewelry_grpo.py`) utilizes Group Relative Policy Optimization, an advanced RL technique designed to stabilize training and improve sample efficiency when teaching LLMs complex, multi-step tasks. Custom reward functions (`rewards.py`) evaluate and guide the model's behavior.
+
+---
+
+## 🚀 Why This Matters
+
+As we move towards autonomous AI agents, we need benchmarks that go beyond static knowledge retrieval. ShopManagerEng evaluates an agent's ability to:
+*   **Plan Long-Term:** A bad purchase in Phase 1 ruins the negotiation in Phase 3.
+*   **Handle Uncertainty:** Dealing with noisy demand forecasts and volatile live markets.
+*   **Negotiate:** Understanding margins and customer psychology.
+
+Whether you are testing the out-of-the-box reasoning of a new foundational model or fine-tuning a specialized RL agent, ShopManagerEng provides a rich, complex sandbox to push AI capabilities forward.

Dockerfile

@@ -59,6 +59,11 @@ FROM ${BASE_IMAGE}
 
 WORKDIR /app
 
+# Install curl (for health checks in start.sh) and streamlit
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends curl && \
+    rm -rf /var/lib/apt/lists/*
+
 # Copy the virtual environment from builder
 COPY --from=builder /app/env/.venv /app/.venv
 
@@ -71,11 +76,16 @@ ENV PATH="/app/.venv/bin:$PATH"
 # Set PYTHONPATH so imports work correctly
 ENV PYTHONPATH="/app/env:$PYTHONPATH"
 
+# Install streamlit into the existing venv
+RUN pip install --no-cache-dir streamlit
+
 # Health check
 HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
     CMD curl -f http://localhost:8000/health || exit 1
 
-# Run the FastAPI server
-# The module path is constructed to work with the /app/env structure
+# Make startup script executable
+RUN chmod +x /app/env/start.sh
+
+# Run both API server (background) + Streamlit UI (foreground)
 ENV ENABLE_WEB_INTERFACE=true
-CMD ["sh", "-c", "cd /app/env && uvicorn server.app:app --host 0.0.0.0 --port 8000"]
+CMD ["/app/env/start.sh"]

README.md

@@ -5,12 +5,15 @@ colorFrom: green
 colorTo: blue
 sdk: docker
 pinned: false
-app_port: 8000
+app_port: 7860
 base_path: /web
 tags:
   - openenv
 ---
 
+Link of the environment: https://huggingface.co/spaces/hard007ik/ShopManagerEng
+Link of Blog.md: https://huggingface.co/spaces/hard007ik/ShopManagerEng/tree/main/Blog.md
+
 # Jewelry Shop Manager — RL Environment
 
 A reinforcement learning environment simulating a **jewelry shop management** pipeline. An AI agent navigates three sequential phases — buying raw materials, selecting products to craft based on demand, and negotiating sales — to maximize profit.

inference.py

@@ -71,7 +71,8 @@ SYSTEM_PROMPT = textwrap.dedent(
     Respond: "ring", "necklace", or "bracelet".
 
     ## Phase 3: SHOWROOM (negotiate)
-    The customer makes an offer; if you counter, they raise it ~5% per round,
+    you makes an offer; if customer counter by telling less price from your offer, you can drop price about ~3-5% per round but make sure to not sell when loss is happening also bring max profit,
+    if customer says less price then your first told price then you have to say the price that lesser than the price you told before but more that the customer told price
     up to 5 rounds. After 5 rounds with no acceptance, the customer leaves
     (no phase-3 reward). Reject also gives 0 phase-3 reward.
     Respond: "I accept" or a counter like "How about $X?". NEVER explicitly reject.

pyproject.toml

@@ -21,6 +21,7 @@ dependencies = [
     "yfinance>=0.2.40",
     "python-dotenv>=1.0.0",
     "requests>=2.28.0",
+    "streamlit>=1.30.0",
 ]
 
 [project.optional-dependencies]
@@ -29,6 +30,26 @@ dev = [
     "pytest-cov>=4.0.0",
 ]
 
+# GRPO training stack. Install only on a GPU host (vLLM is GPU-only).
+#   pip install -e '.[train]'
+# Mirrors openenv-course Module 5 versions. trl>=0.17 is the cutoff that
+# introduced trl.experimental.openenv.generate_rollout_completions.
+train = [
+    "trl>=0.17.0",
+    "transformers>=4.46.0",
+    "datasets>=2.20.0",
+    "accelerate>=1.0.0",
+    # Pin to TRL's supported window. vllm 0.19+ changes the sampling-logprob
+    # path and breaks GRPO importance sampling (ratio collapses to ~0,
+    # grad_norm goes to 0, model stops learning). Keep <=0.18.0 until TRL
+    # bumps its supported range.
+    "vllm>=0.11.0,<=0.18.0",
+    "trackio",
+    "torch>=2.4.0",
+    # Local plotting of loss + reward curves (hackathon submission evidence).
+    "matplotlib>=3.7",
+]
+
 [project.scripts]
 # Server entry point - enables running via: uv run --project . server
 # or: python -m ShopManagerEng.server.app
@@ -36,5 +57,5 @@ server = "ShopManagerEng.server.app:main"
 
 [tool.setuptools]
 include-package-data = true
-packages = ["ShopManagerEng", "ShopManagerEng.server"]
-package-dir = { "ShopManagerEng" = ".", "ShopManagerEng.server" = "server" }
+packages = ["ShopManagerEng", "ShopManagerEng.server", "ShopManagerEng.training"]
+package-dir = { "ShopManagerEng" = ".", "ShopManagerEng.server" = "server", "ShopManagerEng.training" = "training" }

rollout_baseline.py

@@ -0,0 +1,139 @@
+#!/usr/bin/env python3
+"""
+No-TRL baseline: run random vs simple heuristic policies; log mean return.
+Run local server first:  uv run server
+Then:  SHOPMANAGER_MARKET_MODE=synthetic SHOPMANAGER_TRAIN_BASE_URL=http://127.0.0.1:8000 python rollout_baseline.py
+"""
+from __future__ import annotations
+
+import argparse
+import asyncio
+import os
+import random
+import sys
+from pathlib import Path
+from statistics import fmean, pstdev
+from typing import List, Optional
+
+ROOT = Path(__file__).resolve().parent
+if str(ROOT) not in sys.path:
+    sys.path.insert(0, str(ROOT))
+
+from client import JewelryShopEnv
+from models import JewelryAction, PRODUCT_CATALOG
+
+
+def _heuristic_action(obs) -> JewelryAction:
+    ph = obs.phase
+    if ph == "market":
+        g = float(obs.gold_price or 0.0) or 1.0
+        need = 1.0
+        if obs.cash >= need * g + 10:
+            return JewelryAction(
+                market_action="buy", gold_qty=need, target_price_usd=obs.gold_price
+            )
+        return JewelryAction(market_action="wait")
+    if ph == "warehouse":
+        dem = obs.demand or {"ring": 0.5, "necklace": 0.3, "bracelet": 0.2}
+        for name in sorted(dem, key=lambda k: dem.get(k, 0), reverse=True):
+            gneed = float(PRODUCT_CATALOG[name]["gold_oz"])
+            lab = float(PRODUCT_CATALOG[name]["labor"])
+            if obs.gold_oz + 1e-9 >= gneed and obs.cash + 1e-9 >= lab:
+                return JewelryAction(product_choice=name)
+        return JewelryAction(product_choice="ring")
+    if ph == "showroom":
+        if (
+            obs.current_offer
+            and obs.cost_basis > 0
+            and (float(obs.current_offer) / float(obs.cost_basis)) >= 1.15
+        ) or (getattr(obs, "negotiation_round", 0) and int(obs.negotiation_round) >= 3):
+            return JewelryAction(message="I accept")
+        off = float(obs.current_offer or 0.0)
+        return JewelryAction(
+            message=f"How about ${off * 1.08:.2f}?" if off else "I need a better offer"
+        )
+    return JewelryAction()
+
+
+def _random_action(obs) -> JewelryAction:
+    if obs.phase == "market":
+        if random.random() < 0.35:
+            return JewelryAction(
+                market_action="buy", gold_qty=round(random.uniform(0.1, 1.2), 2)
+            )
+        return JewelryAction(market_action="wait")
+    if obs.phase == "warehouse":
+        return JewelryAction(product_choice=random.choice(["ring", "necklace", "bracelet"]))
+    return JewelryAction(
+        message=random.choice(
+            [
+                "I accept",
+                f"How about ${float(obs.current_offer or 0) * 1.1:.0f}?",
+            ]
+        )
+    )
+
+
+async def one_episode(base: str, policy: str, seed: Optional[int], max_steps: int) -> float:
+    """
+    Run one episode under the given policy and return the trajectory return,
+    which is the env's cumulative reward (sum of per-step partials, in [0, 1]).
+    """
+    if seed is not None:
+        random.seed(seed)
+    env = JewelryShopEnv(base_url=base)
+    r = await env.reset(seed=seed, episode_id=None)
+    o = r.observation
+    for _ in range(max_steps):
+        if r.done:
+            break
+        if policy == "heuristic":
+            a = _heuristic_action(o)
+        else:
+            a = _random_action(o)
+        r = await env.step(a)
+        o = r.observation
+    try:
+        await env.close()
+    except Exception:  # noqa: BLE001
+        pass
+    # Authoritative trajectory return from the server (in [0, 1]).
+    return float(getattr(o, "cumulative_reward", 0.0))
+
+
+def main() -> None:
+    p = argparse.ArgumentParser()
+    p.add_argument("--episodes", type=int, default=20)
+    p.add_argument("--max-steps", type=int, default=25)
+    p.add_argument(
+        "--base-url",
+        default=os.environ.get("SHOPMANAGER_TRAIN_BASE_URL", "http://127.0.0.1:8000"),
+    )
+    p.add_argument("--policies", nargs="+", default=["heuristic", "random"])
+    p.add_argument("--out", type=Path, default=Path("rollout_metrics.txt"))
+    args = p.parse_args()
+    base = str(args.base_url)
+    all_lines: List[str] = [f"base_url={base}", f"episodes={args.episodes} max_steps={args.max_steps}", ""]
+
+    for name in args.policies:
+        scores: List[float] = []
+        for epi in range(args.episodes):
+            sc = asyncio.run(
+                one_episode(base, name, seed=epi, max_steps=int(args.max_steps))  # type: ignore[misc]  # noqa: E501
+            )
+            scores.append(sc)
+        m = fmean(scores) if scores else 0.0
+        sd = pstdev(scores) if len(scores) > 1 else 0.0
+        line = f"{name}: mean={m:.4f} std={sd:.4f} scores={scores!s}"
+        all_lines.append(line)
+        print(line)
+    text = "\n".join(all_lines) + "\n"
+    try:
+        args.out.write_text(text, encoding="utf-8")
+        print(f"Wrote {args.out}", flush=True)
+    except OSError as err:
+        print("Could not write out file:", err, flush=True)
+
+
+if __name__ == "__main__":
+    main()

start.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+set -e
+
+# Start the OpenEnv API server in the background (port 8000)
+cd /app/env
+uvicorn server.app:app --host 0.0.0.0 --port 8000 &
+API_PID=$!
+
+# Wait for the API to be ready
+echo "[start] Waiting for API server on port 8000..."
+for i in $(seq 1 30); do
+    if curl -sf http://localhost:8000/health > /dev/null 2>&1; then
+        echo "[start] API server ready."
+        break
+    fi
+    sleep 1
+done
+
+# Start Streamlit UI (port 7860 — HF Spaces default)
+echo "[start] Launching Streamlit UI on port 7860..."
+exec streamlit run ui.py \
+    --server.port 7860 \
+    --server.address 0.0.0.0 \
+    --server.headless true \
+    --browser.gatherUsageStats false \
+    --theme.primaryColor "#7c3aed" \
+    --theme.backgroundColor "#0f0c29" \
+    --theme.secondaryBackgroundColor "#302b63" \
+    --theme.textColor "#e2e8f0"

test_env_smoke.py

@@ -0,0 +1,118 @@
+"""Standalone, no-LLM, no-server smoke tests for the JewelryShop env.
+
+Run from inside the ShopManagerEng folder so `models` / `server` import.
+
+Usage:
+    cd ShopManagerEng
+    SHOPMANAGER_MARKET_MODE=synthetic python test_env_smoke.py            # default: A+B
+    SHOPMANAGER_MARKET_MODE=real      python test_env_smoke.py live       # C only
+    python test_env_smoke.py all                                          # all three
+
+Why a script: putting `f'gold_price=${o.gold_price}/oz'` inside `bash -c "..."`
+makes bash try to expand `${o.gold_price}` as a shell variable and crash with
+`bad substitution`. Running it from a .py file removes that whole class of bugs.
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+
+from server.ShopManagerEng_environment import JewelryShopEnvironment
+from models import JewelryAction
+
+
+def test_reward_path() -> None:
+    """A. Synthetic episode end-to-end, prints per-step partial + cumulative."""
+    print("\n=== A. reward path (synthetic) ===")
+    os.environ["SHOPMANAGER_MARKET_MODE"] = "synthetic"
+    e = JewelryShopEnvironment()
+    o = e.reset(seed=42, task_id="market_timing", starting_cash=10000.0)
+    print(
+        f"  reset: phase={o.phase} cash=${o.cash} gold={o.gold_oz}oz "
+        f"price=${o.gold_price} weights={o.weights}"
+    )
+
+    actions = [
+        JewelryAction(market_action="buy", gold_qty=1.0),
+        JewelryAction(product_choice="ring"),
+        JewelryAction(message="I accept"),
+    ]
+    for i, a in enumerate(actions, 1):
+        o = e.step(a)
+        print(
+            f"  step {i}: phase={o.phase} reward={o.reward:.4f} "
+            f"cum={o.cumulative_reward:.4f} done={o.done}"
+        )
+    print(f"  FINAL cum={o.cumulative_reward:.4f} (must be in [0, 1])")
+
+
+def test_bounce_loop() -> None:
+    """B. Warehouse cannot craft -> agent is bounced back to MARKET."""
+    print("\n=== B. bounce loop (warehouse -> market) ===")
+    os.environ["SHOPMANAGER_MARKET_MODE"] = "synthetic"
+    e = JewelryShopEnvironment()
+    o = e.reset(seed=42, task_id="profit_negotiator", starting_cash=10000.0)
+
+    o = e.step(JewelryAction(market_action="buy", gold_qty=0.2))
+    print(f"  bought 0.2oz: phase={o.phase}, gold={o.gold_oz}oz")
+
+    o = e.step(JewelryAction(product_choice="ring"))
+    print(
+        f"  tried ring: phase={o.phase} reentries={o.market_reentries}"
+        f"/{o.max_market_reentries} urgent={o.inventory_urgent} "
+        f"cannot_wait={o.cannot_wait}"
+    )
+    assert o.phase == "market", "expected bounce back to market"
+    assert o.inventory_urgent is True, "expected urgent flag"
+
+    o = e.step(JewelryAction(market_action="wait"))
+    print(f"  tried wait while urgent: phase={o.phase} (should still be market)")
+    assert o.phase == "market", "wait should be blocked when urgent"
+
+    o = e.step(JewelryAction(market_action="buy", gold_qty=1.0))
+    print(f"  bought 1.0oz more: phase={o.phase} gold={o.gold_oz}oz")
+
+    o = e.step(JewelryAction(product_choice="ring"))
+    print(f"  craft ring: phase={o.phase} cum={o.cumulative_reward:.4f}")
+
+    o = e.step(JewelryAction(message="I accept"))
+    print(f"  FINAL cum={o.cumulative_reward:.4f}")
+
+
+def test_live_quote() -> None:
+    """C. Real mode: live yfinance gold price (needs network)."""
+    print("\n=== C. live yfinance quote (real mode) ===")
+    os.environ["SHOPMANAGER_MARKET_MODE"] = "real"
+    e = JewelryShopEnvironment()
+    o = e.reset(seed=0, task_id="market_timing", starting_cash=10000.0)
+    print(f"  gold_price=${o.gold_price}/oz   source={o.gold_price_source}")
+    print(f"  market_mode={o.market_mode}")
+    print(f"  history(last 5)={o.gold_price_history[-5:]}")
+
+
+def main(argv: list[str]) -> None:
+    arg = (argv[1] if len(argv) > 1 else "default").lower()
+
+    if arg in ("default", "ab"):
+        test_reward_path()
+        test_bounce_loop()
+    elif arg == "live":
+        test_live_quote()
+    elif arg == "all":
+        test_reward_path()
+        test_bounce_loop()
+        test_live_quote()
+    elif arg == "a":
+        test_reward_path()
+    elif arg == "b":
+        test_bounce_loop()
+    elif arg == "c":
+        test_live_quote()
+    else:
+        print(f"unknown arg: {arg}. use one of: a / b / c / ab / live / all")
+        sys.exit(2)
+
+
+if __name__ == "__main__":
+    main(sys.argv)

train_jewelry_grpo.py

@@ -0,0 +1,363 @@
+"""GRPO training entry point for the JewelryShop OpenEnv (Module 5 style).
+
+Canonical TRL >= 0.17 + OpenEnv pattern:
+
+  1. Persistent sync env client            (one WebSocket reused across rollouts)
+  2. rollout_func(prompts, trainer)        (returns prompt_ids/completion_ids/logprobs + rewards)
+  3. Pure reward_funcs(completions, ...)   (read kwargs the rollout puts there)
+  4. GRPOTrainer(..., rollout_func=...)    (NO `environment_factory`)
+
+Design notes:
+- Reward shaping: ONE primary reward (`reward_total = obs.cumulative_reward`)
+  drives gradients; per-phase rewards (market/warehouse/showroom) are exposed
+  for monitoring with weight 0 in the GRPO advantage.
+- Tasks: dataset rows embed [TASK=<task_id>] which the rollout extracts so each
+  episode trains against a specific per-phase weight profile from openenv.yaml.
+- Smoke: --smoke imports everything, builds the rollout func, opens the env
+  and does a single reset() round-trip — no GPU and no model weights needed.
+  Use this to validate wiring before paying for a GPU.
+
+Quick local smoke (no GPU, no model load):
+    python train_jewelry_grpo.py --smoke
+
+Full local quick check (CPU; slow, but verifies trainer.train() starts):
+    TRAIN_MODEL=Qwen/Qwen3-0.6B python train_jewelry_grpo.py --quick
+
+Cloud (HF Jobs) — see README/TRAINING for the exact `hf jobs run` command.
+"""
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+from pathlib import Path
+
+# Make `from ShopManagerEng...` imports work whether you launch this from inside
+# the package directory or one level up.
+ROOT = Path(__file__).resolve().parent
+PARENT = ROOT.parent
+if str(PARENT) not in sys.path:
+    sys.path.insert(0, str(PARENT))
+
+try:
+    from ShopManagerEng.client import JewelryShopEnv
+    from ShopManagerEng.training.plotting import (
+        build_metrics_callback,
+        save_training_artifacts,
+        upload_training_artifacts_to_hub,
+    )
+    from ShopManagerEng.training.prompts import SYSTEM_PROMPT
+    from ShopManagerEng.training.rewards import (
+        ALL_REWARDS,
+        REWARD_WEIGHTS_MONITOR_ONLY,
+    )
+    from ShopManagerEng.training.rollout import VALID_TASKS, build_rollout_func
+except ImportError:  # script-style invocation from inside the folder
+    from client import JewelryShopEnv  # type: ignore
+    from training.plotting import (  # type: ignore
+        build_metrics_callback,
+        save_training_artifacts,
+        upload_training_artifacts_to_hub,
+    )
+    from training.prompts import SYSTEM_PROMPT  # type: ignore
+    from training.rewards import (  # type: ignore
+        ALL_REWARDS,
+        REWARD_WEIGHTS_MONITOR_ONLY,
+    )
+    from training.rollout import VALID_TASKS, build_rollout_func  # type: ignore
+
+
+def _build_dataset(dataset_size: int):
+    """Cycle through the 3 graded tasks; embed task id in prompt for the rollout."""
+    from datasets import Dataset
+
+    rows = []
+    for i in range(dataset_size):
+        task_id = VALID_TASKS[i % len(VALID_TASKS)]
+        rows.append(
+            {
+                "prompt": (
+                    f"[TASK={task_id}] Manage a jewelry shop episode end-to-end. "
+                    f"Maximize the {task_id} task reward."
+                ),
+                "task_id": task_id,
+            }
+        )
+    return Dataset.from_list(rows)
+
+
+def _resolve_precision():
+    """CPU/GPU autodetect; mirror the well-tested defaults."""
+    try:
+        import torch
+        has_cuda = bool(torch.cuda.is_available())
+    except Exception:
+        has_cuda = False
+    if has_cuda:
+        return {"bf16": True}
+    return {"use_cpu": True, "bf16": False, "fp16": False}
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument(
+        "--model",
+        default=os.environ.get("TRAIN_MODEL", "Qwen/Qwen3-1.7B"),
+        help="HF model id (default: Qwen/Qwen3-1.7B; matches openenv-course Module 5).",
+    )
+    ap.add_argument(
+        "--env-url",
+        default=os.environ.get(
+            "ENV_URL", "https://hard007ik-shopmanagereng.hf.space"
+        ),
+        help="Base URL of the running OpenEnv server (Space or http://127.0.0.1:8000).",
+    )
+    ap.add_argument(
+        "--output-dir",
+        default=os.environ.get("TRAIN_OUTPUT_DIR", "shopmanager-grpo-out"),
+    )
+    ap.add_argument("--dataset-size", type=int, default=300)
+    ap.add_argument("--num-generations", type=int, default=2)
+    ap.add_argument("--per-device-batch", type=int, default=1)
+    ap.add_argument("--grad-accum", type=int, default=32)
+    ap.add_argument("--max-completion-length", type=int, default=64)
+    # NOTE: --max-prompt-length intentionally removed. Recent TRL versions
+    # (>=0.20-ish) dropped `max_prompt_length` from GRPOConfig and now infer
+    # it from the tokenizer / model context window. Passing it back errors
+    # with `TypeError: unexpected keyword argument 'max_prompt_length'`.
+    ap.add_argument("--max-turns", type=int, default=15)
+    ap.add_argument("--lr", type=float, default=5e-6)
+    ap.add_argument("--warmup-steps", type=int, default=10)
+    ap.add_argument("--max-steps", type=int, default=-1, help="-1 = epoch-bounded.")
+    ap.add_argument("--epochs", type=int, default=1)
+    ap.add_argument(
+        "--vllm-gpu-mem",
+        type=float,
+        default=0.3,
+        help="Fraction of GPU mem reserved for vLLM. Lower if OOM.",
+    )
+    ap.add_argument("--push-to-hub", action="store_true")
+    ap.add_argument(
+        "--hub-repo-id",
+        default=None,
+        help="HF model repo (user/model-name) for weight push + training-artifact upload. "
+        "If omitted, artifact upload uses {whoami}/{basename of --output-dir}.",
+    )
+    ap.add_argument(
+        "--report-to",
+        default=os.environ.get("TRAIN_REPORT_TO", "trackio"),
+        help="trackio | wandb | none",
+    )
+    ap.add_argument(
+        "--smoke",
+        action="store_true",
+        help="No training. Imports, env connect, one reset() — validates wiring.",
+    )
+    ap.add_argument(
+        "--quick",
+        action="store_true",
+        help="CPU-friendly tiny run (1 step, num_generations=2, max_completion=32).",
+    )
+    args = ap.parse_args()
+
+    if args.smoke:
+        # No transformers / vllm load: just import + connect + reset, then
+        # also exercise the plotting pipeline on a fake log_history so the
+        # submission-artifact path is proven before we burn a GPU on it.
+        env = JewelryShopEnv(base_url=args.env_url)
+        sync_env = env.sync()
+        sync_env.connect()
+        try:
+            r = sync_env.reset(task_id=VALID_TASKS[0])
+            print(f"[SMOKE] connected to {args.env_url}")
+            print(
+                f"[SMOKE] reset OK: phase={r.observation.phase}, "
+                f"done={r.done}, cumulative_reward="
+                f"{getattr(r.observation, 'cumulative_reward', 0)}"
+            )
+            print(f"[SMOKE] system prompt loaded ({len(SYSTEM_PROMPT)} chars)")
+            print("[SMOKE] reward funcs:", [f.__name__ for f in ALL_REWARDS])
+            print("[SMOKE] reward weights:", REWARD_WEIGHTS_MONITOR_ONLY)
+        finally:
+            try:
+                sync_env.close()
+            except Exception:
+                pass
+
+        # Prove the plotting pipeline works (writes PNG + CSV + JSON to
+        # output_dir using a synthetic, monotonic log). This is what the
+        # real run will produce — same code path.
+        fake_history = []
+        for step in range(1, 21):
+            fake_history.append(
+                {
+                    "step": step,
+                    "loss": 1.0 / step,
+                    "reward": 0.05 * step,
+                    "rewards/reward_total": min(0.05 * step, 1.0),
+                    "rewards/reward_market": min(0.02 * step, 0.6),
+                    "rewards/reward_warehouse": min(0.015 * step, 0.6),
+                    "rewards/reward_showroom": min(0.018 * step, 0.6),
+                }
+            )
+        summary = save_training_artifacts(
+            fake_history,
+            args.output_dir,
+            run_config={"smoke": True, "model": args.model, "env_url": args.env_url},
+        )
+        print(
+            f"[SMOKE] plot pipeline OK -> {args.output_dir}/loss_curve.png, "
+            f"{args.output_dir}/reward_curve.png, "
+            f"{args.output_dir}/reward_total_curve.png"
+        )
+        print(
+            f"[SMOKE] summary peak_reward_total={summary['reward_total']['max']:.3f} "
+            f"final_loss={summary['loss']['final']:.4f}"
+        )
+        print("[SMOKE] OK — wiring is sane.")
+        return
+
+    # ── Heavy imports only past the smoke gate ──
+    from transformers import AutoTokenizer
+    from trl import GRPOConfig, GRPOTrainer
+
+    tokenizer = AutoTokenizer.from_pretrained(args.model)
+    if tokenizer.pad_token is None:
+        tokenizer.pad_token = tokenizer.eos_token
+
+    env = JewelryShopEnv(base_url=args.env_url)
+    sync_env = env.sync()
+    sync_env.connect()
+    print(f"[TRAIN] env: {args.env_url}")
+    print(f"[TRAIN] model: {args.model}")
+
+    rollout_func = build_rollout_func(
+        sync_env=sync_env,
+        tokenizer=tokenizer,
+        system_prompt=SYSTEM_PROMPT,
+        max_turns=args.max_turns,
+        model_name=args.model,
+    )
+
+    if args.quick:
+        # Override to tiny CPU-friendly numbers.
+        args.dataset_size = 4
+        args.num_generations = 2
+        args.per_device_batch = 2
+        args.grad_accum = 1
+        args.max_completion_length = 32
+        args.max_steps = 1
+        args.warmup_steps = 0
+
+    dataset = _build_dataset(args.dataset_size)
+    precision = _resolve_precision()
+    use_cpu = precision.get("use_cpu", False)
+
+    grpo_config = GRPOConfig(
+        output_dir=args.output_dir,
+        num_train_epochs=args.epochs,
+        learning_rate=args.lr,
+        gradient_accumulation_steps=args.grad_accum,
+        per_device_train_batch_size=args.per_device_batch,
+        warmup_steps=args.warmup_steps,
+        num_generations=args.num_generations,
+        max_completion_length=args.max_completion_length,
+        # `max_prompt_length` removed: not accepted by recent TRL GRPOConfig.
+        # vLLM is the canonical generation backend on GPU; turn off on CPU smoke.
+        use_vllm=not use_cpu,
+        vllm_mode="colocate" if not use_cpu else None,
+        vllm_gpu_memory_utilization=args.vllm_gpu_mem if not use_cpu else None,
+        gradient_checkpointing=True,
+        gradient_checkpointing_kwargs={"use_reentrant": False},
+        reward_weights=REWARD_WEIGHTS_MONITOR_ONLY,
+        report_to=args.report_to if args.report_to != "none" else "none",
+        logging_steps=1,
+        save_steps=20,
+        push_to_hub=args.push_to_hub,
+        max_steps=args.max_steps if args.max_steps > 0 else -1,
+        **precision,
+    )
+
+    print(f"[TRAIN] device={'cpu' if use_cpu else 'gpu'}  precision={precision}")
+    print(f"[TRAIN] dataset_size={args.dataset_size}  num_generations={args.num_generations}")
+
+    trainer = GRPOTrainer(
+        model=args.model,
+        processing_class=tokenizer,
+        reward_funcs=list(ALL_REWARDS),
+        train_dataset=dataset,
+        args=grpo_config,
+        rollout_func=rollout_func,
+    )
+
+    # Persist loss + reward plots into output_dir every N steps + at the end.
+    # This is the hackathon "evidence you actually trained" artifact set.
+    trainer.add_callback(build_metrics_callback(args.output_dir, snapshot_every=5))
+
+    run_config = {
+        "model": args.model,
+        "env_url": args.env_url,
+        "dataset_size": args.dataset_size,
+        "num_generations": args.num_generations,
+        "per_device_batch": args.per_device_batch,
+        "grad_accum": args.grad_accum,
+        "max_completion_length": args.max_completion_length,
+        "max_turns": args.max_turns,
+        "lr": args.lr,
+        "warmup_steps": args.warmup_steps,
+        "max_steps": args.max_steps,
+        "epochs": args.epochs,
+        "vllm_gpu_mem": args.vllm_gpu_mem,
+        "reward_weights": REWARD_WEIGHTS_MONITOR_ONLY,
+        "precision": precision,
+    }
+
+    try:
+        trainer.train()
+    finally:
+        try:
+            sync_env.close()
+        except Exception:
+            pass
+        # Always persist whatever metrics we have, even on a crash mid-run.
+        try:
+            summary = save_training_artifacts(
+                list(trainer.state.log_history or []),
+                args.output_dir,
+                run_config=run_config,
+            )
+            print(
+                f"[ARTIFACTS] wrote loss/reward plots + metrics to {args.output_dir}\n"
+                f"[ARTIFACTS] final loss={summary['loss']['final']:.4f} "
+                f"max_reward_total={summary['reward_total']['max']:.4f} "
+                f"final_reward_total={summary['reward_total']['final']:.4f}"
+            )
+        except Exception as exc:
+            print(f"[ARTIFACTS] failed to save metrics: {exc}")
+
+    trainer.save_model(args.output_dir)
+    if args.push_to_hub:
+        trainer.push_to_hub()
+        # Default Hub push only ships weights/tokenizer; upload plots + metrics explicitly.
+        try:
+            from huggingface_hub import whoami
+
+            user = whoami().get("name") or whoami().get("preferred_username", "user")
+            rid = (
+                args.hub_repo_id
+                or getattr(trainer.args, "hub_model_id", None)
+                or f"{user}/{Path(args.output_dir).name}"
+            )
+            uploaded = upload_training_artifacts_to_hub(args.output_dir, rid)
+            if uploaded:
+                print(
+                    f"[HUB] training artifacts ({len(uploaded)} files) -> "
+                    f"https://huggingface.co/{rid}/tree/main/training_artifacts"
+                )
+        except Exception as exc:  # noqa: BLE001
+            print(f"[HUB] training artifact upload failed: {exc}")
+    print(f"[DONE] saved to {args.output_dir}")
+
+
+if __name__ == "__main__":
+    main()

training/__init__.py

@@ -0,0 +1,5 @@
+"""ShopManagerEng training package.
+
+Course-style (Module 5) GRPO training scaffolding for the JewelryShop env.
+Uses TRL's `rollout_func=...` entry point with vLLM colocate generation.
+"""

training/parse_action.py

@@ -0,0 +1,52 @@
+"""Parse free-form model text into a typed JewelryAction.
+
+Mirrors inference.py:get_action_from_text so the action surface during
+training matches what was used during evaluation.
+"""
+from __future__ import annotations
+
+from typing import Tuple
+
+try:
+    from ..models import JewelryAction
+except ImportError:
+    from models import JewelryAction
+
+
+def parse_model_text_to_action(phase: str, text: str) -> Tuple[JewelryAction, str]:
+    """Return (action, normalised_text) for the current phase.
+
+    Robust against typical LLM output noise: backticks, quotes, leading/trailing
+    whitespace. Falls back to safe defaults so a single bad token never breaks
+    the rollout.
+    """
+    text = (text or "").strip().replace("`", "").strip(" \t\n\r\"'")
+
+    if phase == "market":
+        lower = text.lower()
+        if lower.startswith("buy"):
+            qty_str = lower.replace("buy", "").strip()
+            try:
+                qty = float(qty_str)
+            except ValueError:
+                qty = 1.0
+            return JewelryAction(market_action="buy", gold_qty=qty), f"buy {qty}"
+        if "wait" in lower:
+            return JewelryAction(market_action="wait"), "wait"
+        try:
+            qty = float(text)
+            return JewelryAction(market_action="buy", gold_qty=qty), f"buy {qty}"
+        except ValueError:
+            return JewelryAction(market_action="wait"), "wait"
+
+    if phase == "warehouse":
+        lower = text.lower()
+        for product in ("necklace", "bracelet", "ring"):
+            if product in lower:
+                return JewelryAction(product_choice=product), product
+        return JewelryAction(product_choice="ring"), "ring"
+
+    if phase == "showroom":
+        return JewelryAction(message=text), text
+
+    return JewelryAction(), text

training/plotting.py

@@ -0,0 +1,325 @@
+"""Persist training metrics + loss/reward plots to disk.
+
+Why this exists: the hackathon submission asks for "evidence you actually
+trained — at minimum loss and reward plots from a real run." Since we run as
+a script (not a notebook), nothing renders automatically. This module:
+
+* Snapshots ``trainer.state.log_history`` every N steps via a TrainerCallback
+  (so a crashed run still leaves partial evidence behind), and
+* Dumps a final set of artifacts (CSV, JSON, PNGs) after ``trainer.train()``.
+
+All artifacts land in the trainer's ``output_dir`` so they ride back to the
+Hugging Face Hub when ``push_to_hub=True``.
+"""
+from __future__ import annotations
+
+import csv
+import json
+import logging
+from pathlib import Path
+from typing import Any, Dict, Iterable, List, Optional
+
+logger = logging.getLogger(__name__)
+
+
+# Reward keys we track. TRL logs reward functions under "rewards/<func_name>"
+# (and a single-scalar "reward" = sum of weighted rewards).
+PRIMARY_REWARD_KEY = "rewards/reward_total"
+PHASE_REWARD_KEYS = (
+    "rewards/reward_market",
+    "rewards/reward_warehouse",
+    "rewards/reward_showroom",
+)
+LOSS_KEY = "loss"
+STEP_KEY = "step"
+
+
+def _flatten_log_history(log_history: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+    """Make sure every row carries a `step` field even when TRL omits it on epoch logs."""
+    cleaned: List[Dict[str, Any]] = []
+    last_step = 0
+    for row in log_history:
+        step = row.get("step", row.get("global_step", last_step))
+        last_step = step or last_step
+        merged = {"step": last_step, **{k: v for k, v in row.items() if k != "step"}}
+        cleaned.append(merged)
+    return cleaned
+
+
+def _series(rows: List[Dict[str, Any]], key: str) -> List[tuple]:
+    """Return ``[(step, value), ...]`` for the given metric key."""
+    out: List[tuple] = []
+    for r in rows:
+        if key in r and r[key] is not None:
+            try:
+                out.append((int(r["step"]), float(r[key])))
+            except (TypeError, ValueError):
+                continue
+    return out
+
+
+def _save_csv(rows: List[Dict[str, Any]], path: Path) -> None:
+    if not rows:
+        return
+    columns: List[str] = []
+    seen = set()
+    for r in rows:
+        for k in r.keys():
+            if k not in seen:
+                seen.add(k)
+                columns.append(k)
+    with path.open("w", newline="") as f:
+        writer = csv.DictWriter(f, fieldnames=columns)
+        writer.writeheader()
+        writer.writerows(rows)
+
+
+def _save_json(rows: List[Dict[str, Any]], path: Path) -> None:
+    with path.open("w") as f:
+        json.dump(rows, f, indent=2, default=str)
+
+
+def _try_plot(
+    series: Iterable[tuple],
+    title: str,
+    ylabel: str,
+    out_path: Path,
+    *,
+    label: Optional[str] = None,
+) -> bool:
+    """Draw a single-series line plot. Silently no-ops if matplotlib is missing."""
+    try:
+        import matplotlib
+
+        matplotlib.use("Agg")
+        import matplotlib.pyplot as plt
+    except Exception as exc:
+        logger.warning("matplotlib unavailable, skipping %s (%s)", out_path.name, exc)
+        return False
+
+    pts = list(series)
+    if not pts:
+        logger.warning("no data for %s, skipping plot", out_path.name)
+        return False
+    xs, ys = zip(*pts)
+    fig, ax = plt.subplots(figsize=(8, 4.5))
+    ax.plot(xs, ys, marker="o", linewidth=1.5, label=label or ylabel)
+    ax.set_xlabel("training step")
+    ax.set_ylabel(ylabel)
+    ax.set_title(title)
+    ax.grid(True, alpha=0.3)
+    if label:
+        ax.legend(loc="best")
+    fig.tight_layout()
+    fig.savefig(out_path, dpi=120)
+    plt.close(fig)
+    return True
+
+
+def _try_plot_multi(
+    name_to_series: Dict[str, Iterable[tuple]],
+    title: str,
+    ylabel: str,
+    out_path: Path,
+) -> bool:
+    """Draw a multi-series line plot."""
+    try:
+        import matplotlib
+
+        matplotlib.use("Agg")
+        import matplotlib.pyplot as plt
+    except Exception as exc:
+        logger.warning("matplotlib unavailable, skipping %s (%s)", out_path.name, exc)
+        return False
+
+    fig, ax = plt.subplots(figsize=(8.5, 5))
+    drew_any = False
+    for label, pts in name_to_series.items():
+        pts = list(pts)
+        if not pts:
+            continue
+        xs, ys = zip(*pts)
+        ax.plot(xs, ys, marker="o", linewidth=1.3, label=label)
+        drew_any = True
+    if not drew_any:
+        plt.close(fig)
+        logger.warning("no data for %s, skipping plot", out_path.name)
+        return False
+    ax.set_xlabel("training step")
+    ax.set_ylabel(ylabel)
+    ax.set_title(title)
+    ax.grid(True, alpha=0.3)
+    ax.legend(loc="best")
+    fig.tight_layout()
+    fig.savefig(out_path, dpi=120)
+    plt.close(fig)
+    return True
+
+
+def _summary_stats(series: List[tuple]) -> Dict[str, float]:
+    if not series:
+        return {"final": 0.0, "max": 0.0, "min": 0.0, "mean": 0.0, "n": 0}
+    ys = [v for _, v in series]
+    return {
+        "final": float(ys[-1]),
+        "max": float(max(ys)),
+        "min": float(min(ys)),
+        "mean": float(sum(ys) / len(ys)),
+        "n": len(ys),
+    }
+
+
+def save_training_artifacts(
+    log_history: List[Dict[str, Any]],
+    output_dir: str | Path,
+    *,
+    run_config: Optional[Dict[str, Any]] = None,
+) -> Dict[str, Any]:
+    """Write metrics + loss/reward plots into ``output_dir``.
+
+    Returns the summary dict that was also written to ``training_summary.json``.
+    """
+    out = Path(output_dir)
+    out.mkdir(parents=True, exist_ok=True)
+
+    rows = _flatten_log_history(log_history)
+    _save_csv(rows, out / "metrics.csv")
+    _save_json(rows, out / "metrics.json")
+
+    loss_series = _series(rows, LOSS_KEY)
+    total_reward_series = _series(rows, PRIMARY_REWARD_KEY)
+    # Some TRL versions log a flat "reward" scalar in addition. Prefer the
+    # named primary; fall back to "reward" if the named one is empty.
+    if not total_reward_series:
+        total_reward_series = _series(rows, "reward")
+
+    phase_series = {
+        "market": _series(rows, "rewards/reward_market"),
+        "warehouse": _series(rows, "rewards/reward_warehouse"),
+        "showroom": _series(rows, "rewards/reward_showroom"),
+    }
+
+    _try_plot(
+        loss_series,
+        title="Training loss (GRPO)",
+        ylabel="loss",
+        out_path=out / "loss_curve.png",
+        label="loss",
+    )
+    _try_plot(
+        total_reward_series,
+        title="Reward (total) — env cumulative_reward in [0, 1]",
+        ylabel="reward",
+        out_path=out / "reward_total_curve.png",
+        label="reward_total",
+    )
+    _try_plot_multi(
+        {
+            "reward_total": total_reward_series,
+            **{f"reward_{k}": v for k, v in phase_series.items()},
+        },
+        title="Rewards over training",
+        ylabel="reward",
+        out_path=out / "reward_curve.png",
+    )
+
+    summary: Dict[str, Any] = {
+        "loss": _summary_stats(loss_series),
+        "reward_total": _summary_stats(total_reward_series),
+        "reward_market": _summary_stats(phase_series["market"]),
+        "reward_warehouse": _summary_stats(phase_series["warehouse"]),
+        "reward_showroom": _summary_stats(phase_series["showroom"]),
+        "n_log_rows": len(rows),
+        "output_dir": str(out.resolve()),
+    }
+    if run_config is not None:
+        summary["run_config"] = run_config
+
+    with (out / "training_summary.json").open("w") as f:
+        json.dump(summary, f, indent=2, default=str)
+
+    logger.info("Wrote training artifacts to %s", out.resolve())
+    return summary
+
+
+def build_metrics_callback(output_dir: str | Path, snapshot_every: int = 5):
+    """Return a TrainerCallback that snapshots metrics every N steps + on end.
+
+    Imported lazily so this module can be inspected on a machine without
+    transformers installed (e.g. for the local --smoke run).
+    """
+    from transformers.trainer_callback import TrainerCallback
+
+    out = Path(output_dir)
+
+    class MetricsSaverCallback(TrainerCallback):
+        """Persist metrics CSV/JSON + plots periodically and at the end."""
+
+        def __init__(self) -> None:
+            self._last_snapshot_step = -1
+
+        def _snapshot(self, state) -> None:
+            try:
+                save_training_artifacts(list(state.log_history or []), out)
+            except Exception as exc:  # never let plotting kill training
+                logger.warning("metrics snapshot failed: %s", exc)
+
+        def on_log(self, args, state, control, **kwargs):
+            step = int(getattr(state, "global_step", 0) or 0)
+            if step <= 0:
+                return control
+            if (step - self._last_snapshot_step) >= max(snapshot_every, 1):
+                self._snapshot(state)
+                self._last_snapshot_step = step
+            return control
+
+        def on_train_end(self, args, state, control, **kwargs):
+            self._snapshot(state)
+            return control
+
+    return MetricsSaverCallback()
+
+
+def upload_training_artifacts_to_hub(
+    output_dir: str | Path,
+    repo_id: str,
+    *,
+    path_in_repo: str = "training_artifacts",
+) -> list[str]:
+    """Upload small evidence files to the same model repo (PNGs, CSV, JSON).
+
+    ``GRPOTrainer.push_to_hub`` typically uploads weights/tokenizer only; this
+    adds ``metrics.csv``, ``loss_curve.png``, and related files under
+    ``path_in_repo/`` on the Hub so they survive ephemeral cloud jobs.
+    """
+    from huggingface_hub import HfApi, create_repo
+
+    out = Path(output_dir)
+    if not out.is_dir():
+        return []
+
+    create_repo(repo_id, repo_type="model", exist_ok=True)
+    api = HfApi()
+    names = (
+        "metrics.csv",
+        "metrics.json",
+        "loss_curve.png",
+        "reward_curve.png",
+        "reward_total_curve.png",
+        "training_summary.json",
+    )
+    prefix = path_in_repo.strip("/")
+    uploaded: list[str] = []
+    for name in names:
+        path = out / name
+        if not path.is_file():
+            continue
+        dest = f"{prefix}/{name}" if prefix else name
+        api.upload_file(
+            path_or_fileobj=str(path),
+            path_in_repo=dest,
+            repo_id=repo_id,
+            repo_type="model",
+        )
+        uploaded.append(dest)
+    return uploaded

training/prompts.py

@@ -0,0 +1,169 @@
+"""System prompt + per-turn user-prompt builder for the JewelryShop env.
+
+Logic mirrors `inference.py`'s `build_user_prompt` so that whatever the model
+saw during inference evaluation it also sees during training. Kept as a plain
+sync function (no asyncio) so it composes cleanly with TRL's rollout_func.
+"""
+from __future__ import annotations
+
+import math
+import textwrap
+from typing import List
+
+
+SYSTEM_PROMPT = textwrap.dedent(
+    """
+    You are an expert agent running a jewelry shop. The episode runs in 3 phases
+    and may loop back to MARKET if the warehouse runs out of gold. The episode
+    reward is the SUM of per-step partial rewards across the whole episode and
+    is bounded in [0, 1]. Each task weights the phases differently:
+      - market_timing     -> phase 1 = 0.6, phase 2 = 0.2, phase 3 = 0.2
+      - demand_crafter    -> phase 1 = 0.2, phase 2 = 0.6, phase 3 = 0.2
+      - profit_negotiator -> phase 1 = 0.2, phase 2 = 0.2, phase 3 = 0.6
+
+    ## Phase 1: MARKET (buy / wait)
+    Two modes:
+      - synthetic mode: gold price moves randomly each WAIT step within a round cap.
+      - real mode:      gold price comes from a live source (yfinance: GC=F),
+                        no round cap; WAIT just refreshes the live quote.
+    Coordination from the warehouse:
+      - inventory_urgent=True / cannot_wait=True means you MUST buy now;
+        WAIT will be blocked. Submit "buy X.XX" with an affordable troy-oz qty.
+    Behavior:
+      - If you can wait, observe the price trend in gold_price_history before buying.
+      - Reserve cash for labor (ring=$200, necklace=$300, bracelet=$100).
+      - Respond: "buy X.XX" (troy oz of gold) or "wait".
+
+    ## Phase 2: WAREHOUSE (choose product)
+    You see two demand fields:
+      - demand          : the TRUE per-product demand for THIS episode (ground truth).
+      - demand_forecast : a NOISY signal you can also lean on for planning.
+    Products: ring (1oz + $200), necklace (2oz + $300), bracelet (0.5oz + $100).
+    If you don't have enough gold to craft your choice, the env may BOUNCE you back
+    to MARKET to buy more (up to max_market_reentries times). After max bounces or
+    when truly broke, the customer leaves and the episode ends.
+    Respond: "ring", "necklace", or "bracelet".
+
+    ## Phase 3: SHOWROOM (negotiate)
+    The customer makes an offer; if you counter, they raise it ~5% per round,
+    up to 5 rounds. After 5 rounds with no acceptance, the customer leaves
+    (no phase-3 reward). Reject also gives 0 phase-3 reward.
+    Respond: "I accept" or a counter like "How about $X?". NEVER explicitly reject.
+
+    CRITICAL: Respond with ONLY the action value. No explanations.
+    """
+).strip()
+
+
+def build_user_prompt(step: int, obs, last_reward: float, history: List[str]) -> str:
+    """Format a single observation into a user prompt the LLM sees this turn.
+
+    Mirrors inference.py:build_user_prompt so the model sees the same input shape
+    during training and at evaluation time.
+    """
+    history_block = "\n".join(history[-4:]) if history else "None"
+
+    if obs.phase == "market":
+        prices = getattr(obs, "gold_price_history", []) or []
+        trend = ""
+        if len(prices) >= 2:
+            if prices[-1] < prices[-2]:
+                trend = "FALLING (might keep dropping, consider waiting)"
+            else:
+                trend = "RISING (buy now before it gets more expensive)"
+
+        if getattr(obs, "cannot_wait", False):
+            trend = (
+                "URGENT: inventory needs gold now — you cannot wait; buy at the current "
+                "live quote with an affordable gold_qty (troy oz)."
+            )
+
+        max_rounds = getattr(obs, "max_market_rounds", None)
+        rounds_left = (max_rounds - getattr(obs, "market_round", 0)) if max_rounds else None
+
+        reserve = 300.0
+        gold_price = getattr(obs, "gold_price", 0) or 0
+        cash = getattr(obs, "cash", 0) or 0
+        if gold_price > 0:
+            raw_qty = (cash - reserve) / gold_price
+            suggested_qty = max(math.floor(raw_qty * 100) / 100, 0.01)
+        else:
+            suggested_qty = 1.0
+
+        rl = "unlimited" if rounds_left is None else str(rounds_left)
+        phase_hint = (
+            f"Price: ${gold_price}/oz ({getattr(obs, 'gold_price_source', '') or 'n/a'}). "
+            f"Price history: {prices}. Trend: {trend}. "
+            f"Rounds / waits so far: {getattr(obs, 'market_round', 0)}; cap: {rl}. "
+            f"Gold on hand: {getattr(obs, 'gold_oz', 0)} troy oz "
+            f"(~{getattr(obs, 'gold_grams', 0):.2f} g). "
+            f"If buying, suggested qty: {suggested_qty} oz (reserves $300 for labor). "
+            f"Respond: 'buy {suggested_qty}' or 'wait'"
+        )
+
+    elif obs.phase == "warehouse":
+        demand = getattr(obs, "demand", {}) or {}
+        forecast = getattr(obs, "demand_forecast", {}) or {}
+        best_product = max(demand, key=demand.get) if demand else "ring"
+        phase_hint = (
+            f"Demand (episode): ring={demand.get('ring', 0):.0%}, "
+            f"necklace={demand.get('necklace', 0):.0%}, "
+            f"bracelet={demand.get('bracelet', 0):.0%}. "
+            f"Forecast (noisy): ring={forecast.get('ring', 0):.0%}, "
+            f"necklace={forecast.get('necklace', 0):.0%}, "
+            f"bracelet={forecast.get('bracelet', 0):.0%}. "
+            f"Highest demand: {best_product}. "
+            f"You have {getattr(obs, 'gold_oz', 0)}oz gold and "
+            f"${getattr(obs, 'cash', 0)} cash. "
+            f"Respond with EXACTLY: {best_product}"
+        )
+
+    elif obs.phase == "showroom":
+        cost_basis = getattr(obs, "cost_basis", 0) or 0
+        current_offer = getattr(obs, "current_offer", 0) or 0
+        negotiation_round = getattr(obs, "negotiation_round", 0) or 0
+
+        margin = ""
+        if current_offer and cost_basis > 0:
+            margin_pct = ((current_offer - cost_basis) / cost_basis) * 100
+            margin = f"Margin: {margin_pct:+.1f}%. "
+
+        should_accept = negotiation_round >= 4 or (
+            current_offer and cost_basis > 0 and current_offer > cost_basis * 1.3
+        )
+
+        if should_accept:
+            phase_hint = (
+                f"Cost: ${cost_basis}. Offer: ${current_offer}. {margin}"
+                f"Round {negotiation_round}/5. "
+                f"Respond with EXACTLY: I accept"
+            )
+        else:
+            counter_msgs = [
+                "I need a better price for this quality piece",
+                "That's too low, this craftsmanship deserves more",
+                f"How about ${round(cost_basis * 1.4, 2)}?",
+                f"I can't go below ${round(cost_basis * 1.3, 2)}",
+            ]
+            msg = counter_msgs[min(negotiation_round, len(counter_msgs) - 1)]
+            phase_hint = (
+                f"Cost: ${cost_basis}. Offer: ${current_offer}. {margin}"
+                f"Round {negotiation_round}/5. "
+                f"DO NOT ACCEPT. Counter-offer. "
+                f"Respond with EXACTLY: {msg}"
+            )
+    else:
+        phase_hint = ""
+
+    return textwrap.dedent(
+        f"""
+        Step: {step} | Phase: {obs.phase} | Last reward: {last_reward:.2f}
+        Cash: ${getattr(obs, 'cash', 0)} | Gold: {getattr(obs, 'gold_oz', 0)}oz | Rings: {getattr(obs, 'inventory', {})}
+        Gold Price: ${getattr(obs, 'gold_price', 0)}/oz
+        Env Message: {getattr(obs, 'message', '')}
+
+        {phase_hint}
+
+        History: {history_block}
+        """
+    ).strip()

training/rewards.py

@@ -0,0 +1,48 @@
+"""Reward functions consumed by GRPOTrainer.
+
+Design choice: ONE primary reward (``reward_total``) drives advantages, while
+the three per-phase rewards are exposed for *monitoring only* via TRL's logged
+reward metrics. Their config weight should be set to 0.0 to avoid double
+counting the cumulative phase sum.
+
+If you actually want phase-level shaping in the gradient, change the
+GRPOConfig ``reward_weights`` to e.g. [1.0, 0.2, 0.2, 0.2].
+"""
+from __future__ import annotations
+
+from typing import Any, List
+
+
+def _pull(kwargs: dict, key: str, n: int) -> List[float]:
+    vals = kwargs.get(key)
+    if not vals:
+        return [0.0] * n
+    return [float(v) for v in vals]
+
+
+def reward_total(completions: List[Any], **kwargs) -> List[float]:
+    """Authoritative trajectory return: env's cumulative_reward in [0, 1]."""
+    return _pull(kwargs, "total_reward", len(completions))
+
+
+def reward_market(completions: List[Any], **kwargs) -> List[float]:
+    """Sum of per-step partials emitted while phase == 'market'. Monitoring."""
+    return _pull(kwargs, "market_reward", len(completions))
+
+
+def reward_warehouse(completions: List[Any], **kwargs) -> List[float]:
+    """Sum of per-step partials emitted while phase == 'warehouse'. Monitoring."""
+    return _pull(kwargs, "warehouse_reward", len(completions))
+
+
+def reward_showroom(completions: List[Any], **kwargs) -> List[float]:
+    """Sum of per-step partials emitted while phase == 'showroom'. Monitoring."""
+    return _pull(kwargs, "showroom_reward", len(completions))
+
+
+# Convenience tuple for single-import use
+ALL_REWARDS = (reward_total, reward_market, reward_warehouse, reward_showroom)
+
+# Matching weights so only `reward_total` contributes to the GRPO advantage.
+# Plug this straight into GRPOConfig(reward_weights=REWARD_WEIGHTS_MONITOR_ONLY).
+REWARD_WEIGHTS_MONITOR_ONLY = [1.0, 0.0, 0.0, 0.0]

training/rollout.py

@@ -0,0 +1,214 @@
+"""Module 5 (TRL OpenEnv Wordle) style rollout for ShopManagerEng.
+
+Two public symbols:
+
+* ``rollout_once(...)``  — plays a single multi-turn jewelry-shop episode
+  against an already-connected sync env client and returns the per-episode
+  signals TRL/GRPO needs.
+* ``build_rollout_func(...)`` — closure factory that returns the
+  ``rollout_func(prompts, trainer=None)`` callable handed to ``GRPOTrainer``.
+
+The pattern (canonical for OpenEnv + TRL >= 0.17):
+
+    sync_env = env.sync(); sync_env.connect()      # one persistent WS
+    trainer = GRPOTrainer(..., rollout_func=rollout_func)
+    trainer.train()
+"""
+from __future__ import annotations
+
+import re
+from typing import Any, Callable, Dict, List, Optional
+
+try:
+    from .parse_action import parse_model_text_to_action
+    from .prompts import build_user_prompt
+except ImportError:
+    from training.parse_action import parse_model_text_to_action
+    from training.prompts import build_user_prompt
+
+
+# Set of valid task ids supported by openenv.yaml; first one is the default.
+VALID_TASKS = ("market_timing", "demand_crafter", "profit_negotiator")
+_TASK_RE = re.compile(r"\[TASK=(\w+)\]")
+
+
+def extract_task_id(prompt_text: str, default: str = VALID_TASKS[0]) -> str:
+    """Pull the [TASK=...] tag the dataset embeds, or fall back to the default."""
+    m = _TASK_RE.search(prompt_text or "")
+    if not m:
+        return default
+    candidate = m.group(1)
+    return candidate if candidate in VALID_TASKS else default
+
+
+def _apply_chat_template(tokenizer, messages, model_name: str = "") -> str:
+    """Apply chat template, opting out of Qwen3 'thinking' mode when applicable."""
+    template_kwargs: Dict[str, Any] = {
+        "add_generation_prompt": True,
+        "tokenize": False,
+    }
+    # Qwen3 family supports the `enable_thinking` switch — disable it for short
+    # action outputs. Other models silently ignore unknown kwargs in newer
+    # transformers; older ones may raise, hence the lower() guard.
+    if "qwen3" in (model_name or "").lower():
+        template_kwargs["enable_thinking"] = False
+    return tokenizer.apply_chat_template(messages, **template_kwargs)
+
+
+def rollout_once(
+    *,
+    trainer,
+    sync_env,
+    tokenizer,
+    dataset_prompt: str,
+    system_prompt: str,
+    max_turns: int,
+    model_name: str = "",
+) -> Dict[str, Any]:
+    """Play one full jewelry-shop episode and return per-episode signals.
+
+    Returns the dict shape TRL's GRPO loop expects: ``prompt_ids``,
+    ``completion_ids``, ``logprobs`` for **a single** vLLM forward (the **last
+    environment turn** in the episode) plus reward signals for reward
+    functions.
+
+    We **do not** concatenate multiple turns into one list. In ``GRPOTrainer``,
+    each batch row is ``cat(prompt_ids, completion_ids)``; vLLM's per-token
+    ``logprobs`` must be for **that** exact sequence, or the importance-sampling
+    ratio (vLLM vs reference forward) collapses. Multi-turn play still runs in
+    the environment; the policy gradient is applied to the **last** action's
+    tokens, while ``total_reward`` remains the full episode return for GRPO
+    group advantages.
+    """
+    # Late import: trl.experimental.openenv only exists for trl >= 0.17.
+    from trl.experimental.openenv import generate_rollout_completions
+
+    task_id = extract_task_id(dataset_prompt)
+    result = sync_env.reset(task_id=task_id)
+    obs = result.observation
+
+    # One (prompt_ids, completion_ids, logprobs) per vLLM call; last turn only
+    # is returned to TRL (see module docstring).
+    turn_traces: List[Dict[str, Any]] = []
+
+    history: List[str] = []
+    last_reward = 0.0
+    phase_rewards = {"market": 0.0, "warehouse": 0.0, "showroom": 0.0}
+
+    for turn in range(1, max_turns + 1):
+        if result.done:
+            break
+
+        user_prompt = build_user_prompt(turn, obs, last_reward, history)
+        messages = [
+            {"role": "system", "content": system_prompt},
+            {"role": "user", "content": user_prompt},
+        ]
+        prompt_text = _apply_chat_template(tokenizer, messages, model_name=model_name)
+
+        rollout_outputs = generate_rollout_completions(trainer, [prompt_text])[0]
+        p_ids = rollout_outputs["prompt_ids"]
+        c_ids = rollout_outputs["completion_ids"]
+        lps = rollout_outputs["logprobs"]
+        p_list = p_ids.tolist() if hasattr(p_ids, "tolist") else list(p_ids)
+        c_list = c_ids.tolist() if hasattr(c_ids, "tolist") else list(c_ids)
+        turn_traces.append(
+            {
+                "prompt_ids": p_list,
+                "completion_ids": c_list,
+                "logprobs": [float(x) for x in lps],
+            }
+        )
+
+        completion_text = rollout_outputs.get("text") or tokenizer.decode(
+            rollout_outputs["completion_ids"], skip_special_tokens=True
+        )
+
+        current_phase = obs.phase
+        action, raw_action_str = parse_model_text_to_action(current_phase, completion_text)
+
+        result = sync_env.step(action)
+        obs = result.observation
+        step_reward = float(result.reward or 0.0)
+        last_reward = step_reward
+
+        if current_phase in phase_rewards:
+            phase_rewards[current_phase] += step_reward
+
+        history.append(
+            f"Step {turn} ({current_phase}): {raw_action_str!r} -> reward {step_reward:+.2f}"
+        )
+
+    total_reward = float(getattr(obs, "cumulative_reward", sum(phase_rewards.values())))
+    total_reward = max(0.0, min(total_reward, 1.0))
+
+    if not turn_traces:
+        raise ValueError(
+            "rollout_once produced no vLLM turns (max_turns too low or env ended "
+            "before the first action)."
+        )
+    last = turn_traces[-1]
+
+    return {
+        "prompt_ids": last["prompt_ids"],
+        "completion_ids": last["completion_ids"],
+        "logprobs": last["logprobs"],
+        "total_reward": total_reward,
+        "market_reward": float(phase_rewards["market"]),
+        "warehouse_reward": float(phase_rewards["warehouse"]),
+        "showroom_reward": float(phase_rewards["showroom"]),
+    }
+
+
+def build_rollout_func(
+    *,
+    sync_env,
+    tokenizer,
+    system_prompt: str,
+    max_turns: int = 15,
+    model_name: str = "",
+) -> Callable[..., Dict[str, List]]:
+    """Return ``rollout_func(prompts, trainer=None)`` closing over the env client.
+
+    A fresh episode is run for each prompt; the same persistent ``sync_env``
+    is reused across all prompts (single WebSocket session — matches Module 5).
+    """
+
+    def rollout_func(prompts: List[str], trainer=None) -> Dict[str, List]:
+        episode_prompt_ids: List[List[int]] = []
+        episode_completion_ids: List[List[int]] = []
+        episode_logprobs: List[List[float]] = []
+        total_rewards: List[float] = []
+        market_rewards: List[float] = []
+        warehouse_rewards: List[float] = []
+        showroom_rewards: List[float] = []
+
+        for prompt_text in prompts:
+            ep = rollout_once(
+                trainer=trainer,
+                sync_env=sync_env,
+                tokenizer=tokenizer,
+                dataset_prompt=prompt_text,
+                system_prompt=system_prompt,
+                max_turns=max_turns,
+                model_name=model_name,
+            )
+            episode_prompt_ids.append(ep["prompt_ids"])
+            episode_completion_ids.append(ep["completion_ids"])
+            episode_logprobs.append(ep["logprobs"])
+            total_rewards.append(ep["total_reward"])
+            market_rewards.append(ep["market_reward"])
+            warehouse_rewards.append(ep["warehouse_reward"])
+            showroom_rewards.append(ep["showroom_reward"])
+
+        return {
+            "prompt_ids": episode_prompt_ids,
+            "completion_ids": episode_completion_ids,
+            "logprobs": episode_logprobs,
+            "total_reward": total_rewards,
+            "market_reward": market_rewards,
+            "warehouse_reward": warehouse_rewards,
+            "showroom_reward": showroom_rewards,
+        }
+
+    return rollout_func

training/training_artifacts_v1/metrics.csv

@@ -0,0 +1,20 @@
+step,loss,grad_norm,learning_rate,num_tokens,completions/mean_length,completions/min_length,completions/max_length,completions/clipped_ratio,completions/mean_terminated_length,completions/min_terminated_length,completions/max_terminated_length,rewards/reward_total/mean,rewards/reward_total/std,rewards/reward_market/mean,rewards/reward_market/std,rewards/reward_warehouse/mean,rewards/reward_warehouse/std,rewards/reward_showroom/mean,rewards/reward_showroom/std,reward,reward_std,frac_reward_zero_std,sampling/sampling_logp_difference/mean,sampling/sampling_logp_difference/max,sampling/importance_sampling_ratio/min,sampling/importance_sampling_ratio/mean,sampling/importance_sampling_ratio/max,entropy,clip_ratio/low_mean,clip_ratio/low_min,clip_ratio/high_mean,clip_ratio/high_max,clip_ratio/region_mean,step_time,epoch,train_runtime,train_samples_per_second,train_steps_per_second,total_flos,train_loss
+1,-0.009,12.235088348388672,0.0,27758.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7793656587600708,0.12745577096939087,0.25,0.13440430164337158,0.42500001192092896,0.20160645246505737,0.10436563193798065,0.069697305560112,0.7793656587600708,0.12745577096939087,0.0,0.005470390431582928,0.22214925289154053,0.8007970452308655,0.9949374794960022,1.076386570930481,0.028439456821217846,0.0,0.0,0.0,0.0,0.0,19.228480510413647,0.05555555555555555,,,,,
+2,0.0723,60.551937103271484,5.000000000000001e-07,55324.0,3.25,3.0,7.0,0.0,3.25,3.0,7.0,0.719434380531311,0.1505809724330902,0.30000001192092896,0.17597654461860657,0.30000001192092896,0.17597654461860657,0.11943437159061432,0.07055392116308212,0.719434380531311,0.1505809724330902,0.0,0.01085888221859932,0.28092825412750244,0.8382877111434937,1.0260276794433594,1.324359655380249,0.036137547835437545,0.0,0.0,0.0,0.0,0.0,17.92062332853675,0.1111111111111111,,,,,
+3,0.1153,232.934814453125,1.0000000000000002e-06,83000.0,3.5,3.0,7.0,0.0,3.5,3.0,7.0,0.7885687351226807,0.1279384195804596,0.32500001788139343,0.18837162852287292,0.375,0.20160646736621857,0.08856874704360962,0.07010025531053543,0.7885687351226807,0.1279384344816208,0.0,0.0251829382032156,0.6850378513336182,0.5226751565933228,1.0358223915100098,1.9838452339172363,0.03248842835137111,0.0,0.0,0.0,0.0,0.0,17.184778176248074,0.16666666666666666,,,,,
+4,0.0243,13.620709419250488,1.5e-06,110708.0,3.125,3.0,7.0,0.0,3.125,3.0,7.0,0.7762374877929688,0.13016164302825928,0.32499998807907104,0.18837164342403412,0.3500000238418579,0.1967477649450302,0.10123749077320099,0.06825561076402664,0.7762374877929688,0.13016162812709808,0.0,0.007013080175966024,0.2646750509738922,0.7674550414085388,0.9837819337844849,1.0260045528411865,0.03743034108288157,0.0,0.0,0.0,0.0,0.0,17.90316915512085,0.2222222222222222,,,,,
+5,0.0004,2.0591225624084473,2.0000000000000003e-06,138452.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7784374952316284,0.12858590483665466,0.30000001192092896,0.17597654461860657,0.375,0.20160646736621857,0.10343749821186066,0.06380020827054977,0.7784374952316284,0.12858593463897705,0.0,0.001465568202547729,0.05673474818468094,0.9749767780303955,1.0017430782318115,1.058376669883728,0.008188390799773515,0.0,0.0,0.0,0.0,0.0,17.8210555203259,0.2777777777777778,,,,,
+6,0.0915,31.611696243286133,2.5e-06,166252.0,3.125,3.0,7.0,0.0,3.125,3.0,7.0,0.7915937900543213,0.12960509955883026,0.375,0.20160646736621857,0.32500001788139343,0.18837162852287292,0.09159374982118607,0.0639258474111557,0.7915937900543213,0.12960509955883026,0.0,0.008641102351248264,0.8236088752746582,0.9935171008110046,1.0399717092514038,2.278707504272461,0.007084679029730978,0.0,0.0,0.0,0.0,0.0,18.056264080107212,0.3333333333333333,,,,,
+7,-0.0,0.281630277633667,3e-06,193950.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7600874900817871,0.13816162943840027,0.32500001788139343,0.18837162852287292,0.32500001788139343,0.18837162852287292,0.11008749902248383,0.07028691470623016,0.7600874900817871,0.13816164433956146,0.0,3.554227441782132e-05,0.0027569520752876997,0.997247576713562,0.9999052286148071,1.0000724792480469,0.0005298306713825696,0.0,0.0,0.0,0.0,0.0,17.43799263238907,0.3888888888888889,,,,,
+8,0.0,0.00018881642608903348,3.5e-06,221677.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7622687816619873,0.1372590959072113,0.4000000059604645,0.20320022106170654,0.25,0.13440430164337158,0.11226874589920044,0.07360353320837021,0.7622687816619873,0.1372590959072113,0.0,2.545601773817907e-07,1.5496943888138048e-06,0.9999986886978149,1.000000238418579,1.0000016689300537,4.1391018498870835e-05,0.0,0.0,0.0,0.0,0.0,17.406394600868225,0.4444444444444444,,,,,
+9,0.0,0.00015632262511644512,4.000000000000001e-06,249549.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.775946855545044,0.137176051735878,0.375,0.20160646736621857,0.30000001192092896,0.17597654461860657,0.10094687342643738,0.058497413992881775,0.775946855545044,0.1371760368347168,0.0,2.918131087881193e-07,3.099441755693988e-06,0.9999978542327881,1.0,1.000001311302185,4.173239403826301e-05,0.0,0.0,0.0,0.0,0.0,18.514019537717104,0.5,,,,,
+10,0.0,9.833038348006085e-05,4.5e-06,277377.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7337374687194824,0.15055809915065765,0.2750000059604645,0.1586231142282486,0.3500000238418579,0.1967477649450302,0.10873749852180481,0.06778524816036224,0.7337374687194824,0.15055811405181885,0.0,2.719489771152439e-07,1.9073031580774114e-06,0.9999985694885254,0.9999998807907104,1.0000019073486328,4.224909940830912e-05,0.0,0.0,0.0,0.0,0.0,18.26371632888913,0.5555555555555556,,,,,
+11,-0.0,0.00015879125567153096,5e-06,305332.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7567968368530273,0.13728150725364685,0.375,0.20160646736621857,0.2750000059604645,0.1586231142282486,0.10679687559604645,0.07404065877199173,0.7567968368530273,0.13728150725364685,0.0,3.067227396513772e-07,2.6226434783893637e-06,0.9999973773956299,0.9999994039535522,1.0000019073486328,4.7876037001515215e-05,0.0,0.0,0.0,0.0,0.0,19.4472255371511,0.6111111111111112,,,,,
+12,-0.0,0.00039661259506829083,4.3750000000000005e-06,333165.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7081500291824341,0.1421954482793808,0.30000001192092896,0.17597654461860657,0.2750000059604645,0.1586231142282486,0.13315001130104065,0.07370516657829285,0.7081500291824341,0.142195463180542,0.0,4.768499479723687e-07,5.602954843197949e-06,0.9999943971633911,0.9999988675117493,1.0000016689300537,5.9777358274004655e-05,0.0,0.0,0.0,0.0,0.0,18.45015063509345,0.6666666666666666,,,,,
+13,0.0,0.0005282312049530447,3.7500000000000005e-06,361006.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7762781381607056,0.13823458552360535,0.32500001788139343,0.18837162852287292,0.3500000238418579,0.1967477649450302,0.10127812623977661,0.06158862262964249,0.7762781381607056,0.13823460042476654,0.0,6.830008487668238e-07,5.60290391149465e-06,0.9999943971633911,0.9999983310699463,1.0000022649765015,7.182803437899565e-05,0.0,0.0,0.0,0.0,0.0,17.955969959497452,0.7222222222222222,,,,,
+14,-0.0,0.000535853614564985,3.125e-06,388862.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7628874778747559,0.1295449286699295,0.30000001192092896,0.17597654461860657,0.3499999940395355,0.19674775004386902,0.11288750171661377,0.073255755007267,0.7628874778747559,0.1295449435710907,0.0,1.1113726259281975e-06,2.0979605324100703e-05,0.9999922513961792,1.0000004768371582,1.0000211000442505,9.972968496185786e-05,0.0,0.0,0.0,0.0,0.0,18.99697282537818,0.7777777777777778,,,,,
+15,0.0,0.005312301218509674,2.5e-06,416636.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7899656295776367,0.1296130269765854,0.3499999940395355,0.19674775004386902,0.3500000238418579,0.1967477649450302,0.08996562659740448,0.05463240668177605,0.7899656295776367,0.12961304187774658,0.0,4.004845322924666e-06,4.589592936099507e-05,0.9999337792396545,0.9999885559082031,1.0000029802322388,0.0001807671503684105,0.0,0.0,0.0,0.0,0.0,19.184648096561432,0.8333333333333334,,,,,
+16,-0.0,0.005995223298668861,1.8750000000000003e-06,444544.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.789996862411499,0.12048782408237457,0.375,0.20160646736621857,0.32039374113082886,0.18316024541854858,0.09460312128067017,0.062435995787382126,0.789996862411499,0.12048781663179398,0.0,4.866625204158481e-06,2.9087463190080598e-05,0.999959409236908,0.9999855160713196,1.0000004768371582,0.00019320984370096994,0.0,0.0,0.0,0.0,0.0,19.476148523390293,0.8888888888888888,,,,,
+17,0.0,0.011378168128430843,1.25e-06,472481.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.8168656826019287,0.09262391924858093,0.375,0.20160646736621857,0.3498094081878662,0.19690066576004028,0.09205625206232071,0.07046373933553696,0.8168656826019287,0.09262390434741974,0.0,5.950785180175444e-06,4.7328881919384e-05,0.9999284744262695,0.9999822974205017,1.000001072883606,0.00021975090658088448,0.0,0.0,0.0,0.0,0.0,19.344543006271124,0.9444444444444444,,,,,
+18,0.0,0.018683306872844696,6.25e-07,500266.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7750625014305115,0.14084643125534058,0.3500000238418579,0.1967477649450302,0.32499998807907104,0.18837164342403412,0.10006250441074371,0.05640558898448944,0.7750625014305115,0.14084644615650177,0.0625,1.1303089195280336e-05,0.00027322862297296524,0.999584436416626,0.9999662637710571,0.9999997615814209,0.000320568448614722,0.0,0.0,0.0,0.0,0.0,17.66909484937787,1.0,,,,,
+18,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,1.0,406.352,0.738,0.044,0.0,0.016376476217475202

training/training_artifacts_v1/metrics.json

@@ -0,0 +1,695 @@
+[
+  {
+    "step": 1,
+    "loss": -0.009,
+    "grad_norm": 12.235088348388672,
+    "learning_rate": 0.0,
+    "num_tokens": 27758.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7793656587600708,
+    "rewards/reward_total/std": 0.12745577096939087,
+    "rewards/reward_market/mean": 0.25,
+    "rewards/reward_market/std": 0.13440430164337158,
+    "rewards/reward_warehouse/mean": 0.42500001192092896,
+    "rewards/reward_warehouse/std": 0.20160645246505737,
+    "rewards/reward_showroom/mean": 0.10436563193798065,
+    "rewards/reward_showroom/std": 0.069697305560112,
+    "reward": 0.7793656587600708,
+    "reward_std": 0.12745577096939087,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 0.005470390431582928,
+    "sampling/sampling_logp_difference/max": 0.22214925289154053,
+    "sampling/importance_sampling_ratio/min": 0.8007970452308655,
+    "sampling/importance_sampling_ratio/mean": 0.9949374794960022,
+    "sampling/importance_sampling_ratio/max": 1.076386570930481,
+    "entropy": 0.028439456821217846,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 19.228480510413647,
+    "epoch": 0.05555555555555555
+  },
+  {
+    "step": 2,
+    "loss": 0.0723,
+    "grad_norm": 60.551937103271484,
+    "learning_rate": 5.000000000000001e-07,
+    "num_tokens": 55324.0,
+    "completions/mean_length": 3.25,
+    "completions/min_length": 3.0,
+    "completions/max_length": 7.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.25,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 7.0,
+    "rewards/reward_total/mean": 0.719434380531311,
+    "rewards/reward_total/std": 0.1505809724330902,
+    "rewards/reward_market/mean": 0.30000001192092896,
+    "rewards/reward_market/std": 0.17597654461860657,
+    "rewards/reward_warehouse/mean": 0.30000001192092896,
+    "rewards/reward_warehouse/std": 0.17597654461860657,
+    "rewards/reward_showroom/mean": 0.11943437159061432,
+    "rewards/reward_showroom/std": 0.07055392116308212,
+    "reward": 0.719434380531311,
+    "reward_std": 0.1505809724330902,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 0.01085888221859932,
+    "sampling/sampling_logp_difference/max": 0.28092825412750244,
+    "sampling/importance_sampling_ratio/min": 0.8382877111434937,
+    "sampling/importance_sampling_ratio/mean": 1.0260276794433594,
+    "sampling/importance_sampling_ratio/max": 1.324359655380249,
+    "entropy": 0.036137547835437545,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 17.92062332853675,
+    "epoch": 0.1111111111111111
+  },
+  {
+    "step": 3,
+    "loss": 0.1153,
+    "grad_norm": 232.934814453125,
+    "learning_rate": 1.0000000000000002e-06,
+    "num_tokens": 83000.0,
+    "completions/mean_length": 3.5,
+    "completions/min_length": 3.0,
+    "completions/max_length": 7.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.5,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 7.0,
+    "rewards/reward_total/mean": 0.7885687351226807,
+    "rewards/reward_total/std": 0.1279384195804596,
+    "rewards/reward_market/mean": 0.32500001788139343,
+    "rewards/reward_market/std": 0.18837162852287292,
+    "rewards/reward_warehouse/mean": 0.375,
+    "rewards/reward_warehouse/std": 0.20160646736621857,
+    "rewards/reward_showroom/mean": 0.08856874704360962,
+    "rewards/reward_showroom/std": 0.07010025531053543,
+    "reward": 0.7885687351226807,
+    "reward_std": 0.1279384344816208,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 0.0251829382032156,
+    "sampling/sampling_logp_difference/max": 0.6850378513336182,
+    "sampling/importance_sampling_ratio/min": 0.5226751565933228,
+    "sampling/importance_sampling_ratio/mean": 1.0358223915100098,
+    "sampling/importance_sampling_ratio/max": 1.9838452339172363,
+    "entropy": 0.03248842835137111,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 17.184778176248074,
+    "epoch": 0.16666666666666666
+  },
+  {
+    "step": 4,
+    "loss": 0.0243,
+    "grad_norm": 13.620709419250488,
+    "learning_rate": 1.5e-06,
+    "num_tokens": 110708.0,
+    "completions/mean_length": 3.125,
+    "completions/min_length": 3.0,
+    "completions/max_length": 7.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.125,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 7.0,
+    "rewards/reward_total/mean": 0.7762374877929688,
+    "rewards/reward_total/std": 0.13016164302825928,
+    "rewards/reward_market/mean": 0.32499998807907104,
+    "rewards/reward_market/std": 0.18837164342403412,
+    "rewards/reward_warehouse/mean": 0.3500000238418579,
+    "rewards/reward_warehouse/std": 0.1967477649450302,
+    "rewards/reward_showroom/mean": 0.10123749077320099,
+    "rewards/reward_showroom/std": 0.06825561076402664,
+    "reward": 0.7762374877929688,
+    "reward_std": 0.13016162812709808,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 0.007013080175966024,
+    "sampling/sampling_logp_difference/max": 0.2646750509738922,
+    "sampling/importance_sampling_ratio/min": 0.7674550414085388,
+    "sampling/importance_sampling_ratio/mean": 0.9837819337844849,
+    "sampling/importance_sampling_ratio/max": 1.0260045528411865,
+    "entropy": 0.03743034108288157,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 17.90316915512085,
+    "epoch": 0.2222222222222222
+  },
+  {
+    "step": 5,
+    "loss": 0.0004,
+    "grad_norm": 2.0591225624084473,
+    "learning_rate": 2.0000000000000003e-06,
+    "num_tokens": 138452.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7784374952316284,
+    "rewards/reward_total/std": 0.12858590483665466,
+    "rewards/reward_market/mean": 0.30000001192092896,
+    "rewards/reward_market/std": 0.17597654461860657,
+    "rewards/reward_warehouse/mean": 0.375,
+    "rewards/reward_warehouse/std": 0.20160646736621857,
+    "rewards/reward_showroom/mean": 0.10343749821186066,
+    "rewards/reward_showroom/std": 0.06380020827054977,
+    "reward": 0.7784374952316284,
+    "reward_std": 0.12858593463897705,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 0.001465568202547729,
+    "sampling/sampling_logp_difference/max": 0.05673474818468094,
+    "sampling/importance_sampling_ratio/min": 0.9749767780303955,
+    "sampling/importance_sampling_ratio/mean": 1.0017430782318115,
+    "sampling/importance_sampling_ratio/max": 1.058376669883728,
+    "entropy": 0.008188390799773515,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 17.8210555203259,
+    "epoch": 0.2777777777777778
+  },
+  {
+    "step": 6,
+    "loss": 0.0915,
+    "grad_norm": 31.611696243286133,
+    "learning_rate": 2.5e-06,
+    "num_tokens": 166252.0,
+    "completions/mean_length": 3.125,
+    "completions/min_length": 3.0,
+    "completions/max_length": 7.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.125,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 7.0,
+    "rewards/reward_total/mean": 0.7915937900543213,
+    "rewards/reward_total/std": 0.12960509955883026,
+    "rewards/reward_market/mean": 0.375,
+    "rewards/reward_market/std": 0.20160646736621857,
+    "rewards/reward_warehouse/mean": 0.32500001788139343,
+    "rewards/reward_warehouse/std": 0.18837162852287292,
+    "rewards/reward_showroom/mean": 0.09159374982118607,
+    "rewards/reward_showroom/std": 0.0639258474111557,
+    "reward": 0.7915937900543213,
+    "reward_std": 0.12960509955883026,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 0.008641102351248264,
+    "sampling/sampling_logp_difference/max": 0.8236088752746582,
+    "sampling/importance_sampling_ratio/min": 0.9935171008110046,
+    "sampling/importance_sampling_ratio/mean": 1.0399717092514038,
+    "sampling/importance_sampling_ratio/max": 2.278707504272461,
+    "entropy": 0.007084679029730978,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 18.056264080107212,
+    "epoch": 0.3333333333333333
+  },
+  {
+    "step": 7,
+    "loss": -0.0,
+    "grad_norm": 0.281630277633667,
+    "learning_rate": 3e-06,
+    "num_tokens": 193950.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7600874900817871,
+    "rewards/reward_total/std": 0.13816162943840027,
+    "rewards/reward_market/mean": 0.32500001788139343,
+    "rewards/reward_market/std": 0.18837162852287292,
+    "rewards/reward_warehouse/mean": 0.32500001788139343,
+    "rewards/reward_warehouse/std": 0.18837162852287292,
+    "rewards/reward_showroom/mean": 0.11008749902248383,
+    "rewards/reward_showroom/std": 0.07028691470623016,
+    "reward": 0.7600874900817871,
+    "reward_std": 0.13816164433956146,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 3.554227441782132e-05,
+    "sampling/sampling_logp_difference/max": 0.0027569520752876997,
+    "sampling/importance_sampling_ratio/min": 0.997247576713562,
+    "sampling/importance_sampling_ratio/mean": 0.9999052286148071,
+    "sampling/importance_sampling_ratio/max": 1.0000724792480469,
+    "entropy": 0.0005298306713825696,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 17.43799263238907,
+    "epoch": 0.3888888888888889
+  },
+  {
+    "step": 8,
+    "loss": 0.0,
+    "grad_norm": 0.00018881642608903348,
+    "learning_rate": 3.5e-06,
+    "num_tokens": 221677.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7622687816619873,
+    "rewards/reward_total/std": 0.1372590959072113,
+    "rewards/reward_market/mean": 0.4000000059604645,
+    "rewards/reward_market/std": 0.20320022106170654,
+    "rewards/reward_warehouse/mean": 0.25,
+    "rewards/reward_warehouse/std": 0.13440430164337158,
+    "rewards/reward_showroom/mean": 0.11226874589920044,
+    "rewards/reward_showroom/std": 0.07360353320837021,
+    "reward": 0.7622687816619873,
+    "reward_std": 0.1372590959072113,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 2.545601773817907e-07,
+    "sampling/sampling_logp_difference/max": 1.5496943888138048e-06,
+    "sampling/importance_sampling_ratio/min": 0.9999986886978149,
+    "sampling/importance_sampling_ratio/mean": 1.000000238418579,
+    "sampling/importance_sampling_ratio/max": 1.0000016689300537,
+    "entropy": 4.1391018498870835e-05,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 17.406394600868225,
+    "epoch": 0.4444444444444444
+  },
+  {
+    "step": 9,
+    "loss": 0.0,
+    "grad_norm": 0.00015632262511644512,
+    "learning_rate": 4.000000000000001e-06,
+    "num_tokens": 249549.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.775946855545044,
+    "rewards/reward_total/std": 0.137176051735878,
+    "rewards/reward_market/mean": 0.375,
+    "rewards/reward_market/std": 0.20160646736621857,
+    "rewards/reward_warehouse/mean": 0.30000001192092896,
+    "rewards/reward_warehouse/std": 0.17597654461860657,
+    "rewards/reward_showroom/mean": 0.10094687342643738,
+    "rewards/reward_showroom/std": 0.058497413992881775,
+    "reward": 0.775946855545044,
+    "reward_std": 0.1371760368347168,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 2.918131087881193e-07,
+    "sampling/sampling_logp_difference/max": 3.099441755693988e-06,
+    "sampling/importance_sampling_ratio/min": 0.9999978542327881,
+    "sampling/importance_sampling_ratio/mean": 1.0,
+    "sampling/importance_sampling_ratio/max": 1.000001311302185,
+    "entropy": 4.173239403826301e-05,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 18.514019537717104,
+    "epoch": 0.5
+  },
+  {
+    "step": 10,
+    "loss": 0.0,
+    "grad_norm": 9.833038348006085e-05,
+    "learning_rate": 4.5e-06,
+    "num_tokens": 277377.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7337374687194824,
+    "rewards/reward_total/std": 0.15055809915065765,
+    "rewards/reward_market/mean": 0.2750000059604645,
+    "rewards/reward_market/std": 0.1586231142282486,
+    "rewards/reward_warehouse/mean": 0.3500000238418579,
+    "rewards/reward_warehouse/std": 0.1967477649450302,
+    "rewards/reward_showroom/mean": 0.10873749852180481,
+    "rewards/reward_showroom/std": 0.06778524816036224,
+    "reward": 0.7337374687194824,
+    "reward_std": 0.15055811405181885,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 2.719489771152439e-07,
+    "sampling/sampling_logp_difference/max": 1.9073031580774114e-06,
+    "sampling/importance_sampling_ratio/min": 0.9999985694885254,
+    "sampling/importance_sampling_ratio/mean": 0.9999998807907104,
+    "sampling/importance_sampling_ratio/max": 1.0000019073486328,
+    "entropy": 4.224909940830912e-05,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 18.26371632888913,
+    "epoch": 0.5555555555555556
+  },
+  {
+    "step": 11,
+    "loss": -0.0,
+    "grad_norm": 0.00015879125567153096,
+    "learning_rate": 5e-06,
+    "num_tokens": 305332.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7567968368530273,
+    "rewards/reward_total/std": 0.13728150725364685,
+    "rewards/reward_market/mean": 0.375,
+    "rewards/reward_market/std": 0.20160646736621857,
+    "rewards/reward_warehouse/mean": 0.2750000059604645,
+    "rewards/reward_warehouse/std": 0.1586231142282486,
+    "rewards/reward_showroom/mean": 0.10679687559604645,
+    "rewards/reward_showroom/std": 0.07404065877199173,
+    "reward": 0.7567968368530273,
+    "reward_std": 0.13728150725364685,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 3.067227396513772e-07,
+    "sampling/sampling_logp_difference/max": 2.6226434783893637e-06,
+    "sampling/importance_sampling_ratio/min": 0.9999973773956299,
+    "sampling/importance_sampling_ratio/mean": 0.9999994039535522,
+    "sampling/importance_sampling_ratio/max": 1.0000019073486328,
+    "entropy": 4.7876037001515215e-05,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 19.4472255371511,
+    "epoch": 0.6111111111111112
+  },
+  {
+    "step": 12,
+    "loss": -0.0,
+    "grad_norm": 0.00039661259506829083,
+    "learning_rate": 4.3750000000000005e-06,
+    "num_tokens": 333165.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7081500291824341,
+    "rewards/reward_total/std": 0.1421954482793808,
+    "rewards/reward_market/mean": 0.30000001192092896,
+    "rewards/reward_market/std": 0.17597654461860657,
+    "rewards/reward_warehouse/mean": 0.2750000059604645,
+    "rewards/reward_warehouse/std": 0.1586231142282486,
+    "rewards/reward_showroom/mean": 0.13315001130104065,
+    "rewards/reward_showroom/std": 0.07370516657829285,
+    "reward": 0.7081500291824341,
+    "reward_std": 0.142195463180542,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 4.768499479723687e-07,
+    "sampling/sampling_logp_difference/max": 5.602954843197949e-06,
+    "sampling/importance_sampling_ratio/min": 0.9999943971633911,
+    "sampling/importance_sampling_ratio/mean": 0.9999988675117493,
+    "sampling/importance_sampling_ratio/max": 1.0000016689300537,
+    "entropy": 5.9777358274004655e-05,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 18.45015063509345,
+    "epoch": 0.6666666666666666
+  },
+  {
+    "step": 13,
+    "loss": 0.0,
+    "grad_norm": 0.0005282312049530447,
+    "learning_rate": 3.7500000000000005e-06,
+    "num_tokens": 361006.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7762781381607056,
+    "rewards/reward_total/std": 0.13823458552360535,
+    "rewards/reward_market/mean": 0.32500001788139343,
+    "rewards/reward_market/std": 0.18837162852287292,
+    "rewards/reward_warehouse/mean": 0.3500000238418579,
+    "rewards/reward_warehouse/std": 0.1967477649450302,
+    "rewards/reward_showroom/mean": 0.10127812623977661,
+    "rewards/reward_showroom/std": 0.06158862262964249,
+    "reward": 0.7762781381607056,
+    "reward_std": 0.13823460042476654,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 6.830008487668238e-07,
+    "sampling/sampling_logp_difference/max": 5.60290391149465e-06,
+    "sampling/importance_sampling_ratio/min": 0.9999943971633911,
+    "sampling/importance_sampling_ratio/mean": 0.9999983310699463,
+    "sampling/importance_sampling_ratio/max": 1.0000022649765015,
+    "entropy": 7.182803437899565e-05,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 17.955969959497452,
+    "epoch": 0.7222222222222222
+  },
+  {
+    "step": 14,
+    "loss": -0.0,
+    "grad_norm": 0.000535853614564985,
+    "learning_rate": 3.125e-06,
+    "num_tokens": 388862.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7628874778747559,
+    "rewards/reward_total/std": 0.1295449286699295,
+    "rewards/reward_market/mean": 0.30000001192092896,
+    "rewards/reward_market/std": 0.17597654461860657,
+    "rewards/reward_warehouse/mean": 0.3499999940395355,
+    "rewards/reward_warehouse/std": 0.19674775004386902,
+    "rewards/reward_showroom/mean": 0.11288750171661377,
+    "rewards/reward_showroom/std": 0.073255755007267,
+    "reward": 0.7628874778747559,
+    "reward_std": 0.1295449435710907,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 1.1113726259281975e-06,
+    "sampling/sampling_logp_difference/max": 2.0979605324100703e-05,
+    "sampling/importance_sampling_ratio/min": 0.9999922513961792,
+    "sampling/importance_sampling_ratio/mean": 1.0000004768371582,
+    "sampling/importance_sampling_ratio/max": 1.0000211000442505,
+    "entropy": 9.972968496185786e-05,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 18.99697282537818,
+    "epoch": 0.7777777777777778
+  },
+  {
+    "step": 15,
+    "loss": 0.0,
+    "grad_norm": 0.005312301218509674,
+    "learning_rate": 2.5e-06,
+    "num_tokens": 416636.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7899656295776367,
+    "rewards/reward_total/std": 0.1296130269765854,
+    "rewards/reward_market/mean": 0.3499999940395355,
+    "rewards/reward_market/std": 0.19674775004386902,
+    "rewards/reward_warehouse/mean": 0.3500000238418579,
+    "rewards/reward_warehouse/std": 0.1967477649450302,
+    "rewards/reward_showroom/mean": 0.08996562659740448,
+    "rewards/reward_showroom/std": 0.05463240668177605,
+    "reward": 0.7899656295776367,
+    "reward_std": 0.12961304187774658,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 4.004845322924666e-06,
+    "sampling/sampling_logp_difference/max": 4.589592936099507e-05,
+    "sampling/importance_sampling_ratio/min": 0.9999337792396545,
+    "sampling/importance_sampling_ratio/mean": 0.9999885559082031,
+    "sampling/importance_sampling_ratio/max": 1.0000029802322388,
+    "entropy": 0.0001807671503684105,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 19.184648096561432,
+    "epoch": 0.8333333333333334
+  },
+  {
+    "step": 16,
+    "loss": -0.0,
+    "grad_norm": 0.005995223298668861,
+    "learning_rate": 1.8750000000000003e-06,
+    "num_tokens": 444544.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.789996862411499,
+    "rewards/reward_total/std": 0.12048782408237457,
+    "rewards/reward_market/mean": 0.375,
+    "rewards/reward_market/std": 0.20160646736621857,
+    "rewards/reward_warehouse/mean": 0.32039374113082886,
+    "rewards/reward_warehouse/std": 0.18316024541854858,
+    "rewards/reward_showroom/mean": 0.09460312128067017,
+    "rewards/reward_showroom/std": 0.062435995787382126,
+    "reward": 0.789996862411499,
+    "reward_std": 0.12048781663179398,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 4.866625204158481e-06,
+    "sampling/sampling_logp_difference/max": 2.9087463190080598e-05,
+    "sampling/importance_sampling_ratio/min": 0.999959409236908,
+    "sampling/importance_sampling_ratio/mean": 0.9999855160713196,
+    "sampling/importance_sampling_ratio/max": 1.0000004768371582,
+    "entropy": 0.00019320984370096994,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 19.476148523390293,
+    "epoch": 0.8888888888888888
+  },
+  {
+    "step": 17,
+    "loss": 0.0,
+    "grad_norm": 0.011378168128430843,
+    "learning_rate": 1.25e-06,
+    "num_tokens": 472481.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.8168656826019287,
+    "rewards/reward_total/std": 0.09262391924858093,
+    "rewards/reward_market/mean": 0.375,
+    "rewards/reward_market/std": 0.20160646736621857,
+    "rewards/reward_warehouse/mean": 0.3498094081878662,
+    "rewards/reward_warehouse/std": 0.19690066576004028,
+    "rewards/reward_showroom/mean": 0.09205625206232071,
+    "rewards/reward_showroom/std": 0.07046373933553696,
+    "reward": 0.8168656826019287,
+    "reward_std": 0.09262390434741974,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 5.950785180175444e-06,
+    "sampling/sampling_logp_difference/max": 4.7328881919384e-05,
+    "sampling/importance_sampling_ratio/min": 0.9999284744262695,
+    "sampling/importance_sampling_ratio/mean": 0.9999822974205017,
+    "sampling/importance_sampling_ratio/max": 1.000001072883606,
+    "entropy": 0.00021975090658088448,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 19.344543006271124,
+    "epoch": 0.9444444444444444
+  },
+  {
+    "step": 18,
+    "loss": 0.0,
+    "grad_norm": 0.018683306872844696,
+    "learning_rate": 6.25e-07,
+    "num_tokens": 500266.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7750625014305115,
+    "rewards/reward_total/std": 0.14084643125534058,
+    "rewards/reward_market/mean": 0.3500000238418579,
+    "rewards/reward_market/std": 0.1967477649450302,
+    "rewards/reward_warehouse/mean": 0.32499998807907104,
+    "rewards/reward_warehouse/std": 0.18837164342403412,
+    "rewards/reward_showroom/mean": 0.10006250441074371,
+    "rewards/reward_showroom/std": 0.05640558898448944,
+    "reward": 0.7750625014305115,
+    "reward_std": 0.14084644615650177,
+    "frac_reward_zero_std": 0.0625,
+    "sampling/sampling_logp_difference/mean": 1.1303089195280336e-05,
+    "sampling/sampling_logp_difference/max": 0.00027322862297296524,
+    "sampling/importance_sampling_ratio/min": 0.999584436416626,
+    "sampling/importance_sampling_ratio/mean": 0.9999662637710571,
+    "sampling/importance_sampling_ratio/max": 0.9999997615814209,
+    "entropy": 0.000320568448614722,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 17.66909484937787,
+    "epoch": 1.0
+  },
+  {
+    "step": 18,
+    "train_runtime": 406.352,
+    "train_samples_per_second": 0.738,
+    "train_steps_per_second": 0.044,
+    "total_flos": 0.0,
+    "train_loss": 0.016376476217475202,
+    "epoch": 1.0
+  }
+]

training/training_artifacts_v1/training_summary.json

@@ -0,0 +1,63 @@
+{
+  "loss": {
+    "final": 0.0,
+    "max": 0.1153,
+    "min": -0.009,
+    "mean": 0.01637777777777778,
+    "n": 18
+  },
+  "reward_total": {
+    "final": 0.7750625014305115,
+    "max": 0.8168656826019287,
+    "min": 0.7081500291824341,
+    "mean": 0.7689822945329878,
+    "n": 18
+  },
+  "reward_market": {
+    "final": 0.0,
+    "max": 0.0,
+    "min": 0.0,
+    "mean": 0.0,
+    "n": 0
+  },
+  "reward_warehouse": {
+    "final": 0.0,
+    "max": 0.0,
+    "min": 0.0,
+    "mean": 0.0,
+    "n": 0
+  },
+  "reward_showroom": {
+    "final": 0.0,
+    "max": 0.0,
+    "min": 0.0,
+    "mean": 0.0,
+    "n": 0
+  },
+  "n_log_rows": 19,
+  "output_dir": "/workspace/shopmanager-grpo-qwen3",
+  "run_config": {
+    "model": "Qwen/Qwen3-1.7B",
+    "env_url": "https://hard007ik-shopmanagereng.hf.space",
+    "dataset_size": 300,
+    "num_generations": 2,
+    "per_device_batch": 1,
+    "grad_accum": 32,
+    "max_completion_length": 64,
+    "max_turns": 15,
+    "lr": 5e-06,
+    "warmup_steps": 10,
+    "max_steps": -1,
+    "epochs": 1,
+    "vllm_gpu_mem": 0.3,
+    "reward_weights": [
+      1.0,
+      0.0,
+      0.0,
+      0.0
+    ],
+    "precision": {
+      "bf16": true
+    }
+  }
+}

ui.py

@@ -0,0 +1,358 @@
+"""
+Streamlit UI for ShopManagerEng — Interactive Jewelry Shop Demo.
+
+An AI heuristic agent automatically plays through each episode.
+Users press "New Episode" and watch the agent navigate all 3 phases.
+"""
+
+import os
+import sys
+import random
+import time
+from pathlib import Path
+
+import streamlit as st
+
+# ── Ensure imports resolve ──────────────────────────────────────────────────
+ROOT = Path(__file__).resolve().parent
+if str(ROOT) not in sys.path:
+    sys.path.insert(0, str(ROOT))
+
+os.environ.setdefault("SHOPMANAGER_MARKET_MODE", "synthetic")
+
+from server.ShopManagerEng_environment import JewelryShopEnvironment
+from models import JewelryAction, PRODUCT_CATALOG
+
+
+# ── Page config ─────────────────────────────────────────────────────────────
+st.set_page_config(
+    page_title="ShopManagerEng — Jewelry Shop RL",
+    page_icon="💎",
+    layout="wide",
+    initial_sidebar_state="expanded",
+)
+
+# ── CSS: clean light-theme styling ──────────────────────────────────────────
+st.markdown("""
+<style>
+@import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;600;700;800&display=swap');
+html, body, [class*="st-"] { font-family: 'Inter', sans-serif; }
+
+.hero-title {
+    font-size: 36px; font-weight: 800;
+    background: linear-gradient(135deg, #7c3aed, #3b82f6, #10b981);
+    -webkit-background-clip: text; -webkit-text-fill-color: transparent;
+    margin-bottom: 2px;
+}
+.hero-sub { font-size: 15px; color: #64748b; margin-bottom: 20px; }
+
+.metric-card {
+    background: #f8fafc; border: 1px solid #e2e8f0; border-radius: 12px;
+    padding: 14px 18px; text-align: center; min-height: 90px;
+}
+.metric-card .label { font-size: 12px; color: #64748b; font-weight: 600; text-transform: uppercase; letter-spacing: 0.5px; }
+.metric-card .value { font-size: 24px; font-weight: 800; color: #1e293b; margin-top: 4px; }
+
+.phase-box {
+    background: #f1f5f9; border: 1px solid #cbd5e1; border-radius: 12px;
+    padding: 18px; margin-bottom: 12px;
+}
+.phase-box.active { border: 2px solid #7c3aed; background: #f5f3ff; }
+.phase-box h4 { margin: 0 0 6px; color: #1e293b; }
+.phase-box p { margin: 0; color: #475569; font-size: 14px; }
+
+.env-msg {
+    background: #eff6ff; border-left: 4px solid #3b82f6;
+    border-radius: 0 10px 10px 0; padding: 12px 16px;
+    margin: 10px 0; font-size: 13px; color: #1e40af; line-height: 1.5;
+    word-wrap: break-word;
+}
+
+.step-row {
+    padding: 8px 12px; margin: 4px 0; border-radius: 8px;
+    font-size: 13px; color: #334155;
+}
+.step-row.market   { background: #fef3c7; border-left: 3px solid #f59e0b; }
+.step-row.warehouse { background: #dbeafe; border-left: 3px solid #3b82f6; }
+.step-row.showroom  { background: #d1fae5; border-left: 3px solid #10b981; }
+
+.reward-big {
+    text-align: center; padding: 24px;
+    background: linear-gradient(135deg, #f5f3ff, #eff6ff);
+    border: 2px solid #7c3aed; border-radius: 16px;
+}
+.reward-big .score { font-size: 52px; font-weight: 800; color: #7c3aed; }
+.reward-big .label { font-size: 14px; color: #64748b; }
+
+.catalog-card {
+    background: #f8fafc; border: 1px solid #e2e8f0; border-radius: 12px;
+    padding: 16px; text-align: center;
+}
+.catalog-card h4 { margin: 0 0 8px; color: #1e293b; }
+.catalog-card p { margin: 2px 0; color: #475569; font-size: 13px; }
+
+.demand-bar {
+    background: #e2e8f0; border-radius: 6px; height: 10px; overflow: hidden; margin-top: 4px;
+}
+.demand-fill { height: 100%; border-radius: 6px; }
+</style>
+""", unsafe_allow_html=True)
+
+
+# ── Heuristic Agent ─────────────────────────────────────────────────────────
+def heuristic_action(obs):
+    """Simple rule-based agent that plays through all 3 phases."""
+    if obs.phase == "market":
+        price = float(obs.gold_price or 300)
+        # Buy if we have enough cash for at least 1 oz
+        if obs.cash >= price + 10:
+            return JewelryAction(
+                market_action="buy",
+                gold_qty=1.0,
+                target_price_usd=obs.gold_price,
+            )
+        return JewelryAction(market_action="wait")
+
+    if obs.phase == "warehouse":
+        # Pick the highest-demand product we can afford
+        demand = obs.demand or {"ring": 0.5, "necklace": 0.3, "bracelet": 0.2}
+        for name in sorted(demand, key=lambda k: demand.get(k, 0), reverse=True):
+            spec = PRODUCT_CATALOG[name]
+            if obs.gold_oz + 1e-8 >= spec["gold_oz"] and obs.cash >= spec["labor"]:
+                return JewelryAction(product_choice=name)
+        return JewelryAction(product_choice="ring")
+
+    if obs.phase == "showroom":
+        # Accept if margin > 15% or after round 3
+        if (
+            obs.current_offer
+            and obs.cost_basis > 0
+            and float(obs.current_offer) / float(obs.cost_basis) >= 1.15
+        ) or (obs.negotiation_round and int(obs.negotiation_round) >= 3):
+            return JewelryAction(message="I accept")
+        offer = float(obs.current_offer or 0)
+        if offer:
+            return JewelryAction(message=f"How about ${offer * 1.08:.2f}?")
+        return JewelryAction(message="I need a better offer")
+
+    return JewelryAction()
+
+
+# ── Session state ───────────────────────────────────────────────────────────
+if "episode_steps" not in st.session_state:
+    st.session_state.episode_steps = None
+    st.session_state.final_reward = None
+    st.session_state.episode_count = 0
+
+
+def run_episode(task_id):
+    """Run a full episode with the heuristic agent and return step logs."""
+    env = JewelryShopEnvironment()
+    seed = random.randint(0, 99999)
+    obs = env.reset(seed=seed, market_mode="synthetic", task_id=task_id)
+
+    steps = [{
+        "step": 0, "phase": obs.phase, "action": "reset",
+        "msg": obs.message, "reward": 0.0,
+        "cash": obs.cash, "gold_oz": obs.gold_oz,
+        "gold_price": obs.gold_price,
+        "cumulative": float(obs.cumulative_reward),
+    }]
+
+    for i in range(1, 20):
+        if obs.done:
+            break
+        action = heuristic_action(obs)
+
+        # Describe the action in human terms
+        if obs.phase == "market":
+            act_str = f"BUY {action.gold_qty} oz" if action.market_action == "buy" else "WAIT"
+        elif obs.phase == "warehouse":
+            act_str = f"CRAFT {action.product_choice}"
+        else:
+            act_str = action.message or "..."
+
+        obs = env.step(action)
+        steps.append({
+            "step": i, "phase": obs.phase, "action": act_str,
+            "msg": obs.message, "reward": float(obs.reward),
+            "cash": obs.cash, "gold_oz": obs.gold_oz,
+            "gold_price": getattr(obs, "gold_price", 0),
+            "product": getattr(obs, "product_for_sale", None),
+            "offer": float(obs.current_offer) if obs.current_offer else None,
+            "cost_basis": float(obs.cost_basis) if obs.cost_basis else None,
+            "cumulative": float(obs.cumulative_reward),
+        })
+
+    return steps, float(obs.cumulative_reward)
+
+
+# ── Sidebar ─────────────────────────────────────────────────────────────────
+with st.sidebar:
+    st.markdown("### 💎 ShopManagerEng")
+    st.markdown("---")
+
+    task = st.selectbox(
+        "🎯 Task Profile",
+        ["profit_negotiator", "market_timing", "demand_crafter"],
+        index=0,
+    )
+    weights = {
+        "profit_negotiator": "Showroom 60% · Market 20% · Warehouse 20%",
+        "market_timing": "Market 60% · Warehouse 20% · Showroom 20%",
+        "demand_crafter": "Warehouse 60% · Market 20% · Showroom 20%",
+    }
+    st.caption(f"**Weights:** {weights[task]}")
+
+    st.markdown("---")
+    if st.button("🚀 New Episode", use_container_width=True, type="primary"):
+        steps, reward = run_episode(task)
+        st.session_state.episode_steps = steps
+        st.session_state.final_reward = reward
+        st.session_state.episode_count += 1
+        st.rerun()
+
+    st.markdown("---")
+    st.markdown("#### How It Works")
+    st.markdown("""
+    An AI **heuristic agent** automatically plays
+    through all 3 phases of the jewelry shop:
+
+    1. 📈 **Market** — Buy gold at the right price
+    2. 🏭 **Warehouse** — Craft the most demanded product
+    3. 🤝 **Showroom** — Negotiate the best sale price
+
+    Press **🚀 New Episode** to watch the agent play!
+    """)
+
+    if st.session_state.final_reward is not None:
+        st.markdown("---")
+        reward = st.session_state.final_reward
+        color = "#10b981" if reward >= 0.6 else "#f59e0b" if reward >= 0.4 else "#ef4444"
+        st.markdown(f"**Final Score:** :{'green' if reward >= 0.6 else 'orange' if reward >= 0.4 else 'red'}[{reward:.4f}]")
+        st.metric("Episodes Played", st.session_state.episode_count)
+
+
+# ── Main area ───────────────────────────────────────────────────────────────
+st.markdown('<p class="hero-title">💎 Jewelry Shop Manager</p>', unsafe_allow_html=True)
+st.markdown('<p class="hero-sub">An RL environment for training LLMs on multi-step business decisions</p>', unsafe_allow_html=True)
+
+
+# ── No episode yet — show welcome ──────────────────────────────────────────
+if st.session_state.episode_steps is None:
+    st.info("👋 **Welcome!** Press **🚀 New Episode** in the sidebar to watch the AI agent play through the jewelry shop simulation.")
+
+    st.markdown("### 📦 Product Catalog")
+    cols = st.columns(3)
+    items = [("💍 Ring", "ring"), ("📿 Necklace", "necklace"), ("⌚ Bracelet", "bracelet")]
+    for i, (icon_name, key) in enumerate(items):
+        spec = PRODUCT_CATALOG[key]
+        with cols[i]:
+            st.markdown(f"""
+            <div class="catalog-card">
+                <h4>{icon_name}</h4>
+                <p>🪙 Gold: {spec['gold_oz']} oz</p>
+                <p>🔧 Labor: ${spec['labor']:.0f}</p>
+                <p>📊 Base demand: {spec['base_demand']:.0%}</p>
+            </div>
+            """, unsafe_allow_html=True)
+
+    st.markdown("### 🧠 Three Business Phases")
+    c1, c2, c3 = st.columns(3)
+    with c1:
+        st.markdown("""
+        <div class="phase-box">
+            <h4>📈 Phase 1: Market</h4>
+            <p>Buy raw gold at the best price. Prices fluctuate — time your purchase wisely!</p>
+        </div>
+        """, unsafe_allow_html=True)
+    with c2:
+        st.markdown("""
+        <div class="phase-box">
+            <h4>🏭 Phase 2: Warehouse</h4>
+            <p>Craft a product (ring, necklace, bracelet) that matches market demand.</p>
+        </div>
+        """, unsafe_allow_html=True)
+    with c3:
+        st.markdown("""
+        <div class="phase-box">
+            <h4>🤝 Phase 3: Showroom</h4>
+            <p>Negotiate with a customer over 5 rounds to maximize your selling price.</p>
+        </div>
+        """, unsafe_allow_html=True)
+
+
+# ── Episode results ─────────────────────────────────────────────────────────
+else:
+    steps = st.session_state.episode_steps
+    reward = st.session_state.final_reward
+
+    # ── Score banner ────────────────────────────────────────────────────────
+    if reward >= 0.8:
+        grade, grade_emoji = "Excellent", "🏆"
+    elif reward >= 0.6:
+        grade, grade_emoji = "Good", "👍"
+    elif reward >= 0.4:
+        grade, grade_emoji = "Fair", "😐"
+    else:
+        grade, grade_emoji = "Poor", "😬"
+
+    st.markdown(f"""
+    <div class="reward-big">
+        <div class="label">{grade_emoji} Episode #{st.session_state.episode_count} — {grade}</div>
+        <div class="score">{reward:.4f}</div>
+        <div class="label">cumulative reward (out of 1.0)</div>
+    </div>
+    """, unsafe_allow_html=True)
+
+    st.markdown("")
+
+    # ── Summary metrics ─────────────────────────────────────────────────────
+    last = steps[-1]
+    m1, m2, m3, m4 = st.columns(4)
+    with m1:
+        st.markdown(f"""<div class="metric-card"><div class="label">Steps</div><div class="value">{len(steps)-1}</div></div>""", unsafe_allow_html=True)
+    with m2:
+        st.markdown(f"""<div class="metric-card"><div class="label">Final Cash</div><div class="value">${last['cash']:,.0f}</div></div>""", unsafe_allow_html=True)
+    with m3:
+        gold_price = steps[0].get("gold_price", 0)
+        st.markdown(f"""<div class="metric-card"><div class="label">Gold Price</div><div class="value">${gold_price:,.0f}/oz</div></div>""", unsafe_allow_html=True)
+    with m4:
+        product = None
+        for s in steps:
+            if s.get("product"):
+                product = s["product"]
+        st.markdown(f"""<div class="metric-card"><div class="label">Product</div><div class="value">{(product or 'N/A').title()}</div></div>""", unsafe_allow_html=True)
+
+    st.markdown("---")
+
+    # ── Step-by-step log ────────────────────────────────────────────────────
+    st.markdown("### 📋 Agent Decision Log")
+
+    for s in steps:
+        phase = s.get("phase", "market")
+        icon = {"market": "📈", "warehouse": "🏭", "showroom": "🤝"}.get(phase, "⬜")
+        step_num = s["step"]
+        action = s.get("action", "")
+        rw = s.get("reward", 0)
+        cum = s.get("cumulative", 0)
+
+        rw_badge = f" · reward: `{rw:.4f}`" if rw else ""
+        cum_str = f" · cumulative: `{cum:.4f}`" if cum else ""
+
+        # Use pure markdown — no raw HTML divs
+        st.markdown(f"**Step {step_num}** {icon} **{action}**{rw_badge}{cum_str}")
+
+        # Show environment message
+        if s.get("msg"):
+            st.caption(s["msg"])
+
+        st.divider()
+
+    # ── Reward progression chart ────────────────────────────────────────────
+    st.markdown("---")
+    st.markdown("### 📊 Cumulative Reward Over Steps")
+    chart_data = [s["cumulative"] for s in steps]
+    st.line_chart(chart_data, use_container_width=True, height=200)
+
+    st.info("Press **🚀 New Episode** in the sidebar to run another episode with a different seed!")