complete almost

Blog.md

@@ -0,0 +1,58 @@
+# 🏬 ShopManagerEng: Training LLMs to Run a Business
+
+Welcome to **ShopManagerEng**, a Reinforcement Learning (RL) environment designed to test and train Large Language Models (LLMs) on complex, multi-step business operations. 
+
+While LLMs have become incredibly adept at chatting and writing code, evaluating their ability to make strategic, long-term decisions in a dynamic environment remains a challenge. ShopManagerEng tackles this by putting the AI in the shoes of a jewelry shop manager. Dont get into name 'Eng',just typos mistake of 'Env'.
+
+This blog post breaks down how the environment works, its architecture, and how it can be used to train smarter, more strategic agents.
+
+---
+
+## 🎮 The Environment: A 3-Phase Business Simulation
+
+ShopManagerEng is not a simple Q&A benchmark. It is a continuous simulation where the agent must manage inventory, respond to market fluctuations, and negotiate with customers. Every "episode" (a full game loop) consists of three distinct phases:
+
+### Phase 1: 📈 The Market (Supply)
+The agent starts with a limited budget and must decide when to buy raw gold. 
+*   **The Catch:** Gold prices fluctuate. The agent can observe price trends and decide to "wait" for a better price or "buy" immediately. 
+*   **Real-World Noise:** The environment supports a "real" market mode that pulls live gold prices from Yahoo Finance, testing the agent against real-world economic volatility.
+
+### Phase 2: 🏭 The Warehouse (Production)
+Once gold is acquired, the agent must decide what to craft: Rings, Necklaces, or Bracelets.
+*   **The Catch:** The agent must balance raw material costs, labor costs, and market demand. It receives a "demand forecast" (which includes noise) and must make production choices that maximize potential profit without running out of cash or gold.
+
+### Phase 3: 🤝 The Showroom (Sales)
+The final phase tests the agent's negotiation skills.
+*   **The Catch:** Customers make initial offers. The agent can accept, reject, or counter-offer over a maximum of 5 rounds. Push too hard, and the customer walks away (resulting in zero profit). Settle too early, and the agent leaves money on the table.
+
+The agent's final score is a cumulative reward based on how well it navigated all three phases.
+
+---
+
+## 🏗️ Architecture Under the Hood
+
+ShopManagerEng is built on a robust Client-Server architecture, making it highly scalable for Reinforcement Learning tasks.
+
+1.  **The Server (`/server/`):** The core simulation engine runs as a standalone web application (often deployed via Docker to a Hugging Face Space). It tracks the hidden "true" state of the world, handles state transitions, fetches real market data, and manages an SQLite database to log inventory and invoices.
+2.  **The Client (`client.py` & `models.py`):** Provides a clean, typed Python interface. Agents use this client to send actions (e.g., `{"market_action": "buy", "gold_qty": 2.0}`) and receive structured observations in return.
+3.  **The AI Player (`inference.py`):** A script demonstrating how to plug an LLM (like Llama 3) into the environment. It dynamically builds text prompts explaining the current state (e.g., *"Gold is $2000/oz and trending down. You have $1000. Buy or wait?"*) and parses the LLM's text output back into valid game actions.
+4.  **The Training Suite (`/training/`):** The real power of this project lies in its RL training capabilities. Using algorithms like GRPO (Group Relative Policy Optimization), the framework can run massive batches of episodes, evaluate the agent using custom reward functions, and iteratively update the model's weights to forge a master negotiator and supply-chain manager.
+
+### 🛠️ Technical Stack & Data Flow
+
+*   **FastAPI & OpenEnv:** The server is powered by FastAPI, leveraging the `openenv` framework to standardize interactions. This ensures the environment can be easily hosted (e.g., on Hugging Face Spaces) and queried by agents remotely.
+*   **Pydantic Models:** All actions and observations are strongly typed using Pydantic (`models.py`). This guarantees that agents send properly formatted JSON payloads (like `target_price_usd` or `inventory_urgent` flags) and receive structured state data.
+*   **State Persistence:** A built-in SQLite store (`sqlite_store.py`) tracks the complex state across episodes, maintaining ledgers for cash, gold (in troy ounces and grams), and specific inventory items (rings, necklaces, bracelets).
+*   **yfinance Integration:** For the "real" market mode, the environment dynamically fetches live GC=F (Gold Futures) ticker data via `market_data.py`, injecting true market volatility into the simulation rather than relying purely on synthetic random walks.
+*   **GRPO Training:** The training pipeline (`train_jewelry_grpo.py`) utilizes Group Relative Policy Optimization, an advanced RL technique designed to stabilize training and improve sample efficiency when teaching LLMs complex, multi-step tasks. Custom reward functions (`rewards.py`) evaluate and guide the model's behavior.
+
+---
+
+## 🚀 Why This Matters
+
+As we move towards autonomous AI agents, we need benchmarks that go beyond static knowledge retrieval. ShopManagerEng evaluates an agent's ability to:
+*   **Plan Long-Term:** A bad purchase in Phase 1 ruins the negotiation in Phase 3.
+*   **Handle Uncertainty:** Dealing with noisy demand forecasts and volatile live markets.
+*   **Negotiate:** Understanding margins and customer psychology.
+
+Whether you are testing the out-of-the-box reasoning of a new foundational model or fine-tuning a specialized RL agent, ShopManagerEng provides a rich, complex sandbox to push AI capabilities forward.

README.md

@@ -5,12 +5,15 @@ colorFrom: green
 colorTo: blue
 sdk: docker
 pinned: false
-app_port: 8000
+app_port: 7860
 base_path: /web
 tags:
   - openenv
 ---
 
+Link of the environment: https://huggingface.co/spaces/hard007ik/ShopManagerEng
+Link of Blog.md: https://huggingface.co/spaces/hard007ik/ShopManagerEng/tree/main/Blog.md
+
 # Jewelry Shop Manager — RL Environment
 
 A reinforcement learning environment simulating a **jewelry shop management** pipeline. An AI agent navigates three sequential phases — buying raw materials, selecting products to craft based on demand, and negotiating sales — to maximize profit.

inference.py

@@ -71,7 +71,8 @@ SYSTEM_PROMPT = textwrap.dedent(
     Respond: "ring", "necklace", or "bracelet".
 
     ## Phase 3: SHOWROOM (negotiate)
-    The customer makes an offer; if you counter, they raise it ~5% per round,
+    you makes an offer; if customer counter by telling less price from your offer, you can drop price about ~3-5% per round but make sure to not sell when loss is happening also bring max profit,
+    if customer says less price then your first told price then you have to say the price that lesser than the price you told before but more that the customer told price
     up to 5 rounds. After 5 rounds with no acceptance, the customer leaves
     (no phase-3 reward). Reject also gives 0 phase-3 reward.
     Respond: "I accept" or a counter like "How about $X?". NEVER explicitly reject.
@@ -344,8 +345,8 @@ async def main() -> None:
     # ── ENV SERVER URL ──────────────────────────────────────────────────────
     # LOCAL:  start server with `uv run --project . server`, then use localhost
     # REMOTE: comment the localhost line and uncomment the HF Space line
-    # base_url = "http://localhost:8000"
-    base_url = "https://hard007ik-shopmanagereng.hf.space"
+    base_url = "http://localhost:8000"
+    # base_url = "https://hard007ik-shopmanagereng.hf.space"
     # ───────────────────────────────────────────────────────────────────────
 
     # print(f"[CONFIG] base_url={base_url}  model={MODEL_NAME}", flush=True)

pyproject.toml

@@ -21,6 +21,7 @@ dependencies = [
     "yfinance>=0.2.40",
     "python-dotenv>=1.0.0",
     "requests>=2.28.0",
+    "streamlit>=1.30.0",
 ]
 
 [project.optional-dependencies]

server/Dockerfile

@@ -59,6 +59,11 @@ FROM ${BASE_IMAGE}
 
 WORKDIR /app
 
+# Install curl (for health checks in start.sh) and streamlit
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends curl && \
+    rm -rf /var/lib/apt/lists/*
+
 # Copy the virtual environment from builder
 COPY --from=builder /app/env/.venv /app/.venv
 
@@ -71,10 +76,15 @@ ENV PATH="/app/.venv/bin:$PATH"
 # Set PYTHONPATH so imports work correctly
 ENV PYTHONPATH="/app/env:$PYTHONPATH"
 
+# Install streamlit into the existing venv
+RUN pip install --no-cache-dir streamlit
+
 # Health check
 HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
     CMD curl -f http://localhost:8000/health || exit 1
 
-# Run the FastAPI server
-# The module path is constructed to work with the /app/env structure
-CMD ["sh", "-c", "cd /app/env && uvicorn server.app:app --host 0.0.0.0 --port 8000"]
+# Make startup script executable
+RUN chmod +x /app/env/start.sh
+
+# Run both API server (background) + Streamlit UI (foreground)
+CMD ["/app/env/start.sh"]

start.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+set -e
+
+# Start the OpenEnv API server in the background (port 8000)
+cd /app/env
+uvicorn server.app:app --host 0.0.0.0 --port 8000 &
+API_PID=$!
+
+# Wait for the API to be ready
+echo "[start] Waiting for API server on port 8000..."
+for i in $(seq 1 30); do
+    if curl -sf http://localhost:8000/health > /dev/null 2>&1; then
+        echo "[start] API server ready."
+        break
+    fi
+    sleep 1
+done
+
+# Start Streamlit UI (port 7860 — HF Spaces default)
+echo "[start] Launching Streamlit UI on port 7860..."
+exec streamlit run ui.py \
+    --server.port 7860 \
+    --server.address 0.0.0.0 \
+    --server.headless true \
+    --browser.gatherUsageStats false \
+    --theme.primaryColor "#7c3aed" \
+    --theme.backgroundColor "#0f0c29" \
+    --theme.secondaryBackgroundColor "#302b63" \
+    --theme.textColor "#e2e8f0"

train_jewelry_grpo.py

@@ -44,6 +44,7 @@ try:
     from ShopManagerEng.training.plotting import (
         build_metrics_callback,
         save_training_artifacts,
+        upload_training_artifacts_to_hub,
     )
     from ShopManagerEng.training.prompts import SYSTEM_PROMPT
     from ShopManagerEng.training.rewards import (
@@ -56,6 +57,7 @@ except ImportError:  # script-style invocation from inside the folder
     from training.plotting import (  # type: ignore
         build_metrics_callback,
         save_training_artifacts,
+        upload_training_artifacts_to_hub,
     )
     from training.prompts import SYSTEM_PROMPT  # type: ignore
     from training.rewards import (  # type: ignore
@@ -135,6 +137,12 @@ def main() -> None:
         help="Fraction of GPU mem reserved for vLLM. Lower if OOM.",
     )
     ap.add_argument("--push-to-hub", action="store_true")
+    ap.add_argument(
+        "--hub-repo-id",
+        default=None,
+        help="HF model repo (user/model-name) for weight push + training-artifact upload. "
+        "If omitted, artifact upload uses {whoami}/{basename of --output-dir}.",
+    )
     ap.add_argument(
         "--report-to",
         default=os.environ.get("TRAIN_REPORT_TO", "trackio"),
@@ -330,6 +338,24 @@ def main() -> None:
     trainer.save_model(args.output_dir)
     if args.push_to_hub:
         trainer.push_to_hub()
+        # Default Hub push only ships weights/tokenizer; upload plots + metrics explicitly.
+        try:
+            from huggingface_hub import whoami
+
+            user = whoami().get("name") or whoami().get("preferred_username", "user")
+            rid = (
+                args.hub_repo_id
+                or getattr(trainer.args, "hub_model_id", None)
+                or f"{user}/{Path(args.output_dir).name}"
+            )
+            uploaded = upload_training_artifacts_to_hub(args.output_dir, rid)
+            if uploaded:
+                print(
+                    f"[HUB] training artifacts ({len(uploaded)} files) -> "
+                    f"https://huggingface.co/{rid}/tree/main/training_artifacts"
+                )
+        except Exception as exc:  # noqa: BLE001
+            print(f"[HUB] training artifact upload failed: {exc}")
     print(f"[DONE] saved to {args.output_dir}")
 
 

training/plotting.py

@@ -278,3 +278,48 @@ def build_metrics_callback(output_dir: str | Path, snapshot_every: int = 5):
             return control
 
     return MetricsSaverCallback()
+
+
+def upload_training_artifacts_to_hub(
+    output_dir: str | Path,
+    repo_id: str,
+    *,
+    path_in_repo: str = "training_artifacts",
+) -> list[str]:
+    """Upload small evidence files to the same model repo (PNGs, CSV, JSON).
+
+    ``GRPOTrainer.push_to_hub`` typically uploads weights/tokenizer only; this
+    adds ``metrics.csv``, ``loss_curve.png``, and related files under
+    ``path_in_repo/`` on the Hub so they survive ephemeral cloud jobs.
+    """
+    from huggingface_hub import HfApi, create_repo
+
+    out = Path(output_dir)
+    if not out.is_dir():
+        return []
+
+    create_repo(repo_id, repo_type="model", exist_ok=True)
+    api = HfApi()
+    names = (
+        "metrics.csv",
+        "metrics.json",
+        "loss_curve.png",
+        "reward_curve.png",
+        "reward_total_curve.png",
+        "training_summary.json",
+    )
+    prefix = path_in_repo.strip("/")
+    uploaded: list[str] = []
+    for name in names:
+        path = out / name
+        if not path.is_file():
+            continue
+        dest = f"{prefix}/{name}" if prefix else name
+        api.upload_file(
+            path_or_fileobj=str(path),
+            path_in_repo=dest,
+            repo_id=repo_id,
+            repo_type="model",
+        )
+        uploaded.append(dest)
+    return uploaded

training/rollout.py

@@ -68,9 +68,17 @@ def rollout_once(
     """Play one full jewelry-shop episode and return per-episode signals.
 
     Returns the dict shape TRL's GRPO loop expects: ``prompt_ids``,
-    ``completion_ids``, ``logprobs`` (concatenated across turns of the episode)
-    plus reward signals consumed by reward functions (``total_reward``,
-    ``market_reward``, ``warehouse_reward``, ``showroom_reward``).
+    ``completion_ids``, ``logprobs`` for **a single** vLLM forward (the **last
+    environment turn** in the episode) plus reward signals for reward
+    functions.
+
+    We **do not** concatenate multiple turns into one list. In ``GRPOTrainer``,
+    each batch row is ``cat(prompt_ids, completion_ids)``; vLLM's per-token
+    ``logprobs`` must be for **that** exact sequence, or the importance-sampling
+    ratio (vLLM vs reference forward) collapses. Multi-turn play still runs in
+    the environment; the policy gradient is applied to the **last** action's
+    tokens, while ``total_reward`` remains the full episode return for GRPO
+    group advantages.
     """
     # Late import: trl.experimental.openenv only exists for trl >= 0.17.
     from trl.experimental.openenv import generate_rollout_completions
@@ -79,9 +87,9 @@ def rollout_once(
     result = sync_env.reset(task_id=task_id)
     obs = result.observation
 
-    prompt_ids: List[int] = []
-    completion_ids: List[int] = []
-    logprobs: List[float] = []
+    # One (prompt_ids, completion_ids, logprobs) per vLLM call; last turn only
+    # is returned to TRL (see module docstring).
+    turn_traces: List[Dict[str, Any]] = []
 
     history: List[str] = []
     last_reward = 0.0
@@ -99,9 +107,18 @@ def rollout_once(
         prompt_text = _apply_chat_template(tokenizer, messages, model_name=model_name)
 
         rollout_outputs = generate_rollout_completions(trainer, [prompt_text])[0]
-        prompt_ids.extend(rollout_outputs["prompt_ids"])
-        completion_ids.extend(rollout_outputs["completion_ids"])
-        logprobs.extend(rollout_outputs["logprobs"])
+        p_ids = rollout_outputs["prompt_ids"]
+        c_ids = rollout_outputs["completion_ids"]
+        lps = rollout_outputs["logprobs"]
+        p_list = p_ids.tolist() if hasattr(p_ids, "tolist") else list(p_ids)
+        c_list = c_ids.tolist() if hasattr(c_ids, "tolist") else list(c_ids)
+        turn_traces.append(
+            {
+                "prompt_ids": p_list,
+                "completion_ids": c_list,
+                "logprobs": [float(x) for x in lps],
+            }
+        )
 
         completion_text = rollout_outputs.get("text") or tokenizer.decode(
             rollout_outputs["completion_ids"], skip_special_tokens=True
@@ -125,10 +142,17 @@ def rollout_once(
     total_reward = float(getattr(obs, "cumulative_reward", sum(phase_rewards.values())))
     total_reward = max(0.0, min(total_reward, 1.0))
 
+    if not turn_traces:
+        raise ValueError(
+            "rollout_once produced no vLLM turns (max_turns too low or env ended "
+            "before the first action)."
+        )
+    last = turn_traces[-1]
+
     return {
-        "prompt_ids": prompt_ids,
-        "completion_ids": completion_ids,
-        "logprobs": logprobs,
+        "prompt_ids": last["prompt_ids"],
+        "completion_ids": last["completion_ids"],
+        "logprobs": last["logprobs"],
         "total_reward": total_reward,
         "market_reward": float(phase_rewards["market"]),
         "warehouse_reward": float(phase_rewards["warehouse"]),

training/training_artifacts_v1/metrics.csv

@@ -0,0 +1,20 @@
+step,loss,grad_norm,learning_rate,num_tokens,completions/mean_length,completions/min_length,completions/max_length,completions/clipped_ratio,completions/mean_terminated_length,completions/min_terminated_length,completions/max_terminated_length,rewards/reward_total/mean,rewards/reward_total/std,rewards/reward_market/mean,rewards/reward_market/std,rewards/reward_warehouse/mean,rewards/reward_warehouse/std,rewards/reward_showroom/mean,rewards/reward_showroom/std,reward,reward_std,frac_reward_zero_std,sampling/sampling_logp_difference/mean,sampling/sampling_logp_difference/max,sampling/importance_sampling_ratio/min,sampling/importance_sampling_ratio/mean,sampling/importance_sampling_ratio/max,entropy,clip_ratio/low_mean,clip_ratio/low_min,clip_ratio/high_mean,clip_ratio/high_max,clip_ratio/region_mean,step_time,epoch,train_runtime,train_samples_per_second,train_steps_per_second,total_flos,train_loss
+1,-0.009,12.235088348388672,0.0,27758.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7793656587600708,0.12745577096939087,0.25,0.13440430164337158,0.42500001192092896,0.20160645246505737,0.10436563193798065,0.069697305560112,0.7793656587600708,0.12745577096939087,0.0,0.005470390431582928,0.22214925289154053,0.8007970452308655,0.9949374794960022,1.076386570930481,0.028439456821217846,0.0,0.0,0.0,0.0,0.0,19.228480510413647,0.05555555555555555,,,,,
+2,0.0723,60.551937103271484,5.000000000000001e-07,55324.0,3.25,3.0,7.0,0.0,3.25,3.0,7.0,0.719434380531311,0.1505809724330902,0.30000001192092896,0.17597654461860657,0.30000001192092896,0.17597654461860657,0.11943437159061432,0.07055392116308212,0.719434380531311,0.1505809724330902,0.0,0.01085888221859932,0.28092825412750244,0.8382877111434937,1.0260276794433594,1.324359655380249,0.036137547835437545,0.0,0.0,0.0,0.0,0.0,17.92062332853675,0.1111111111111111,,,,,
+3,0.1153,232.934814453125,1.0000000000000002e-06,83000.0,3.5,3.0,7.0,0.0,3.5,3.0,7.0,0.7885687351226807,0.1279384195804596,0.32500001788139343,0.18837162852287292,0.375,0.20160646736621857,0.08856874704360962,0.07010025531053543,0.7885687351226807,0.1279384344816208,0.0,0.0251829382032156,0.6850378513336182,0.5226751565933228,1.0358223915100098,1.9838452339172363,0.03248842835137111,0.0,0.0,0.0,0.0,0.0,17.184778176248074,0.16666666666666666,,,,,
+4,0.0243,13.620709419250488,1.5e-06,110708.0,3.125,3.0,7.0,0.0,3.125,3.0,7.0,0.7762374877929688,0.13016164302825928,0.32499998807907104,0.18837164342403412,0.3500000238418579,0.1967477649450302,0.10123749077320099,0.06825561076402664,0.7762374877929688,0.13016162812709808,0.0,0.007013080175966024,0.2646750509738922,0.7674550414085388,0.9837819337844849,1.0260045528411865,0.03743034108288157,0.0,0.0,0.0,0.0,0.0,17.90316915512085,0.2222222222222222,,,,,
+5,0.0004,2.0591225624084473,2.0000000000000003e-06,138452.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7784374952316284,0.12858590483665466,0.30000001192092896,0.17597654461860657,0.375,0.20160646736621857,0.10343749821186066,0.06380020827054977,0.7784374952316284,0.12858593463897705,0.0,0.001465568202547729,0.05673474818468094,0.9749767780303955,1.0017430782318115,1.058376669883728,0.008188390799773515,0.0,0.0,0.0,0.0,0.0,17.8210555203259,0.2777777777777778,,,,,
+6,0.0915,31.611696243286133,2.5e-06,166252.0,3.125,3.0,7.0,0.0,3.125,3.0,7.0,0.7915937900543213,0.12960509955883026,0.375,0.20160646736621857,0.32500001788139343,0.18837162852287292,0.09159374982118607,0.0639258474111557,0.7915937900543213,0.12960509955883026,0.0,0.008641102351248264,0.8236088752746582,0.9935171008110046,1.0399717092514038,2.278707504272461,0.007084679029730978,0.0,0.0,0.0,0.0,0.0,18.056264080107212,0.3333333333333333,,,,,
+7,-0.0,0.281630277633667,3e-06,193950.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7600874900817871,0.13816162943840027,0.32500001788139343,0.18837162852287292,0.32500001788139343,0.18837162852287292,0.11008749902248383,0.07028691470623016,0.7600874900817871,0.13816164433956146,0.0,3.554227441782132e-05,0.0027569520752876997,0.997247576713562,0.9999052286148071,1.0000724792480469,0.0005298306713825696,0.0,0.0,0.0,0.0,0.0,17.43799263238907,0.3888888888888889,,,,,
+8,0.0,0.00018881642608903348,3.5e-06,221677.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7622687816619873,0.1372590959072113,0.4000000059604645,0.20320022106170654,0.25,0.13440430164337158,0.11226874589920044,0.07360353320837021,0.7622687816619873,0.1372590959072113,0.0,2.545601773817907e-07,1.5496943888138048e-06,0.9999986886978149,1.000000238418579,1.0000016689300537,4.1391018498870835e-05,0.0,0.0,0.0,0.0,0.0,17.406394600868225,0.4444444444444444,,,,,
+9,0.0,0.00015632262511644512,4.000000000000001e-06,249549.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.775946855545044,0.137176051735878,0.375,0.20160646736621857,0.30000001192092896,0.17597654461860657,0.10094687342643738,0.058497413992881775,0.775946855545044,0.1371760368347168,0.0,2.918131087881193e-07,3.099441755693988e-06,0.9999978542327881,1.0,1.000001311302185,4.173239403826301e-05,0.0,0.0,0.0,0.0,0.0,18.514019537717104,0.5,,,,,
+10,0.0,9.833038348006085e-05,4.5e-06,277377.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7337374687194824,0.15055809915065765,0.2750000059604645,0.1586231142282486,0.3500000238418579,0.1967477649450302,0.10873749852180481,0.06778524816036224,0.7337374687194824,0.15055811405181885,0.0,2.719489771152439e-07,1.9073031580774114e-06,0.9999985694885254,0.9999998807907104,1.0000019073486328,4.224909940830912e-05,0.0,0.0,0.0,0.0,0.0,18.26371632888913,0.5555555555555556,,,,,
+11,-0.0,0.00015879125567153096,5e-06,305332.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7567968368530273,0.13728150725364685,0.375,0.20160646736621857,0.2750000059604645,0.1586231142282486,0.10679687559604645,0.07404065877199173,0.7567968368530273,0.13728150725364685,0.0,3.067227396513772e-07,2.6226434783893637e-06,0.9999973773956299,0.9999994039535522,1.0000019073486328,4.7876037001515215e-05,0.0,0.0,0.0,0.0,0.0,19.4472255371511,0.6111111111111112,,,,,
+12,-0.0,0.00039661259506829083,4.3750000000000005e-06,333165.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7081500291824341,0.1421954482793808,0.30000001192092896,0.17597654461860657,0.2750000059604645,0.1586231142282486,0.13315001130104065,0.07370516657829285,0.7081500291824341,0.142195463180542,0.0,4.768499479723687e-07,5.602954843197949e-06,0.9999943971633911,0.9999988675117493,1.0000016689300537,5.9777358274004655e-05,0.0,0.0,0.0,0.0,0.0,18.45015063509345,0.6666666666666666,,,,,
+13,0.0,0.0005282312049530447,3.7500000000000005e-06,361006.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7762781381607056,0.13823458552360535,0.32500001788139343,0.18837162852287292,0.3500000238418579,0.1967477649450302,0.10127812623977661,0.06158862262964249,0.7762781381607056,0.13823460042476654,0.0,6.830008487668238e-07,5.60290391149465e-06,0.9999943971633911,0.9999983310699463,1.0000022649765015,7.182803437899565e-05,0.0,0.0,0.0,0.0,0.0,17.955969959497452,0.7222222222222222,,,,,
+14,-0.0,0.000535853614564985,3.125e-06,388862.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7628874778747559,0.1295449286699295,0.30000001192092896,0.17597654461860657,0.3499999940395355,0.19674775004386902,0.11288750171661377,0.073255755007267,0.7628874778747559,0.1295449435710907,0.0,1.1113726259281975e-06,2.0979605324100703e-05,0.9999922513961792,1.0000004768371582,1.0000211000442505,9.972968496185786e-05,0.0,0.0,0.0,0.0,0.0,18.99697282537818,0.7777777777777778,,,,,
+15,0.0,0.005312301218509674,2.5e-06,416636.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7899656295776367,0.1296130269765854,0.3499999940395355,0.19674775004386902,0.3500000238418579,0.1967477649450302,0.08996562659740448,0.05463240668177605,0.7899656295776367,0.12961304187774658,0.0,4.004845322924666e-06,4.589592936099507e-05,0.9999337792396545,0.9999885559082031,1.0000029802322388,0.0001807671503684105,0.0,0.0,0.0,0.0,0.0,19.184648096561432,0.8333333333333334,,,,,
+16,-0.0,0.005995223298668861,1.8750000000000003e-06,444544.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.789996862411499,0.12048782408237457,0.375,0.20160646736621857,0.32039374113082886,0.18316024541854858,0.09460312128067017,0.062435995787382126,0.789996862411499,0.12048781663179398,0.0,4.866625204158481e-06,2.9087463190080598e-05,0.999959409236908,0.9999855160713196,1.0000004768371582,0.00019320984370096994,0.0,0.0,0.0,0.0,0.0,19.476148523390293,0.8888888888888888,,,,,
+17,0.0,0.011378168128430843,1.25e-06,472481.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.8168656826019287,0.09262391924858093,0.375,0.20160646736621857,0.3498094081878662,0.19690066576004028,0.09205625206232071,0.07046373933553696,0.8168656826019287,0.09262390434741974,0.0,5.950785180175444e-06,4.7328881919384e-05,0.9999284744262695,0.9999822974205017,1.000001072883606,0.00021975090658088448,0.0,0.0,0.0,0.0,0.0,19.344543006271124,0.9444444444444444,,,,,
+18,0.0,0.018683306872844696,6.25e-07,500266.0,3.0,3.0,3.0,0.0,3.0,3.0,3.0,0.7750625014305115,0.14084643125534058,0.3500000238418579,0.1967477649450302,0.32499998807907104,0.18837164342403412,0.10006250441074371,0.05640558898448944,0.7750625014305115,0.14084644615650177,0.0625,1.1303089195280336e-05,0.00027322862297296524,0.999584436416626,0.9999662637710571,0.9999997615814209,0.000320568448614722,0.0,0.0,0.0,0.0,0.0,17.66909484937787,1.0,,,,,
+18,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,1.0,406.352,0.738,0.044,0.0,0.016376476217475202

training/training_artifacts_v1/metrics.json

@@ -0,0 +1,695 @@
+[
+  {
+    "step": 1,
+    "loss": -0.009,
+    "grad_norm": 12.235088348388672,
+    "learning_rate": 0.0,
+    "num_tokens": 27758.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7793656587600708,
+    "rewards/reward_total/std": 0.12745577096939087,
+    "rewards/reward_market/mean": 0.25,
+    "rewards/reward_market/std": 0.13440430164337158,
+    "rewards/reward_warehouse/mean": 0.42500001192092896,
+    "rewards/reward_warehouse/std": 0.20160645246505737,
+    "rewards/reward_showroom/mean": 0.10436563193798065,
+    "rewards/reward_showroom/std": 0.069697305560112,
+    "reward": 0.7793656587600708,
+    "reward_std": 0.12745577096939087,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 0.005470390431582928,
+    "sampling/sampling_logp_difference/max": 0.22214925289154053,
+    "sampling/importance_sampling_ratio/min": 0.8007970452308655,
+    "sampling/importance_sampling_ratio/mean": 0.9949374794960022,
+    "sampling/importance_sampling_ratio/max": 1.076386570930481,
+    "entropy": 0.028439456821217846,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 19.228480510413647,
+    "epoch": 0.05555555555555555
+  },
+  {
+    "step": 2,
+    "loss": 0.0723,
+    "grad_norm": 60.551937103271484,
+    "learning_rate": 5.000000000000001e-07,
+    "num_tokens": 55324.0,
+    "completions/mean_length": 3.25,
+    "completions/min_length": 3.0,
+    "completions/max_length": 7.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.25,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 7.0,
+    "rewards/reward_total/mean": 0.719434380531311,
+    "rewards/reward_total/std": 0.1505809724330902,
+    "rewards/reward_market/mean": 0.30000001192092896,
+    "rewards/reward_market/std": 0.17597654461860657,
+    "rewards/reward_warehouse/mean": 0.30000001192092896,
+    "rewards/reward_warehouse/std": 0.17597654461860657,
+    "rewards/reward_showroom/mean": 0.11943437159061432,
+    "rewards/reward_showroom/std": 0.07055392116308212,
+    "reward": 0.719434380531311,
+    "reward_std": 0.1505809724330902,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 0.01085888221859932,
+    "sampling/sampling_logp_difference/max": 0.28092825412750244,
+    "sampling/importance_sampling_ratio/min": 0.8382877111434937,
+    "sampling/importance_sampling_ratio/mean": 1.0260276794433594,
+    "sampling/importance_sampling_ratio/max": 1.324359655380249,
+    "entropy": 0.036137547835437545,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 17.92062332853675,
+    "epoch": 0.1111111111111111
+  },
+  {
+    "step": 3,
+    "loss": 0.1153,
+    "grad_norm": 232.934814453125,
+    "learning_rate": 1.0000000000000002e-06,
+    "num_tokens": 83000.0,
+    "completions/mean_length": 3.5,
+    "completions/min_length": 3.0,
+    "completions/max_length": 7.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.5,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 7.0,
+    "rewards/reward_total/mean": 0.7885687351226807,
+    "rewards/reward_total/std": 0.1279384195804596,
+    "rewards/reward_market/mean": 0.32500001788139343,
+    "rewards/reward_market/std": 0.18837162852287292,
+    "rewards/reward_warehouse/mean": 0.375,
+    "rewards/reward_warehouse/std": 0.20160646736621857,
+    "rewards/reward_showroom/mean": 0.08856874704360962,
+    "rewards/reward_showroom/std": 0.07010025531053543,
+    "reward": 0.7885687351226807,
+    "reward_std": 0.1279384344816208,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 0.0251829382032156,
+    "sampling/sampling_logp_difference/max": 0.6850378513336182,
+    "sampling/importance_sampling_ratio/min": 0.5226751565933228,
+    "sampling/importance_sampling_ratio/mean": 1.0358223915100098,
+    "sampling/importance_sampling_ratio/max": 1.9838452339172363,
+    "entropy": 0.03248842835137111,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 17.184778176248074,
+    "epoch": 0.16666666666666666
+  },
+  {
+    "step": 4,
+    "loss": 0.0243,
+    "grad_norm": 13.620709419250488,
+    "learning_rate": 1.5e-06,
+    "num_tokens": 110708.0,
+    "completions/mean_length": 3.125,
+    "completions/min_length": 3.0,
+    "completions/max_length": 7.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.125,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 7.0,
+    "rewards/reward_total/mean": 0.7762374877929688,
+    "rewards/reward_total/std": 0.13016164302825928,
+    "rewards/reward_market/mean": 0.32499998807907104,
+    "rewards/reward_market/std": 0.18837164342403412,
+    "rewards/reward_warehouse/mean": 0.3500000238418579,
+    "rewards/reward_warehouse/std": 0.1967477649450302,
+    "rewards/reward_showroom/mean": 0.10123749077320099,
+    "rewards/reward_showroom/std": 0.06825561076402664,
+    "reward": 0.7762374877929688,
+    "reward_std": 0.13016162812709808,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 0.007013080175966024,
+    "sampling/sampling_logp_difference/max": 0.2646750509738922,
+    "sampling/importance_sampling_ratio/min": 0.7674550414085388,
+    "sampling/importance_sampling_ratio/mean": 0.9837819337844849,
+    "sampling/importance_sampling_ratio/max": 1.0260045528411865,
+    "entropy": 0.03743034108288157,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 17.90316915512085,
+    "epoch": 0.2222222222222222
+  },
+  {
+    "step": 5,
+    "loss": 0.0004,
+    "grad_norm": 2.0591225624084473,
+    "learning_rate": 2.0000000000000003e-06,
+    "num_tokens": 138452.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7784374952316284,
+    "rewards/reward_total/std": 0.12858590483665466,
+    "rewards/reward_market/mean": 0.30000001192092896,
+    "rewards/reward_market/std": 0.17597654461860657,
+    "rewards/reward_warehouse/mean": 0.375,
+    "rewards/reward_warehouse/std": 0.20160646736621857,
+    "rewards/reward_showroom/mean": 0.10343749821186066,
+    "rewards/reward_showroom/std": 0.06380020827054977,
+    "reward": 0.7784374952316284,
+    "reward_std": 0.12858593463897705,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 0.001465568202547729,
+    "sampling/sampling_logp_difference/max": 0.05673474818468094,
+    "sampling/importance_sampling_ratio/min": 0.9749767780303955,
+    "sampling/importance_sampling_ratio/mean": 1.0017430782318115,
+    "sampling/importance_sampling_ratio/max": 1.058376669883728,
+    "entropy": 0.008188390799773515,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 17.8210555203259,
+    "epoch": 0.2777777777777778
+  },
+  {
+    "step": 6,
+    "loss": 0.0915,
+    "grad_norm": 31.611696243286133,
+    "learning_rate": 2.5e-06,
+    "num_tokens": 166252.0,
+    "completions/mean_length": 3.125,
+    "completions/min_length": 3.0,
+    "completions/max_length": 7.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.125,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 7.0,
+    "rewards/reward_total/mean": 0.7915937900543213,
+    "rewards/reward_total/std": 0.12960509955883026,
+    "rewards/reward_market/mean": 0.375,
+    "rewards/reward_market/std": 0.20160646736621857,
+    "rewards/reward_warehouse/mean": 0.32500001788139343,
+    "rewards/reward_warehouse/std": 0.18837162852287292,
+    "rewards/reward_showroom/mean": 0.09159374982118607,
+    "rewards/reward_showroom/std": 0.0639258474111557,
+    "reward": 0.7915937900543213,
+    "reward_std": 0.12960509955883026,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 0.008641102351248264,
+    "sampling/sampling_logp_difference/max": 0.8236088752746582,
+    "sampling/importance_sampling_ratio/min": 0.9935171008110046,
+    "sampling/importance_sampling_ratio/mean": 1.0399717092514038,
+    "sampling/importance_sampling_ratio/max": 2.278707504272461,
+    "entropy": 0.007084679029730978,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 18.056264080107212,
+    "epoch": 0.3333333333333333
+  },
+  {
+    "step": 7,
+    "loss": -0.0,
+    "grad_norm": 0.281630277633667,
+    "learning_rate": 3e-06,
+    "num_tokens": 193950.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7600874900817871,
+    "rewards/reward_total/std": 0.13816162943840027,
+    "rewards/reward_market/mean": 0.32500001788139343,
+    "rewards/reward_market/std": 0.18837162852287292,
+    "rewards/reward_warehouse/mean": 0.32500001788139343,
+    "rewards/reward_warehouse/std": 0.18837162852287292,
+    "rewards/reward_showroom/mean": 0.11008749902248383,
+    "rewards/reward_showroom/std": 0.07028691470623016,
+    "reward": 0.7600874900817871,
+    "reward_std": 0.13816164433956146,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 3.554227441782132e-05,
+    "sampling/sampling_logp_difference/max": 0.0027569520752876997,
+    "sampling/importance_sampling_ratio/min": 0.997247576713562,
+    "sampling/importance_sampling_ratio/mean": 0.9999052286148071,
+    "sampling/importance_sampling_ratio/max": 1.0000724792480469,
+    "entropy": 0.0005298306713825696,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 17.43799263238907,
+    "epoch": 0.3888888888888889
+  },
+  {
+    "step": 8,
+    "loss": 0.0,
+    "grad_norm": 0.00018881642608903348,
+    "learning_rate": 3.5e-06,
+    "num_tokens": 221677.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7622687816619873,
+    "rewards/reward_total/std": 0.1372590959072113,
+    "rewards/reward_market/mean": 0.4000000059604645,
+    "rewards/reward_market/std": 0.20320022106170654,
+    "rewards/reward_warehouse/mean": 0.25,
+    "rewards/reward_warehouse/std": 0.13440430164337158,
+    "rewards/reward_showroom/mean": 0.11226874589920044,
+    "rewards/reward_showroom/std": 0.07360353320837021,
+    "reward": 0.7622687816619873,
+    "reward_std": 0.1372590959072113,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 2.545601773817907e-07,
+    "sampling/sampling_logp_difference/max": 1.5496943888138048e-06,
+    "sampling/importance_sampling_ratio/min": 0.9999986886978149,
+    "sampling/importance_sampling_ratio/mean": 1.000000238418579,
+    "sampling/importance_sampling_ratio/max": 1.0000016689300537,
+    "entropy": 4.1391018498870835e-05,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 17.406394600868225,
+    "epoch": 0.4444444444444444
+  },
+  {
+    "step": 9,
+    "loss": 0.0,
+    "grad_norm": 0.00015632262511644512,
+    "learning_rate": 4.000000000000001e-06,
+    "num_tokens": 249549.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.775946855545044,
+    "rewards/reward_total/std": 0.137176051735878,
+    "rewards/reward_market/mean": 0.375,
+    "rewards/reward_market/std": 0.20160646736621857,
+    "rewards/reward_warehouse/mean": 0.30000001192092896,
+    "rewards/reward_warehouse/std": 0.17597654461860657,
+    "rewards/reward_showroom/mean": 0.10094687342643738,
+    "rewards/reward_showroom/std": 0.058497413992881775,
+    "reward": 0.775946855545044,
+    "reward_std": 0.1371760368347168,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 2.918131087881193e-07,
+    "sampling/sampling_logp_difference/max": 3.099441755693988e-06,
+    "sampling/importance_sampling_ratio/min": 0.9999978542327881,
+    "sampling/importance_sampling_ratio/mean": 1.0,
+    "sampling/importance_sampling_ratio/max": 1.000001311302185,
+    "entropy": 4.173239403826301e-05,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 18.514019537717104,
+    "epoch": 0.5
+  },
+  {
+    "step": 10,
+    "loss": 0.0,
+    "grad_norm": 9.833038348006085e-05,
+    "learning_rate": 4.5e-06,
+    "num_tokens": 277377.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7337374687194824,
+    "rewards/reward_total/std": 0.15055809915065765,
+    "rewards/reward_market/mean": 0.2750000059604645,
+    "rewards/reward_market/std": 0.1586231142282486,
+    "rewards/reward_warehouse/mean": 0.3500000238418579,
+    "rewards/reward_warehouse/std": 0.1967477649450302,
+    "rewards/reward_showroom/mean": 0.10873749852180481,
+    "rewards/reward_showroom/std": 0.06778524816036224,
+    "reward": 0.7337374687194824,
+    "reward_std": 0.15055811405181885,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 2.719489771152439e-07,
+    "sampling/sampling_logp_difference/max": 1.9073031580774114e-06,
+    "sampling/importance_sampling_ratio/min": 0.9999985694885254,
+    "sampling/importance_sampling_ratio/mean": 0.9999998807907104,
+    "sampling/importance_sampling_ratio/max": 1.0000019073486328,
+    "entropy": 4.224909940830912e-05,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 18.26371632888913,
+    "epoch": 0.5555555555555556
+  },
+  {
+    "step": 11,
+    "loss": -0.0,
+    "grad_norm": 0.00015879125567153096,
+    "learning_rate": 5e-06,
+    "num_tokens": 305332.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7567968368530273,
+    "rewards/reward_total/std": 0.13728150725364685,
+    "rewards/reward_market/mean": 0.375,
+    "rewards/reward_market/std": 0.20160646736621857,
+    "rewards/reward_warehouse/mean": 0.2750000059604645,
+    "rewards/reward_warehouse/std": 0.1586231142282486,
+    "rewards/reward_showroom/mean": 0.10679687559604645,
+    "rewards/reward_showroom/std": 0.07404065877199173,
+    "reward": 0.7567968368530273,
+    "reward_std": 0.13728150725364685,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 3.067227396513772e-07,
+    "sampling/sampling_logp_difference/max": 2.6226434783893637e-06,
+    "sampling/importance_sampling_ratio/min": 0.9999973773956299,
+    "sampling/importance_sampling_ratio/mean": 0.9999994039535522,
+    "sampling/importance_sampling_ratio/max": 1.0000019073486328,
+    "entropy": 4.7876037001515215e-05,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 19.4472255371511,
+    "epoch": 0.6111111111111112
+  },
+  {
+    "step": 12,
+    "loss": -0.0,
+    "grad_norm": 0.00039661259506829083,
+    "learning_rate": 4.3750000000000005e-06,
+    "num_tokens": 333165.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7081500291824341,
+    "rewards/reward_total/std": 0.1421954482793808,
+    "rewards/reward_market/mean": 0.30000001192092896,
+    "rewards/reward_market/std": 0.17597654461860657,
+    "rewards/reward_warehouse/mean": 0.2750000059604645,
+    "rewards/reward_warehouse/std": 0.1586231142282486,
+    "rewards/reward_showroom/mean": 0.13315001130104065,
+    "rewards/reward_showroom/std": 0.07370516657829285,
+    "reward": 0.7081500291824341,
+    "reward_std": 0.142195463180542,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 4.768499479723687e-07,
+    "sampling/sampling_logp_difference/max": 5.602954843197949e-06,
+    "sampling/importance_sampling_ratio/min": 0.9999943971633911,
+    "sampling/importance_sampling_ratio/mean": 0.9999988675117493,
+    "sampling/importance_sampling_ratio/max": 1.0000016689300537,
+    "entropy": 5.9777358274004655e-05,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 18.45015063509345,
+    "epoch": 0.6666666666666666
+  },
+  {
+    "step": 13,
+    "loss": 0.0,
+    "grad_norm": 0.0005282312049530447,
+    "learning_rate": 3.7500000000000005e-06,
+    "num_tokens": 361006.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7762781381607056,
+    "rewards/reward_total/std": 0.13823458552360535,
+    "rewards/reward_market/mean": 0.32500001788139343,
+    "rewards/reward_market/std": 0.18837162852287292,
+    "rewards/reward_warehouse/mean": 0.3500000238418579,
+    "rewards/reward_warehouse/std": 0.1967477649450302,
+    "rewards/reward_showroom/mean": 0.10127812623977661,
+    "rewards/reward_showroom/std": 0.06158862262964249,
+    "reward": 0.7762781381607056,
+    "reward_std": 0.13823460042476654,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 6.830008487668238e-07,
+    "sampling/sampling_logp_difference/max": 5.60290391149465e-06,
+    "sampling/importance_sampling_ratio/min": 0.9999943971633911,
+    "sampling/importance_sampling_ratio/mean": 0.9999983310699463,
+    "sampling/importance_sampling_ratio/max": 1.0000022649765015,
+    "entropy": 7.182803437899565e-05,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 17.955969959497452,
+    "epoch": 0.7222222222222222
+  },
+  {
+    "step": 14,
+    "loss": -0.0,
+    "grad_norm": 0.000535853614564985,
+    "learning_rate": 3.125e-06,
+    "num_tokens": 388862.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7628874778747559,
+    "rewards/reward_total/std": 0.1295449286699295,
+    "rewards/reward_market/mean": 0.30000001192092896,
+    "rewards/reward_market/std": 0.17597654461860657,
+    "rewards/reward_warehouse/mean": 0.3499999940395355,
+    "rewards/reward_warehouse/std": 0.19674775004386902,
+    "rewards/reward_showroom/mean": 0.11288750171661377,
+    "rewards/reward_showroom/std": 0.073255755007267,
+    "reward": 0.7628874778747559,
+    "reward_std": 0.1295449435710907,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 1.1113726259281975e-06,
+    "sampling/sampling_logp_difference/max": 2.0979605324100703e-05,
+    "sampling/importance_sampling_ratio/min": 0.9999922513961792,
+    "sampling/importance_sampling_ratio/mean": 1.0000004768371582,
+    "sampling/importance_sampling_ratio/max": 1.0000211000442505,
+    "entropy": 9.972968496185786e-05,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 18.99697282537818,
+    "epoch": 0.7777777777777778
+  },
+  {
+    "step": 15,
+    "loss": 0.0,
+    "grad_norm": 0.005312301218509674,
+    "learning_rate": 2.5e-06,
+    "num_tokens": 416636.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7899656295776367,
+    "rewards/reward_total/std": 0.1296130269765854,
+    "rewards/reward_market/mean": 0.3499999940395355,
+    "rewards/reward_market/std": 0.19674775004386902,
+    "rewards/reward_warehouse/mean": 0.3500000238418579,
+    "rewards/reward_warehouse/std": 0.1967477649450302,
+    "rewards/reward_showroom/mean": 0.08996562659740448,
+    "rewards/reward_showroom/std": 0.05463240668177605,
+    "reward": 0.7899656295776367,
+    "reward_std": 0.12961304187774658,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 4.004845322924666e-06,
+    "sampling/sampling_logp_difference/max": 4.589592936099507e-05,
+    "sampling/importance_sampling_ratio/min": 0.9999337792396545,
+    "sampling/importance_sampling_ratio/mean": 0.9999885559082031,
+    "sampling/importance_sampling_ratio/max": 1.0000029802322388,
+    "entropy": 0.0001807671503684105,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 19.184648096561432,
+    "epoch": 0.8333333333333334
+  },
+  {
+    "step": 16,
+    "loss": -0.0,
+    "grad_norm": 0.005995223298668861,
+    "learning_rate": 1.8750000000000003e-06,
+    "num_tokens": 444544.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.789996862411499,
+    "rewards/reward_total/std": 0.12048782408237457,
+    "rewards/reward_market/mean": 0.375,
+    "rewards/reward_market/std": 0.20160646736621857,
+    "rewards/reward_warehouse/mean": 0.32039374113082886,
+    "rewards/reward_warehouse/std": 0.18316024541854858,
+    "rewards/reward_showroom/mean": 0.09460312128067017,
+    "rewards/reward_showroom/std": 0.062435995787382126,
+    "reward": 0.789996862411499,
+    "reward_std": 0.12048781663179398,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 4.866625204158481e-06,
+    "sampling/sampling_logp_difference/max": 2.9087463190080598e-05,
+    "sampling/importance_sampling_ratio/min": 0.999959409236908,
+    "sampling/importance_sampling_ratio/mean": 0.9999855160713196,
+    "sampling/importance_sampling_ratio/max": 1.0000004768371582,
+    "entropy": 0.00019320984370096994,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 19.476148523390293,
+    "epoch": 0.8888888888888888
+  },
+  {
+    "step": 17,
+    "loss": 0.0,
+    "grad_norm": 0.011378168128430843,
+    "learning_rate": 1.25e-06,
+    "num_tokens": 472481.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.8168656826019287,
+    "rewards/reward_total/std": 0.09262391924858093,
+    "rewards/reward_market/mean": 0.375,
+    "rewards/reward_market/std": 0.20160646736621857,
+    "rewards/reward_warehouse/mean": 0.3498094081878662,
+    "rewards/reward_warehouse/std": 0.19690066576004028,
+    "rewards/reward_showroom/mean": 0.09205625206232071,
+    "rewards/reward_showroom/std": 0.07046373933553696,
+    "reward": 0.8168656826019287,
+    "reward_std": 0.09262390434741974,
+    "frac_reward_zero_std": 0.0,
+    "sampling/sampling_logp_difference/mean": 5.950785180175444e-06,
+    "sampling/sampling_logp_difference/max": 4.7328881919384e-05,
+    "sampling/importance_sampling_ratio/min": 0.9999284744262695,
+    "sampling/importance_sampling_ratio/mean": 0.9999822974205017,
+    "sampling/importance_sampling_ratio/max": 1.000001072883606,
+    "entropy": 0.00021975090658088448,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 19.344543006271124,
+    "epoch": 0.9444444444444444
+  },
+  {
+    "step": 18,
+    "loss": 0.0,
+    "grad_norm": 0.018683306872844696,
+    "learning_rate": 6.25e-07,
+    "num_tokens": 500266.0,
+    "completions/mean_length": 3.0,
+    "completions/min_length": 3.0,
+    "completions/max_length": 3.0,
+    "completions/clipped_ratio": 0.0,
+    "completions/mean_terminated_length": 3.0,
+    "completions/min_terminated_length": 3.0,
+    "completions/max_terminated_length": 3.0,
+    "rewards/reward_total/mean": 0.7750625014305115,
+    "rewards/reward_total/std": 0.14084643125534058,
+    "rewards/reward_market/mean": 0.3500000238418579,
+    "rewards/reward_market/std": 0.1967477649450302,
+    "rewards/reward_warehouse/mean": 0.32499998807907104,
+    "rewards/reward_warehouse/std": 0.18837164342403412,
+    "rewards/reward_showroom/mean": 0.10006250441074371,
+    "rewards/reward_showroom/std": 0.05640558898448944,
+    "reward": 0.7750625014305115,
+    "reward_std": 0.14084644615650177,
+    "frac_reward_zero_std": 0.0625,
+    "sampling/sampling_logp_difference/mean": 1.1303089195280336e-05,
+    "sampling/sampling_logp_difference/max": 0.00027322862297296524,
+    "sampling/importance_sampling_ratio/min": 0.999584436416626,
+    "sampling/importance_sampling_ratio/mean": 0.9999662637710571,
+    "sampling/importance_sampling_ratio/max": 0.9999997615814209,
+    "entropy": 0.000320568448614722,
+    "clip_ratio/low_mean": 0.0,
+    "clip_ratio/low_min": 0.0,
+    "clip_ratio/high_mean": 0.0,
+    "clip_ratio/high_max": 0.0,
+    "clip_ratio/region_mean": 0.0,
+    "step_time": 17.66909484937787,
+    "epoch": 1.0
+  },
+  {
+    "step": 18,
+    "train_runtime": 406.352,
+    "train_samples_per_second": 0.738,
+    "train_steps_per_second": 0.044,
+    "total_flos": 0.0,
+    "train_loss": 0.016376476217475202,
+    "epoch": 1.0
+  }
+]

training/training_artifacts_v1/training_summary.json

@@ -0,0 +1,63 @@
+{
+  "loss": {
+    "final": 0.0,
+    "max": 0.1153,
+    "min": -0.009,
+    "mean": 0.01637777777777778,
+    "n": 18
+  },
+  "reward_total": {
+    "final": 0.7750625014305115,
+    "max": 0.8168656826019287,
+    "min": 0.7081500291824341,
+    "mean": 0.7689822945329878,
+    "n": 18
+  },
+  "reward_market": {
+    "final": 0.0,
+    "max": 0.0,
+    "min": 0.0,
+    "mean": 0.0,
+    "n": 0
+  },
+  "reward_warehouse": {
+    "final": 0.0,
+    "max": 0.0,
+    "min": 0.0,
+    "mean": 0.0,
+    "n": 0
+  },
+  "reward_showroom": {
+    "final": 0.0,
+    "max": 0.0,
+    "min": 0.0,
+    "mean": 0.0,
+    "n": 0
+  },
+  "n_log_rows": 19,
+  "output_dir": "/workspace/shopmanager-grpo-qwen3",
+  "run_config": {
+    "model": "Qwen/Qwen3-1.7B",
+    "env_url": "https://hard007ik-shopmanagereng.hf.space",
+    "dataset_size": 300,
+    "num_generations": 2,
+    "per_device_batch": 1,
+    "grad_accum": 32,
+    "max_completion_length": 64,
+    "max_turns": 15,
+    "lr": 5e-06,
+    "warmup_steps": 10,
+    "max_steps": -1,
+    "epochs": 1,
+    "vllm_gpu_mem": 0.3,
+    "reward_weights": [
+      1.0,
+      0.0,
+      0.0,
+      0.0
+    ],
+    "precision": {
+      "bf16": true
+    }
+  }
+}

ui.py

@@ -0,0 +1,358 @@
+"""
+Streamlit UI for ShopManagerEng — Interactive Jewelry Shop Demo.
+
+An AI heuristic agent automatically plays through each episode.
+Users press "New Episode" and watch the agent navigate all 3 phases.
+"""
+
+import os
+import sys
+import random
+import time
+from pathlib import Path
+
+import streamlit as st
+
+# ── Ensure imports resolve ──────────────────────────────────────────────────
+ROOT = Path(__file__).resolve().parent
+if str(ROOT) not in sys.path:
+    sys.path.insert(0, str(ROOT))
+
+os.environ.setdefault("SHOPMANAGER_MARKET_MODE", "synthetic")
+
+from server.ShopManagerEng_environment import JewelryShopEnvironment
+from models import JewelryAction, PRODUCT_CATALOG
+
+
+# ── Page config ─────────────────────────────────────────────────────────────
+st.set_page_config(
+    page_title="ShopManagerEng — Jewelry Shop RL",
+    page_icon="💎",
+    layout="wide",
+    initial_sidebar_state="expanded",
+)
+
+# ── CSS: clean light-theme styling ──────────────────────────────────────────
+st.markdown("""
+<style>
+@import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;600;700;800&display=swap');
+html, body, [class*="st-"] { font-family: 'Inter', sans-serif; }
+
+.hero-title {
+    font-size: 36px; font-weight: 800;
+    background: linear-gradient(135deg, #7c3aed, #3b82f6, #10b981);
+    -webkit-background-clip: text; -webkit-text-fill-color: transparent;
+    margin-bottom: 2px;
+}
+.hero-sub { font-size: 15px; color: #64748b; margin-bottom: 20px; }
+
+.metric-card {
+    background: #f8fafc; border: 1px solid #e2e8f0; border-radius: 12px;
+    padding: 14px 18px; text-align: center; min-height: 90px;
+}
+.metric-card .label { font-size: 12px; color: #64748b; font-weight: 600; text-transform: uppercase; letter-spacing: 0.5px; }
+.metric-card .value { font-size: 24px; font-weight: 800; color: #1e293b; margin-top: 4px; }
+
+.phase-box {
+    background: #f1f5f9; border: 1px solid #cbd5e1; border-radius: 12px;
+    padding: 18px; margin-bottom: 12px;
+}
+.phase-box.active { border: 2px solid #7c3aed; background: #f5f3ff; }
+.phase-box h4 { margin: 0 0 6px; color: #1e293b; }
+.phase-box p { margin: 0; color: #475569; font-size: 14px; }
+
+.env-msg {
+    background: #eff6ff; border-left: 4px solid #3b82f6;
+    border-radius: 0 10px 10px 0; padding: 12px 16px;
+    margin: 10px 0; font-size: 13px; color: #1e40af; line-height: 1.5;
+    word-wrap: break-word;
+}
+
+.step-row {
+    padding: 8px 12px; margin: 4px 0; border-radius: 8px;
+    font-size: 13px; color: #334155;
+}
+.step-row.market   { background: #fef3c7; border-left: 3px solid #f59e0b; }
+.step-row.warehouse { background: #dbeafe; border-left: 3px solid #3b82f6; }
+.step-row.showroom  { background: #d1fae5; border-left: 3px solid #10b981; }
+
+.reward-big {
+    text-align: center; padding: 24px;
+    background: linear-gradient(135deg, #f5f3ff, #eff6ff);
+    border: 2px solid #7c3aed; border-radius: 16px;
+}
+.reward-big .score { font-size: 52px; font-weight: 800; color: #7c3aed; }
+.reward-big .label { font-size: 14px; color: #64748b; }
+
+.catalog-card {
+    background: #f8fafc; border: 1px solid #e2e8f0; border-radius: 12px;
+    padding: 16px; text-align: center;
+}
+.catalog-card h4 { margin: 0 0 8px; color: #1e293b; }
+.catalog-card p { margin: 2px 0; color: #475569; font-size: 13px; }
+
+.demand-bar {
+    background: #e2e8f0; border-radius: 6px; height: 10px; overflow: hidden; margin-top: 4px;
+}
+.demand-fill { height: 100%; border-radius: 6px; }
+</style>
+""", unsafe_allow_html=True)
+
+
+# ── Heuristic Agent ─────────────────────────────────────────────────────────
+def heuristic_action(obs):
+    """Simple rule-based agent that plays through all 3 phases."""
+    if obs.phase == "market":
+        price = float(obs.gold_price or 300)
+        # Buy if we have enough cash for at least 1 oz
+        if obs.cash >= price + 10:
+            return JewelryAction(
+                market_action="buy",
+                gold_qty=1.0,
+                target_price_usd=obs.gold_price,
+            )
+        return JewelryAction(market_action="wait")
+
+    if obs.phase == "warehouse":
+        # Pick the highest-demand product we can afford
+        demand = obs.demand or {"ring": 0.5, "necklace": 0.3, "bracelet": 0.2}
+        for name in sorted(demand, key=lambda k: demand.get(k, 0), reverse=True):
+            spec = PRODUCT_CATALOG[name]
+            if obs.gold_oz + 1e-8 >= spec["gold_oz"] and obs.cash >= spec["labor"]:
+                return JewelryAction(product_choice=name)
+        return JewelryAction(product_choice="ring")
+
+    if obs.phase == "showroom":
+        # Accept if margin > 15% or after round 3
+        if (
+            obs.current_offer
+            and obs.cost_basis > 0
+            and float(obs.current_offer) / float(obs.cost_basis) >= 1.15
+        ) or (obs.negotiation_round and int(obs.negotiation_round) >= 3):
+            return JewelryAction(message="I accept")
+        offer = float(obs.current_offer or 0)
+        if offer:
+            return JewelryAction(message=f"How about ${offer * 1.08:.2f}?")
+        return JewelryAction(message="I need a better offer")
+
+    return JewelryAction()
+
+
+# ── Session state ───────────────────────────────────────────────────────────
+if "episode_steps" not in st.session_state:
+    st.session_state.episode_steps = None
+    st.session_state.final_reward = None
+    st.session_state.episode_count = 0
+
+
+def run_episode(task_id):
+    """Run a full episode with the heuristic agent and return step logs."""
+    env = JewelryShopEnvironment()
+    seed = random.randint(0, 99999)
+    obs = env.reset(seed=seed, market_mode="synthetic", task_id=task_id)
+
+    steps = [{
+        "step": 0, "phase": obs.phase, "action": "reset",
+        "msg": obs.message, "reward": 0.0,
+        "cash": obs.cash, "gold_oz": obs.gold_oz,
+        "gold_price": obs.gold_price,
+        "cumulative": float(obs.cumulative_reward),
+    }]
+
+    for i in range(1, 20):
+        if obs.done:
+            break
+        action = heuristic_action(obs)
+
+        # Describe the action in human terms
+        if obs.phase == "market":
+            act_str = f"BUY {action.gold_qty} oz" if action.market_action == "buy" else "WAIT"
+        elif obs.phase == "warehouse":
+            act_str = f"CRAFT {action.product_choice}"
+        else:
+            act_str = action.message or "..."
+
+        obs = env.step(action)
+        steps.append({
+            "step": i, "phase": obs.phase, "action": act_str,
+            "msg": obs.message, "reward": float(obs.reward),
+            "cash": obs.cash, "gold_oz": obs.gold_oz,
+            "gold_price": getattr(obs, "gold_price", 0),
+            "product": getattr(obs, "product_for_sale", None),
+            "offer": float(obs.current_offer) if obs.current_offer else None,
+            "cost_basis": float(obs.cost_basis) if obs.cost_basis else None,
+            "cumulative": float(obs.cumulative_reward),
+        })
+
+    return steps, float(obs.cumulative_reward)
+
+
+# ── Sidebar ─────────────────────────────────────────────────────────────────
+with st.sidebar:
+    st.markdown("### 💎 ShopManagerEng")
+    st.markdown("---")
+
+    task = st.selectbox(
+        "🎯 Task Profile",
+        ["profit_negotiator", "market_timing", "demand_crafter"],
+        index=0,
+    )
+    weights = {
+        "profit_negotiator": "Showroom 60% · Market 20% · Warehouse 20%",
+        "market_timing": "Market 60% · Warehouse 20% · Showroom 20%",
+        "demand_crafter": "Warehouse 60% · Market 20% · Showroom 20%",
+    }
+    st.caption(f"**Weights:** {weights[task]}")
+
+    st.markdown("---")
+    if st.button("🚀 New Episode", use_container_width=True, type="primary"):
+        steps, reward = run_episode(task)
+        st.session_state.episode_steps = steps
+        st.session_state.final_reward = reward
+        st.session_state.episode_count += 1
+        st.rerun()
+
+    st.markdown("---")
+    st.markdown("#### How It Works")
+    st.markdown("""
+    An AI **heuristic agent** automatically plays
+    through all 3 phases of the jewelry shop:
+
+    1. 📈 **Market** — Buy gold at the right price
+    2. 🏭 **Warehouse** — Craft the most demanded product
+    3. 🤝 **Showroom** — Negotiate the best sale price
+
+    Press **🚀 New Episode** to watch the agent play!
+    """)
+
+    if st.session_state.final_reward is not None:
+        st.markdown("---")
+        reward = st.session_state.final_reward
+        color = "#10b981" if reward >= 0.6 else "#f59e0b" if reward >= 0.4 else "#ef4444"
+        st.markdown(f"**Final Score:** :{'green' if reward >= 0.6 else 'orange' if reward >= 0.4 else 'red'}[{reward:.4f}]")
+        st.metric("Episodes Played", st.session_state.episode_count)
+
+
+# ── Main area ───────────────────────────────────────────────────────────────
+st.markdown('<p class="hero-title">💎 Jewelry Shop Manager</p>', unsafe_allow_html=True)
+st.markdown('<p class="hero-sub">An RL environment for training LLMs on multi-step business decisions</p>', unsafe_allow_html=True)
+
+
+# ── No episode yet — show welcome ──────────────────────────────────────────
+if st.session_state.episode_steps is None:
+    st.info("👋 **Welcome!** Press **🚀 New Episode** in the sidebar to watch the AI agent play through the jewelry shop simulation.")
+
+    st.markdown("### 📦 Product Catalog")
+    cols = st.columns(3)
+    items = [("💍 Ring", "ring"), ("📿 Necklace", "necklace"), ("⌚ Bracelet", "bracelet")]
+    for i, (icon_name, key) in enumerate(items):
+        spec = PRODUCT_CATALOG[key]
+        with cols[i]:
+            st.markdown(f"""
+            <div class="catalog-card">
+                <h4>{icon_name}</h4>
+                <p>🪙 Gold: {spec['gold_oz']} oz</p>
+                <p>🔧 Labor: ${spec['labor']:.0f}</p>
+                <p>📊 Base demand: {spec['base_demand']:.0%}</p>
+            </div>
+            """, unsafe_allow_html=True)
+
+    st.markdown("### 🧠 Three Business Phases")
+    c1, c2, c3 = st.columns(3)
+    with c1:
+        st.markdown("""
+        <div class="phase-box">
+            <h4>📈 Phase 1: Market</h4>
+            <p>Buy raw gold at the best price. Prices fluctuate — time your purchase wisely!</p>
+        </div>
+        """, unsafe_allow_html=True)
+    with c2:
+        st.markdown("""
+        <div class="phase-box">
+            <h4>🏭 Phase 2: Warehouse</h4>
+            <p>Craft a product (ring, necklace, bracelet) that matches market demand.</p>
+        </div>
+        """, unsafe_allow_html=True)
+    with c3:
+        st.markdown("""
+        <div class="phase-box">
+            <h4>🤝 Phase 3: Showroom</h4>
+            <p>Negotiate with a customer over 5 rounds to maximize your selling price.</p>
+        </div>
+        """, unsafe_allow_html=True)
+
+
+# ── Episode results ─────────────────────────────────────────────────────────
+else:
+    steps = st.session_state.episode_steps
+    reward = st.session_state.final_reward
+
+    # ── Score banner ────────────────────────────────────────────────────────
+    if reward >= 0.8:
+        grade, grade_emoji = "Excellent", "🏆"
+    elif reward >= 0.6:
+        grade, grade_emoji = "Good", "👍"
+    elif reward >= 0.4:
+        grade, grade_emoji = "Fair", "😐"
+    else:
+        grade, grade_emoji = "Poor", "😬"
+
+    st.markdown(f"""
+    <div class="reward-big">
+        <div class="label">{grade_emoji} Episode #{st.session_state.episode_count} — {grade}</div>
+        <div class="score">{reward:.4f}</div>
+        <div class="label">cumulative reward (out of 1.0)</div>
+    </div>
+    """, unsafe_allow_html=True)
+
+    st.markdown("")
+
+    # ── Summary metrics ─────────────────────────────────────────────────────
+    last = steps[-1]
+    m1, m2, m3, m4 = st.columns(4)
+    with m1:
+        st.markdown(f"""<div class="metric-card"><div class="label">Steps</div><div class="value">{len(steps)-1}</div></div>""", unsafe_allow_html=True)
+    with m2:
+        st.markdown(f"""<div class="metric-card"><div class="label">Final Cash</div><div class="value">${last['cash']:,.0f}</div></div>""", unsafe_allow_html=True)
+    with m3:
+        gold_price = steps[0].get("gold_price", 0)
+        st.markdown(f"""<div class="metric-card"><div class="label">Gold Price</div><div class="value">${gold_price:,.0f}/oz</div></div>""", unsafe_allow_html=True)
+    with m4:
+        product = None
+        for s in steps:
+            if s.get("product"):
+                product = s["product"]
+        st.markdown(f"""<div class="metric-card"><div class="label">Product</div><div class="value">{(product or 'N/A').title()}</div></div>""", unsafe_allow_html=True)
+
+    st.markdown("---")
+
+    # ── Step-by-step log ────────────────────────────────────────────────────
+    st.markdown("### 📋 Agent Decision Log")
+
+    for s in steps:
+        phase = s.get("phase", "market")
+        icon = {"market": "📈", "warehouse": "🏭", "showroom": "🤝"}.get(phase, "⬜")
+        step_num = s["step"]
+        action = s.get("action", "")
+        rw = s.get("reward", 0)
+        cum = s.get("cumulative", 0)
+
+        rw_badge = f" · reward: `{rw:.4f}`" if rw else ""
+        cum_str = f" · cumulative: `{cum:.4f}`" if cum else ""
+
+        # Use pure markdown — no raw HTML divs
+        st.markdown(f"**Step {step_num}** {icon} **{action}**{rw_badge}{cum_str}")
+
+        # Show environment message
+        if s.get("msg"):
+            st.caption(s["msg"])
+
+        st.divider()
+
+    # ── Reward progression chart ────────────────────────────────────────────
+    st.markdown("---")
+    st.markdown("### 📊 Cumulative Reward Over Steps")
+    chart_data = [s["cumulative"] for s in steps]
+    st.line_chart(chart_data, use_container_width=True, height=200)
+
+    st.info("Press **🚀 New Episode** in the sidebar to run another episode with a different seed!")