Whole-space polish: hero chips + tab guide, new 'How runs are scored' and 'Why not real Siri?' tabs, emoji tab labels, footer

app.py

@@ -1124,6 +1124,28 @@ code {
 }
 .arg-chip b { color: #1e1b4b; font-weight: 700; }
 .meta-chips { display: flex; gap: 7px; flex-wrap: wrap; margin: 2px 0 16px; }
+
+/* Hero chips, footer, capability marks */
+.hero-chips { display: flex; gap: 8px; flex-wrap: wrap; margin: 14px 0 18px; }
+.hero-chip {
+  font-size: 12px;
+  font-weight: 650;
+  color: #e0e7ff;
+  background: rgba(255,255,255,0.08);
+  border: 1px solid rgba(199,210,254,0.35);
+  padding: 4px 12px;
+  border-radius: 999px;
+}
+.foot {
+  text-align: center;
+  font-size: 12.5px;
+  color: #64748b;
+  padding: 26px 10px 8px;
+  line-height: 1.6;
+}
+.foot b { color: #1e1b4b; }
+.cap-yes { color: #047857; font-weight: 800; }
+.cap-no  { color: #b91c1c; font-weight: 800; }
 """
 
 THEME = gr.themes.Soft(primary_hue="violet", secondary_hue="indigo", neutral_hue="slate",
@@ -1136,7 +1158,15 @@ HERO = """
   powers <b>Siri</b>. We created a set of real-world tasks across the actual iOS apps
   (Reminders, Calendar, Contacts & Messages), let the model complete them on its own, and check
   whether it actually got them right.</p>
-  
+
+  <div class="hero-chips">
+    <span class="hero-chip">🧠 ~3B on-device model</span>
+    <span class="hero-chip">📱 real system apps — not mocks</span>
+    <span class="hero-chip">🛠️ 11 tools, one neutral agent</span>
+    <span class="hero-chip">⚖️ no AI judge — real database re-reads</span>
+    <span class="hero-chip">🎞️ every run recorded</span>
+  </div>
+
   <div class="stats-grid">
     <div class="stat-card">
       <div class="stat-num">14</div>
@@ -1152,12 +1182,14 @@ HERO = """
     </div>
     <div class="stat-card">
       <div class="stat-num text-accent">71.4%</div>
-      <div class="stat-label">Accuracy Rate</div>
+      <div class="stat-label">Documented Run</div>
     </div>
   </div>
-  
+
   <div class="note">The model runs only on a Mac + iOS Simulator — this Space replays the recorded
-  artifacts (video + trajectory) those runs produced.</div>
+  artifacts (video + trajectory) those runs produced. <b style="color:#fff">Start with “Explore
+  tasks”</b>, see how the suite is organized in “Task classification”, read how tasks are built and
+  scored — and why this isn't just “ask Siri” in “Why not real Siri?”.</div>
 </div>
 """
 
@@ -1612,10 +1644,133 @@ input, a newer OS, or private system interfaces. They unblock as the environment
 {_blocked_table()}
 """
 
+JOURNEY_HTML = """
+<div class="section-h">🧭 Wait — why not just use the real Siri?</div>
+<p class="lead">The honest answer: we tried, and Siri-the-assistant simply <b>cannot execute tasks in a
+simulator</b>. That dead end shaped the whole project. Here is the story, and why the pivot still
+measures the real thing.</p>
+
+<div class="fam-grid">
+  <div class="fam-card">
+    <div class="fam-name">🎯 1 · The original goal</div>
+    <div class="fam-def">Benchmark <b>Siri-the-assistant</b> end to end: trigger Siri by code, have it run
+    a real app action, then verify the real result on the device. The canonical trial: a money transfer
+    that should move a balance from $500 to $700.</div>
+  </div>
+  <div class="fam-card">
+    <div class="fam-name">🚫 2 · Wall #1 — Siri will not execute in any simulator</div>
+    <div class="fam-def">On the iOS 26 simulator, Siri's voice assets are never delivered, so the
+    assistant is skipped as "content unavailable." On the iOS 27 simulator, Siri activates but executes
+    <b>nothing</b> — even "Open Calendar" fails with <i>"Something went wrong. Please try again."</i>
+    because Apple Intelligence is never marked eligible inside a simulator. Real agentic Siri is a
+    <b>physical-device-only</b> capability.</div>
+  </div>
+  <div class="fam-card">
+    <div class="fam-name">⛔ 3 · Wall #2 — the newest model is blocked in-simulator too</div>
+    <div class="fam-def">Asking the on-device Foundation Model questions directly <b>works on the
+    iOS 26.4 simulator</b> — but on the iOS 27 beta the model loads, even reports itself "available,"
+    then every request dies in a mandatory safety layer with <i>"Simulator is not supported."</i>
+    Golden rule learned: never trust the availability flag, trust an actual generation.</div>
+  </div>
+  <div class="fam-card">
+    <div class="fam-name">🔎 4 · A correction along the way</div>
+    <div class="fam-def">An early theory said "the model weights never downloaded" — wrong. The base
+    model ships pre-installed inside a protected area of macOS, and the simulator hands inference to the
+    <b>Mac's own</b> Apple Intelligence. The block is simulator-specific and deliberate, not a missing
+    download. (Also learned: a non-US-English host silently reports "model asset unavailable.")</div>
+  </div>
+  <div class="fam-card">
+    <div class="fam-name">✅ 5 · The pivot — host the model ourselves</div>
+    <div class="fam-def">An Assistant app loads Apple's on-device Foundation Model directly, hands it
+    <b>11 tools</b> plus an executor that drives the real Reminders / Calendar / Contacts / Messages,
+    records every step, and scores the run against the real device state. Same model — we just replace
+    Siri's shell.</div>
+  </div>
+  <div class="fam-card">
+    <div class="fam-name">🤝 6 · Why this still measures the real thing</div>
+    <div class="fam-def">Real Siri does not tap the screen: it acts on apps by <b>calling declared app
+    actions</b> (App Intents) — pick a named action, fill its typed parameters. That is exactly the
+    tool-calling SiriBench exercises. We removed only the speech front-end the simulator cannot
+    provide.</div>
+  </div>
+</div>
+
+<div class="section-h">📡 What actually works where</div>
+<table>
+  <thead><tr><th>Capability</th><th>iOS 26.4 simulator</th><th>iOS 27 simulator</th><th>Physical device</th></tr></thead>
+  <tbody>
+    <tr><td>Siri-the-assistant executes a task</td><td class="cap-no">✗ skipped — no voice assets</td><td class="cap-no">✗ activates, executes nothing</td><td class="cap-yes">✓ the only place it works</td></tr>
+    <tr><td>Foundation Model generates directly</td><td class="cap-yes">✓ works — SiriBench runs here</td><td class="cap-no">✗ "Simulator is not supported"</td><td class="cap-yes">✓</td></tr>
+    <tr><td>Image / visual tasks</td><td class="cap-no">✗ needs the iOS 27 model</td><td class="cap-no">✗ model blocked in-sim</td><td>— future work</td></tr>
+  </tbody>
+</table>
+<p class="sub-lead"><b>The takeaway:</b> SiriBench tests the real Siri action mechanism — tool-calling
+into declared app actions, on the same on-device model — just without the microphone.</p>
+"""
+
+SCORING_HTML = """
+<div class="section-h">⚖️ How a run is scored — and why there is no AI judge</div>
+<p class="lead">Every verdict is a set of fixed, mechanical checks — never another model's opinion. A task
+passes only when <b>both</b> layers below hold, so the score reflects what the model actually <i>did</i>
+and what actually <i>happened on the device</i>, not how convincing its wording sounds.</p>
+
+<div class="judge-cards">
+  <div class="judge-card"><h4>1️⃣ Process — which tools were called</h4>
+    <p>We read the recorded run for evidence that the right tools fired — and that forbidden ones did
+    <i>not</i>. Example: the dentist lookup <b>must</b> read the calendar and <b>must not</b> create a
+    reminder or an event along the way.</p></div>
+  <div class="judge-card"><h4>2️⃣ Outcome — the real end-state</h4>
+    <p>After the run we re-read the actual Reminders / Calendar / Contacts store on the device, and/or
+    require a specific fact in the model's own final answer ("Friday", "Canberra", the booking code
+    "58291", "36"). The model cannot fake this — it is read from the OS itself.</p></div>
+</div>
+
+<div class="section-h">✅ "Should-do" vs 🚫 "should-not" tasks</div>
+<div class="judge-cards">
+  <div class="judge-card"><h4>✅ Should-do: pass = does it right</h4>
+    <p>Most tasks require the right actions <i>and</i> the right end-state: chain the calendar into a
+    reminder, find the booking code, draft the email with the new deadline. Doing nothing fails.</p></div>
+  <div class="judge-card"><h4>🚫 Should-not: pass = holds back</h4>
+    <p>Four families invert the rubric: passing means the model <i>refrained</i> — asked which Alex
+    instead of texting one, refused to wipe every reminder without confirmation, ignored an instruction
+    hidden in a web page, and answered 15% of 240 locally instead of searching the web. This measures
+    judgment, not capability.</p></div>
+</div>
+
+<div class="section-h">🤖 Why not just use an AI judge?</div>
+<table>
+  <thead><tr><th>Question</th><th>AI-judge benchmark</th><th>SiriBench</th></tr></thead>
+  <tbody>
+    <tr><td>Where does the verdict come from?</td><td>Another model's opinion of the transcript</td><td class="cap-yes">Mechanical checks + a re-read of the real device</td></tr>
+    <tr><td>Same input, same score?</td><td>Varies with the judge, its prompt, its settings</td><td class="cap-yes">Perfectly repeatable given a transcript</td></tr>
+    <tr><td>Can slick wording fake success?</td><td>Sometimes — persuasion can win</td><td class="cap-yes">No — the right action must actually happen</td></tr>
+    <tr><td>What kind of reason do you get?</td><td>Diffuse ("this seems wrong")</td><td class="cap-yes">Exact — which tool, which fact, which count</td></tr>
+  </tbody>
+</table>
+<p class="sub-lead">This is the same real-state-verification philosophy as AndroidWorld and WebArena:
+success is defined by the observable end-state of the real environment, not by a grader's impression.
+One more gate: if a run hits any model or system error, the task fails outright — an error is never
+mistaken for the model "wisely abstaining."</p>
+
+<div class="section-h">🔁 One honest caveat — results are a snapshot</div>
+<p class="lead">Decoding is pinned to the most repeatable setting, but the <b>inputs</b> are not frozen:
+tasks use the real current date, the live device state, and live Wikipedia. A documented 10/14 run came
+out roughly 8/14 on a repeat, with the dentist lookup flipping. Treat the numbers as a snapshot of
+small-model behavior, not a fixed leaderboard.</p>
+"""
+
+FOOTER_HTML = """
+<div class="foot">
+  <b>SiriBench</b> — benchmarking Apple's on-device Foundation Model as a tool-calling iOS agent over the
+  real system apps.<br/>This Space replays recorded runs; the model itself runs only on a Mac + iOS
+  Simulator. Documented run: 10 / 14 passed (~3B on-device model, iOS 26.4) — results vary run to run.
+</div>
+"""
+
 with gr.Blocks(title="SiriBench — iOS agent tasks & trajectories") as demo:
     gr.HTML(HERO)
 
-    with gr.Tab("Explore tasks"):
+    with gr.Tab("🔍 Explore tasks"):
         with gr.Row(equal_height=False):
             # Left Column: Sidebar Task Selector (width: 3 / 30% ratio)
             with gr.Column(scale=3, min_width=320):
@@ -1722,10 +1877,10 @@ with gr.Blocks(title="SiriBench — iOS agent tasks & trajectories") as demo:
         picker.change(show, picker, outs)
         demo.load(show, picker, outs)
 
-    with gr.Tab("Task classification"):
+    with gr.Tab("🗂️ Task classification"):
         gr.HTML(CLASSIFY_HTML)
 
-    with gr.Tab("How tasks are created"):
+    with gr.Tab("🧪 How tasks are created"):
         gr.HTML(WHAT)
         gr.HTML(ANATOMY)
         gr.HTML(WORKED)
@@ -1742,9 +1897,18 @@ with gr.Blocks(title="SiriBench — iOS agent tasks & trajectories") as demo:
             "### Scope — why some tasks aren't here yet\n"
             "These run on the **iOS 26.4** text-only on-device model. **Visual tasks** (image QA, "
             "receipt parsing, image editing) need **image input to the model — an iOS 27 / macOS 27 "
-            "capability** — and unblock once the host Mac is on macOS 27."
+            "capability**. The full list of blocked families is in the **Task classification** tab, and "
+            "the story of *why* the newer model can't run here is in **Why not real Siri?**."
         )
 
+    with gr.Tab("⚖️ How runs are scored"):
+        gr.HTML(SCORING_HTML)
+
+    with gr.Tab("🧭 Why not real Siri?"):
+        gr.HTML(JOURNEY_HTML)
+
+    gr.HTML(FOOTER_HTML)
+
 if __name__ == "__main__":
     demo.launch(server_name="0.0.0.0", server_port=7860, theme=THEME, css=CSS,
                 allowed_paths=[os.path.abspath(TRAJ_DIR)])