naman-cen commited on
Commit
ee3dd09
·
verified ·
1 Parent(s): dfa1578

Remove 'Why not real Siri?' tab and its cross-references

Browse files
Files changed (1) hide show
  1. app.py +3 -59
app.py CHANGED
@@ -1180,8 +1180,8 @@ HERO = """
1180
 
1181
  <div class="note">The model runs only on a Mac + iOS Simulator — this Space replays the recorded
1182
  artifacts (video + trajectory) those runs produced. <b style="color:#fff">Start with “Explore
1183
- tasks”</b>, see how the suite is organized in “Task classification”, read how tasks are built and
1184
- scored — and why this isn't just “ask Siri” in “Why not real Siri?”.</div>
1185
  </div>
1186
  """
1187
 
@@ -1636,58 +1636,6 @@ input, a newer OS, or private system interfaces. They unblock as the environment
1636
  {_blocked_table()}
1637
  """
1638
 
1639
- JOURNEY_HTML = """
1640
- <div class="section-h">🧭 Wait — why not just use the real Siri?</div>
1641
- <p class="lead">Because Siri-the-assistant <b>cannot execute tasks in a simulator</b> — that's an Apple
1642
- platform constraint, not a choice. Here is what does and doesn't run, and why driving the on-device
1643
- model directly still measures the same mechanism Siri uses.</p>
1644
-
1645
- <div class="fam-grid">
1646
- <div class="fam-card">
1647
- <div class="fam-name">🚫 1 · Siri won't execute in any simulator</div>
1648
- <div class="fam-def">On the iOS 26 simulator, Siri's voice assets are never delivered, so the
1649
- assistant reports its content unavailable and can't start. On the iOS 27 simulator, Siri activates
1650
- but executes <b>nothing</b> — even "Open Calendar" fails with <i>"Something went wrong. Please try
1651
- again,"</i> because Apple Intelligence is never marked eligible inside a simulator. Agentic Siri is
1652
- a <b>physical-device-only</b> capability.</div>
1653
- </div>
1654
- <div class="fam-card">
1655
- <div class="fam-name">⛔ 2 · The newest model is simulator-blocked too</div>
1656
- <div class="fam-def">Querying the on-device Foundation Model directly <b>works on the iOS 26.4
1657
- simulator</b> — but on the iOS 27 beta, the model loads and even reports itself "available," yet
1658
- every request is refused by a mandatory safety layer with <i>"Simulator is not supported."</i>
1659
- (Which is why generation is always tested with a real call, never trusted from the availability
1660
- flag.)</div>
1661
- </div>
1662
- <div class="fam-card">
1663
- <div class="fam-name">🧠 3 · So the benchmark hosts the model directly</div>
1664
- <div class="fam-def">An assistant app loads Apple's on-device Foundation Model — the same model
1665
- behind Apple Intelligence — hands it <b>11 tools</b> plus an executor that drives the real
1666
- Reminders / Calendar / Contacts / Messages, records every step, and scores the run against the
1667
- real device state.</div>
1668
- </div>
1669
- <div class="fam-card">
1670
- <div class="fam-name">🤝 4 · Why this measures the real mechanism</div>
1671
- <div class="fam-def">Real Siri does not tap the screen: it acts on apps by <b>calling declared app
1672
- actions</b> (App Intents) — pick a named action, fill its typed parameters. That is exactly the
1673
- tool-calling this benchmark exercises. Only the speech front-end — which the simulator cannot
1674
- provide — is absent.</div>
1675
- </div>
1676
- </div>
1677
-
1678
- <div class="section-h">📡 What actually works where</div>
1679
- <table>
1680
- <thead><tr><th>Capability</th><th>iOS 26.4 simulator</th><th>iOS 27 simulator</th><th>Physical device</th></tr></thead>
1681
- <tbody>
1682
- <tr><td>Siri-the-assistant executes a task</td><td class="cap-no">✗ skipped — no voice assets</td><td class="cap-no">✗ activates, executes nothing</td><td class="cap-yes">✓ the only place it works</td></tr>
1683
- <tr><td>Foundation Model generates directly</td><td class="cap-yes">✓ works — SiriBench runs here</td><td class="cap-no">✗ "Simulator is not supported"</td><td class="cap-yes">✓</td></tr>
1684
- <tr><td>Image / visual tasks</td><td class="cap-no">✗ needs the iOS 27 model</td><td class="cap-no">✗ model blocked in-sim</td><td>— future work</td></tr>
1685
- </tbody>
1686
- </table>
1687
- <p class="sub-lead"><b>The takeaway:</b> SiriBench tests the real Siri action mechanism — tool-calling
1688
- into declared app actions, on the same on-device model — just without the microphone.</p>
1689
- """
1690
-
1691
  SCORING_HTML = """
1692
  <div class="section-h">⚖️ How a run is scored — and why there is no AI judge</div>
1693
  <p class="lead">Every verdict is a set of fixed, mechanical checks — never another model's opinion. A task
@@ -1871,16 +1819,12 @@ with gr.Blocks(title="SiriBench — iOS agent tasks & trajectories") as demo:
1871
  "### Scope — why some tasks aren't here yet\n"
1872
  "These run on the **iOS 26.4** text-only on-device model. **Visual tasks** (image QA, "
1873
  "receipt parsing, image editing) need **image input to the model — an iOS 27 / macOS 27 "
1874
- "capability**. The full list of blocked families is in the **Task classification** tab, and "
1875
- "the story of *why* the newer model can't run here is in **Why not real Siri?**."
1876
  )
1877
 
1878
  with gr.Tab("⚖️ How runs are scored"):
1879
  gr.HTML(SCORING_HTML)
1880
 
1881
- with gr.Tab("🧭 Why not real Siri?"):
1882
- gr.HTML(JOURNEY_HTML)
1883
-
1884
  gr.HTML(FOOTER_HTML)
1885
 
1886
  if __name__ == "__main__":
 
1180
 
1181
  <div class="note">The model runs only on a Mac + iOS Simulator — this Space replays the recorded
1182
  artifacts (video + trajectory) those runs produced. <b style="color:#fff">Start with “Explore
1183
+ tasks”</b>, see how the suite is organized in “Task classification”, then read how tasks are built
1184
+ and scored.</div>
1185
  </div>
1186
  """
1187
 
 
1636
  {_blocked_table()}
1637
  """
1638
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1639
  SCORING_HTML = """
1640
  <div class="section-h">⚖️ How a run is scored — and why there is no AI judge</div>
1641
  <p class="lead">Every verdict is a set of fixed, mechanical checks — never another model's opinion. A task
 
1819
  "### Scope — why some tasks aren't here yet\n"
1820
  "These run on the **iOS 26.4** text-only on-device model. **Visual tasks** (image QA, "
1821
  "receipt parsing, image editing) need **image input to the model — an iOS 27 / macOS 27 "
1822
+ "capability**. The full list of blocked families is in the **Task classification** tab."
 
1823
  )
1824
 
1825
  with gr.Tab("⚖️ How runs are scored"):
1826
  gr.HTML(SCORING_HTML)
1827
 
 
 
 
1828
  gr.HTML(FOOTER_HTML)
1829
 
1830
  if __name__ == "__main__":