Spaces:
Sleeping
Sleeping
Remove 'Why not real Siri?' tab and its cross-references
Browse files
app.py
CHANGED
|
@@ -1180,8 +1180,8 @@ HERO = """
|
|
| 1180 |
|
| 1181 |
<div class="note">The model runs only on a Mac + iOS Simulator — this Space replays the recorded
|
| 1182 |
artifacts (video + trajectory) those runs produced. <b style="color:#fff">Start with “Explore
|
| 1183 |
-
tasks”</b>, see how the suite is organized in “Task classification”, read how tasks are built
|
| 1184 |
-
|
| 1185 |
</div>
|
| 1186 |
"""
|
| 1187 |
|
|
@@ -1636,58 +1636,6 @@ input, a newer OS, or private system interfaces. They unblock as the environment
|
|
| 1636 |
{_blocked_table()}
|
| 1637 |
"""
|
| 1638 |
|
| 1639 |
-
JOURNEY_HTML = """
|
| 1640 |
-
<div class="section-h">🧭 Wait — why not just use the real Siri?</div>
|
| 1641 |
-
<p class="lead">Because Siri-the-assistant <b>cannot execute tasks in a simulator</b> — that's an Apple
|
| 1642 |
-
platform constraint, not a choice. Here is what does and doesn't run, and why driving the on-device
|
| 1643 |
-
model directly still measures the same mechanism Siri uses.</p>
|
| 1644 |
-
|
| 1645 |
-
<div class="fam-grid">
|
| 1646 |
-
<div class="fam-card">
|
| 1647 |
-
<div class="fam-name">🚫 1 · Siri won't execute in any simulator</div>
|
| 1648 |
-
<div class="fam-def">On the iOS 26 simulator, Siri's voice assets are never delivered, so the
|
| 1649 |
-
assistant reports its content unavailable and can't start. On the iOS 27 simulator, Siri activates
|
| 1650 |
-
but executes <b>nothing</b> — even "Open Calendar" fails with <i>"Something went wrong. Please try
|
| 1651 |
-
again,"</i> because Apple Intelligence is never marked eligible inside a simulator. Agentic Siri is
|
| 1652 |
-
a <b>physical-device-only</b> capability.</div>
|
| 1653 |
-
</div>
|
| 1654 |
-
<div class="fam-card">
|
| 1655 |
-
<div class="fam-name">⛔ 2 · The newest model is simulator-blocked too</div>
|
| 1656 |
-
<div class="fam-def">Querying the on-device Foundation Model directly <b>works on the iOS 26.4
|
| 1657 |
-
simulator</b> — but on the iOS 27 beta, the model loads and even reports itself "available," yet
|
| 1658 |
-
every request is refused by a mandatory safety layer with <i>"Simulator is not supported."</i>
|
| 1659 |
-
(Which is why generation is always tested with a real call, never trusted from the availability
|
| 1660 |
-
flag.)</div>
|
| 1661 |
-
</div>
|
| 1662 |
-
<div class="fam-card">
|
| 1663 |
-
<div class="fam-name">🧠 3 · So the benchmark hosts the model directly</div>
|
| 1664 |
-
<div class="fam-def">An assistant app loads Apple's on-device Foundation Model — the same model
|
| 1665 |
-
behind Apple Intelligence — hands it <b>11 tools</b> plus an executor that drives the real
|
| 1666 |
-
Reminders / Calendar / Contacts / Messages, records every step, and scores the run against the
|
| 1667 |
-
real device state.</div>
|
| 1668 |
-
</div>
|
| 1669 |
-
<div class="fam-card">
|
| 1670 |
-
<div class="fam-name">🤝 4 · Why this measures the real mechanism</div>
|
| 1671 |
-
<div class="fam-def">Real Siri does not tap the screen: it acts on apps by <b>calling declared app
|
| 1672 |
-
actions</b> (App Intents) — pick a named action, fill its typed parameters. That is exactly the
|
| 1673 |
-
tool-calling this benchmark exercises. Only the speech front-end — which the simulator cannot
|
| 1674 |
-
provide — is absent.</div>
|
| 1675 |
-
</div>
|
| 1676 |
-
</div>
|
| 1677 |
-
|
| 1678 |
-
<div class="section-h">📡 What actually works where</div>
|
| 1679 |
-
<table>
|
| 1680 |
-
<thead><tr><th>Capability</th><th>iOS 26.4 simulator</th><th>iOS 27 simulator</th><th>Physical device</th></tr></thead>
|
| 1681 |
-
<tbody>
|
| 1682 |
-
<tr><td>Siri-the-assistant executes a task</td><td class="cap-no">✗ skipped — no voice assets</td><td class="cap-no">✗ activates, executes nothing</td><td class="cap-yes">✓ the only place it works</td></tr>
|
| 1683 |
-
<tr><td>Foundation Model generates directly</td><td class="cap-yes">✓ works — SiriBench runs here</td><td class="cap-no">✗ "Simulator is not supported"</td><td class="cap-yes">✓</td></tr>
|
| 1684 |
-
<tr><td>Image / visual tasks</td><td class="cap-no">✗ needs the iOS 27 model</td><td class="cap-no">✗ model blocked in-sim</td><td>— future work</td></tr>
|
| 1685 |
-
</tbody>
|
| 1686 |
-
</table>
|
| 1687 |
-
<p class="sub-lead"><b>The takeaway:</b> SiriBench tests the real Siri action mechanism — tool-calling
|
| 1688 |
-
into declared app actions, on the same on-device model — just without the microphone.</p>
|
| 1689 |
-
"""
|
| 1690 |
-
|
| 1691 |
SCORING_HTML = """
|
| 1692 |
<div class="section-h">⚖️ How a run is scored — and why there is no AI judge</div>
|
| 1693 |
<p class="lead">Every verdict is a set of fixed, mechanical checks — never another model's opinion. A task
|
|
@@ -1871,16 +1819,12 @@ with gr.Blocks(title="SiriBench — iOS agent tasks & trajectories") as demo:
|
|
| 1871 |
"### Scope — why some tasks aren't here yet\n"
|
| 1872 |
"These run on the **iOS 26.4** text-only on-device model. **Visual tasks** (image QA, "
|
| 1873 |
"receipt parsing, image editing) need **image input to the model — an iOS 27 / macOS 27 "
|
| 1874 |
-
"capability**. The full list of blocked families is in the **Task classification** tab
|
| 1875 |
-
"the story of *why* the newer model can't run here is in **Why not real Siri?**."
|
| 1876 |
)
|
| 1877 |
|
| 1878 |
with gr.Tab("⚖️ How runs are scored"):
|
| 1879 |
gr.HTML(SCORING_HTML)
|
| 1880 |
|
| 1881 |
-
with gr.Tab("🧭 Why not real Siri?"):
|
| 1882 |
-
gr.HTML(JOURNEY_HTML)
|
| 1883 |
-
|
| 1884 |
gr.HTML(FOOTER_HTML)
|
| 1885 |
|
| 1886 |
if __name__ == "__main__":
|
|
|
|
| 1180 |
|
| 1181 |
<div class="note">The model runs only on a Mac + iOS Simulator — this Space replays the recorded
|
| 1182 |
artifacts (video + trajectory) those runs produced. <b style="color:#fff">Start with “Explore
|
| 1183 |
+
tasks”</b>, see how the suite is organized in “Task classification”, then read how tasks are built
|
| 1184 |
+
and scored.</div>
|
| 1185 |
</div>
|
| 1186 |
"""
|
| 1187 |
|
|
|
|
| 1636 |
{_blocked_table()}
|
| 1637 |
"""
|
| 1638 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1639 |
SCORING_HTML = """
|
| 1640 |
<div class="section-h">⚖️ How a run is scored — and why there is no AI judge</div>
|
| 1641 |
<p class="lead">Every verdict is a set of fixed, mechanical checks — never another model's opinion. A task
|
|
|
|
| 1819 |
"### Scope — why some tasks aren't here yet\n"
|
| 1820 |
"These run on the **iOS 26.4** text-only on-device model. **Visual tasks** (image QA, "
|
| 1821 |
"receipt parsing, image editing) need **image input to the model — an iOS 27 / macOS 27 "
|
| 1822 |
+
"capability**. The full list of blocked families is in the **Task classification** tab."
|
|
|
|
| 1823 |
)
|
| 1824 |
|
| 1825 |
with gr.Tab("⚖️ How runs are scored"):
|
| 1826 |
gr.HTML(SCORING_HTML)
|
| 1827 |
|
|
|
|
|
|
|
|
|
|
| 1828 |
gr.HTML(FOOTER_HTML)
|
| 1829 |
|
| 1830 |
if __name__ == "__main__":
|