diff --git "a/rollout.jsonl" "b/rollout.jsonl" deleted file mode 100644--- "a/rollout.jsonl" +++ /dev/null @@ -1,119 +0,0 @@ -Reading additional input from stdin... -{"type":"thread.started","thread_id":"019f125a-6d51-76d1-af24-6642c27f6f23"} -{"type":"turn.started"} -{"type":"item.completed","item":{"id":"item_0","type":"agent_message","text":"I’ll fetch the requested HF Spaces playbook first, then I’ll read the relevant local skills for HF CLI and Gradio before touching the Space implementation."}} -{"type":"item.started","item":{"id":"item_1","type":"command_execution","command":"/bin/zsh -lc 'curl -L --fail --silent https://gist.githubusercontent.com/gary149/37c955b832558837c40e1c14ff6d955d/raw/ad35807f8466378afd04d7653d53683a847b96c4/hf-spaces-agent-quickstart-compact.md'","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_1","type":"command_execution","command":"/bin/zsh -lc 'curl -L --fail --silent https://gist.githubusercontent.com/gary149/37c955b832558837c40e1c14ff6d955d/raw/ad35807f8466378afd04d7653d53683a847b96c4/hf-spaces-agent-quickstart-compact.md'","aggregated_output":"# HF Spaces Agent Quickstart (Compact)\n\n---\n\n## DO / NEVER\n\n**DO:** Iterate ON the Space. Push early, verify against live URL. Stream logs. Read logs first, act once, make one targeted fix. Use cheapest iteration rung. Set all three cache env vars at module top, before any import. Measure `@spaces.GPU(duration=)` empirically. Call `c.view_api()` before any `predict()` call. `python3 -m py_compile app.py` is the maximum local check before pushing.\n\n**NEVER:** Sleep-poll. Build mock modes, `SKIP_MODEL_LOAD` env vars, Playwright harnesses, or local Gradio servers. Restart before reading the error. Issue concurrent uploads. Restart while uploading. Stack restarts while `runtime.sha` is still flipping. Use integer device IDs (`.to(0)`, `device_map={\"\": 0}`, `set_device(0)`). Pin `torch` or `torchaudio`. Proxy another community ZeroGPU Space via `gradio_client`. Wrap a `gradio_client` proxy call in `@spaces.GPU`. Stack `@spaces.GPU` and `@app.api` on the same function. Edit files in dev-mode SSH expecting them to survive restarts.\n\n---\n\n## §1 Fast Path\n\n```bash\nhf spaces search \"\" --sdk gradio --limit 10 # search reference Space first\n\nhf repos create / --type space \\\n --space-sdk [--flavor zero-a10g] --exist-ok\n\nhf upload / . . --type space \\\n --exclude '.git/*' --exclude '__pycache__/*' --exclude '.venv/*' --exclude '*.pyc' \\\n --commit-message 'init'\n\nhf spaces logs / --build --follow\nhf spaces logs / --follow\nhf spaces logs --build --tail 200 # BUILD_ERROR: find the FIRST error\nhf spaces logs --tail 200 # RUNTIME_ERROR or stuck APP_STARTING\n```\n\nREADME frontmatter (required fields):\n```yaml\n---\ntitle: ...\nemoji: 🚀\ncolorFrom: blue\ncolorTo: indigo\nsdk: gradio\nsdk_version: 6.10.0 # omit for docker/static\napp_file: app.py # gradio/streamlit only\nshort_description: ... # <= 60 chars\nstartup_duration_timeout: 1h # default 30m; bump for big LLMs\n---\n```\n\n`hardware:` in frontmatter is silently ignored. Set hardware via `--flavor` at create time OR `hf spaces settings --hardware zero-a10g`. A silently wrong hardware shows up later as `RuntimeError: Found no NVIDIA driver`.\n\n---\n\n## §2 Iteration Ladder\n\n| Rung | When | Command |\n|---|---|---|\n| 1. Hot-reload | Pure Python edit, Gradio >= 6.1, no new deps | `hf spaces hot-reload -f app.py` |\n| 2. Dev SSH | Diagnostics only (edits don't persist) | `ssh -i ~/.ssh/id_rsa @ssh.hf.space ''` |\n| 3. Targeted upload | Code-only, non-Gradio file, `gr.Server` | `hf upload / --include '' && hf spaces logs --follow` |\n| 4. Full rebuild | `requirements.txt`, Dockerfile, frontmatter, hardware | `hf spaces logs --build --follow` |\n| 5. Factory reboot | Container in inconsistent state, last resort | `hf spaces restart --factory-reboot` |\n\nHot-reload poisoning: `--factory-reboot` after a hot-reload-only commit fails with `could not read Username for https://huggingface.co`. Push any normal `hf upload` commit first, then restart. After upload, `runtime.sha` lags; do NOT restart again until it flips.\n\nStates: `BUILDING -> APP_STARTING -> RUNNING`.\n\n---\n\n## §3 Dev Mode (SSH)\n\n```bash\nhf spaces dev-mode / # triggers rebuild; PRO/Team/Enterprise only\nhf spaces info / --format json \\\n | python3 -c \"import json,sys; print(json.load(sys.stdin)['runtime']['raw']['domains'][0]['domain'])\"\ngit add . && git commit && git push # from inside SSH to persist edits\n```\n\nTreat all SSH edits as throwaway. Use dev mode for: reading logs, `pip list`, ad-hoc imports, `curl localhost:7860`. Not as an editor.\n\n---\n\n## §4 ZeroGPU Specifics\n\nZeroGPU is Gradio-only. Flavor: `zero-a10g`. Actual GPU: NVIDIA RTX PRO 6000 Blackwell.\n\n**Stub probe (required for zero-a10g Spaces):**\n```python\n@spaces.GPU(duration=1)\ndef _zerogpu_probe(): return \"ready\"\n```\nWithout at least one `@spaces.GPU` function, the Space `RUNTIME_ERROR`s immediately.\n\n**Cache env vars + import order (set at top of app.py, before any import):**\n```python\nimport os\nos.environ.setdefault(\"HF_HOME\", \"/data/.cache/huggingface\") # or /tmp on non-bucket spaces\nos.environ.setdefault(\"HF_MODULES_CACHE\", \"/tmp/hf_modules\")\nos.environ.setdefault(\"MPLCONFIGDIR\", \"/tmp/matplotlib\")\nimport spaces # before torch; avoids libcudart.so.X errors\n```\n\n**NeMo/RNNT exception (import order matters):**\n```python\nos.environ.setdefault(\"NUMBA_DISABLE_CUDA\", \"1\")\nimport spaces # immediately after NUMBA_DISABLE_CUDA, before any nemo/torch import\n```\nLoad model on CPU at startup; inside `@spaces.GPU` do `model.to(\"cuda\")` then infer then `model.to(\"cpu\")`.\n\n**Model loading:** Load at module level including `.to(\"cuda\")`. Only compute goes inside `@spaces.GPU`.\n```python\nmodel = Model.from_pretrained(..., torch_dtype=torch.bfloat16).to(\"cuda\")\n# NOT device_map=\"cuda\" for plain from_pretrained - crashes with \"Found no NVIDIA driver\"\n# device_map=\"cuda\" is OK only with bitsandbytes quantization_config (Pattern B)\n```\nException: frameworks that touch CUDA at import (pynvml, NCCL, vllm, NVFP4 quantizers) must defer construction into the first `@spaces.GPU` call with duration set high enough to cover load + inference.\n\n**Multiple checkpoints:**\n```python\nBUNDLES = {}\nfor family in (\"image\", \"video\"):\n BUNDLES[family] = load(family).to(\"cuda\")\n```\nNever unload/reload between requests.\n\n**Duration:** Never hardcode without measuring. Ship placeholder, instrument with `time.perf_counter()`, run 2-3 calls (include at least one cold start, which is 1.5-3x slower), set `duration = round(measured_max * 1.4)`. Too-high surfaces as `\"duration above maximum\"` at call time. Too-low silently truncates. Neither shows at deploy.\n\nCallable duration (use `*args, **kwargs` to absorb `gr.Progress` positional arg):\n```python\ndef _estimate(prompt, history, max_new_tokens, *args, **kwargs):\n return min(240, 60 + int(max_new_tokens / 64))\n@spaces.GPU(duration=_estimate)\ndef chat(prompt, history, max_new_tokens, ..., progress=gr.Progress(track_tqdm=True)): ...\n```\n\n**Two-ceiling:** `@spaces.GPU(duration=)` and `@app.api(time_limit=)` both apply; lower wins. Put them on SEPARATE functions:\n```python\n@spaces.GPU(duration=60)\ndef _run_gpu(prompt): return inference(prompt)\n\n@app.api(name=\"generate\", concurrency_limit=1, time_limit=180)\ndef generate(prompt: str) -> str: return _run_gpu(prompt)\n```\n\n**Size:** default `\"large\"` = 48 GB, 1x quota. Use `size=\"xlarge\"` only when exceeding 48 GB:\n```python\n@spaces.GPU(duration=120, size=\"xlarge\") # xlarge = 96 GB, 2x quota\ndef heavy_fn(): ...\n```\nbf16 costs `params_B * 2` GB (27B=54 GB overflows large). NF4 4-bit costs `params_B * ~0.55` GB (27B=~15 GB fits large, 70B=~40 GB fits large).\n\n**Flash-attn / xformers:** bitsandbytes works on ZeroGPU. Legacy flash-attn 1.x/2.x, triton, and xformers do not build cleanly; flash-attn 3 is officially recommended. Wrap unconditional top-level imports:\n```python\ntry:\n from flash_attn import flash_attn_func; HAS_FA = True\nexcept ImportError:\n flash_attn_func = None; HAS_FA = False\n```\n**xformers SDPA shim** (for models with hard `import xformers.ops`): install at boot before model imports.\n```python\nxformers.ops.memory_efficient_attention = _meff # full shim code in §4 item 5 of full gist\n```\n\n**Misc:** `torch.compile` NOT supported on ZeroGPU; use AoTI (torch >= 2.8). Gemma2 / transformers >= 4.50: `attn_implementation=\"eager\"`. Thinking models: `max_new_tokens >= 512`; suppress with `chat_template_kwargs={\"enable_thinking\": False}` (not `enable_thinking=` directly, raises `TypeError` in transformers >= 5). ONNX: use `onnxruntime-gpu`; rewrite custom ops to opset-20. TorchScript: `torch._C._set_graph_executor_optimize(False)`.\n\n---\n\n## §5 Per-SDK Gotchas\n\n**Gradio slow startup:** set `GRADIO_SSR_MODE=false` via env var (NOT `launch(ssr_mode=False)`, ignored on HF) AND `startup_duration_timeout: 1h` in README YAML. Both required.\n```bash\nhf spaces variables add --env GRADIO_SSR_MODE=false\n```\n\n**`gr.Examples`:** use `cache_examples=True, cache_mode=\"lazy\"`. `\"eager\"` runs every example at startup and burns ZeroGPU daily quota. Cache is keyed by file path, not content hash; bump a `cache_version` constant to wipe stale cache.\n\n**Streamlit:** port 8501 only. **Docker:** set `app_port:` in README YAML if not 7860 (`EXPOSE` not auto-read); build with `--platform=linux/amd64`. **Static:** set `app_build_command: npm run build` and `app_file: dist/index.html` in README YAML.\n\n---\n\n## §6 requirements.txt Rules\n\nPin everything except `torch`/`torchaudio` (base layer). Add `torchvision` unpinned if needed. Include `accelerate` whenever using `device_map=`. No local paths or editable installs. Non-pip-installable model code: include the directory in the upload; it lands in `/home/user/app/`. CUDA-extension build failures: vendor a same-named pure-PyTorch shim in the Space root. Verify with `python3 -m py_compile fast_hadamard_transform.py` before pushing. Build hangs = pip backtracking; read `--build` logs and pin the conflicting transitive dep.\n\n---\n\n## §8 Minimum Agent Loop\n\n0. Search reference Space: `hf spaces search \"\" --sdk gradio --limit 10`. Read its `app.py` + `requirements.txt`. Cap pre-push research at one reference Space.\n1. Decide SDK + hardware. Write minimal `app.py` + `requirements.txt` + README frontmatter.\n2. Push immediately: `hf repos create --exist-ok` then `hf upload` then `hf spaces logs --build --follow`.\n3. Once `RUNNING`: verify with `gradio_client` (see §11).\n4. Iterate via cheapest rung in §2.\n5. On error: read the FIRST error, make ONE targeted fix, use smallest rung.\n\n---\n\n## §9 Big LLM Cookbook\n\n**Pattern A (Inference Provider proxy):** stateless chat, any model size, zero VRAM, no `@spaces.GPU`, `cpu-basic` hardware.\n```python\ngr.load(\"models//\", accept_token=button, provider=\"fireworks-ai\")\n```\nREADME needs `hf_oauth: true` and `hf_oauth_scopes: [inference-api]`.\n\n**Pattern B (NF4 4-bit on ZeroGPU):** custom inference, tool use, multimodal, stateful.\n```python\nbnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type=\"nf4\",\n bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)\nmodel = AutoModelForCausalLM.from_pretrained(MODEL_ID, trust_remote_code=True,\n device_map=\"cuda\", torch_dtype=torch.bfloat16, quantization_config=bnb,\n attn_implementation=\"sdpa\", low_cpu_mem_usage=True).eval()\n```\nStreaming (non-negotiable for chat UX):\n```python\nstreamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True, timeout=120)\nThread(target=model.generate, kwargs=dict(**inputs, streamer=streamer, ...)).start()\n```\nPattern B minimum `requirements.txt`:\n```\ngradio>=6.10\nspaces>=0.41\ntransformers>=4.57\naccelerate>=1.10\nbitsandbytes>=0.48\nsentencepiece\n```\nBefore deploying Pattern B: set `GRADIO_SSR_MODE=false` and `startup_duration_timeout: 1h`. Full Pattern B code in full gist §9.\n\n---\n\n## §10 Custom Frontend + Storage\n\n```python\ndemo = app # HF runtime expects this name\n\n@spaces.GPU(duration=60)\ndef _run_gpu(prompt): return inference(prompt) # separate function\n\n@app.api(name=\"generate\", concurrency_limit=1, time_limit=180)\ndef generate(prompt: str) -> str: return _run_gpu(prompt) # never stack @app.api + @spaces.GPU\n```\nHot-reload (rung 1) does NOT work with `gr.Server`. Files in `@app.api` routes are plain dicts: `isinstance(v, dict) and (v.get(\"path\") or v.get(\"name\"))`.\n\n**Persistent storage (HF Buckets):**\n```bash\nhf buckets create /\nhf spaces volumes set / -v hf://buckets//:/data\n```\n`/data` is ephemeral by default; bucket mount makes it durable. Do NOT load model weights from `/data`; bucket I/O is S3-paced and stalls past any `@spaces.GPU` duration cap. Use bucket for user-generated content only.\n\n---\n\n## §11 Verifying a Deployed Space\n\n```bash\nhf spaces info --expand runtime --format json | python3 -c \\\n \"import json,sys; r=json.load(sys.stdin)['runtime']; print(r['stage'], r.get('hardware','?'), '| requested:', r.get('requested_hardware','?'))\"\n# expect: RUNNING zero-a10g | requested: zero-a10g\n```\n\n```python\nfrom gradio_client import Client, handle_file\nimport os\nc = Client(\"ns/name\", token=os.environ[\"HF_TOKEN\"], # token=, not hf_token=\n httpx_kwargs={\"timeout\": 600}) # >= @spaces.GPU duration + 60\nprint(c.view_api()) # always call first; never guess api_name\nresult = c.predict(handle_file(\"input.png\"), \"prompt\", api_name=\"/generate\")\n# Streaming:\njob = c.submit(\"prompt\", api_name=\"/chat\")\nfor chunk in job: print(chunk, end=\"\")\n# Custom @app.get/post routes (not in view_api()):\nimport httpx\nr = httpx.post(f\"{base}/your_route\", json={...}, headers=headers, timeout=600)\n```\n\nNo Playwright. No mock-mode local servers. No anonymous calls to private Spaces (returns 404, not an app bug).\n\n---\n\n## §12 HF Jobs / AOTI\n\n```bash\nhf jobs logs --follow # stream; never sleep+poll\n```\n`zero-a10g` flavor name is historical. Actual GPU is RTX PRO 6000 Blackwell (`sm_120`). Compile AOTI artifacts for `sm_120`, not A10G `sm_86`.\n\n---\n\n## Stuck-State Cheatsheet\n\n```\nBUILDING > 5min -> logs --build --tail 500, find FIRST error\nAPP_STARTING 10-25min (big LLM) -> NORMAL: model download/load; tail logs to see progress\nAPP_STARTING forever (small app) -> logs --tail 500; usually missing import or model OOM\nRUNTIME_ERROR right after long -> startup timed out: (1) set startup_duration_timeout: 1h\n APP_STARTING, sparse logs in README YAML AND (2) set GRADIO_SSR_MODE=false env var\nRUNNING but 404 from public URL -> Space is private; auth gradio_client with token=\nSSH hangs / Permission denied -> check ~/.ssh/id_rsa, key on /settings/keys, PRO/Team plan\n@spaces.GPU \"duration above max\" -> lower the value\nZeroGPU init spinner / GPU abort -> CUDA touched before allocation; for NeMo/RNNT set\n before app logs anything NUMBA_DISABLE_CUDA=1 then 'import spaces' immediately;\n for others reorder so 'import spaces' precedes torch\nlibcudart.so.X at startup -> reorder: import spaces before import torch\nFactory reboot fails \"could not -> previous commit was hot-reload-only; push any normal\n read Username...\" hf upload commit first, then restart\nruntime.sha stuck on old commit -> container still loading; poll SHA, do NOT re-restart\nStale example after asset regen -> gr.Examples cache keyed by path not hash; bump cache_version\n```\n\n---\n\n## Footgun Lookup Table\n\n| Symptom | Fix | Ref |\n|---|---|---|\n| Space on cpu-basic despite `hardware:` in README YAML | `--flavor` at create or `hf spaces settings --hardware zero-a10g`; frontmatter silently ignored | §1 |\n| `RuntimeError: Found no NVIDIA driver` inside `@spaces.GPU` | cpu-basic hardware (above) OR `device_map='cuda'` on plain `from_pretrained`; use `.to('cuda')` | §1/§4 |\n| factory-reboot fails `could not read Username` | Push any normal `hf upload` commit first, then restart | §2 |\n| `runtime.sha` stuck after upload | Container loading; do NOT issue another restart | §2 |\n| SSH Permission denied or hangs | Key on `/settings/keys`, PRO/Team plan, subdomain looked up dynamically | §3 |\n| SSH edits lost after restart | `git add . && git commit && git push` from inside SSH session | §3 |\n| `RUNTIME_ERROR` after long `APP_STARTING` | `startup_duration_timeout: 1h` in README YAML AND `GRADIO_SSR_MODE=false` env var | §5/§7 |\n| `RUNNING` but 404 | Private Space; `token=` (not `hf_token=`) in `gradio_client` | §11 |\n| `duration above maximum` at call time | Lower `@spaces.GPU(duration=)` value; not visible at deploy time | §4 |\n| ZeroGPU init spinner or aborted before app logs | NeMo: `NUMBA_DISABLE_CUDA=1` then `import spaces` immediately; others: `import spaces` before torch | §4 |\n| `libcudart.so.X` at startup | `import spaces` before `import torch` | §4 |\n| xformers CUDA extension mismatch | SDPA shim before model imports; full shim in full gist §4 item 5 | §4 |\n| Gemma2 `sdpa_mask` / `_vmap_for_bhqkv` failures | `attn_implementation='eager'` in `from_pretrained` | §4 |\n| `gr.Examples` stale output after in-place asset regen | Cache keyed by path not hash; bump `cache_version` | §5 |\n| `gr.Examples` eager burns ZeroGPU quota at startup | `cache_examples=True, cache_mode='lazy'` | §5 |\n| `TypeError: N+1 positional args` in callable duration | Add `*args, **kwargs` to duration function signature | §4 |\n| Files in `@app.api` routes are plain dicts not FileData | `isinstance(v, dict) and (v.get('path') or v.get('name'))` | §10 |\n| AOTI artifacts for `sm_86` fail on ZeroGPU | Compile for `sm_120`; `zero-a10g` name is historical | §12 |\n| ONNX custom op errors | `onnxruntime-gpu`; rewrite to opset-20; dynamic batch dim | §4 |\n| `enable_thinking=` raises `TypeError` in transformers >= 5 | `chat_template_kwargs={\"enable_thinking\": False}` in `apply_chat_template` | §4 |\n| `gradio_client` proxy in `@spaces.GPU` burns quota | Never wrap HTTP-only proxy in `@spaces.GPU`; proxy Space uses cpu-basic | §9 |\n| Hot-reload has no effect with `gr.Server` | Use `hf upload` (rung 3) or real commit | §10 |\n| `HF_HOME` set but cache fails silently | Set all three: `HF_HOME`, `HF_MODULES_CACHE=/tmp/hf_modules`, `MPLCONFIGDIR=/tmp/matplotlib`, before any import | §4 |\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_2","type":"agent_message","text":"I’ve read the requested playbook and will apply its deployment loop: reference search, early Space push, log-first debugging, and no local Gradio server or Playwright checks. Next I’m loading the HF CLI and Gradio skill instructions so the implementation and publish flow match the installed tooling."}} -{"type":"item.started","item":{"id":"item_3","type":"command_execution","command":"/bin/zsh -lc \"sed -n '1,240p' /root/.codex/plugins/cache/openai-curated-remote/hugging-face/1.0.0/skills/cli/SKILL.md\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_3","type":"command_execution","command":"/bin/zsh -lc \"sed -n '1,240p' /root/.codex/plugins/cache/openai-curated-remote/hugging-face/1.0.0/skills/cli/SKILL.md\"","aggregated_output":"---\nname: hf-cli\ndescription: \"Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing repositories, models, datasets, and Spaces on the Hugging Face Hub. Replaces now deprecated `huggingface-cli` command.\"\n---\n\nInstall: `curl -LsSf https://hf.co/cli/install.sh | bash -s`.\n\nThe Hugging Face Hub CLI tool `hf` is available. IMPORTANT: The `hf` command replaces the deprecated `huggingface-cli` command.\n\nUse `hf --help` to view available functions. Note that auth commands are now all under `hf auth` e.g. `hf auth whoami`.\n\n## Commands\n\n- `hf download REPO_ID` — Download files from the Hub. `[--type CHOICE --revision TEXT --include TEXT --exclude TEXT --cache-dir TEXT --local-dir TEXT --force-download --dry-run --quiet --max-workers INTEGER]`\n- `hf env` — Print information about the environment.\n- `hf sync` — Sync files between local directory and a bucket. `[--delete --ignore-times --ignore-sizes --plan TEXT --apply TEXT --dry-run --include TEXT --exclude TEXT --filter-from TEXT --existing --ignore-existing --verbose --quiet]`\n- `hf upload REPO_ID` — Upload a file or a folder to the Hub. Recommended for single-commit uploads. `[--type CHOICE --revision TEXT --private --include TEXT --exclude TEXT --delete TEXT --commit-message TEXT --commit-description TEXT --create-pr --every FLOAT --quiet]`\n- `hf upload-large-folder REPO_ID LOCAL_PATH` — Upload a large folder to the Hub. Recommended for resumable uploads. `[--type CHOICE --revision TEXT --private --include TEXT --exclude TEXT --num-workers INTEGER --no-report --no-bars]`\n- `hf version` — Print information about the hf version.\n\n### `hf auth` — Manage authentication (login, logout, etc.).\n\n- `hf auth list` — List all stored access tokens.\n- `hf auth login` — Login using a token from huggingface.co/settings/tokens. `[--add-to-git-credential --force]`\n- `hf auth logout` — Logout from a specific token. `[--token-name TEXT]`\n- `hf auth switch` — Switch between access tokens. `[--token-name TEXT --add-to-git-credential]`\n- `hf auth whoami` — Find out which huggingface.co account you are logged in as. `[--format CHOICE]`\n\n### `hf buckets` — Commands to interact with buckets.\n\n- `hf buckets cp SRC` — Copy a single file to or from a bucket. `[--quiet]`\n- `hf buckets create BUCKET_ID` — Create a new bucket. `[--private --exist-ok --quiet]`\n- `hf buckets delete BUCKET_ID` — Delete a bucket. `[--yes --missing-ok --quiet]`\n- `hf buckets info BUCKET_ID` — Get info about a bucket. `[--quiet]`\n- `hf buckets list` — List buckets or files in a bucket. `[--human-readable --tree --recursive --format CHOICE --quiet]`\n- `hf buckets move FROM_ID TO_ID` — Move (rename) a bucket to a new name or namespace.\n- `hf buckets remove ARGUMENT` — Remove files from a bucket. `[--recursive --yes --dry-run --include TEXT --exclude TEXT --quiet]`\n- `hf buckets sync` — Sync files between local directory and a bucket. `[--delete --ignore-times --ignore-sizes --plan TEXT --apply TEXT --dry-run --include TEXT --exclude TEXT --filter-from TEXT --existing --ignore-existing --verbose --quiet]`\n\n### `hf cache` — Manage local cache directory.\n\n- `hf cache list` — List cached repositories or revisions. `[--cache-dir TEXT --revisions --filter TEXT --format CHOICE --quiet --sort CHOICE --limit INTEGER]`\n- `hf cache prune` — Remove detached revisions from the cache. `[--cache-dir TEXT --yes --dry-run]`\n- `hf cache rm TARGETS` — Remove cached repositories or revisions. `[--cache-dir TEXT --yes --dry-run]`\n- `hf cache verify REPO_ID` — Verify checksums for a single repo revision from cache or a local directory. `[--type CHOICE --revision TEXT --cache-dir TEXT --local-dir TEXT --fail-on-missing-files --fail-on-extra-files]`\n\n### `hf collections` — Interact with collections on the Hub.\n\n- `hf collections add-item COLLECTION_SLUG ITEM_ID ITEM_TYPE` — Add an item to a collection. `[--note TEXT --exists-ok]`\n- `hf collections create TITLE` — Create a new collection on the Hub. `[--namespace TEXT --description TEXT --private --exists-ok]`\n- `hf collections delete COLLECTION_SLUG` — Delete a collection from the Hub. `[--missing-ok]`\n- `hf collections delete-item COLLECTION_SLUG ITEM_OBJECT_ID` — Delete an item from a collection. `[--missing-ok]`\n- `hf collections info COLLECTION_SLUG` — Get info about a collection on the Hub. Output is in JSON format.\n- `hf collections list` — List collections on the Hub. `[--owner TEXT --item TEXT --sort CHOICE --limit INTEGER --format CHOICE --quiet]`\n- `hf collections update COLLECTION_SLUG` — Update a collection's metadata on the Hub. `[--title TEXT --description TEXT --position INTEGER --private --theme TEXT]`\n- `hf collections update-item COLLECTION_SLUG ITEM_OBJECT_ID` — Update an item in a collection. `[--note TEXT --position INTEGER]`\n\n### `hf datasets` — Interact with datasets on the Hub.\n\n- `hf datasets info DATASET_ID` — Get info about a dataset on the Hub. Output is in JSON format. `[--revision TEXT --expand TEXT]`\n- `hf datasets list` — List datasets on the Hub. `[--search TEXT --author TEXT --filter TEXT --sort CHOICE --limit INTEGER --expand TEXT --format CHOICE --quiet]`\n- `hf datasets parquet DATASET_ID` — List parquet file URLs available for a dataset. `[--subset TEXT --split TEXT --format CHOICE --quiet]`\n- `hf datasets sql SQL` — Execute a raw SQL query with DuckDB against dataset parquet URLs. `[--format CHOICE]`\n\n### `hf discussions` — Manage discussions and pull requests on the Hub.\n\n- `hf discussions close REPO_ID NUM` — Close a discussion or pull request. `[--comment TEXT --yes --type CHOICE]`\n- `hf discussions comment REPO_ID NUM` — Comment on a discussion or pull request. `[--body TEXT --body-file PATH --type CHOICE]`\n- `hf discussions create REPO_ID --title TEXT` — Create a new discussion or pull request on a repo. `[--body TEXT --body-file PATH --pull-request --type CHOICE]`\n- `hf discussions diff REPO_ID NUM` — Show the diff of a pull request. `[--type CHOICE]`\n- `hf discussions info REPO_ID NUM` — Get info about a discussion or pull request. `[--comments --diff --no-color --type CHOICE --format CHOICE]`\n- `hf discussions list REPO_ID` — List discussions and pull requests on a repo. `[--status CHOICE --kind CHOICE --author TEXT --limit INTEGER --type CHOICE --format CHOICE --quiet]`\n- `hf discussions merge REPO_ID NUM` — Merge a pull request. `[--comment TEXT --yes --type CHOICE]`\n- `hf discussions rename REPO_ID NUM NEW_TITLE` — Rename a discussion or pull request. `[--type CHOICE]`\n- `hf discussions reopen REPO_ID NUM` — Reopen a closed discussion or pull request. `[--comment TEXT --yes --type CHOICE]`\n\n### `hf endpoints` — Manage Hugging Face Inference Endpoints.\n\n- `hf endpoints catalog deploy --repo TEXT` — Deploy an Inference Endpoint from the Model Catalog. `[--name TEXT --accelerator TEXT --namespace TEXT]`\n- `hf endpoints catalog list` — List available Catalog models.\n- `hf endpoints delete NAME` — Delete an Inference Endpoint permanently. `[--namespace TEXT --yes]`\n- `hf endpoints deploy NAME --repo TEXT --framework TEXT --accelerator TEXT --instance-size TEXT --instance-type TEXT --region TEXT --vendor TEXT` — Deploy an Inference Endpoint from a Hub repository. `[--namespace TEXT --task TEXT --min-replica INTEGER --max-replica INTEGER --scale-to-zero-timeout INTEGER --scaling-metric CHOICE --scaling-threshold FLOAT]`\n- `hf endpoints describe NAME` — Get information about an existing endpoint. `[--namespace TEXT]`\n- `hf endpoints list` — Lists all Inference Endpoints for the given namespace. `[--namespace TEXT --format CHOICE --quiet]`\n- `hf endpoints pause NAME` — Pause an Inference Endpoint. `[--namespace TEXT]`\n- `hf endpoints resume NAME` — Resume an Inference Endpoint. `[--namespace TEXT --fail-if-already-running]`\n- `hf endpoints scale-to-zero NAME` — Scale an Inference Endpoint to zero. `[--namespace TEXT]`\n- `hf endpoints update NAME` — Update an existing endpoint. `[--namespace TEXT --repo TEXT --accelerator TEXT --instance-size TEXT --instance-type TEXT --framework TEXT --revision TEXT --task TEXT --min-replica INTEGER --max-replica INTEGER --scale-to-zero-timeout INTEGER --scaling-metric CHOICE --scaling-threshold FLOAT]`\n\n### `hf extensions` — Manage hf CLI extensions.\n\n- `hf extensions exec NAME` — Execute an installed extension.\n- `hf extensions install REPO_ID` — Install an extension from a public GitHub repository. `[--force]`\n- `hf extensions list` — List installed extension commands. `[--format CHOICE --quiet]`\n- `hf extensions remove NAME` — Remove an installed extension.\n- `hf extensions search` — Search extensions available on GitHub (tagged with 'hf-extension' topic). `[--format CHOICE --quiet]`\n\n### `hf jobs` — Run and manage Jobs on the Hub.\n\n- `hf jobs cancel JOB_ID` — Cancel a Job `[--namespace TEXT]`\n- `hf jobs hardware` — List available hardware options for Jobs\n- `hf jobs inspect JOB_IDS` — Display detailed information on one or more Jobs `[--namespace TEXT]`\n- `hf jobs logs JOB_ID` — Fetch the logs of a Job. `[--follow --tail INTEGER --namespace TEXT]`\n- `hf jobs ps` — List Jobs. `[--all --namespace TEXT --filter TEXT --format TEXT --quiet]`\n- `hf jobs run IMAGE COMMAND` — Run a Job. `[--env TEXT --secrets TEXT --label TEXT --env-file TEXT --secrets-file TEXT --flavor CHOICE --timeout TEXT --detach --namespace TEXT]`\n- `hf jobs scheduled delete SCHEDULED_JOB_ID` — Delete a scheduled Job. `[--namespace TEXT]`\n- `hf jobs scheduled inspect SCHEDULED_JOB_IDS` — Display detailed information on one or more scheduled Jobs `[--namespace TEXT]`\n- `hf jobs scheduled ps` — List scheduled Jobs `[--all --namespace TEXT --filter TEXT --format TEXT --quiet]`\n- `hf jobs scheduled resume SCHEDULED_JOB_ID` — Resume (unpause) a scheduled Job. `[--namespace TEXT]`\n- `hf jobs scheduled run SCHEDULE IMAGE COMMAND` — Schedule a Job. `[--suspend --concurrency --env TEXT --secrets TEXT --label TEXT --env-file TEXT --secrets-file TEXT --flavor CHOICE --timeout TEXT --namespace TEXT]`\n- `hf jobs scheduled suspend SCHEDULED_JOB_ID` — Suspend (pause) a scheduled Job. `[--namespace TEXT]`\n- `hf jobs scheduled uv run SCHEDULE SCRIPT` — Run a UV script (local file or URL) on HF infrastructure `[--suspend --concurrency --image TEXT --flavor CHOICE --env TEXT --secrets TEXT --label TEXT --env-file TEXT --secrets-file TEXT --timeout TEXT --namespace TEXT --with TEXT --python TEXT]`\n- `hf jobs stats` — Fetch the resource usage statistics and metrics of Jobs `[--namespace TEXT]`\n- `hf jobs uv run SCRIPT` — Run a UV script (local file or URL) on HF infrastructure `[--image TEXT --flavor CHOICE --env TEXT --secrets TEXT --label TEXT --env-file TEXT --secrets-file TEXT --timeout TEXT --detach --namespace TEXT --with TEXT --python TEXT]`\n\n### `hf models` — Interact with models on the Hub.\n\n- `hf models info MODEL_ID` — Get info about a model on the Hub. Output is in JSON format. `[--revision TEXT --expand TEXT]`\n- `hf models list` — List models on the Hub. `[--search TEXT --author TEXT --filter TEXT --num-parameters TEXT --sort CHOICE --limit INTEGER --expand TEXT --format CHOICE --quiet]`\n\n### `hf papers` — Interact with papers on the Hub.\n\n- `hf papers list` — List daily papers on the Hub. `[--date TEXT --sort CHOICE --limit INTEGER --format CHOICE --quiet]`\n\n### `hf repos` — Manage repos on the Hub.\n\n- `hf repos branch create REPO_ID BRANCH` — Create a new branch for a repo on the Hub. `[--revision TEXT --type CHOICE --exist-ok]`\n- `hf repos branch delete REPO_ID BRANCH` — Delete a branch from a repo on the Hub. `[--type CHOICE]`\n- `hf repos create REPO_ID` — Create a new repo on the Hub. `[--type CHOICE --space-sdk TEXT --private --exist-ok --resource-group-id TEXT]`\n- `hf repos delete REPO_ID` — Delete a repo from the Hub. This is an irreversible operation. `[--type CHOICE --missing-ok]`\n- `hf repos delete-files REPO_ID PATTERNS` — Delete files from a repo on the Hub. `[--type CHOICE --revision TEXT --commit-message TEXT --commit-description TEXT --create-pr]`\n- `hf repos duplicate FROM_ID` — Duplicate a repo on the Hub (model, dataset, or Space). `[--type CHOICE --private --exist-ok]`\n- `hf repos move FROM_ID TO_ID` — Move a repository from a namespace to another namespace. `[--type CHOICE]`\n- `hf repos settings REPO_ID` — Update the settings of a repository. `[--gated CHOICE --private --type CHOICE]`\n- `hf repos tag create REPO_ID TAG` — Create a tag for a repo. `[--message TEXT --revision TEXT --type CHOICE]`\n- `hf repos tag delete REPO_ID TAG` — Delete a tag for a repo. `[--yes --type CHOICE]`\n- `hf repos tag list REPO_ID` — List tags for a repo. `[--type CHOICE]`\n\n### `hf skills` — Manage skills for AI assistants.\n\n- `hf skills add` — Download a skill and install it for an AI assistant. `[--claude --codex --cursor --opencode --global --dest PATH --force]`\n- `hf skills preview` — Print the generated SKILL.md to stdout.\n\n### `hf spaces` — Interact with spaces on the Hub.\n\n- `hf spaces dev-mode SPACE_ID` — Enable or disable dev mode on a Space. `[--stop]`\n- `hf spaces hot-reload SPACE_ID` — Hot-reload any Python file of a Space without a full rebuild + restart. `[--local-file TEXT --skip-checks --skip-summary]`\n- `hf spaces info SPACE_ID` — Get info about a space on the Hub. Output is in JSON format. `[--revision TEXT --expand TEXT]`\n- `hf spaces list` — List spaces on the Hub. `[--search TEXT --author TEXT --filter TEXT --sort CHOICE --limit INTEGER --expand TEXT --format CHOICE --quiet]`\n\n### `hf webhooks` — Manage webhooks on the Hub.\n\n- `hf webhooks create --watch TEXT` — Create a new webhook. `[--url TEXT --job-id TEXT --domain CHOICE --secret TEXT]`\n- `hf webhooks delete WEBHOOK_ID` — Delete a webhook permanently. `[--yes]`\n- `hf webhooks disable WEBHOOK_ID` — Disable an active webhook.\n- `hf webhooks enable WEBHOOK_ID` — Enable a disabled webhook.\n- `hf webhooks info WEBHOOK_ID` — Show full details for a single webhook as JSON.\n- `hf webhooks list` — List all webhooks for the current user. `[--format CHOICE --quiet]`\n- `hf webhooks update WEBHOOK_ID` — Update an existing webhook. Only provided options are changed. `[--url TEXT --watch TEXT --domain CHOICE --secret TEXT]`\n\n## Common options\n\n- `--format` — Output format: `--format json` (or `--json`) or `--format table` (default).\n- `-q / --quiet` — Minimal output.\n- `--revision` — Git revision id which can be a branch name, a tag, or a commit hash.\n- `--token` — Use a User Access Token. Prefer setting `HF_TOKEN` env var instead of passing `--token`.\n- `--type` — The type of repository (model, dataset, or space).\n\n## Tips\n\n- Use `hf --help` for full options, descriptions, usage, and real-world examples\n- Authenticate with `HF_TOKEN` env var (recommended) or with `--token`\n","exit_code":0,"status":"completed"}} -{"type":"item.started","item":{"id":"item_4","type":"command_execution","command":"/bin/zsh -lc \"sed -n '1,260p' /root/.codex/plugins/cache/openai-curated-remote/hugging-face/1.0.0/skills/gradio/SKILL.md\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_4","type":"command_execution","command":"/bin/zsh -lc \"sed -n '1,260p' /root/.codex/plugins/cache/openai-curated-remote/hugging-face/1.0.0/skills/gradio/SKILL.md\"","aggregated_output":"---\nname: huggingface-gradio\ndescription: Build Gradio web UIs and demos in Python. Use when creating or editing Gradio apps, components, event listeners, layouts, or chatbots.\n---\n\n# Gradio\n\nGradio is a Python library for building interactive web UIs and ML demos. This skill covers the core API, patterns, and examples.\n\n## Guides\n\nDetailed guides on specific topics (read these when relevant):\n\n- [Quickstart](https://www.gradio.app/guides/quickstart)\n- [The Interface Class](https://www.gradio.app/guides/the-interface-class)\n- [Blocks and Event Listeners](https://www.gradio.app/guides/blocks-and-event-listeners)\n- [Controlling Layout](https://www.gradio.app/guides/controlling-layout)\n- [More Blocks Features](https://www.gradio.app/guides/more-blocks-features)\n- [Custom CSS and JS](https://www.gradio.app/guides/custom-CSS-and-JS)\n- [Streaming Outputs](https://www.gradio.app/guides/streaming-outputs)\n- [Streaming Inputs](https://www.gradio.app/guides/streaming-inputs)\n- [Sharing Your App](https://www.gradio.app/guides/sharing-your-app)\n- [Custom HTML Components](https://www.gradio.app/guides/custom-HTML-components)\n- [Getting Started with the Python Client](https://www.gradio.app/guides/getting-started-with-the-python-client)\n- [Getting Started with the JS Client](https://www.gradio.app/guides/getting-started-with-the-js-client)\n\n## Core Patterns\n\n**Interface** (high-level): wraps a function with input/output components.\n\n```python\nimport gradio as gr\n\ndef greet(name):\n return f\"Hello {name}!\"\n\ngr.Interface(fn=greet, inputs=\"text\", outputs=\"text\").launch()\n```\n\n**Blocks** (low-level): flexible layout with explicit event wiring.\n\n```python\nimport gradio as gr\n\nwith gr.Blocks() as demo:\n name = gr.Textbox(label=\"Name\")\n output = gr.Textbox(label=\"Greeting\")\n btn = gr.Button(\"Greet\")\n btn.click(fn=lambda n: f\"Hello {n}!\", inputs=name, outputs=output)\n\ndemo.launch()\n```\n\n**ChatInterface**: high-level wrapper for chatbot UIs.\n\n```python\nimport gradio as gr\n\ndef respond(message, history):\n return f\"You said: {message}\"\n\ngr.ChatInterface(fn=respond).launch()\n```\n\n## Key Component Signatures\n\n### `Textbox(value: str | I18nData | Callable | None = None, type: Literal['text', 'password', 'email'] = \"text\", lines: int = 1, max_lines: int | None = None, placeholder: str | I18nData | None = None, label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, autofocus: bool = False, autoscroll: bool = True, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = \"value\", text_align: Literal['left', 'right'] | None = None, rtl: bool = False, buttons: list[Literal['copy'] | Button] | None = None, max_length: int | None = None, submit_btn: str | bool | None = False, stop_btn: str | bool | None = False, html_attributes: InputHTMLAttributes | None = None)`\nCreates a textarea for user to enter string input or display string output..\n\n### `Number(value: float | Callable | None = None, label: str | I18nData | None = None, placeholder: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = \"value\", buttons: list[Button] | None = None, precision: int | None = None, minimum: float | None = None, maximum: float | None = None, step: float = 1)`\nCreates a numeric field for user to enter numbers as input or display numeric output..\n\n### `Slider(minimum: float = 0, maximum: float = 100, value: float | Callable | None = None, step: float | None = None, precision: int | None = None, label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = \"value\", randomize: bool = False, buttons: list[Literal['reset']] | None = None)`\nCreates a slider that ranges from {minimum} to {maximum} with a step size of {step}..\n\n### `Checkbox(value: bool | Callable = False, label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = \"value\", buttons: list[Button] | None = None)`\nCreates a checkbox that can be set to `True` or `False`.\n\n### `Dropdown(choices: Sequence[str | int | float | tuple[str, str | int | float]] | None = None, value: str | int | float | Sequence[str | int | float] | Callable | DefaultValue | None = DefaultValue(), type: Literal['value', 'index'] = \"value\", multiselect: bool | None = None, allow_custom_value: bool = False, max_choices: int | None = None, filterable: bool = True, label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = \"value\", buttons: list[Button] | None = None)`\nCreates a dropdown of choices from which a single entry or multiple entries can be selected (as an input component) or displayed (as an output component)..\n\n### `Radio(choices: Sequence[str | int | float | tuple[str, str | int | float]] | None = None, value: str | int | float | Callable | None = None, type: Literal['value', 'index'] = \"value\", label: str | I18nData | None = None, info: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = \"value\", rtl: bool = False, buttons: list[Button] | None = None)`\nCreates a set of (string or numeric type) radio buttons of which only one can be selected..\n\n### `Image(value: str | PIL.Image.Image | np.ndarray | Callable | None = None, format: str = \"webp\", height: int | str | None = None, width: int | str | None = None, image_mode: Literal['1', 'L', 'P', 'RGB', 'RGBA', 'CMYK', 'YCbCr', 'LAB', 'HSV', 'I', 'F'] | None = \"RGB\", sources: list[Literal['upload', 'webcam', 'clipboard']] | Literal['upload', 'webcam', 'clipboard'] | None = None, type: Literal['numpy', 'pil', 'filepath'] = \"numpy\", label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, buttons: list[Literal['download', 'share', 'fullscreen'] | Button] | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, streaming: bool = False, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = \"value\", webcam_options: WebcamOptions | None = None, placeholder: str | None = None, watermark: WatermarkOptions | None = None)`\nCreates an image component that can be used to upload images (as an input) or display images (as an output)..\n\n### `Audio(value: str | Path | tuple[int, np.ndarray] | Callable | None = None, sources: list[Literal['upload', 'microphone']] | Literal['upload', 'microphone'] | None = None, type: Literal['numpy', 'filepath'] = \"numpy\", label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, streaming: bool = False, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = \"value\", format: Literal['wav', 'mp3'] | None = None, autoplay: bool = False, editable: bool = True, buttons: list[Literal['download', 'share'] | Button] | None = None, waveform_options: WaveformOptions | dict | None = None, loop: bool = False, recording: bool = False, subtitles: str | Path | list[dict[str, Any]] | None = None, playback_position: float = 0)`\nCreates an audio component that can be used to upload/record audio (as an input) or display audio (as an output)..\n\n### `Video(value: str | Path | Callable | None = None, format: str | None = None, sources: list[Literal['upload', 'webcam']] | Literal['upload', 'webcam'] | None = None, height: int | str | None = None, width: int | str | None = None, label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = \"value\", webcam_options: WebcamOptions | None = None, include_audio: bool | None = None, autoplay: bool = False, buttons: list[Literal['download', 'share'] | Button] | None = None, loop: bool = False, streaming: bool = False, watermark: WatermarkOptions | None = None, subtitles: str | Path | list[dict[str, Any]] | None = None, playback_position: float = 0)`\nCreates a video component that can be used to upload/record videos (as an input) or display videos (as an output).\n\n### `File(value: str | list[str] | Callable | None = None, file_count: Literal['single', 'multiple', 'directory'] = \"single\", file_types: list[str] | None = None, type: Literal['filepath', 'binary'] = \"filepath\", label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, height: int | str | float | None = None, interactive: bool | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = \"value\", allow_reordering: bool = False, buttons: list[Button] | None = None)`\nCreates a file component that allows uploading one or more generic files (when used as an input) or displaying generic files or URLs for download (as output).\n\n### `Chatbot(value: list[MessageDict | Message] | Callable | None = None, label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, container: bool = True, scale: int | None = None, min_width: int = 160, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, autoscroll: bool = True, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = \"value\", height: int | str | None = 400, resizable: bool = False, max_height: int | str | None = None, min_height: int | str | None = None, editable: Literal['user', 'all'] | None = None, latex_delimiters: list[dict[str, str | bool]] | None = None, rtl: bool = False, buttons: list[Literal['share', 'copy', 'copy_all'] | Button] | None = None, watermark: str | None = None, avatar_images: tuple[str | Path | None, str | Path | None] | None = None, sanitize_html: bool = True, render_markdown: bool = True, feedback_options: list[str] | tuple[str, ...] | None = ('Like', 'Dislike'), feedback_value: Sequence[str | None] | None = None, line_breaks: bool = True, layout: Literal['panel', 'bubble'] | None = None, placeholder: str | None = None, examples: list[ExampleMessage] | None = None, allow_file_downloads: = True, group_consecutive_messages: bool = True, allow_tags: list[str] | bool = True, reasoning_tags: list[tuple[str, str]] | None = None, like_user_message: bool = False)`\nCreates a chatbot that displays user-submitted messages and responses.\n\n### `Button(value: str | I18nData | Callable = \"Run\", every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, variant: Literal['primary', 'secondary', 'stop', 'huggingface'] = \"secondary\", size: Literal['sm', 'md', 'lg'] = \"lg\", icon: str | Path | None = None, link: str | None = None, link_target: Literal['_self', '_blank', '_parent', '_top'] = \"_self\", visible: bool | Literal['hidden'] = True, interactive: bool = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = \"value\", scale: int | None = None, min_width: int | None = None)`\nCreates a button that can be assigned arbitrary .click() events.\n\n### `Markdown(value: str | I18nData | Callable | None = None, label: str | I18nData | None = None, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool | None = None, rtl: bool = False, latex_delimiters: list[dict[str, str | bool]] | None = None, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = \"value\", sanitize_html: bool = True, line_breaks: bool = False, header_links: bool = False, height: int | str | None = None, max_height: int | str | None = None, min_height: int | str | None = None, buttons: list[Literal['copy']] | None = None, container: bool = False, padding: bool = False)`\nUsed to render arbitrary Markdown output.\n\n### `HTML(value: Any | Callable | None = None, label: str | I18nData | None = None, html_template: str = \"${value}\", css_template: str = \"\", js_on_load: str | None = \"element.addEventListener('click', function() { trigger('click') });\", apply_default_css: bool = True, every: Timer | float | None = None, inputs: Component | Sequence[Component] | set[Component] | None = None, show_label: bool = False, visible: bool | Literal['hidden'] = True, elem_id: str | None = None, elem_classes: list[str] | str | None = None, render: bool = True, key: int | str | tuple[int | str, ...] | None = None, preserved_by_key: list[str] | str | None = \"value\", min_height: int | None = None, max_height: int | None = None, container: bool = False, padding: bool = False, autoscroll: bool = False, buttons: list[Button] | None = None, props: Any)`\nCreates a component with arbitrary HTML.\n\n\n## Custom HTML Components\n\nIf a task requires significant customization of an existing component or a component that doesn't exist in Gradio, you can create one with `gr.HTML`. It supports `html_template` (with `${}` JS expressions and `{{}}` Handlebars syntax), `css_template` for scoped styles, and `js_on_load` for interactivity — where `props.value` updates the component value and `trigger('event_name')` fires Gradio events. For reuse, subclass `gr.HTML` and define `api_info()` for API/MCP support. See the [full guide](https://www.gradio.app/guides/custom-HTML-components).\n\nHere's an example that shows how to create and use these kinds of components:\n\n```python\nimport gradio as gr\n\nclass StarRating(gr.HTML):\n def __init__(self, label, value=0, **kwargs):\n html_template = \"\"\"\n

${label} rating:

\n ${Array.from({length: 5}, (_, i) => ``).join('')}\n \"\"\"\n css_template = \"\"\"\n img { height: 50px; display: inline-block; cursor: pointer; }\n .faded { filter: grayscale(100%); opacity: 0.3; }\n \"\"\"\n js_on_load = \"\"\"\n const imgs = element.querySelectorAll('img');\n imgs.forEach((img, index) => {\n img.addEventListener('click', () => {\n props.value = index + 1;\n });\n });\n \"\"\"\n super().__init__(value=value, label=label, html_template=html_template, css_template=css_template, js_on_load=js_on_load, **kwargs)\n\n def api_info(self):\n return {\"type\": \"integer\", \"minimum\": 0, \"maximum\": 5}\n\n\nwith gr.Blocks() as demo:\n gr.Markdown(\"# Restaurant Review\")\n food_rating = StarRating(label=\"Food\", value=3)\n service_rating = StarRating(label=\"Service\", value=3)\n ambience_rating = StarRating(label=\"Ambience\", value=3)\n average_btn = gr.Button(\"Calculate Average Rating\")\n rating_output = StarRating(label=\"Average\", value=3)\n def calculate_average(food, service, ambience):\n return round((food + service + ambience) / 3)\n average_btn.click(\n fn=calculate_average,\n inputs=[food_rating, service_rating, ambience_rating],\n outputs=rating_output\n )\n\ndemo.launch()\n```\n\n## Event Listeners\n\nAll event listeners share the same signature:\n\n```python\ncomponent.event_name(\n fn: Callable | None | Literal[\"decorator\"] = \"decorator\",\n inputs: Component | Sequence[Component] | set[Component] | None = None,\n outputs: Component | Sequence[Component] | set[Component] | None = None,\n api_name: str | None = None,\n api_description: str | None | Literal[False] = None,\n scroll_to_output: bool = False,\n show_progress: Literal[\"full\", \"minimal\", \"hidden\"] = \"full\",\n show_progress_on: Component | Sequence[Component] | None = None,\n queue: bool = True,\n batch: bool = False,\n max_batch_size: int = 4,\n preprocess: bool = True,\n postprocess: bool = True,\n cancels: dict[str, Any] | list[dict[str, Any]] | None = None,\n trigger_mode: Literal[\"once\", \"multiple\", \"always_last\"] | None = None,\n js: str | Literal[True] | None = None,\n concurrency_limit: int | None | Literal[\"default\"] = \"default\",\n concurrency_id: str | None = None,\n api_visibility: Literal[\"public\", \"private\", \"undocumented\"] = \"public\",\n time_limit: int | None = None,\n stream_every: float = 0.5,\n key: int | str | tuple[int | str, ...] | None = None,\n validator: Callable | None = None,\n) -> Dependency\n```\n\nSupported events per component:\n\n- **AnnotatedImage**: select\n- **Audio**: stream, change, clear, play, pause, stop, pause, start_recording, pause_recording, stop_recording, upload, input\n- **BarPlot**: select, double_click\n- **BrowserState**: change\n- **Button**: click\n- **Chatbot**: change, select, like, retry, undo, example_select, option_select, clear, copy, edit\n- **Checkbox**: change, input, select\n- **CheckboxGroup**: change, input, select\n- **ClearButton**: click\n- **Code**: change, input, focus, blur\n- **ColorPicker**: change, input, submit, focus, blur\n- **Dataframe**: change, input, select, edit\n- **Dataset**: click, select\n- **DateTime**: change, submit\n- **DeepLinkButton**: click\n- **Dialogue**: change, input, submit\n- **DownloadButton**: click\n- **Dropdown**: change, input, select, focus, blur, key_up\n- **DuplicateButton**: click\n- **File**: change, select, clear, upload, delete, download\n- **FileExplorer**: change, input, select\n- **Gallery**: select, upload, change, delete, preview_close, preview_open\n- **HTML**: change, input, click, double_click, submit, stop, edit, clear, play, pause, end, start_recording, pause_recording, stop_recording, focus, blur, upload, release, select, stream, like, example_select, option_select, load, key_up, apply, delete, tick, undo, retry, expand, collapse, download, copy\n- **HighlightedText**: change, select\n- **Image**: clear, change, stream, select, upload, input\n- **ImageEditor**: clear, change, input, select, upload, apply\n- **ImageSlider**: clear, change, stream, select, upload, input\n- **JSON**: change\n- **Label**: change, select\n- **LinePlot**: select, double_click\n- **LoginButton**: click\n- **Markdown**: change, copy\n- **Model3D**: change, upload, edit, clear\n- **MultimodalTextbox**: change, input, select, submit, focus, blur, stop\n- **Navbar**: change\n- **Number**: change, input, submit, focus, blur\n- **ParamViewer**: change, upload\n- **Plot**: change\n- **Radio**: select, change, input\n- **ScatterPlot**: select, double_click\n- **SimpleImage**: clear, change, upload\n- **Slider**: change, input, release\n- **State**: change\n- **Textbox**: change, input, select, submit, focus, blur, stop, copy\n- **Timer**: tick\n- **UploadButton**: click, upload\n- **Video**: change, clear, start_recording, stop_recording, stop, play, pause, end, upload, input\n\n## Additional Reference\n\n- [End-to-End Examples](examples.md) — complete working apps\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_5","type":"agent_message","text":"I’m checking the current workspace and HF account, and I’m doing the required reference Space search before deeper model research. That should tell me whether there’s an existing demo pattern worth borrowing, without spending much time on it."}} -{"type":"item.started","item":{"id":"item_6","type":"command_execution","command":"/bin/zsh -lc 'git status --short'","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.started","item":{"id":"item_7","type":"command_execution","command":"/bin/zsh -lc \"pwd && rg --files -g '\"'!__pycache__'\"' -g '\"'!.git'\"'\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.started","item":{"id":"item_8","type":"command_execution","command":"/bin/zsh -lc 'hf auth whoami --format json'","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.started","item":{"id":"item_9","type":"command_execution","command":"/bin/zsh -lc 'hf spaces search \"DeepSpec speculative decoding deepseek\" --sdk gradio --limit 10'","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.started","item":{"id":"item_10","type":"command_execution","command":"/bin/zsh -lc 'hf collections info deepseek-ai/deepspec --format json'","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_6","type":"command_execution","command":"/bin/zsh -lc 'git status --short'","aggregated_output":"fatal: not a git repository (or any of the parent directories): .git\n","exit_code":128,"status":"failed"}} -{"type":"item.completed","item":{"id":"item_7","type":"command_execution","command":"/bin/zsh -lc \"pwd && rg --files -g '\"'!__pycache__'\"' -g '\"'!.git'\"'\"","aggregated_output":"PROMPT.txt\nrollout.jsonl\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_8","type":"command_execution","command":"/bin/zsh -lc 'hf auth whoami --format json'","aggregated_output":"{\"user\": \"Mike0021\", \"orgs\": \"gemma-challenge\", \"endpoint\": null}\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_10","type":"command_execution","command":"/bin/zsh -lc 'hf collections info deepseek-ai/deepspec --format json'","aggregated_output":"{\"slug\": \"deepseek-ai/deepspec-6a410e3f1831ca8ca801b88b\", \"title\": \"DeepSpec\", \"owner\": {\"_id\": \"652faff917096ceb6bf53f3f\", \"avatarUrl\": \"https://cdn-avatars.huggingface.co/v1/production/uploads/6538815d1bdb3c40db94fbfa/xMBly9PUMphrFVMxLX4kq.png\", \"fullname\": \"DeepSeek\", \"name\": \"deepseek-ai\", \"type\": \"org\", \"isHf\": false, \"isHfAdmin\": false, \"isMod\": false, \"followerCount\": 136307, \"isUserFollowing\": false}, \"items\": [{\"item_object_id\": \"6a41126725ecbea1adc33d6a\", \"item_id\": \"deepseek-ai/dspark_qwen3_4b_block7\", \"item_type\": \"model\", \"position\": 0, \"note\": \"dspark_qwen3_4b_block7\"}, {\"item_object_id\": \"6a4112bc9d9bac32330339ad\", \"item_id\": \"deepseek-ai/dspark_qwen3_8b_block7\", \"item_type\": \"model\", \"position\": 1, \"note\": \"dspark_qwen3_8b_block7\"}, {\"item_object_id\": \"6a4113371831ca8ca802316d\", \"item_id\": \"deepseek-ai/dspark_qwen3_14b_block7\", \"item_type\": \"model\", \"position\": 2, \"note\": \"dspark_qwen3_14b_block7\"}, {\"item_object_id\": \"6a41139703d8215f35ac29f0\", \"item_id\": \"deepseek-ai/dspark_gemma4_12b_block7\", \"item_type\": \"model\", \"position\": 3, \"note\": \"dspark_gemma4_12b_block7\"}, {\"item_object_id\": \"6a4113ced6f7a065b898d384\", \"item_id\": \"deepseek-ai/dflash_qwen3_4b_block7\", \"item_type\": \"model\", \"position\": 4, \"note\": \"dflash_qwen3_4b_block7\"}, {\"item_object_id\": \"6a4114218791003d8e886ee4\", \"item_id\": \"deepseek-ai/dflash_qwen3_8b_block7\", \"item_type\": \"model\", \"position\": 5, \"note\": \"dflash_qwen3_8b_block7\"}, {\"item_object_id\": \"6a411493665348b0c8b4f984\", \"item_id\": \"deepseek-ai/dflash_qwen3_14b_block7\", \"item_type\": \"model\", \"position\": 6, \"note\": \"dflash_qwen3_14b_block7\"}, {\"item_object_id\": \"6a4114f2730bbee3abcc366a\", \"item_id\": \"deepseek-ai/dflash_gemma4_12b_block7\", \"item_type\": \"model\", \"position\": 7, \"note\": \"dflash_gemma4_12b_block7\"}, {\"item_object_id\": \"6a4115108791003d8e88883e\", \"item_id\": \"deepseek-ai/eagle3_qwen3_4b_ttt7\", \"item_type\": \"model\", \"position\": 8, \"note\": \"eagle3_qwen3_4b_ttt7\"}, {\"item_object_id\": \"6a4115437a3e55a06214e222\", \"item_id\": \"deepseek-ai/eagle3_qwen3_8b_ttt7\", \"item_type\": \"model\", \"position\": 9, \"note\": \"eagle3_qwen3_8b_ttt7\"}, {\"item_object_id\": \"6a41157c762f2302e0c77210\", \"item_id\": \"deepseek-ai/eagle3_qwen3_14b_ttt7\", \"item_type\": \"model\", \"position\": 10, \"note\": \"eagle3_qwen3_14b_ttt7\"}, {\"item_object_id\": \"6a4115b54d64a10aa84dea4f\", \"item_id\": \"deepseek-ai/eagle3_gemma4_12b_ttt7\", \"item_type\": \"model\", \"position\": 11, \"note\": \"eagle3_gemma4_12b_ttt7\"}], \"last_updated\": \"2026-06-28T12:42:42.271000+00:00\", \"position\": 0, \"private\": false, \"theme\": \"pink\", \"upvotes\": 42}\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_9","type":"command_execution","command":"/bin/zsh -lc 'hf spaces search \"DeepSpec speculative decoding deepseek\" --sdk gradio --limit 10'","aggregated_output":"No results found.\nHint: Use --description to show AI-generated descriptions.\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_11","type":"agent_message","text":"The collection resolves cleanly under `deepseek-ai/deepspec-6a410e3f1831ca8ca801b88b` and the authenticated HF namespace is `Mike0021`. I’m pulling the actual model-card content next; the reference Space search returned no existing DeepSpec Gradio Space to use as a template."}} -{"type":"item.started","item":{"id":"item_12","type":"command_execution","command":"/bin/zsh -lc \"sed -n '1,220p' PROMPT.txt\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_12","type":"command_execution","command":"/bin/zsh -lc \"sed -n '1,220p' PROMPT.txt\"","aggregated_output":"/goal Research the DeepSpec collection by deepseek-ai on HuggingFace (https://huggingface.co/collections/deepseek-ai/deepspec) and build the most impressive HuggingFace Space possible from it.\n\nThe collection contains 12 models across three speculative decoding families:\n- dSpark (qwen3_4b, qwen3_8b, qwen3_14b, gemma4_12b) - block7 variants\n- dFlash (qwen3_4b, qwen3_8b, qwen3_14b, gemma4_12b) - block7 variants \n- EAGLE3 (qwen3_4b, qwen3_8b, qwen3_14b, gemma4_12b) - ttt7 variants\n\nSteps:\n1. Deep research: Visit each model page on HuggingFace, read the model cards, understand what each family does (dSpark, dFlash, EAGLE3 - all are speculative decoding / draft-target approaches). Read any linked papers. Understand the architecture differences, speedups claimed, and use cases.\n2. Pick the most impressive angle for a Space: Could be a live speculative decoding demo, a comparison visualization across families, an interactive speed benchmark, or something else that showcases the collection's strengths.\n3. Implement the Space as a Gradio app. Make it visually impressive and functional.\n4. Push to a HuggingFace Space repo.\n\nBefore doing anything else, fetch and follow these HF Spaces operating guidelines for the entire task:\ncurl -L --fail --silent https://gist.githubusercontent.com/gary149/37c955b832558837c40e1c14ff6d955d/raw/ad35807f8466378afd04d7653d53683a847b96c4/hf-spaces-agent-quickstart-compact.md\nRead them fully first, then apply them throughout (iteration ladder, push early, ZeroGPU duration measurement, and the listed anti-patterns).\n","exit_code":0,"status":"completed"}} -{"type":"item.started","item":{"id":"item_13","type":"command_execution","command":"/bin/zsh -lc \"wc -l rollout.jsonl && sed -n '1,5p' rollout.jsonl\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_13","type":"command_execution","command":"/bin/zsh -lc \"wc -l rollout.jsonl && sed -n '1,5p' rollout.jsonl\"","aggregated_output":"25 rollout.jsonl\nReading additional input from stdin...\n{\"type\":\"thread.started\",\"thread_id\":\"019f125a-6d51-76d1-af24-6642c27f6f23\"}\n{\"type\":\"turn.started\"}\n{\"type\":\"item.completed\",\"item\":{\"id\":\"item_0\",\"type\":\"agent_message\",\"text\":\"I’ll fetch the requested HF Spaces playbook first, then I’ll read the relevant local skills for HF CLI and Gradio before touching the Space implementation.\"}}\n{\"type\":\"item.started\",\"item\":{\"id\":\"item_1\",\"type\":\"command_execution\",\"command\":\"/bin/zsh -lc 'curl -L --fail --silent https://gist.githubusercontent.com/gary149/37c955b832558837c40e1c14ff6d955d/raw/ad35807f8466378afd04d7653d53683a847b96c4/hf-spaces-agent-quickstart-compact.md'\",\"aggregated_output\":\"\",\"exit_code\":null,\"status\":\"in_progress\"}}\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_14","type":"agent_message","text":"The workspace is just the saved prompt and turn log, so I’m starting from a clean Space implementation. I’m going to gather model-card metadata and paper links now, then decide whether a live GPU demo is feasible or whether a research-backed visual simulator will showcase the collection better."}} -{"type":"item.started","item":{"id":"item_15","type":"mcp_tool_call","server":"codex_apps","tool":"hugging_face.hub_repo_details","arguments":{"repo_type":"model","include_readme":true,"repo_ids":["deepseek-ai/dspark_qwen3_4b_block7","deepseek-ai/dspark_qwen3_8b_block7","deepseek-ai/dspark_qwen3_14b_block7","deepseek-ai/dspark_gemma4_12b_block7","deepseek-ai/dflash_qwen3_4b_block7","deepseek-ai/dflash_qwen3_8b_block7","deepseek-ai/dflash_qwen3_14b_block7","deepseek-ai/dflash_gemma4_12b_block7","deepseek-ai/eagle3_qwen3_4b_ttt7","deepseek-ai/eagle3_qwen3_8b_ttt7","deepseek-ai/eagle3_qwen3_14b_ttt7","deepseek-ai/eagle3_gemma4_12b_ttt7"]},"result":null,"error":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_15","type":"mcp_tool_call","server":"codex_apps","tool":"hugging_face.hub_repo_details","arguments":{"repo_type":"model","include_readme":true,"repo_ids":["deepseek-ai/dspark_qwen3_4b_block7","deepseek-ai/dspark_qwen3_8b_block7","deepseek-ai/dspark_qwen3_14b_block7","deepseek-ai/dspark_gemma4_12b_block7","deepseek-ai/dflash_qwen3_4b_block7","deepseek-ai/dflash_qwen3_8b_block7","deepseek-ai/dflash_qwen3_14b_block7","deepseek-ai/dflash_gemma4_12b_block7","deepseek-ai/eagle3_qwen3_4b_ttt7","deepseek-ai/eagle3_qwen3_8b_ttt7","deepseek-ai/eagle3_qwen3_14b_ttt7","deepseek-ai/eagle3_gemma4_12b_ttt7"]},"result":{"content":[{"type":"text","text":"RuntimeException: Error calling MCP tool: [TextContent(type='text', text='MCP error -32602: Input validation error: Invalid arguments for tool hub_repo_details: [\\n {\\n \"code\": \"too_big\",\\n \"maximum\": 10,\\n \"type\": \"array\",\\n \"inclusive\": true,\\n \"exact\": false,\\n \"message\": \"Provide at most 10 repo ids\",\\n \"path\": [\\n \"repo_ids\"\\n ]\\n }\\n]', annotations=None, meta=None)]"}],"structured_content":{"error":"RuntimeException: Error calling MCP tool: [TextContent(type='text', text='MCP error -32602: Input validation error: Invalid arguments for tool hub_repo_details: [\\n {\\n \"code\": \"too_big\",\\n \"maximum\": 10,\\n \"type\": \"array\",\\n \"inclusive\": true,\\n \"exact\": false,\\n \"message\": \"Provide at most 10 repo ids\",\\n \"path\": [\\n \"repo_ids\"\\n ]\\n }\\n]', annotations=None, meta=None)]","error_code":"INVALID_ARGUMENT","error_data":{"type":"mcp_tool_execution_error","result":{"meta":null,"content":[{"type":"text","text":"MCP error -32602: Input validation error: Invalid arguments for tool hub_repo_details: [\n {\n \"code\": \"too_big\",\n \"maximum\": 10,\n \"type\": \"array\",\n \"inclusive\": true,\n \"exact\": false,\n \"message\": \"Provide at most 10 repo ids\",\n \"path\": [\n \"repo_ids\"\n ]\n }\n]","annotations":null,"meta":null}],"structuredContent":null,"isError":true},"upstream_error":null}}},"error":null,"status":"failed"}} -{"type":"item.completed","item":{"id":"item_16","type":"agent_message","text":"The repo-details tool only accepts up to 10 repos at once; I’m splitting the 12-model read into two batches."}} -{"type":"item.started","item":{"id":"item_17","type":"mcp_tool_call","server":"codex_apps","tool":"hugging_face.hub_repo_details","arguments":{"repo_type":"model","include_readme":true,"repo_ids":["deepseek-ai/dflash_qwen3_14b_block7","deepseek-ai/dflash_gemma4_12b_block7","deepseek-ai/eagle3_qwen3_4b_ttt7","deepseek-ai/eagle3_qwen3_8b_ttt7","deepseek-ai/eagle3_qwen3_14b_ttt7","deepseek-ai/eagle3_gemma4_12b_ttt7"]},"result":null,"error":null,"status":"in_progress"}} -{"type":"item.started","item":{"id":"item_18","type":"mcp_tool_call","server":"codex_apps","tool":"hugging_face.hub_repo_details","arguments":{"repo_type":"model","include_readme":true,"repo_ids":["deepseek-ai/dspark_qwen3_4b_block7","deepseek-ai/dspark_qwen3_8b_block7","deepseek-ai/dspark_qwen3_14b_block7","deepseek-ai/dspark_gemma4_12b_block7","deepseek-ai/dflash_qwen3_4b_block7","deepseek-ai/dflash_qwen3_8b_block7"]},"result":null,"error":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_17","type":"mcp_tool_call","server":"codex_apps","tool":"hugging_face.hub_repo_details","arguments":{"repo_type":"model","include_readme":true,"repo_ids":["deepseek-ai/dflash_qwen3_14b_block7","deepseek-ai/dflash_gemma4_12b_block7","deepseek-ai/eagle3_qwen3_4b_ttt7","deepseek-ai/eagle3_qwen3_8b_ttt7","deepseek-ai/eagle3_qwen3_14b_ttt7","deepseek-ai/eagle3_gemma4_12b_ttt7"]},"result":{"content":[{"type":"text","text":"# deepseek-ai/dflash_qwen3_14b_block7\n\n## Overview\n- **Author:** deepseek-ai\n- **Likes:** 2\n- **Updated:** 28 Jun, 2026\n\n## Technical Details\n- **Parameters:** 3338.5M\n- **Architecture:** qwen3\n\n## Tags\n`safetensors` `qwen3` `region:us`\n\n**Link:** [https://hf.co/deepseek-ai/dflash_qwen3_14b_block7](https://hf.co/deepseek-ai/dflash_qwen3_14b_block7)\n\n---\n\n# deepseek-ai/dflash_gemma4_12b_block7\n\n## Overview\n- **Author:** deepseek-ai\n- **Likes:** 5\n- **Updated:** 28 Jun, 2026\n\n## Technical Details\n- **Parameters:** 3296.2M\n- **Architecture:** gemma4_text\n\n## Tags\n`safetensors` `gemma4_text` `region:us`\n\n**Link:** [https://hf.co/deepseek-ai/dflash_gemma4_12b_block7](https://hf.co/deepseek-ai/dflash_gemma4_12b_block7)\n\n---\n\n# deepseek-ai/eagle3_qwen3_4b_ttt7\n\n## Overview\n- **Author:** deepseek-ai\n- **Likes:** 1\n- **Updated:** 28 Jun, 2026\n\n## Technical Details\n- **Parameters:** 927.3M\n- **Architecture:** qwen3\n\n## Tags\n`safetensors` `qwen3` `region:us`\n\n**Link:** [https://hf.co/deepseek-ai/eagle3_qwen3_4b_ttt7](https://hf.co/deepseek-ai/eagle3_qwen3_4b_ttt7)\n\n---\n\n# deepseek-ai/eagle3_qwen3_8b_ttt7\n\n## Overview\n- **Author:** deepseek-ai\n- **Likes:** 1\n- **Updated:** 28 Jun, 2026\n\n## Technical Details\n- **Parameters:** 1546.7M\n- **Architecture:** qwen3\n\n## Tags\n`safetensors` `qwen3` `region:us`\n\n**Link:** [https://hf.co/deepseek-ai/eagle3_qwen3_8b_ttt7](https://hf.co/deepseek-ai/eagle3_qwen3_8b_ttt7)\n\n---\n\n# deepseek-ai/eagle3_qwen3_14b_ttt7\n\n## Overview\n- **Author:** deepseek-ai\n- **Likes:** 1\n- **Updated:** 28 Jun, 2026\n\n## Technical Details\n- **Parameters:** 2053.9M\n- **Architecture:** qwen3\n\n## Tags\n`safetensors` `qwen3` `region:us`\n\n**Link:** [https://hf.co/deepseek-ai/eagle3_qwen3_14b_ttt7](https://hf.co/deepseek-ai/eagle3_qwen3_14b_ttt7)\n\n---\n\n# deepseek-ai/eagle3_gemma4_12b_ttt7\n\n## Overview\n- **Author:** deepseek-ai\n- **Likes:** 5\n- **Updated:** 28 Jun, 2026\n\n## Technical Details\n- **Parameters:** 2362.3M\n- **Architecture:** gemma4_unified_text\n\n## Tags\n`safetensors` `gemma4_unified_text` `region:us`\n\n**Link:** [https://hf.co/deepseek-ai/eagle3_gemma4_12b_ttt7](https://hf.co/deepseek-ai/eagle3_gemma4_12b_ttt7)"}],"structured_content":null},"error":null,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_18","type":"mcp_tool_call","server":"codex_apps","tool":"hugging_face.hub_repo_details","arguments":{"repo_type":"model","include_readme":true,"repo_ids":["deepseek-ai/dspark_qwen3_4b_block7","deepseek-ai/dspark_qwen3_8b_block7","deepseek-ai/dspark_qwen3_14b_block7","deepseek-ai/dspark_gemma4_12b_block7","deepseek-ai/dflash_qwen3_4b_block7","deepseek-ai/dflash_qwen3_8b_block7"]},"result":{"content":[{"type":"text","text":"# deepseek-ai/dspark_qwen3_4b_block7\n\n## Overview\n- **Author:** deepseek-ai\n- **Likes:** 11\n- **Updated:** 28 Jun, 2026\n\n## Technical Details\n- **Parameters:** 1393.1M\n- **Architecture:** qwen3\n\n## Tags\n`safetensors` `qwen3` `region:us`\n\n**Link:** [https://hf.co/deepseek-ai/dspark_qwen3_4b_block7](https://hf.co/deepseek-ai/dspark_qwen3_4b_block7)\n\n---\n\n# deepseek-ai/dspark_qwen3_8b_block7\n\n## Overview\n- **Author:** deepseek-ai\n- **Likes:** 2\n- **Updated:** 28 Jun, 2026\n\n## Technical Details\n- **Parameters:** 2371.1M\n- **Architecture:** qwen3\n\n## Tags\n`safetensors` `qwen3` `region:us`\n\n**Link:** [https://hf.co/deepseek-ai/dspark_qwen3_8b_block7](https://hf.co/deepseek-ai/dspark_qwen3_8b_block7)\n\n---\n\n# deepseek-ai/dspark_qwen3_14b_block7\n\n## Overview\n- **Author:** deepseek-ai\n- **Likes:** 3\n- **Updated:** 28 Jun, 2026\n\n## Technical Details\n- **Parameters:** 3416.3M\n- **Architecture:** qwen3\n\n## Tags\n`safetensors` `qwen3` `region:us`\n\n**Link:** [https://hf.co/deepseek-ai/dspark_qwen3_14b_block7](https://hf.co/deepseek-ai/dspark_qwen3_14b_block7)\n\n---\n\n# deepseek-ai/dspark_gemma4_12b_block7\n\n## Overview\n- **Author:** deepseek-ai\n- **Likes:** 8\n- **Updated:** 28 Jun, 2026\n\n## Technical Details\n- **Parameters:** 3430.4M\n- **Architecture:** gemma4_text\n\n## Tags\n`safetensors` `gemma4_text` `region:us`\n\n**Link:** [https://hf.co/deepseek-ai/dspark_gemma4_12b_block7](https://hf.co/deepseek-ai/dspark_gemma4_12b_block7)\n\n---\n\n# deepseek-ai/dflash_qwen3_4b_block7\n\n## Overview\n- **Author:** deepseek-ai\n- **Likes:** 1\n- **Updated:** 28 Jun, 2026\n\n## Technical Details\n- **Parameters:** 1315.3M\n- **Architecture:** qwen3\n\n## Tags\n`safetensors` `qwen3` `region:us`\n\n**Link:** [https://hf.co/deepseek-ai/dflash_qwen3_4b_block7](https://hf.co/deepseek-ai/dflash_qwen3_4b_block7)\n\n---\n\n# deepseek-ai/dflash_qwen3_8b_block7\n\n## Overview\n- **Author:** deepseek-ai\n- **Likes:** 1\n- **Updated:** 28 Jun, 2026\n\n## Technical Details\n- **Parameters:** 2293.3M\n- **Architecture:** qwen3\n\n## Tags\n`safetensors` `qwen3` `region:us`\n\n**Link:** [https://hf.co/deepseek-ai/dflash_qwen3_8b_block7](https://hf.co/deepseek-ai/dflash_qwen3_8b_block7)"}],"structured_content":null},"error":null,"status":"completed"}} -{"type":"item.started","item":{"id":"item_19","type":"web_search","id":"ws_0ec1c722ea10094d016a4223d066d881918f98ae7ad9a490a4","query":"","action":{"type":"other"}}} -{"type":"item.completed","item":{"id":"item_19","type":"web_search","id":"ws_0ec1c722ea10094d016a4223d066d881918f98ae7ad9a490a4","query":"site:huggingface.co/deepseek-ai/dspark_qwen3_4b_block7 DeepSpec dSpark model card","action":{"type":"search","query":"site:huggingface.co/deepseek-ai/dspark_qwen3_4b_block7 DeepSpec dSpark model card","queries":["site:huggingface.co/deepseek-ai/dspark_qwen3_4b_block7 DeepSpec dSpark model card","DeepSpec DeepSeek dSpark dFlash EAGLE3 speculative decoding paper","dSpark dFlash DeepSpec DeepSeek speculative decoding"]}}} -{"type":"item.started","item":{"id":"item_20","type":"web_search","id":"ws_0ec1c722ea10094d016a4223dc59c48191ad72e130c6aceb77","query":"","action":{"type":"other"}}} -{"type":"item.completed","item":{"id":"item_20","type":"web_search","id":"ws_0ec1c722ea10094d016a4223dc59c48191ad72e130c6aceb77","query":"DeepSeek DeepSpec GitHub DSpark paper","action":{"type":"search","query":"DeepSeek DeepSpec GitHub DSpark paper","queries":["DeepSeek DeepSpec GitHub DSpark paper","DeepSeek DSpark paper arxiv dSpark dFlash EAGLE3","DeepSpec deepseek-ai GitHub speculative decoding"]}}} -{"type":"item.started","item":{"id":"item_21","type":"web_search","id":"ws_0ec1c722ea10094d016a4223e74b1c8191a5efbf8e2dc251cd","query":"","action":{"type":"other"}}} -{"type":"item.completed","item":{"id":"item_21","type":"web_search","id":"ws_0ec1c722ea10094d016a4223e74b1c8191a5efbf8e2dc251cd","query":"https://github.com/deepseek-ai/DeepSpec","action":{"type":"other"}}} -{"type":"item.started","item":{"id":"item_22","type":"web_search","id":"ws_0ec1c722ea10094d016a4223ebada88191a7edc98aecdace8c","query":"","action":{"type":"other"}}} -{"type":"item.completed","item":{"id":"item_22","type":"web_search","id":"ws_0ec1c722ea10094d016a4223ebada88191a7edc98aecdace8c","query":"https://github.com/deepseek-ai/DeepSpec","action":{"type":"other"}}} -{"type":"item.completed","item":{"id":"item_23","type":"agent_message","text":"The DeepSpec repo README is more informative than the individual HF model pages: the pages themselves report “No model card,” while the repo README maps all 12 released checkpoints to Table 1 in the paper and explains the train/eval workflow. I’m reading the linked DSpark PDF plus the DFlash and EAGLE-3 papers now so the app’s comparisons are grounded in the actual methods."}} -{"type":"item.started","item":{"id":"item_24","type":"command_execution","command":"/bin/zsh -lc \"curl -L --fail --silent https://raw.githubusercontent.com/deepseek-ai/DeepSpec/main/DSpark_paper.pdf -o /tmp/DSpark_paper.pdf && pdftotext /tmp/DSpark_paper.pdf - | sed -n '1,220p'\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.started","item":{"id":"item_25","type":"command_execution","command":"/bin/zsh -lc \"curl -L --fail --silent https://raw.githubusercontent.com/deepseek-ai/DeepSpec/main/README.md | sed -n '1,220p'\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.started","item":{"id":"item_26","type":"command_execution","command":"/bin/zsh -lc \"curl -L --fail --silent https://raw.githubusercontent.com/deepseek-ai/DeepSpec/main/scripts/data/README.md | sed -n '1,220p'\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_24","type":"command_execution","command":"/bin/zsh -lc \"curl -L --fail --silent https://raw.githubusercontent.com/deepseek-ai/DeepSpec/main/DSpark_paper.pdf -o /tmp/DSpark_paper.pdf && pdftotext /tmp/DSpark_paper.pdf - | sed -n '1,220p'\"","aggregated_output":"zsh:1: command not found: pdftotext\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_25","type":"command_execution","command":"/bin/zsh -lc \"curl -L --fail --silent https://raw.githubusercontent.com/deepseek-ai/DeepSpec/main/README.md | sed -n '1,220p'\"","aggregated_output":"# DeepSpec\n\nDeepSpec is a full-stack codebase for training and evaluating draft models for speculative decoding. It contains data preparation utilities, draft model implementations, training code, and evaluation scripts.\n\n## Environment\n\nInstall the Python dependencies:\n\n```bash\npython -m pip install -r requirements.txt\n```\n\nData preparation additionally requires an inference engine to serve the target model when regenerating answers; see [scripts/data/README.md](./scripts/data/README.md) for details.\n\n## Workflow\n\nRun the stages in order — each stage's output feeds the next:\n\n1. **Data Preparation** — download prompts, regenerate target answers, and build the target cache.\n2. **Training** — train a draft model against the cached target outputs.\n3. **Evaluation** — measure speculative-decoding acceptance on benchmark tasks.\n\n## Data Preparation\n\nSee [scripts/data/README.md](./scripts/data/README.md) for the step-by-step data pipeline:\n\n1. download and split training data,\n2. regenerate answers,\n3. prepare the target cache (storage warning: this can be very large — roughly 38 TB for the default `Qwen/Qwen3-4B` setting).\n\n## Training\n\n```bash\nbash scripts/train/train.sh\n```\n\n`train.sh` launches `train.py`, which spawns one worker per visible GPU. Select the algorithm and target model by pointing `config_path` at one of the configs under [config/](./config/) (e.g. `config/dspark/dspark_qwen3_4b.py`); see the script header for the full list of configs, how to override `config_path` / `target_cache_dir`, and how to use `--opts` to override individual config fields. Checkpoints are written to `~/checkpoints///step_*`.\n\nHardware: the default configs and scripts assume a single node with 8 GPUs. For fewer GPUs, reduce `CUDA_VISIBLE_DEVICES`.\n\n\n## Evaluation\n\n```bash\nbash scripts/eval/eval.sh\n```\n\n`eval.sh` runs `eval.py` against a trained draft checkpoint over the speculative-decoding benchmarks in [eval_datasets/](./eval_datasets/) (gsm8k, math500, aime25, humaneval, mbpp, livecodebench, mt-bench, alpaca, arena-hard-v2). Set:\n\n- `target_name_or_path` — the target model the draft was trained against (e.g. `Qwen/Qwen3-4B`),\n- `draft_name_or_path` — the draft checkpoint, e.g. `~/checkpoints/deepspec/dspark_block8_qwen3_4b/step_latest`, or one of the Hugging Face repo IDs listed in [Released Checkpoints](#released-checkpoints).\n\n### Released Checkpoints\n\nThe checkpoints below are the ones used for Table 1 in the [paper](./DSpark_paper.pdf). Each checkpoint was trained on [open-perfectblend](https://huggingface.co/datasets/mlabonne/open-perfectblend) data generated by its corresponding target model in non-thinking mode, and is the direct output of the corresponding training configuration under [config/](./config/).\n\n\n| Algorithm | `Qwen/Qwen3-4B` | `Qwen/Qwen3-8B` | `Qwen/Qwen3-14B` | `google/gemma-4-12B-it` |\n| --- | --- | --- | --- | --- |\n| Eagle3 | [deepseek-ai/eagle3_qwen3_4b_ttt7](https://huggingface.co/deepseek-ai/eagle3_qwen3_4b_ttt7) | [deepseek-ai/eagle3_qwen3_8b_ttt7](https://huggingface.co/deepseek-ai/eagle3_qwen3_8b_ttt7) | [deepseek-ai/eagle3_qwen3_14b_ttt7](https://huggingface.co/deepseek-ai/eagle3_qwen3_14b_ttt7) | [deepseek-ai/eagle3_gemma4_12b_ttt7](https://huggingface.co/deepseek-ai/eagle3_gemma4_12b_ttt7) |\n| DFlash | [deepseek-ai/dflash_qwen3_4b_block7](https://huggingface.co/deepseek-ai/dflash_qwen3_4b_block7) | [deepseek-ai/dflash_qwen3_8b_block7](https://huggingface.co/deepseek-ai/dflash_qwen3_8b_block7) | [deepseek-ai/dflash_qwen3_14b_block7](https://huggingface.co/deepseek-ai/dflash_qwen3_14b_block7) | [deepseek-ai/dflash_gemma4_12b_block7](https://huggingface.co/deepseek-ai/dflash_gemma4_12b_block7) |\n| DSpark | [deepseek-ai/dspark_qwen3_4b_block7](https://huggingface.co/deepseek-ai/dspark_qwen3_4b_block7) | [deepseek-ai/dspark_qwen3_8b_block7](https://huggingface.co/deepseek-ai/dspark_qwen3_8b_block7) | [deepseek-ai/dspark_qwen3_14b_block7](https://huggingface.co/deepseek-ai/dspark_qwen3_14b_block7) | [deepseek-ai/dspark_gemma4_12b_block7](https://huggingface.co/deepseek-ai/dspark_gemma4_12b_block7) |\n\n> [!IMPORTANT]\n> If you cite these results in a new paper, align your setup with the training settings in this repository; otherwise, the comparison is not meaningful. For domain-specific use, fine-tune the draft model again for better results, especially if the target model is expected to run in thinking mode.\n\n## Supported Algorithms\n\nCurrently, DeepSpec includes three draft models: [DSpark](./DSpark_paper.pdf), [DFlash](https://arxiv.org/abs/2602.06036) and [Eagle3](https://arxiv.org/abs/2503.01840).\n\n## License\n\nDeepSpec is released under the [MIT License](./LICENSE). It includes code adapted\nfrom third-party projects under their own licenses; see [NOTICE](./NOTICE) for the\nfull attribution.\n\n## Acknowledgements\n\nDeepSpec builds on the ideas and code of several excellent open-source projects:\n\n- [SpecForge](https://github.com/sgl-project/SpecForge) (Apache-2.0) — the overall training framework and Eagle3 implementation; portions of the Eagle3 modeling, loss, optimizer, attention, and evaluation code are adapted from it. Adapted files carry an in-file attribution comment, and the full notice is recorded in [NOTICE](./NOTICE).\n- [DFlash](https://github.com/z-lab/dflash) (MIT) — the DFlash draft-model design and training recipe.\n- [Qwen3](https://github.com/QwenLM/Qwen3) and [Gemma](https://github.com/google-deepmind/gemma) — the target model families supported in this repo.\n\nWe thank the authors and maintainers of these projects. Contributions of new algorithms are welcome.\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_26","type":"command_execution","command":"/bin/zsh -lc \"curl -L --fail --silent https://raw.githubusercontent.com/deepseek-ai/DeepSpec/main/scripts/data/README.md | sed -n '1,220p'\"","aggregated_output":"# Data Preparation\n\nThis directory contains an example data preparation pipeline using `Qwen/Qwen3-4B` as the target model.\n\nDeepSpec trains draft models against a target model. The data pipeline does three things:\n\n1. download and split prompt data,\n2. regenerate assistant answers with the target model,\n3. precompute the target cache used by training.\n\nThe example below targets `Qwen/Qwen3-4B`, but the same pipeline applies to other models (e.g. Gemma). To switch targets, change the model name (`--model` / `model_path`) and adjust the sampling parameters (`--temperature`, `--top-p`, `--top-k` and `--min-p`) to match the recommended generation settings for that model. Output paths in the examples reference `qwen3_4b`; rename them as needed.\n\nThe wrapper script [prepare_data.sh](./prepare_data.sh) records the default settings. The individual Python scripts are also documented below for users who want to run each stage manually.\n\n## Outputs\n\nDefault outputs:\n\n```text\ntrain_datasets/perfectblend_train.jsonl\ntrain_datasets/qwen3_4b/perfectblend_train_regen.jsonl\n~/.cache/deepspec/qwen3_4b_target_cache\n```\n\nThe example scripts assume a single machine with eight visible GPUs by default. For fewer GPUs, edit `num_workers` and `CUDA_VISIBLE_DEVICES` in the shell scripts.\n\n## Step 1: Download And Split Data\n\nThe source dataset is `mlabonne/open-perfectblend`. The train split is written as JSONL, and the held-out user turns are written under `eval_datasets/`.\n\n```bash\npython scripts/data/download_and_split.py \\\n --dataset-name mlabonne/open-perfectblend \\\n --test-size 0.05 \\\n --train-output-path train_datasets/perfectblend_train.jsonl \\\n --test-output-dir eval_datasets \\\n --skip-existing\n```\n\nThis produces:\n\n```text\ntrain_datasets/perfectblend_train.jsonl\neval_datasets/perfectblend.jsonl\n```\n\n## Step 2: Regenerate Answers With Qwen3-4B\n\nThis step serves the target model and regenerates assistant answers against it. Any OpenAI-compatible inference engine works (SGLang, vLLM, TGI, etc.) — the example below uses [SGLang](https://github.com/sgl-project/sglang), but you can swap in whatever engine you prefer as long as it exposes an OpenAI-compatible `/v1` endpoint. SGLang is not in `requirements.txt`; install it separately, e.g. `pip install \"sglang[all]\"`.\n\nStart local sglang servers in one terminal:\n\n```bash\nbash scripts/data/launch_sglang_server.sh\n```\n\nBy default this starts eight `Qwen/Qwen3-4B` workers on ports `30000` to `30007` and writes logs to:\n\n```text\nlogs/sglang_qwen3_4b/\n```\n\nIn another terminal, regenerate the assistant answers:\n\n```bash\npython scripts/data/generate_train_data.py \\\n --model Qwen/Qwen3-4B \\\n --server-address \\\n 127.0.0.1:30000 \\\n 127.0.0.1:30001 \\\n 127.0.0.1:30002 \\\n 127.0.0.1:30003 \\\n 127.0.0.1:30004 \\\n 127.0.0.1:30005 \\\n 127.0.0.1:30006 \\\n 127.0.0.1:30007 \\\n --concurrency 32 \\\n --temperature 0.7 \\\n --top-p 0.8 \\\n --top-k 20 \\\n --min-p 0 \\\n --max-tokens 4096 \\\n --disable-thinking \\\n --resume \\\n --input-file-path train_datasets/perfectblend_train.jsonl \\\n --output-file-path train_datasets/qwen3_4b/perfectblend_train_regen.jsonl\n```\n\nThis produces:\n\n```text\ntrain_datasets/qwen3_4b/perfectblend_train_regen.jsonl\n```\n\nIf any samples fail, the script writes them to:\n\n```text\ntrain_datasets/qwen3_4b/perfectblend_train_regen_error.jsonl\n```\n\nStop the sglang servers before the next step if they are using the same GPUs.\n\n## Step 3: Prepare Target Cache\n\nThe training loop reads a precomputed target cache instead of repeatedly running the target model. Prepare it with:\n\n```bash\nexport CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}\nexport MASTER_ADDR=${MASTER_ADDR:-127.0.0.1}\nexport MASTER_PORT=${MASTER_PORT:-29500}\nexport RANK=${RANK:-0}\nexport WORLD_SIZE=${WORLD_SIZE:-1}\n\npython scripts/data/prepare_target_cache.py \\\n --config config/dspark/dspark_qwen3_4b.py \\\n --train-data-path train_datasets/qwen3_4b/perfectblend_train_regen.jsonl \\\n --output-dir ${HOME}/.cache/deepspec/qwen3_4b_target_cache \\\n --local-batch-size 16\n```\n\n> **Storage warning:** The target cache stores per-token hidden states for the\n> full training set and can be very large. With the default `Qwen/Qwen3-4B`\n> setting it takes roughly **38 TB** of disk. Make sure the `--output-dir`\n> filesystem has enough free space (scaling with dataset size, sequence length,\n> and target hidden dimension) before running this step. If storage is limited,\n> use a smaller training set and/or reduce `model.target_layer_ids` in the config\n> (fewer captured layers means proportionally less cache).\n\nThis produces the cache consumed by [scripts/train/train.sh](../train/train.sh):\n\n```text\n~/.cache/deepspec/qwen3_4b_target_cache\n```\n\n## Wrapper Script\n\nThe wrapper script combines the default public commands:\n\n```bash\nbash scripts/data/prepare_data.sh\n```\n\nUse the manual commands above if you want to stop and restart services between stages, change sampling parameters, use fewer GPUs, or inspect intermediate outputs.\n","exit_code":0,"status":"completed"}} -{"type":"item.started","item":{"id":"item_27","type":"web_search","id":"ws_0ec1c722ea10094d016a4223febbb88191920bb4c126167d54","query":"","action":{"type":"other"}}} -{"type":"item.completed","item":{"id":"item_27","type":"web_search","id":"ws_0ec1c722ea10094d016a4223febbb88191920bb4c126167d54","query":"arxiv 2606.19348 DSpark","action":{"type":"search","query":"arxiv 2606.19348 DSpark","queries":["arxiv 2606.19348 DSpark","2606.19348 DSpark speculative decoding","DSpark Adaptive speculative decoding arxiv 2606.19348"]}}} -{"type":"item.completed","item":{"id":"item_28","type":"agent_message","text":"`pdftotext` is not installed in this container, so I’m switching to a lightweight Python PDF extractor for the DSpark paper. I’m keeping the deployment app itself dependency-light; this is just for research."}} -{"type":"item.started","item":{"id":"item_29","type":"command_execution","command":"/bin/zsh -lc \"python3 - <<'PY'\ntry:\n import pypdf\n print('pypdf available')\nexcept Exception as e:\n print('missing', type(e).__name__, e)\nPY\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_29","type":"command_execution","command":"/bin/zsh -lc \"python3 - <<'PY'\ntry:\n import pypdf\n print('pypdf available')\nexcept Exception as e:\n print('missing', type(e).__name__, e)\nPY\"","aggregated_output":"","exit_code":0,"status":"completed"}} -{"type":"item.started","item":{"id":"item_30","type":"command_execution","command":"/bin/zsh -lc \"python3 -m pip install --quiet pypdf && python3 - <<'PY'\nfrom pypdf import PdfReader\nreader = PdfReader('/tmp/DSpark_paper.pdf')\nfor i, page in enumerate(reader.pages[:10]):\n text = page.extract_text() or ''\n print(f'\\\\n--- PAGE {i+1} ---\\\\n')\n print(text[:4500])\nPY\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_30","type":"command_execution","command":"/bin/zsh -lc \"python3 -m pip install --quiet pypdf && python3 - <<'PY'\nfrom pypdf import PdfReader\nreader = PdfReader('/tmp/DSpark_paper.pdf')\nfor i, page in enumerate(reader.pages[:10]):\n text = page.extract_text() or ''\n print(f'\\\\n--- PAGE {i+1} ---\\\\n')\n print(text[:4500])\nPY\"","aggregated_output":"error: externally-managed-environment\n\n× This environment is externally managed\n╰─> To install Python packages system-wide, try apt install\n python3-xyz, where xyz is the package you are trying to\n install.\n \n If you wish to install a non-Debian-packaged Python package,\n create a virtual environment using python3 -m venv path/to/venv.\n Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make\n sure you have python3-full installed.\n \n If you wish to install a non-Debian packaged Python application,\n it may be easiest to use pipx install xyz, which will manage a\n virtual environment for you. Make sure you have pipx installed.\n \n See /usr/share/doc/python3.12/README.venv for more information.\n\nnote: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.\nhint: See PEP 668 for the detailed specification.\n","exit_code":1,"status":"failed"}} -{"type":"item.started","item":{"id":"item_31","type":"command_execution","command":"/bin/zsh -lc \"python3 -m venv /tmp/deepspec-pdf-venv && /tmp/deepspec-pdf-venv/bin/pip install --quiet pypdf && /tmp/deepspec-pdf-venv/bin/python - <<'PY'\nfrom pypdf import PdfReader\nreader = PdfReader('/tmp/DSpark_paper.pdf')\nprint('pages', len(reader.pages))\nfor i, page in enumerate(reader.pages):\n text = page.extract_text() or ''\n print(f'\\\\n--- PAGE {i+1} ---\\\\n')\n print(text[:5000])\nPY\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_31","type":"command_execution","command":"/bin/zsh -lc \"python3 -m venv /tmp/deepspec-pdf-venv && /tmp/deepspec-pdf-venv/bin/pip install --quiet pypdf && /tmp/deepspec-pdf-venv/bin/python - <<'PY'\nfrom pypdf import PdfReader\nreader = PdfReader('/tmp/DSpark_paper.pdf')\nprint('pages', len(reader.pages))\nfor i, page in enumerate(reader.pages):\n text = page.extract_text() or ''\n print(f'\\\\n--- PAGE {i+1} ---\\\\n')\n print(text[:5000])\nPY\"","aggregated_output":"pages 33\n\n--- PAGE 1 ---\n\nDSpark: Confidence-Scheduled Speculative Decoding with\nSemi-Autoregressive Generation\nXin Cheng1,2,∗, Xingkai Yu2,∗, Chenze Shao2,∗, Jiashi Li2,∗, Yunfan Xiong2,∗\nYi Qian2, Jiaqi Zhu2, Shirong Ma2, Xiaokang Zhang2, Jiasheng Ye2, Qinyu Chen2,\nChengqi Deng2, Jiping Yu2, Damai Dai2, Zhengyan Zhang2, Yixuan Wei2, Yixuan Tan2,\nWenkai Yang2, Runxin Xu2, Yu Wu2, Zhean Xu2, Xuanyu Wang2, Muyang Chen2,\nRui Tian2, Xiao Bi2, Zhewen Hao2, Shaoyuan Chen2, Huanqi Cao2, Wentao Zhang2,\nAnyi Xu2, Huishuai Zhang1, Dongyan Zhao1, Wenfeng Liang2\n1Peking University 2DeepSeek-AI\n{chengxin, xingkai, shaochenze, js.li, yunfanxiong}@deepseek.com\nAbstract\nSpeculative decoding accelerates Large Language Model (LLM) inference by decoupling draft\ngeneration from target verification. While recent parallel drafters efficiently propose long token\nsequences in a single forward pass, they suffer from rapid acceptance decay due to a lack\nof inter-token dependencies. Furthermore, indiscriminately verifying these extended blocks\nwastes critical batch capacity on tokens with high rejection risks, severely degrading throughput\nin high-concurrency serving systems. We introduce DSpark, a speculative decoding framework\nthat unifies high-throughput parallel generation with adaptive, load-aware verification. To\nmaintain draft quality, DSpark utilizes a semi-autoregressive architecture—coupling a parallel\nbackbone with a lightweight sequential module—to introduce intra-block dependency modeling\nand mitigate suffix decay. To optimize system efficiency, DSpark employs confidence-scheduled\nverification, dynamically tailoring the verification length for each request based on estimated\nprefix survival probabilities and engine-specific throughput profiles. On offline benchmarks\nacross diverse domains, DSpark substantially improves the accepted length over state-of-the-art\nautoregressive and parallel drafters. When deployed within the DeepSeek-V4 serving system\nunder live user traffic, DSpark successfully mitigates verification waste. Compared to the\nestablished production baseline (MTP-1), DSpark accelerates per-user generation speeds by\n60%–85% at matched throughput levels. More importantly, by preventing severe throughput\ndegradation under strict interactivity constraints, it enables performance tiers that were previ-\nously unattainable, shifting the Pareto frontier of our serving system. To facilitate community\nprogress, we open-source the DSpark checkpoints alongside DeepSpec, an algorithm-driven\ntraining repository for speculative decoding.\n1. Introduction\nLarge Language Models (LLMs) generate text autoregressively: each new token requires a full\nforward pass conditioned on all preceding tokens, making inference latency proportional to the\noutput length. The resulting low GPU utilization and high user-perceived waiting time constitute\n*Equal contribution.\n\n--- PAGE 2 ---\n\na primary bottleneck in production LLM serving, particularly for latency-sensitive scenarios\nsuch as real-time conversational assistants and multi-turn agentic workflows. Speculative\ndecoding (Chen et al., 2023; Leviathan et al., 2023) offers a principled solution: a lightweight\ndraft modelproposes a block of candidate tokens, and the full-sizetarget modelverifies the entire\nblock in a single forward pass via rejection sampling, accepting the longest prefix consistent\nwith the target distribution and appending one bonus token. Because verification is parallel and\nthe acceptance rule preserves the target distribution exactly, speculative decoding accelerates\ngeneration without any quality loss.\nThe design of the draft model governs the trade-off between drafting latency and acceptance\nrate. Early drafters are autoregressive (Cheng et al., 2024; Li et al., 2024b), conditioning each\nposition on previously sampled tokens. However, their drafting latency grows linearly with\nthe block size, forcing these methods to use short blocks and shallow architectures. To break\nthis sequential bottleneck, parallel drafters (Cai et al., 2024; Chen et al., 2026; Liu et al., 2026a)\nhave emerged as a compelling alternative: all draft positions are produced in a single forward\npass, making drafting latency nearly independent of block size. This structural advantage\ntheoretically allows parallel drafters to efficiently generate substantially longer draft blocks.\nHowever, fully unlocking the potential of large parallel draft blocks introduces two critical\nbottlenecks—one in generation quality, and the other in system efficiency. First, because parallel\ndrafters predict each position independently, they cannot model inter-token dependencies\nwithin a block. This independence leads to multi-modal collisions and rapid acceptance decay\nat later positions (Gu et al., 2018; Huang et al., 2022b). Second, determining the optimal\nverification length remains a challenge. While parallel generation easily produces long draft\nblocks, indiscriminately verifying all proposed tokens degrades system throughput, particularly\nunder high-concurrency workloads (Hu et al., 2026; Liu et al., 2024c). The ideal verification\nlength varies along two axes. On the data side, structured requests like code naturally sustain\nhigher acceptance rates than open-ended chat (Abramovich et al., 2026; Xia et al., 2024). On the\nsystem side, verifying extra tokens is nearly free under light loads. Under heavy loads, however,\nverifying tokens with a high rejection risk occupies critical batch capacity that could otherwise\nserve other active requests (Liu et al., 2024b; Wu et al., 2025).\nTo address these bottlenecks, we introduceDSpark, a speculative decoding framework that\nunifies high-throughput parallel generation with adaptive, load-aware verification. At its core,\nDSpark is designed to resolve the inherent trade-offs in draft generation and verification through\ntwo complementary mechanisms.\n• First, to overcome the lack of inter-token dependencies, DSpark adopts a semi-autoregressive\narchitecture. It keeps the computationally expensive draft backbone fully parallel, append-\ning only a lightweight serial output head to inject local transition information. This design\npreserves the drafting speed of parallel models while significantly mitigating suffix decay.\n• Second, to resolve the system-level bottleneck, DSpark employs confidence-scheduled\nverification. By coupling a confidence head—which estimates per-position prefix survival\nprobabilities—with a hardware-aware scheduler, DSpark dynamically tailors the verifica-\ntion length for each request. This scheduler leverages real-time engine throughput profiles\nto route target verification budget only toward tokens with the highest expected return.\nWe extensively evaluate DSpark across both controlled offline benchmarks and production-\nscale online deployments. On controlled offline benchmarks—spanning mathematical reasoning,\ncode generation, and daily chat—DSpark consistently outperforms strong baselines. Specifically,\n2\n\n--- PAGE 3 ---\n\nacross the Qwen3-4B, 8B, and 14B target models (Yang et al., 2025), it improves the macro-average\naccepted length over the autoregressive Eagle3 (Li et al., 2026b) by 30.9%, 26.7%, and 30.0%, and\nover the parallel DFlash (Chen et al., 2026) by 16.3%, 18.4%, and 18.3%, respectively. Beyond top-\nline metrics, our fine-grained position-wise analysis reveals the distinct generation characteristics\nof different drafters, empirically demonstrating how DSpark successfully combines the high\ninitial-token capacity of parallel models with the suffix coherence of autoregressive models.\nBeyond offline evaluation, we deployed DSpark within the DeepSeek-V4 (DeepSeek-AI, 2026)\nserving system to assess its performance under live user traffic. Compared to the prior MTP-1\nproduction baseline (DeepSeek-AI, 2024), DSpark significantly broadens the system’s operational\nenvelope. Specifically, it consistently accelerates per-user generation speeds by 60%–85% (V4-\nFlash) and 57%–78% (V4-Pro) at matched aggregate throughput capacities. Furthermore, under\nstrict Service Level Agreements (SLAs) where the baseline’s capacity deteriorates severely—such\nas 120 TPS for Flash and 50 TPS for Pro—DSpark mitigates verification overhead to maintain\nrobust throughput. By overcoming this performance cliff, DSpark unlocks strict interactivity\ntiers that were previously unattainable, effectively shifting the Pareto frontier of LLM serving.\nTo foster collective advancement within the open-source community, we are making our\nartifacts publicly available. Specifically, we release the trained DSpark checkpoints for both\nthe DeepSeek-V4-Flash (preview) and DeepSeek-V4-Pro (preview) models. Furthermore, we\nopen-source DeepSpec, an algorithm-driven training repository, including Eagle3, DFlash and\nDSpark. These artifacts are intended to support further research on efficient LLM serving.\n2. Background\n2.1. Speculative Decoding\nAutoregressive language models generate one token per forward pass, making inference latency\nproportional to output length. Speculative decoding (Chen et al., 2023; Ge et al., 2022; Leviathan\net al., 2023) accelerates the inference of a target model𝑀𝑡 using a lightweight draft model𝑀𝑑. At\neach decoding cycle, the draft model proposes𝛾 candidate tokens𝑥1,... ,𝑥𝛾. The target model\nverifies all candidates in a single forward pass, accepting the longest prefix consistent with its\nown distribution.\nConcretely, at each draft position 𝑘, the target model computes its own distribution 𝑝𝑡\n𝑘\nand compares it against the draft distribution 𝑝𝑑\n𝑘. The token 𝑥𝑘 is accepted with probability\nmin(1, 𝑝𝑡\n𝑘(𝑥𝑘)/𝑝𝑑\n𝑘(𝑥𝑘)). Verification proceeds left to right: the first rejection at position𝑘 discards\nall subsequent tokens𝑥 𝑘+1,...,𝑥 𝛾, regardless of their quality.\nLet𝜏 denote the number of accepted tokens per cycle, and let𝑇draft and𝑇verify be the wall-clock\ntimes of the drafting and verification passes, respectively. The average latency per generated\ntoken is:\n𝐿=\n𝑇draft+𝑇 verify\n𝜏 . (1)\nImproving speedup therefore reduces to three levers: lowering 𝑇draft (draft faster), raising 𝜏\n(draft better), or reducing the effective𝑇 verify (verify smarter).\n2.2. Drafter Architectures\nThe design of the draft model determines how 𝑇draft and𝜏 trade off. Existing approaches fall\ninto two categories.\n3\n\n--- PAGE 4 ---\n\nAutoregressive drafters.Autoregressive drafters generate draft tokens sequentially, condition-\ning each position on previously sampled tokens (DeepSeek-AI, 2024; Li et al., 2024b,c, 2026b;\nZhang et al., 2025). This explicit dependency gives strong modeling capacity, but the drafting\ncost grows linearly with block size:𝑇 draft∝𝛾, which forces autoregressive drafters to use small\n𝛾 and shallow architectures to keep 𝑇draft low. To compensate for the short block, tree-based\nverification (Miao et al., 2024) expands candidates into a tree and verifies multiple paths via tree\nattention, but the large number of verification tokens reduces overall serving throughput.\nParallel drafters.Parallel drafters produce all 𝛾 draft tokens in a single forward pass, making\n𝑇draft nearly independent of the block size (Cai et al., 2024; Chen et al., 2026; Li et al., 2025a; Liu\net al., 2026a; Sandler et al., 2026). This allows substantially larger blocks (e.g., 𝛾=16) without\nproportionally increasing latency.\nAmong them, DFlash (Chen et al., 2026) is a state-of-the-art parallel drafter, which conditions\nits draft model on rich context features extracted from the target model (KV injection). During\nprefill, hidden states from a set of target layers{𝑙1,... ,𝑙𝑚} are concatenated and projected into\nthe draft hidden space:\n𝐻ctx =RMSNorm \u0000\n𝑊𝑐[𝐻(𝑙 1);...;𝐻 (𝑙𝑚)]\u0001, (2)\nwhere𝑊𝑐∈R 𝑑×𝑚𝑑 is a shared projection. These context features are injected into every draft\nlayer by concatenating them with the draft block representations along the sequence dimension\nof keys and values:\n𝐾𝑖 =[𝑊 𝐾\n𝑖 𝐻ctx;𝑊 𝐾\n𝑖 𝐻𝑑],𝑉 𝑖 =[𝑊 𝑉\n𝑖 𝐻ctx;𝑊 𝑉\n𝑖 𝐻𝑑]. (3)\nAll positions within a block attend bidirectionally to each other and to the injected target context.\nThe draft model shares the target model’s embedding layer and language modeling head\n(both frozen). It takes as input the embedding of an anchor token 1 followed by 𝛾 mask to-\nken embeddings, and produces logits for all mask positions in a single forward pass. Since\ndrafting requires only a single forward pass regardless of block size, DFlash can afford deeper\narchitectures and larger blocks than autoregressive drafters under the same latency budget.\n3. Architecture\nThe overview of DSpark is shown in Figure 1. Recall from Equation 1 that the per-token latency\nof speculative decoding is 𝐿=(𝑇 draft+𝑇 verify)/𝜏. Autoregressive drafters achieve high𝜏 but pay\n𝑇draft∝𝛾; parallel drafters collapse𝑇 draft to a single pass but sacrifice𝜏because each position is\npredicted independently. Meanwhile, fixed-length verification wastes𝑇verify on low-confidence\nsuffix tokens that are almost certain to be rejected. DSpark addresses these limitations with two\ncomplementary components:\n• Semi-autoregressive generation(Section 3.1). A parallel backbone handles the bulk of draft\ncomputation, which keeps𝑇draft nearly independent of𝛾. A lightweight sequential block then\ninjects dependency among draft tokens, improving𝜏at minimal additional latency.\n• Confidence-scheduled verification(Section 3.2). A confidence head estimates per-position\nacceptance probabilities, and a hardware-aware scheduler uses these estimates to prune\nlow-confidence suffix tokens, cutting unnecessary verification compute.\n1We use the termsanchor tokenandbonus tokeninterchangeably in this paper to denote the final token generated\nby the target model in the previous decoding round.\n4\n\n--- PAGE 5 ---\n\nD Mask Mask Mask\n2\nE F G H\nLogits Logits Logits Logits\nC1 C2 C3 C4\nParallel Block \nA B C\nD\n1 Target Model\nE F G\nHardware-Aware\nPrefix Scheduler\nH\nKeep Drop\nSequential Block\nD\nE F G*\nE F G\nnext round\n3 Target Model\nFigure 1| The DSpark architecture and decoding cycle.Given prompt tokens ABC , the\ntarget model executes one step to generate the next token D , which serves as the anchor for\nthe drafting phase. Using D as the input, DSpark employs a heavy parallel backbone and a\nlightweight sequential head to generate draft tokens EFGH along with their corresponding\nconfidence scores 𝑐1–𝑐4 . The Hardware-Aware Prefix Scheduler then evaluates these scores to\nretain the prefix EFG and drop the low-confidence token H . Finally, the target model verifies\nthe scheduled prefix in parallel. As illustrated, E and F are accepted while G is rejected,\nprompting the model to generate a corrected token G∗ to complete the current round.\nIn combination, the two components let DSpark draft better and verify smarter. We detail each\nbelow.\n3.1. Semi-Autoregressive Generation\nA parallel drafter produces all 𝛾 draft logits in one forward pass, so each prediction cannot\ncondition on tokens sampled elsewhere in the block. When the context admits multiple plausible\ncontinuations, e.g., “of course” and “no problem”, a parallel drafter may produce incoherent\ncombinations such as “of problem” or “no course”, because each position marginalizes over\nall possible predecessors rather than conditioning on the one actually sampled (Gu et al., 2018;\nHuang et al., 2022a). Acceptance rate thus decays rapidly along the block, wasting both draft\nand verification compute. We therefore adopt asemi-autoregressivestructure that splits draft\ngeneration into two stages:\nParallel stage.A parallel backbone (in our instantiation, DFlash (Chen et al., 2026)) runs a\nsingle forward pass over the entire block, producing hidden states ℎ1,... ,ℎ𝛾 and base logits\n𝑈1,... ,𝑈𝛾. We make only a minor modification to the original DFlash backbone: instead of\nfeeding an anchor token plus𝛾 mask tokens and predicting only the mask positions, we treat\n5\n\n--- PAGE 6 ---\n\nthe anchor itself as the first prediction position, so𝛾 input tokens (anchor+𝛾− 1 masks) yield𝛾\ndraft logits. This reduces draft computation while maintaining similar draft quality.\nSequential stage.The sequential stage supplements the base logits with a prefix-dependent\ntransition bias 𝐵𝑘(𝑥0,𝑥<𝑘,𝑥𝑘), allowing each draft position to condition on previously sampled\ntokens within the block. Rather than defining a globally normalized energy model, the sequential\nstage induces a causal block distribution through an autoregressive factorization:\n𝑃(𝑋|𝑥 0)=\n𝛾Ö\n𝑘=1\n𝑝𝑘(𝑥𝑘|𝑥 0,𝑥 <𝑘),𝑝 𝑘(𝑣|𝑥 0,𝑥 <𝑘)= exp(𝑈𝑘(𝑣)+𝐵 𝑘(𝑥0,𝑥 <𝑘,𝑣))Í\n𝑢∈V exp(𝑈𝑘(𝑢)+𝐵 𝑘(𝑥0,𝑥 <𝑘,𝑢)) . (4)\nHere,𝑥0 denotes the anchor token from the previous verification cycle,𝑈𝑘 is the base logit vector\nproduced by the parallel backbone at position𝑘, andV is the vocabulary. At inference time, the\nsequential block samples left to right according to 𝑝𝑘(·|𝑥 0,𝑥<𝑘). Because this sampling process\nis inherently sequential, the block must be computationally lightweight (𝑇sequential≪𝑇 parallel)\nso that the overall draft latency remains dominated by the parallel stage. We describe two\ninstantiations of the sequential block below.\n• Markov head.The simplest instantiation restricts 𝐵𝑘 to depend only on the immediately\npreceding token, reducing it to a first-order transition 𝐵(𝑥𝑘−1,𝑥𝑘). In principle this is a\nfull𝑉×𝑉 matrix 𝐵; we approximate it with a low-rank factorization 𝐵=𝑊 1𝑊2, where\n𝑊1∈R 𝑉×𝑟 and𝑊2∈R 𝑟×𝑉 . Given the preceding token 𝑥𝑘−1, the transition bias for position\n𝑘is:\n𝐵(𝑥𝑘−1,·)=𝑊 1[𝑥𝑘−1]𝑊 2 ∈R 𝑉, (5)\nwhere𝑊1 serves as an embedding lookup table and𝑊2 as a logit projection. The low-rank\nfactorization (𝑟=256 by default) keeps both storage and per-step compute small, making\nthe sequential loop efficient even for large vocabularies. Returning to the earlier example:\nonce position 1 samples “of”, the Markov head boosts “course” and suppresses “problem”\nat position 2, which mitigates the cross-mode collision.\n• RNN head.The Markov head is memoryless beyond one step—position 𝑘 cannot access\ntokens before 𝑥𝑘−1. The RNN head relaxes this by maintaining a recurrent state 𝑠𝑘 that\naccumulates the full prefix history within a block. At each step, the module concatenates\nthe current state𝑠𝑘−1∈R 𝑟, the previous token embedding𝑊1[𝑥𝑘−1]∈R 𝑟, and the backbone\nhiddenℎ𝑘∈R 𝑑 into an input vector 𝑧𝑘 =[𝑠 𝑘−1; 𝑊1[𝑥𝑘−1]; ℎ𝑘]∈R 2𝑟+𝑑, then applies a single\ngated update:\n𝑠𝑘 =𝜎(𝑊 𝑔𝑧𝑘)⊙𝑠 𝑘−1 + \u00001−𝜎(𝑊 𝑔𝑧𝑘)\u0001⊙tanh(𝑊 𝑐𝑧𝑘),\n𝐵𝑘(𝑥<𝑘,·)=𝑊 ⊤\n2 tanh(𝑊𝑜𝑧𝑘), (6)\nwhere𝑊𝑔,𝑊𝑐,𝑊𝑜∈R(2𝑟+𝑑)×𝑟 are jointly parameterized by a single linear projection that is\nsplit into gate, candidate, and output components. The state𝑠 0 is initialized to zero.\n3.2. Confidence-Scheduled Verification\nThe semi-autoregressive architecture enables DSpark to generate large draft blocks efficiently.\nHowever, producing more draft tokens does not automatically translate to higher end-to-end\nspeedups. Indiscriminately verifying the full draft block can actually degrade overall system\nthroughput, especially in high-concurrency scenarios (Hu et al., 2026; Liu et al., 2024c).\n6\n\n--- PAGE 7 ---\n\nThis performance bottleneck stems from two interacting factors. First, on the data side,\ndraft acceptance rates inherently vary across domains: structured text like code naturally yields\nhigh acceptance, whereas open-ended chat has significantly lower acceptance (Abramovich\net al., 2026; Xia et al., 2024). Second, on the system side, the actual cost of verifying an extra\ntoken depends strictly on the engine load. Under light system load, an extra verification\nincurs minimal penalty even if rejected. However, under high-concurrency deployments, every\nunnecessary verification occupies target model batch capacity that could otherwise serve other\nactive requests (Liu et al., 2024b; Wu et al., 2025).\nTherefore, fully unlocking the potential of large draft blocks requires a unified mechanism\nthat routes target model compute only toward tokens with a positive expected return. DSpark\nachieves this by coupling aconfidence head(Section 3.2.1) that predicts prefix survival proba-\nbilities, with ahardware-aware prefix scheduler(Section 3.2.2) that dynamically determines the\noptimal verification lengths based on current system load.\n3.2.1. Confidence Head\nDrawing inspiration from Huang et al. (2024); Wang et al. (2026), the confidence head outputs a\nscalar estimate𝑐𝑘∈( 0, 1) for each draft position𝑘. Crucially,𝑐𝑘 models theconditionalprobability\nthat the draft token at position𝑘 will survive target verification, given that all preceding tokens in\nthe block have been accepted. The architecture features a lightweight linear projection followed\nby a sigmoid function:\n𝑐𝑘 =𝜎\n\u0000\n𝑤⊤[ℎ𝑘;𝑊 1[𝑥𝑘−1]]\u0001, (7)\nwhereℎ𝑘 is the hidden state of the backbone and𝑊1[𝑥𝑘−1] is the Markov Embedding from the\nprevious draft token. We supervise𝑐𝑘 using the analytical acceptance rate per-step𝑐∗\n𝑘. This rate\nis determined by the total variation distance between the draft distribution 𝑝𝑑\n𝑘 and the target\ndistribution𝑝 𝑡\n𝑘:\n𝑐∗\n𝑘 =1− 1\n2∥𝑝𝑑\n𝑘−𝑝 𝑡\n𝑘∥1. (8)\nPost-hoc Calibration.Unlike threshold-based verification heuristics (Huang et al., 2024; Li\net al., 2024b; Zhang et al., 2026b), which only require confidence scores to correctly rank draft\ntoken qualities, our hardware-aware scheduling approach (detailed in Section 3.2.2) precisely\nrequires the absolute magnitudes of the cumulative acceptance probabilities to compute the\nexpected acceptance length𝜏. Because neural confidence estimates are often overconfident (Guo\net al., 2017; Ovadia et al., 2019), using the raw confidence scores directly would distort the\nthroughput estimation, leading to suboptimal scheduling.\nTo address this, we introduceSequential Temperature Scaling (STS). Because each𝑐𝑖 models\na conditional probability, the chain rule dictates that the joint probability of a draft prefix being\naccepted factorizes into the cumulative product Î\n𝑖⩽𝑘𝑐𝑖. Using a held-out validation set, STS\ncalibrates this joint probability consecutively from left to right. Specifically, at each position\n𝑘∈{ 1,... ,𝛾}, we perform a simple 1D grid search to find the optimal temperature scalar that\nminimizes the Expected Calibration Error (ECE) (Naeini et al., 2015) of the cumulative product,\nkeeping the already-calibrated scores of all preceding positions fixed. Crucially, temperature\nscaling is an order-preserving transformation: it rectifies the predicted probabilities to match\nempirical acceptance rates without disrupting the relative draft token rankings learned by the\nconfidence head.\n3.2.2. Hardware-Aware Prefix Scheduler\n7\n\n--- PAGE 8 ---\n\nAlgorithm 1Hardware-Aware Prefix Scheduler\nRequire: Active requests𝑟∈{ 1,... ,𝑅}; confidence sequence𝑐𝑟,1,... ,𝑐𝑟,𝛾 per request; profiled\nstep curve SPS(𝐵)\nEnsure:Selected per-request prefix lengthsℓ ∗\n1,...,ℓ ∗\n𝑅\n1:for𝑟=1to𝑅do\n2:Compute prefix survival probabilities:𝑎 𝑟,𝑗← Î\n𝑖⩽𝑗 𝑐𝑟,𝑖 for𝑗=1,...,𝛾\n3:end for\n4:Construct candidate spaceE←{(𝑟,𝑗)|𝑎 𝑟,𝑗 >0}and sort descending by𝑎 𝑟,𝑗\n5:Initialize states:ℓ 𝑟←0 for all𝑟; Batch size𝐵←𝑅; Expected accepts𝜏 ∗←𝑅\n6:Initialize tracking:Θ best←𝑅·SPS(𝑅); Selected lengthsℓ ∗\n𝑟←0 for all𝑟\n7:foreach(𝑟,𝑗)∈Ein sorted orderdo\n8:ℓ 𝑟←𝑗;𝐵←𝐵+1;𝜏 ∗←𝜏∗+𝑎 𝑟,𝑗\n9:Current throughputΘ←𝜏 ∗·SPS(𝐵)\n10:ifΘ>Θ best then\n11:Θ best←Θ; Update selected lengthsℓ ∗\n𝑟←ℓ 𝑟\n12:else\n13:break\n14:end if\n15:end for\n16:return(ℓ ∗\n1,...,ℓ ∗\n𝑅)achievingΘ best\nPrior methods (Huang et al., 2024; Li et al., 2024b) typically apply a static threshold to\nconfidence scores to determine verification length. While effective under isolated, single-request\nassumptions, static thresholds can be suboptimal in high-concurrency production systems,\nwhere the utility of verifying a draft token depends heavily on the current system load.\nTo address this, we formulate verification length selection as a global throughput maximiza-\ntion problem (Algorithm 1). Consider a batch of 𝑅 active requests. For request𝑟, let𝑐𝑟,1,... ,𝑐𝑟,𝛾\nbe the per-position confidence estimates, and letℓ𝑟∈{ 0,... ,𝛾} denote the scheduled verification\nlength. Because speculative decoding dynamically accepts draft tokens only as a continuous\nprefix, the survival probability of a token at position𝑗is the cumulative product𝑎 𝑟,𝑗 = Î\n𝑖⩽𝑗 𝑐𝑟,𝑖.\nIn a single verification step, the total batch size (measured in tokens) sent to the target\nmodel is 𝐵= Í𝑅\n𝑟=1(1+ℓ𝑟), and the expected number of successfully accepted tokens is 𝜏=Í𝑅\n𝑟=1\n\u00001+ Íℓ𝑟\n𝑗=1𝑎𝑟,𝑗\n\u0001. Let SPS(𝐵) denote the engine throughput, measured in steps per second, for\na given forward-pass batch size𝐵. Crucially, this capacity curve is profiled once during engine\ninitialization and stored as a lightweight cost table. Our scheduler then aims to maximize the\nexpected system-wide token throughput Θ=𝜏·SPS(𝐵) by dynamically selecting verification\nlengthsℓ 1,...,ℓ 𝑅.\nAlthough finding the global maximum of Θ appears to be a combinatorial search, the\nobjective structure allows for an efficient greedy solution. Because 𝑎𝑟,𝑗 is monotonically non-\nincreasing with respect to 𝑗 (i.e.,𝑎𝑟,𝑗≤𝑎 𝑟,𝑗−1 ), the marginal gain in expected accepted tokens for\nextending request𝑟’s verification length from 𝑗− 1 to 𝑗 is exactly𝑎𝑟,𝑗 . This monotonicity ensures\nthat sorting candidate tokens globally by 𝑎𝑟,𝑗 naturally respects intra-block prefix dependencies.\nConsequently, if the total verification batch size𝐵 were fixed, the optimal allocation{ℓ𝑟} would\nbe determined by greedily selecting the draft tokens with the highest survival probabilities from\nthe global pool of all{𝑎 𝑟,𝑗}.\nBuilding on this insight, the optimization can be evaluated along this greedy admission path.\n8\n\n--- PAGE 9 ---\n\nWe first globally sort all valid prefix extensions in descending order of survival probability. To\ndynamically determine the optimal target batch size 𝐵, we incrementally admit tokens from this\nsorted pool, updating the expected throughput Θ via an𝑂(1) lookup from the pre-profiled cost\ntable.\nLossless speculative decoding strictly requires thenon-anticipating property: admission de-\ncisions must not depend on future candidate tokens (Chen et al., 2023; Leviathan et al., 2023).\nBecause our confidence head relies on the Markov feature of the previously sampled token,\ncomputing the next survival probability𝑎𝑟,𝑘+1 explicitly requires the instantiated candidate𝑥𝑟,𝑘.\nA retrospective global search would thus inadvertently leak𝑥𝑟,𝑘 into the admission decision for\nstep𝑘, introducing selection bias (we provide a concrete counterexample demonstrating this\ntheoretical violation in Appendix A).\nTo enforce strict causality, the scheduler (Algorithm 1) employs an early-stopping mech-\nanism. By breaking the greedy search immediately when the throughput drops ( Θ≤Θ best),\nthe truncation decision relies solely on the prefix processed up to that exact step. This isolates\nthe admission event from future tokens, ensuring exact target-distribution recovery. Note that\nthis stepwise early-stopping yields the global maximum throughput if and only if the objective\nΘ is unimodal, which implicitly assumes a smoothly decaying hardware capacity curve. We\naddress the engineering adaptations required for real-world, non-smooth SPS characteristics\nand asynchronous system pipelines in Section 5.2.\n3.3. Training\nDuring training, we randomly sample multiple anchor positions from each target sequence\nto form 𝛾-token blocks as training data. The target model is frozen throughout training; the\ndraft model shares its embedding layer and language modeling head and keeps them frozen,\nupdating only the backbone drafter, sequential block, and confidence head.\nThe training objective consists of three terms: a cross-entropy loss Lce, a distribution-\nmatching loss Ltv, and a confidence loss Lconf. All three are position-weighted by 𝑤𝑘 =\nexp(−(𝑘− 1)/𝛾) (Chen et al., 2026), which emphasizes earlier block positions that contribute\nmore to the expected acceptance length under prefix-based verification. The cross-entropy loss\nLce trains the drafter to predict the correct next token:\nLce =−\n𝛾∑︁\n𝑘=1\n𝑤𝑘 log𝑝 𝑑\n𝑘(𝑥∗\n𝑘), (9)\nwhere𝑥∗\n𝑘 is the ground-truth token and 𝑝𝑑\n𝑘 is the draft distribution. The distribution-matching\nlossL tv penalizes the total variation distance between the draft and target distributions:\nLtv =\n𝛾∑︁\n𝑘=1\n𝑤𝑘∥𝑝𝑑\n𝑘−𝑝 𝑡\n𝑘∥1. (10)\nSince the total variation distance is a direct proxy for the acceptance rate: the per-step acceptance\nprobability equals 1− 1\n2∥𝑝𝑑−𝑝 𝑡∥1 (Leviathan et al., 2023), minimizingLtv directly maximizes\nthe expected acceptance rate. The confidence lossLconf is a binary cross-entropy that trains the\nconfidence head to predict the soft acceptance label𝑐∗\n𝑘 from Equation 8:\nLconf =−\n𝛾∑︁\n𝑘=1\n𝑤𝑘\n\u0002\n𝑐∗\n𝑘 log𝑐 𝑘+(1−𝑐 ∗\n𝑘)log(1−𝑐 𝑘)\n\u0003\n. (11)\n9\n\n--- PAGE 10 ---\n\nThe overall objective is a weighted combination of the three terms (with default weights𝛼ce = 0.1,\n𝛼tv =0.9,𝛼 conf =1.0):\nL=𝛼 ceLce+𝛼 tvLtv+𝛼 confLconf (12)\n4. Experiments\nIn this section, we validate the draft quality of DSpark using offline benchmarks and report\nthe effectiveness of confidence scheduler under online production traffic in Section 5. The\nexperimental setup is described in Section 4.1, main results in Section 4.2, and additional\nanalyses are included in Section 4.3.\n4.1. Experimental Setup\nTarget and draft models.We evaluate DSpark on four target models spanning different\nscales and model families: Qwen3-{4B, 8B, 14B} (Yang et al., 2025), and Gemma4-12B (Google\nDeepMind, 2026). For draft models, we compare DSpark with two representative drafters:\nDFlash (Chen et al., 2026), a state-of-the-art parallel drafter, and Eagle3 (Li et al., 2026b), an\nautoregressive drafter based on Training-Time Test (TTT). For fair comparison, we retrain all\ndrafters in the same training framework and on the same data. We align Eagle3’s TTT horizon (7)\nwith the block size (7) used by DFlash and DSpark, and we use the same target-model feature\nlayers for all drafters. For the number of draft model layers, we set 1 for Eagle3 and 5 for DSpark\nand DFlash (Chen et al., 2026). Unless otherwise stated, DSpark denotes the Markov-head\nvariant; we study the RNN-head variant in Section 4.3.2.\nTraining data.We use Open-PerfectBlend 2, an open-sourced version of PerfectBlend (Xu et al.,\n2024) consisting of 1.3 million samples. It is a general-purpose instruction dataset containing\nchat (17.6%), math (39.4%), code (38.9%), and instruction-following data (4.1%). We only\nuse the prompts from Open-PerfectBlend; responses are regenerated by each target model\nwith recommended sampling parameters. Each drafter is trained for 10 epochs to ensure full\nconvergence. For data generation and evaluation, we adopt the non-thinking mode.\nEvaluation protocol.We evaluate the performance of different algorithms on three domains:\n1. Mathematical Reasoning, including GSM8K (Cobbe et al., 2021), MATH500 (Lightman\net al., 2024) and AIME25 (Zhang and Math-AI, 2025).\n2. Code Generation, including MBPP (Austin et al., 2021b), HumanEval (Chen et al., 2021)\nand Live-CodeBench (Jain et al., 2025).\n3. Daily Chat, including MT-Bench (Zheng et al., 2023), Alpaca (Taori et al., 2023) and\nArena-Hard (Li et al., 2024a, 2025b).\nFor all benchmarks, we use standard speculative decoding (Chen et al., 2023; Leviathan et al.,\n2023) with the sampling temperature set to 1.0. We report the accepted length (𝜏) per decoding\nround3. For all drafters, we use chain-based drafting.\n2https://huggingface.co/datasets/mlabonne/open-perfectblend\n3For clarity, unless otherwise stated, all reported metrics for accepted length and acceptance rate include the\ntarget-generated bonus token.\n10\n\n--- PAGE 11 ---\n\nTable 1| Main speculative decoding results.We report accepted length ( 𝜏) per decoding\nround (higher is better) for different target models and domains.Boldmarks the best results.\nTarget Drafter Math Code Chat\nGSM8K MATH AIME25 MBPP HumanEval LCB MT-Bench Alpaca Arena-Hard\nQwen3-4B\nEagle3 5.14 4.62 3.92 3.69 4.16 3.77 2.39 2.26 2.55\nDFlash 5.40 4.85 4.15 4.40 4.74 4.18 3.07 2.96 2.83\nDSpark6.11 5.70 4.89 5.13 5.38 4.86 3.64 3.54 3.29\nQwen3-8B\nEagle3 5.30 4.77 3.91 3.96 4.33 4.17 2.66 2.54 2.54\nDFlash 5.33 4.91 4.07 4.36 4.64 4.39 3.11 2.98 2.81\nDSpark6.17 5.78 5.01 5.16 5.52 5.17 3.72 3.58 3.21\nQwen3-14B\nEagle3 5.24 4.60 3.71 3.81 4.14 4.01 2.62 2.47 2.48\nDFlash 5.41 4.84 3.98 4.44 4.59 4.33 3.10 2.94 2.72\nDSpark6.21 5.74 4.94 5.26 5.43 5.02 3.70 3.58 3.13\nGemma4-12B\nEagle3 5.87 5.46 4.83 4.72 5.37 4.16 3.19 3.06 2.72\nDFlash 5.45 5.04 4.22 4.39 4.95 3.70 2.98 2.84 2.59\nDSpark6.05 5.78 5.12 5.11 5.64 4.51 3.49 3.35 2.92\n4.2. Experimental Results\nTo isolate the raw draft quality from system-level scheduling policies, our offline evaluation\ndisables the confidence scheduler, forcing all drafters to propose a fixed block of tokens. The\nmain results, measured by the average accepted length (𝜏) per round, are reported in Table 1.\nDSpark consistently outperforms both the autoregressive baseline (Eagle3) and the parallel\nbaseline (DFlash) across all evaluated target models and benchmark domains. Specifically,\nacross the Qwen3-4B, 8B, and 14B models, DSpark improves the macro-average accepted length\nover Eagle3 by 30.9%, 26.7%, and 30.0%, respectively. Similarly, compared to DFlash, DSpark\nyields relative improvements of 16.3%, 18.4%, and 18.3% across the three scales. Crucially, this\nadvantage generalizes across model families, as demonstrated by the consistent performance\ngains on the Gemma4-12B target.\nBeyond the average improvements, Table 1 reveals a strong domain effect: the accepted\nlength is naturally higher on structured tasks (e.g., 5.57 on math and 5.12 on code for Qwen3-4B)\nthan on open-ended chat (3.49). This inherent variance in data predictability means a static\nverification length often wastes compute on trailing tokens that are highly likely to be rejected.\nThis directly motivates our confidence-scheduled verification, which dynamically prunes the\ndraft block based on expected acceptance.\n4.3. Experimental Analysis\n4.3.1. Why Can Parallel Generation Outperform Autoregression?\nTable 1 presents a counter-intuitive observation: the parallel drafter (DFlash) and the semi-\nautoregressive drafter (DSpark) often yield longer accepted lengths than the fully autoregressive\ndrafter (Eagle3). This finding contrasts with the standard expectation that step-by-step autore-\ngression produces higher-quality sequences than parallel models (Israel et al., 2026; Ren et al.,\n2020; Zheng et al., 2025).\nTo analyze this behavior, we examine performance beyond the macro-level accepted length.\nUsing the Qwen3-4B target model and the benchmark sets described in Section 4.1, we introduce\nposition-wise conditional acceptancetracked during actual speculative decoding rollouts. Specifi-\ncally, for a given draft position𝑘, the evaluation denominator counts only the instances where\n11\n\n--- PAGE 12 ---\n\n1 2 3 4 5 6 7\n0.75\n0.80\n0.85\n0.90\n0.95Cond. acceptance\nMath\n1 2 3 4 5 6 7\n0.70\n0.75\n0.80\n0.85\n0.90\n0.95\nCode\n1 2 3 4 5 6 7\n0.50\n0.55\n0.60\n0.65\n0.70\n0.75\n0.80\nChat\nDraft position\nDFlash Eagle3 DSpark\nFigure 2| Position-wise conditional acceptance.We report the empirical conditional acceptance\nrate for each draft position, averaged across benchmarks within each domain using the Qwen3-\n4B target model. Unlike standard prefix survival, this metric isolates the baseline predictive\nquality at position𝑘 by removing the penalty of previous rejections. Notice that the autoregres-\nsive drafter (Eagle3) remains stable or trends upward, while the parallel drafter (DFlash) suffers\nsuffix decay.\nthe target model successfully verifies and accepts all preceding draft tokens from 1 to𝑘− 1. The\nmetric then calculates the proportion of these valid instances where the token at position𝑘 is\nalso accepted. This approach ensures that the evaluation of position 𝑘 is not penalized by earlier\nprefix errors, revealing the underlying predictive quality at each specific step. Figure 2 details\nthese measurements, demonstrating clear behavioral differences across the architectures.\nThe Capacity Advantage at Position 1.At the first draft position, both architectures predict\nthe next token based solely on the target context. The performance divergence here stems\nstrictly from architectural capacity: autoregressive models like Eagle3 are constrained to shallow\nnetworks due to their 𝑂(𝛾) latency, whereas 𝑂(1) parallel drafters can afford much deeper\nnetworks. This structural gap yields a substantial accuracy margin at position 1, with DFlash\nstarting noticeably higher than Eagle3 (e.g., 0.88 vs. 0.81 on Math, and 0.72 vs. 0.53 on Chat).\nBecause speculative decoding operates as a strict prefix-matching survival process, the first\ntoken carries the highest leverage—a rejection here immediately invalidates the entire block.\nConsequently, this initial capacity advantage disproportionately boosts the final accepted length,\nexplaining why parallel drafters ultimately outperform autoregressive ones globally despite\nrapid acceptance decay at later positions.\nThe Limitation of Independence at Later Positions.Examining the tail of the curves (positions\n2 through 7) exposes the inherent limitation of independent parallel generation. As earlier\ntokens lock in a specific semantic path, subsequent tokens naturally become more predictable.\nAutoregressive models like Eagle3 effectively leverage this conditional certainty, maintaining or\neven increasing conditional acceptance deeper into the block (e.g., from 0.53 to 0.74 on Chat). In\ncontrast, DFlash suffers from rapid acceptance decay, dropping from 0.87 to 0.78 on Code and\n0.72 to 0.63 on Chat. Because each parallel position marginalizes over all possible prior tokens\nrather than conditioning on an exact sampled prefix, the model frequently proposes inconsistent\nsuffix combinations—a mode known as multi-modal collision (Gu et al., 2018; Stern et al., 2018).\nMitigating Suffix Decay with Semi-Autoregression.The preceding analysis highlights a clear\narchitectural objective: combining the high capacity of a parallel backbone for the initial token\nwith the dependency modeling of an autoregressive model for subsequent tokens. This directly\n12\n\n--- PAGE 13 ---\n\nGSM8K MATH500 AIME25 MBPP\nHumanEval\nLiveCodeBench\nMT-Bench Alpaca\nArena-Hard v2\n2\n3\n4\n5\n6\nAccepted Length\nMethod DSpark DFlash\nDepth 1L 2L 5L\nFigure 3| Effect of drafter depth.With proposal length fixed, DSpark’s performance improves\nas drafter layers are added. Notably, a shallow 2-layer DSpark outperforms a deeper 5-layer\nDFlash baseline, highlighting the parameter efficiency of sequential modeling.\n4 8 12 16\n4\n5\n6\n7Accepted Length\nMath\n+16%\n+23%\n+30%\n4 8 12 16\n3.2\n4.0\n4.8\n5.6\n6.4\nCode\n+15%\n+21%\n+26%\n4 8 12 16\n2.7\n3.0\n3.3\n3.6\nChat\n+18%\n+21% +22%\n4 8 12 16\n188\n190\n192\n194Latency (ms)\nLatency\n+0.6%\n+0.9%\n+1.3%\nDraft Length\nDSpark (markov) DSpark (rnn) DFlash\nFigure 4| Effect of proposal length and latency overhead.DSpark consistently outperforms\nDFlash across various block sizes (left three panels). The rightmost panel demonstrates that the\nsequential head introduces minimal latency overhead during serving.\nmotivates DSpark’s semi-autoregressive design. As shown in Figure 2, DSpark inherits the high\ninitial acceptance of the deep parallel drafter (e.g., starting at 0.93 on Math). Simultaneously, its\nlightweight sequential head mitigates the rapid acceptance decay typical of parallel generation.\nBy resolving this trade-off, DSpark maintains a high and stable conditional acceptance rate\nthroughout the entire draft block.\n4.3.2. A Little Autoregression Goes a Long Way\nBuilding on the insights from Section 4.3.1, we explore the architectural design space of DSpark\nalong two dimensions: drafter depth (number of transformer layers) and proposal length (block\nsize 𝛾). Unless otherwise stated, all experiments in this section use Qwen3-4B as the target\nmodel and follow the evaluation protocol detailed in Section 4.1.\nDrafter Depth.Increasing the number of transformer layers naturally expands a draft model’s\npredictive capacity. To isolate this effect, we fix the block size to 7 and vary the number of\nDSpark layers from 1 to 5, comparing it against a 5-layer DFlash baseline. Figure 3 aggregates the\naccepted lengths across the math, code, and chat domains. As expected, DSpark’s performance\nimproves monotonically with depth, with the steepest marginal gain occurring from one to two\nlayers. Notably, a 2-layer DSpark outperforms the 5-layer DFlash baseline across all domains.\n13\n\n--- PAGE 14 ---\n\nThis demonstrates that injecting local auto-regression via a lightweight sequential head offers a\nhighly favorable accuracy-parameter trade-off, achieving better sequence coherence than simply\nstacking deeper parallel layers.\nProposal Length.Next, we fix the drafter depth to 5 layers and scale the draft length (proposal\nlength 𝛾 plus one anchor token) across{4, 8, 12, 16} to evaluate performance on longer draft\nblocks. For DSpark, we evaluate both the default Markov head and the RNN head. The first\nthree panels of Figure 4 show that DSpark consistently outperforms DFlash at every proposal\nlength. More importantly, the performance gap steadily widens as𝛾 increases. Because pure\nparallel generation (DFlash) suffers from rapid acceptance decay (Figure 2), its marginal utility\ndiminishes for long blocks. DSpark mitigates this decay, causing its relative gain over DFlash\nto grow. For instance, at𝛾= 7, DSpark improves the accepted length by 16% on math, 15% on\ncode, and 18% on chat; at𝛾= 15, these gains expand to 30%, 26%, and 22%, respectively. Also,\nRNN head provides only marginal additional gains over the Markov head, mainly at longer\nproposal lengths. Given its higher implementation complexity and less favorable deployment\nproperties, we use the Markov head as the default.\nLatency Overhead.We quantify the overhead of the sequential generation loop in DSpark.\nThe rightmost panel of Figure 4 reports the per-round engine latency—comprising one target\nverification pass, the parallel draft block forward, and the serial sampling loop—measured\nat a batch size of 128. To prevent sequence-length bias, the reported latency represents the\narithmetic mean across varying context lengths ({512, 1024, 2048, 4096}tokens ). Since the target\nmodel dominates the verification compute time at this batch size, the sequential block’s latency\noverhead is negligible. Consequently, scaling the draft length from 4 to 16 adds a marginal\n0.2% to 1.3% to the full-round latency over the DFlash baseline, despite delivering up to a 30%\nimprovement in accepted length.\n4.3.3. Verify Smarter, Not Longer: The Role of Confidence Head\nWhile DSpark sustains high acceptance over long draft blocks, verifying the entire proposal\nremains inefficient (Hu et al., 2026; Huang et al., 2024). Due to the inherent domain variance\nnoted in Section 4.2, trailing tokens in open-ended chat still face high rejection risks, making\nblind verification a waste of target compute. To evaluate whether the confidence head can\neffectively prune these unpromising suffixes, we conduct an offline threshold sweep using\nQwen3-4B. We validate the estimator in isolation here, reserving the hardware-aware prefix\nscheduler (Section 3.2.2) for live production evaluation in Section 5.\nDiagnostic: Static Threshold Sweep.Figure 5 plots the average tokens per step (bars) and\nthe overall acceptance rate (line) across confidence thresholds. As the threshold increases, the\nacceptance rate steadily rises because the estimator filters out tokens that would ultimately be\nrejected (hashed bars). This suggests that the confidence head can identify lower-value suffix\ntokens and this pruning is most pronounced on chat workloads, where higher-entropy token\ndistributions limit the efficiency of fixed-length verification. In the Chat subplot, raising the\nthreshold significantly reduces rejected tokens, increasing the acceptance rate from 45.7% to\n95.7%. In contrast, structured tasks (Math and Code) experience milder pruning and retain more\ndraft tokens, with acceptance rates rising from 76.9% to 92.5% and 67.6% to 92.0%, respectively.\n14\n\n--- PAGE 15 ---\n\n0.0 0.2 0.4 0.6 0.8\n0\n2\n4\n6\n8Avg. Tokens per Step\nMath\n0.0 0.2 0.4 0.6 0.8\nCode\n0.0 0.2 0.4 0.6 0.8\nChat\n76.9%\n92.5%\n67.6%\n92.0%\n40%\n60%\n80%\n100%\nAcceptance Rate\n45.7%\n95.7%\nConfidence Threshold\nAccepted Tokens Rejected Tokens Acceptance Rate\nFigure 5| Confidence threshold sweep.A threshold of 0 corresponds to standard fixed-length\nverification. As the threshold increases, the overall acceptance rate steadily rises because the\nconfidence head effectively prunes tokens that would ultimately be rejected (hashed bars).\n0.00 0.25 0.50 0.75 1.00\n0.00\n0.25\n0.50\n0.75\n1.00Observed Acceptance\nPosition 1\nECE: 5.7% 2.0%\nAUC: 0.818\n0.00 0.25 0.50 0.75 1.00\nPosition 3\nECE: 8.2% 1.7%\nAUC: 0.812\n0.00 0.25 0.50 0.75 1.00\nPosition 5\nECE: 5.8% 0.8%\nAUC: 0.864\n0.00 0.25 0.50 0.75 1.00\nPosition 7\nECE: 3.3% 0.4%\nAUC: 0.907\nPredicted Acceptance\nPerfect calibration Before calibration After calibration\nFigure 6| The Reliability Diagram on Alpaca Dataset.While the raw confidence estimator\nachieves strong discrimination, its predictions are inherently overconfident. Applying post-hoc\ncalibration helps to align the prefix survival probabilities with empirical acceptance rates. The\nshaded background histogram represents the frequency distribution of sample counts across\ndifferent confidence bins.\nFrom Static Thresholds to Calibrated Scheduling.While useful for diagnostics, a static\nthreshold is sub-optimal in dynamic serving environments because it ignores system load:\nverifying low-confidence tokens incurs minimal opportunity cost under low concurrency, but\nwastes critical batch capacity under high concurrency. This load dependency motivates the\nhardware-aware prefix scheduler. As formulated in Section 3.2, maximizing system-level\nthroughput requires the confidence model to exhibit both strong predictive discrimination\nand precise calibration to accurately estimate cumulative survival probabilities. The reliability\ndiagram (Figure 6) demonstrates that while the raw model achieves strong discrimination\n(ROC-AUC (Hanley and McNeil, 1982) ranging from 0.81 to 0.90), it is overly confident (ECE\n3%–8%). Applying post-hoc STS (Section 3.2.1) mitigates this overconfidence, reducing the\naverage ECE to∼1% and yielding reliable survival estimates.\n5. Real-World Deployment of DSpark\nWhile Section 4 establishes the algorithmic gains of DSpark on offline benchmarks, deploying\nit alongside large-scale models like DeepSeek-V4 (DeepSeek-AI, 2026) introduces additional\n15\n\n--- PAGE 16 ---\n\nsystem-level challenges across both training and inference. In this section, we present the end-to-\nend production pipeline of DSpark. We detail our scalable training mechanisms, the system-level\noptimizations necessary to deploy the hardware-aware prefix scheduler (Section 3.2.2), and the\nframework’s end-to-end performance under live user traffic.\n5.1. Scalable and Flexible Training\nThe DSpark draft models are co-deployed with the preview versions of DeepSeek-V4-Flash and\nDeepSeek-V4-Pro (DeepSeek-AI, 2026). The parallel backbone comprises three MoE layers (Dai\net al., 2024) with mHC (Xie et al., 2026) and a sliding window attention of 128. We configure the\nmaximum block size to𝛾= 5 and utilize the Markov head for sequential modeling. Furthermore,\nthe confidence head is trained end-to-end alongside the draft model and subsequently calibrated\nvia STS to provide reliable scheduling signals.\nTraining the draft model requires the target model’s output distributions for supervision.\nEvaluating both models over the full document context incurs substantial memory footprints\nand inter-worker communication overhead. To address these bottlenecks, we implement two\nsystem-level optimizations within our internal training framework (HAI-LLM)4:\n• Hidden state communication.Transferring the target model’s full-vocabulary logits (𝑉≈ 105)\nacross parallel workers creates a significant bandwidth bottleneck. Instead, we temporarily\ncache the target model’s forward-pass activations and communicate only the hidden states\nimmediately preceding the language modeling (LM) head. The LM head projection is then\nexecuted locally on the draft model’s workers only for the sampled target positions. This\nreduces the per-token communication complexity to𝑂(𝑑), where𝑑is the hidden dimension.\n• Anchor-bounded sequence packing.To decouple the draft model’s computational cost\nfrom the target model’s context length, we sample a fixed number of draft anchors from\nthe training sequence and pack these isolated prediction blocks into dense training batches.\nWe manage this packing via token-level attention indices rather than standard 2D masks.\nThis maintains exact causal masking across multiple independent sequences and anchors,\navoiding the computational and memory overhead associated with standard padding.\n5.2. Hardware-Aware Prefix Scheduler in Practice\nIn Section 3.2.2, Algorithm 1 provides a theoretically sound and lossless scheduling mechanism.\nHowever, directly deploying this algorithm into a production environment exposes two funda-\nmental conflicts with real-world infrastructure. First, the algorithm assumes a smooth, unimodal\ncapacity curve, whereas the true hardware capacity SPS(𝐵) is inherently discrete, exhibiting a\njagged, step-wise degradation (Yan et al., 2020). Second, the algorithm requires scheduling of\ndynamic draft tokens per step, which clashes with continuous CUDA graph replay (Fireworks\nAI, 2023) and Zero-Overhead Scheduling (ZOS) (Zheng et al., 2024; Zhu et al., 2025).\nTo navigate the trade-offs among system compatibility, throughput, and algorithmic correct-\nness, we adapt the scheduler to operate asynchronously. Because ZOS requires the batch size for\nthe next step to be known before the current step completes, synchronous scheduling would\ninevitably stall the GPU pipeline. Instead, we approximate the upcoming verification capacity\nusing the confidence head outputs from two steps prior. Mechanically, the candidate tokens in\nthe current step are still strictly sorted by their actual, up-to-date cumulative confidence scores;\nthe historical prediction from two steps prior is used solely to determine the dynamic truncation\n4https://www.high-flyer.cn/en/blog/hai-llm/\n16\n\n--- PAGE 17 ---\n\nlength (i.e., the batch capacity limit 𝐾). This effectively casts the admission process as a dynamic\ntop-𝐾 selection. While approximating the capacity 𝐾 introduces a slight temporal offset, the\nselection mechanism is fundamentally rank-preserving: the most confident draft tokens are\nalways prioritized for verification. This adaptation fully hides scheduling latency and ensures\nseamless ZOS integration.\nBuilding on this asynchronous pipeline, we resolve the hardware utilization bottleneck. To\nprevent the scheduler from being trapped in local minima by jagged SPS cliffs, we remove the\nearly-stopping break, enabling an unconstrained global search. Ordinarily, this retrospective\nsearch would leak future token information and violate the lossless guarantee (Appendix A).\nHowever, our ZOS-driven adaptation naturally prevents this. Because the unconstrained search\nevaluates only historical predictions from two steps prior, the admission decision is isolated\nfrom the realization of the current token 𝑥𝑟,𝑘. The truncation length inherently depends only\non information available from two steps prior. Thus, asynchronous design forms a causal\nbarrier, maximizing physical throughput across hardware cliffs while preserving the exact target\ndistribution.\n5.3. High-Throughput and Low-Latency Inference\nDuring decoding, production serving systems must simultaneously optimize two competing\nobjectives: per-request latency and aggregate throughput (Kwon et al., 2023; Zhao et al., 2025a;\nZhong et al., 2024). The former governs the quality of service for individual users—a factor\nincreasingly critical in agent-based workloads (Tiwari et al., 2026)—while the latter determines\nthe total number of concurrently served users. Because speculative decoding inevitably incurs\nwasted verification compute, it inherently navigates this trade-off, trading extra system compute\nfor faster per-request generation.\nIn our deployment setting, however, the number of requests processed per step is frequently\nconstrained by resource limits (e.g., fixed KV-cache capacity per request) and the pool of available\nuser traffic (e.g., RL long-tail loads). Consequently, the effective batch size persistently remains\nwell below the GPU’s compute-saturating threshold. Under this regime, the traditional trade-off\nsimplifies: given a fixed concurrency limit, maximizing per-GPU total token throughput and\nmaximizing the generation speed per user (tok/s/user) become highly correlated objectives rather\nthan competing ones.\nTo achieve this maximum throughput, the asynchronous scheduler (Section 5.2) actively\nroutes idle compute toward the most promising draft tokens. However, executing this dynamic\nrouting introduces a severe challenge at the physical execution layer: the inference framework\nmust efficiently support variable-length queries within a single batch. Standard decode ker-\nnels are heavily optimized for fixed query lengths; naively processing variable-length verified\nprefixes leads to severe GPU under-utilization due to padding and uneven workload distribu-\ntion. We resolve this by decoupling physical execution from logical sequence tracking. In our\ncompute kernels, all tokens across different requests are flattened and processed identically as\nindependent elements. The complex intra-sequence dependencies are then strictly conveyed\nvia a marker tensor integrated into our sparse attention implementation. Specifically on the\nDeepSeek-V4 architecture, only the index-attention and compress kernels require modification\nto support this variable-length routing, allowing the dynamic scheduler to operate seamlessly\nwithout introducing low-level execution overhead.\n17\n\n--- PAGE 18 ---\n\n50 75 100 125 150 175 200 225\n0\n5k\n10k\n15k\n20k\nThroughput (token/s/gpu)\nDeepSeek-V4-Flash\n+51% throughput\n+60% TPS\n+661% throughput\n+85% TPS\n20 40 60 80 100 120\n0\n1k\n2k\n3k\n4k\n5k\n6k\nDeepSeek-V4-Pro\nMTP\nDSpark\n+52% throughput\n+57% TPS\n+406% throughput\n+78% TPS\nTPS (token/s/user)\nFigure 7| Throughput vs. TPS.Aggregate output token throughput against per-request genera-\ntion speed (tok/s/user) under live traffic. In our production deployment, DSpark improves the\nobserved throughput–interactivity frontier relative to the MTP-1 baseline under the measured\ntraffic and engine configurations.\n5.4. Performance under Live User Traffic\nWe evaluate DSpark-5 (configured with a maximum draft length of 𝛾= 5) against the MTP-\n1 (DeepSeek-AI, 2024) baseline within the production serving engines of DeepSeek-V4-Flash (pre-\nview) and DeepSeek-V4-Pro (preview). MTP-1 represents the former production setup, having\nbeen superseded by DSpark two weeks following the DeepSeek-V4-preview release. This single-\ntoken setup was historically maintained in production because deploying a static multi-token\ndrafter (e.g., MTP-3/5) strictly degrades aggregate throughput under high concurrency due to\nexcessive verification overhead. Therefore, comparing DSpark against this established baseline\ndirectly demonstrates its ability to safely unlock the performance potential of larger draft blocks\nin dynamic serving environments. In all figures, the scatter points represent raw telemetry data\nsampled directly from live user traffic, capturing complex, real-world request distributions,\nwhile the solid lines represent the fitted performance frontiers.\nThe Serving Pareto Frontier.Figure 7 illustrates the trade-off between aggregate system\nthroughput and per-user generation speed (interactivity). To quantify DSpark’s behavior under\npractical deployment constraints, we evaluate the system at several interactivity SLA anchors.\nHere, an SLA (Service Level Agreement) specifies the minimum per-user generation speed (in\ntokens per second) that the system must guarantee.\nFor the V4-Flash engine, we evaluate the system at SLA anchors of 80 and 120 tok/s/user.\nAt the moderate 80 tok/s/user SLA, DSpark improves aggregate throughput by 51% over the\nMTP-1 baseline. The stricter 120 tok/s/user SLA represents a qualitatively different regime:\nunder this constraint, the single-token MTP-1 baseline approaches its operational boundary and\ncan sustain only a very small concurrent batch. Consequently, the relative throughput ratio\nat this point is numerically large, with DSpark achieving a nominal 661% higher aggregate\nthroughput. We therefore interpret this high-SLA point primarily as evidence that DSpark\nextends the feasible interactivity frontier, rather than as a representative multiplicative speedup\nover a well-utilized baseline. At matched practical throughput levels, which provide a more\nstable comparison, DSpark accelerates per-user generation speeds by 60% to 85%.\n18\n\n--- PAGE 19 ---\n\nThe V4-Pro deployment shows the same pattern. At the moderate 35 tok/s/user SLA,\nDSpark improves aggregate throughput by 52%. At the stricter 50 tok/s/user SLA, MTP-1\nagain enters a low-concurrency regime, yielding a nominal 406% relative throughput advantage\nfor DSpark. As with V4-Flash, we treat this point as an indication that DSpark sustains useful\nthroughput under an interactivity target that the baseline cannot efficiently support. At matched\nsystem capacities, DSpark delivers 57% to 78% faster per-user generation. Overall, these results\nshow that DSpark shifts the observed throughput–interactivity frontier outward: it improves\nthroughput in moderate-SLA regimes and, more importantly, preserves non-degenerate serving\ncapacity under strict interactivity constraints.\n0\n5k\n10k\n15k\nThroughput (token/s/gpu)\nDeepSeek-V4-Flash\nMTP\nDSpark\n0\n2k\n4k\n6k\nDeepSeek-V4-Pro\n0 50 100 150 200\n1\n2\n3\n4\n5\n6\nVerification Budget\n0 25 50 75 100 125 150 175 200\n1\n2\n3\n4\n5\n6\nNumber of Concurrent Requests\nFigure 8| Load-adaptive throughput and verification budgets.Top row (a, b): Aggregate output\nthroughput across varying levels of system concurrency. Bottom row (c, d): The average target\nverification budget allocated per request. As concurrent load increases, the dynamic scheduler\nautomatically restricts the per-request verification length to prevent resource contention.\nThroughput Dynamics under Load.Figure 8 analyzes the underlying mechanism driving\nthese gains by plotting aggregate throughput (top row) and the dynamic verification budget\n(bottom row) against system concurrency.\n• Under the moderate concurrency regimes typical of our production deployment (fewer\nthan 200 concurrent requests for V4-Flash and 150 for V4-Pro), the hardware-aware sched-\nuler leverages available target compute capacity by allocating longer verification budgets,\nexpanding from MTP-1’s static 2 tokens to roughly 4–6 tokens per request. This extended\nverification yields more accepted tokens per forward pass, directly contributing to the\nthroughput gains observed on the Pareto frontier.\n• As system concurrency scales and target capacity saturates, the scheduler dynamically\nrestricts this budget. The average verification length decreases smoothly with load, en-\nsuring that low-confidence draft tokens are pruned before they consume critical batch\ncapacity. This load-aware behavior stabilizes production deployment: DSpark maximizes\n19\n\n--- PAGE 20 ---\n\nthe utility of idle compute under light traffic, while effectively preserving critical batch\ncapacity under heavy traffic.\nLimitations.Although the prefix scheduler minimizes wasted target-model verification, DSpark\nstill incurs a fixed draft-side cost to generate the initial𝛾-token block via the parallel backbone.\nFor complex queries with inherently low acceptance rates, this upfront drafting compute is\nunrecoverable. Future optimizations could introduce difficulty-aware early exiting within the\ndraft model, enabling such requests to bypass full-block generation.\n6. Related Work\nSpeculative Decoding Algorithms.Speculative decoding accelerates autoregressive genera-\ntion by decoupling token proposal from verification. Building on early blockwise methods (Ge\net al., 2022; Stern et al., 2018; Sun et al., 2021; Xia et al., 2023), modern approaches employ rejec-\ntion sampling to exactly preserve the target model’s distribution (Chen et al., 2023; Leviathan\net al., 2023). Because inference speedup directly depends on the drafter’s efficiency and accu-\nracy, extensive research has focused on optimizing its architecture. Beyond using standalone\nsmall language models (Chen et al., 2023; Leviathan et al., 2023), subsequent work integrates\nmulti-token heads or feature extrapolators directly into the target model (Ankner et al., 2024;\nCai et al., 2024, 2025; DeepSeek-AI, 2024; Gloeckle et al., 2024; Li et al., 2024b,c, 2026b; Zhang\net al., 2025). Other strategies include self-speculation via early exits (Elhoushi et al., 2024; Liu\net al., 2024a; Xia et al., 2025; Zhang et al., 2024), dynamic vocabulary compression (Williams\net al., 2026; Zhao et al., 2025b), prompt lookup (Saxena, 2023; Somasundaram et al., 2025), suffix\nautomata (Hu et al., 2025), and retrieval (He et al., 2023; Shen et al., 2026). To remove the\nsequential bottleneck of drafting itself, recent methods propose parallel or blockwise genera-\ntion. P-EAGLE parallelizes EAGLE-style drafting (Hui et al., 2026), while PARD, DART, and\nDFlash use diffusion-inspired prediction to generate entire blocks in a single forward pass (An\net al., 2026; Chen et al., 2026; Liu et al., 2026a), which DDTree then extends into verifiable draft\ntrees (Ringel and Romano, 2026). Concurrent efforts also improve DFlash: Domino (Huang et al.,\n2026a) introduces a CausalEncoder conceptually similar to our RNN Head, while DFlare (Zhang\net al., 2026a) addresses conditioning bottlenecks via layer-wise fusion.\nSystem-Aware Scheduling for Speculative Decoding.Beyond drafter architecture, another\nline of work focuses on determining the optimal number of speculative tokens to generate\nor verify in each round. To this end, various approaches adapt draft lengths on the fly using\nconfidence heuristics (Du et al., 2024; Li et al., 2024b; Liu et al., 2026c; Mamou et al., 2024; Wen and\nFeng, 2026), learned acceptance predictors (Huang et al., 2024; Zacks917, 2026), or bandit-style\npolicies (Liu et al., 2026b). Furthermore, recognizing speculative decoding as inherently a system-\nlevel scheduling problem, recent works optimize overall goodput and latency by adjusting\nspeculation budgets according to real-time system load and request priority (AngelSlim Team,\n2026; Hu et al., 2026; Huang et al., 2026b; Li et al., 2026a; Liu et al., 2024c; Miao et al., 2024;\nSadhukhan et al., 2025; Wu et al., 2025).\nParallel Generation.Models that generate tokens in parallel offer a decoding latency nearly\nindependent of output length, making them an attractive alternative to autoregressive decoding.\nNon-Autoregressive Transformers (NATs, Gu et al., 2018) pioneered this direction by predicting\nall positions independently in a single pass. However, this forces the model to average over all\n20\n\n--- PAGE 21 ---\n\nplausible modes, often producing outputs that mix fragments from different valid sequences.\nTwo broad lines of work have emerged to address this limitation. One direction retains the\nsingle-pass architecture but changes what the model sees or how it is trained: introducing\nlatent variables as conditioning input to steer all positions toward a consistent output (Gu et al.,\n2018; Kaiser et al., 2018; Ma et al., 2019), or relaxing the training objective so that the model\nfocuses on producing a single coherent output rather than modeling the full distribution over all\nvalid alternatives (Du et al., 2021; Qian et al., 2021; Shao et al., 2021, 2023). The other direction\nreintroduces limited sequential dependency through iterative re-prediction (Austin et al., 2021a;\nGhazvininejad et al., 2019; Li et al., 2022), block-level autoregression (Arriola et al., 2025; Wang\net al., 2018), or structured output layers such as CRF (Sun et al., 2019), CTC (Libovický and Helcl,\n2018; Saharia et al., 2020), HMM (Huang et al., 2022b), and PCFG (Gui et al., 2023).\nSpeculative decoding places a further demand that the drafter must provide exact per-token\nprobabilities for the rejection sampling rule. Most techniques above cannot readily provide such\nprobabilities due to iterative refinement, latent marginalization, or global normalization. For\ninstance, in a design closely related to ours, CRF-NAT (Sun et al., 2019) also places a sequential\nmodule over parallel hidden states, but its globally normalized partition function prevents exact\nper-token probability computation. Similarly, when adapting the CTC output layer to parallel\nspeculative decoding, CTC-drafter (Wen et al., 2024) is restricted to greedy verification due to\nthe latent marginalization of alignment paths. DSpark circumvents these limitations by keeping\nthe sequential correction local, so per-token probabilities remain exact softmax evaluations.\n7. Conclusion\nIn this paper, we present DSpark, a speculative decoding framework designed to overcome the\nstructural and system-level bottlenecks of large language model inference in high-concurrency\nproduction environments. Algorithmically, DSpark introduces a semi-autoregressive generation\nparadigm—coupling a computationally heavy parallel backbone with a lightweight sequential\nhead—to mitigate the rapid suffix decay of independent parallel drafters. At the system level, we\nformulate verification length selection as a global throughput maximization problem, employing\na hardware-aware prefix scheduler that dynamically tailors the target model’s verification\nbudget based on calibrated survival probabilities and real-time engine load. Extensive offline\nevaluations demonstrate that DSpark substantially outperforms state-of-the-art autoregressive\nand parallel baselines across diverse domains. Furthermore, its real-world deployment within\nthe DeepSeek-V4 validates its practical value in production serving: by intelligently managing\nverification overhead, DSpark sustains robust concurrency under heavy load, consistently\naccelerates per-user generation speeds, and effectively shifts the Pareto frontier of LLM serving\noutward.\nReferences\nT. Abramovich, M. Ashkenazi, I. Putterman, B. Chislett, T. Mitra, B. D. Rouhani, R. Zilberstein,\nand Y. Geifman. Speed-bench: A unified and diverse benchmark for speculative decoding.\narXiv preprint arXiv:2604.09557, 2026.\nZ. An, H. Bai, Z. Liu, D. Li, and E. Barsoum. PARD: Accelerating LLM inference with low-cost\nPARallel draft model adaptation. In The Fourteenth International Conference on Learning\nRepresentations, 2026. URLhttps://openreview.net/forum?id=XbOyv7iVGL.\n21\n\n--- PAGE 22 ---\n\nAngelSlim Team. D-Cut: Adaptive verification depth pruning for speculative decoding, 2026.\nURLhttps://angelslim.readthedocs.io/zh-cn/latest/dcut.html.\nZ. Ankner, R. Parthasarathy, A. Nrusimha, C. Rinard, J. Ragan-Kelley, and W. Brandon. Hydra:\nSequentially-dependent draft heads for medusa decoding. In First Conference on Language\nModeling, 2024. URLhttps://openreview.net/forum?id=FbhjirzvJG.\nM. Arriola, S. S. Sahoo, A. Gokaslan, Z. Yang, Z. Qi, J. Han, J. T. Chiu, and V . Kuleshov.\nBlock diffusion: Interpolating between autoregressive and diffusion language models. In\nThe Thirteenth International Conference on Learning Representations, 2025. URL https:\n//openreview.net/forum?id=tyEyYT267x.\nJ. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. van den Berg. Structured denoising diffusion\nmodels in discrete state-spaces. In A. Beygelzimer, Y. Dauphin, P . Liang, and J. W. Vaughan,\neditors, Advances in Neural Information Processing Systems, 2021a. URL https://openre\nview.net/forum?id=h7-XixPCAL.\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry,\nQ. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732,\n2021b.\nT. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao. Medusa: Simple LLM infer-\nence acceleration framework with multiple decoding heads. In R. Salakhutdinov, Z. Kolter,\nK. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the\n41st International Conference on Machine Learning, volume 235 of Proceedings of Machine\nLearning Research, pages 5209–5235. PMLR, 21–27 Jul 2024. URL https://proceedings.\nmlr.press/v235/cai24b.html.\nY. Cai, X. Liang, X. Wang, J. Ma, H. Liang, J. Luo, X. Zuo, L. Duan, Y. Yin, and X. Chen.\nFastmtp: Accelerating llm inference with enhanced multi-token prediction, 2025. URL https:\n//arxiv.org/abs/2509.18362.\nC. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper. Accelerating large language\nmodel decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.\nJ. Chen, Y. Liang, and Z. Liu. Dflash: Block diffusion for flash speculative decoding. arXiv\npreprint arXiv:2602.06036, 2026.\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P . de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda,\nN. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P . Mishkin,\nB. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P . Tillet,\nF. P . Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss,\nA. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse,\nA. N. Carr, J. Leike, J. Achiam, V . Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage,\nM. Murati, K. Mayer, P . Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and\nW. Zaremba. Evaluating large language models trained on code, 2021.\nY. Cheng, A. Zhang, X. Zhang, C. Wang, and Y. Wang. Recurrent drafter for fast speculative\ndecoding in large language models, 2024. URLhttps://arxiv.org/abs/2403.09919.\nK. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton,\nR. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv\npreprint arXiv:2110.14168, 2021.\n22\n\n--- PAGE 23 ---\n\nD. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, et al. Deepseekmoe:\nTowards ultimate expert specialization in mixture-of-experts language models. InProceedings\nof the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:\nLong Papers), pages 1280–1297, 2024.\nDeepSeek-AI. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.\nDeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence, 2026.\nURLhttps://arxiv.org/abs/2606.19348.\nC. Du, Z. Tu, and J. Jiang. Order-agnostic cross entropy for non-autoregressive machine trans-\nlation. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference\non Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2849–\n2859. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/du21c.html.\nC. Du, J. Jiang, X. Yuanchen, J. Wu, S. Yu, Y. Li, S. Li, K. Xu, L. Nie, Z. Tu, and Y. You. GliDe with a\nCaPE: A low-hassle method to accelerate speculative decoding. In R. Salakhutdinov, Z. Kolter,\nK. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the\n41st International Conference on Machine Learning, volume 235 of Proceedings of Machine\nLearning Research, pages 11704–11720. PMLR, 21–27 Jul 2024. URL https://proceeding\ns.mlr.press/v235/du24c.html.\nM. Elhoushi, A. Shrivastava, D. Liskovich, B. Hosmer, B. Wasti, L. Lai, A. Mahmoud, B. Acun,\nS. Agarwal, A. Roman, A. Aly, B. Chen, and C.-J. Wu. Layerskip: Enabling early exit inference\nand self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association\nfor Computational Linguistics (Volume 1: Long Papers), page 12622–12642. Association for\nComputational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.681. URL http://dx.doi\n.org/10.18653/v1/2024.acl-long.681.\nFireworks AI. Speed, Python: Pick Two. How CUDA Graphs Enable Fast Python Code for Deep\nLearning. https://fireworks.ai/blog/speed-python-pick-two-how-cuda-graph\ns-enable-fast-python-code-for-deep-learning, Aug. 2023. Accessed: 2026-06-22.\nT. Ge, H. Xia, X. Sun, S.-Q. Chen, and F. Wei. Lossless acceleration for seq2seq generation with\naggressive decoding. arXiv preprint arXiv:2205.10350, 2022.\nM. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer. Mask-predict: Parallel decoding of con-\nditional masked language models. In K. Inui, J. Jiang, V . Ng, and X. Wan, editors,Proceedings\nof the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th\nInternational Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages\n6112–6121, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi:\n10.18653/v1/D19-1633. URLhttps://aclanthology.org/D19-1633/.\nF. Gloeckle, B. Youbi Idrissi, B. Roziere, D. Lopez-Paz, and G. Synnaeve. Better & faster large lan-\nguage models via multi-token prediction. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller,\nN. Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the 41st International\nConference on Machine Learning, volume 235 of Proceedings of Machine Learning Research,\npages 15706–15734. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235\n/gloeckle24a.html.\nGoogle DeepMind. Gemma 4 model card. https://ai.google.dev/gemma/docs/core/\nmodel_card_4, 2026. Accessed: 2026-06-11.\n23\n\n--- PAGE 24 ---\n\nJ. Gu, J. Bradbury, C. Xiong, V . O. Li, and R. Socher. Non-autoregressive neural machine\ntranslation. In International Conference on Learning Representations, 2018. URL https:\n//openreview.net/forum?id=B1l8BtlCb.\nS. Gui, C. Shao, Z. Ma, X. Zhang, Y. Chen, and Y. Feng. Non-autoregressive machine translation\nwith probabilistic context-free grammar. InThirty-seventh Conference on Neural Information\nProcessing Systems, 2023. URLhttps://openreview.net/forum?id=LloZFVwWvj.\nC. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In\nInternational conference on machine learning, pages 1321–1330. PMLR, 2017.\nJ. A. Hanley and B. J. McNeil. The meaning and use of the area under a receiver operating\ncharacteristic (roc) curve. Radiology, 143(1):29–36, 1982. doi: 10.1148/radiology.143.1.7063747.\nZ. He, Z. Zhong, T. Cai, J. D. Lee, and D. He. Rest: Retrieval-based speculative decoding, 2023.\nX. Hu, Y. Shen, B. Zhang, H. Zhang, J. Dai, S. Ge, L. Chen, Y. Li, and M. Wan. Echo: Elastic\nspeculative decoding with sparse gating for high-concurrency scenarios. arXiv preprint\narXiv:2604.09603, 2026.\nY. Hu, K. Wang, X. Zhang, F. Zhang, C. Li, H. Chen, and J. Zhang. SAM decoding: Speculative\ndecoding via suffix automaton. In W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, editors,\nProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics\n(Volume 1: Long Papers), pages 12187–12204, Vienna, Austria, July 2025. Association for\nComputational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.595.\nURLhttps://aclanthology.org/2025.acl-long.595/.\nF. Huang, T. Tao, H. Zhou, L. Li, and M. Huang. On the learning of non-autoregressive\ntransformers. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato,\neditors, Proceedings of the 39th International Conference on Machine Learning, volume 162\nof Proceedings of Machine Learning Research, pages 9356–9376. PMLR, 17–23 Jul 2022a. URL\nhttps://proceedings.mlr.press/v162/huang22k.html.\nF. Huang, H. Zhou, Y. Liu, H. Li, and M. Huang. Directed acyclic transformer for non-\nautoregressive machine translation. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari,\nG. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine\nLearning, volume 162 of Proceedings of Machine Learning Research, pages 9410–9428. PMLR,\n17–23 Jul 2022b. URLhttps://proceedings.mlr.press/v162/huang22m.html.\nJ. Huang, Y. Zhang, Q. Zhang, H. Lin, H. Xu, and L. Zhang. Domino: Decoupling causal\nmodeling from autoregressive drafting in speculative decoding, 2026a. URL https://arxi\nv.org/abs/2605.29707.\nK. Huang, X. Guo, and M. Wang. Specdec++: Boosting speculative decoding via adaptive\ncandidate lengths. arXiv preprint arXiv:2405.19715, 2024.\nK. Huang, H. Wu, Z. Shi, H. Zou, M. Yu, and Q. Shi. Adaspec: Adaptive speculative decoding for\nfast, slo-aware large language model serving. In Proceedings of the 2025 ACM Symposium\non Cloud Computing, SoCC ’25, page 361–374, New York, NY, USA, 2026b. Association\nfor Computing Machinery. ISBN 9798400722769. doi: 10.1145/3772052.3772239. URL\nhttps://doi.org/10.1145/3772052.3772239.\nM. Hui, X. Huang, J. C. Salas, Y. Sun, N. Pemberton, X. Song, A. Khetan, and G. Karypis. P-eagle:\nParallel-drafting eagle with scalable training, 2026. URL https://arxiv.org/abs/2602\n.01469.\n24\n\n--- PAGE 25 ---\n\nD. Israel, G. Van den Broeck, and A. Grover. Accelerating diffusion llms via adaptive parallel\ndecoding. Advances in neural information processing systems, 38:52870–52888, 2026.\nN. Jain, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica.\nLivecodebench: Holistic and contamination free evaluation of large language models for code.\nIn International Conference on Learning Representations, volume 2025, pages 58791–58831,\n2025.\nL. Kaiser, S. Bengio, A. Roy, A. Vaswani, N. Parmar, J. Uszkoreit, and N. Shazeer. Fast decoding in\nsequence models using discrete latent variables. In J. Dy and A. Krause, editors, Proceedings\nof the 35th International Conference on Machine Learning, volume 80 of Proceedings of\nMachine Learning Research, pages 2390–2399. PMLR, 10–15 Jul 2018. URL https://proc\needings.mlr.press/v80/kaiser18a.html.\nW. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica.\nEfficient memory management for large language model serving with pagedattention. In\nProceedings of the 29th symposium on operating systems principles, pages 611–626, 2023.\nY. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative\ndecoding. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, edi-\ntors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of\nProceedings of Machine Learning Research, pages 19274–19286. PMLR, 23–29 Jul 2023. URL\nhttps://proceedings.mlr.press/v202/leviathan23a.html.\nG. Li, Z. Fu, M. Fang, Q. Zhao, M. Tang, C. Yuan, and J. Wang. Diffuspec: Unlocking diffusion\nlanguage models for speculative decoding, 2025a. URL https://arxiv.org/abs/2510.0\n2358.\nR. Li, Z. Zhang, L. Zhang, H. Wang, X. Fu, and Z. Lai. Nightjar: Dynamic adaptive speculative\ndecoding for large language models serving, 2026a. URL https://arxiv.org/abs/2512\n.22420.\nT. Li, W.-L. Chiang, E. Frick, L. Dunlap, B. Zhu, J. E. Gonzalez, and I. Stoica. From live data to\nhigh-quality benchmarks: The arena-hard pipeline, April 2024a. URL https://lmsys.org/\nblog/2024-04-19-arena-hard/.\nT. Li, W.-L. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica. From\ncrowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.\nIn A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff,\nand J. Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning,\nvolume 267 of Proceedings of Machine Learning Research, pages 34209–34231. PMLR, 13–19\nJul 2025b. URLhttps://proceedings.mlr.press/v267/li25h.html.\nX. L. Li, J. Thickstun, I. Gulrajani, P . Liang, and T. Hashimoto. Diffusion-LM improves control-\nlable text generation. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in\nNeural Information Processing Systems, 2022. URL https://openreview.net/forum?i\nd=3s9IrEsjLyk.\nY. Li, F. Wei, C. Zhang, and H. Zhang. EAGLE-2: Faster inference of language models with\ndynamic draft trees. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Proceedings of the\n2024 Conference on Empirical Methods in Natural Language Processing, pages 7421–7432,\nMiami, Florida, USA, Nov. 2024b. Association for Computational Linguistics. doi: 10.18653/v\n1/2024.emnlp-main.422. URLhttps://aclanthology.org/2024.emnlp-main.422/.\n25\n\n--- PAGE 26 ---\n\nY. Li, F. Wei, C. Zhang, and H. Zhang. EAGLE: Speculative sampling requires rethinking\nfeature uncertainty. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett,\nand F. Berkenkamp, editors, Proceedings of the 41st International Conference on Machine\nLearning, volume 235 of Proceedings of Machine Learning Research, pages 28935–28948.\nPMLR, 21–27 Jul 2024c. URLhttps://proceedings.mlr.press/v235/li24bt.html.\nY. Li, F. Wei, C. Zhang, and H. Zhang. EAGLE-3: Scaling up inference acceleration of large\nlanguage models via training-time test. In The Thirty-ninth Annual Conference on Neural\nInformation Processing Systems, 2026b. URL https://openreview.net/forum?id=4e\nxx1hUffq.\nJ. Libovický and J. Helcl. End-to-end non-autoregressive neural machine translation with con-\nnectionist temporal classification. In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors,\nProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,\npages 3016–3021, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguis-\ntics. doi: 10.18653/v1/D18-1336. URLhttps://aclanthology.org/D18-1336/.\nH. Lightman, V . Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman,\nI. Sutskever, and K. Cobbe. Let’s verify step by step. In The Twelfth International Conference\non Learning Representations, 2024. URL https://openreview.net/forum?id=v8L0pN\n6EOi.\nF. Liu, Y. Tang, Z. Liu, Y. Ni, D. Tang, K. Han, and Y. Wang. Kangaroo: Lossless self-speculative\ndecoding for accelerating llms via double early exiting. In A. Globerson, L. Mackey, D. Bel-\ngrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information\nProcessing Systems, volume 37, pages 11946–11965. Curran Associates, Inc., 2024a. doi:\n10.52202/079017-0381. URL https://proceedings.neurips.cc/paper_files/paper\n/2024/file/16336d94a5ffca8de019087ab7fe403f-Paper-Conference.pdf.\nF. Liu, X. Li, K. Zhao, Y. Gao, Z. Zhou, Z. Zhang, Z. Wang, W. Dou, S. Zhong, and C. Tian.\nDart: Diffusion-inspired speculative decoding for fast llm inference, 2026a. URL https:\n//arxiv.org/abs/2601.19278.\nH. Liu, J. Huang, Z. Jia, Y. Park, and Y.-X. Wang. Not-a-bandit: Provably no-regret drafter\nselection in speculative decoding for llms, 2026b. URL https://arxiv.org/abs/2510.2\n0064.\nT. Liu, Q. Lv, Y. Shen, X. Sun, and X. Sun. Talon: Confidence-aware speculative decoding with\nadaptive token trees. arXiv preprint arXiv:2601.07353, 2026c.\nX. Liu, C. Daniel, L. Hu, W. Kwon, Z. Li, X. Mo, A. Cheung, Z. Deng, I. Stoica, and H. Zhang.\nOptimizing speculative decoding for serving large language models using goodput. arXiv\ne-prints, pages arXiv–2406, 2024b.\nX. Liu, J. Park, L. Hu, W. Kwon, Z. Li, C. Zhang, K. Du, X. Mo, K. You, A. Cheung, et al.\nTurbospec: Closed-loop speculation control system for optimizing llm serving goodput. arXiv\npreprint arXiv:2406.14066, 2024c.\nX. Ma, C. Zhou, X. Li, G. Neubig, and E. Hovy. FlowSeq: Non-autoregressive condi-\ntional sequence generation with generative flow. In K. Inui, J. Jiang, V . Ng, and X. Wan,\neditors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language\nProcessing and the 9th International Joint Conference on Natural Language Processing\n26\n\n--- PAGE 27 ---\n\n(EMNLP-IJCNLP), pages 4282–4292, Hong Kong, China, Nov. 2019. Association for Computa-\ntional Linguistics. doi: 10.18653/v1/D19-1437. URL https://aclanthology.org/D19-1\n437/.\nJ. Mamou, O. Pereg, D. Korat, M. Berchansky, N. Timor, M. Wasserblat, and R. Schwartz.\nDynamic speculation lookahead accelerates speculative decoding of large language models.\nIn M. Rezagholizadeh, P . Passban, S. Samiee, V . Partovi Nia, Y. Cheng, Y. Deng, Q. Liu, and\nB. Chen, editors, Proceedings of The 4th NeurIPS Efficient Natural Language and Speech\nProcessing Workshop, volume 262 of Proceedings of Machine Learning Research, pages 456–\n467. PMLR, 14 Dec 2024. URL https://proceedings.mlr.press/v262/mamou24a.ht\nml.\nX. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, Z. Zhang, R. Y. Y. Wong, A. Zhu, L. Yang,\nX. Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative\ninference and verification. In Proceedings of the 29th ACM International Conference on\nArchitectural Support for Programming Languages and Operating Systems, Volume3, pages\n932–949, 2024.\nM. P . Naeini, G. Cooper, and M. Hauskrecht. Obtaining well calibrated probabilities using\nbayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29,\n2015.\nY. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan,\nand J. Snoek. Can you trust your model’s uncertainty? evaluating predictive uncertainty\nunder dataset shift. Advances in neural information processing systems, 32, 2019.\nL. Qian, H. Zhou, Y. Bao, M. Wang, L. Qiu, W. Zhang, Y. Yu, and L. Li. Glancing transformer\nfor non-autoregressive neural machine translation. In C. Zong, F. Xia, W. Li, and R. Nav-\nigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational\nLinguistics and the 11th International Joint Conference on Natural Language Processing\n(Volume 1: Long Papers), pages 1993–2003, Online, Aug. 2021. Association for Computa-\ntional Linguistics. doi: 10.18653/v1/2021.acl-long.155. URL https://aclanthology.org\n/2021.acl-long.155/.\nY. Ren, J. Liu, X. Tan, Z. Zhao, S. Zhao, and T.-Y. Liu. A study of non-autoregressive model\nfor sequence generation. In Proceedings of the 58th Annual Meeting of the Association for\nComputational Linguistics, pages 149–159, 2020.\nL. Ringel and Y. Romano. Accelerating speculative decoding with block diffusion draft trees.\narXiv preprint arXiv:2604.12989, 2026.\nR. Sadhukhan, J. Chen, Z. Chen, V . Tiwari, R. Lai, J. Shi, I. E.-H. Yen, A. May, T. Chen, and B. Chen.\nMagicdec: Breaking the latency-throughput tradeoff for long context generation with spec-\nulative decoding. In The Thirteenth International Conference on Learning Representations,\n2025. URLhttps://openreview.net/forum?id=CS2JWaziYr.\nC. Saharia, W. Chan, S. Saxena, and M. Norouzi. Non-autoregressive machine translation with\nlatent alignments. In B. Webber, T. Cohn, Y. He, and Y. Liu, editors,Proceedings of the 2020\nConference on Empirical Methods in Natural Language Processing (EMNLP), pages 1098–\n1108, Online, Nov. 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.e\nmnlp-main.83. URLhttps://aclanthology.org/2020.emnlp-main.83/.\n27\n\n--- PAGE 28 ---\n\nJ. Sandler, J. Christopher, T. Hartvigsen, and F. Fioretto. Specdiff-2: Scaling diffusion drafter\nalignment for faster speculative decoding. In Ninth Conference on Machine Learning and\nSystems, 2026. URLhttps://openreview.net/forum?id=o42VU86ZsV.\nA. Saxena. Prompt lookup decoding, November 2023. URL https://github.com/apoorvu\nmang/prompt-lookup-decoding/.\nC. Shao, Y. Feng, J. Zhang, F. Meng, and J. Zhou. Sequence-level training for non-autoregressive\nneural machine translation. Computational Linguistics, 47(4):891–925, Dec. 2021. doi: 10.116\n2/coli_a_00421. URLhttps://aclanthology.org/2021.cl-4.29/.\nC. Shao, Z. Ma, M. Zhang, and Y. Feng. Beyond MLE: Convex learning for text generation.\nIn Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:\n//openreview.net/forum?id=sla7V80uWA.\nY. Shen, T. Liu, X. Hu, Q. Kong, B. Zhang, J. Dai, J. Zhang, S. Ge, L. Chen, Y. Li, M. Wan, and\nC. Wang. Draft less, retrieve more: Hybrid tree construction for speculative decoding, 2026.\nURLhttps://arxiv.org/abs/2605.20104.\nS. Somasundaram, A. Phukan, and A. Saxena. PLD+: Accelerating LLM inference by leveraging\nlanguage model artifacts. In L. Chiruzzo, A. Ritter, and L. Wang, editors, Findings of the\nAssociation for Computational Linguistics: NAACL 2025, pages 6090–6104, Albuquerque,\nNew Mexico, Apr. 2025. Association for Computational Linguistics. ISBN 979-8-89176-195-7.\ndoi: 10.18653/v1/2025.findings-naacl.338. URL https://aclanthology.org/2025.find\nings-naacl.338/.\nM. Stern, N. Shazeer, and J. Uszkoreit. Blockwise parallel decoding for deep autoregressive\nmodels. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett,\neditors, Advances in Neural Information Processing Systems, volume 31. Curran Associates,\nInc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file\n/c4127b9194fe8562c64dc0f5bf2c93bc-Paper.pdf.\nX. Sun, T. Ge, F. Wei, and H. Wang. Instantaneous grammatical error correction with shallow\naggressive decoding. In C. Zong, F. Xia, W. Li, and R. Navigli, editors,Proceedings of the 59th\nAnnual Meeting of the Association for Computational Linguistics and the 11th International\nJoint Conference on Natural Language Processing (Volume1: Long Papers), pages 5937–5947,\nOnline, Aug. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-lon\ng.462. URLhttps://aclanthology.org/2021.acl-long.462/.\nZ. Sun, Z. Li, H. Wang, D. He, Z. Lin, and Z. Deng. Fast structured decoding for sequence\nmodels. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett,\neditors, Advances in Neural Information Processing Systems, volume 32. Curran Associates,\nInc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file\n/74563ba21a90da13dacf2a73e3ddefa7-Paper.pdf.\nR. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P . Liang, and T. B. Hashimoto.\nStanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab\n/stanford_alpaca, 2023.\nS. Tiwari, T. Chugh, N. Rickert, S. Peter, R. Mahajan, and H. Shen. Cachewise: Understanding\nworkloads and optimizing kvcache management for efficiently serving llm coding agents.\narXiv preprint arXiv:2606.16824, 2026.\n28\n\n--- PAGE 29 ---\n\nC. Wang, J. Zhang, and H. Chen. Semi-autoregressive neural machine translation. In E. Riloff,\nD. Chiang, J. Hockenmaier, and J. Tsujii, editors, Proceedings of the 2018 Conference on\nEmpirical Methods in Natural Language Processing, pages 479–488, Brussels, Belgium, Oct.-\nNov. 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1044. URL\nhttps://aclanthology.org/D18-1044/.\nZ. Wang, D. Ma, X. Huang, D. Cai, T. Lan, J. Xu, H. Mi, X. Tang, and Y. Wang. THE END\nOF MANUAL DECODING: TOWARDS TRULY END-TO-END LANGUAGE MODELS. In\nThe Fourteenth International Conference on Learning Representations, 2026. URL https:\n//openreview.net/forum?id=cPTgQDMD5p.\nZ. Wen and Y. Feng. Specbound: Adaptive bounded self-speculation with layer-wise confidence\ncalibration, 2026. URLhttps://arxiv.org/abs/2604.12247.\nZ. Wen, S. Gui, and Y. Feng. Speculative decoding with ctc-based draft model for llm inference\nacceleration. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and\nC. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages\n92082–92100. Curran Associates, Inc., 2024. doi: 10.52202/079017-2923. URL https:\n//proceedings.neurips.cc/paper_files/paper/2024/file/a79054a9da91d73ed\n3cb1a9e87d7cd2d-Paper-Conference.pdf.\nM. Williams, Y. D. Kwon, R. Li, A. Kouris, and S. I. Venieris. Speculative decoding with a\nspeculative vocabulary. arXiv preprint arXiv:2602.13836, 2026.\nZ. Wu, Z. Zhou, A. Verma, A. Prakash, D. Rus, and B. K. H. Low. TETRIS: Optimal draft token\nselection for batch speculative decoding. In W. Che, J. Nabende, E. Shutova, and M. T. Pile-\nhvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational\nLinguistics (Volume 1: Long Papers), pages 33329–33345, Vienna, Austria, July 2025. Associa-\ntion for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.\n1598. URLhttps://aclanthology.org/2025.acl-long.1598/.\nH. Xia, T. Ge, P . Wang, S.-Q. Chen, F. Wei, and Z. Sui. Speculative decoding: Exploiting\nspeculative execution for accelerating seq2seq generation. In H. Bouamor, J. Pino, and K. Bali,\neditors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3909–\n3925, Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023\n.findings-emnlp.257. URLhttps://aclanthology.org/2023.findings-emnlp.257/.\nH. Xia, Z. Yang, Q. Dong, P . Wang, Y. Li, T. Ge, T. Liu, W. Li, and Z. Sui. Unlocking efficiency\nin large language model inference: A comprehensive survey of speculative decoding. In\nL.-W. Ku, A. Martins, and V . Srikumar, editors,Findings of the Association for Computational\nLinguistics ACL 2024, pages 7655–7671, Bangkok, Thailand and virtual meeting, Aug. 2024.\nAssociation for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.456. URL\nhttps://aclanthology.org/2024.findings-acl.456.\nH. Xia, Y. Li, J. Zhang, C. Du, and W. Li. SWIFT: On-the-fly self-speculative decoding for LLM in-\nference acceleration. InThe Thirteenth International Conference on Learning Representations,\n2025. URLhttps://openreview.net/forum?id=EKJhH5D5wA.\nZ. Xie, Y. Wei, H. Cao, C. Zhao, C. Deng, J. Li, D. Dai, H. Gao, M. Xu, K. Yu, L. Zhao, S. Zhou,\nZ. Xu, Z. Zhang, W. Zeng, S. Hu, Y. Wang, J. Yuan, L. Wang, and W. Liang. mHC: Manifold-\nconstrained hyper-connections. In Forty-third International Conference on Machine Learning,\n2026. URLhttps://openreview.net/forum?id=mDhyxu8WRb.\n29\n\n--- PAGE 30 ---\n\nT. Xu, E. Helenowski, K. A. Sankararaman, D. Jin, K. Peng, E. Han, S. Nie, C. Zhu, H. Zhang,\nW. Zhou, et al. The perfect blend: Redefining rlhf with mixture of judges. arXiv preprint\narXiv:2409.20370, 2024.\nD. Yan, W. Wang, and X. Chu. Demystifying tensor cores to optimize half-precision matrix\nmultiply. 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS),\npages 634–643, 2020. URL https://api.semanticscholar.org/CorpusID:220604999.\nA. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3\ntechnical report. arXiv preprint arXiv:2505.09388, 2025.\nZacks917. AutoMTP_vLLM: Adapt vllm to automtp (early stop for multi-token prediction).\nhttps://github.com/Zacks917/AutoMTP_vLLM, 2026. Accessed: 2026-06-21.\nJ. Zhang, J. Wang, H. Li, L. Shou, K. Chen, G. Chen, and S. Mehrotra. Draft& verify: Lossless\nlarge language model acceleration via self-speculative decoding. In Proceedings of the 62nd\nAnnual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),\npage 11263–11282. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.a\ncl-long.607. URLhttp://dx.doi.org/10.18653/v1/2024.acl-long.607.\nJ. Zhang, Z. Yu, S. Liu, E. J. Yu, Z. Li, D. Zhu, J. Duo, W. Xiong, Y. Song, G. Yu, J. Zhu, and\nS. Li. Dflare: Scaling up draft capacity for block diffusion speculative decoding, 2026a. URL\nhttps://arxiv.org/abs/2606.02091.\nL. Zhang, X. Wang, Y. Huang, and R. Xu. Learning harmonized representations for speculative\nsampling, 2025. URLhttps://arxiv.org/abs/2408.15766.\nS. Zhang, Y. Zhang, Z. Zhu, H. Wang, D. Ma, D. Zhang, L. Chen, and K. Yu. Pacer: Blockwise pre-\nverification for speculative decoding with adaptive length. arXiv preprint arXiv:2602.01274,\n2026b.\nY. Zhang and T. Math-AI. American invitational mathematics examination (aime) 2025, 2025.\nC. Zhao, C. Deng, C. Ruan, D. Dai, H. Gao, J. Li, L. Zhang, P . Huang, S. Zhou, S. Ma, et al. Insights\ninto deepseek-v3: Scaling challenges and reflections on hardware for ai architectures. In\nProceedings of the 52nd Annual International Symposium on Computer Architecture, pages\n1731–1745, 2025a.\nW. Zhao, T. Pan, X. Han, Y. Zhang, S. Ao, Y. Huang, K. Zhang, W. Zhao, Y. Li, J. Zhou, et al.\nFr-spec: Accelerating large-vocabulary language models via frequency-ranked speculative\nsampling. In Proceedings of the 63rd Annual Meeting of the Association for Computational\nLinguistics (Volume 1: Long Papers), pages 3909–3921, 2025b.\nK. Zheng, Y. Chen, H. Mao, M.-Y. Liu, J. Zhu, and Q. Zhang. Masked diffusion models are\nsecretly time-agnostic masked models and exploit inaccurate categorical sampling. In The\nThirteenth International Conference on Learning Representations, 2025. URL https://op\nenreview.net/forum?id=CTC7CmirNr.\nL. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al.\nJudging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information\nprocessing systems, 36:46595–46623, 2023.\nL. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E.\nGonzalez, C. Barrett, and Y. Sheng. Sglang: Efficient execution of structured language model\n30\n\n--- PAGE 31 ---\n\nprograms. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and\nC. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages\n62557–62583. Curran Associates, Inc., 2024. doi: 10.52202/079017-2000. URL https:\n//proceedings.neurips.cc/paper_files/paper/2024/file/724be4472168f31ba\n1c9ac630f15dec8-Paper-Conference.pdf.\nY. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang.{DistServe}: Disaggregating\nprefill and decoding for goodput-optimized large language model serving. In 18th USENIX\nSymposium on Operating Systems Design and Implementation (OSDI 24), pages 193–210,\n2024.\nK. Zhu, Y. Gao, Y. Zhao, L. Zhao, G. Zuo, Y. Gu, D. Xie, T. Tang, Q. Xu, Z. Ye, K. Kamahori,\nC.-Y. Lin, Z. Wang, S. Wang, A. Krishnamurthy, and B. Kasikci. Nanoflow: towards optimal\nlarge language model serving throughput. In Proceedings of the 19th USENIX Conference on\nOperating Systems Design and Implementation, OSDI ’25, USA, 2025. USENIX Association.\nISBN 978-1-939133-47-2.\n31\n\n--- PAGE 32 ---\n\nAppendices\nA. Counterexample: Selection Bias Without Early-Stopping\nWe provide a simple counterexample to illustrate how an offline global search, i.e., operating\nwithout the break condition in Algorithm 1, violates the non-anticipating property required\nby lossless speculative decoding. Formally, the admission event for the𝑘-th draft token,ℓ𝑟≥𝑘 ,\nmust be determined by scheduler-visible information available before the token𝑥𝑟,𝑘 is sampled.\nIt must not depend on the realization of 𝑥𝑟,𝑘 itself. Consider a scenario with a single request\n(𝑅= 1) and maximum draft length (𝛾= 2). Suppose the pre-token confidence for the first position\nis𝑎 1 =0.8, and the profiled capacity curve is\nSPS(1)=1.0, SPS(2)=0.5, SPS(3)=0.45.\nThe expected throughputs for verifying 0 and 1 draft tokens are\nΘ0 =1·SPS(1)=1.0,\nΘ1 =(1+0.8)·SPS(2)=0.9.\nWithout early-stopping, the scheduler proceeds to evaluate Θ2 before committing any admission\ndecisions. Because the Markov confidence head uses the previously sampled token, the next\nconfidence score𝑐2 explicitly depends on the realization of𝑥1. Consequently, the second-prefix\nsurvival probability\n𝑎2 =𝑎 1𝑐2\nalso depends on𝑥 1. Consider two possible realizations of𝑥 1:\n•Case 1 (𝑥 1 yields a high𝑐 2):Suppose𝑥 1 results in𝑐 2 =0.9. Then\n𝑎2 =0.8×0.9=0.72.\nThe expected throughput for length 2 is\nΘ2 =(1+0.8+0.72)×0.45=1.134.\nSince Θ2 is the global maximum among{1.0, 0.9, 1.134}, the scheduler returnsℓ= 2. The\nfirst token𝑥 1 is admitted into the verification prefix.\n•Case 2 (𝑥 1 yields a low𝑐 2):Suppose𝑥 1 results in𝑐 2 =0. Then\n𝑎2 =0.\nThe expected throughput for length 2 is\nΘ2 =(1+0.8+0)×0.45=0.81.\nHere, the global maximum remains Θ0 = 1.0, so the scheduler returnsℓ= 0. The first token\n𝑥1 is not admitted into the verification prefix.\nThus, the admission of the first draft token dynamically depends on the value of the first draft\ntoken itself. This retrospective dependence introduces selection bias: the scheduler favors tokens\nthat lead to highly confident continuations, even though the admission decision for𝑥1 should\nhave been made before observing 𝑥1. We now make the distributional bias explicit. Let the\nvocabulary be{𝐴,𝐵}, and consider the target and draft distributions at the first position:\n𝑝t(𝐴)=0.7,𝑝 t(𝐵)=0.3,\n32\n\n--- PAGE 33 ---\n\n𝑝d(𝐴)=0.5,𝑝 d(𝐵)=0.5.\nThe standard speculative acceptance probability at the first position is\n∑︁\n𝑥∈{𝐴,𝐵}\nmin\u0000\n𝑝t(𝑥),𝑝 d(𝑥) \u0001 =min(0.7, 0.5)+min(0.3, 0.5)=0.8,\nmatching the assumed value (𝑎1 = 0.8). Suppose the retrospective scheduler behaves as above:\n𝑥1 =𝐴 yields a high continuation confidence and hence ℓ= 2, while 𝑥1 =𝐵 yields a low\ncontinuation confidence and henceℓ= 0. Then the first output token is distributed as follows. If\n𝑥1 =𝐴, the draft token is admitted and accepted with probability\nmin\n\u0012\n1, 𝑝t(𝐴)\n𝑝d(𝐴)\n\u0013\n=min\n\u0012\n1, 0.7\n0.5\n\u0013\n=1,\nso the output token is 𝐴. If 𝑥1 =𝐵 , the draft token is not admitted; the target model instead\ngenerates a fresh token from𝑝 t. Therefore,\nPr(𝑌=𝐴)=Pr(𝑥 1 =𝐴)·1+Pr(𝑥 1 =𝐵)·𝑝 t(𝐴)=0.5+0.5×0.7=0.85,\nand hence\nPr(𝑌=𝐵)=0.15.\nThis output distribution ((0.85, 0.15)) differs from the target distribution ((0.7, 0.3)), proving\nthat the retrospective scheduler is not lossless. The early-stopping mechanism prevents this\nissue in the causal greedy scheduler. Since Θ1 <Θ 0, the scheduler halts immediately and returns\nℓ= 0 before evaluating any continuation-dependent quantity such as𝑐2. The admission decision\nfor the first position therefore depends only on pre-token information and cannot be biased\nby the realization of 𝑥1. This restores the non-anticipating property required by the standard\nlosslessness argument.\n33\n","exit_code":0,"status":"completed"}} -{"type":"item.started","item":{"id":"item_32","type":"command_execution","command":"/bin/zsh -lc \"/tmp/deepspec-pdf-venv/bin/python - <<'PY' >/tmp/dspark_paper.txt\nfrom pypdf import PdfReader\nreader=PdfReader('/tmp/DSpark_paper.pdf')\nfor i,page in enumerate(reader.pages, start=1):\n print(f'\\\\n\\\\n--- PAGE {i} ---')\n print(page.extract_text() or '')\nPY\nrg -n \\\"Table 1|Table 2|Table 3|macro-average|accepted length|V4-Flash|V4-Pro|60%|85%|57%|78%|DFlash|Eagle3|throughput|confidence|scheduler|Qwen3\\\" /tmp/dspark_paper.txt | sed -n '1,240p'\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_32","type":"command_execution","command":"/bin/zsh -lc \"/tmp/deepspec-pdf-venv/bin/python - <<'PY' >/tmp/dspark_paper.txt\nfrom pypdf import PdfReader\nreader=PdfReader('/tmp/DSpark_paper.pdf')\nfor i,page in enumerate(reader.pages, start=1):\n print(f'\\\\n\\\\n--- PAGE {i} ---')\n print(page.extract_text() or '')\nPY\nrg -n \\\"Table 1|Table 2|Table 3|macro-average|accepted length|V4-Flash|V4-Pro|60%|85%|57%|78%|DFlash|Eagle3|throughput|confidence|scheduler|Qwen3\\\" /tmp/dspark_paper.txt | sed -n '1,240p'\"","aggregated_output":"binary file matches (found \"\\0\" byte around offset 11852)\n","exit_code":0,"status":"completed"}} -{"type":"item.started","item":{"id":"item_33","type":"command_execution","command":"/bin/zsh -lc \"tr -d '\\\\000' < /tmp/dspark_paper.txt > /tmp/dspark_paper_clean.txt\nrg -n \\\"Table 1|Table 2|Table 3|macro-average|accepted length|V4-Flash|V4-Pro|60%|85%|57%|78%|DFlash|Eagle3|throughput|confidence|scheduler|Qwen3\\\" /tmp/dspark_paper_clean.txt | sed -n '1,260p'\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_33","type":"command_execution","command":"/bin/zsh -lc \"tr -d '\\\\000' < /tmp/dspark_paper.txt > /tmp/dspark_paper_clean.txt\nrg -n \\\"Table 1|Table 2|Table 3|macro-average|accepted length|V4-Flash|V4-Pro|60%|85%|57%|78%|DFlash|Eagle3|throughput|confidence|scheduler|Qwen3\\\" /tmp/dspark_paper_clean.txt | sed -n '1,260p'\"","aggregated_output":"19:wastes critical batch capacity on tokens with high rejection risks, severely degrading throughput\n21:that unifies high-throughput parallel generation with adaptive, load-aware verification. To\n24:and mitigate suffix decay. To optimize system efficiency, DSpark employs confidence-scheduled\n26:prefix survival probabilities and engine-specific throughput profiles. On offline benchmarks\n27:across diverse domains, DSpark substantially improves the accepted length over state-of-the-art\n31:60%–85% at matched throughput levels. More importantly, by preventing severe throughput\n66:blocks, indiscriminately verifying all proposed tokens degrades system throughput, particularly\n74:unifies high-throughput parallel generation with adaptive, load-aware verification. At its core,\n81:• Second, to resolve the system-level bottleneck, DSpark employs confidence-scheduled\n82:verification. By coupling a confidence head—which estimates per-position prefix survival\n83:probabilities—with a hardware-aware scheduler, DSpark dynamically tailors the verifica-\n84:tion length for each request. This scheduler leverages real-time engine throughput profiles\n93:across the Qwen3-4B, 8B, and 14B target models (Yang et al., 2025), it improves the macro-average\n94:accepted length over the autoregressive Eagle3 (Li et al., 2026b) by 30.9%, 26.7%, and 30.0%, and\n95:over the parallel DFlash (Chen et al., 2026) by 16.3%, 18.4%, and 18.3%, respectively. Beyond top-\n102:envelope. Specifically, it consistently accelerates per-user generation speeds by 60%–85% (V4-\n103:Flash) and 57%–78% (V4-Pro) at matched aggregate throughput capacities. Furthermore, under\n106:robust throughput. By overcoming this performance cliff, DSpark unlocks strict interactivity\n110:the DeepSeek-V4-Flash (preview) and DeepSeek-V4-Pro (preview) models. Furthermore, we\n111:open-source DeepSpec, an algorithm-driven training repository, including Eagle3, DFlash and\n150:attention, but the large number of verification tokens reduces overall serving throughput.\n155:Among them, DFlash (Chen et al., 2026) is a state-of-the-art parallel drafter, which conditions\n173:drafting requires only a single forward pass regardless of block size, DFlash can afford deeper\n179:predicted independently. Meanwhile, fixed-length verification wastes𝑇verify on low-confidence\n185:• Confidence-scheduled verification(Section 3.2). A confidence head estimates per-position\n186:acceptance probabilities, and a hardware-aware scheduler uses these estimates to prune\n187:low-confidence suffix tokens, cutting unnecessary verification compute.\n218:confidence scores 𝑐1–𝑐4 . The Hardware-Aware Prefix Scheduler then evaluates these scores to\n219:retain the prefix EFG and drop the low-confidence token H . Finally, the target model verifies\n233:Parallel stage.A parallel backbone (in our instantiation, DFlash (Chen et al., 2026)) runs a\n235:𝑈1,... ,𝑈𝛾. We make only a minor modification to the original DFlash backbone: instead of\n284:throughput, especially in high-concurrency scenarios (Hu et al., 2026; Liu et al., 2024c).\n299:achieves this by coupling aconfidence head(Section 3.2.1) that predicts prefix survival proba-\n300:bilities, with ahardware-aware prefix scheduler(Section 3.2.2) that dynamically determines the\n303:Drawing inspiration from Huang et al. (2024); Wang et al. (2026), the confidence head outputs a\n324:et al., 2024b; Zhang et al., 2026b), which only require confidence scores to correctly rank draft\n327:expected acceptance length𝜏. Because neural confidence estimates are often overconfident (Guo\n328:et al., 2017; Ovadia et al., 2019), using the raw confidence scores directly would distort the\n329:throughput estimation, leading to suboptimal scheduling.\n340:confidence head.\n347:Require: Active requests𝑟∈{ 1,... ,𝑅}; confidence sequence𝑐𝑟,1,... ,𝑐𝑟,𝛾 per request; profiled\n362:9:Current throughputΘ←𝜏 ∗·SPS(𝐵)\n374:confidence scores to determine verification length. While effective under isolated, single-request\n377:To address this, we formulate verification length selection as a global throughput maximiza-\n379:be the per-position confidence estimates, and letℓ𝑟∈{ 0,... ,𝛾} denote the scheduled verification\n389:\u0001. Let SPS(𝐵) denote the engine throughput, measured in steps per second, for\n391:initialization and stored as a lightweight cost table. Our scheduler then aims to maximize the\n392:expected system-wide token throughput Θ=𝜏·SPS(𝐵) by dynamically selecting verification\n409:sorted pool, updating the expected throughput Θ via an𝑂(1) lookup from the pre-profiled cost\n413:Because our confidence head relies on the Markov feature of the previously sampled token,\n418:To enforce strict causality, the scheduler (Algorithm 1) employs an early-stopping mech-\n419:anism. By breaking the greedy search immediately when the throughput drops ( Θ≤Θ best),\n422:this stepwise early-stopping yields the global maximum throughput if and only if the objective\n430:updating only the backbone drafter, sequential block, and confidence head.\n432:matching loss Ltv, and a confidence loss Lconf. All three are position-weighted by 𝑤𝑘 =\n455:the expected acceptance rate. The confidence lossLconf is a binary cross-entropy that trains the\n456:confidence head to predict the soft acceptance label𝑐∗\n477:the effectiveness of confidence scheduler under online production traffic in Section 5. The\n482:scales and model families: Qwen3-{4B, 8B, 14B} (Yang et al., 2025), and Gemma4-12B (Google\n484:DFlash (Chen et al., 2026), a state-of-the-art parallel drafter, and Eagle3 (Li et al., 2026b), an\n486:drafters in the same training framework and on the same data. We align Eagle3’s TTT horizon (7)\n487:with the block size (7) used by DFlash and DSpark, and we use the same target-model feature\n488:layers for all drafters. For the number of draft model layers, we set 1 for Eagle3 and 5 for DSpark\n489:and DFlash (Chen et al., 2026). Unless otherwise stated, DSpark denotes the Markov-head\n505:2023) with the sampling temperature set to 1.0. We report the accepted length (𝜏) per decoding\n508:3For clarity, unless otherwise stated, all reported metrics for accepted length and acceptance rate include the\n514:Table 1| Main speculative decoding results.We report accepted length ( 𝜏) per decoding\n518:Qwen3-4B\n519:Eagle3 5.14 4.62 3.92 3.69 4.16 3.77 2.39 2.26 2.55\n520:DFlash 5.40 4.85 4.15 4.40 4.74 4.18 3.07 2.96 2.83\n522:Qwen3-8B\n523:Eagle3 5.30 4.77 3.91 3.96 4.33 4.17 2.66 2.54 2.54\n524:DFlash 5.33 4.91 4.07 4.36 4.64 4.39 3.11 2.98 2.81\n526:Qwen3-14B\n527:Eagle3 5.24 4.60 3.71 3.81 4.14 4.01 2.62 2.47 2.48\n528:DFlash 5.41 4.84 3.98 4.44 4.59 4.33 3.10 2.94 2.72\n531:Eagle3 5.87 5.46 4.83 4.72 5.37 4.16 3.19 3.06 2.72\n532:DFlash 5.45 5.04 4.22 4.39 4.95 3.70 2.98 2.84 2.59\n536:disables the confidence scheduler, forcing all drafters to propose a fixed block of tokens. The\n537:main results, measured by the average accepted length (𝜏) per round, are reported in Table 1.\n538:DSpark consistently outperforms both the autoregressive baseline (Eagle3) and the parallel\n539:baseline (DFlash) across all evaluated target models and benchmark domains. Specifically,\n540:across the Qwen3-4B, 8B, and 14B models, DSpark improves the macro-average accepted length\n541:over Eagle3 by 30.9%, 26.7%, and 30.0%, respectively. Similarly, compared to DFlash, DSpark\n545:Beyond the average improvements, Table 1 reveals a strong domain effect: the accepted\n546:length is naturally higher on structured tasks (e.g., 5.57 on math and 5.12 on code for Qwen3-4B)\n549:This directly motivates our confidence-scheduled verification, which dynamically prunes the\n553:Table 1 presents a counter-intuitive observation: the parallel drafter (DFlash) and the semi-\n554:autoregressive drafter (DSpark) often yield longer accepted lengths than the fully autoregressive\n555:drafter (Eagle3). This finding contrasts with the standard expectation that step-by-step autore-\n558:To analyze this behavior, we examine performance beyond the macro-level accepted length.\n559:Using the Qwen3-4B target model and the benchmark sets described in Section 4.1, we introduce\n591:DFlash Eagle3 DSpark\n593:rate for each draft position, averaged across benchmarks within each domain using the Qwen3-\n596:sive drafter (Eagle3) remains stable or trends upward, while the parallel drafter (DFlash) suffers\n605:strictly from architectural capacity: autoregressive models like Eagle3 are constrained to shallow\n607:networks. This structural gap yields a substantial accuracy margin at position 1, with DFlash\n608:starting noticeably higher than Eagle3 (e.g., 0.88 vs. 0.81 on Math, and 0.72 vs. 0.53 on Chat).\n611:Consequently, this initial capacity advantage disproportionately boosts the final accepted length,\n617:Autoregressive models like Eagle3 effectively leverage this conditional certainty, maintaining or\n619:contrast, DFlash suffers from rapid acceptance decay, dropping from 0.87 to 0.78 on Code and\n641:Method DSpark DFlash\n645:DFlash baseline, highlighting the parameter efficiency of sequential modeling.\n683:DSpark (markov) DSpark (rnn) DFlash\n685:DFlash across various block sizes (left three panels). The rightmost panel demonstrates that the\n695:size 𝛾). Unless otherwise stated, all experiments in this section use Qwen3-4B as the target\n699:DSpark layers from 1 to 5, comparing it against a 5-layer DFlash baseline. Figure 3 aggregates the\n700:accepted lengths across the math, code, and chat domains. As expected, DSpark’s performance\n702:layers. Notably, a 2-layer DSpark outperforms the 5-layer DFlash baseline across all domains.\n713:three panels of Figure 4 show that DSpark consistently outperforms DFlash at every proposal\n715:parallel generation (DFlash) suffers from rapid acceptance decay (Figure 2), its marginal utility\n716:diminishes for long blocks. DSpark mitigates this decay, causing its relative gain over DFlash\n717:to grow. For instance, at𝛾= 7, DSpark improves the accepted length by 16% on math, 15% on\n729:0.2% to 1.3% to the full-round latency over the DFlash baseline, despite delivering up to a 30%\n730:improvement in accepted length.\n735:blind verification a waste of target compute. To evaluate whether the confidence head can\n737:Qwen3-4B. We validate the estimator in isolation here, reserving the hardware-aware prefix\n738:scheduler (Section 3.2.2) for live production evaluation in Section 5.\n740:the overall acceptance rate (line) across confidence thresholds. As the threshold increases, the\n742:rejected (hashed bars). This suggests that the confidence head can identify lower-value suffix\n768:60%\n778:confidence head effectively prunes tokens that would ultimately be rejected (hashed bars).\n802:Figure 6| The Reliability Diagram on Alpaca Dataset.While the raw confidence estimator\n806:different confidence bins.\n809:verifying low-confidence tokens incurs minimal opportunity cost under low concurrency, but\n811:hardware-aware prefix scheduler. As formulated in Section 3.2, maximizing system-level\n812:throughput requires the confidence model to exhibit both strong predictive discrimination\n816:3%–8%). Applying post-hoc STS (Section 3.2.1) mitigates this overconfidence, reducing the\n827:optimizations necessary to deploy the hardware-aware prefix scheduler (Section 3.2.2), and the\n830:The DSpark draft models are co-deployed with the preview versions of DeepSeek-V4-Flash and\n831:DeepSeek-V4-Pro (DeepSeek-AI, 2026). The parallel backbone comprises three MoE layers (Dai\n834:the confidence head is trained end-to-end alongside the draft model and subsequently calibrated\n860:To navigate the trade-offs among system compatibility, throughput, and algorithmic correct-\n861:ness, we adapt the scheduler to operate asynchronously. Because ZOS requires the batch size for\n864:using the confidence head outputs from two steps prior. Mechanically, the candidate tokens in\n865:the current step are still strictly sorted by their actual, up-to-date cumulative confidence scores;\n878:prevent the scheduler from being trapped in local minima by jagged SPS cliffs, we remove the\n885:barrier, maximizing physical throughput across hardware cliffs while preserving the exact target\n889:objectives: per-request latency and aggregate throughput (Kwon et al., 2023; Zhao et al., 2025a;\n899:simplifies: given a fixed concurrency limit, maximizing per-GPU total token throughput and\n902:To achieve this maximum throughput, the asynchronous scheduler (Section 5.2) actively\n913:to support this variable-length routing, allowing the dynamic scheduler to operate seamlessly\n926:DeepSeek-V4-Flash\n927:+51% throughput\n928:+60% TPS\n929:+661% throughput\n930:+85% TPS\n939:DeepSeek-V4-Pro\n942:+52% throughput\n943:+57% TPS\n944:+406% throughput\n945:+78% TPS\n947:Figure 7| Throughput vs. TPS.Aggregate output token throughput against per-request genera-\n949:observed throughput–interactivity frontier relative to the MTP-1 baseline under the measured\n953:1 (DeepSeek-AI, 2024) baseline within the production serving engines of DeepSeek-V4-Flash (pre-\n954:view) and DeepSeek-V4-Pro (preview). MTP-1 represents the former production setup, having\n957:drafter (e.g., MTP-3/5) strictly degrades aggregate throughput under high concurrency due to\n964:throughput and per-user generation speed (interactivity). To quantify DSpark’s behavior under\n968:For the V4-Flash engine, we evaluate the system at SLA anchors of 80 and 120 tok/s/user.\n969:At the moderate 80 tok/s/user SLA, DSpark improves aggregate throughput by 51% over the\n972:can sustain only a very small concurrent batch. Consequently, the relative throughput ratio\n974:throughput. We therefore interpret this high-SLA point primarily as evidence that DSpark\n976:over a well-utilized baseline. At matched practical throughput levels, which provide a more\n977:stable comparison, DSpark accelerates per-user generation speeds by 60% to 85%.\n982:The V4-Pro deployment shows the same pattern. At the moderate 35 tok/s/user SLA,\n983:DSpark improves aggregate throughput by 52%. At the stricter 50 tok/s/user SLA, MTP-1\n984:again enters a low-concurrency regime, yielding a nominal 406% relative throughput advantage\n985:for DSpark. As with V4-Flash, we treat this point as an indication that DSpark sustains useful\n986:throughput under an interactivity target that the baseline cannot efficiently support. At matched\n987:system capacities, DSpark delivers 57% to 78% faster per-user generation. Overall, these results\n988:show that DSpark shifts the observed throughput–interactivity frontier outward: it improves\n989:throughput in moderate-SLA regimes and, more importantly, preserves non-degenerate serving\n996:DeepSeek-V4-Flash\n1003:DeepSeek-V4-Pro\n1020:Figure 8| Load-adaptive throughput and verification budgets.Top row (a, b): Aggregate output\n1021:throughput across varying levels of system concurrency. Bottom row (c, d): The average target\n1022:verification budget allocated per request. As concurrent load increases, the dynamic scheduler\n1025:these gains by plotting aggregate throughput (top row) and the dynamic verification budget\n1028:than 200 concurrent requests for V4-Flash and 150 for V4-Pro), the hardware-aware sched-\n1032:throughput gains observed on the Pareto frontier.\n1033:• As system concurrency scales and target capacity saturates, the scheduler dynamically\n1035:suring that low-confidence draft tokens are pruned before they consume critical batch\n1043:Limitations.Although the prefix scheduler minimizes wasted target-model verification, DSpark\n1064:DFlash use diffusion-inspired prediction to generate entire blocks in a single forward pass (An\n1066:trees (Ringel and Romano, 2026). Concurrent efforts also improve DFlash: Domino (Huang et al.,\n1072:confidence heuristics (Du et al., 2024; Li et al., 2024b; Liu et al., 2026c; Mamou et al., 2024; Wen and\n1113:formulate verification length selection as a global throughput maximization problem, employing\n1114:a hardware-aware prefix scheduler that dynamically tailors the target model’s verification\n1399:Magicdec: Breaking the latency-throughput tradeoff for long context generation with spec-\n1466:Z. Wen and Y. Feng. Specbound: Adaptive bounded self-speculation with layer-wise confidence\n1510:A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3\n1560:large language model serving throughput. In Proceedings of the 19th USENIX Conference on\n1572:must be determined by scheduler-visible information available before the token𝑥𝑟,𝑘 is sampled.\n1574:(𝑅= 1) and maximum draft length (𝛾= 2). Suppose the pre-token confidence for the first position\n1577:The expected throughputs for verifying 0 and 1 draft tokens are\n1580:Without early-stopping, the scheduler proceeds to evaluate Θ2 before committing any admission\n1581:decisions. Because the Markov confidence head uses the previously sampled token, the next\n1582:confidence score𝑐2 explicitly depends on the realization of𝑥1. Consequently, the second-prefix\n1588:The expected throughput for length 2 is\n1590:Since Θ2 is the global maximum among{1.0, 0.9, 1.134}, the scheduler returnsℓ= 2. The\n1594:The expected throughput for length 2 is\n1596:Here, the global maximum remains Θ0 = 1.0, so the scheduler returnsℓ= 0. The first token\n1599:token itself. This retrospective dependence introduces selection bias: the scheduler favors tokens\n1614:matching the assumed value (𝑎1 = 0.8). Suppose the retrospective scheduler behaves as above:\n1615:𝑥1 =𝐴 yields a high continuation confidence and hence ℓ= 2, while 𝑥1 =𝐵 yields a low\n1616:continuation confidence and henceℓ= 0. Then the first output token is distributed as follows. If\n1635:that the retrospective scheduler is not lossless. The early-stopping mechanism prevents this\n1636:issue in the causal greedy scheduler. Since Θ1 <Θ 0, the scheduler halts immediately and returns\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_34","type":"agent_message","text":"The DSpark paper is usable now. The key app-worthy finding is the “why”: DSpark is not just another checkpoint row; it combines DFlash-style parallel block drafting with a tiny sequential dependency head, then schedules verification length by confidence and live serving load."}} -{"type":"item.started","item":{"id":"item_35","type":"command_execution","command":"/bin/zsh -lc \"sed -n '500,550p' /tmp/dspark_paper_clean.txt\nsed -n '920,990p' /tmp/dspark_paper_clean.txt\nsed -n '630,650p' /tmp/dspark_paper_clean.txt\nsed -n '680,732p' /tmp/dspark_paper_clean.txt\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_35","type":"command_execution","command":"/bin/zsh -lc \"sed -n '500,550p' /tmp/dspark_paper_clean.txt\nsed -n '920,990p' /tmp/dspark_paper_clean.txt\nsed -n '630,650p' /tmp/dspark_paper_clean.txt\nsed -n '680,732p' /tmp/dspark_paper_clean.txt\"","aggregated_output":"2. Code Generation, including MBPP (Austin et al., 2021b), HumanEval (Chen et al., 2021)\nand Live-CodeBench (Jain et al., 2025).\n3. Daily Chat, including MT-Bench (Zheng et al., 2023), Alpaca (Taori et al., 2023) and\nArena-Hard (Li et al., 2024a, 2025b).\nFor all benchmarks, we use standard speculative decoding (Chen et al., 2023; Leviathan et al.,\n2023) with the sampling temperature set to 1.0. We report the accepted length (𝜏) per decoding\nround3. For all drafters, we use chain-based drafting.\n2https://huggingface.co/datasets/mlabonne/open-perfectblend\n3For clarity, unless otherwise stated, all reported metrics for accepted length and acceptance rate include the\ntarget-generated bonus token.\n10\n\n\n--- PAGE 11 ---\nTable 1| Main speculative decoding results.We report accepted length ( 𝜏) per decoding\nround (higher is better) for different target models and domains.Boldmarks the best results.\nTarget Drafter Math Code Chat\nGSM8K MATH AIME25 MBPP HumanEval LCB MT-Bench Alpaca Arena-Hard\nQwen3-4B\nEagle3 5.14 4.62 3.92 3.69 4.16 3.77 2.39 2.26 2.55\nDFlash 5.40 4.85 4.15 4.40 4.74 4.18 3.07 2.96 2.83\nDSpark6.11 5.70 4.89 5.13 5.38 4.86 3.64 3.54 3.29\nQwen3-8B\nEagle3 5.30 4.77 3.91 3.96 4.33 4.17 2.66 2.54 2.54\nDFlash 5.33 4.91 4.07 4.36 4.64 4.39 3.11 2.98 2.81\nDSpark6.17 5.78 5.01 5.16 5.52 5.17 3.72 3.58 3.21\nQwen3-14B\nEagle3 5.24 4.60 3.71 3.81 4.14 4.01 2.62 2.47 2.48\nDFlash 5.41 4.84 3.98 4.44 4.59 4.33 3.10 2.94 2.72\nDSpark6.21 5.74 4.94 5.26 5.43 5.02 3.70 3.58 3.13\nGemma4-12B\nEagle3 5.87 5.46 4.83 4.72 5.37 4.16 3.19 3.06 2.72\nDFlash 5.45 5.04 4.22 4.39 4.95 3.70 2.98 2.84 2.59\nDSpark6.05 5.78 5.12 5.11 5.64 4.51 3.49 3.35 2.92\n4.2. Experimental Results\nTo isolate the raw draft quality from system-level scheduling policies, our offline evaluation\ndisables the confidence scheduler, forcing all drafters to propose a fixed block of tokens. The\nmain results, measured by the average accepted length (𝜏) per round, are reported in Table 1.\nDSpark consistently outperforms both the autoregressive baseline (Eagle3) and the parallel\nbaseline (DFlash) across all evaluated target models and benchmark domains. Specifically,\nacross the Qwen3-4B, 8B, and 14B models, DSpark improves the macro-average accepted length\nover Eagle3 by 30.9%, 26.7%, and 30.0%, respectively. Similarly, compared to DFlash, DSpark\nyields relative improvements of 16.3%, 18.4%, and 18.3% across the three scales. Crucially, this\nadvantage generalizes across model families, as demonstrated by the consistent performance\ngains on the Gemma4-12B target.\nBeyond the average improvements, Table 1 reveals a strong domain effect: the accepted\nlength is naturally higher on structured tasks (e.g., 5.57 on math and 5.12 on code for Qwen3-4B)\nthan on open-ended chat (3.49). This inherent variance in data predictability means a static\nverification length often wastes compute on trailing tokens that are highly likely to be rejected.\nThis directly motivates our confidence-scheduled verification, which dynamically prunes the\ndraft block based on expected acceptance.\n0\n5k\n10k\n15k\n20k\nThroughput (token/s/gpu)\nDeepSeek-V4-Flash\n+51% throughput\n+60% TPS\n+661% throughput\n+85% TPS\n20 40 60 80 100 120\n0\n1k\n2k\n3k\n4k\n5k\n6k\nDeepSeek-V4-Pro\nMTP\nDSpark\n+52% throughput\n+57% TPS\n+406% throughput\n+78% TPS\nTPS (token/s/user)\nFigure 7| Throughput vs. TPS.Aggregate output token throughput against per-request genera-\ntion speed (tok/s/user) under live traffic. In our production deployment, DSpark improves the\nobserved throughput–interactivity frontier relative to the MTP-1 baseline under the measured\ntraffic and engine configurations.\n5.4. Performance under Live User Traffic\nWe evaluate DSpark-5 (configured with a maximum draft length of 𝛾= 5) against the MTP-\n1 (DeepSeek-AI, 2024) baseline within the production serving engines of DeepSeek-V4-Flash (pre-\nview) and DeepSeek-V4-Pro (preview). MTP-1 represents the former production setup, having\nbeen superseded by DSpark two weeks following the DeepSeek-V4-preview release. This single-\ntoken setup was historically maintained in production because deploying a static multi-token\ndrafter (e.g., MTP-3/5) strictly degrades aggregate throughput under high concurrency due to\nexcessive verification overhead. Therefore, comparing DSpark against this established baseline\ndirectly demonstrates its ability to safely unlock the performance potential of larger draft blocks\nin dynamic serving environments. In all figures, the scatter points represent raw telemetry data\nsampled directly from live user traffic, capturing complex, real-world request distributions,\nwhile the solid lines represent the fitted performance frontiers.\nThe Serving Pareto Frontier.Figure 7 illustrates the trade-off between aggregate system\nthroughput and per-user generation speed (interactivity). To quantify DSpark’s behavior under\npractical deployment constraints, we evaluate the system at several interactivity SLA anchors.\nHere, an SLA (Service Level Agreement) specifies the minimum per-user generation speed (in\ntokens per second) that the system must guarantee.\nFor the V4-Flash engine, we evaluate the system at SLA anchors of 80 and 120 tok/s/user.\nAt the moderate 80 tok/s/user SLA, DSpark improves aggregate throughput by 51% over the\nMTP-1 baseline. The stricter 120 tok/s/user SLA represents a qualitatively different regime:\nunder this constraint, the single-token MTP-1 baseline approaches its operational boundary and\ncan sustain only a very small concurrent batch. Consequently, the relative throughput ratio\nat this point is numerically large, with DSpark achieving a nominal 661% higher aggregate\nthroughput. We therefore interpret this high-SLA point primarily as evidence that DSpark\nextends the feasible interactivity frontier, rather than as a representative multiplicative speedup\nover a well-utilized baseline. At matched practical throughput levels, which provide a more\nstable comparison, DSpark accelerates per-user generation speeds by 60% to 85%.\n18\n\n\n--- PAGE 19 ---\nThe V4-Pro deployment shows the same pattern. At the moderate 35 tok/s/user SLA,\nDSpark improves aggregate throughput by 52%. At the stricter 50 tok/s/user SLA, MTP-1\nagain enters a low-concurrency regime, yielding a nominal 406% relative throughput advantage\nfor DSpark. As with V4-Flash, we treat this point as an indication that DSpark sustains useful\nthroughput under an interactivity target that the baseline cannot efficiently support. At matched\nsystem capacities, DSpark delivers 57% to 78% faster per-user generation. Overall, these results\nshow that DSpark shifts the observed throughput–interactivity frontier outward: it improves\nthroughput in moderate-SLA regimes and, more importantly, preserves non-degenerate serving\ncapacity under strict interactivity constraints.\nGSM8K MATH500 AIME25 MBPP\nHumanEval\nLiveCodeBench\nMT-Bench Alpaca\nArena-Hard v2\n2\n3\n4\n5\n6\nAccepted Length\nMethod DSpark DFlash\nDepth 1L 2L 5L\nFigure 3| Effect of drafter depth.With proposal length fixed, DSpark’s performance improves\nas drafter layers are added. Notably, a shallow 2-layer DSpark outperforms a deeper 5-layer\nDFlash baseline, highlighting the parameter efficiency of sequential modeling.\n4 8 12 16\n4\n5\n6\n7Accepted Length\n+0.9%\n+1.3%\nDraft Length\nDSpark (markov) DSpark (rnn) DFlash\nFigure 4| Effect of proposal length and latency overhead.DSpark consistently outperforms\nDFlash across various block sizes (left three panels). The rightmost panel demonstrates that the\nsequential head introduces minimal latency overhead during serving.\nmotivates DSpark’s semi-autoregressive design. As shown in Figure 2, DSpark inherits the high\ninitial acceptance of the deep parallel drafter (e.g., starting at 0.93 on Math). Simultaneously, its\nlightweight sequential head mitigates the rapid acceptance decay typical of parallel generation.\nBy resolving this trade-off, DSpark maintains a high and stable conditional acceptance rate\nthroughout the entire draft block.\n4.3.2. A Little Autoregression Goes a Long Way\nBuilding on the insights from Section 4.3.1, we explore the architectural design space of DSpark\nalong two dimensions: drafter depth (number of transformer layers) and proposal length (block\nsize 𝛾). Unless otherwise stated, all experiments in this section use Qwen3-4B as the target\nmodel and follow the evaluation protocol detailed in Section 4.1.\nDrafter Depth.Increasing the number of transformer layers naturally expands a draft model’s\npredictive capacity. To isolate this effect, we fix the block size to 7 and vary the number of\nDSpark layers from 1 to 5, comparing it against a 5-layer DFlash baseline. Figure 3 aggregates the\naccepted lengths across the math, code, and chat domains. As expected, DSpark’s performance\nimproves monotonically with depth, with the steepest marginal gain occurring from one to two\nlayers. Notably, a 2-layer DSpark outperforms the 5-layer DFlash baseline across all domains.\n13\n\n\n--- PAGE 14 ---\nThis demonstrates that injecting local auto-regression via a lightweight sequential head offers a\nhighly favorable accuracy-parameter trade-off, achieving better sequence coherence than simply\nstacking deeper parallel layers.\nProposal Length.Next, we fix the drafter depth to 5 layers and scale the draft length (proposal\nlength 𝛾 plus one anchor token) across{4, 8, 12, 16} to evaluate performance on longer draft\nblocks. For DSpark, we evaluate both the default Markov head and the RNN head. The first\nthree panels of Figure 4 show that DSpark consistently outperforms DFlash at every proposal\nlength. More importantly, the performance gap steadily widens as𝛾 increases. Because pure\nparallel generation (DFlash) suffers from rapid acceptance decay (Figure 2), its marginal utility\ndiminishes for long blocks. DSpark mitigates this decay, causing its relative gain over DFlash\nto grow. For instance, at𝛾= 7, DSpark improves the accepted length by 16% on math, 15% on\ncode, and 18% on chat; at𝛾= 15, these gains expand to 30%, 26%, and 22%, respectively. Also,\nRNN head provides only marginal additional gains over the Markov head, mainly at longer\nproposal lengths. Given its higher implementation complexity and less favorable deployment\nproperties, we use the Markov head as the default.\nLatency Overhead.We quantify the overhead of the sequential generation loop in DSpark.\nThe rightmost panel of Figure 4 reports the per-round engine latency—comprising one target\nverification pass, the parallel draft block forward, and the serial sampling loop—measured\nat a batch size of 128. To prevent sequence-length bias, the reported latency represents the\narithmetic mean across varying context lengths ({512, 1024, 2048, 4096}tokens ). Since the target\nmodel dominates the verification compute time at this batch size, the sequential block’s latency\noverhead is negligible. Consequently, scaling the draft length from 4 to 16 adds a marginal\n0.2% to 1.3% to the full-round latency over the DFlash baseline, despite delivering up to a 30%\nimprovement in accepted length.\n4.3.3. Verify Smarter, Not Longer: The Role of Confidence Head\nWhile DSpark sustains high acceptance over long draft blocks, verifying the entire proposal\n","exit_code":0,"status":"completed"}} -{"type":"item.started","item":{"id":"item_36","type":"web_search","id":"ws_0ec1c722ea10094d016a422445a4a481918a5081f702630f85","query":"","action":{"type":"other"}}} -{"type":"item.completed","item":{"id":"item_36","type":"web_search","id":"ws_0ec1c722ea10094d016a422445a4a481918a5081f702630f85","query":"","action":{"type":"other"}}} -{"type":"item.started","item":{"id":"item_37","type":"web_search","id":"ws_0ec1c722ea10094d016a42244a24088191a92883538c1c1c5d","query":"","action":{"type":"other"}}} -{"type":"item.completed","item":{"id":"item_37","type":"web_search","id":"ws_0ec1c722ea10094d016a42244a24088191a92883538c1c1c5d","query":"https://ar5iv.labs.arxiv.org/html/2503.01840","action":{"type":"other"}}} -{"type":"item.started","item":{"id":"item_38","type":"web_search","id":"ws_0ec1c722ea10094d016a42244f12ec8191ae6aa1eadd7cdfc1","query":"","action":{"type":"other"}}} -{"type":"item.completed","item":{"id":"item_38","type":"web_search","id":"ws_0ec1c722ea10094d016a42244f12ec8191ae6aa1eadd7cdfc1","query":"https://ar5iv.labs.arxiv.org/html/2602.06036","action":{"type":"other"}}} -{"type":"item.started","item":{"id":"item_39","type":"web_search","id":"ws_0ec1c722ea10094d016a422456682c819180e853d4c9331abc","query":"","action":{"type":"other"}}} -{"type":"item.completed","item":{"id":"item_39","type":"web_search","id":"ws_0ec1c722ea10094d016a422456682c819180e853d4c9331abc","query":"https://ar5iv.org/abs/2602.06036","action":{"type":"other"}}} -{"type":"item.started","item":{"id":"item_40","type":"web_search","id":"ws_0ec1c722ea10094d016a422460d00881919bb7dd32d45d3c8d","query":"","action":{"type":"other"}}} -{"type":"item.completed","item":{"id":"item_40","type":"web_search","id":"ws_0ec1c722ea10094d016a422460d00881919bb7dd32d45d3c8d","query":"https://huggingface.co/deepseek-ai/dspark_qwen3_4b_block7","action":{"type":"other"}}} -{"type":"item.started","item":{"id":"item_41","type":"web_search","id":"ws_0ec1c722ea10094d016a4224670a808191a448ce6abbb07361","query":"","action":{"type":"other"}}} -{"type":"item.completed","item":{"id":"item_41","type":"web_search","id":"ws_0ec1c722ea10094d016a4224670a808191a448ce6abbb07361","query":"https://huggingface.co/deepseek-ai/eagle3_gemma4_12b_ttt7","action":{"type":"other"}}} -{"type":"item.started","item":{"id":"item_42","type":"command_execution","command":"/bin/zsh -lc 'which jq || true'","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_42","type":"command_execution","command":"/bin/zsh -lc 'which jq || true'","aggregated_output":"/usr/bin/jq\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_43","type":"agent_message","text":"I’ve confirmed the individual released checkpoint pages are sparse: they are BF16 safetensors repos, not hosted inference endpoints, and the model cards are absent. For the Space, I’m going to avoid pretending these can all run interactively on a free CPU Space; the stronger angle is an interactive research dashboard and speculative-decoding simulator backed by the paper’s metrics."}} -{"type":"item.started","item":{"id":"item_44","type":"command_execution","command":"/bin/zsh -lc 'for repo in deepseek-ai/dspark_qwen3_4b_block7 deepseek-ai/dspark_qwen3_8b_block7 deepseek-ai/dspark_qwen3_14b_block7 deepseek-ai/dspark_gemma4_12b_block7 deepseek-ai/dflash_qwen3_4b_block7 deepseek-ai/dflash_qwen3_8b_block7 deepseek-ai/dflash_qwen3_14b_block7 deepseek-ai/dflash_gemma4_12b_block7 deepseek-ai/eagle3_qwen3_4b_ttt7 deepseek-ai/eagle3_qwen3_8b_ttt7 deepseek-ai/eagle3_qwen3_14b_ttt7 deepseek-ai/eagle3_gemma4_12b_ttt7; do curl -L --silent --fail \"https://huggingface.co/api/models/$repo\" | jq -r '\"'[.id, (.safetensors.parameters|to_entries|map(.value)|add // null), .config.model_type, (.siblings|map(.rfilename)|join(\\\",\\\")), ((.cardData // {})|keys|join(\\\",\\\"))] | @tsv'; done\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_44","type":"command_execution","command":"/bin/zsh -lc 'for repo in deepseek-ai/dspark_qwen3_4b_block7 deepseek-ai/dspark_qwen3_8b_block7 deepseek-ai/dspark_qwen3_14b_block7 deepseek-ai/dspark_gemma4_12b_block7 deepseek-ai/dflash_qwen3_4b_block7 deepseek-ai/dflash_qwen3_8b_block7 deepseek-ai/dflash_qwen3_14b_block7 deepseek-ai/dflash_gemma4_12b_block7 deepseek-ai/eagle3_qwen3_4b_ttt7 deepseek-ai/eagle3_qwen3_8b_ttt7 deepseek-ai/eagle3_qwen3_14b_ttt7 deepseek-ai/eagle3_gemma4_12b_ttt7; do curl -L --silent --fail \"https://huggingface.co/api/models/$repo\" | jq -r '\"'[.id, (.safetensors.parameters|to_entries|map(.value)|add // null), .config.model_type, (.siblings|map(.rfilename)|join(\\\",\\\")), ((.cardData // {})|keys|join(\\\",\\\"))] | @tsv'; done\"","aggregated_output":"deepseek-ai/dspark_qwen3_4b_block7\t1393133569\tqwen3\t.gitattributes,config.json,model.safetensors\t\ndeepseek-ai/dspark_qwen3_8b_block7\t2371081729\tqwen3\t.gitattributes,config.json,model.safetensors\t\ndeepseek-ai/dspark_qwen3_14b_block7\t3416263169\tqwen3\t.gitattributes,config.json,model.safetensors\t\ndeepseek-ai/dspark_gemma4_12b_block7\t3430444550\tgemma4_text\t.gitattributes,config.json,model.safetensors\t\ndeepseek-ai/dflash_qwen3_4b_block7\t1315339520\tqwen3\t.gitattributes,config.json,model.safetensors\t\ndeepseek-ai/dflash_qwen3_8b_block7\t2293286144\tqwen3\t.gitattributes,config.json,model.safetensors\t\ndeepseek-ai/dflash_qwen3_14b_block7\t3338466560\tqwen3\t.gitattributes,config.json,model.safetensors\t\ndeepseek-ai/dflash_gemma4_12b_block7\t3296222725\tgemma4_text\t.gitattributes,config.json,model.safetensors\t\ndeepseek-ai/eagle3_qwen3_4b_ttt7\t927344896\tqwen3\t.gitattributes,config.json,model.safetensors\t\ndeepseek-ai/eagle3_qwen3_8b_ttt7\t1546666240\tqwen3\t.gitattributes,config.json,model.safetensors\t\ndeepseek-ai/eagle3_qwen3_14b_ttt7\t2053918976\tqwen3\t.gitattributes,config.json,model.safetensors\t\ndeepseek-ai/eagle3_gemma4_12b_ttt7\t2362269185\tgemma4_unified_text\t.gitattributes,config.json,model.safetensors\t\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_45","type":"agent_message","text":"The HF API confirms all 12 repos contain only `.gitattributes`, `config.json`, and `model.safetensors`, with exact parameter counts ranging from about 0.93B to 3.43B for the draft modules. I’m sampling the configs next to turn the family differences into concrete UI data rather than generic descriptions."}} -{"type":"item.started","item":{"id":"item_46","type":"command_execution","command":"/bin/zsh -lc 'curl -L --silent --fail https://huggingface.co/deepseek-ai/dspark_qwen3_4b_block7/raw/main/config.json | jq .'","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.started","item":{"id":"item_47","type":"command_execution","command":"/bin/zsh -lc 'curl -L --silent --fail https://huggingface.co/deepseek-ai/eagle3_qwen3_4b_ttt7/raw/main/config.json | jq .'","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.started","item":{"id":"item_48","type":"command_execution","command":"/bin/zsh -lc 'curl -L --silent --fail https://huggingface.co/deepseek-ai/dflash_qwen3_4b_block7/raw/main/config.json | jq .'","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.started","item":{"id":"item_49","type":"command_execution","command":"/bin/zsh -lc 'curl -L --silent --fail https://huggingface.co/deepseek-ai/dspark_gemma4_12b_block7/raw/main/config.json | jq .'","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_46","type":"command_execution","command":"/bin/zsh -lc 'curl -L --silent --fail https://huggingface.co/deepseek-ai/dspark_qwen3_4b_block7/raw/main/config.json | jq .'","aggregated_output":"{\n \"architectures\": [\n \"Qwen3DSparkModel\"\n ],\n \"attention_bias\": false,\n \"attention_dropout\": 0.0,\n \"block_size\": 7,\n \"bos_token_id\": 151643,\n \"confidence_head_with_markov\": true,\n \"dtype\": \"bfloat16\",\n \"enable_confidence_head\": true,\n \"eos_token_id\": 151645,\n \"head_dim\": 128,\n \"hidden_act\": \"silu\",\n \"hidden_size\": 2560,\n \"initializer_range\": 0.02,\n \"intermediate_size\": 9728,\n \"layer_types\": [\n \"full_attention\",\n \"full_attention\",\n \"full_attention\",\n \"full_attention\",\n \"full_attention\"\n ],\n \"markov_head_type\": \"vanilla\",\n \"markov_rank\": 256,\n \"mask_token_id\": 151669,\n \"max_position_embeddings\": 40960,\n \"max_window_layers\": 36,\n \"model_type\": \"qwen3\",\n \"num_anchors\": 512,\n \"num_attention_heads\": 32,\n \"num_hidden_layers\": 5,\n \"num_key_value_heads\": 8,\n \"num_target_layers\": 36,\n \"pad_token_id\": null,\n \"rms_norm_eps\": 0.000001,\n \"rope_parameters\": {\n \"rope_theta\": 1000000,\n \"rope_type\": \"default\"\n },\n \"sliding_window\": null,\n \"target_layer_ids\": [\n 1,\n 9,\n 17,\n 25,\n 33\n ],\n \"tie_word_embeddings\": false,\n \"transformers_version\": \"5.10.2\",\n \"use_cache\": true,\n \"use_sliding_window\": false,\n \"vocab_size\": 151936\n}\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_47","type":"command_execution","command":"/bin/zsh -lc 'curl -L --silent --fail https://huggingface.co/deepseek-ai/eagle3_qwen3_4b_ttt7/raw/main/config.json | jq .'","aggregated_output":"{\n \"architectures\": [\n \"Qwen3Eagle3Model\"\n ],\n \"attention_bias\": false,\n \"attention_dropout\": 0.0,\n \"bos_token_id\": 151643,\n \"draft_num_hidden_layers\": 1,\n \"dtype\": \"bfloat16\",\n \"eos_token_id\": 151645,\n \"head_dim\": 128,\n \"hidden_act\": \"silu\",\n \"hidden_size\": 2560,\n \"initializer_range\": 0.02,\n \"intermediate_size\": 9728,\n \"layer_types\": [\n \"full_attention\"\n ],\n \"max_position_embeddings\": 40960,\n \"max_window_layers\": 36,\n \"model_type\": \"qwen3\",\n \"num_attention_heads\": 32,\n \"num_hidden_layers\": 1,\n \"num_key_value_heads\": 8,\n \"num_target_layers\": 36,\n \"pad_token_id\": null,\n \"rms_norm_eps\": 0.000001,\n \"rope_parameters\": {\n \"rope_theta\": 1000000,\n \"rope_type\": \"default\"\n },\n \"sliding_window\": null,\n \"step_loss_decay\": 0.8,\n \"target_layer_ids\": [\n 1,\n 9,\n 17,\n 25,\n 33\n ],\n \"target_model_name_or_path\": \"Qwen/Qwen3-4B\",\n \"tie_word_embeddings\": false,\n \"transformers_version\": \"5.10.2\",\n \"ttt_length\": 7,\n \"use_cache\": true,\n \"use_sliding_window\": false,\n \"vocab_size\": 151936\n}\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_48","type":"command_execution","command":"/bin/zsh -lc 'curl -L --silent --fail https://huggingface.co/deepseek-ai/dflash_qwen3_4b_block7/raw/main/config.json | jq .'","aggregated_output":"{\n \"architectures\": [\n \"Qwen3DSparkModel\"\n ],\n \"attention_bias\": false,\n \"attention_dropout\": 0.0,\n \"block_size\": 7,\n \"bos_token_id\": 151643,\n \"dtype\": \"bfloat16\",\n \"enable_confidence_head\": false,\n \"eos_token_id\": 151645,\n \"head_dim\": 128,\n \"hidden_act\": \"silu\",\n \"hidden_size\": 2560,\n \"initializer_range\": 0.02,\n \"intermediate_size\": 9728,\n \"layer_types\": [\n \"full_attention\",\n \"full_attention\",\n \"full_attention\",\n \"full_attention\",\n \"full_attention\"\n ],\n \"markov_rank\": 0,\n \"mask_token_id\": 151669,\n \"max_position_embeddings\": 40960,\n \"max_window_layers\": 36,\n \"model_type\": \"qwen3\",\n \"num_anchors\": 512,\n \"num_attention_heads\": 32,\n \"num_hidden_layers\": 5,\n \"num_key_value_heads\": 8,\n \"num_target_layers\": 36,\n \"pad_token_id\": null,\n \"rms_norm_eps\": 0.000001,\n \"rope_parameters\": {\n \"rope_theta\": 1000000,\n \"rope_type\": \"default\"\n },\n \"sliding_window\": null,\n \"target_layer_ids\": [\n 1,\n 9,\n 17,\n 25,\n 33\n ],\n \"tie_word_embeddings\": false,\n \"transformers_version\": \"5.10.2\",\n \"use_cache\": true,\n \"use_sliding_window\": false,\n \"vocab_size\": 151936\n}\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_49","type":"command_execution","command":"/bin/zsh -lc 'curl -L --silent --fail https://huggingface.co/deepseek-ai/dspark_gemma4_12b_block7/raw/main/config.json | jq .'","aggregated_output":"{\n \"architectures\": [\n \"Gemma4DSparkModel\"\n ],\n \"attention_bias\": false,\n \"attention_dropout\": 0.0,\n \"attention_k_eq_v\": true,\n \"block_size\": 7,\n \"bos_token_id\": 2,\n \"confidence_head_with_markov\": true,\n \"dtype\": \"bfloat16\",\n \"enable_confidence_head\": true,\n \"enable_moe_block\": false,\n \"eos_token_id\": 1,\n \"final_logit_softcapping\": 30.0,\n \"global_head_dim\": 512,\n \"head_dim\": 256,\n \"hidden_activation\": \"gelu_pytorch_tanh\",\n \"hidden_size\": 3840,\n \"hidden_size_per_layer_input\": 0,\n \"initializer_range\": 0.02,\n \"intermediate_size\": 15360,\n \"layer_types\": [\n \"full_attention\",\n \"full_attention\",\n \"full_attention\",\n \"full_attention\",\n \"full_attention\"\n ],\n \"markov_head_type\": \"vanilla\",\n \"markov_rank\": 256,\n \"mask_token_id\": 4,\n \"max_position_embeddings\": 262144,\n \"model_type\": \"gemma4_text\",\n \"moe_intermediate_size\": null,\n \"num_anchors\": 512,\n \"num_attention_heads\": 16,\n \"num_experts\": null,\n \"num_global_key_value_heads\": 1,\n \"num_hidden_layers\": 5,\n \"num_key_value_heads\": 8,\n \"num_kv_shared_layers\": 0,\n \"num_target_layers\": 48,\n \"pad_token_id\": 0,\n \"rms_norm_eps\": 0.000001,\n \"rope_parameters\": {\n \"full_attention\": {\n \"partial_rotary_factor\": 0.25,\n \"rope_theta\": 1000000.0,\n \"rope_type\": \"proportional\"\n },\n \"rope_theta\": null,\n \"rope_type\": \"default\",\n \"sliding_attention\": {\n \"rope_theta\": 10000.0,\n \"rope_type\": \"default\"\n }\n },\n \"sliding_window\": 1024,\n \"target_layer_ids\": [\n 5,\n 17,\n 29,\n 41,\n 46\n ],\n \"target_model_type\": \"gemma4_unified\",\n \"target_text_model_type\": \"gemma4_unified_text\",\n \"tie_word_embeddings\": false,\n \"top_k_experts\": null,\n \"transformers_version\": \"5.10.2\",\n \"use_bidirectional_attention\": \"vision\",\n \"use_cache\": true,\n \"use_double_wide_mlp\": false,\n \"vocab_size\": 262144,\n \"vocab_size_per_layer_input\": 262144\n}\n","exit_code":0,"status":"completed"}} -{"type":"item.started","item":{"id":"item_50","type":"web_search","id":"ws_0ec1c722ea10094d016a4224c109288191ac941152ebe81f28","query":"","action":{"type":"other"}}} -{"type":"item.completed","item":{"id":"item_50","type":"web_search","id":"ws_0ec1c722ea10094d016a4224c109288191ac941152ebe81f28","query":"","action":{"type":"other"}}} -{"type":"item.started","item":{"id":"item_51","type":"web_search","id":"ws_0ec1c722ea10094d016a4224c3c44c8191851678678c1d6a06","query":"","action":{"type":"other"}}} -{"type":"item.completed","item":{"id":"item_51","type":"web_search","id":"ws_0ec1c722ea10094d016a4224c3c44c8191851678678c1d6a06","query":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-DSpark","action":{"type":"other"}}} -{"type":"item.completed","item":{"id":"item_52","type":"agent_message","text":"I have enough research to implement a first version. Plan:\n1. Build a Gradio “DeepSpec Decoding Lab” with the 12-checkpoint inventory, paper table metrics, architecture comparisons, production Pareto claims, and a deterministic speculative-decoding simulator.\n2. Push it early to `Mike0021/deepspec-decoding-lab` as a CPU Gradio Space.\n3. Stream build/runtime logs, make targeted fixes only from the first error, then verify with `gradio_client.view_api()` and a `/simulate` call."}} -{"type":"item.started","item":{"id":"item_53","type":"todo_list","items":[{"text":"Research DeepSpec models and linked papers","completed":true},{"text":"Implement Gradio Space files","completed":false},{"text":"Create and upload Hugging Face Space","completed":false},{"text":"Stream logs and fix first failure if needed","completed":false},{"text":"Verify live Space with gradio_client","completed":false}]}} -{"type":"item.completed","item":{"id":"item_54","type":"agent_message","text":"I’m adding the Space files now: `app.py` for the interactive dashboard/simulator, `README.md` with proper Space frontmatter and source notes, pinned `requirements.txt`, and `.hfignore` so the saved prompt/log do not get uploaded."}} -{"type":"item.started","item":{"id":"item_55","type":"file_change","changes":[{"path":"/root/code/deepspec-space/.hfignore","kind":"add"},{"path":"/root/code/deepspec-space/README.md","kind":"add"},{"path":"/root/code/deepspec-space/app.py","kind":"add"},{"path":"/root/code/deepspec-space/requirements.txt","kind":"add"}],"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_55","type":"file_change","changes":[{"path":"/root/code/deepspec-space/.hfignore","kind":"add"},{"path":"/root/code/deepspec-space/README.md","kind":"add"},{"path":"/root/code/deepspec-space/app.py","kind":"add"},{"path":"/root/code/deepspec-space/requirements.txt","kind":"add"}],"status":"completed"}} -{"type":"item.completed","item":{"id":"item_56","type":"agent_message","text":"The first implementation is in place. I’m doing the allowed local syntax check now, then I’ll create/upload the Space and use the live logs as the source of truth."}} -{"type":"item.started","item":{"id":"item_57","type":"command_execution","command":"/bin/zsh -lc 'python3 -m py_compile app.py'","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.started","item":{"id":"item_58","type":"command_execution","command":"/bin/zsh -lc \"rg --files -g '\"'!__pycache__'\"' -g '\"'!.git'\"'\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_58","type":"command_execution","command":"/bin/zsh -lc \"rg --files -g '\"'!__pycache__'\"' -g '\"'!.git'\"'\"","aggregated_output":"PROMPT.txt\nrequirements.txt\napp.py\nREADME.md\nrollout.jsonl\n","exit_code":0,"status":"completed"}} -{"type":"item.started","item":{"id":"item_59","type":"command_execution","command":"/bin/zsh -lc \"sed -n '1,80p' README.md && sed -n '1,20p' requirements.txt\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_59","type":"command_execution","command":"/bin/zsh -lc \"sed -n '1,80p' README.md && sed -n '1,20p' requirements.txt\"","aggregated_output":"---\ntitle: DeepSpec Decoding Lab\nemoji: ⚡\ncolorFrom: teal\ncolorTo: orange\nsdk: gradio\nsdk_version: 6.10.0\napp_file: app.py\nshort_description: Interactive DeepSpec speculative decoding lab\nstartup_duration_timeout: 1h\n---\n\n# DeepSpec Decoding Lab\n\nAn interactive research dashboard for the DeepSeek-AI DeepSpec collection:\nDSpark, DFlash, and EAGLE-3 draft modules across Qwen3 and Gemma4 targets.\n\nThe Space uses the DSpark paper's Table 1 accepted-length measurements, public\nHugging Face checkpoint metadata, and a deterministic speculative-decoding\nsimulator to make the collection easy to inspect without requiring a multi-GPU\nserving stack.\n\nSources:\n\n- DeepSpec collection: https://huggingface.co/collections/deepseek-ai/deepspec-6a410e3f1831ca8ca801b88b\n- DeepSpec repository: https://github.com/deepseek-ai/DeepSpec\n- DSpark paper: https://github.com/deepseek-ai/DeepSpec/blob/main/DSpark_paper.pdf\n- DFlash paper: https://arxiv.org/abs/2602.06036\n- EAGLE-3 paper: https://arxiv.org/abs/2503.01840\ngradio==6.10.0\nplotly==5.24.1\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_57","type":"command_execution","command":"/bin/zsh -lc 'python3 -m py_compile app.py'","aggregated_output":"","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_60","type":"agent_message","text":"`app.py` compiles cleanly. I’m creating the Space under `Mike0021/deepspec-decoding-lab` and uploading only the app files next."}} -{"type":"item.updated","item":{"id":"item_53","type":"todo_list","items":[{"text":"Research DeepSpec models and linked papers","completed":true},{"text":"Implement Gradio Space files","completed":true},{"text":"Create and upload Hugging Face Space","completed":false},{"text":"Stream logs and fix first failure if needed","completed":false},{"text":"Verify live Space with gradio_client","completed":false}]}} -{"type":"item.started","item":{"id":"item_61","type":"command_execution","command":"/bin/zsh -lc 'hf repos create Mike0021/deepspec-decoding-lab --type space --space-sdk gradio --exist-ok'","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_61","type":"command_execution","command":"/bin/zsh -lc 'hf repos create Mike0021/deepspec-decoding-lab --type space --space-sdk gradio --exist-ok'","aggregated_output":"repo_id=Mike0021/deepspec-decoding-lab url=https://huggingface.co/spaces/Mike0021/deepspec-decoding-lab\n","exit_code":0,"status":"completed"}} -{"type":"item.completed","item":{"id":"item_62","type":"agent_message","text":"The Space repo exists at `https://huggingface.co/spaces/Mike0021/deepspec-decoding-lab`. Uploading the initial Gradio build now."}} -{"type":"item.started","item":{"id":"item_63","type":"command_execution","command":"/bin/zsh -lc \"hf upload Mike0021/deepspec-decoding-lab . . --type space --exclude '.git/*' --exclude '__pycache__/*' --exclude '.venv/*' --exclude '*.pyc' --commit-message 'Initial DeepSpec decoding lab'\"","aggregated_output":"","exit_code":null,"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_63","type":"command_execution","command":"/bin/zsh -lc \"hf upload Mike0021/deepspec-decoding-lab . . --type space --exclude '.git/*' --exclude '__pycache__/*' --exclude '.venv/*' --exclude '*.pyc' --commit-message 'Initial DeepSpec decoding lab'\"","aggregated_output":"Error: Invalid value. Invalid metadata in README.md.\n- \"colorFrom\" must be one of [red, yellow, green, blue, indigo, purple, pink, gray]\n- \"colorTo\" must be one of [red, yellow, green, blue, indigo, purple, pink, gray]\nHint: set HF_DEBUG=1 as environment variable for full traceback.\n","exit_code":1,"status":"failed"}} -{"type":"item.completed","item":{"id":"item_64","type":"agent_message","text":"The first upload failed before build because HF README metadata only accepts a fixed color list. I’m making that single frontmatter fix, then I’ll upload again."}} -{"type":"item.started","item":{"id":"item_65","type":"file_change","changes":[{"path":"/root/code/deepspec-space/README.md","kind":"update"}],"status":"in_progress"}} -{"type":"item.completed","item":{"id":"item_65","type":"file_change","changes":[{"path":"/root/code/deepspec-space/README.md","kind":"update"}],"status":"completed"}} -{"type":"item.started","item":{"id":"item_66","type":"command_execution","command":"/bin/zsh -lc \"hf upload Mike0021/deepspec-decoding-lab . . --type space --exclude '.git/*' --exclude '__pycache__/*' --exclude '.venv/*' --exclude '*.pyc' --commit-message 'Initial DeepSpec decoding lab'\"","aggregated_output":"","exit_code":null,"status":"in_progress"}}