Spaces:
Running
Running
v0.5.0: more local model options and a model picker
Browse files- README.md +22 -4
- css/app.css +19 -1
- index.html +26 -2
- js/agent.js +5 -2
- js/config.js +41 -9
- js/lazy-weights.js +22 -12
- js/llm-engine.js +15 -5
- js/llm-worker.js +5 -4
- js/llm.js +7 -3
- js/main.js +87 -13
- js/setup.js +23 -3
- sw.js +1 -1
README.md
CHANGED
|
@@ -7,9 +7,13 @@ sdk: static
|
|
| 7 |
app_file: index.html
|
| 8 |
pinned: false
|
| 9 |
license: apache-2.0
|
| 10 |
-
short_description:
|
| 11 |
models:
|
| 12 |
- LiquidAI/LFM3-1.7B-ONNX
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
custom_headers:
|
| 14 |
cross-origin-embedder-policy: require-corp
|
| 15 |
cross-origin-opener-policy: same-origin
|
|
@@ -20,7 +24,7 @@ custom_headers:
|
|
| 20 |
|
| 21 |
An AI agent with its own Linux computer, entirely inside one web page.
|
| 22 |
|
| 23 |
-
- **Model:** [LFM3-1.7B](https://huggingface.co/LiquidAI/LFM3-1.7B)
|
| 24 |
- **Computer:** a real Debian (i386) machine running in the browser on [CheerpX](https://cheerpx.io), the engine behind [WebVM](https://webvm.io). It always runs on your device, whichever model you use.
|
| 25 |
- **Agent harness:** the model calls `run_command`, `write_file` and `read_file` tools. Commands run in the same terminal you see, so you can watch and take over at any time.
|
| 26 |
- **File viewer:** browse the machine's files and open code (syntax highlighted), text, Markdown, images, audio and video. HTML pages open as sandboxed live previews together with the stylesheets, scripts, images and data files they reference, and links between pages work.
|
|
@@ -28,9 +32,23 @@ An AI agent with its own Linux computer, entirely inside one web page.
|
|
| 28 |
|
| 29 |
**Requirements for the local model:** desktop Chrome or Edge with WebGPU and `shader-f16` support, about 2.5 GB of free disk space, and 8 GB or more RAM. Open the app at its own URL (`https://<owner>-localwebagent.static.hf.space`), because the embedded view on huggingface.co can't enable the cross-origin isolation CheerpX needs.
|
| 30 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
## Phones and low-memory devices
|
| 32 |
|
| 33 |
-
The
|
| 34 |
|
| 35 |
To make the local model fit on more devices, the weights are kept on disk and read tensor by tensor while they are copied to the GPU, instead of being held in memory all at once. On phones and devices reporting less than 8 GB of RAM, the model also loads before Linux starts, so the two don't compete for memory.
|
| 36 |
|
|
@@ -59,6 +77,6 @@ The key stays only until you close the tab, unless you turn on "Remember key on
|
|
| 59 |
## Licenses
|
| 60 |
|
| 61 |
- This app: Apache-2.0.
|
| 62 |
-
- LFM3-1.7B: [LFM Open License v1.0](https://huggingface.co/LiquidAI/LFM3-1.7B-ONNX/blob/main/LICENSE).
|
| 63 |
- CheerpX: proprietary, free for personal and open-source projects. It is **not** included in this Space. Each visitor's browser loads it from Leaning Technologies' CDN and caches it locally.
|
| 64 |
- The Debian disk image (`disk/`) contains Debian packages under their respective free licenses.
|
|
|
|
| 7 |
app_file: index.html
|
| 8 |
pinned: false
|
| 9 |
license: apache-2.0
|
| 10 |
+
short_description: Local AI agent driving a real Linux VM, fully offline
|
| 11 |
models:
|
| 12 |
- LiquidAI/LFM3-1.7B-ONNX
|
| 13 |
+
- onnx-community/Qwen3.5-0.8B-Text-ONNX
|
| 14 |
+
- onnx-community/Qwen3-1.7B-ONNX
|
| 15 |
+
- onnx-community/granite-4.0-1b-ONNX-web
|
| 16 |
+
- onnx-community/Qwen3-4B-ONNX
|
| 17 |
custom_headers:
|
| 18 |
cross-origin-embedder-policy: require-corp
|
| 19 |
cross-origin-opener-policy: same-origin
|
|
|
|
| 24 |
|
| 25 |
An AI agent with its own Linux computer, entirely inside one web page.
|
| 26 |
|
| 27 |
+
- **Model:** [LFM3-1.7B](https://huggingface.co/LiquidAI/LFM3-1.7B) by default, or one of [several other local models](#local-models), running on your GPU through WebGPU with [transformers.js](https://github.com/huggingface/transformers.js). If your device can't run them, you can [use an API model instead](#use-an-api-model).
|
| 28 |
- **Computer:** a real Debian (i386) machine running in the browser on [CheerpX](https://cheerpx.io), the engine behind [WebVM](https://webvm.io). It always runs on your device, whichever model you use.
|
| 29 |
- **Agent harness:** the model calls `run_command`, `write_file` and `read_file` tools. Commands run in the same terminal you see, so you can watch and take over at any time.
|
| 30 |
- **File viewer:** browse the machine's files and open code (syntax highlighted), text, Markdown, images, audio and video. HTML pages open as sandboxed live previews together with the stylesheets, scripts, images and data files they reference, and links between pages work.
|
|
|
|
| 32 |
|
| 33 |
**Requirements for the local model:** desktop Chrome or Edge with WebGPU and `shader-f16` support, about 2.5 GB of free disk space, and 8 GB or more RAM. Open the app at its own URL (`https://<owner>-localwebagent.static.hf.space`), because the embedded view on huggingface.co can't enable the cross-origin isolation CheerpX needs.
|
| 34 |
|
| 35 |
+
## Local models
|
| 36 |
+
|
| 37 |
+
Pick one in **Settings → Model → Change**, or with "Choose a different local model" on the setup screen. All are 4-bit (q4f16) ONNX builds loaded straight from their Hugging Face repositories; switching downloads the new model once, and the model picker can delete models you no longer use.
|
| 38 |
+
|
| 39 |
+
| Model | Download | Context | Reasoning | Notes |
|
| 40 |
+
|---|---:|---:|:---:|---|
|
| 41 |
+
| [LFM3 1.7B](https://huggingface.co/LiquidAI/LFM3-1.7B-ONNX) (default) | 1.38 GB | 256k | yes | Fast hybrid model by Liquid AI, good at tool use. |
|
| 42 |
+
| [Qwen3.5 0.8B](https://huggingface.co/onnx-community/Qwen3.5-0.8B-Text-ONNX) | 0.47 GB | 256k | yes | Smallest. Best chance on weak GPUs and phones; less capable. |
|
| 43 |
+
| [Qwen3 1.7B](https://huggingface.co/onnx-community/Qwen3-1.7B-ONNX) | 1.43 GB | 40k | yes | Similar size to LFM3, strong reasoning. |
|
| 44 |
+
| [Granite 4.0 1B](https://huggingface.co/onnx-community/granite-4.0-1b-ONNX-web) | 1.25 GB | 128k | no | IBM's hybrid model built for tool calling. |
|
| 45 |
+
| [Qwen3 4B](https://huggingface.co/onnx-community/Qwen3-4B-ONNX) | 2.83 GB | 40k | yes | Most capable, but slow. Needs a GPU with about 4 GB of memory. |
|
| 46 |
+
|
| 47 |
+
The context size setting is capped at each model's own window.
|
| 48 |
+
|
| 49 |
## Phones and low-memory devices
|
| 50 |
|
| 51 |
+
The default model needs about 1.4 GB of graphics memory. Most phones can't give that much to a browser tab, so the app warns you on phones and suggests an API model or the smaller Qwen3.5 0.8B instead. You can still try any local model. If it fails to load, the app explains why (out of graphics memory, no WebGPU support, or a failed download), keeps Linux running, and offers to switch to an API model, pick a smaller model, or try again. The raw error is under "Technical details".
|
| 52 |
|
| 53 |
To make the local model fit on more devices, the weights are kept on disk and read tensor by tensor while they are copied to the GPU, instead of being held in memory all at once. On phones and devices reporting less than 8 GB of RAM, the model also loads before Linux starts, so the two don't compete for memory.
|
| 54 |
|
|
|
|
| 77 |
## Licenses
|
| 78 |
|
| 79 |
- This app: Apache-2.0.
|
| 80 |
+
- LFM3-1.7B: [LFM Open License v1.0](https://huggingface.co/LiquidAI/LFM3-1.7B-ONNX/blob/main/LICENSE). Qwen3, Qwen3.5 and Granite 4.0: Apache-2.0. No weights are re-hosted here: your browser downloads them directly from the repositories linked above.
|
| 81 |
- CheerpX: proprietary, free for personal and open-source projects. It is **not** included in this Space. Each visitor's browser loads it from Leaning Technologies' CDN and caches it locally.
|
| 82 |
- The Debian disk image (`disk/`) contains Debian packages under their respective free licenses.
|
css/app.css
CHANGED
|
@@ -497,7 +497,25 @@ dialog[open] { animation: pop .14s ease-out; }
|
|
| 497 |
.field-hint { margin: -4px 0 8px; font-size: 12px; color: var(--text-3); }
|
| 498 |
.note.ok { color: var(--ok); }
|
| 499 |
.dlg-foot { display: flex; align-items: center; gap: 8px; padding: 12px 22px 18px; border-top: 1px solid var(--border); }
|
| 500 |
-
#api-dlg { max-height: calc(100dvh - 32px); overflow-y: auto; }
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 501 |
|
| 502 |
/* ---------- narrow screens ---------- */
|
| 503 |
@media (max-width: 900px) {
|
|
|
|
| 497 |
.field-hint { margin: -4px 0 8px; font-size: 12px; color: var(--text-3); }
|
| 498 |
.note.ok { color: var(--ok); }
|
| 499 |
.dlg-foot { display: flex; align-items: center; gap: 8px; padding: 12px 22px 18px; border-top: 1px solid var(--border); }
|
| 500 |
+
#api-dlg, #model-dlg { max-height: calc(100dvh - 32px); overflow-y: auto; }
|
| 501 |
+
#model-dlg { width: min(560px, calc(100% - 32px)); }
|
| 502 |
+
.model-list { display: flex; flex-direction: column; gap: 8px; }
|
| 503 |
+
.model-opt {
|
| 504 |
+
display: flex; gap: 12px; align-items: flex-start; padding: 12px 14px; cursor: pointer;
|
| 505 |
+
border: 1px solid var(--border); border-radius: var(--radius); background: var(--surface);
|
| 506 |
+
}
|
| 507 |
+
.model-opt:hover { background: var(--surface-2); }
|
| 508 |
+
.model-opt:has(input:checked) { border-color: var(--primary); box-shadow: 0 0 0 1px var(--primary); }
|
| 509 |
+
.model-opt input { margin: 3px 0 0; accent-color: var(--primary); flex: none; }
|
| 510 |
+
.mo-main { display: flex; flex-direction: column; gap: 2px; min-width: 0; }
|
| 511 |
+
.mo-name { font-weight: 600; display: flex; flex-wrap: wrap; align-items: center; gap: 6px; }
|
| 512 |
+
.mo-tag { font-size: 10.5px; font-weight: 500; color: var(--text-2); border: 1px solid var(--border-strong); border-radius: 99px; padding: 0 7px; }
|
| 513 |
+
.mo-note { font-size: 12.5px; color: var(--text-2); }
|
| 514 |
+
.mo-meta { font-size: 11.5px; color: var(--text-3); }
|
| 515 |
+
.mo-state { font-size: 11.5px; color: var(--text-3); display: flex; gap: 10px; align-items: center; margin-top: 2px; }
|
| 516 |
+
.mo-dl { color: var(--ok); }
|
| 517 |
+
.linkish { border: 0; background: none; padding: 0; color: var(--bad); font-size: 11.5px; text-decoration: underline; cursor: pointer; }
|
| 518 |
+
#tog-thinking:disabled { opacity: .45; cursor: not-allowed; }
|
| 519 |
|
| 520 |
/* ---------- narrow screens ---------- */
|
| 521 |
@media (max-width: 900px) {
|
index.html
CHANGED
|
@@ -135,7 +135,7 @@
|
|
| 135 |
<svg><use href="#i-alert"/></svg>
|
| 136 |
<div>
|
| 137 |
<strong>Phones usually can't run the local model.</strong>
|
| 138 |
-
It needs about 1.4 GB of graphics memory, more than most phones give a browser tab. Use an API model instead: the Linux machine still runs on your phone, only the model runs on the provider's servers. You can also
|
| 139 |
</div>
|
| 140 |
</div>
|
| 141 |
<h2>Install for offline use</h2>
|
|
@@ -147,7 +147,7 @@
|
|
| 147 |
<div class="section-label">Components</div>
|
| 148 |
<div class="components">
|
| 149 |
<div class="component" data-row="model" data-state="todo">
|
| 150 |
-
<div class="c-main"><span class="c-name">LFM3
|
| 151 |
<div class="c-side"><span class="c-size mono" id="model-size"></span><span class="row-state">Not installed</span></div>
|
| 152 |
<progress max="1" value="0"></progress>
|
| 153 |
</div>
|
|
@@ -165,6 +165,7 @@
|
|
| 165 |
|
| 166 |
<button class="btn primary block" id="install" disabled>Install</button>
|
| 167 |
<button class="btn block secondary" id="setup-api" type="button">Use an API model instead</button>
|
|
|
|
| 168 |
<p class="note" id="setup-note"></p>
|
| 169 |
<p class="fine">
|
| 170 |
The model is <a href="https://huggingface.co/LiquidAI/LFM3-1.7B" target="_blank" rel="noopener">LFM3-1.7B</a> (LFM Open License), loaded from Hugging Face.
|
|
@@ -223,6 +224,29 @@
|
|
| 223 |
</form>
|
| 224 |
</dialog>
|
| 225 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 226 |
<dialog id="api-dlg" aria-labelledby="api-title">
|
| 227 |
<form id="api-form" novalidate>
|
| 228 |
<div class="dlg-head">
|
|
|
|
| 135 |
<svg><use href="#i-alert"/></svg>
|
| 136 |
<div>
|
| 137 |
<strong>Phones usually can't run the local model.</strong>
|
| 138 |
+
It needs about 1.4 GB of graphics memory, more than most phones give a browser tab. Use an API model instead: the Linux machine still runs on your phone, only the model runs on the provider's servers. You can also pick the smaller Qwen3.5 0.8B (0.47 GB) and try it anyway.
|
| 139 |
</div>
|
| 140 |
</div>
|
| 141 |
<h2>Install for offline use</h2>
|
|
|
|
| 147 |
<div class="section-label">Components</div>
|
| 148 |
<div class="components">
|
| 149 |
<div class="component" data-row="model" data-state="todo">
|
| 150 |
+
<div class="c-main"><span class="c-name">LFM3 1.7B</span><span class="c-desc">Language model by Liquid AI, q4f16 ONNX</span></div>
|
| 151 |
<div class="c-side"><span class="c-size mono" id="model-size"></span><span class="row-state">Not installed</span></div>
|
| 152 |
<progress max="1" value="0"></progress>
|
| 153 |
</div>
|
|
|
|
| 165 |
|
| 166 |
<button class="btn primary block" id="install" disabled>Install</button>
|
| 167 |
<button class="btn block secondary" id="setup-api" type="button">Use an API model instead</button>
|
| 168 |
+
<button class="btn block secondary" id="setup-model" type="button">Choose a different local model</button>
|
| 169 |
<p class="note" id="setup-note"></p>
|
| 170 |
<p class="fine">
|
| 171 |
The model is <a href="https://huggingface.co/LiquidAI/LFM3-1.7B" target="_blank" rel="noopener">LFM3-1.7B</a> (LFM Open License), loaded from Hugging Face.
|
|
|
|
| 224 |
</form>
|
| 225 |
</dialog>
|
| 226 |
|
| 227 |
+
<dialog id="model-dlg" aria-labelledby="model-title">
|
| 228 |
+
<div class="dlg-head">
|
| 229 |
+
<h2 id="model-title">Choose a model</h2>
|
| 230 |
+
<button class="icon-btn" type="button" id="model-close" aria-label="Close"><svg><use href="#i-x"/></svg></button>
|
| 231 |
+
</div>
|
| 232 |
+
<div class="dlg-section">
|
| 233 |
+
<div class="section-label">On this device</div>
|
| 234 |
+
<p class="dlg-lede">Runs on your GPU with WebGPU and works offline once downloaded. Bigger models are more capable but slower and need more graphics memory.</p>
|
| 235 |
+
<div class="model-list" id="model-list"></div>
|
| 236 |
+
</div>
|
| 237 |
+
<div class="dlg-section">
|
| 238 |
+
<div class="section-label">Through an API</div>
|
| 239 |
+
<div class="row-setting">
|
| 240 |
+
<span><span class="rs-title">API model</span><span class="rs-desc" id="model-api-desc"></span></span>
|
| 241 |
+
<button type="button" class="btn" id="model-api">Set up</button>
|
| 242 |
+
</div>
|
| 243 |
+
</div>
|
| 244 |
+
<div class="dlg-foot">
|
| 245 |
+
<span class="spacer"></span>
|
| 246 |
+
<button type="button" class="btn primary" id="model-save">Use this model</button>
|
| 247 |
+
</div>
|
| 248 |
+
</dialog>
|
| 249 |
+
|
| 250 |
<dialog id="api-dlg" aria-labelledby="api-title">
|
| 251 |
<form id="api-form" novalidate>
|
| 252 |
<div class="dlg-head">
|
js/agent.js
CHANGED
|
@@ -53,7 +53,7 @@ export class Agent {
|
|
| 53 |
for (let step = 1; step <= this.settings.maxSteps; step++) {
|
| 54 |
if (this.stopRequested) break;
|
| 55 |
this.#compact();
|
| 56 |
-
const thinking = this.settings.thinking;
|
| 57 |
const view = this.ui.startTurn(step);
|
| 58 |
let raw = "";
|
| 59 |
const res = await this.llm.generate(
|
|
@@ -81,7 +81,10 @@ export class Agent {
|
|
| 81 |
}
|
| 82 |
// Store exactly what was generated (plus the "<think>" the prompt ended with),
|
| 83 |
// so the next prompt re-tokenizes to the same prefix and the KV cache is reused.
|
| 84 |
-
|
|
|
|
|
|
|
|
|
|
| 85 |
|
| 86 |
const calls = parsed.calls.filter((c) => c.complete || !res.hitLimit);
|
| 87 |
if (!calls.length) {
|
|
|
|
| 53 |
for (let step = 1; step <= this.settings.maxSteps; step++) {
|
| 54 |
if (this.stopRequested) break;
|
| 55 |
this.#compact();
|
| 56 |
+
const thinking = this.settings.thinking && this.llm.supportsThinking !== false;
|
| 57 |
const view = this.ui.startTurn(step);
|
| 58 |
let raw = "";
|
| 59 |
const res = await this.llm.generate(
|
|
|
|
| 81 |
}
|
| 82 |
// Store exactly what was generated (plus the "<think>" the prompt ended with),
|
| 83 |
// so the next prompt re-tokenizes to the same prefix and the KV cache is reused.
|
| 84 |
+
// Some chat templates end the prompt with "<think>" (LFM3), others let the model
|
| 85 |
+
// write it (Qwen3); don't double it.
|
| 86 |
+
const prefix = /^\s*<think>/.test(text) ? "" : thinking ? "<think>" : "<think></think>";
|
| 87 |
+
this.history.push({ role: "assistant", content: prefix + text });
|
| 88 |
|
| 89 |
const calls = parsed.calls.filter((c) => c.complete || !res.hitLimit);
|
| 90 |
if (!calls.length) {
|
js/config.js
CHANGED
|
@@ -1,17 +1,49 @@
|
|
| 1 |
// Central configuration. Everything that pins a version or a location lives here.
|
| 2 |
-
export const APP_VERSION = "0.
|
| 3 |
|
| 4 |
-
// ---
|
| 5 |
-
|
| 6 |
-
|
|
|
|
| 7 |
export const MODEL_DEVICE = "webgpu";
|
| 8 |
-
|
| 9 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
"config.json", "generation_config.json", "tokenizer.json", "tokenizer_config.json",
|
| 11 |
-
`onnx/model_${
|
| 12 |
];
|
| 13 |
-
export const
|
| 14 |
-
export const hubUrl = (file) => `https://huggingface.co/${MODEL_ID}/resolve/main/${file}`;
|
| 15 |
|
| 16 |
// --- CheerpX (WebVM engine). Proprietary: loaded from Leaning Technologies' CDN,
|
| 17 |
// never redistributed by this Space. The visitor's browser caches it for offline use.
|
|
|
|
| 1 |
// Central configuration. Everything that pins a version or a location lives here.
|
| 2 |
+
export const APP_VERSION = "0.5.0";
|
| 3 |
|
| 4 |
+
// --- Local language models (loaded from the Hugging Face Hub, cached in Cache Storage) ---
|
| 5 |
+
// Each runs on WebGPU through transformers.js. `chunks` is how many external weight files
|
| 6 |
+
// (model_<dtype>.onnx_data, _data_1, ...) the ONNX graph uses; 0 means the weights are inside
|
| 7 |
+
// the .onnx file itself. `bytes` is the approximate download, `contextMax` the model's window.
|
| 8 |
export const MODEL_DEVICE = "webgpu";
|
| 9 |
+
export const LOCAL_MODELS = [
|
| 10 |
+
{
|
| 11 |
+
id: "lfm3-1.7b", label: "LFM3 1.7B", repo: "LiquidAI/LFM3-1.7B-ONNX", dtype: "q4f16", chunks: 1,
|
| 12 |
+
bytes: 1_383_000_000, contextMax: 262_144, thinking: true, by: "Liquid AI", license: "LFM Open License",
|
| 13 |
+
note: "The default. Fast hybrid model, good at tool use.",
|
| 14 |
+
},
|
| 15 |
+
{
|
| 16 |
+
id: "qwen3.5-0.8b", label: "Qwen3.5 0.8B", repo: "onnx-community/Qwen3.5-0.8B-Text-ONNX", dtype: "q4f16", chunks: 1,
|
| 17 |
+
bytes: 470_000_000, contextMax: 262_144, thinking: true, by: "Qwen", license: "Apache-2.0",
|
| 18 |
+
note: "Smallest download and GPU memory use. Best chance on weak GPUs and phones; less capable.",
|
| 19 |
+
},
|
| 20 |
+
{
|
| 21 |
+
id: "qwen3-1.7b", label: "Qwen3 1.7B", repo: "onnx-community/Qwen3-1.7B-ONNX", dtype: "q4f16", chunks: 0,
|
| 22 |
+
bytes: 1_426_000_000, contextMax: 40_960, thinking: true, by: "Qwen", license: "Apache-2.0",
|
| 23 |
+
note: "Similar size to LFM3, strong reasoning.",
|
| 24 |
+
},
|
| 25 |
+
{
|
| 26 |
+
id: "granite-4.0-1b", label: "Granite 4.0 1B", repo: "onnx-community/granite-4.0-1b-ONNX-web", dtype: "q4f16", chunks: 1,
|
| 27 |
+
bytes: 1_247_000_000, contextMax: 131_072, thinking: false, by: "IBM", license: "Apache-2.0",
|
| 28 |
+
note: "Hybrid model built for tool calling. No reasoning mode.",
|
| 29 |
+
},
|
| 30 |
+
{
|
| 31 |
+
id: "qwen3-4b", label: "Qwen3 4B", repo: "onnx-community/Qwen3-4B-ONNX", dtype: "q4f16", chunks: 2,
|
| 32 |
+
bytes: 2_833_000_000, contextMax: 40_960, thinking: true, by: "Qwen", license: "Apache-2.0",
|
| 33 |
+
note: "Most capable, but slow. Needs a desktop GPU with about 4 GB of memory.",
|
| 34 |
+
},
|
| 35 |
+
];
|
| 36 |
+
export const DEFAULT_MODEL_ID = LOCAL_MODELS[0].id;
|
| 37 |
+
export const findModel = (id) => LOCAL_MODELS.find((m) => m.id === id) ?? LOCAL_MODELS[0];
|
| 38 |
+
|
| 39 |
+
export const dataFileNames = (m) =>
|
| 40 |
+
Array.from({ length: m.chunks }, (_, i) => `model_${m.dtype}.onnx_data${i === 0 ? "" : "_" + i}`);
|
| 41 |
+
// Files transformers.js fetches for a model; used to check the offline cache.
|
| 42 |
+
export const modelFiles = (m) => [
|
| 43 |
"config.json", "generation_config.json", "tokenizer.json", "tokenizer_config.json",
|
| 44 |
+
`onnx/model_${m.dtype}.onnx`, ...dataFileNames(m).map((f) => "onnx/" + f),
|
| 45 |
];
|
| 46 |
+
export const hubUrl = (m, file) => `https://huggingface.co/${m.repo}/resolve/main/${file}`;
|
|
|
|
| 47 |
|
| 48 |
// --- CheerpX (WebVM engine). Proprietary: loaded from Leaning Technologies' CDN,
|
| 49 |
// never redistributed by this Space. The visitor's browser caches it for offline use.
|
js/lazy-weights.js
CHANGED
|
@@ -53,16 +53,10 @@ async function download(cache, url, file, progress) {
|
|
| 53 |
progress?.({ status: "done", file, loaded, total: total || loaded });
|
| 54 |
}
|
| 55 |
|
| 56 |
-
/**
|
| 57 |
-
|
| 58 |
-
* lazily from Cache Storage, downloading them first if needed.
|
| 59 |
-
*/
|
| 60 |
-
export async function lazyWeightsOptions(modelId, dtype, progress) {
|
| 61 |
-
if (typeof FileReaderSync === "undefined" || !self.caches) throw new Error("lazy weights unsupported here");
|
| 62 |
-
const name = `model_${dtype}.onnx_data`;
|
| 63 |
const file = `onnx/${name}`;
|
| 64 |
const url = `https://huggingface.co/${modelId}/resolve/main/${file}`;
|
| 65 |
-
const cache = await caches.open(CACHE_NAME);
|
| 66 |
let res = await cache.match(url);
|
| 67 |
if (!res) {
|
| 68 |
await download(cache, url, file, progress);
|
|
@@ -73,11 +67,27 @@ export async function lazyWeightsOptions(modelId, dtype, progress) {
|
|
| 73 |
progress?.({ status: "done", file, loaded: size, total: size });
|
| 74 |
}
|
| 75 |
const blob = await res.blob(); // disk-backed in Chromium; not read into memory
|
| 76 |
-
if (!blob.size) throw new Error(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 77 |
return {
|
| 78 |
-
// Stop transformers.js from fetching the .onnx_data itself...
|
| 79 |
use_external_data_format: false,
|
| 80 |
-
// ...and mount ours under the
|
| 81 |
-
session_options: { externalData
|
| 82 |
};
|
| 83 |
}
|
|
|
|
| 53 |
progress?.({ status: "done", file, loaded, total: total || loaded });
|
| 54 |
}
|
| 55 |
|
| 56 |
+
/** Gets one external-data file as a disk-backed BlobBytes, downloading it first if needed. */
|
| 57 |
+
async function lazyFile(cache, modelId, name, progress) {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
const file = `onnx/${name}`;
|
| 59 |
const url = `https://huggingface.co/${modelId}/resolve/main/${file}`;
|
|
|
|
| 60 |
let res = await cache.match(url);
|
| 61 |
if (!res) {
|
| 62 |
await download(cache, url, file, progress);
|
|
|
|
| 67 |
progress?.({ status: "done", file, loaded: size, total: size });
|
| 68 |
}
|
| 69 |
const blob = await res.blob(); // disk-backed in Chromium; not read into memory
|
| 70 |
+
if (!blob.size) throw new Error(`cached ${name} is empty`);
|
| 71 |
+
return new BlobBytes(blob);
|
| 72 |
+
}
|
| 73 |
+
|
| 74 |
+
/**
|
| 75 |
+
* Returns from_pretrained options that make onnxruntime read the external weights
|
| 76 |
+
* (model_<dtype>.onnx_data, _data_1, ...) lazily from Cache Storage, downloading them
|
| 77 |
+
* first if needed.
|
| 78 |
+
*/
|
| 79 |
+
export async function lazyWeightsOptions(modelId, dtype, chunks, progress) {
|
| 80 |
+
if (typeof FileReaderSync === "undefined" || !self.caches) throw new Error("lazy weights unsupported here");
|
| 81 |
+
const cache = await caches.open(CACHE_NAME);
|
| 82 |
+
const externalData = [];
|
| 83 |
+
for (let i = 0; i < chunks; i++) {
|
| 84 |
+
const name = `model_${dtype}.onnx_data${i === 0 ? "" : "_" + i}`;
|
| 85 |
+
externalData.push({ path: name, data: await lazyFile(cache, modelId, name, progress) });
|
| 86 |
+
}
|
| 87 |
return {
|
| 88 |
+
// Stop transformers.js from fetching the .onnx_data files itself...
|
| 89 |
use_external_data_format: false,
|
| 90 |
+
// ...and mount ours under the names the ONNX graph refers to.
|
| 91 |
+
session_options: { externalData },
|
| 92 |
};
|
| 93 |
}
|
js/llm-engine.js
CHANGED
|
@@ -15,6 +15,13 @@ const THINK_CLOSE = {
|
|
| 15 |
budget: "\n\nI have thought about this enough. Time to act.\n</think>\n",
|
| 16 |
};
|
| 17 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
export function createEngine(tf) {
|
| 19 |
const { AutoTokenizer, AutoModelForCausalLM, DynamicCache, Tensor, TextStreamer, InterruptableStoppingCriteria, StoppingCriteria } = tf;
|
| 20 |
let tokenizer = null;
|
|
@@ -72,10 +79,13 @@ export function createEngine(tf) {
|
|
| 72 |
model = await AutoModelForCausalLM.from_pretrained(modelId, { dtype, device, progress_callback, ...modelOptions });
|
| 73 |
const te = tokenizer.encode("</think>", { add_special_tokens: false });
|
| 74 |
thinkEndId = te.length === 1 ? te[0] : -1;
|
| 75 |
-
//
|
| 76 |
-
// fills `num_logits_to_keep`, so inject it on every forward pass.
|
| 77 |
-
const
|
| 78 |
-
|
|
|
|
|
|
|
|
|
|
| 79 |
onStatus?.("Warming up");
|
| 80 |
await generate([{ role: "user", content: "Hi" }], [], { thinking: false, maxNewTokens: 1 });
|
| 81 |
await reset();
|
|
@@ -158,7 +168,7 @@ export function createEngine(tf) {
|
|
| 158 |
});
|
| 159 |
const entries = {};
|
| 160 |
for (const k in out) {
|
| 161 |
-
if (k.startsWith("present")) entries[
|
| 162 |
}
|
| 163 |
cache.update(entries);
|
| 164 |
out.logits?.dispose?.();
|
|
|
|
| 15 |
budget: "\n\nI have thought about this enough. Time to act.\n</think>\n",
|
| 16 |
};
|
| 17 |
|
| 18 |
+
// Maps a graph output (present_*) to the matching cache input, for every cache kind the
|
| 19 |
+
// supported architectures use: attention KV, conv (LFM), SSM (Granite), recurrent (Qwen3.5).
|
| 20 |
+
// Mirrors transformers.js' own mapping.
|
| 21 |
+
const pastName = (k) => k
|
| 22 |
+
.replace("present_ssm", "past_ssm").replace("present_conv", "past_conv")
|
| 23 |
+
.replace("present_recurrent", "past_recurrent").replace("present", "past_key_values");
|
| 24 |
+
|
| 25 |
export function createEngine(tf) {
|
| 26 |
const { AutoTokenizer, AutoModelForCausalLM, DynamicCache, Tensor, TextStreamer, InterruptableStoppingCriteria, StoppingCriteria } = tf;
|
| 27 |
let tokenizer = null;
|
|
|
|
| 79 |
model = await AutoModelForCausalLM.from_pretrained(modelId, { dtype, device, progress_callback, ...modelOptions });
|
| 80 |
const te = tokenizer.encode("</think>", { add_special_tokens: false });
|
| 81 |
thinkEndId = te.length === 1 ? te[0] : -1;
|
| 82 |
+
// Some graphs (LFM3) take `logits_to_keep` (rank-1 int64). transformers.js only
|
| 83 |
+
// fills `num_logits_to_keep`, so inject it on every forward pass for those.
|
| 84 |
+
const inputNames = model.sessions?.model?.inputNames ?? [];
|
| 85 |
+
if (inputNames.includes("logits_to_keep")) {
|
| 86 |
+
const forward = model.forward.bind(model);
|
| 87 |
+
model.forward = (inputs) => forward({ ...inputs, logits_to_keep: inputs.logits_to_keep ?? int64([1], [1]) });
|
| 88 |
+
}
|
| 89 |
onStatus?.("Warming up");
|
| 90 |
await generate([{ role: "user", content: "Hi" }], [], { thinking: false, maxNewTokens: 1 });
|
| 91 |
await reset();
|
|
|
|
| 168 |
});
|
| 169 |
const entries = {};
|
| 170 |
for (const k in out) {
|
| 171 |
+
if (k.startsWith("present")) entries[pastName(k)] = out[k];
|
| 172 |
}
|
| 173 |
cache.update(entries);
|
| 174 |
out.logits?.dispose?.();
|
js/llm-worker.js
CHANGED
|
@@ -27,7 +27,7 @@ self.onmessage = async ({ data }) => {
|
|
| 27 |
}
|
| 28 |
};
|
| 29 |
|
| 30 |
-
async function load({ modelId, dtype, device }) {
|
| 31 |
if (device === "webgpu") {
|
| 32 |
const adapter = await navigator.gpu?.requestAdapter({ powerPreference: "high-performance" });
|
| 33 |
if (!adapter) throw new Error("WebGPU is not available in this browser. Use a recent desktop Chrome or Edge.");
|
|
@@ -42,11 +42,12 @@ async function load({ modelId, dtype, device }) {
|
|
| 42 |
post({ type: "progress", file: p.file, loaded: p.loaded ?? 0, total: p.total ?? 0, status: p.status });
|
| 43 |
}
|
| 44 |
};
|
| 45 |
-
// Read
|
| 46 |
-
//
|
|
|
|
| 47 |
let modelOptions;
|
| 48 |
try {
|
| 49 |
-
modelOptions = await lazyWeightsOptions(modelId, dtype, progress_callback);
|
| 50 |
} catch (err) {
|
| 51 |
console.warn("Low-memory weight loading unavailable, using the default path:", err);
|
| 52 |
}
|
|
|
|
| 27 |
}
|
| 28 |
};
|
| 29 |
|
| 30 |
+
async function load({ modelId, dtype, chunks = 1, device }) {
|
| 31 |
if (device === "webgpu") {
|
| 32 |
const adapter = await navigator.gpu?.requestAdapter({ powerPreference: "high-performance" });
|
| 33 |
if (!adapter) throw new Error("WebGPU is not available in this browser. Use a recent desktop Chrome or Edge.");
|
|
|
|
| 42 |
post({ type: "progress", file: p.file, loaded: p.loaded ?? 0, total: p.total ?? 0, status: p.status });
|
| 43 |
}
|
| 44 |
};
|
| 45 |
+
// Read external weights from disk tensor by tensor instead of holding them all in
|
| 46 |
+
// memory while they are copied to the GPU (see lazy-weights.js). Models whose weights
|
| 47 |
+
// are inside the .onnx file itself (chunks = 0) load the default way.
|
| 48 |
let modelOptions;
|
| 49 |
try {
|
| 50 |
+
if (chunks > 0) modelOptions = await lazyWeightsOptions(modelId, dtype, chunks, progress_callback);
|
| 51 |
} catch (err) {
|
| 52 |
console.warn("Low-memory weight loading unavailable, using the default path:", err);
|
| 53 |
}
|
js/llm.js
CHANGED
|
@@ -1,8 +1,11 @@
|
|
| 1 |
// Main-thread client for the model worker.
|
| 2 |
-
import {
|
| 3 |
|
| 4 |
export class LLM {
|
| 5 |
-
|
|
|
|
|
|
|
|
|
|
| 6 |
this.worker = new Worker(new URL("./llm-worker.js", import.meta.url), { type: "module" });
|
| 7 |
this.pending = new Map();
|
| 8 |
this.nextId = 1;
|
|
@@ -17,7 +20,8 @@ export class LLM {
|
|
| 17 |
load() {
|
| 18 |
this.loading ??= new Promise((resolve, reject) => {
|
| 19 |
this.pending.set("load", { resolve, reject });
|
| 20 |
-
|
|
|
|
| 21 |
});
|
| 22 |
return this.loading;
|
| 23 |
}
|
|
|
|
| 1 |
// Main-thread client for the model worker.
|
| 2 |
+
import { MODEL_DEVICE } from "./config.js";
|
| 3 |
|
| 4 |
export class LLM {
|
| 5 |
+
/** model: an entry of LOCAL_MODELS (config.js). */
|
| 6 |
+
constructor(model) {
|
| 7 |
+
this.model = model;
|
| 8 |
+
this.supportsThinking = model.thinking;
|
| 9 |
this.worker = new Worker(new URL("./llm-worker.js", import.meta.url), { type: "module" });
|
| 10 |
this.pending = new Map();
|
| 11 |
this.nextId = 1;
|
|
|
|
| 20 |
load() {
|
| 21 |
this.loading ??= new Promise((resolve, reject) => {
|
| 22 |
this.pending.set("load", { resolve, reject });
|
| 23 |
+
const { repo, dtype, chunks } = this.model;
|
| 24 |
+
this.worker.postMessage({ type: "load", modelId: repo, dtype, chunks, device: MODEL_DEVICE });
|
| 25 |
});
|
| 26 |
return this.loading;
|
| 27 |
}
|
js/main.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
import { DEFAULTS,
|
| 2 |
import * as setup from "./setup.js";
|
| 3 |
import { LLM } from "./llm.js";
|
| 4 |
import { RemoteLLM, API_PRESETS, loadApiKey, saveApiKey, apiKeyRemembered } from "./remote-llm.js";
|
|
@@ -58,12 +58,14 @@ const apiConfig = () => ({ baseUrl: settings.api.baseUrl, model: settings.api.mo
|
|
| 58 |
const apiHost = () => { try { return new URL(settings.api.baseUrl).host; } catch { return settings.api?.baseUrl ?? ""; } };
|
| 59 |
|
| 60 |
// The active context budget follows the backend.
|
|
|
|
|
|
|
| 61 |
const syncContextBudget = () => {
|
| 62 |
-
settings.contextBudgetTokens = usingApi() ? settings.contextApiTokens : settings.contextLocalTokens;
|
| 63 |
};
|
| 64 |
syncContextBudget();
|
| 65 |
|
| 66 |
-
let llm = usingApi() ? new RemoteLLM(apiConfig()) : new LLM();
|
| 67 |
let modelError = null; // why the local model failed to load, if it did
|
| 68 |
const vm = new VM();
|
| 69 |
let agent = null;
|
|
@@ -75,7 +77,10 @@ let files = null;
|
|
| 75 |
async function init() {
|
| 76 |
$("#version").textContent = "v" + APP_VERSION;
|
| 77 |
setBackendLabels();
|
| 78 |
-
|
|
|
|
|
|
|
|
|
|
| 79 |
syncToggles();
|
| 80 |
|
| 81 |
const checks = await environmentChecks();
|
|
@@ -85,7 +90,7 @@ async function init() {
|
|
| 85 |
try { await setup.ensureServiceWorker(); }
|
| 86 |
catch (err) { checks.sw = { ok: false, text: err.message }; renderChecks(checks); return showSetup(false); }
|
| 87 |
|
| 88 |
-
const [cx, disk, model] = await Promise.all([setup.cheerpxStatus(), setup.installedDisk(), setup.modelStatus()]);
|
| 89 |
diskManifest = disk;
|
| 90 |
const installed = { cx: cx.ready, disk: !!disk, model: model.ready };
|
| 91 |
for (const k of Object.keys(installed)) if (installed[k]) setRow(k, 1, "Installed");
|
|
@@ -205,7 +210,7 @@ function loadModel(onProgress) {
|
|
| 205 |
files.set(p.file, { loaded: p.status === "done" ? (prev?.total ?? p.loaded) : p.loaded, total: p.total || prev?.total || 0 });
|
| 206 |
let loaded = 0, total = 0;
|
| 207 |
for (const f of files.values()) { loaded += f.loaded; total += f.total; }
|
| 208 |
-
total = Math.max(total,
|
| 209 |
const frac = Math.min(0.99, loaded / total);
|
| 210 |
onProgress?.(frac, `${fmtBytes(loaded)} of ${fmtBytes(total)}`);
|
| 211 |
status("model", "busy", `Loading ${Math.round(frac * 100)}%`);
|
|
@@ -324,6 +329,8 @@ function showModelFailure(err) {
|
|
| 324 |
`${friendlyModelError(err)} Linux is still running on this device. You can connect an API model instead and the agent will keep using this machine.`,
|
| 325 |
[
|
| 326 |
h("button", { class: "btn primary small", type: "button", onclick: () => openApiDialog() }, "Use an API model"),
|
|
|
|
|
|
|
| 327 |
h("button", { class: "btn small", type: "button", onclick: () => { el.remove(); retryLocalModel(); } }, "Try again"),
|
| 328 |
],
|
| 329 |
String(err?.message ?? err));
|
|
@@ -332,21 +339,25 @@ function showModelFailure(err) {
|
|
| 332 |
|
| 333 |
function showMobileWarning() {
|
| 334 |
callout("warn", "You're on a phone or tablet",
|
| 335 |
-
|
| 336 |
-
[
|
|
|
|
|
|
|
|
|
|
|
|
|
| 337 |
}
|
| 338 |
|
| 339 |
function retryLocalModel() {
|
| 340 |
modelError = null;
|
| 341 |
llm.dispose?.();
|
| 342 |
-
llm = new LLM();
|
| 343 |
if (agent) agent.llm = llm;
|
| 344 |
setComposerEnabled();
|
| 345 |
loadModelOnce();
|
| 346 |
}
|
| 347 |
|
| 348 |
function setBackendLabels() {
|
| 349 |
-
$("#model-name").textContent = usingApi() ? settings.api.model :
|
| 350 |
$("#gpu-name").textContent = usingApi() ? "API" : MODEL_DEVICE === "webgpu" ? "WebGPU" : MODEL_DEVICE;
|
| 351 |
}
|
| 352 |
|
|
@@ -544,7 +555,10 @@ function newConversation() {
|
|
| 544 |
|
| 545 |
// ---------- composer toggles ----------
|
| 546 |
function syncToggles() {
|
| 547 |
-
|
|
|
|
|
|
|
|
|
|
| 548 |
$("#tog-confirm").setAttribute("aria-pressed", String(!!settings.confirmCommands));
|
| 549 |
}
|
| 550 |
$("#tog-thinking").addEventListener("click", () => { settings.thinking = !settings.thinking; saveSettings(); syncToggles(); });
|
|
@@ -634,7 +648,7 @@ for (const tab of document.querySelectorAll("[data-tab]")) tab.addEventListener(
|
|
| 634 |
|
| 635 |
const dlg = $("#settings");
|
| 636 |
$("#open-settings").addEventListener("click", () => {
|
| 637 |
-
$("#set-model-desc").textContent = usingApi() ? `${settings.api.model} via ${apiHost()}` :
|
| 638 |
$("#set-thinking").checked = settings.thinking;
|
| 639 |
$("#set-confirm").checked = settings.confirmCommands;
|
| 640 |
$("#set-steps").value = settings.maxSteps;
|
|
@@ -662,7 +676,67 @@ dlg.addEventListener("close", () => {
|
|
| 662 |
syncToggles();
|
| 663 |
});
|
| 664 |
dlg.addEventListener("click", (e) => { if (e.target === dlg) dlg.close(); }); // click on backdrop
|
| 665 |
-
$("#set-model").addEventListener("click", () => { dlg.close();
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 666 |
|
| 667 |
// ---------- API model dialog ----------
|
| 668 |
const apiDlg = $("#api-dlg");
|
|
|
|
| 1 |
+
import { DEFAULTS, MODEL_DEVICE, APP_VERSION, LOCAL_MODELS, findModel } from "./config.js";
|
| 2 |
import * as setup from "./setup.js";
|
| 3 |
import { LLM } from "./llm.js";
|
| 4 |
import { RemoteLLM, API_PRESETS, loadApiKey, saveApiKey, apiKeyRemembered } from "./remote-llm.js";
|
|
|
|
| 58 |
const apiHost = () => { try { return new URL(settings.api.baseUrl).host; } catch { return settings.api?.baseUrl ?? ""; } };
|
| 59 |
|
| 60 |
// The active context budget follows the backend.
|
| 61 |
+
const currentModel = () => findModel(settings.localModel);
|
| 62 |
+
const smallestModel = () => LOCAL_MODELS.reduce((a, b) => (b.bytes < a.bytes ? b : a));
|
| 63 |
const syncContextBudget = () => {
|
| 64 |
+
settings.contextBudgetTokens = usingApi() ? settings.contextApiTokens : Math.min(settings.contextLocalTokens, currentModel().contextMax);
|
| 65 |
};
|
| 66 |
syncContextBudget();
|
| 67 |
|
| 68 |
+
let llm = usingApi() ? new RemoteLLM(apiConfig()) : new LLM(currentModel());
|
| 69 |
let modelError = null; // why the local model failed to load, if it did
|
| 70 |
const vm = new VM();
|
| 71 |
let agent = null;
|
|
|
|
| 77 |
async function init() {
|
| 78 |
$("#version").textContent = "v" + APP_VERSION;
|
| 79 |
setBackendLabels();
|
| 80 |
+
const m = currentModel();
|
| 81 |
+
$("#model-size").textContent = fmtBytes(m.bytes);
|
| 82 |
+
$(".c-name", $('[data-row="model"]')).textContent = m.label;
|
| 83 |
+
$(".c-desc", $('[data-row="model"]')).textContent = `Language model by ${m.by}, ${m.dtype} ONNX`;
|
| 84 |
syncToggles();
|
| 85 |
|
| 86 |
const checks = await environmentChecks();
|
|
|
|
| 90 |
try { await setup.ensureServiceWorker(); }
|
| 91 |
catch (err) { checks.sw = { ok: false, text: err.message }; renderChecks(checks); return showSetup(false); }
|
| 92 |
|
| 93 |
+
const [cx, disk, model] = await Promise.all([setup.cheerpxStatus(), setup.installedDisk(), setup.modelStatus(currentModel())]);
|
| 94 |
diskManifest = disk;
|
| 95 |
const installed = { cx: cx.ready, disk: !!disk, model: model.ready };
|
| 96 |
for (const k of Object.keys(installed)) if (installed[k]) setRow(k, 1, "Installed");
|
|
|
|
| 210 |
files.set(p.file, { loaded: p.status === "done" ? (prev?.total ?? p.loaded) : p.loaded, total: p.total || prev?.total || 0 });
|
| 211 |
let loaded = 0, total = 0;
|
| 212 |
for (const f of files.values()) { loaded += f.loaded; total += f.total; }
|
| 213 |
+
total = Math.max(total, currentModel().bytes);
|
| 214 |
const frac = Math.min(0.99, loaded / total);
|
| 215 |
onProgress?.(frac, `${fmtBytes(loaded)} of ${fmtBytes(total)}`);
|
| 216 |
status("model", "busy", `Loading ${Math.round(frac * 100)}%`);
|
|
|
|
| 329 |
`${friendlyModelError(err)} Linux is still running on this device. You can connect an API model instead and the agent will keep using this machine.`,
|
| 330 |
[
|
| 331 |
h("button", { class: "btn primary small", type: "button", onclick: () => openApiDialog() }, "Use an API model"),
|
| 332 |
+
currentModel() !== smallestModel()
|
| 333 |
+
? h("button", { class: "btn small", type: "button", onclick: () => openModelDialog() }, "Choose a smaller model") : null,
|
| 334 |
h("button", { class: "btn small", type: "button", onclick: () => { el.remove(); retryLocalModel(); } }, "Try again"),
|
| 335 |
],
|
| 336 |
String(err?.message ?? err));
|
|
|
|
| 339 |
|
| 340 |
function showMobileWarning() {
|
| 341 |
callout("warn", "You're on a phone or tablet",
|
| 342 |
+
`${currentModel().label} needs about ${fmtBytes(currentModel().bytes)} of graphics memory, which most phones can't give a browser tab, so it will probably fail to load. Trying anyway. To skip the wait, use an API model (only the model moves to the provider, Linux keeps running here)${currentModel() !== smallestModel() ? `, or try the smaller ${smallestModel().label}` : ""}.`,
|
| 343 |
+
[
|
| 344 |
+
h("button", { class: "btn primary small", type: "button", onclick: () => openApiDialog() }, "Use an API model"),
|
| 345 |
+
currentModel() !== smallestModel()
|
| 346 |
+
? h("button", { class: "btn small", type: "button", onclick: () => openModelDialog() }, "Choose a smaller model") : null,
|
| 347 |
+
]);
|
| 348 |
}
|
| 349 |
|
| 350 |
function retryLocalModel() {
|
| 351 |
modelError = null;
|
| 352 |
llm.dispose?.();
|
| 353 |
+
llm = new LLM(currentModel());
|
| 354 |
if (agent) agent.llm = llm;
|
| 355 |
setComposerEnabled();
|
| 356 |
loadModelOnce();
|
| 357 |
}
|
| 358 |
|
| 359 |
function setBackendLabels() {
|
| 360 |
+
$("#model-name").textContent = usingApi() ? settings.api.model : currentModel().label;
|
| 361 |
$("#gpu-name").textContent = usingApi() ? "API" : MODEL_DEVICE === "webgpu" ? "WebGPU" : MODEL_DEVICE;
|
| 362 |
}
|
| 363 |
|
|
|
|
| 555 |
|
| 556 |
// ---------- composer toggles ----------
|
| 557 |
function syncToggles() {
|
| 558 |
+
const canThink = usingApi() || currentModel().thinking;
|
| 559 |
+
$("#tog-thinking").disabled = !canThink;
|
| 560 |
+
$("#tog-thinking").title = canThink ? "Let the model reason before each step" : `${currentModel().label} has no reasoning mode`;
|
| 561 |
+
$("#tog-thinking").setAttribute("aria-pressed", String(!!settings.thinking && canThink));
|
| 562 |
$("#tog-confirm").setAttribute("aria-pressed", String(!!settings.confirmCommands));
|
| 563 |
}
|
| 564 |
$("#tog-thinking").addEventListener("click", () => { settings.thinking = !settings.thinking; saveSettings(); syncToggles(); });
|
|
|
|
| 648 |
|
| 649 |
const dlg = $("#settings");
|
| 650 |
$("#open-settings").addEventListener("click", () => {
|
| 651 |
+
$("#set-model-desc").textContent = usingApi() ? `${settings.api.model} via ${apiHost()}` : `${currentModel().label} on this device (WebGPU)`;
|
| 652 |
$("#set-thinking").checked = settings.thinking;
|
| 653 |
$("#set-confirm").checked = settings.confirmCommands;
|
| 654 |
$("#set-steps").value = settings.maxSteps;
|
|
|
|
| 676 |
syncToggles();
|
| 677 |
});
|
| 678 |
dlg.addEventListener("click", (e) => { if (e.target === dlg) dlg.close(); }); // click on backdrop
|
| 679 |
+
$("#set-model").addEventListener("click", () => { dlg.close(); openModelDialog(); });
|
| 680 |
+
|
| 681 |
+
// ---------- model picker ----------
|
| 682 |
+
const modelDlg = $("#model-dlg");
|
| 683 |
+
let pickedModel = null;
|
| 684 |
+
|
| 685 |
+
async function openModelDialog() {
|
| 686 |
+
pickedModel = usingApi() ? null : currentModel().id;
|
| 687 |
+
$("#model-api-desc").textContent = usingApi() ? `In use: ${settings.api.model} via ${apiHost()}` : "OpenRouter, OpenAI, Hugging Face or any OpenAI-compatible API. Linux stays on this device.";
|
| 688 |
+
renderModelList();
|
| 689 |
+
modelDlg.showModal();
|
| 690 |
+
// Fill in what's downloaded (slow-ish: reads Cache Storage).
|
| 691 |
+
for (const m of LOCAL_MODELS) {
|
| 692 |
+
const bytes = await setup.modelCachedBytes(m).catch(() => 0);
|
| 693 |
+
const el = $(`[data-model="${m.id}"] .mo-state`, modelDlg);
|
| 694 |
+
if (!el) continue;
|
| 695 |
+
const loaded = !usingApi() && m.id === currentModel().id && llm.ready;
|
| 696 |
+
el.replaceChildren(bytes > 1e6
|
| 697 |
+
? h("span", { class: "mo-dl" }, `Downloaded · ${fmtBytes(bytes)}`)
|
| 698 |
+
: h("span", {}, `${fmtBytes(m.bytes)} download`));
|
| 699 |
+
if (bytes > 1e6 && !loaded) {
|
| 700 |
+
el.append(h("button", { class: "linkish", type: "button", onclick: async (e) => {
|
| 701 |
+
e.preventDefault();
|
| 702 |
+
if (!confirm(`Delete the downloaded files of ${m.label}? It can be downloaded again later.`)) return;
|
| 703 |
+
await setup.removeModel(m);
|
| 704 |
+
el.replaceChildren(h("span", {}, `${fmtBytes(m.bytes)} download`));
|
| 705 |
+
} }, "Delete"));
|
| 706 |
+
}
|
| 707 |
+
}
|
| 708 |
+
}
|
| 709 |
+
|
| 710 |
+
function renderModelList() {
|
| 711 |
+
$("#model-list").replaceChildren(...LOCAL_MODELS.map((m) => h("label", { class: "model-opt", "data-model": m.id },
|
| 712 |
+
h("input", { type: "radio", name: "local-model", value: m.id, checked: m.id === pickedModel, onchange: () => { pickedModel = m.id; syncModelSave(); } }),
|
| 713 |
+
h("span", { class: "mo-main" },
|
| 714 |
+
h("span", { class: "mo-name" }, m.label, m.id === LOCAL_MODELS[0].id ? h("span", { class: "mo-tag" }, "Default") : null,
|
| 715 |
+
!m.thinking ? h("span", { class: "mo-tag" }, "No reasoning") : null),
|
| 716 |
+
h("span", { class: "mo-note" }, m.note),
|
| 717 |
+
h("span", { class: "mo-meta" }, `${m.by} · ${m.license} · up to ${Math.round(m.contextMax / 1024)}k context`),
|
| 718 |
+
h("span", { class: "mo-state" }, `${fmtBytes(m.bytes)} download`)))));
|
| 719 |
+
syncModelSave();
|
| 720 |
+
}
|
| 721 |
+
|
| 722 |
+
function syncModelSave() {
|
| 723 |
+
const same = !usingApi() && pickedModel === currentModel().id;
|
| 724 |
+
$("#model-save").disabled = !pickedModel || same;
|
| 725 |
+
$("#model-save").textContent = same ? "In use" : "Use this model";
|
| 726 |
+
}
|
| 727 |
+
|
| 728 |
+
$("#model-close").addEventListener("click", () => modelDlg.close());
|
| 729 |
+
modelDlg.addEventListener("click", (e) => { if (e.target === modelDlg) modelDlg.close(); });
|
| 730 |
+
$("#model-api").addEventListener("click", () => { modelDlg.close(); openApiDialog(); });
|
| 731 |
+
$("#model-save").addEventListener("click", () => {
|
| 732 |
+
if (!pickedModel) return;
|
| 733 |
+
if (agent?.running) return alert("Stop the agent before switching models.");
|
| 734 |
+
settings.localModel = pickedModel;
|
| 735 |
+
settings.backend = "local";
|
| 736 |
+
saveSettings();
|
| 737 |
+
location.reload(); // a fresh worker loads (and if needed downloads) the new model
|
| 738 |
+
});
|
| 739 |
+
$("#setup-model").addEventListener("click", () => openModelDialog());
|
| 740 |
|
| 741 |
// ---------- API model dialog ----------
|
| 742 |
const apiDlg = $("#api-dlg");
|
js/setup.js
CHANGED
|
@@ -2,7 +2,7 @@
|
|
| 2 |
// Cache Storage: the app shell and CheerpX (via sw.js), the disk image chunks
|
| 3 |
// (stored decompressed by downloadDisk), and the model (by transformers.js).
|
| 4 |
import {
|
| 5 |
-
CX_BASE, CX_FILES, CACHE_CX, CACHE_DISK, CACHE_MODEL, DISK_BASE,
|
| 6 |
} from "./config.js";
|
| 7 |
|
| 8 |
const scopeUrl = (p) => new URL(p, document.baseURI).href;
|
|
@@ -48,12 +48,32 @@ export async function installedDisk() {
|
|
| 48 |
return res ? res.json() : null;
|
| 49 |
}
|
| 50 |
|
| 51 |
-
export async function modelStatus() {
|
| 52 |
const cache = await caches.open(CACHE_MODEL);
|
| 53 |
-
const have = await Promise.all(
|
| 54 |
return { ready: have.every(Boolean) };
|
| 55 |
}
|
| 56 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
export async function fetchRemoteManifest() {
|
| 58 |
const res = await fetch(DISK_BASE + "manifest.json", { cache: "no-store" });
|
| 59 |
if (!res.ok) throw new Error(`disk manifest: HTTP ${res.status}`);
|
|
|
|
| 2 |
// Cache Storage: the app shell and CheerpX (via sw.js), the disk image chunks
|
| 3 |
// (stored decompressed by downloadDisk), and the model (by transformers.js).
|
| 4 |
import {
|
| 5 |
+
CX_BASE, CX_FILES, CACHE_CX, CACHE_DISK, CACHE_MODEL, DISK_BASE, modelFiles, hubUrl,
|
| 6 |
} from "./config.js";
|
| 7 |
|
| 8 |
const scopeUrl = (p) => new URL(p, document.baseURI).href;
|
|
|
|
| 48 |
return res ? res.json() : null;
|
| 49 |
}
|
| 50 |
|
| 51 |
+
export async function modelStatus(model) {
|
| 52 |
const cache = await caches.open(CACHE_MODEL);
|
| 53 |
+
const have = await Promise.all(modelFiles(model).map((f) => cache.match(hubUrl(model, f))));
|
| 54 |
return { ready: have.every(Boolean) };
|
| 55 |
}
|
| 56 |
|
| 57 |
+
/** Bytes this model's files take in Cache Storage (0 if not downloaded). */
|
| 58 |
+
export async function modelCachedBytes(model) {
|
| 59 |
+
const cache = await caches.open(CACHE_MODEL);
|
| 60 |
+
const prefix = `https://huggingface.co/${model.repo}/`;
|
| 61 |
+
let total = 0;
|
| 62 |
+
for (const req of await cache.keys()) {
|
| 63 |
+
if (!req.url.startsWith(prefix)) continue;
|
| 64 |
+
const res = await cache.match(req);
|
| 65 |
+
total += Number(res?.headers.get("Content-Length")) || 0;
|
| 66 |
+
}
|
| 67 |
+
return total;
|
| 68 |
+
}
|
| 69 |
+
|
| 70 |
+
/** Deletes one model's downloaded files. */
|
| 71 |
+
export async function removeModel(model) {
|
| 72 |
+
const cache = await caches.open(CACHE_MODEL);
|
| 73 |
+
const prefix = `https://huggingface.co/${model.repo}/`;
|
| 74 |
+
for (const req of await cache.keys()) if (req.url.startsWith(prefix)) await cache.delete(req);
|
| 75 |
+
}
|
| 76 |
+
|
| 77 |
export async function fetchRemoteManifest() {
|
| 78 |
const res = await fetch(DISK_BASE + "manifest.json", { cache: "no-store" });
|
| 79 |
if (!res.ok) throw new Error(`disk manifest: HTTP ${res.status}`);
|
sw.js
CHANGED
|
@@ -6,7 +6,7 @@
|
|
| 6 |
// (__vm__/disk.ext2), backed by the chunks the setup screen stored in Cache Storage.
|
| 7 |
// 4. Adds the cross-origin isolation headers (COOP/COEP) that SharedArrayBuffer,
|
| 8 |
// CheerpX and multi-threaded WASM need, even when a host forgets to send them.
|
| 9 |
-
const SHELL_VERSION = "lwa-shell-0.
|
| 10 |
const CX_ORIGIN = "https://cxrtnc.leaningtech.com";
|
| 11 |
const CX_CACHE_PREFIX = "lwa-cheerpx-";
|
| 12 |
const DISK_CACHE = "lwa-disk";
|
|
|
|
| 6 |
// (__vm__/disk.ext2), backed by the chunks the setup screen stored in Cache Storage.
|
| 7 |
// 4. Adds the cross-origin isolation headers (COOP/COEP) that SharedArrayBuffer,
|
| 8 |
// CheerpX and multi-threaded WASM need, even when a host forgets to send them.
|
| 9 |
+
const SHELL_VERSION = "lwa-shell-0.5.0";
|
| 10 |
const CX_ORIGIN = "https://cxrtnc.leaningtech.com";
|
| 11 |
const CX_CACHE_PREFIX = "lwa-cheerpx-";
|
| 12 |
const DISK_CACHE = "lwa-disk";
|