Spaces:
Running
Running
NakliTechie commited on
Commit ·
869ba2d
1
Parent(s): b0d59cd
Sync from GitHub 2026-10-30T05:02:25Z
Browse files- index.html +168 -27
index.html
CHANGED
|
@@ -866,6 +866,7 @@
|
|
| 866 |
height: 64px;
|
| 867 |
}
|
| 868 |
|
|
|
|
| 869 |
.attachment-chip.audio-chip {
|
| 870 |
width: 180px;
|
| 871 |
height: 48px;
|
|
@@ -2501,7 +2502,7 @@
|
|
| 2501 |
<li><strong>Artifact preview</strong> — <code>```html</code> / <code>```svg</code> / <code>```artifact</code> code blocks get a live sandboxed iframe below the code (<code>sandbox="allow-scripts"</code>, no same-origin). Safe to run model-generated UI inline.</li>
|
| 2502 |
<li><strong>Voice to text</strong> — 🗣 button left of the input records mic audio, decodes to 16 kHz mono PCM on-device, and runs Whisper-base via WebGPU to transcribe into the input. ~80 MB first-use download; nothing leaves the device.</li>
|
| 2503 |
<li><strong>Read aloud</strong> — 🔊 button under any reply speaks it via Kokoro-82M (runs on WASM, so it never competes with the chat model). Streams sentence-by-sentence; English voices in Settings, and a Hindi reply uses the device’s own Hindi voice when it has one. ~80 MB first-use download; nothing leaves the device.</li>
|
| 2504 |
-
<li><strong>Voice mode</strong> — 🎙 chip opens a hands-free loop: hold the mic (or hold <strong>V</strong>) to talk, release to send. Moonshine transcribes on-device (English; other languages fall back to Whisper), the model answers, and Kokoro reads the reply back. Turn-based; the chat model stays loaded. ~180 MB first-use download.</li>
|
| 2505 |
<li><strong>Python code tool</strong> (agent-capable models) — model can call <code>run_python</code> to execute Python in a sandboxed Pyodide worker. numpy / pandas / matplotlib auto-install on import. ~10 MB first-use download.</li>
|
| 2506 |
<li><strong>Cache management</strong> — view/clear cached models in Settings</li>
|
| 2507 |
<li><strong>Custom models</strong> — Settings → paste a Hugging Face ONNX repo id (causal LMs, runs on WebGPU) <em>or a direct GGUF URL</em> (runs via wllama, WebGPU when present / CPU otherwise). ONNX repos are probed via the HF API for the best quant; oversized models warn (but aren’t blocked) so you can try on a capable GPU. GGUF URLs are sized via a HEAD request; point at the first shard of a split GGUF (…-00001-of-…) for files over wllama’s 2 GB per-file / ~4 GB total limits.</li>
|
|
@@ -2680,7 +2681,7 @@
|
|
| 2680 |
<li><strong>Gemma 4 E2B · WebGPU kernels</strong> (~2 GB) — text + agent. Google's Gemma 4 E2B (QAT mobile, int4) on a from-scratch WebGPU engine: hand-written WGSL for QAT int4 matmul, embed-gather-norm, RoPE/RMSNorm and GQA + sliding-window attention, reading the QAT-mobile weights directly. Ported from <em>webml-community/gemma-4-webgpu-kernels</em>. WebGPU-only. About 4× the ONNX Gemma 4 E2B’s decode speed on the same Mac (171 vs 42 tok/s, M4 Pro); text-only.</li>
|
| 2681 |
<li><strong>Gemma 4 E4B · WebGPU kernels</strong> (~3.6 GB) — text + agent. The larger Gemma 4 (QAT mobile) on the same hand-written WebGPU engine. Its per-layer embedding table can also live on disk (Settings → Models, below).</li>
|
| 2682 |
<li><strong>LFM3-VL 2B</strong> (~1.9 GB) — text and images, with reasoning. Liquid AI's LFM3-VL-2B (October 2026): the LFM3 language model with a new vision encoder, through Transformers.js on WebGPU. Attach an image and ask about it. Show reasoning turns its step-by-step thinking on.</li>
|
| 2683 |
-
<li><strong>LFM3-Omni 2B</strong> (~2.1 GB) — text, images and speech in, with reasoning and tools. Liquid AI's LFM3-Omni-2B (October 2026), one model that reads text, pictures and spoken questions (up to 30 seconds a clip). Transformers.js does not run it yet, so it comes through anyformer on diskformer's own onnxruntime-web engine; the first load stores the files in this browser once.
|
| 2684 |
<li><strong>Gemma 4 E2B</strong> (~1.5 GB) — multimodal (image + audio) + agent.</li>
|
| 2685 |
<li><strong>Gemma 4 E4B</strong> (~4.9 GB) — multimodal + agent, best quality.</li>
|
| 2686 |
<li><strong>LFM2.5 230M · GGUF</strong> (~170 MB) — the same LFM2.5 230M through wllama (llama.cpp → WebAssembly), reading a Q5_K_M GGUF. WebGPU when your browser has it, CPU otherwise, so it also runs where WebGPU is missing.</li>
|
|
@@ -2850,6 +2851,10 @@
|
|
| 2850 |
<input type="checkbox" id="toolApprovalToggle" checked>
|
| 2851 |
Ask before running custom and MCP tools <span style="font-weight:400;color:var(--gray-500)">(they send the model's arguments to another server; also asks before weather and Wikipedia lookups)</span>
|
| 2852 |
</label>
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2853 |
<label class="toggle-row" id="keylessWebRow">
|
| 2854 |
<input type="checkbox" id="keylessWebToggle" checked>
|
| 2855 |
Keyless web tools: weather and Wikipedia <span style="font-weight:400;color:var(--gray-500)">(agent-capable models; sends the place name to open-meteo.com or the topic to wikipedia.org, no key or account)</span>
|
|
@@ -3005,7 +3010,7 @@
|
|
| 3005 |
<option value="lfm3-1.7b">LFM3 1.7B · Agent (~1.4 GB)</option>
|
| 3006 |
<option value="gemini-nano">Gemini Nano · Built into Chrome (0 MB)</option>
|
| 3007 |
<option value="lfm3-vl-2b">LFM3-VL 2B · Text and images (~1.9 GB)</option>
|
| 3008 |
-
<option value="lfm3-omni-2b-anyformer">LFM3-Omni 2B ·
|
| 3009 |
<option value="gemma4-e2b">Gemma 4 E2B · Multimodal (~1.5 GB)</option>
|
| 3010 |
<option value="gemma4-e4b">Gemma 4 E4B · Multimodal (~4.9 GB)</option>
|
| 3011 |
<option value="lfm2-230m-gguf">LFM2.5 230M · GGUF (~170 MB)</option>
|
|
@@ -3170,7 +3175,11 @@
|
|
| 3170 |
</div>
|
| 3171 |
</div>
|
| 3172 |
<div class="mode-extra" id="voicePanel" role="region" aria-label="Voice conversation">
|
| 3173 |
-
<div class="mode-extra-head">Voice · <span class="memory-count">Moonshine STT + Kokoro TTS · on-device</span>
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3174 |
<div class="voice-stage">
|
| 3175 |
<button type="button" class="voice-mic" id="voiceMicBtn" aria-label="Hold to talk, release to send">
|
| 3176 |
<span class="voice-mic-icon" id="voiceMicIcon">🎙️</span>
|
|
@@ -8031,7 +8040,7 @@ const engineFetch = async (url, init = {}) => {
|
|
| 8031 |
// The model loads through anyformer (convergence plan, 2026-10-22): anyformer resolves the repo to diskformer's copy
|
| 8032 |
// of the engine (Gemma4Mobile) and, with ple, builds the OPFS per-layer-embedding table itself. The worker then drives
|
| 8033 |
// session.engine exactly as it drove its own engine. Pinned to the same anyformer as anyformer_models.js.
|
| 8034 |
-
const ANYFORMER_URL = 'https://cdn.jsdelivr.net/npm/anyformer@0.4.
|
| 8035 |
|
| 8036 |
let anyformerLoad = null;
|
| 8037 |
let session = null;
|
|
@@ -8176,7 +8185,7 @@ self.onmessage = async (e) => {
|
|
| 8176 |
// rolled-in module URLs (moduleUrl()).
|
| 8177 |
// The model loads through anyformer (convergence plan, 2026-10-22): anyformer resolves the repo to diskformer's copy of
|
| 8178 |
// the engine (TernaryBonsai2); the worker drives session.engine as before and attaches DFlash / MTP itself.
|
| 8179 |
-
const ANYFORMER_URL = 'https://cdn.jsdelivr.net/npm/anyformer@0.4.
|
| 8180 |
const DFLASH_URL = '__TERNARY_BONSAI2_DFLASH_URL__';
|
| 8181 |
const MTP_URL = '__TERNARY_BONSAI2_MTP_URL__';
|
| 8182 |
// MTP speculative decoding (Settings → Models, default OFF; posted as `mtp` + `mtpDepth` on the
|
|
@@ -8699,6 +8708,12 @@ self.onmessage = async (e) => {
|
|
| 8699 |
try { model.reset(); } catch (_) {}
|
| 8700 |
}
|
| 8701 |
lastHistoryLen = msgs.length;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8702 |
abortController = new AbortController();
|
| 8703 |
const open = model.thinkOpenTokenId, close = model.thinkCloseTokenId;
|
| 8704 |
// A reasoning-only model (LFM2.5 24B: alwaysThinks) opens <think> in every prompt, so its reply always starts as reasoning.
|
|
@@ -8710,11 +8725,18 @@ self.onmessage = async (e) => {
|
|
| 8710 |
maxNewTokens: cfg.max_new_tokens || cfg.max_tokens || 1024,
|
| 8711 |
signal: abortController.signal,
|
| 8712 |
enableThinking: wantThink,
|
|
|
|
| 8713 |
});
|
|
|
|
| 8714 |
for await (const out of stream) {
|
| 8715 |
if (stopFlag) break;
|
| 8716 |
const full = out && typeof out.text === 'string' ? out.text : '';
|
| 8717 |
last = full;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8718 |
// A multi-byte character split across tokens decodes as a trailing U+FFFD until its last byte
|
| 8719 |
// arrives. Hold the delta back until then: posting it sent '\uFFFD' and then cut the next delta
|
| 8720 |
// one code unit late, so 🍂 arrived as '\uFFFD\uDF42' (2026-10-14).
|
|
@@ -9488,7 +9510,7 @@ self.addEventListener('message', async (e) => {
|
|
| 9488 |
id: 'LiquidAI/LFM3-Omni-2B-ONNX',
|
| 9489 |
label: 'LFM3-Omni 2B',
|
| 9490 |
backend: 'anyformer-lfm3-omni',
|
| 9491 |
-
size: '~2.
|
| 9492 |
type: 'causal',
|
| 9493 |
multimodal: true,
|
| 9494 |
imageInput: true,
|
|
@@ -19035,6 +19057,10 @@ self.addEventListener('message', async (e) => {
|
|
| 19035 |
}
|
| 19036 |
}
|
| 19037 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19038 |
else if (data.type === 'token') {
|
| 19039 |
// Always accumulate so non-streaming callers (chat UI, non-streaming
|
| 19040 |
// API) still receive the full text via the 'complete' handler.
|
|
@@ -19847,6 +19873,7 @@ self.addEventListener('message', async (e) => {
|
|
| 19847 |
attachments: attData || null,
|
| 19848 |
enableThinking,
|
| 19849 |
generationConfig: genConfig,
|
|
|
|
| 19850 |
});
|
| 19851 |
});
|
| 19852 |
}
|
|
@@ -20143,10 +20170,14 @@ self.addEventListener('message', async (e) => {
|
|
| 20143 |
// a flag for whether any tool was actually called.
|
| 20144 |
// TOOL-APPROVAL (Open Q#11): custom and MCP tools send the model's arguments to another server, so each call
|
| 20145 |
// waits for Allow / Always / Deny on a card in the chat, unless the gate is off (Settings → Tools) or the user
|
| 20146 |
-
// chose "Always" for that tool
|
| 20147 |
-
//
|
| 20148 |
-
|
| 20149 |
-
const
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20150 |
function toolApprovalOn() { try { return localStorage.getItem(TOOL_APPROVAL_KEY) !== 'off'; } catch { return true; } }
|
| 20151 |
function toolNeedsApproval(name, tool, gateOn, approved) {
|
| 20152 |
return !!(tool && (tool._custom || tool._mcp || tool.approvalHost)) && !!gateOn && !approved.has(name);
|
|
@@ -20159,6 +20190,14 @@ self.addEventListener('message', async (e) => {
|
|
| 20159 |
toolApprovalToggle.checked = toolApprovalOn();
|
| 20160 |
toolApprovalToggle.addEventListener('change', () => { try { localStorage.setItem(TOOL_APPROVAL_KEY, toolApprovalToggle.checked ? 'on' : 'off'); } catch {} });
|
| 20161 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20162 |
const keylessWebToggle = document.getElementById('keylessWebToggle');
|
| 20163 |
if (keylessWebToggle) {
|
| 20164 |
keylessWebToggle.checked = keylessWebOn();
|
|
@@ -20237,7 +20276,7 @@ self.addEventListener('message', async (e) => {
|
|
| 20237 |
clearInterval(watch);
|
| 20238 |
card.classList.add('decided');
|
| 20239 |
card.replaceChildren(Object.assign(document.createElement('div'), {
|
| 20240 |
-
textContent: v === 'deny' ? `Declined: ${name} did not run.` : `Allowed: ${name}${v === 'always' ? ' (
|
| 20241 |
}));
|
| 20242 |
resolve(v);
|
| 20243 |
}
|
|
@@ -20245,9 +20284,9 @@ self.addEventListener('message', async (e) => {
|
|
| 20245 |
});
|
| 20246 |
}
|
| 20247 |
async function executeToolWithApproval(tc, tool, ctx) {
|
| 20248 |
-
if (toolNeedsApproval(tc.name, tool, toolApprovalOn(),
|
| 20249 |
const v = await askToolApproval(tc.name, tool, tc.arguments);
|
| 20250 |
-
if (v === 'always')
|
| 20251 |
if (v === 'deny') return toolDeclinedResult(tc.name);
|
| 20252 |
}
|
| 20253 |
return tool.execute(tc.arguments, ctx);
|
|
@@ -21336,11 +21375,14 @@ self.addEventListener('message', async (e) => {
|
|
| 21336 |
|
| 21337 |
async function voiceStartRecording() {
|
| 21338 |
if (VM.recording || VM.state === 'thinking' || VM.state === 'speaking') return;
|
| 21339 |
-
|
|
|
|
| 21340 |
// Stop any read-back that's still playing (barge-in on a new turn is allowed pre-record).
|
| 21341 |
ttsStopAll();
|
| 21342 |
-
|
| 21343 |
-
|
|
|
|
|
|
|
| 21344 |
try { VM.stream = await navigator.mediaDevices.getUserMedia({ audio: true }); }
|
| 21345 |
catch (e) { voiceSetState('idle', 'Mic access denied — you can still type below.'); return; }
|
| 21346 |
VM.chunks = [];
|
|
@@ -21351,7 +21393,7 @@ self.addEventListener('message', async (e) => {
|
|
| 21351 |
VM.recording = true;
|
| 21352 |
const t = voiceEl('voiceTranscript'); if (t) t.textContent = '';
|
| 21353 |
voiceSetState('listening', 'Listening… release to send.');
|
| 21354 |
-
VM.partialTimer = setInterval(voiceDoPartial, 1200);
|
| 21355 |
}
|
| 21356 |
|
| 21357 |
async function voiceDoPartial() {
|
|
@@ -21379,6 +21421,7 @@ self.addEventListener('message', async (e) => {
|
|
| 21379 |
const blob = new Blob(VM.chunks, { type: (VM.recorder && VM.recorder.mimeType) || 'audio/webm' });
|
| 21380 |
VM.recorder = null;
|
| 21381 |
if (!blob.size) { voiceSetState('idle', 'No audio captured — hold the mic and speak.'); return; }
|
|
|
|
| 21382 |
voiceSetState('thinking', 'Transcribing…');
|
| 21383 |
let text = '';
|
| 21384 |
try { const pcm = await decodeToWhisperPCM(blob); text = (await voiceTranscribe(pcm)).trim(); }
|
|
@@ -21396,8 +21439,96 @@ self.addEventListener('message', async (e) => {
|
|
| 21396 |
if (VM.state !== 'speaking') voiceSetState('idle', 'Ready — hold the mic to talk.');
|
| 21397 |
}
|
| 21398 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21399 |
async function onEnterVoiceMode() {
|
| 21400 |
voiceSetState('idle');
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21401 |
const [sttC, ttsC] = await Promise.all([sttModelCached(), ttsModelCached()]);
|
| 21402 |
VM.needsDownload = !(sttC && ttsC);
|
| 21403 |
if (VM.needsDownload) {
|
|
@@ -21512,7 +21643,11 @@ self.addEventListener('message', async (e) => {
|
|
| 21512 |
// Auto-read the reply: ALWAYS in Voice mode (v1.1), else only if the user
|
| 21513 |
// enabled auto-play. Skip (and reset the mic) if the language isn't voiced.
|
| 21514 |
const voiceReadback = (activeMode === 'voice');
|
| 21515 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21516 |
const sb = bubble.querySelector('.speak-btn:not(:disabled)');
|
| 21517 |
if (sb) speakBubble(sb, response);
|
| 21518 |
} else if (voiceReadback) {
|
|
@@ -23967,7 +24102,7 @@ self.addEventListener('message', async (e) => {
|
|
| 23967 |
*/
|
| 23968 |
const THINK_CLOSE = -2;
|
| 23969 |
|
| 23970 |
-
export const ANYFORMER_URL = 'https://cdn.jsdelivr.net/npm/anyformer@0.4.
|
| 23971 |
|
| 23972 |
// anyformer's progress events → the engine events the SSD worker posts. The llama.cpp path reports 'download' (bytes
|
| 23973 |
// into OPFS) and 'load' / 'plan'; a hand engine (diskformer back end) reports its own engine events, 'weights' with
|
|
@@ -23997,6 +24132,7 @@ export class AnyformerModel {
|
|
| 23997 |
static file = undefined; // a GGUF in the repo; undefined lets anyformer pick (the file diskformer checked)
|
| 23998 |
static budget = 4e9; // GPU bytes: the llama.cpp path plans its expert slots from it; for a hand engine, the fit check
|
| 23999 |
static poolBytes = undefined; // a hand engine's expert cache (LocalMind's own engines ran with 4 GiB)
|
|
|
|
| 24000 |
|
| 24001 |
static async load(_id, opts = {}) {
|
| 24002 |
const { load, MODELS } = await import(ANYFORMER_URL);
|
|
@@ -24005,6 +24141,7 @@ export class AnyformerModel {
|
|
| 24005 |
const session = await load(entry || this.repo, {
|
| 24006 |
...(opts.budget || !entry ? { budget: opts.budget || this.budget } : {}), maxCtx: opts.maxCtx, signal: opts.signal, ...(this.file ? { file: this.file } : {}),
|
| 24007 |
...(this.poolBytes ? { poolBytes: this.poolBytes } : {}), ...(opts.localFile ? { localFile: opts.localFile } : {}),
|
|
|
|
| 24008 |
onProgress: (e) => { const ev = toEngineProgress(e, files); if (ev && opts.onProgress) opts.onProgress(ev); },
|
| 24009 |
});
|
| 24010 |
return new this(session);
|
|
@@ -24025,20 +24162,22 @@ export class AnyformerModel {
|
|
| 24025 |
|
| 24026 |
// A hand engine's chunks pass through. On the llama.cpp path: the reply text without its <think> / </think> tags; when
|
| 24027 |
// </think> first appears, the text up to it goes out, then the same text again with token THINK_CLOSE (an empty
|
| 24028 |
-
// delta), so the worker closes the thought with no text lost.
|
| 24029 |
-
|
|
|
|
|
|
|
| 24030 |
if (this.think) {
|
| 24031 |
-
for await (const c of this.session.generate(messages,
|
| 24032 |
return;
|
| 24033 |
}
|
| 24034 |
let closed = false;
|
| 24035 |
-
for await (const c of this.session.generate(messages,
|
| 24036 |
const raw = String(c.text || '').replace(/^\s*<think>\n?/, '');
|
| 24037 |
const at = raw.indexOf('</think>');
|
| 24038 |
// The worker posts no text on its close event, so the thought's last text goes out first.
|
| 24039 |
if (at >= 0 && !closed) { closed = true; yield { text: raw.slice(0, at) }; yield { text: raw.slice(0, at), token: THINK_CLOSE }; }
|
| 24040 |
const text = at >= 0 ? raw.slice(0, at) + raw.slice(at + '</think>'.length).replace(/^\n+/, '') : raw;
|
| 24041 |
-
yield { text, tokens: c.tokens, tokPerSec: c.tokPerSec };
|
| 24042 |
}
|
| 24043 |
}
|
| 24044 |
|
|
@@ -24125,11 +24264,13 @@ export class Lfm25_24B extends AnyformerModel {
|
|
| 24125 |
// LiquidAI's LFM3-Omni-2B (LiquidAI/LFM3-Omni-2B-ONNX: q4f16 decoder, int8 vision encoder, fp32 audio encoder, 2.14 GB)
|
| 24126 |
// by anyformer's catalog entry, on diskformer's onnxruntime-web engine (lfm3_omni.js, gated 2026-10-30: 32 of 32
|
| 24127 |
// greedy tokens equal Liquid's CPU reference for text, image and speech in). transformers.js 4.3.2 has no lfm3_omni
|
| 24128 |
-
// class; when it adds one, anyformer moves the entry and LocalMind changes nothing.
|
|
|
|
| 24129 |
// mediaParts: the SSD worker passes image parts as { type:'image', rgb, width, height } and audio parts as
|
| 24130 |
// { type:'audio', pcm } (16 kHz mono, at most 30 s); this class turns them into anyformer's content blocks.
|
| 24131 |
export class Lfm3Omni2B extends AnyformerModel {
|
| 24132 |
static entry = 'LFM3_OMNI_2B';
|
|
|
|
| 24133 |
get mediaParts() { return true; }
|
| 24134 |
async *generate(messages, opts) {
|
| 24135 |
const blocks = messages.map((m) => !Array.isArray(m.content) ? m : {
|
|
@@ -32371,7 +32512,7 @@ const GGUF_FILE = 'LFM2.5-230M-Q4_0.gguf';
|
|
| 32371 |
// The model loads through anyformer (convergence plan, 2026-10-22): anyformer resolves the repo to diskformer's copy of
|
| 32372 |
// the LFM2.5 engine (Lfm2Mobile) and returns a session with the same generate / reset / dispose. Pinned to an exact
|
| 32373 |
// version; LocalMind's anyformer_models.js pins the same one.
|
| 32374 |
-
const ANYFORMER_URL = 'https://cdn.jsdelivr.net/npm/anyformer@0.4.
|
| 32375 |
|
| 32376 |
let anyformerLoad = null;
|
| 32377 |
let model = null;
|
|
|
|
| 866 |
height: 64px;
|
| 867 |
}
|
| 868 |
|
| 869 |
+
.voice-omni-toggle { margin-left: 10px; font-weight: 400; font-size: 0.78rem; cursor: pointer; white-space: nowrap; }
|
| 870 |
.attachment-chip.audio-chip {
|
| 871 |
width: 180px;
|
| 872 |
height: 48px;
|
|
|
|
| 2502 |
<li><strong>Artifact preview</strong> — <code>```html</code> / <code>```svg</code> / <code>```artifact</code> code blocks get a live sandboxed iframe below the code (<code>sandbox="allow-scripts"</code>, no same-origin). Safe to run model-generated UI inline.</li>
|
| 2503 |
<li><strong>Voice to text</strong> — 🗣 button left of the input records mic audio, decodes to 16 kHz mono PCM on-device, and runs Whisper-base via WebGPU to transcribe into the input. ~80 MB first-use download; nothing leaves the device.</li>
|
| 2504 |
<li><strong>Read aloud</strong> — 🔊 button under any reply speaks it via Kokoro-82M (runs on WASM, so it never competes with the chat model). Streams sentence-by-sentence; English voices in Settings, and a Hindi reply uses the device’s own Hindi voice when it has one. ~80 MB first-use download; nothing leaves the device.</li>
|
| 2505 |
+
<li><strong>Voice mode</strong> — 🎙 chip opens a hands-free loop: hold the mic (or hold <strong>V</strong>) to talk, release to send. Moonshine transcribes on-device (English; other languages fall back to Whisper), the model answers, and Kokoro reads the reply back. Turn-based; the chat model stays loaded. ~180 MB first-use download. <strong>Omni voice</strong> (the switch in the Voice panel) swaps all three for LFM3-Omni 2B: it hears the recording itself (no transcription, so no misheard words in between) and answers in its own voice; it switches the chat model to LFM3-Omni and back when you turn it off.</li>
|
| 2506 |
<li><strong>Python code tool</strong> (agent-capable models) — model can call <code>run_python</code> to execute Python in a sandboxed Pyodide worker. numpy / pandas / matplotlib auto-install on import. ~10 MB first-use download.</li>
|
| 2507 |
<li><strong>Cache management</strong> — view/clear cached models in Settings</li>
|
| 2508 |
<li><strong>Custom models</strong> — Settings → paste a Hugging Face ONNX repo id (causal LMs, runs on WebGPU) <em>or a direct GGUF URL</em> (runs via wllama, WebGPU when present / CPU otherwise). ONNX repos are probed via the HF API for the best quant; oversized models warn (but aren’t blocked) so you can try on a capable GPU. GGUF URLs are sized via a HEAD request; point at the first shard of a split GGUF (…-00001-of-…) for files over wllama’s 2 GB per-file / ~4 GB total limits.</li>
|
|
|
|
| 2681 |
<li><strong>Gemma 4 E2B · WebGPU kernels</strong> (~2 GB) — text + agent. Google's Gemma 4 E2B (QAT mobile, int4) on a from-scratch WebGPU engine: hand-written WGSL for QAT int4 matmul, embed-gather-norm, RoPE/RMSNorm and GQA + sliding-window attention, reading the QAT-mobile weights directly. Ported from <em>webml-community/gemma-4-webgpu-kernels</em>. WebGPU-only. About 4× the ONNX Gemma 4 E2B’s decode speed on the same Mac (171 vs 42 tok/s, M4 Pro); text-only.</li>
|
| 2682 |
<li><strong>Gemma 4 E4B · WebGPU kernels</strong> (~3.6 GB) — text + agent. The larger Gemma 4 (QAT mobile) on the same hand-written WebGPU engine. Its per-layer embedding table can also live on disk (Settings → Models, below).</li>
|
| 2683 |
<li><strong>LFM3-VL 2B</strong> (~1.9 GB) — text and images, with reasoning. Liquid AI's LFM3-VL-2B (October 2026): the LFM3 language model with a new vision encoder, through Transformers.js on WebGPU. Attach an image and ask about it. Show reasoning turns its step-by-step thinking on.</li>
|
| 2684 |
+
<li><strong>LFM3-Omni 2B</strong> (~2.1 GB) — text, images and speech in, with reasoning and tools. Liquid AI's LFM3-Omni-2B (October 2026), one model that reads text, pictures and spoken questions (up to 30 seconds a clip). Transformers.js does not run it yet, so it comes through anyformer on diskformer's own onnxruntime-web engine; the first load stores the files in this browser once. In chat it answers in text; in Voice mode, turn on <strong>Omni voice</strong> and it hears your recording directly and answers in its own voice (the speech part adds about 0.5 GB to the download).</li>
|
| 2685 |
<li><strong>Gemma 4 E2B</strong> (~1.5 GB) — multimodal (image + audio) + agent.</li>
|
| 2686 |
<li><strong>Gemma 4 E4B</strong> (~4.9 GB) — multimodal + agent, best quality.</li>
|
| 2687 |
<li><strong>LFM2.5 230M · GGUF</strong> (~170 MB) — the same LFM2.5 230M through wllama (llama.cpp → WebAssembly), reading a Q5_K_M GGUF. WebGPU when your browser has it, CPU otherwise, so it also runs where WebGPU is missing.</li>
|
|
|
|
| 2851 |
<input type="checkbox" id="toolApprovalToggle" checked>
|
| 2852 |
Ask before running custom and MCP tools <span style="font-weight:400;color:var(--gray-500)">(they send the model's arguments to another server; also asks before weather and Wikipedia lookups)</span>
|
| 2853 |
</label>
|
| 2854 |
+
<div class="toggle-row" id="toolAlwaysRow" style="display:none">
|
| 2855 |
+
<span>Always allowed: <span id="toolAlwaysList"></span></span>
|
| 2856 |
+
<button type="button" class="btn-icon" id="toolAlwaysForget">Forget</button>
|
| 2857 |
+
</div>
|
| 2858 |
<label class="toggle-row" id="keylessWebRow">
|
| 2859 |
<input type="checkbox" id="keylessWebToggle" checked>
|
| 2860 |
Keyless web tools: weather and Wikipedia <span style="font-weight:400;color:var(--gray-500)">(agent-capable models; sends the place name to open-meteo.com or the topic to wikipedia.org, no key or account)</span>
|
|
|
|
| 3010 |
<option value="lfm3-1.7b">LFM3 1.7B · Agent (~1.4 GB)</option>
|
| 3011 |
<option value="gemini-nano">Gemini Nano · Built into Chrome (0 MB)</option>
|
| 3012 |
<option value="lfm3-vl-2b">LFM3-VL 2B · Text and images (~1.9 GB)</option>
|
| 3013 |
+
<option value="lfm3-omni-2b-anyformer">LFM3-Omni 2B · Hears and speaks (~2.6 GB)</option>
|
| 3014 |
<option value="gemma4-e2b">Gemma 4 E2B · Multimodal (~1.5 GB)</option>
|
| 3015 |
<option value="gemma4-e4b">Gemma 4 E4B · Multimodal (~4.9 GB)</option>
|
| 3016 |
<option value="lfm2-230m-gguf">LFM2.5 230M · GGUF (~170 MB)</option>
|
|
|
|
| 3175 |
</div>
|
| 3176 |
</div>
|
| 3177 |
<div class="mode-extra" id="voicePanel" role="region" aria-label="Voice conversation">
|
| 3178 |
+
<div class="mode-extra-head">Voice · <span class="memory-count" id="voiceEngineLabel">Moonshine STT + Kokoro TTS · on-device</span>
|
| 3179 |
+
<label class="voice-omni-toggle" title="LFM3-Omni 2B hears your recording directly and answers in its own voice: no transcription step, no separate voice model. Switches the chat model to LFM3-Omni (~2.6 GB once); off switches back.">
|
| 3180 |
+
<input type="checkbox" id="omniVoiceToggle"> Omni voice
|
| 3181 |
+
</label>
|
| 3182 |
+
</div>
|
| 3183 |
<div class="voice-stage">
|
| 3184 |
<button type="button" class="voice-mic" id="voiceMicBtn" aria-label="Hold to talk, release to send">
|
| 3185 |
<span class="voice-mic-icon" id="voiceMicIcon">🎙️</span>
|
|
|
|
| 8040 |
// The model loads through anyformer (convergence plan, 2026-10-22): anyformer resolves the repo to diskformer's copy
|
| 8041 |
// of the engine (Gemma4Mobile) and, with ple, builds the OPFS per-layer-embedding table itself. The worker then drives
|
| 8042 |
// session.engine exactly as it drove its own engine. Pinned to the same anyformer as anyformer_models.js.
|
| 8043 |
+
const ANYFORMER_URL = 'https://cdn.jsdelivr.net/npm/anyformer@0.4.14/index.js';
|
| 8044 |
|
| 8045 |
let anyformerLoad = null;
|
| 8046 |
let session = null;
|
|
|
|
| 8185 |
// rolled-in module URLs (moduleUrl()).
|
| 8186 |
// The model loads through anyformer (convergence plan, 2026-10-22): anyformer resolves the repo to diskformer's copy of
|
| 8187 |
// the engine (TernaryBonsai2); the worker drives session.engine as before and attaches DFlash / MTP itself.
|
| 8188 |
+
const ANYFORMER_URL = 'https://cdn.jsdelivr.net/npm/anyformer@0.4.14/index.js';
|
| 8189 |
const DFLASH_URL = '__TERNARY_BONSAI2_DFLASH_URL__';
|
| 8190 |
const MTP_URL = '__TERNARY_BONSAI2_MTP_URL__';
|
| 8191 |
// MTP speculative decoding (Settings → Models, default OFF; posted as `mtp` + `mtpDepth` on the
|
|
|
|
| 8708 |
try { model.reset(); } catch (_) {}
|
| 8709 |
}
|
| 8710 |
lastHistoryLen = msgs.length;
|
| 8711 |
+
// Speech out (LFM3-Omni, Voice mode's Omni toggle): the engine sets up Liquid's speech-to-speech prompt (its own
|
| 8712 |
+
// system line and <|mixed_start|>) only when there is no system turn. With LocalMind's system prompt (tool
|
| 8713 |
+
// instructions, ~1,900 tokens) in front, the first in-app run answered a spoken question about mountains with a
|
| 8714 |
+
// refusal about weather tools (2026-10-30), so a spoken turn goes without it: no tools in Omni voice for now.
|
| 8715 |
+
const speech = d.speech === true && model.mediaParts === true;
|
| 8716 |
+
if (speech) while (msgs.length && msgs[0].role === 'system') msgs.shift();
|
| 8717 |
abortController = new AbortController();
|
| 8718 |
const open = model.thinkOpenTokenId, close = model.thinkCloseTokenId;
|
| 8719 |
// A reasoning-only model (LFM2.5 24B: alwaysThinks) opens <think> in every prompt, so its reply always starts as reasoning.
|
|
|
|
| 8725 |
maxNewTokens: cfg.max_new_tokens || cfg.max_tokens || 1024,
|
| 8726 |
signal: abortController.signal,
|
| 8727 |
enableThinking: wantThink,
|
| 8728 |
+
...(speech ? { speech: true } : {}),
|
| 8729 |
});
|
| 8730 |
+
let toolMarkup = false;
|
| 8731 |
for await (const out of stream) {
|
| 8732 |
if (stopFlag) break;
|
| 8733 |
const full = out && typeof out.text === 'string' ? out.text : '';
|
| 8734 |
last = full;
|
| 8735 |
+
// The model's speech, until a tool call starts (the interleave writes its text before its sound, so the
|
| 8736 |
+
// markup is never voiced); the page plays it.
|
| 8737 |
+
if (speech && !toolMarkup && (full.includes('<tool_call>') || full.includes('<function='))) toolMarkup = true;
|
| 8738 |
+
// A copy: out.audio may be a view of the engine's own buffer, which transferring would detach (2026-10-30).
|
| 8739 |
+
if (speech && !toolMarkup && out.audio && out.audio.length) { const pcm = Float32Array.from(out.audio); self.postMessage({ type: 'speech', pcm, sampleRate: out.sampleRate || 24000 }, [pcm.buffer]); }
|
| 8740 |
// A multi-byte character split across tokens decodes as a trailing U+FFFD until its last byte
|
| 8741 |
// arrives. Hold the delta back until then: posting it sent '\uFFFD' and then cut the next delta
|
| 8742 |
// one code unit late, so 🍂 arrived as '\uFFFD\uDF42' (2026-10-14).
|
|
|
|
| 9510 |
id: 'LiquidAI/LFM3-Omni-2B-ONNX',
|
| 9511 |
label: 'LFM3-Omni 2B',
|
| 9512 |
backend: 'anyformer-lfm3-omni',
|
| 9513 |
+
size: '~2.6 GB',
|
| 9514 |
type: 'causal',
|
| 9515 |
multimodal: true,
|
| 9516 |
imageInput: true,
|
|
|
|
| 19057 |
}
|
| 19058 |
}
|
| 19059 |
|
| 19060 |
+
else if (data.type === 'speech') {
|
| 19061 |
+
if (generating && activeMode === 'voice') omniSpeechChunk(data.pcm, data.sampleRate);
|
| 19062 |
+
}
|
| 19063 |
+
|
| 19064 |
else if (data.type === 'token') {
|
| 19065 |
// Always accumulate so non-streaming callers (chat UI, non-streaming
|
| 19066 |
// API) still receive the full text via the 'complete' handler.
|
|
|
|
| 19873 |
attachments: attData || null,
|
| 19874 |
enableThinking,
|
| 19875 |
generationConfig: genConfig,
|
| 19876 |
+
speech: typeof omniVoiceActive === 'function' && omniVoiceActive(),
|
| 19877 |
});
|
| 19878 |
});
|
| 19879 |
}
|
|
|
|
| 20170 |
// a flag for whether any tool was actually called.
|
| 20171 |
// TOOL-APPROVAL (Open Q#11): custom and MCP tools send the model's arguments to another server, so each call
|
| 20172 |
// waits for Allow / Always / Deny on a card in the chat, unless the gate is off (Settings → Tools) or the user
|
| 20173 |
+
// chose "Always" for that tool. "Always" is stored (lm_tool_always, a JSON list of tool names), so it holds across
|
| 20174 |
+
// chats and reloads until Settings → Tools → Forget. The keyless web built-ins (get_weather, wikipedia_lookup)
|
| 20175 |
+
// carry an approvalHost and ask the same way; every other built-in runs without a card.
|
| 20176 |
+
const TOOL_APPROVAL_KEY = 'lm_tool_approval', TOOL_ALWAYS_KEY = 'lm_tool_always';
|
| 20177 |
+
function toolAlwaysLoad() { try { const a = JSON.parse(localStorage.getItem(TOOL_ALWAYS_KEY) || '[]'); return new Set(Array.isArray(a) ? a.filter((x) => typeof x === 'string') : []); } catch { return new Set(); } }
|
| 20178 |
+
const toolAlwaysAllowed = toolAlwaysLoad();
|
| 20179 |
+
function toolAlwaysAdd(name) { toolAlwaysAllowed.add(name); try { localStorage.setItem(TOOL_ALWAYS_KEY, JSON.stringify([...toolAlwaysAllowed])); } catch {} }
|
| 20180 |
+
function toolAlwaysClear() { toolAlwaysAllowed.clear(); try { localStorage.removeItem(TOOL_ALWAYS_KEY); } catch {} }
|
| 20181 |
function toolApprovalOn() { try { return localStorage.getItem(TOOL_APPROVAL_KEY) !== 'off'; } catch { return true; } }
|
| 20182 |
function toolNeedsApproval(name, tool, gateOn, approved) {
|
| 20183 |
return !!(tool && (tool._custom || tool._mcp || tool.approvalHost)) && !!gateOn && !approved.has(name);
|
|
|
|
| 20190 |
toolApprovalToggle.checked = toolApprovalOn();
|
| 20191 |
toolApprovalToggle.addEventListener('change', () => { try { localStorage.setItem(TOOL_APPROVAL_KEY, toolApprovalToggle.checked ? 'on' : 'off'); } catch {} });
|
| 20192 |
}
|
| 20193 |
+
function renderToolAlways() {
|
| 20194 |
+
const row = document.getElementById('toolAlwaysRow'), list = document.getElementById('toolAlwaysList');
|
| 20195 |
+
if (!row || !list) return;
|
| 20196 |
+
list.textContent = [...toolAlwaysAllowed].join(', ');
|
| 20197 |
+
row.style.display = toolAlwaysAllowed.size ? '' : 'none';
|
| 20198 |
+
}
|
| 20199 |
+
renderToolAlways();
|
| 20200 |
+
document.getElementById('toolAlwaysForget')?.addEventListener('click', () => { toolAlwaysClear(); renderToolAlways(); });
|
| 20201 |
const keylessWebToggle = document.getElementById('keylessWebToggle');
|
| 20202 |
if (keylessWebToggle) {
|
| 20203 |
keylessWebToggle.checked = keylessWebOn();
|
|
|
|
| 20276 |
clearInterval(watch);
|
| 20277 |
card.classList.add('decided');
|
| 20278 |
card.replaceChildren(Object.assign(document.createElement('div'), {
|
| 20279 |
+
textContent: v === 'deny' ? `Declined: ${name} did not run.` : `Allowed: ${name}${v === 'always' ? ' (always; Settings → Tools forgets it)' : ''}.`,
|
| 20280 |
}));
|
| 20281 |
resolve(v);
|
| 20282 |
}
|
|
|
|
| 20284 |
});
|
| 20285 |
}
|
| 20286 |
async function executeToolWithApproval(tc, tool, ctx) {
|
| 20287 |
+
if (toolNeedsApproval(tc.name, tool, toolApprovalOn(), toolAlwaysAllowed)) {
|
| 20288 |
const v = await askToolApproval(tc.name, tool, tc.arguments);
|
| 20289 |
+
if (v === 'always') { toolAlwaysAdd(tc.name); renderToolAlways(); }
|
| 20290 |
if (v === 'deny') return toolDeclinedResult(tc.name);
|
| 20291 |
}
|
| 20292 |
return tool.execute(tc.arguments, ctx);
|
|
|
|
| 21375 |
|
| 21376 |
async function voiceStartRecording() {
|
| 21377 |
if (VM.recording || VM.state === 'thinking' || VM.state === 'speaking') return;
|
| 21378 |
+
const omni = omniVoiceActive(); // LFM3-Omni hears the clip itself: no Moonshine, no Kokoro
|
| 21379 |
+
if (VM.needsDownload && !omni) { await voiceLoadModels(); return; } // first tap = consent to download
|
| 21380 |
// Stop any read-back that's still playing (barge-in on a new turn is allowed pre-record).
|
| 21381 |
ttsStopAll();
|
| 21382 |
+
if (!omni) {
|
| 21383 |
+
try { await sttEnsureWorker(); }
|
| 21384 |
+
catch (e) { voiceSetState('idle', 'Voice recognizer failed to load: ' + (e.message || e)); return; }
|
| 21385 |
+
}
|
| 21386 |
try { VM.stream = await navigator.mediaDevices.getUserMedia({ audio: true }); }
|
| 21387 |
catch (e) { voiceSetState('idle', 'Mic access denied — you can still type below.'); return; }
|
| 21388 |
VM.chunks = [];
|
|
|
|
| 21393 |
VM.recording = true;
|
| 21394 |
const t = voiceEl('voiceTranscript'); if (t) t.textContent = '';
|
| 21395 |
voiceSetState('listening', 'Listening… release to send.');
|
| 21396 |
+
if (!omni) VM.partialTimer = setInterval(voiceDoPartial, 1200);
|
| 21397 |
}
|
| 21398 |
|
| 21399 |
async function voiceDoPartial() {
|
|
|
|
| 21421 |
const blob = new Blob(VM.chunks, { type: (VM.recorder && VM.recorder.mimeType) || 'audio/webm' });
|
| 21422 |
VM.recorder = null;
|
| 21423 |
if (!blob.size) { voiceSetState('idle', 'No audio captured — hold the mic and speak.'); return; }
|
| 21424 |
+
if (omniVoiceActive()) { await omniVoiceSend(blob); return; }
|
| 21425 |
voiceSetState('thinking', 'Transcribing…');
|
| 21426 |
let text = '';
|
| 21427 |
try { const pcm = await decodeToWhisperPCM(blob); text = (await voiceTranscribe(pcm)).trim(); }
|
|
|
|
| 21439 |
if (VM.state !== 'speaking') voiceSetState('idle', 'Ready — hold the mic to talk.');
|
| 21440 |
}
|
| 21441 |
|
| 21442 |
+
// ── Omni voice (2026-10-30): LFM3-Omni 2B hears the recording and answers in its own voice ──
|
| 21443 |
+
// The clip goes to the model as an audio attachment (16 kHz, at most 30 s); the worker streams the model's 24 kHz
|
| 21444 |
+
// speech as 'speech' messages, played through Kokoro's audio context and registered in TTS.sessions, so the mic
|
| 21445 |
+
// (ttsStopAll) and leaving Voice mode stop it like a read-back. Kokoro does not read a reply the model spoke.
|
| 21446 |
+
const OMNI_VOICE_KEY = 'lm_omni_voice', OMNI_MODEL_KEY = 'lfm3-omni-2b-anyformer', OMNI_MAX_SECONDS = 30;
|
| 21447 |
+
let omniPrevModel = null;
|
| 21448 |
+
function omniVoiceOn() { try { return localStorage.getItem(OMNI_VOICE_KEY) === 'on'; } catch { return false; } }
|
| 21449 |
+
function omniVoiceActive() { return activeMode === 'voice' && omniVoiceOn() && activeModelKey === OMNI_MODEL_KEY && modelReady; }
|
| 21450 |
+
function omniVoiceLabel() {
|
| 21451 |
+
const l = voiceEl('voiceEngineLabel');
|
| 21452 |
+
if (l) l.innerHTML = omniVoiceOn() ? 'LFM3-Omni 2B hears and speaks · on-device' : 'Moonshine STT + Kokoro TTS · on-device';
|
| 21453 |
+
const t = voiceEl('omniVoiceToggle'); if (t) t.checked = omniVoiceOn();
|
| 21454 |
+
}
|
| 21455 |
+
function omniPickModel(key) {
|
| 21456 |
+
if (!key || (key === activeModelKey && modelReady)) return;
|
| 21457 |
+
if (generating) { setTimeout(() => omniPickModel(key), 500); return; } // the switch waits for the reply in flight
|
| 21458 |
+
modelSelect.value = key;
|
| 21459 |
+
modelSelect.dispatchEvent(new Event('change', { bubbles: true }));
|
| 21460 |
+
}
|
| 21461 |
+
function omniVoiceSet(on) {
|
| 21462 |
+
try { localStorage.setItem(OMNI_VOICE_KEY, on ? 'on' : 'off'); } catch {}
|
| 21463 |
+
omniVoiceLabel();
|
| 21464 |
+
if (on && activeModelKey !== OMNI_MODEL_KEY) { omniPrevModel = activeModelKey; omniPickModel(OMNI_MODEL_KEY); }
|
| 21465 |
+
else if (!on && activeModelKey === OMNI_MODEL_KEY && omniPrevModel) { const k = omniPrevModel; omniPrevModel = null; omniPickModel(k); }
|
| 21466 |
+
if (activeMode === 'voice') voiceSetState('idle', on ? 'Hold the mic (or press and hold V) to talk. LFM3-Omni hears you and answers in its own voice.' : 'Ready — hold the mic to talk.');
|
| 21467 |
+
}
|
| 21468 |
+
voiceEl('omniVoiceToggle')?.addEventListener('change', (e) => omniVoiceSet(e.target.checked));
|
| 21469 |
+
omniVoiceLabel();
|
| 21470 |
+
|
| 21471 |
+
async function omniVoiceSend(blob) {
|
| 21472 |
+
let pcm;
|
| 21473 |
+
try { pcm = await decodeToWhisperPCM(blob); }
|
| 21474 |
+
catch (e) { voiceSetState('idle', 'Could not read the recording: ' + (e.message || e)); return; }
|
| 21475 |
+
const secs = pcm.length / 16000;
|
| 21476 |
+
if (secs > OMNI_MAX_SECONDS) { voiceSetState('idle', `That clip is ${secs.toFixed(0)} s; LFM3-Omni takes up to ${OMNI_MAX_SECONDS} s. Try a shorter question.`); return; }
|
| 21477 |
+
const t = voiceEl('voiceTranscript'); if (t) t.textContent = `🎙 ${secs.toFixed(1)} s clip`;
|
| 21478 |
+
addAttachment({ type: 'audio', blob, pcmData: pcm, thumb: URL.createObjectURL(blob), name: 'voice clip' });
|
| 21479 |
+
chatInput.value = '';
|
| 21480 |
+
voiceSetState('thinking', 'Listening to your clip…');
|
| 21481 |
+
try { await sendMessage(); }
|
| 21482 |
+
catch (e) { voiceSetState('idle', 'Send failed: ' + (e.message || e)); return; }
|
| 21483 |
+
if (VM.state !== 'speaking') voiceSetState('idle', 'Ready — hold the mic to talk.');
|
| 21484 |
+
}
|
| 21485 |
+
|
| 21486 |
+
// One playback session per reply: chunks scheduled back to back on Kokoro's audio context.
|
| 21487 |
+
const OMNI_SPEECH = { session: null };
|
| 21488 |
+
function omniSpeechChunk(pcm, sr) {
|
| 21489 |
+
let sp = OMNI_SPEECH.session;
|
| 21490 |
+
if (!sp) {
|
| 21491 |
+
const ctx = ttsGetCtx();
|
| 21492 |
+
try { if (ctx.state === 'suspended') { ctx.resume().catch(() => {}); ttsArmResumeOnGesture(ctx); } } catch (_) {}
|
| 21493 |
+
const id = ++TTS.seq, sources = [];
|
| 21494 |
+
let nextTime = 0, pending = 0, ended = false, done = false;
|
| 21495 |
+
const finish = () => {
|
| 21496 |
+
if (done) return; done = true;
|
| 21497 |
+
TTS.sessions.delete(id); if (TTS.active && TTS.active.id === id) TTS.active = null;
|
| 21498 |
+
if (OMNI_SPEECH.session === sp) OMNI_SPEECH.session = null;
|
| 21499 |
+
if (activeMode === 'voice' && !generating) voiceSetState('idle', 'Ready — hold the mic to talk.');
|
| 21500 |
+
};
|
| 21501 |
+
sp = {
|
| 21502 |
+
sawAudio: false,
|
| 21503 |
+
onChunk(p, rate) {
|
| 21504 |
+
if (done) return;
|
| 21505 |
+
const buf = ctx.createBuffer(1, p.length, rate);
|
| 21506 |
+
buf.copyToChannel(p, 0);
|
| 21507 |
+
const node = ctx.createBufferSource();
|
| 21508 |
+
node.buffer = buf; node.connect(ctx.destination);
|
| 21509 |
+
const at = Math.max(ctx.currentTime + 0.02, nextTime);
|
| 21510 |
+
node.start(at); nextTime = at + buf.duration;
|
| 21511 |
+
pending++; sources.push(node); sp.sawAudio = true;
|
| 21512 |
+
node.onended = () => { pending--; if (ended && pending <= 0) finish(); };
|
| 21513 |
+
if (activeMode === 'voice') voiceSetState('speaking', 'Speaking…');
|
| 21514 |
+
},
|
| 21515 |
+
end() { ended = true; if (pending <= 0) finish(); },
|
| 21516 |
+
stop() { sources.forEach((n) => { try { n.stop(); } catch (_) {} }); finish(); },
|
| 21517 |
+
};
|
| 21518 |
+
OMNI_SPEECH.session = sp;
|
| 21519 |
+
TTS.sessions.set(id, sp); TTS.active = { id, session: sp, button: null };
|
| 21520 |
+
}
|
| 21521 |
+
sp.onChunk(pcm instanceof Float32Array ? pcm : new Float32Array(pcm), sr || 24000);
|
| 21522 |
+
}
|
| 21523 |
+
|
| 21524 |
async function onEnterVoiceMode() {
|
| 21525 |
voiceSetState('idle');
|
| 21526 |
+
omniVoiceLabel();
|
| 21527 |
+
if (omniVoiceOn()) {
|
| 21528 |
+
if (activeModelKey !== OMNI_MODEL_KEY) { omniPrevModel = activeModelKey; omniPickModel(OMNI_MODEL_KEY); }
|
| 21529 |
+
voiceSetState('idle', 'Hold the mic (or press and hold V) to talk. LFM3-Omni hears you and answers in its own voice.');
|
| 21530 |
+
return;
|
| 21531 |
+
}
|
| 21532 |
const [sttC, ttsC] = await Promise.all([sttModelCached(), ttsModelCached()]);
|
| 21533 |
VM.needsDownload = !(sttC && ttsC);
|
| 21534 |
if (VM.needsDownload) {
|
|
|
|
| 21643 |
// Auto-read the reply: ALWAYS in Voice mode (v1.1), else only if the user
|
| 21644 |
// enabled auto-play. Skip (and reset the mic) if the language isn't voiced.
|
| 21645 |
const voiceReadback = (activeMode === 'voice');
|
| 21646 |
+
const omniSpoke = !!(OMNI_SPEECH.session && OMNI_SPEECH.session.sawAudio);
|
| 21647 |
+
if (OMNI_SPEECH.session) OMNI_SPEECH.session.end();
|
| 21648 |
+
if (omniSpoke) {
|
| 21649 |
+
// LFM3-Omni already said it; the 🔊 button still reads the text with Kokoro on request.
|
| 21650 |
+
} else if (response && response.trim() && ttsVoiceForText(response, 'af_heart') && (voiceReadback || ttsSettings().autoplay)) {
|
| 21651 |
const sb = bubble.querySelector('.speak-btn:not(:disabled)');
|
| 21652 |
if (sb) speakBubble(sb, response);
|
| 21653 |
} else if (voiceReadback) {
|
|
|
|
| 24102 |
*/
|
| 24103 |
const THINK_CLOSE = -2;
|
| 24104 |
|
| 24105 |
+
export const ANYFORMER_URL = 'https://cdn.jsdelivr.net/npm/anyformer@0.4.14/index.js';
|
| 24106 |
|
| 24107 |
// anyformer's progress events → the engine events the SSD worker posts. The llama.cpp path reports 'download' (bytes
|
| 24108 |
// into OPFS) and 'load' / 'plan'; a hand engine (diskformer back end) reports its own engine events, 'weights' with
|
|
|
|
| 24132 |
static file = undefined; // a GGUF in the repo; undefined lets anyformer pick (the file diskformer checked)
|
| 24133 |
static budget = 4e9; // GPU bytes: the llama.cpp path plans its expert slots from it; for a hand engine, the fit check
|
| 24134 |
static poolBytes = undefined; // a hand engine's expert cache (LocalMind's own engines ran with 4 GiB)
|
| 24135 |
+
static speech = false; // load the speech graphs too (LFM3-Omni: depth and codec decoders, 515 MB)
|
| 24136 |
|
| 24137 |
static async load(_id, opts = {}) {
|
| 24138 |
const { load, MODELS } = await import(ANYFORMER_URL);
|
|
|
|
| 24141 |
const session = await load(entry || this.repo, {
|
| 24142 |
...(opts.budget || !entry ? { budget: opts.budget || this.budget } : {}), maxCtx: opts.maxCtx, signal: opts.signal, ...(this.file ? { file: this.file } : {}),
|
| 24143 |
...(this.poolBytes ? { poolBytes: this.poolBytes } : {}), ...(opts.localFile ? { localFile: opts.localFile } : {}),
|
| 24144 |
+
...(this.speech ? { speech: true } : {}),
|
| 24145 |
onProgress: (e) => { const ev = toEngineProgress(e, files); if (ev && opts.onProgress) opts.onProgress(ev); },
|
| 24146 |
});
|
| 24147 |
return new this(session);
|
|
|
|
| 24162 |
|
| 24163 |
// A hand engine's chunks pass through. On the llama.cpp path: the reply text without its <think> / </think> tags; when
|
| 24164 |
// </think> first appears, the text up to it goes out, then the same text again with token THINK_CLOSE (an empty
|
| 24165 |
+
// delta), so the worker closes the thought with no text lost. With speech (LFM3-Omni) a chunk also carries `audio`,
|
| 24166 |
+
// the new 24 kHz PCM since the last chunk, and `sampleRate`.
|
| 24167 |
+
async *generate(messages, { maxNewTokens = 1024, signal, enableThinking = false, speech = false } = {}) {
|
| 24168 |
+
const opts = { maxNewTokens, signal, thinking: enableThinking, ...(speech ? { speech: true } : {}) };
|
| 24169 |
if (this.think) {
|
| 24170 |
+
for await (const c of this.session.generate(messages, opts)) yield c;
|
| 24171 |
return;
|
| 24172 |
}
|
| 24173 |
let closed = false;
|
| 24174 |
+
for await (const c of this.session.generate(messages, opts)) {
|
| 24175 |
const raw = String(c.text || '').replace(/^\s*<think>\n?/, '');
|
| 24176 |
const at = raw.indexOf('</think>');
|
| 24177 |
// The worker posts no text on its close event, so the thought's last text goes out first.
|
| 24178 |
if (at >= 0 && !closed) { closed = true; yield { text: raw.slice(0, at) }; yield { text: raw.slice(0, at), token: THINK_CLOSE }; }
|
| 24179 |
const text = at >= 0 ? raw.slice(0, at) + raw.slice(at + '</think>'.length).replace(/^\n+/, '') : raw;
|
| 24180 |
+
yield { text, tokens: c.tokens, tokPerSec: c.tokPerSec, ...(c.audio ? { audio: c.audio, sampleRate: c.sampleRate } : {}) };
|
| 24181 |
}
|
| 24182 |
}
|
| 24183 |
|
|
|
|
| 24264 |
// LiquidAI's LFM3-Omni-2B (LiquidAI/LFM3-Omni-2B-ONNX: q4f16 decoder, int8 vision encoder, fp32 audio encoder, 2.14 GB)
|
| 24265 |
// by anyformer's catalog entry, on diskformer's onnxruntime-web engine (lfm3_omni.js, gated 2026-10-30: 32 of 32
|
| 24266 |
// greedy tokens equal Liquid's CPU reference for text, image and speech in). transformers.js 4.3.2 has no lfm3_omni
|
| 24267 |
+
// class; when it adds one, anyformer moves the entry and LocalMind changes nothing. Speech out from anyformer 0.4.14
|
| 24268 |
+
// (diskformer 0.6.12), used by Voice mode's Omni toggle; text chat stays text out.
|
| 24269 |
// mediaParts: the SSD worker passes image parts as { type:'image', rgb, width, height } and audio parts as
|
| 24270 |
// { type:'audio', pcm } (16 kHz mono, at most 30 s); this class turns them into anyformer's content blocks.
|
| 24271 |
export class Lfm3Omni2B extends AnyformerModel {
|
| 24272 |
static entry = 'LFM3_OMNI_2B';
|
| 24273 |
+
static speech = true; // Voice mode's Omni toggle speaks with it; the speech graphs load with the model (2.6 GB in all)
|
| 24274 |
get mediaParts() { return true; }
|
| 24275 |
async *generate(messages, opts) {
|
| 24276 |
const blocks = messages.map((m) => !Array.isArray(m.content) ? m : {
|
|
|
|
| 32512 |
// The model loads through anyformer (convergence plan, 2026-10-22): anyformer resolves the repo to diskformer's copy of
|
| 32513 |
// the LFM2.5 engine (Lfm2Mobile) and returns a session with the same generate / reset / dispose. Pinned to an exact
|
| 32514 |
// version; LocalMind's anyformer_models.js pins the same one.
|
| 32515 |
+
const ANYFORMER_URL = 'https://cdn.jsdelivr.net/npm/anyformer@0.4.14/index.js';
|
| 32516 |
|
| 32517 |
let anyformerLoad = null;
|
| 32518 |
let model = null;
|