index.html

<!doctype html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover" />
  <meta name="referrer" content="no-referrer" />
  <meta name="theme-color" content="#f5f7fb" />
  <title>Decide Live</title>
  <meta name="description" content="Decide Live: on-device decisions from camera and microphone. A multimodal embedding model runs in your browser and turns every frame into calibrated probabilities." />
  <link rel="preconnect" href="https://fonts.googleapis.com" />
  <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
  <link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Bricolage+Grotesque:opsz,wght@12..96,700;12..96,800&family=DM+Sans:wght@400;500;600;700&family=JetBrains+Mono:wght@400;700&display=swap" />
  <link rel="stylesheet" href="./style.css?v=7" />
</head>
<body>
  <header class="top">
    <span class="glow a" aria-hidden="true"></span><span class="glow b" aria-hidden="true"></span><span class="glow c" aria-hidden="true"></span>
    <div class="wrap topbar">
      <div class="brand">
        <span class="eyebrow"><span class="pulse"></span>Decide Live</span>
        <h1>See it. <span class="lit">Decide it.</span><br />Right in your browser.</h1>
        <p class="sub">On-device decisions from camera and microphone. Every frame is embedded in your browser by EmbeddingGemma 2, fine-tuned for decisions, and scored against the options you write. No text generation, no tokens, no server.</p>
        <div class="stats">
          <span class="stat"><b>95%</b> on 100 held-out egg photos</span>
          <span class="stat"><b>0</b> bytes uploaded</span>
          <span class="stat"><b>744M</b> parameters, 4-bit</span>
          <span class="stat"><b>0</b> tokens generated</span>
        </div>
      </div>
      <div class="statusbox">
        <div class="status" id="status"><span class="dot"></span><span id="statusText">Model not loaded</span></div>
        <div class="chips"><span id="backend" class="chip"></span><span class="chip safe">0 bytes uploaded</span></div>
        <div class="progress" id="progressWrap" hidden><div id="progressBar"></div></div>
        <p class="warn" id="wasmNote" hidden>WebGPU is not available here, so the model runs on the CPU (WASM). The live rate is lowered automatically.</p>
      </div>
    </div>
    <div class="reel" aria-hidden="true"><div class="reel-track" id="reel"></div></div>
    <nav class="wrap tabs" role="tablist">
      <button class="tab active" data-tab="live" role="tab">Live</button>
      <button class="tab" data-tab="single" role="tab">Single input</button>
      <button class="tab" data-tab="egg" role="tab">Egg check</button>
      <button class="tab" data-tab="samples" role="tab">Sample tests</button>
      <button class="tab" data-tab="about" role="tab">How it works</button>
    </nav>
  </header>

  <!-- LIVE -->
  <main class="wrap view" data-view="live">
    <div class="livegrid">
      <section class="stagecol">
        <div class="sources">
          <button class="src active" data-src="demo">Demo feed</button>
          <button class="src" data-src="camera">Camera</button>
          <button class="src" data-src="mic">Microphone</button>
          <button class="ghost" id="flipBtn" hidden title="Switch front and back camera">Flip camera</button>
        </div>
        <div class="stage" id="stage">
          <video id="cam" playsinline muted autoplay></video>
          <canvas id="feed"></canvas>
          <div class="scan" aria-hidden="true"></div>
          <div class="badge"><span class="pulse"></span>Runs on your device. 0 bytes uploaded.</div>
          <div class="hud" id="hud"><span id="fpsVal">0.0</span> fps <span class="sep">|</span> <span id="latVal">0</span> <span id="latUnit" class="unit">ms / frame</span></div>
          <div class="overlay" id="overlay" hidden>
            <svg class="ring" viewBox="0 0 120 120" aria-hidden="true">
              <circle class="ring-bg" cx="60" cy="60" r="52" />
              <circle class="ring-fg" id="ringFg" cx="60" cy="60" r="52" />
            </svg>
            <div class="ovtext">
              <div class="ovq" id="ovQuestion"></div>
              <div class="ovlabel" id="ovLabel">...</div>
              <div class="ovpct" id="ovPct"></div>
            </div>
          </div>
          <div class="idle" id="idle">
            <button class="big" id="startBtn">Start live decisions</button>
            <p>Loads the model on first start (about 260 to 290 MB at 4 bits, cached afterwards).</p>
          </div>
        </div>
        <div class="controls">
          <button id="stopBtn" class="ghost" disabled>Stop</button>
          <label>Speed
            <select id="rateSel">
              <option value="auto">Auto</option>
              <option value="5">up to 5 per second</option>
              <option value="2">up to 2 per second</option>
              <option value="1">1 per second</option>
            </select>
          </label>
          <label>Detail
            <select id="budgetSel">
              <option value="70" selected>fast (70 image tokens)</option>
              <option value="140">balanced (140)</option>
              <option value="280">fine (280)</option>
            </select>
          </label>
          <label>Prompt
            <select id="livePrompt">
              <option value="trained" selected>question + frame, as trained (one pass per card)</option>
              <option value="alone">frame alone (one pass for all cards)</option>
            </select>
          </label>
          <label>Smoothing
            <select id="smoothSel">
              <option value="0">off</option>
              <option value="0.5" selected>light</option>
              <option value="0.8">strong</option>
            </select>
          </label>
        </div>
        <div class="gate">
          <div><span class="gnum" id="gOnDevice">0</span><span class="glab">decided on device</span></div>
          <div><span class="gnum warnc" id="gEscalate">0</span><span class="glab">would escalate</span></div>
          <div><span class="gnum accent" id="gRate">-</span><span class="glab">on-device rate</span></div>
          <button class="ghost small" id="resetGate">Reset</button>
        </div>
        <div class="stream">
          <div class="streamhead"><h3>Decision stream</h3><span class="note">Each row is a real decision made in this browser just now.</span></div>
          <ol class="streamlist" id="stream"></ol>
        </div>
        <p class="note">When a card's top probability is below its threshold the frame is marked uncertain: in a production system it would go to a larger model. Here nothing is sent anywhere; it is only counted.</p>
      </section>

      <section class="cardcol">
        <div class="cardhead">
          <h2 id="cardsTitle">Decisions on every frame</h2>
          <span class="note" id="cardsCount"></span>
        </div>
        <div id="cards" class="cards"></div>
        <div class="templates">
          <h3>Add a decision (up to 4)</h3>
          <div id="templates" class="tpls"></div>
          <button class="ghost small" id="customBtn">+ Custom decision</button>
        </div>
      </section>
    </div>
  </main>

  <!-- SINGLE -->
  <main class="wrap view grid2" data-view="single" hidden>
    <section class="panel">
      <h2>1. Input</h2>
      <div class="subtabs" role="tablist">
        <button class="subtab active" data-kind="image">Image</button>
        <button class="subtab" data-kind="text">Text</button>
        <button class="subtab" data-kind="audio">Audio</button>
        <button class="subtab" data-kind="video">Video</button>
      </div>
      <div class="pane" data-pane="image">
        <label class="col">Sample
          <select id="sampleSelect"><option value="">(choose one of the 18 sample tests)</option></select>
        </label>
        <label class="drop"><input type="file" id="imageFile" accept="image/*" /><span>or upload an image (stays on this device)</span></label>
        <img id="imagePreview" alt="" hidden />
      </div>
      <div class="pane" data-pane="text" hidden>
        <textarea id="textInput" rows="6" placeholder="Paste the text to decide about"></textarea>
      </div>
      <div class="pane" data-pane="audio" hidden>
        <label class="drop"><input type="file" id="audioFile" accept="audio/*" /><span>Upload an audio clip (up to 60 s, resampled to mono 16 kHz)</span></label>
        <audio id="audioPreview" controls hidden></audio>
        <p class="note">Audio needs the extra audio encoder (about 190 MB at q4), loaded on first use.</p>
      </div>
      <div class="pane" data-pane="video" hidden>
        <label class="drop"><input type="file" id="videoFile" accept="video/*" /><span>Upload a short video (up to 16 frames are sampled evenly)</span></label>
        <video id="videoPreview" controls muted playsinline hidden></video>
      </div>
    </section>

    <section class="panel">
      <h2>2. Question and options</h2>
      <div class="seg" role="radiogroup">
        <label><input type="radio" name="mode" value="choice" checked /> Options</label>
        <label><input type="radio" name="mode" value="yesno" /> Yes / no</label>
      </div>
      <label class="col">Context (optional)
        <input id="contextInput" type="text" placeholder="e.g. Candling photo: a light is shone through the egg" />
      </label>
      <label class="col"><span id="questionLabel">Question</span>
        <input id="questionInput" type="text" placeholder="e.g. Is this egg good or bad?" />
      </label>
      <label class="col" id="optionsWrap">Options, 2 to 8, one per line as <code>key: description</code>
        <textarea id="optionsInput" rows="6" placeholder="good: Normal, healthy egg&#10;bad: Defective, spoiled or damaged egg"></textarea>
      </label>
      <p class="note" id="yesnoNote" hidden>In the trained format (default), the question stays in the query and the two options are "Yes" and "No". The older formats build two options from the statement: the statement and its negation, or for a question, the question followed by "Yes." and by "No.".</p>
      <details class="adv">
        <summary>Settings</summary>
        <label class="col">Temperature <span id="tempVal">0.0501</span> <button type="button" class="ghost small" id="resetTemp">Use trained value</button>
          <input type="range" id="temp" min="0.005" max="0.5" step="0.0001" value="0.0501" />
        </label>
        <label class="col">Prompt format
          <select id="preset"></select>
        </label>
        <label class="col">Image detail (also used by the sample tests)
          <select id="budgetSingle">
            <option value="auto">Auto: 280 image tokens on WebGPU, 70 on CPU</option>
            <option value="70">70 (fast)</option>
            <option value="140">140</option>
            <option value="280">280 (model default)</option>
          </select>
        </label>
        <label class="col">Precision (applies on next load)
          <select id="dtype">
            <option value="auto">Auto: 4-bit (q4f16 on WebGPU with fp16 support, else q4)</option>
            <option value="q4">q4 (4-bit, works everywhere, also on the CPU)</option>
            <option value="q4f16">q4f16 (4-bit weights, fp16 math; WebGPU with fp16 only)</option>
          </select>
        </label>
        <label class="col">Backend (applies on next load)
          <select id="device">
            <option value="auto">Auto (WebGPU, else WASM)</option>
            <option value="webgpu">WebGPU</option>
            <option value="wasm">WASM (CPU)</option>
          </select>
        </label>
      </details>
      <button id="decideBtn" class="primary">Decide</button>
    </section>

    <section class="panel wide" id="resultCard" hidden>
      <h2>3. Decision</h2>
      <div id="result"></div>
    </section>
  </main>

  <!-- EGG CHECK -->
  <main class="wrap view" data-view="egg" hidden>
    <section class="panel">
      <h2>Egg check</h2>
      <p class="note">Candling photos (a light shone through the egg in a dark room). The decision uses the format the model was trained with:
        the photo with <code>Is this egg good or bad?</code> and the candling context, scored against <code>Normal, healthy egg</code> and
        <code>Defective, spoiled or damaged egg</code>. The 8 photos below are held out: they were not used in training. Pick one, run all 8, or use your own photo.</p>
      <div class="egggrid" id="eggGrid"></div>
      <div class="btnrow">
        <button id="eggRunAll" class="primary inline">Check all 8</button>
        <label class="drop inline"><input type="file" id="eggFile" accept="image/*" /><span>Use your own candling photo (stays on this device)</span></label>
      </div>
      <div id="eggSummary" class="summary"></div>
      <div class="eggresult" id="eggResult" hidden>
        <img id="eggPreview" alt="Selected egg photo" />
        <div id="eggOut"></div>
      </div>
      <div class="tablewrap"><table id="eggTable" hidden>
        <thead><tr><th>Photo</th><th>Gold</th><th>Decision</th><th>p(good)</th><th>p(bad)</th><th>OK</th><th>ms</th></tr></thead>
        <tbody></tbody>
      </table></div>
      <p class="note">Photos: Good and Bad Eggs Identification Image Dataset (Mendeley Data, doi:10.17632/mdty358x8m.1), CC BY 4.0. See <a href="./CREDITS.md" target="_blank">CREDITS.md</a>.</p>
    </section>
  </main>

  <!-- SAMPLES -->
  <main class="wrap view" data-view="samples" hidden>
    <section class="panel">
      <h2>Sample tests (18)</h2>
      <p class="note">Counting, yes / no, spatial relations, a chart, a receipt, app screens, scene text, common sense and egg candling (the egg photos are held out from training; the other images may come from datasets the training mix drew on). Each test runs in this browser with the prompt format, image detail and temperature chosen under Single input &gt; Settings. In the trained format, yes / no tests keep the question in the query and score the options "Yes" and "No"; the older formats use the yes / no mode (the question followed by "Yes." or "No.").</p>
      <div class="btnrow">
        <button id="runAllBtn" class="primary inline">Run all samples</button>
        <button id="compareBtn" class="ghost">Compare all prompt formats</button>
      </div>
      <div id="runAllSummary" class="summary"></div>
      <div class="tablewrap"><table id="compareTable" hidden>
        <thead><tr><th>Prompt format</th><th>Accuracy</th><th>Mean ms / test</th></tr></thead><tbody></tbody>
      </table></div>
      <div class="tablewrap"><table id="runAllTable" hidden>
        <thead><tr><th>#</th><th>Test</th><th>Type</th><th>Gold</th><th>Decision</th><th>p</th><th>OK</th><th>ms</th></tr></thead>
        <tbody></tbody>
      </table></div>
    </section>
  </main>

  <!-- ABOUT -->
  <main class="wrap view" data-view="about" hidden>
    <section class="panel prose">
      <h2>How it works</h2>
      <ol>
        <li>The input (a camera frame, an image, a text, a 2 second sound window or a short video) and the question are turned into one 768-number vector (mean pooling, then L2 normalization) by <a href="https://huggingface.co/google/embeddinggemma-2" target="_blank" rel="noopener">EmbeddingGemma 2</a> (Apache-2.0), fine-tuned for decisions: the question and the media go into one embedding and each option into another, and training pushes the right option closest. It runs as ONNX (same file layout as <a href="https://huggingface.co/onnx-community/embeddinggemma-2-ONNX" target="_blank" rel="noopener">onnx-community/embeddinggemma-2-ONNX</a>, with the fine-tuned weights) with <a href="https://github.com/huggingface/transformers.js" target="_blank" rel="noopener">transformers.js</a> 4.3.1 on WebGPU (or WASM on the CPU). The model files are served with this page.</li>
        <li>Each option description is embedded once and cached.</li>
        <li>The probabilities are softmax(scale &times; cosine) over the options. The scale was learned in training (about 19.96), so the default temperature is 1 / scale, about 0.050.</li>
        <li>In Live mode each card embeds the frame together with its own question (one pass per card). The alternative "frame alone" setting embeds the frame once and scores all cards from that vector, which is faster but is not the format the model was trained with.</li>
        <li>A card whose top probability is below its threshold is marked uncertain. That is where a larger model would take over; this page only counts it.</li>
      </ol>
      <h3>Privacy</h3>
      <p>The page is static. The model weights are downloaded from the Hugging Face Hub and cached by the browser. Camera frames, microphone audio, images and text are processed on this device and are never uploaded. You can check this in the browser's network panel: after the model is cached, decisions make no requests.</p>
      <h3>Prompts</h3>
      <div id="promptDoc"></div>
      <h3>Limits</h3>
      <ul>
        <li>An embedding model compares meanings; it does not read, count or reason step by step. Questions about exact counts, small text or negation are hard for it.</li>
        <li>Probabilities depend on the temperature. They are comparable between options of one card, not calibrated against ground truth.</li>
        <li>WebGPU needs a recent Chrome, Edge or Safari. Without it the page falls back to the CPU (WASM): it works with the same 4-bit files (about 340 MB), but it takes several seconds per image (about 6 s at 70 image tokens, about 30 s at 280 in our test), so the live rate drops to a frame every few seconds.</li>
        <li>The fine-tuning targeted image decisions (including egg candling). The audio encoder is the unchanged base one, so sound decisions are not tuned.</li>
      </ul>
      <p class="note">Sample image credits: <a href="./CREDITS.md" target="_blank">CREDITS.md</a>. Code: Apache-2.0.</p>
    </section>
  </main>

  <footer class="wrap foot">
    <p>Based on EmbeddingGemma 2 by Google DeepMind (Apache-2.0), fine-tuned for decisions, run with transformers.js. This page sends no data anywhere.</p>
  </footer>
  <script type="module" src="./app.js?v=7"></script>
</body>
</html>