web/templates/about.html

{% extends "base.html" %}
{% set nav = 'about' %}
{% block title %}How it works — NanoDex{% endblock %}
{% block body %}

<div style="max-width:760px">
  <span class="eyebrow">How it works</span>
  <h1 style="font-size:30px;margin:6px 0 10px">What actually happens when you
    press the button</h1>
  <p class="sub" style="font-size:16px">No fine-tuning, no adapters, no
    distillation. A transformer is created with random weights and taught to
    predict the next token, from zero.</p>
</div>

<section style="margin-top:36px">
  <div class="card pad-lg">
    <div class="step-n">1</div>
    <h2 style="font-size:19px">Architecture</h2>
    <p class="sub" style="margin-top:8px">A standard modern decoder-only
      transformer — <span class="mono">LlamaForCausalLM</span>: SiLU MLPs, RMSNorm
      (ε=1e-5), rotary position embeddings (θ=10000), grouped-query attention,
      tied input/output embeddings, no biases. The four tiers are that same
      recipe, narrowed and shortened until it fits the parameter budget.</p>

    <div style="overflow-x:auto;margin-top:18px">
      <table>
        <thead><tr><th>Tier</th><th>Parameters</th><th>Layers</th><th>Hidden</th>
          <th>Heads (KV)</th><th>FFN</th><th>Tokens / step</th></tr></thead>
        <tbody>
        {% for t in tiers %}
          <tr><td><b>{{ t.label }}</b></td>
            <td class="mono">{{ "{:,}".format(t.params) }}</td>
            <td>{{ t.layers }}</td><td>{{ t.hidden }}</td>
            <td>{{ t.heads }} ({{ t.kv_heads }})</td><td>{{ t.ffn }}</td>
            <td class="mono">{{ "{:,}".format(t.batch_tokens) }}</td></tr>
        {% endfor %}
        </tbody>
      </table>
    </div>
  </div>
</section>

<section style="margin-top:16px">
  <div class="card pad-lg">
    <div class="step-n">2</div>
    <h2 style="font-size:19px">Tokenizer</h2>
    <p class="sub" style="margin-top:8px">A byte-level BPE with
      <b>{{ "{:,}".format(vocab) }} tokens</b>, trained on fineweb-edu itself
      (~2.7 characters per token). This matters more than it sounds: a typical
      small-LM vocabulary of ~49k tokens would cost 28M embedding parameters on
      its own — more than three times the largest model here. A lean vocabulary is
      what makes "a 500k-parameter language model" a sentence that means something. The
      parameter counts above are real totals, embeddings included.</p>
    <p class="sub" style="margin-top:12px">The vocabulary is deliberately the
      same for every tier, and stays at 2,048 even for the 8M model. Two reasons:
      at a 500k total budget anything larger would be almost entirely
      embeddings, and a shared vocabulary is the only thing that makes the
      losses on the <a href="/explore" style="color:var(--accent)">leaderboard</a>
      comparable at all — change the tokenizer and cross-entropy stops meaning
      the same thing between two models.</p>
  </div>
</section>

<section style="margin-top:16px">
  <div class="card pad-lg">
    <div class="step-n">3</div>
    <h2 style="font-size:19px">Data</h2>
    <p class="sub" style="margin-top:8px">
      <a href="https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu"
         target="_blank" rel="noopener" style="color:var(--accent)">HuggingFaceFW/fineweb-edu</a>,
      config <span class="mono">sample-10BT</span> — filtered educational web text.
      Instead of every run re-streaming and re-tokenizing the corpus, this site
      tokenizes a {{ "%.2fB"|format(cache_target / 1000000000) }}-token slice
      <b>once</b> into a flat <span class="mono">uint16</span> file
      (~{{ "%.1f"|format(cache_target * 2 / 1024**3) }} GB) that all runs read
      from, so the workers are never waiting on data. Building it is a one-time
      cost per container; a run that asks for more tokens than are cached yet
      waits for the writer to catch up, and says so in its log. Every run starts at
      its own random offset, so two identical configurations don't see identical
      token order.</p>
  </div>
</section>

<section style="margin-top:16px">
  <div class="card pad-lg">
    <div class="step-n">4</div>
    <h2 style="font-size:19px">Training</h2>
    <div class="grid g2" style="margin-top:14px">
      <div>
        <div class="kv"><span>Optimizer</span><b>AdamW β=(0.9, 0.95)</b></div>
        <div class="kv"><span>Weight decay</span><b>0.1</b></div>
        <div class="kv"><span>Gradient clipping</span><b>1.0</b></div>
        <div class="kv"><span>Schedule</span><b>2% warmup → cosine to 10%</b></div>
      </div>
      <div>
        <div class="kv"><span>Precision</span><b>bf16 autocast on GPU</b></div>
        <div class="kv"><span>Context length</span><b>{{ seq_len }} tokens</b></div>
        <div class="kv"><span>Token budget</span>
          <b>{{ min_tokens // 10**6 }}M – 1.5B</b></div>
        <div class="kv"><span>Concurrent runs</span>
          <b>{{ max_active }} per person</b></div>
      </div>
    </div>
  </div>
</section>

<section style="margin-top:16px">
  <div class="card pad-lg">
    <div class="step-n">5</div>
    <h2 style="font-size:19px">Queue</h2>
    <p class="sub" style="margin-top:8px">One worker thread per available device
      claims the oldest queued run atomically, so two workers can never grab the
      same job. This deployment currently has
      <b>{{ n_devices }} {{ 'GPU' if on_gpu else 'CPU' }} worker{{ 's' if n_devices != 1 }}</b>,
      which means {{ n_devices }} run{{ 's' if n_devices != 1 }} can train at
      once. Estimated times shown in the wizard start from a prior and then
      recalibrate from the median throughput of runs that actually finished here.</p>
  </div>
</section>

<section style="margin-top:16px">
  <div class="card pad-lg">
    <div class="step-n">6</div>
    <h2 style="font-size:19px">Publishing</h2>
    <p class="sub" style="margin-top:8px">When a run finishes you can push it to
      your own Hugging Face namespace with the OAuth token from your session:
      <span class="mono">config.json</span>,
      <span class="mono">model.safetensors</span>, tokenizer files,
      <span class="mono">generation_config.json</span>, a
      <span class="mono">training_run.json</span> recording the complete recipe,
      and a generated model card. It loads with
      <span class="mono">transformers</span> like any other model on the Hub.</p>
  </div>
</section>

<section style="margin-top:16px">
  <div class="card pad-lg">
    <div class="step-n">7</div>
    <h2 style="font-size:19px">Durable storage</h2>
    <p class="sub" style="margin-top:8px">A Space's filesystem is wiped on every
      rebuild and restart, so finished runs, their loss curves and the token
      cache are mirrored to a Hugging Face bucket. The archive is append-only
      and keyed by run id — a finished run is written once and never mutated,
      which is why a rolling deploy running two containers at once cannot
      corrupt it. History comes back on boot; weights are pulled on demand the
      first time somebody opens the playground.</p>
    <p class="sub" style="margin-top:12px">Runs that are mid-training when a
      container dies are <b>not</b> recoverable — the optimizer state only ever
      exists in that process. They come back marked interrupted.</p>
  </div>
</section>

<section style="margin-top:36px">
  <div class="card pad-lg" style="border-color:var(--line-2)">
    <span class="eyebrow">A note on scale</span>
    <h2 style="font-size:20px;margin:8px 0 10px">Deliberately over-trained</h2>
    <p class="sub">1.5 billion tokens into an 8M-parameter model is roughly 9×
      past the Chinchilla compute-optimal ratio of 20 tokens per parameter —
      and into a 500k-parameter model, over 150× past it. That's intentional.
      Tiny models keep improving long after the
      compute-optimal point, and the goal here was never FLOP efficiency — it's
      watching cross-entropy fall from <b>7.6</b> (uniform noise over
      {{ "{:,}".format(vocab) }} tokens) toward <b>4</b>, and seeing a network
      that started as pure randomness begin to emit English words.</p>
    <p class="sub" style="margin-top:12px">Nothing trained here is a useful
      assistant, and nothing it says is factual. That was never the point.</p>
  </div>
</section>

{% endblock %}