Add CI, licences, samples, and benchmark scripts

.github/workflows/ci.yml

@@ -0,0 +1,22 @@
+name: ci
+
+on:
+  push:
+    branches: [ main, master ]
+  pull_request:
+    branches: [ main, master ]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Setup Node
+        uses: actions/setup-node@v4
+        with:
+          node-version: 20
+          cache: npm
+      - name: Install
+        run: npm ci
+      - name: Test
+        run: npm test

CHANGELOG.md

@@ -0,0 +1,10 @@
+# Changelog
+
+## 1.1.0 - 2025-11-27
+- Add Apache-2.0 LICENSE.
+- Add instruct pipeline toggle (`INSTRUCT_PIPELINE`, `INSTRUCT_GENERATOR_MODEL/PROVIDER`) with separate cache/output helpers.
+- Provide continuous runner scripts for thinking/instruct pipelines.
+- Improve cache reporting for thinking vs instruct caches.
+- Expand docs and `.env.example` for new env vars, random walk, overnight runs, and instruct flows.
+- Add sample gold JSONL and Hugging Face/GitHub distribution guidance.
+- Add GitHub Actions CI (npm test).

LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+   "License" shall mean the terms and conditions for use, reproduction,
+   and distribution as defined by Sections 1 through 9 of this document.
+
+   "Licensor" shall mean the copyright owner or entity authorized by
+   the copyright owner that is granting the License.
+
+   "Legal Entity" shall mean the union of the acting entity and all
+   other entities that control, are controlled by, or are under common
+   control with that entity. For the purposes of this definition,
+   "control" means (i) the power, direct or indirect, to cause the
+   direction or management of such entity, whether by contract or
+   otherwise, or (ii) ownership of fifty percent (50%) or more of the
+   outstanding shares, or (iii) beneficial ownership of such entity.
+
+   "You" (or "Your") shall mean an individual or Legal Entity
+   exercising permissions granted by this License.
+
+   "Source" form shall mean the preferred form for making modifications,
+   including but not limited to software source code, documentation
+   source, and configuration files.
+
+   "Object" form shall mean any form resulting from mechanical
+   transformation or translation of a Source form, including but
+   not limited to compiled object code, generated documentation,
+   and conversions to other media types.
+
+   "Work" shall mean the work of authorship, whether in Source or
+   Object form, made available under the License, as indicated by a
+   copyright notice that is included in or attached to the work
+   (an example is provided in the Appendix below).
+
+   "Derivative Works" shall mean any work, whether in Source or Object
+   form, that is based on (or derived from) the Work and for which the
+   editorial revisions, annotations, elaborations, or other modifications
+   represent, as a whole, an original work of authorship. For the purposes
+   of this License, Derivative Works shall not include works that remain
+   separable from, or merely link (or bind by name) to the interfaces of,
+   the Work and Derivative Works thereof.
+
+   "Contribution" shall mean any work of authorship, including
+   the original version of the Work and any modifications or additions
+   to that Work or Derivative Works thereof, that is intentionally
+   submitted to Licensor for inclusion in the Work by the copyright owner
+   or by an individual or Legal Entity authorized to submit on behalf of
+   the copyright owner. For the purposes of this definition, "submitted"
+   means any form of electronic, verbal, or written communication sent
+   to the Licensor or its representatives, including but not limited to
+   communication on electronic mailing lists, source code control systems,
+   and issue tracking systems that are managed by, or on behalf of, the
+   Licensor for the purpose of discussing and improving the Work, but
+   excluding communication that is conspicuously marked or otherwise
+   designated in writing by the copyright owner as "Not a Contribution."
+
+   "Contributor" shall mean Licensor and any individual or Legal Entity
+   on behalf of whom a Contribution has been received by Licensor and
+   subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of
+   this License, each Contributor hereby grants to You a perpetual,
+   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+   copyright license to reproduce, prepare Derivative Works of,
+   publicly display, publicly perform, sublicense, and distribute the
+   Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of
+   this License, each Contributor hereby grants to You a perpetual,
+   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+   (except as stated in this section) patent license to make, have made,
+   use, offer to sell, sell, import, and otherwise transfer the Work,
+   where such license applies only to those patent claims licensable
+   by such Contributor that are necessarily infringed by their
+   Contribution(s) alone or by combination of their Contribution(s)
+   with the Work to which such Contribution(s) was submitted. If You
+   institute patent litigation against any entity (including a
+   cross-claim or counterclaim in a lawsuit) alleging that the Work
+   or a Contribution incorporated within the Work constitutes direct
+   or contributory patent infringement, then any patent licenses
+   granted to You under this License for that Work shall terminate
+   as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the
+   Work or Derivative Works thereof in any medium, with or without
+   modifications, and in Source or Object form, provided that You
+   meet the following conditions:
+
+   (a) You must give any other recipients of the Work or
+       Derivative Works a copy of this License; and
+
+   (b) You must cause any modified files to carry prominent notices
+       stating that You changed the files; and
+
+   (c) You must retain, in the Source form of any Derivative Works
+       that You distribute, all copyright, patent, trademark, and
+       attribution notices from the Source form of the Work,
+       excluding those notices that do not pertain to any part of
+       the Derivative Works; and
+
+   (d) If the Work includes a "NOTICE" text file as part of its
+       distribution, then any Derivative Works that You distribute must
+       include a readable copy of the attribution notices contained
+       within such NOTICE file, excluding those notices that do not
+       pertain to any part of the Derivative Works, in at least one
+       of the following places: within a NOTICE text file distributed
+       as part of the Derivative Works; within the Source form or
+       documentation, if provided along with the Derivative Works; or,
+       within a display generated by the Derivative Works, if and
+       wherever such third-party notices normally appear. The contents
+       of the NOTICE file are for informational purposes only and
+       do not modify the License. You may add Your own attribution
+       notices within Derivative Works that You distribute, alongside
+       or as an addendum to the NOTICE text from the Work, provided
+       that such additional attribution notices cannot be construed
+       as modifying the License.
+
+   You may add Your own copyright statement to Your modifications and
+   may provide additional or different license terms and conditions
+   for use, reproduction, or distribution of Your modifications, or
+   for any such Derivative Works as a whole, provided Your use,
+   reproduction, and distribution of the Work otherwise complies with
+   the conditions stated in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise,
+   any Contribution intentionally submitted for inclusion in the Work
+   by You to the Licensor shall be under the terms and conditions of
+   this License, without any additional terms or conditions.
+   Notwithstanding the above, nothing herein shall supersede or modify
+   the terms of any separate license agreement you may have executed
+   with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade
+   names, trademarks, service marks, or product names of the Licensor,
+   except as required for reasonable and customary use in describing the
+   origin of the Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or
+   agreed to in writing, Licensor provides the Work (and each
+   Contributor provides its Contributions) on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+   implied, including, without limitation, any warranties or conditions
+   of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+   PARTICULAR PURPOSE. You are solely responsible for determining the
+   appropriateness of using or redistributing the Work and assume any
+   risks associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory,
+   whether in tort (including negligence), contract, or otherwise,
+   unless required by applicable law (such as deliberate and grossly
+   negligent acts) or agreed to in writing, shall any Contributor be
+   liable to You for damages, including any direct, indirect, special,
+   incidental, or consequential damages of any character arising as a
+   result of this License or out of the use or inability to use the
+   Work (including but not limited to damages for loss of goodwill,
+   work stoppage, computer failure or malfunction, or any and all
+   other commercial damages or losses), even if such Contributor
+   has been advised of the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing
+   the Work or Derivative Works thereof, You may choose to offer,
+   and charge a fee for, acceptance of support, warranty, indemnity,
+   or other liability obligations and/or rights consistent with this
+   License. However, in accepting such obligations, You may act only
+   on Your own behalf and on Your sole responsibility, not on behalf
+   of any other Contributor, and only if You agree to indemnify,
+   defend, and hold each Contributor harmless for any liability
+   incurred by, or claims asserted against, such Contributor by reason
+   of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+APPENDIX: How to apply the Apache License to your work.
+
+   To apply the Apache License to your work, attach the following
+   boilerplate notice, with the fields enclosed by brackets "[]"
+   replaced with your own identifying information. (Don't include
+   the brackets!)  The text should be enclosed in the appropriate
+   comment syntax for the file format. We also recommend that a
+   file or class name and description of purpose be included on the
+   same "printed page" as the copyright notice for easier
+   identification within third-party archives.
+
+Copyright [yyyy] [name of copyright owner]
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

README.md

@@ -1,3 +1,4 @@
+---
 license: apache-2.0
 title: distill-pipeline 🌟
 tags:

samples/pipeline_gold_sample.jsonl

@@ -0,0 +1 @@
+{"question":"What is the meaning of life?","context":[{"id":"c1","content":"ctx content"}],"sample":{"answer":"42","thought":"reasoning goes here","raw":"42"},"verifier":{"ok":true,"score":0.9},"reward":{"ok":true,"score":0.8}}

scripts/bench_pipeline.mjs

@@ -0,0 +1,114 @@
+#!/usr/bin/env node
+// scripts/bench_pipeline.mjs
+// Quick micro-benchmark for the pipeline using mock providers.
+// Measures throughput (questions/sec) over a limited run.
+
+import { performance } from 'perf_hooks';
+import path from 'path';
+import os from 'os';
+import { fileURLToPath } from 'url';
+import { runPipelineBatch } from '../src/pipeline/pipeline.mjs';
+
+const __filename = fileURLToPath(import.meta.url);
+const __dirname = path.dirname(__filename);
+const PROJECT_ROOT = path.join(__dirname, '..');
+
+function parseArgs(argv) {
+  const args = argv.slice(2);
+  let limit = 50;
+  let chunkLimit;
+  let cacheDir;
+  let randomWalk = false;
+  for (let i = 0; i < args.length; i++) {
+    const a = args[i];
+    if (a === '--limit' || a === '-n') {
+      const v = Number(args[i + 1]);
+      if (!Number.isNaN(v)) limit = v;
+      i++;
+    } else if (a === '--chunk-limit') {
+      const v = Number(args[i + 1]);
+      if (!Number.isNaN(v)) chunkLimit = v;
+      i++;
+    } else if (a === '--cache-dir') {
+      cacheDir = args[i + 1];
+      i++;
+    } else if (a === '--random-walk') {
+      randomWalk = true;
+    }
+  }
+  return { limit, chunkLimit, cacheDir, randomWalk };
+}
+
+function bar(label, fraction, width = 30) {
+  const clamped = Math.max(0, Math.min(1, fraction));
+  const filled = Math.round(clamped * width);
+  const empty = width - filled;
+  return `${label} [${'#'.repeat(filled)}${'.'.repeat(empty)}] ${(clamped * 100).toFixed(1)}%`;
+}
+
+async function main() {
+  const { limit, chunkLimit, cacheDir, randomWalk } = parseArgs(process.argv);
+
+  // Force mock providers for speed and determinism
+  process.env.GENERATOR_PROVIDER = 'mock';
+  process.env.VERIFIER_PROVIDER = 'mock';
+  process.env.REWARD_PROVIDER = 'mock';
+  process.env.QUESTION_PROVIDER = 'mock';
+  process.env.PROVIDER_TYPE = 'mock';
+
+  // Seed mode: question-first avoids ES by using rag chunks JSONL
+  process.env.PIPELINE_SEED_MODE = 'question-first';
+
+  // Optional random walk over chunks
+  if (randomWalk) process.env.PIPELINE_RANDOM_WALK = '1';
+
+  // Isolate cache/output
+  const cachePath =
+    cacheDir ||
+    path.join(os.tmpdir(), `distill-cache-bench-${Date.now()}`);
+  process.env.PIPELINE_CACHE_DIR = cachePath;
+  const outPath = path.join(
+    os.tmpdir(),
+    `pipeline_gold_bench_${Date.now()}.jsonl`,
+  );
+
+  console.log('🏎️  Benchmarking pipeline (mock providers)');
+  console.log(`   limit:       ${limit}`);
+  console.log(`   chunkLimit:  ${chunkLimit ?? 'default'}`);
+  console.log(`   randomWalk:  ${randomWalk ? 'yes' : 'no'}`);
+  console.log(`   cache:       ${cachePath}`);
+  console.log(`   out:         ${outPath}`);
+  console.log('');
+
+  const start = performance.now();
+  const silentLogger = { log: () => {}, error: console.error };
+  const result = await runPipelineBatch({
+    limit,
+    chunkLimit,
+    verbose: false,
+    outPath,
+    seedMode: 'question-first',
+    logger: silentLogger,
+  });
+  const end = performance.now();
+  const ms = end - start;
+
+  const qps = result.processed > 0 ? (result.processed / ms) * 1000 : 0;
+  const acceptRatio = result.processed > 0 ? result.accepted / result.processed : 0;
+
+  console.log('🎯 Benchmark complete');
+  console.log(`   mode:        ${result.mode}`);
+  console.log(`   processed:   ${result.processed}`);
+  console.log(`   accepted:    ${result.accepted}`);
+  console.log(`   duration:    ${ms.toFixed(1)} ms`);
+  console.log(`   throughput:  ${qps.toFixed(2)} q/s`);
+  console.log(`   ${bar('accept rate ', acceptRatio)}`);
+  console.log(`   ${bar('throughput ', Math.min(1, qps / 50))} (normalized vs 50 q/s)`);
+  console.log(`   cache dir:   ${cachePath}`);
+  console.log(`   out file:    ${outPath}`);
+}
+
+main().catch((err) => {
+  console.error('Benchmark error:', err);
+  process.exit(1);
+});

scripts/live_bench.mjs

@@ -0,0 +1,182 @@
+#!/usr/bin/env node
+// scripts/live_bench.mjs
+// Live HUD for pipeline throughput/latency using readline (no deps).
+// Defaults to mock providers for speed; can run real providers by env overrides.
+
+import readline from 'readline';
+import { performance } from 'perf_hooks';
+import path from 'path';
+import os from 'os';
+import { runPipelineBatch } from '../src/pipeline/pipeline.mjs';
+
+function parseArgs(argv) {
+  const args = argv.slice(2);
+  let limit;
+  let chunkLimit;
+  let randomWalk = false;
+  let mockMode = true;
+  for (let i = 0; i < args.length; i++) {
+    const a = args[i];
+    if (a === '--limit' || a === '-n') {
+      const v = Number(args[i + 1]);
+      if (!Number.isNaN(v)) limit = v;
+      i++;
+    } else if (a === '--chunk-limit') {
+      const v = Number(args[i + 1]);
+      if (!Number.isNaN(v)) chunkLimit = v;
+      i++;
+    } else if (a === '--random-walk') {
+      randomWalk = true;
+    } else if (a === '--real') {
+      mockMode = false;
+    }
+  }
+  return { limit, chunkLimit, randomWalk, mockMode };
+}
+
+function createHud() {
+  const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
+  rl.pause();
+  function render(lines) {
+    readline.cursorTo(process.stdout, 0, 0);
+    readline.clearScreenDown(process.stdout);
+    process.stdout.write(lines.join('\n') + '\n');
+  }
+  return { render };
+}
+
+function formatBar(label, fraction, width = 30) {
+  const clamped = Math.max(0, Math.min(1, fraction));
+  const filled = Math.round(clamped * width);
+  const empty = width - filled;
+  return `${label.padEnd(12)} [${'#'.repeat(filled)}${'.'.repeat(empty)}] ${(clamped * 100).toFixed(1)}%`;
+}
+
+function humanMs(ms) {
+  if (ms < 1000) return `${ms.toFixed(1)} ms`;
+  return `${(ms / 1000).toFixed(2)} s`;
+}
+
+async function main() {
+  const { limit, chunkLimit, randomWalk, mockMode } = parseArgs(process.argv);
+
+  // Defaults: mock providers for speed/determinism
+  if (mockMode) {
+    process.env.GENERATOR_PROVIDER = 'mock';
+    process.env.VERIFIER_PROVIDER = 'mock';
+    process.env.REWARD_PROVIDER = 'mock';
+    process.env.QUESTION_PROVIDER = 'mock';
+    process.env.PROVIDER_TYPE = 'mock';
+  }
+
+  // question-first by default
+  process.env.PIPELINE_SEED_MODE = process.env.PIPELINE_SEED_MODE || 'question-first';
+  if (randomWalk) process.env.PIPELINE_RANDOM_WALK = '1';
+
+  const cacheDir =
+    process.env.PIPELINE_CACHE_DIR ||
+    path.join(os.tmpdir(), `distill-live-bench-cache-${Date.now()}`);
+  const outPath =
+    process.env.BENCH_OUT ||
+    path.join(os.tmpdir(), `pipeline_gold_live_bench_${Date.now()}.jsonl`);
+
+  const hud = createHud();
+  const t0 = performance.now();
+  const samples = [];
+  const statusCounts = {};
+  const stageTimes = { gen: [], ver: [], rew: [], end2end: [] };
+
+  function avg(arr) {
+    if (!arr.length) return 0;
+    return arr.reduce((a, b) => a + b, 0) / arr.length;
+  }
+
+  function throughput(windowMs = 60000) {
+    const now = performance.now();
+    const recent = samples.filter((s) => now - s.ts <= windowMs);
+    if (recent.length === 0) return 0;
+    const spanMs = Math.min(windowMs, now - recent[0].ts);
+    if (spanMs <= 0) return 0;
+    return recent.length / (spanMs / 1000);
+  }
+
+  function acceptRate() {
+    const accepted = statusCounts.accepted || 0;
+    const processed = samples.length;
+    return processed ? accepted / processed : 0;
+  }
+
+  function summarizeStatus() {
+    const keys = Object.keys(statusCounts);
+    return keys.map((k) => `${k}:${statusCounts[k]}`).join(' ');
+  }
+
+  const logger = {
+    log: () => {},
+    error: console.error,
+  };
+
+  const hudInterval = setInterval(() => {
+    const now = performance.now();
+    const totalMs = now - t0;
+    const proc = samples.length;
+    const qps = throughput();
+    const acc = statusCounts.accepted || 0;
+    const lines = [
+      '📊 Live Bench (minimal logging)',
+      `mode: ${process.env.PIPELINE_SEED_MODE} | mock: ${mockMode ? 'yes' : 'no'} | random walk: ${randomWalk ? 'yes' : 'no'}`,
+      `processed: ${proc} | accepted: ${acc} | elapsed: ${humanMs(totalMs)}`,
+      `throughput: ${qps.toFixed(2)} pipeline cycles/s (60s window)`,
+      `gen avg: ${humanMs(avg(stageTimes.gen))} | ver avg: ${humanMs(avg(stageTimes.ver))} | rew avg: ${humanMs(avg(stageTimes.rew))}`,
+      `end2end avg: ${humanMs(avg(stageTimes.end2end))}`,
+      `status: ${summarizeStatus() || 'n/a'}`,
+      `cache: ${cacheDir}`,
+      `out:   ${outPath}`,
+    ];
+    hud.render(lines);
+  }, 750);
+
+  function onProgress({ status, elapsedMs }) {
+    const ts = performance.now();
+    statusCounts[status] = (statusCounts[status] || 0) + 1;
+    samples.push({ ts, status, end2endMs: elapsedMs });
+    if (samples.length > 2000) samples.shift();
+    if (elapsedMs != null) {
+      stageTimes.end2end.push(elapsedMs);
+      if (stageTimes.end2end.length > 500) stageTimes.end2end.shift();
+    }
+  }
+
+  const result = await runPipelineBatch({
+    limit,
+    chunkLimit,
+    verbose: false,
+    outPath,
+    seedMode: process.env.PIPELINE_SEED_MODE,
+    logger,
+    onProgress,
+  });
+
+  const totalMs = performance.now() - t0;
+  clearInterval(hudInterval);
+
+  // Final render
+  const proc = result.processed;
+  const acc = result.accepted;
+  const qps = proc > 0 ? (proc / totalMs) * 1000 : 0;
+  const lines = [
+    '✅ Bench complete',
+    `mode: ${result.mode} | mock: ${mockMode ? 'yes' : 'no'} | random walk: ${randomWalk ? 'yes' : 'no'}`,
+    `processed: ${proc} | accepted: ${acc} | elapsed: ${humanMs(totalMs)}`,
+    `throughput: ${qps.toFixed(2)} pipeline cycles/s overall`,
+    `status: ${summarizeStatus() || 'n/a'}`,
+    `cache: ${cacheDir}`,
+    `out:   ${outPath}`,
+  ];
+  hud.render(lines);
+}
+
+main().catch((err) => {
+  console.error('Live bench error:', err);
+  process.exit(1);
+});

src/pipeline/batch.mjs

@@ -74,6 +74,7 @@ export async function runPipelineBatch({
   verbose = false,
   logger = console,
   seedMode = process.env.PIPELINE_SEED_MODE || 'question-first',
+  onProgress = () => {},
 } = {}) {
   const log = logger?.log?.bind(logger) || console.log;
   const errLog = logger?.error?.bind(logger) || console.error;
@@ -105,6 +106,7 @@ export async function runPipelineBatch({
 
       log(`→ ${label} Running pipeline for: "${question}"`);
 
+      const tStart = Date.now();
       try {
         const result = await runPipelineStep({
           question,
@@ -116,6 +118,15 @@ export async function runPipelineBatch({
         statusCounts[result.status] =
           (statusCounts[result.status] || 0) + 1;
 
+        onProgress({
+          processed,
+          accepted,
+          status: result.status,
+          elapsedMs: Date.now() - tStart,
+          question,
+          mode: seedMode,
+        });
+
         if (verbose) {
           log(`   ↳ status: ${result.status}`);
         }
@@ -145,6 +156,14 @@ export async function runPipelineBatch({
         processed += 1;
         statusCounts.pipeline_error =
           (statusCounts.pipeline_error || 0) + 1;
+        onProgress({
+          processed,
+          accepted,
+          status: 'pipeline_error',
+          elapsedMs: Date.now() - tStart,
+          question,
+          mode: seedMode,
+        });
         errLog('   [pipeline] ERROR:', msg);
       }
     }
@@ -349,6 +368,14 @@ export async function runPipelineBatch({
           accepted += 1;
           statusCounts.cached_reward =
             (statusCounts.cached_reward || 0) + 1;
+          onProgress({
+            processed,
+            accepted,
+            status: 'cached_reward',
+            elapsedMs: 0,
+            question: q,
+            mode: seedMode,
+          });
           if (verbose)
             log(
               `   → [q ${processed}] using cached reward, skipping stages`,
@@ -363,6 +390,7 @@ export async function runPipelineBatch({
           `   → ${qLabel} Running pipeline for generated question: "${q}"`,
         );
 
+        const tStart = Date.now();
         try {
           const result = await runPipelineStep({
             question: q,
@@ -376,6 +404,15 @@ export async function runPipelineBatch({
           statusCounts[result.status] =
             (statusCounts[result.status] || 0) + 1;
 
+          onProgress({
+            processed,
+            accepted,
+            status: result.status,
+            elapsedMs: Date.now() - tStart,
+            question: q,
+            mode: seedMode,
+          });
+
           if (verbose) {
             log(`     ↳ status: ${result.status}`);
           }
@@ -443,6 +480,14 @@ export async function runPipelineBatch({
           processed += 1;
           statusCounts.pipeline_error =
             (statusCounts.pipeline_error || 0) + 1;
+          onProgress({
+            processed,
+            accepted,
+            status: 'pipeline_error',
+            elapsedMs: Date.now() - tStart,
+            question: q,
+            mode: seedMode,
+          });
           errLog('     [pipeline] ERROR:', msg);
         }
       }