| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"/> |
| <meta name="viewport" content="width=device-width, initial-scale=1"/> |
| <title>Slipstream forecasting benchmark</title> |
| <script src="https://code.highcharts.com/highcharts.js"></script> |
| <script src="https://code.highcharts.com/modules/heatmap.js"></script> |
| <script src="bench.js"></script> |
| <style> |
| body { font: 14px/1.45 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; |
| margin: 0; padding: 24px 28px; color: #1a1a1a; background: #fff; } |
| h1 { font-size: 20px; margin: 0 0 2px; } |
| .sub { color: #666; margin: 0 0 18px; } |
| h2 { font-size: 15px; margin: 26px 0 8px; border-top: 1px solid #eee; padding-top: 18px; } |
| .row { display: flex; flex-wrap: wrap; gap: 18px; } |
| .panel { flex: 1 1 460px; min-width: 360px; } |
| .controls { margin: 6px 0 4px; } |
| select { font-size: 14px; padding: 3px 6px; } |
| .legend { font-size: 12px; color: #555; margin: 6px 0 0; } |
| .legend span { display: inline-block; margin-right: 12px; white-space: nowrap; } |
| .sw { display: inline-block; width: 11px; height: 11px; border-radius: 2px; vertical-align: -1px; margin-right: 4px; } |
| .legend span[title] { cursor: help; } |
| .note { color: #888; font-size: 12px; margin-top: 4px; } |
| .howto { background: #f6f8fa; border: 1px solid #e1e4e8; border-radius: 8px; padding: 10px 16px; margin: 0 0 18px; font-size: 13px; max-width: 1000px; } |
| .howto h3 { font-weight: 600; font-size: 14px; margin: 0 0 4px; } |
| .howto p { margin: 10px 0 0; color: #333; } |
| .howto dl { margin: 10px 0 2px; display: grid; grid-template-columns: max-content 1fr; gap: 4px 14px; } |
| .howto dt { font-weight: 600; color: #111; } |
| .howto dd { margin: 0; color: #444; } |
| .desc { color: #555; font-size: 13px; max-width: 1000px; margin: 2px 0 10px; } |
| .info { color: #2c7fb8; cursor: help; font-style: normal; font-weight: 700; } |
| .highcharts-tooltip-container { z-index: 10000 !important; } |
| .highcharts-tooltip > span { max-width: 340px; white-space: normal !important; line-height: 1.35; } |
| </style> |
| </head> |
| <body> |
| <h1>Slipstream: project-controls forecasting benchmark</h1> |
| <p class="sub" id="subtitle"></p> |
| <div class="legend" id="legend"></div> |
|
|
| <div class="howto"> |
| <h3>How to read this dashboard</h3> |
| <p>We take real projects that are <b>partway done</b> and try to predict two things: the <b>total cost when the project finishes</b> (the jargon for this is "EAC"), and <b>which time-period it will finish in</b>. We run many forecasting methods on <b>107 real projects</b> and measure how close each one gets. Hover over any bar, dot, label or grid cell for a plain-English explanation.</p> |
| <dl> |
| <dt>Cost error %</dt><dd>How far the predicted final cost was from the real final cost, as a percentage. 3% means the forecast was within 3% of the actual bill. Lower is better.</dd> |
| <dt>Finish-date error</dt><dd>How many time-periods the predicted finish was off by. 0.7 means it was less than one period out. Lower is better.</dd> |
| <dt>"Typical" (median)</dt><dd>We report the middle project (half do better, half do worse), so one unusual project does not skew the result.</dd> |
| <dt>Completion level</dt><dd>How far into the project we were when we made the forecast: 25% is very early (hard, little information), 75% is nearly done (easier). "Average" blends all four.</dd> |
| <dt>Project length</dt><dd>Short to very long. Longer projects are harder to forecast, so we also break the results down by length.</dd> |
| <dt>"base" vs "sft", and the "teacher"</dt><dd>A <b>base</b> model is a small AI straight out of the box. An <b>sft</b> model is that same small AI <b>after we taught it</b>, using worked examples from a big, expensive <b>teacher</b> AI (shown here as agent_deepseek...). The whole goal is to make a small, cheap model forecast almost as well as the big one.</dd> |
| </dl> |
| </div> |
|
|
| <h2>Headline: accuracy by method <span style="font-weight:400;color:#888">(ordered best-overall: cost-rank + schedule-rank)</span></h2> |
| <p class="desc">Each row is one forecasting method. The <b>left</b> bars show how accurate its <b>cost</b> forecast is; the <b>right</b> bars show how accurate its <b>finish-date</b> forecast is. <b>Shorter bars are better.</b> Methods are sorted with the best all-rounder (good at both) at the top. Pick a point in the project's life below, or choose "Average" to combine them.</p> |
| <div class="controls">completion level: |
| <select id="stage"></select> |
| </div> |
| <div class="row"> |
| <div class="panel"><div id="hl-cost" style="height:520px"></div></div> |
| <div class="panel"><div id="hl-sched" style="height:520px"></div></div> |
| </div> |
|
|
| <h2>Cost vs schedule trade-off <span style="font-weight:400;color:#888">(lower-left = better; ★ best, ● Pareto frontier)</span></h2> |
| <p class="desc">Every dot is one method. <b>Further left</b> = better cost forecasts; <b>further down</b> = better finish-date forecasts, so the <b>bottom-left corner is best</b>. The dashed line links the "best trade-offs" - methods where you cannot get better at one thing without getting worse at the other. The big diamond is the best all-rounder. Hover any dot for details.</p> |
| <div class="controls">completion level: |
| <select id="stage_sc"></select> |
| </div> |
| <div id="scatter" style="height:560px"></div> |
| <p class="note" id="scatter_off"></p> |
|
|
| <h2>Accuracy by project length <span style="font-weight:400;color:#888">(darker = worse; rows best-overall first)</span></h2> |
| <p class="desc">The same methods (rows, best at the top) split by <b>how long the project runs</b> (columns). Colour shows the error: <b>pale = accurate, dark red = inaccurate</b>. This reveals which methods stay reliable on <b>long projects</b> (usually the hardest) and which fall apart. Left grid = cost error, right grid = finish-date error.</p> |
| <div class="controls">completion level: |
| <select id="stage_st"></select> |
| </div> |
| <div class="row"> |
| <div class="panel"><div id="st-cost" style="height:640px"></div></div> |
| <div class="panel"><div id="st-sched" style="height:640px"></div></div> |
| </div> |
|
|
| <script> |
| Highcharts.setOptions({ |
| accessibility: { enabled: false }, credits: { enabled: false }, |
| tooltip: { |
| useHTML: true, outside: true, backgroundColor: "#ffffff", borderColor: "#9aa0a6", |
| borderRadius: 6, borderWidth: 1, shadow: true, style: { color: "#1a1a1a", fontSize: "12px" } |
| } |
| }); |
| const R = window.BENCH, CFG = window.CHART_CFG; |
| const FC = CFG.familyColor, FL = CFG.familyLabel, STRATA = CFG.strata; |
| const CAP_COST = CFG.capCost, CAP_FIN = CFG.capFin, ST_COST = CFG.stratCapCost, ST_FIN = CFG.stratCapFin; |
| const fam = m => R.methods[m].family; |
| const at = (m, s) => (R.methods[m].stages || {})[s] || null; |
| const mean = a => a.reduce((x, y) => x + y, 0) / a.length; |
| const AVG = "avg"; |
| |
| |
| const FAM_DESC = { |
| classical: "A traditional Earned Value Management formula from project controls - simple and fast, no learning.", |
| ml: "A machine-learning model trained on many past projects to spot patterns (a 'reference class').", |
| foundation: "A general-purpose time-series forecasting AI applied to the project's cost and progress curves.", |
| naive: "A deliberately simple baseline (e.g. assume the project finishes exactly as planned) - the bar everything should clear.", |
| control: "A null control: a non-informative guess, included to confirm the other methods genuinely add value.", |
| agent: "An AI agent that writes and runs code to reason its way to the forecast.", |
| other: "A forecasting method." |
| }; |
| const MODEL_DESC = { |
| "MiniCPM5-1B": "MiniCPM5, a tiny 1-billion-parameter open model", |
| "Qwen3.5-2B": "Qwen3.5 (Alibaba), a small 2-billion-parameter open model", |
| "Qwen3.5-4B": "Qwen3.5 (Alibaba), a 4-billion-parameter open model", |
| "gemma-4-E2B": "Gemma 4 (Google), a compact open model", |
| "Nemotron-4B": "Nemotron (NVIDIA), a 4-billion-parameter open model" |
| }; |
| const CV_NOTE = " Scored with cross-validation on the real projects (we repeatedly hide some projects, fit on the rest, then test on the hidden ones) - a stricter, leak-free test."; |
| const METHOD_DESC = { |
| agent_deepseek_v4_flash: "The big, expensive 'teacher' AI (DeepSeek V4). It writes and runs code to reason through each forecast. Too costly to deploy everywhere, so we use it to teach the small 'student' models.", |
| earned_schedule: "Earned Schedule, the standard project-controls method: compares work actually done against the plan to project the finish, and scales the budget by how efficiently money has been spent for the cost.", |
| evm_cpi_spi: "A classic Earned Value formula that blends cost-efficiency and schedule-efficiency to project the final cost. Tends to be pessimistic (over-estimates the overrun).", |
| xsm: "An Earned Value variant that weights the project's recent performance more than its running average when extrapolating.", |
| exp_smoothing: "Exponential smoothing: extrapolates progress while weighting recent periods more heavily than older ones.", |
| growth_curve: "Fits an S-shaped growth curve to the work done so far and extends it to predict the finish.", |
| logistic: "Fits a logistic (S-shaped) curve to the project's progress and extrapolates it to the finish.", |
| naive_onplan: "Assumes the project finishes exactly on its original schedule and budget - the 'nothing goes wrong' baseline.", |
| last_value: "Assumes the project carries on at its most recent rate - a 'more of the same' guess.", |
| linear: "Draws a straight line through the progress so far and extends it - the simplest possible extrapolation.", |
| naive_dist: "A null control: ignores this particular project and just guesses the average outcome across all projects. Any genuinely useful method should beat it.", |
| chronos_2: "Chronos, a general-purpose AI trained to forecast any time-series, applied here to the project's cost/progress curve.", |
| chronos_2_cov: "Chronos (a general time-series forecasting AI) given extra side-information about the project as well as its curve.", |
| "timesfm_2.5": "TimesFM (Google), a general-purpose time-series forecasting AI, applied to the project's curve.", |
| ml_catboost: "A 'gradient-boosted trees' model (CatBoost): learns from thousands of past projects by combining many simple decision rules.", |
| ml_lightgbm: "A 'gradient-boosted trees' model (LightGBM): learns patterns from many past projects by combining simple decision rules.", |
| ml_lightgbm_quantile: "LightGBM (gradient-boosted trees) tuned to predict a likely range rather than a single number.", |
| ml_xgboost: "A 'gradient-boosted trees' model (XGBoost): a popular method that learns from many past projects.", |
| ml_histgbm: "A histogram-based 'gradient-boosted trees' model: learns project patterns from many past examples.", |
| ml_rf: "A 'random forest': averages the predictions of many independent decision trees trained on past projects.", |
| ml_ridge: "Ridge regression: a straightforward linear model fitted to features of past projects.", |
| ml_svr: "A support-vector regression model fitted to features of past projects.", |
| ml_mlp: "A small neural network trained on features of past projects.", |
| ml_tabpfn: "TabPFN: a pre-trained AI that makes table-based predictions instantly, without training a new model per dataset." |
| }; |
| function methodBlurb(m) { |
| if (m.indexOf("student_") === 0) { |
| const variant = m.slice(-4) === "_sft" ? "sft" : "base"; |
| const mid = m.replace(/^student_/, "").replace(/_(base|sft)$/, ""); |
| const name = MODEL_DESC[mid] || ("a small open model (" + mid + ")"); |
| return variant === "sft" |
| ? "A small 'student': " + name + ", AFTER we fine-tuned it on the teacher AI's worked examples - a distilled model we are shipping." |
| : "A small 'student': " + name + ", straight out of the box with no training from us - the 'before' picture, for comparison."; |
| } |
| let key = m, cv = ""; |
| if (m.slice(-7) === "_realcv") { key = m.slice(0, -7); cv = CV_NOTE; } |
| if (METHOD_DESC[key]) return METHOD_DESC[key] + cv; |
| return FAM_DESC[fam(m)] || FAM_DESC.other; |
| } |
| const COST_HELP = "How far the predicted FINAL COST was from the project's real final cost, as a percentage, for the typical (median) project. 3% means within 3% of the actual bill. Lower is better."; |
| const FIN_HELP = "How many time-periods the predicted FINISH was off by, for the typical (median) project. 0.7 means less than one period out. Lower is better."; |
| const LEN_HELP = "Projects grouped by how long they run. Longer projects are harder to forecast, so we check that methods hold up as length grows."; |
| const esc = s => String(s).replace(/"/g, """); |
| const axisTitle = (text, help) => text + ' <span class="info" title="' + esc(help) + '">ⓘ</span>'; |
| const labelHTML = (val, help) => '<span title="' + esc(help) + '">' + val + "</span>"; |
| |
| function statsFor(stage) { |
| const out = {}; |
| if (stage === AVG) { |
| Object.keys(R.methods).forEach(m => { |
| const v = R.stages.map(s => at(m, s)); |
| if (v.every(x => x && x.eac_ape_med != null && x.finish_err_med != null)) |
| out[m] = { cost: mean(v.map(x => x.eac_ape_med)), fin: mean(v.map(x => x.finish_err_med)) }; |
| }); |
| } else { |
| Object.keys(R.methods).forEach(m => { |
| const a = at(m, stage); |
| if (a && a.eac_ape_med != null) out[m] = { cost: a.eac_ape_med, fin: a.finish_err_med }; |
| }); |
| } |
| return out; |
| } |
| function orderFromStats(st) { |
| const names = Object.keys(st); |
| const cr = {}, fr = {}; |
| names.slice().sort((a, b) => st[a].cost - st[b].cost).forEach((m, i) => cr[m] = i); |
| names.slice().sort((a, b) => st[a].fin - st[b].fin).forEach((m, i) => fr[m] = i); |
| return names.sort((a, b) => (cr[a] + fr[a]) - (cr[b] + fr[b])); |
| } |
| const pctLabel = s => s === AVG ? "Average (all splits)" : Math.round(parseFloat(s) * 100) + "%"; |
| |
| function barPanel(elId, order, valOf, cap, title, axisText, axisHelp, kind) { |
| Highcharts.chart(elId, { |
| chart: { type: "bar" }, title: { text: title, style: { fontSize: "13px" } }, |
| credits: { enabled: false }, legend: { enabled: false }, |
| xAxis: { categories: order, reversed: true, labels: { useHTML: true, style: { fontSize: "11px" }, |
| formatter: function () { return labelHTML(this.value, methodBlurb(this.value)); } } }, |
| yAxis: { min: 0, max: cap * 1.12, title: { useHTML: true, text: axisTitle(axisText, axisHelp) } }, |
| tooltip: { useHTML: true, formatter: function () { |
| const a = this.point.actual, m = this.point.mname; |
| const line = kind === "cost" |
| ? "Cost forecast was within <b>" + a.toFixed(a < 10 ? 2 : 1) + "%</b> of the real final cost" |
| : "Finish-date forecast was off by <b>" + a.toFixed(2) + "</b> period(s)"; |
| return "<b>" + m + "</b><br/><span style='color:#777;font-size:11px'>" + methodBlurb(m) + |
| "</span><br/>" + line + " for the typical project."; } }, |
| plotOptions: { bar: { dataLabels: { enabled: true, formatter: function () { |
| const a = this.point.actual; return a.toFixed(a < 10 ? 2 : 1) + (a > cap ? " →" : ""); }, |
| style: { fontSize: "10px", textOutline: "none" } } } }, |
| series: [{ data: order.map(m => ({ y: Math.min(valOf(m), cap), actual: valOf(m), color: FC[fam(m)], mname: m })) }] |
| }); |
| } |
| function renderHeadline(stage) { |
| const st = statsFor(stage), order = orderFromStats(st); |
| barPanel("hl-cost", order, m => st[m].cost, CAP_COST, "Cost accuracy", |
| "Cost forecast error % (lower = better)", COST_HELP, "cost"); |
| barPanel("hl-sched", order, m => st[m].fin, CAP_FIN, "Schedule accuracy", |
| "Finish-date error, periods (lower = better)", FIN_HELP, "sched"); |
| } |
| |
| function renderScatter(stage) { |
| const st = statsFor(stage), names = Object.keys(st); |
| const cost = m => st[m].cost, fin = m => st[m].fin; |
| const pareto = names.filter(m => !names.some(o => |
| cost(o) <= cost(m) && fin(o) <= fin(m) && (cost(o) < cost(m) || fin(o) < fin(m)))); |
| const best = orderFromStats(st)[0]; |
| const byFam = {}, off = []; |
| names.forEach(m => { |
| if (cost(m) > CAP_COST || fin(m) > CAP_FIN) { off.push(m + " (" + cost(m).toFixed(1) + "%, " + fin(m).toFixed(1) + ")"); return; } |
| (byFam[fam(m)] = byFam[fam(m)] || []).push({ |
| x: cost(m), y: fin(m), name: m, |
| marker: m === best ? { symbol: "diamond", radius: 9, lineColor: "#000", lineWidth: 1 } |
| : pareto.includes(m) ? { symbol: "circle", radius: 6, lineColor: "#000", lineWidth: 1 } |
| : { symbol: "circle", radius: 4, fillOpacity: 0.45 } }); |
| }); |
| const series = Object.keys(byFam).map(f => ({ name: FL[f] || f, color: FC[f], data: byFam[f], |
| marker: { symbol: "circle" } })); |
| const pf = pareto.filter(m => cost(m) <= CAP_COST && fin(m) <= CAP_FIN).sort((a, b) => cost(a) - cost(b)); |
| series.push({ type: "line", name: "Pareto frontier", color: "#888", dashStyle: "Dash", |
| enableMouseTracking: false, showInLegend: false, marker: { enabled: false }, |
| data: pf.map(m => [cost(m), fin(m)]) }); |
| Highcharts.chart("scatter", { |
| chart: { type: "scatter", zoomType: "xy" }, credits: { enabled: false }, |
| title: { text: "Cost vs schedule at " + pctLabel(stage) + " complete", style: { fontSize: "13px" } }, |
| xAxis: { min: 0, max: CAP_COST, gridLineWidth: 1, |
| title: { useHTML: true, text: axisTitle("Cost forecast error %", COST_HELP) } }, |
| yAxis: { min: 0, max: CAP_FIN, |
| title: { useHTML: true, text: axisTitle("Finish-date error (periods)", FIN_HELP) } }, |
| tooltip: { useHTML: true, formatter: function () { |
| return "<b>" + this.point.name + "</b><br/><span style='color:#777;font-size:11px'>" + |
| methodBlurb(this.point.name) + "</span><br/>Cost within <b>" + this.point.x.toFixed(2) + |
| "%</b>, finish off by <b>" + this.point.y.toFixed(2) + "</b> period(s)."; } }, |
| series: series |
| }); |
| document.getElementById("scatter_off").textContent = off.length ? "off-chart: " + off.join(" ") : ""; |
| } |
| |
| function renderStratified(stage) { |
| const order = orderFromStats(statsFor(stage)); |
| const pretty = s => s.replace(/_/g, " "); |
| function heat(elId, key, cap, title, unit) { |
| const data = []; |
| order.forEach((m, yi) => STRATA.forEach((s, xi) => { |
| const cell = (R.methods[m].by_strata || {})[s] || {}; |
| const v = cell[key]; |
| if (v != null && isFinite(v)) data.push({ x: xi, y: yi, value: Math.min(v, cap), actual: v }); |
| })); |
| Highcharts.chart(elId, { |
| chart: { type: "heatmap" }, credits: { enabled: false }, |
| title: { useHTML: true, text: axisTitle(title, key === "eac_ape_med" ? COST_HELP : FIN_HELP), |
| style: { fontSize: "13px" } }, |
| xAxis: { categories: STRATA, title: { useHTML: true, text: axisTitle("Project length", LEN_HELP) }, |
| labels: { useHTML: true, formatter: function () { return labelHTML(pretty(this.value), LEN_HELP); } } }, |
| yAxis: { categories: order, reversed: true, title: null, |
| labels: { useHTML: true, style: { fontSize: "10px" }, |
| formatter: function () { return labelHTML(this.value, methodBlurb(this.value)); } } }, |
| colorAxis: { min: 0, max: cap, stops: [[0, "#ffffcc"], [0.35, "#fed976"], [0.6, "#fd8d3c"], |
| [0.8, "#f03b20"], [1, "#bd0026"]] }, legend: { enabled: false }, |
| tooltip: { useHTML: true, formatter: function () { const m = order[this.point.y]; |
| return "<b>" + m + "</b><br/><span style='color:#777;font-size:11px'>" + methodBlurb(m) + |
| "</span><br/>" + pretty(STRATA[this.point.x]) + " projects: <b>" + this.point.actual.toFixed(2) + |
| "</b> " + unit; } }, |
| series: [{ borderWidth: 1, borderColor: "#fff", data: data, |
| dataLabels: { enabled: true, formatter: function () { return this.point.actual.toFixed(1); }, |
| style: { fontSize: "9px", textOutline: "none" } } }] |
| }); |
| } |
| heat("st-cost", "eac_ape_med", ST_COST, "Cost forecast error %", "% off the real cost"); |
| heat("st-sched", "finish_err_med", ST_FIN, "Finish-date error (periods)", "periods off the real finish"); |
| } |
| |
| |
| const STAGE_OPTS = R.stages.concat([AVG]); |
| function fillSelect(el, withAvg, dflt) { |
| (withAvg ? STAGE_OPTS : R.stages).forEach(s => { |
| const o = document.createElement("option"); o.value = s; o.textContent = pctLabel(s); |
| if (s === dflt) o.selected = true; el.appendChild(o); |
| }); |
| } |
| document.getElementById("subtitle").textContent = |
| "How well different methods forecast a project's final cost and finish date, tested on " + R.n_eval + |
| " real projects the models never saw during training. Colour = type of method (hover for what each is)."; |
| document.getElementById("legend").innerHTML = Object.keys(FL) |
| .filter(f => Object.keys(R.methods).some(m => fam(m) === f)) |
| .map(f => '<span title="' + esc(FAM_DESC[f] || "") + '"><i class="sw" style="background:' + |
| FC[f] + '"></i>' + FL[f] + "</span>").join(""); |
| |
| const primary = String(R.primary); |
| const selHl = document.getElementById("stage"), selSc = document.getElementById("stage_sc"), selSt = document.getElementById("stage_st"); |
| fillSelect(selHl, true, AVG); fillSelect(selSc, false, primary); fillSelect(selSt, false, primary); |
| selHl.onchange = () => renderHeadline(selHl.value); |
| selSc.onchange = () => renderScatter(selSc.value); |
| selSt.onchange = () => renderStratified(selSt.value); |
| renderHeadline(AVG); renderScatter(primary); renderStratified(primary); |
| </script> |
| </body> |
| </html> |
|
|
