Update TERN package, release test questions with private SQL, and accept integer prediction IDs
028437a verified Download index.html from THULab/TERN: direct link, hf CLI and curl.
- Browser
- Download file 19.5 kB
-
https://huggingface.co/spaces/THULab/TERN/resolve/main/index.html
- Command line
-
hf download hf://spaces/THULab/TERN/index.html
-
curl -L -o index.html https://huggingface.co/spaces/THULab/TERN/resolve/main/index.html
19.5 kB
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <meta name="description" content="TERN is a time-series text-to-SQL benchmark generated by TSSDA, with 13,182 samples across 32 Apache IoTDB database instances and three evaluation levels."> | |
| <title>TERN — Time-Series Text-to-SQL Benchmark</title> | |
| <link rel="icon" href="assets/favicon.svg" type="image/svg+xml"> | |
| <link rel="stylesheet" href="assets/styles.css?v=20261117-download-tooltip"> | |
| <script defer src="assets/leaderboard-data.js"></script> | |
| <script defer src="assets/submission-config.js"></script> | |
| <script defer src="assets/email-submission.js?v=20261117-submission-setting"></script> | |
| <script defer src="assets/app.js?v=20261119-package-update"></script> | |
| <script defer src="assets/download-counter-config.js?v=20261117-1"></script> | |
| <script defer src="assets/download-counter.js?v=20261117-1"></script> | |
| </head> | |
| <body data-page="home"> | |
| <a class="skip-link" href="#main">Skip to content</a> | |
| <header data-site-header></header> | |
| <main id="main" class="container"> | |
| <section class="intro" id="overview" data-nav-section> | |
| <div class="project-name"><img src="assets/favicon.svg" width="56" height="56" alt=""><h1>TERN</h1></div> | |
| <p class="project-subtitle">A Benchmark for Time-Series Text-to-SQL</p> | |
| <div class="intro-links"><a class="button button-resource" href="https://arxiv.org/abs/2611.02132" target="_blank" rel="noopener noreferrer"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false"><path d="M14 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V8Z"/><path d="M14 2v6h6M8 13h8M8 17h6"/></svg><span>Paper</span></a><a class="button button-resource" href="https://github.com/chris265100/TSSDA" target="_blank" rel="noopener noreferrer"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false"><path fill="currentColor" stroke="none" d="M12 .8a11.2 11.2 0 0 0-3.54 21.83c.56.1.77-.24.77-.54v-2.1c-3.13.68-3.79-1.33-3.79-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.03-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.4-1.22.71-1.5-2.5-.28-5.12-1.25-5.12-5.56 0-1.23.44-2.23 1.15-3.01-.12-.28-.5-1.43.11-2.97 0 0 .94-.3 3.08 1.15a10.7 10.7 0 0 1 5.6 0c2.14-1.45 3.08-1.15 3.08-1.15.61 1.54.23 2.69.11 2.97.72.78 1.15 1.78 1.15 3.01 0 4.32-2.63 5.27-5.14 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.2 11.2 0 0 0 12 .8Z"/></svg><span>Code</span></a><a class="button button-resource" href="#dataset"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false"><ellipse cx="12" cy="5" rx="8" ry="3"/><path d="M4 5v7c0 1.66 3.58 3 8 3s8-1.34 8-3V5M4 12v7c0 1.66 3.58 3 8 3s8-1.34 8-3v-7"/></svg><span>Dataset</span></a><a class="button button-resource" href="#submit"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false"><path d="m22 2-7 20-4-9-9-4Z"/><path d="m22 2-11 11"/></svg><span>Submission</span></a></div> | |
| <div class="project-summary" aria-labelledby="about-title"> | |
| <h2 id="about-title">About TERN</h2> | |
| <p>TERN is a text-to-SQL benchmark for time-series databases, automatically generated by <a href="https://arxiv.org/abs/2611.02132" target="_blank" rel="noopener noreferrer">TSSDA</a>, an autonomous data generation agent framework. Built on <strong>Apache IoTDB</strong>, it spans basic temporal queries through complex time-series analysis.</p> | |
| <p>The benchmark contains <strong>13,182 samples</strong> across <strong>32 database instances</strong>, with test sets organized into three levels: Foundation, Standard, and Analytical.</p> | |
| </div> | |
| </section> | |
| <section class="page-section" id="leaderboard" data-nav-section aria-labelledby="leaderboard-title"> | |
| <div class="section-heading"><h2 id="leaderboard-title">Leaderboard</h2><span class="metric-note">Execution accuracy (EX %) · Higher is better</span></div> | |
| <div class="board-toolbar"><div class="tab-list" role="tablist" aria-label="Rank by tier"><button type="button" class="tab-button is-active" role="tab" aria-selected="true" aria-controls="board-panel" id="tier-overall" data-tier="overall">Overall</button><button type="button" class="tab-button" role="tab" aria-selected="false" aria-controls="board-panel" id="tier-foundation" tabindex="-1" data-tier="foundation">Foundation</button><button type="button" class="tab-button" role="tab" aria-selected="false" aria-controls="board-panel" id="tier-standard" tabindex="-1" data-tier="standard">Standard</button><button type="button" class="tab-button" role="tab" aria-selected="false" aria-controls="board-panel" id="tier-analytical" tabindex="-1" data-tier="analytical">Analytical</button></div><div class="board-controls"><label class="search-field"><span aria-hidden="true">⌕</span><input type="search" id="method-search" placeholder="Search methods or papers" aria-label="Search methods or papers"></label></div></div> | |
| <div class="table-card" id="board-panel" role="tabpanel" aria-labelledby="tier-overall"><div class="table-scroll"><table class="leaderboard-table full-table"><caption class="sr-only">TERN leaderboard. Execution accuracy scores are percentages.</caption><thead><tr><th scope="col">Rank</th><th scope="col">Method / Model</th><th scope="col" class="numeric" data-sort-header="overall" aria-sort="descending"><button type="button" data-sort="overall">Overall EX <span>↓</span></button></th><th scope="col" class="numeric" data-sort-header="foundation"><button type="button" data-sort="foundation">Foundation EX <span>↕</span></button></th><th scope="col" class="numeric" data-sort-header="standard"><button type="button" data-sort="standard">Standard EX <span>↕</span></button></th><th scope="col" class="numeric" data-sort-header="analytical"><button type="button" data-sort="analytical">Analytical EX <span>↕</span></button></th><th scope="col">Details</th></tr></thead><tbody data-full-leaderboard></tbody></table></div><div class="empty-state" data-empty-state hidden><span aria-hidden="true">⌕</span><h3>No matching methods</h3><p>Try another method name or paper title.</p><button type="button" class="button button-outline" data-reset-filters>Clear search</button></div><div class="table-footer"><span data-result-count role="status" aria-live="polite"></span><span>Scores in % · Higher is better</span></div></div> | |
| <p class="section-note">Select a tier to rank by its EX score. Bold values mark the best score in each column. Click a method for details or an author citation to read its paper.</p> | |
| </section> | |
| <section class="page-section" id="dataset" data-nav-section aria-labelledby="dataset-title"> | |
| <div class="section-heading"><h2 id="dataset-title">Dataset</h2></div> | |
| <p class="dataset-description">Includes train/dev NL–SQL pairs, test questions and full database contents; test gold SQL is withheld.<br class="desktop-break">Submit predictions via <a href="#submit">Submission</a> for results by email; online EX evaluation is coming soon.</p> | |
| <div class="dataset-packages" id="downloads"> | |
| <article class="dataset-package" aria-labelledby="dataset-package-title"> | |
| <div class="package-heading"><h3 id="dataset-package-title">TERN Dataset</h3><span class="package-kind">Complete package</span></div> | |
| <p>The complete dataset, database contents, and Apache IoTDB distribution in one package for local deployment and execution accuracy (EX) evaluation on train/dev.</p> | |
| <div class="package-download"> | |
| <a class="button button-download" href="downloads/TERN.zip" download="TERN.zip" aria-describedby="dataset-download-status"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false"><path d="M12 3v12m-5-5 5 5 5-5M5 16v4h14v-4"/></svg>Download Dataset</a> | |
| <span class="download-meta"> | |
| <span class="download-status" id="dataset-download-status">ZIP · 526 MB</span> | |
| <span class="download-count-hint"> | |
| <button type="button" class="download-count-trigger" aria-label="Download statistics" aria-describedby="dataset-download-count"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6" stroke-linecap="round" aria-hidden="true" focusable="false"><circle cx="12" cy="12" r="8.5"/><path d="M12 11v5M12 7.5h.01"/></svg></button> | |
| <span class="download-count-tooltip" id="dataset-download-count" role="tooltip"><span>Download requests</span><strong data-download-count>—</strong><span class="download-count-note">Count not available yet</span></span> | |
| </span> | |
| </span> | |
| </div> | |
| </article> | |
| </div> | |
| </section> | |
| <section class="page-section" id="submit" data-nav-section aria-labelledby="submit-title"> | |
| <div class="section-heading"><h2 id="submit-title">Submission</h2><span class="release-tag" data-email-submission-status>Email submissions coming soon</span></div> | |
| <p>Submit your prediction file and method details by email for official review. The organizers will contact you about verification and leaderboard inclusion. Instant EX feedback for the test set is coming soon.</p> | |
| <ol class="submission-steps"><li><span>1</span> Prepare predictions</li><li><span>2</span> Check file format</li><li><span>3</span> Submit for review</li></ol> | |
| <p class="section-note" id="email-submission-notice" role="status">Email submissions will open once the receiving address is confirmed. You can prepare and validate your file below.</p> | |
| <details class="expandable submission-workspace" id="prepare-submission"><summary>Prepare and submit for leaderboard review</summary> | |
| <div class="submit-layout"><div><form id="submission-form" novalidate> | |
| <section class="form-section"><div class="form-section-title"><span>01</span><div><h3>Tell us about your method</h3><p>Name your method and provide a contact email for review feedback.</p></div></div><div class="form-grid"><label>Method name <span class="required">*</span><input name="method" id="method-name" required maxlength="120" placeholder="e.g. Your method name" autocomplete="off"></label><label>Team / Organization <span class="required">*</span><input name="team" id="team-name" required maxlength="120" placeholder="e.g. Your research team" autocomplete="organization"></label><label>Contact email <span class="required">*</span><input name="email" id="contact-email" type="email" required maxlength="254" placeholder="For submission feedback" autocomplete="email"></label><label>Model & version <span class="optional">Optional</span><input name="model" id="model-name" maxlength="160" placeholder="Exact model or checkpoint version" autocomplete="off"></label><label>Method setting <span class="optional">Optional</span><select name="setting" id="method-setting"><option value="" selected aria-label="Not specified"></option><option value="SFT">SFT</option><option value="ICL">ICL</option></select></label><label>Paper / Code URL <span class="optional">Optional</span><input name="url" id="resource-url" type="url" placeholder="https://…" autocomplete="url"></label></div></section> | |
| <section class="form-section"><div class="form-section-title"><span>02</span><div><h3>Add your predictions</h3><p>One final SQL prediction per question ID.</p></div></div><label class="dropzone" id="prediction-dropzone" for="prediction-file"><input type="file" id="prediction-file" name="predictions" accept=".json,application/json"><span class="upload-icon" aria-hidden="true">↥</span><strong data-file-label>Drop your predictions.json here</strong><span>or <u>browse files</u> · JSON · Up to 10 MB</span></label><div class="file-message" id="file-message" role="status" aria-live="polite" hidden></div></section> | |
| <section class="form-section"><div class="form-section-title"><span>03</span><div><h3>Check and submit</h3><p>Check the format, then send your file for organizer review.</p></div></div><label class="checkbox-label agreement"><input type="checkbox" id="submission-agreement" required><span>I confirm that these predictions were generated by the described method without private reference answers, and agree to send this file and submission details for leaderboard review. <span class="required" aria-hidden="true">*</span></span></label><div class="submission-actions"><button type="button" class="button button-outline" id="validate-submission">Validate locally</button><button type="submit" class="button button-primary" id="send-submission" disabled aria-describedby="email-submission-notice">Submit for leaderboard <span aria-hidden="true">→</span></button></div><p class="small muted">Local checks validate file structure and submission metadata. They do not determine SQL correctness or benchmark coverage.</p><div class="validation-result" id="validation-result" role="status" aria-live="polite" hidden></div></section> | |
| </form></div><aside class="submission-sidebar"><div class="sidebar-panel" id="submission-format"><div class="eyebrow">PREDICTION FORMAT</div><h3>Just IDs and SQL.</h3><p>Keep question IDs unchanged. Include one final query for each test question.</p><div class="code-panel"><div class="code-heading"><span>predictions.json</span><button type="button" data-copy="predictions">Copy <span aria-hidden="true">⧉</span></button></div><pre><code data-prediction-example></code></pre></div><a class="text-link" href="templates/submission.schema.json" download="tern-submission.schema.json">Download JSON Schema <span aria-hidden="true">↓</span></a></div><div class="sidebar-panel subtle-panel"><div class="eyebrow">BEFORE YOU SUBMIT</div><ul class="check-list"><li>Keep each question ID unique.</li><li>Include SQL as a non-empty string.</li><li>Disclose your model and tools.</li></ul><a class="text-link" href="#examples">Dataset examples <span aria-hidden="true">→</span></a></div><div class="privacy-note"><span aria-hidden="true">◇</span><p>When you submit, your prediction file and contact details are forwarded via FormSubmit to the TERN team for review.</p></div></aside></div> | |
| </details> | |
| </section> | |
| <section class="page-section dataset-examples" id="examples" data-nav-section aria-labelledby="examples-title"> | |
| <h2 id="examples-title">Dataset Examples</h2> | |
| <p>Two representative questions from the TERN training set, with their gold SQL. Times use Asia/Shanghai (UTC+08:00); query ranges include the start and exclude the end.</p> | |
| <article class="dataset-example" aria-labelledby="example-1-title"> | |
| <header class="example-heading"><div><p class="example-number">EXAMPLE 01</p><h3 id="example-1-title">Machine temperature stability</h3></div><span class="example-source">Training · train_001054</span></header> | |
| <div class="example-body"> | |
| <div class="example-question"><h4>Question</h4><p>For the machine, compute the population variance of process temperature and the average tool wear across overlapping 1-day windows that advance every 6 hours from 2020-01-01T00:00:00 to 2020-01-07T22:39:00 (end exclusive). Keep only records where tool wear is at least 180 minutes or torque exceeds 65 Nm, while air temperature stays at or below 305 K, process temperature is at least 305 K, and rotational speed is under 2800 RPM. Fill gaps with the previous value within a 1-hour limit, order the windows by time ascending, skip the first 6 rows, and return at most 20 rows. Name the results process_temperature_variance and avg_tool_wear_min.</p><div class="example-context"><span>Database</span><code>root.maintenance</code><dl class="example-units" aria-label="Measurement units"><div><dt>Process and air temperature</dt><dd>K</dd></div><div><dt>Tool wear</dt><dd>min</dd></div><div><dt>Torque</dt><dd>Nm</dd></div><div><dt>Rotational speed</dt><dd>RPM</dd></div></dl></div></div> | |
| <div class="code-panel example-sql"><div class="code-heading"><span>Gold SQL</span><button type="button" data-copy-source="example-sql-1" aria-label="Copy SQL for machine temperature stability">Copy SQL</button></div><pre tabindex="0" aria-label="SQL for machine temperature stability"><code id="example-sql-1">SELECT | |
| VAR_POP(Process_Temperature_K) AS process_temperature_variance, | |
| AVG(Tool_Wear_Min) AS avg_tool_wear_min | |
| FROM root.maintenance.machine | |
| WHERE (Tool_Wear_Min >= 180 OR Torque_Nm > 65) | |
| AND Air_Temperature_K <= 305 | |
| AND Process_Temperature_K >= 305 | |
| AND Rotational_Speed_RPM < 2800 | |
| GROUP BY( | |
| [2020-01-01T00:00:00, 2020-01-07T22:39:00), 1d, 6h | |
| ) | |
| ORDER BY TIME ASC | |
| FILL(PREVIOUS, 1h) | |
| LIMIT 20 OFFSET 6</code></pre></div> | |
| </div> | |
| </article> | |
| <article class="dataset-example" aria-labelledby="example-2-title"> | |
| <header class="example-heading"><div><p class="example-number">EXAMPLE 02</p><h3 id="example-2-title">Hourly bike-share demand</h3></div><span class="example-source">Training · train_000163</span></header> | |
| <div class="example-body"> | |
| <div class="example-question"><h4>Question</h4><p>For the NewportPkwy station from June 5 through June 11, 2023, show the hourly trend of total activity, departures, and arrivals as a percentage of total activity, considering only hours with positive total activity; fill missing hourly values with 0, order by time ascending, and return the first 24 rows.</p><div class="example-context"><span>Database</span><code>root.bikeshare</code><p>Total activity = departures + arrivals; all three series count trips.</p></div></div> | |
| <div class="code-panel example-sql"><div class="code-heading"><span>Gold SQL</span><button type="button" data-copy-source="example-sql-2" aria-label="Copy SQL for hourly bike-share demand">Copy SQL</button></div><pre tabindex="0" aria-label="SQL for hourly bike-share demand"><code id="example-sql-2">SELECT | |
| SUM(total_activity) AS total_activity, | |
| SUM(departures) AS departures_count, | |
| (SUM(total_activity) - SUM(departures)) | |
| / SUM(total_activity) * 100.0 AS arrival_percentage | |
| FROM root.bikeshare.NewportPkwy | |
| WHERE total_activity > 0 | |
| GROUP BY( | |
| [2023-06-05T00:00:00, 2023-06-12T00:00:00), 1h | |
| ) | |
| ORDER BY TIME ASC | |
| FILL(0) | |
| LIMIT 24</code></pre></div> | |
| </div> | |
| </article> | |
| </section> | |
| <section class="citation-section" id="citation"><h2>Citation & resources</h2><p>Read <a href="https://arxiv.org/abs/2611.02132" target="_blank" rel="noopener noreferrer">TSSDA: An Autonomous Agent for Constructing a Time-Series Text-to-SQL Benchmark</a> and explore the <a href="https://github.com/chris265100/TSSDA" target="_blank" rel="noopener noreferrer">code repository</a>.</p></section> | |
| </main> | |
| <footer data-site-footer></footer> | |
| <dialog class="method-dialog" id="method-dialog" aria-labelledby="method-dialog-title"><div data-method-details></div></dialog> | |
| <div class="toast" role="status" aria-live="polite" hidden></div> | |
| </body> | |
| </html> | |