Spaces:
Running
Running
cleaner showcase v1: static build, real sample report; gradio app kept for PRO upgrade
b485a88 verified | """Static showcase page for the cleaner (free-tier Space). | |
| Renders the REAL issue report from running modelbrew_cleaner on the bundled | |
| dirty sample — no fabricated output. Run inside the package venv: | |
| PYTHONPATH= ../modelbrew-cleaner/.venv/bin/python build_static.py | |
| """ | |
| import html | |
| from pathlib import Path | |
| from modelbrew_cleaner import Severity, clean_file, issue_summary | |
| HERE = Path(__file__).parent | |
| rows = clean_file(str(HERE / "sample.jsonl")) | |
| summary = issue_summary(rows) | |
| n_critical_rows = sum(1 for r in rows if any(i.severity == Severity.critical for i in r.issues)) | |
| SEV_ORDER = {"critical": 0, "warning": 1, "suggestion": 2} | |
| records = sorted( | |
| ((r.row_index, i) for r in rows for i in r.issues), | |
| key=lambda x: (SEV_ORDER[x[1].severity.value], x[0]), | |
| ) | |
| table_rows = "".join( | |
| f"<tr><td>{idx}</td><td><span class='pill {i.severity.value}'>{i.severity.value}</span></td>" | |
| f"<td><code>{html.escape(i.code)}</code></td><td>{html.escape(i.message)}</td>" | |
| f"<td>{'✔' if i.auto_fixable else ''}</td></tr>" | |
| for idx, i in records | |
| ) | |
| page = f"""<!doctype html> | |
| <html lang="en"><head><meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>ModelBrew Dataset Cleaner</title> | |
| <style> | |
| :root {{ --bg:#fff; --fg:#1a1a1a; --muted:#6a6a6a; --line:#e3e3e3; --accent:#e17100; --code:#f4f4f4; | |
| --crit-bg:#fdecea; --crit-fg:#8c1d18; --warn-bg:#fff4e0; --warn-fg:#7a4b00; --sugg-bg:#e8f0fe; --sugg-fg:#1a4b8c; }} | |
| @media (prefers-color-scheme: dark) {{ | |
| :root {{ --bg:#101010; --fg:#eaeaea; --muted:#9a9a9a; --line:#2b2b2b; --accent:#ff9e2c; --code:#1d1d1d; | |
| --crit-bg:#3a1512; --crit-fg:#ff9d94; --warn-bg:#3a2c10; --warn-fg:#ffce7a; --sugg-bg:#12233a; --sugg-fg:#9cc4ff; }} }} | |
| body {{ margin:0; background:var(--bg); color:var(--fg); font:16px/1.55 system-ui,-apple-system,Segoe UI,Roboto,sans-serif; }} | |
| main {{ max-width:960px; margin:0 auto; padding:2.5rem 1.25rem 4rem; }} | |
| h1 {{ font-size:1.7rem; margin:0 0 .3rem; }} h2 {{ margin:2rem 0 .6rem; font-size:1.15rem; }} | |
| .sub {{ color:var(--muted); margin:0 0 1.2rem; }} | |
| a {{ color:var(--accent); text-decoration:none; }} a:hover {{ text-decoration:underline; }} | |
| pre {{ background:var(--code); padding:.9rem 1rem; border-radius:8px; overflow-x:auto; font-size:.85rem; }} | |
| code {{ background:var(--code); padding:.1rem .35rem; border-radius:4px; font-size:.85em; }} | |
| pre code {{ padding:0; background:none; }} | |
| .tablewrap {{ overflow-x:auto; border:1px solid var(--line); border-radius:8px; }} | |
| table {{ border-collapse:collapse; width:100%; font-size:.85rem; }} | |
| th,td {{ text-align:left; padding:.5rem .65rem; border-bottom:1px solid var(--line); vertical-align:top; }} | |
| th {{ color:var(--muted); font-weight:600; }} tr:last-child td {{ border-bottom:none; }} | |
| .pill {{ padding:.12rem .5rem; border-radius:99px; font-size:.75rem; white-space:nowrap; }} | |
| .pill.critical {{ background:var(--crit-bg); color:var(--crit-fg); }} | |
| .pill.warning {{ background:var(--warn-bg); color:var(--warn-fg); }} | |
| .pill.suggestion {{ background:var(--sugg-bg); color:var(--sugg-fg); }} | |
| .cta {{ display:inline-block; margin:.3rem .6rem .3rem 0; padding:.55rem 1rem; border-radius:8px; | |
| background:var(--accent); color:#fff !important; font-weight:600; }} | |
| footer {{ margin-top:3rem; color:var(--muted); font-size:.85rem; border-top:1px solid var(--line); padding-top:1rem; }} | |
| </style></head><body><main> | |
| <h1>🧹 ModelBrew Dataset Cleaner</h1> | |
| <p class="sub">90+ quality checks for fine-tuning datasets: PII with real checksum validation, | |
| exact/near duplicates, prompt-injection & jailbreak patterns, label errors, truncated | |
| responses, and more. Free and open source.</p> | |
| <a class="cta" href="https://app.modelbrew.ai/clean">Clean your dataset in the browser →</a> | |
| <a class="cta" href="https://github.com/ackerman404/modelbrew-cleaner" style="background:#333">GitHub</a> | |
| <h2>Or in your pipeline</h2> | |
| <pre><code>pip install modelbrew-cleaner | |
| from modelbrew_cleaner import clean_file, issue_summary, export_clean | |
| rows = clean_file("train.jsonl") # .jsonl, .json, or .csv | |
| print(issue_summary(rows)) # {summary} | |
| cleaned = export_clean(rows) # critical rows dropped</code></pre> | |
| <h2>Real output — the bundled dirty sample ({len(rows)} rows)</h2> | |
| <p class="sub">This report is generated by actually running the cleaner on | |
| <a href="sample.jsonl">sample.jsonl</a> at build time — {summary['critical']} critical / | |
| {summary['warning']} warnings / {summary['suggestion']} suggestions; | |
| {n_critical_rows} rows dropped from the cleaned export.</p> | |
| <div class="tablewrap"><table> | |
| <thead><tr><th>row</th><th>severity</th><th>check</th><th>message</th><th>auto-fix</th></tr></thead> | |
| <tbody>{table_rows}</tbody></table></div> | |
| <h2>Why we built it</h2> | |
| <p>We work on fine-tuning without catastrophic forgetting (patent-pending CRMA adapters). | |
| Measuring forgetting honestly forced us to fix our data first — in our measurements, dataset | |
| confounds alone accounted for a 96.9-percentage-point swing in measured forgetting. | |
| Read: <a href="{{{{HF_ARTICLE_URL}}}}">Your forgetting benchmark is lying to you</a> · | |
| Browse the <a href="https://huggingface.co/spaces/ModelBrew/forgetting-leaderboard">forgetting leaderboard</a>.</p> | |
| <footer><a href="https://modelbrew.ai">ModelBrew</a> · Apache-2.0 · every number we publish links to a raw results file.</footer> | |
| </main></body></html> | |
| """ | |
| (HERE / "index.html").write_text(page) | |
| print(f"wrote index.html ({len(page)} bytes) — report rows: {len(records)}") | |