README.md

metadata

title: Dataset Creator
emoji: 🧩
colorFrom: indigo
colorTo: pink
sdk: gradio
sdk_version: 6.19.0
app_file: app.py
pinned: false
hf_oauth: true
hf_oauth_scopes:
  - write-repos
short_description: Combine chat-format data from multiple Hugging Face datasets

Dataset Creator

Combine chat-format conversation data from multiple Hugging Face datasets into one shuffled, ready-to-train dataset - no notebook required.

What it does

• Add any number of HF datasets by repo id (dynamic list, no cap).
• Each one gets a schema check via the lightweight datasets-server API (falls back to a short streaming pull for gated or script-based datasets). Known ShareGPT/OpenAI-style conversation lists and a handful of common flat field pairs (prompt/chosen, query/answer, etc.) are detected automatically.
• Anything that doesn't match a known shape shows you the raw columns so you can map them by hand - no silent guesses, no half-right auto-mapping.
• Per-dataset sample limit and an optional system-prompt override.
• Build the combined, shuffled dataset, then either push it straight to your own HF account (private or public, via your own OAuth login) or download it as JSONL.

Auth

Sign in with the button at the top. Pushing to the Hub uses your account through HF's OAuth flow - there's no token to paste in, and nothing is stored server-side between sessions.

Project layout

Everything lives flat in the repo root (no subfolders, so it uploads fine through the Spaces web UI one file at a time):

• app.py - Gradio UI and event wiring.
• models.py - the DatasetEntry / FieldMapping dataclasses shared by everything else.
• schema_detect.py - pure schema-detection logic. No network calls in here, which makes it the easy part to unit test on its own.
• field_mapper.py - pure row-to-triplet extraction logic, also network-free.
• hf_inspect.py - peeks at a dataset's shape via the lightweight datasets-server API (with a streaming fallback) without downloading it.
• hf_dataset_loader.py - pulls the real rows, up to the per-dataset limit.
• hf_publish.py - pushes the combined dataset to the Hub, or writes it out as JSONL.

Known limitations

• Auto-detection is intentionally conservative. It only auto-applies a mapping when it recognizes both conversation-role tags (human/gpt, user/assistant, user/model) or an exact known flat-field-name pair. Anything else routes to manual mapping on purpose - a near-miss column name (e.g. question/chosen instead of prompt/chosen) won't get silently mis-mapped. The known-pairs list is a plain Python list at the top of schema_detect.py if you want to extend it for patterns you hit a lot.
• Datasets that bundle extra per-row enrichment beyond a simple triplet (for example, appending a vulnerability note to an answer, or synthesizing a prompt from a code snippet plus a language tag) won't be replicated by the generic mapper - it pulls the two fields you point it at, verbatim. If you need that kind of enrichment again, it's a small addition to field_mapper.py.
• Gated datasets need the signed-in user's token to actually have read access; there's no separate "request access" flow built in here.
• The gr.OAuthToken / gr.OAuthProfile injection is wired to the documented Gradio pattern (annotated trailing parameters, not listed in inputs=) but could only be validated structurally in this environment - there was no live Spaces OAuth session available to confirm end to end. Worth a quick sign-in-and-push smoke test right after deploying, before relying on it.