--- title: Curation emoji: 🧹 colorFrom: blue colorTo: indigo --- # Curation ### Selecting, filtering and refining high-quality data for better AI systems **Curation** is an independent Hugging Face organization focused on the methods, infrastructure and evaluation practices used to transform raw data into reliable, useful and well-governed datasets for **pretraining, post-training, fine-tuning, retrieval, evaluation, agents, multimodal AI and Physical AI**. The organization treats curation as a first-class AI systems discipline. > **Better models do not begin with more data. They begin with better data.** --- # What Is AI Data Curation? AI data curation is the process of **selecting, cleaning, filtering, deduplicating, labeling, balancing, validating and documenting data** so that it becomes more useful for machine learning. A simplified pipeline: ```text Raw Data β”‚ β–Ό Collection / Ingestion β”‚ β–Ό Normalization β”‚ β–Ό Filtering β”‚ β–Ό Deduplication β”‚ β–Ό Quality Scoring β”‚ β–Ό Decontamination β”‚ β–Ό PII / Safety / Policy Filtering β”‚ β–Ό Selection & Balancing β”‚ β–Ό Metadata & Provenance β”‚ β–Ό Curated Dataset β”‚ β”œβ”€β”€ Pretraining β”œβ”€β”€ Post-Training β”œβ”€β”€ Fine-Tuning β”œβ”€β”€ Evaluation └── Retrieval / Agents ``` Curation is not a single cleaning step. It is a **system of decisions about what data should remain, what should be removed, how examples should be weighted, how provenance should be tracked and whether a dataset is fit for a specific AI objective**. --- # Why Curation Matters Modern AI pipelines are often constrained less by raw data availability than by **data usefulness**. Large uncurated corpora can contain: - duplicates - near-duplicates - boilerplate - spam - malformed text - low-information content - benchmark contamination - personal information - unsafe content - licensing uncertainty - inconsistent formatting - language noise - machine-generated repetition - irrelevant domains - incorrect labels - skewed distributions - stale information Curation attempts to reduce these problems before they become model problems. --- # Curation as an AI Infrastructure Layer A useful way to view the AI stack: ```text Sources ↓ Raw Data ↓ CURATION ↓ Training / Post-Training Data ↓ Models ↓ Evaluation ↓ Deployment ``` Curation sits between **data acquisition** and **model optimization**. It affects: - model quality - reasoning performance - factuality - style - domain specialization - robustness - safety - memorization - privacy - evaluation reliability - training efficiency - cost --- # Core Curation Operations ## 1. Ingestion The first stage collects or imports data from one or more sources. Possible sources include: - web documents - code repositories - books - research papers - documentation - enterprise knowledge bases - support logs - conversations - sensor data - images - audio - video - synthetic data - agent trajectories Important ingestion questions: - Where did the data come from? - When was it collected? - Under what terms can it be used? - Has it already been transformed? - Can its provenance be preserved? --- ## 2. Normalization Raw data often arrives in inconsistent formats. Normalization may include: - encoding repair - whitespace cleanup - Unicode normalization - document segmentation - metadata normalization - timestamp normalization - field mapping - schema conversion - text extraction - media conversion The goal is not to remove information unnecessarily, but to create a stable representation for downstream processing. --- ## 3. Rule-Based Filtering Rule-based filters remove examples that clearly fail predefined requirements. Examples: - minimum or maximum length - excessive symbol ratios - broken encoding - repeated characters - malformed markup - missing fields - unsupported languages - unwanted file types - boilerplate patterns Rule-based filtering is often inexpensive and interpretable. --- ## 4. Language Identification Multilingual corpora require reliable language detection. Potential decisions include: - include only target languages - balance language proportions - separate code-switching - preserve low-resource languages - remove uncertain examples Language identification itself can introduce bias, especially for: - dialects - mixed-language text - transliterated text - low-resource languages --- # Quality Filtering Quality is not one universal score. A high-quality example for one task may be irrelevant for another. Possible quality signals include: - linguistic coherence - information density - domain relevance - factual structure - formatting quality - source credibility - educational value - reasoning depth - code validity - conversational usefulness A generic quality pipeline: ```text Document β”‚ β”œβ”€β”€ Heuristic Signals β”œβ”€β”€ Statistical Signals β”œβ”€β”€ Model-Based Score β”œβ”€β”€ Source Signals └── Task Relevance β”‚ β–Ό Quality Score β”‚ β–Ό Keep / Downweight / Remove ``` --- # Heuristic Quality Signals Useful heuristics may include: - average sentence length - punctuation ratios - duplicate-line frequency - stop-word ratios - alphabetic character ratios - link density - markup density - document length - paragraph structure Heuristics are useful because they are: - fast - explainable - cheap to compute - easy to audit But they should not be treated as universal measures of quality. --- # Model-Based Quality Scoring Learned quality models can estimate whether an example resembles a target distribution. Possible approaches include: - binary classifiers - reward models - embedding similarity - LLM-based scoring - pairwise preference models - domain relevance models Example: ```text Raw Example β”‚ β–Ό Quality Model β”‚ β”œβ”€β”€ 0.91 β†’ Keep β”œβ”€β”€ 0.62 β†’ Review / Downweight └── 0.18 β†’ Remove ``` Important questions: - What was the scorer trained on? - What biases does it encode? - Does it favor particular writing styles? - Does it penalize rare but valuable data? - Can the score be reproduced? --- # Deduplication Duplicate data can waste compute and amplify memorization. Curation may therefore remove: - exact duplicates - near-duplicates - repeated documents - repeated paragraphs - repeated code files - template-generated variants --- # Exact Deduplication Exact deduplication can use: - hashes - normalized hashes - canonicalized text Example: ```text Document ↓ Normalize ↓ Hash ↓ Duplicate? β”œβ”€β”€ Yes β†’ Remove / Reference └── No β†’ Keep ``` --- # Near-Duplicate Detection Near-duplicate detection may use: - MinHash - locality-sensitive hashing - n-gram similarity - embeddings - edit distance - semantic similarity Near-duplicate removal matters when documents differ only slightly. --- # Semantic Deduplication Two examples can be semantically redundant without sharing identical text. Embedding-based approaches can help identify: - paraphrases - mirrored content - repeated explanations - translated duplicates - highly similar synthetic examples Semantic deduplication is more expensive and must be tuned carefully to avoid removing legitimate diversity. --- # Benchmark Decontamination Training data should not silently contain evaluation answers. Decontamination attempts to identify overlap between training corpora and benchmarks. A simplified process: ```text Training Corpus β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β–Ό β–Ό Benchmark Items Similarity Search β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜ β–Ό Overlap Detection β”‚ β–Ό Remove / Flag ``` Possible overlap checks include: - exact string matches - n-gram overlap - fuzzy matching - code similarity - semantic similarity Evaluation contamination can create misleading performance estimates. --- # PII Detection and Privacy Filtering Curated datasets may need to identify or remove personally identifiable information. Examples include: - names - email addresses - phone numbers - account identifiers - addresses - private credentials - government identifiers - personal records Possible actions: - remove - mask - hash - tokenize - restrict - retain only with appropriate justification Privacy filtering is particularly important for: - enterprise datasets - conversational data - user-generated content - support logs - healthcare or financial workflows --- # Safety Filtering Safety filtering may target data involving: - explicit abuse - illegal content - extreme violence - harmful instructions - malware - harassment - exploitative content However, aggressive safety filtering can also remove valuable examples needed for: - moderation research - red teaming - safety evaluation - refusal training The correct decision depends on the intended use. --- # Licensing and Usage Rights A technically useful dataset may still be unsuitable if its usage rights are unclear. Curation should track: - source - license - redistribution rights - commercial-use restrictions - attribution requirements - derivative-work rules - jurisdictional constraints Metadata should remain connected to the data throughout the pipeline. --- # Provenance Provenance answers: > **Where did this example come from, and what happened to it before it reached the dataset?** Useful provenance metadata can include: ```text Source ↓ Collection Date ↓ Original Identifier ↓ License ↓ Transformation History ↓ Filtering Decisions ↓ Quality Score ↓ Dataset Version ``` Provenance supports: - reproducibility - auditing - governance - deletion requests - contamination analysis - licensing review --- # Data Lineage Data lineage extends provenance across multiple processing stages. Example: ```text raw_000184 ↓ normalize norm_000184 ↓ language-filter lang_000184 ↓ deduplicate dedup_000184 ↓ quality-score score_000184 ↓ final-selection train_000184 ``` A mature curation pipeline should be able to explain why an example was included. --- # Data Selection More data is not always better. Selection decides which examples are most useful for a target objective. Possible selection signals include: - quality - diversity - domain relevance - difficulty - novelty - uncertainty - model loss - gradient influence - embedding coverage - source quality - task utility --- # Diversity A highly filtered dataset can become too homogeneous. Useful diversity dimensions may include: - topic - language - geography - domain - writing style - source type - difficulty - reasoning pattern - code language - modality Curation therefore requires balancing **quality** and **coverage**. --- # Distribution Balancing Datasets often contain overrepresented sources or domains. Balancing can use: - caps - quotas - reweighting - stratified sampling - temperature sampling - source-level mixing Example: ```text Web 70% Code 15% Books 8% Research 4% Dialogue 3% ``` A curation system may intentionally adjust this distribution before training. --- # Mixture Design Model training often uses several datasets simultaneously. Mixture design determines: - which datasets are included - how much each contributes - how sampling changes over time - whether difficult examples are introduced later - whether domain data is oversampled Mixture design is therefore a form of curation. --- # Curriculum Curation Training data can be ordered or weighted by difficulty. Possible progression: ```text Simple Examples ↓ General Examples ↓ Difficult Examples ↓ Specialized Examples ``` Curriculum strategies can be useful when the order of exposure matters. --- # Data Curation for Pretraining Pretraining datasets may require: - large-scale deduplication - quality filtering - language balancing - source filtering - safety filtering - PII removal - contamination checks - mixture design - provenance tracking At pretraining scale, even inexpensive filters can have significant computational consequences. --- # Data Curation for Post-Training Post-training curation has different priorities. Relevant data may include: - instruction-response pairs - preference data - critiques - reasoning traces - tool-use examples - multi-turn conversations - agent trajectories - verification examples Key questions include: - Is the instruction clear? - Is the answer correct? - Is the preference reliable? - Is the reasoning useful? - Is the task representative? - Is the example too easy? - Is the label consistent? --- # Instruction Data Curation Instruction datasets can be filtered for: - task diversity - clarity - response correctness - difficulty - style - duplication - safety - domain coverage Example: ```text Raw Instruction Pair β”‚ β”œβ”€β”€ Instruction Quality β”œβ”€β”€ Answer Quality β”œβ”€β”€ Difficulty β”œβ”€β”€ Domain β”œβ”€β”€ Safety └── Duplication β”‚ β–Ό Curated SFT Data ``` --- # Preference Data Curation Preference optimization depends on reliable comparisons. Potential issues include: - noisy annotators - ambiguous pairs - trivial differences - style bias - length bias - inconsistent criteria - reward hacking Preference curation may include: - pair filtering - annotator agreement - judge calibration - difficulty balancing - confidence thresholds --- # Reasoning Data Curation Reasoning-oriented datasets can be curated by: - correctness - intermediate consistency - verification - diversity of solution paths - problem difficulty - answer uniqueness - self-correction quality For reasoning systems, **verified trajectories** can be more valuable than unverified chain-like text. --- # Agent Data Curation Agentic AI introduces new data types. Examples: - tool calls - API traces - browser actions - multi-agent messages - planning steps - environment states - task outcomes - retries - failures - recoveries Agent trajectory curation can evaluate: - task completion - tool correctness - efficiency - unnecessary actions - recovery behavior - safety - cost - latency --- # Tool-Use Data Tool-use examples may include: ```text User Goal ↓ Tool Selection ↓ Arguments ↓ Tool Result ↓ Reasoning / State Update ↓ Final Response ``` Curation should identify: - invalid tool calls - unnecessary tool calls - malformed arguments - incorrect interpretation of tool results - unsafe actions --- # Retrieval and RAG Curation Retrieval systems depend on curated source material. RAG curation may include: - document quality - chunk quality - metadata quality - freshness - duplication - source trust - access permissions - semantic coverage A retrieval pipeline can fail even when the generation model is strong if the knowledge base is poorly curated. --- # Evaluation Data Curation Evaluation data requires especially careful treatment. Good evaluation datasets should consider: - benchmark contamination - task relevance - difficulty - clear ground truth - judge reliability - distribution coverage - adversarial cases - temporal freshness Evaluation curation determines what β€œgood performance” actually means. --- # Synthetic Data Curation Synthetic data still needs curation. Generation alone does not guarantee quality. A synthetic pipeline may look like: ```text Seed Data ↓ Generator ↓ Synthetic Examples ↓ Verification ↓ Deduplication ↓ Difficulty / Diversity Scoring ↓ Curated Synthetic Dataset ``` Potential synthetic-data problems include: - mode collapse - stylistic repetition - factual errors - contamination - easy-example bias - self-reinforcing model artifacts --- # Human + Model Curation Modern curation often combines automation with human judgment. ```text Automated Filters ↓ Model Scoring ↓ Human Review ↓ Final Selection ``` Humans are particularly useful when: - context is ambiguous - quality criteria are subjective - domain expertise is required - high-risk data is involved - the model scorer is uncertain --- # Active Learning Active learning can prioritize examples where annotation is most valuable. Typical workflow: ```text Unlabeled Pool ↓ Current Model ↓ Uncertainty / Utility Score ↓ Select Examples ↓ Human Labeling ↓ Retrain ``` This connects curation directly to model improvement. --- # Data Difficulty Not every example contributes equally. Difficulty signals may include: - model loss - disagreement between models - human disagreement - reasoning depth - number of steps - rare concepts - adversarial structure Difficulty-aware curation can help create more informative training mixtures. --- # Domain Curation Enterprise and specialized AI systems often require domain-specific data. Examples: - legal - finance - medicine - engineering - manufacturing - cybersecurity - scientific research - customer support Domain curation requires: - terminology preservation - source validation - expert review - provenance - licensing - domain-specific quality criteria --- # Code Data Curation Code datasets introduce additional signals. Possible filters include: - parseability - compilation - test success - repository quality - license - generated files - vendored dependencies - duplicate code - comments - documentation quality Code curation may operate at: - file level - function level - repository level - commit level --- # Multimodal Curation Multimodal AI expands curation beyond text. Possible modalities: - image - audio - video - depth - LiDAR - radar - tactile signals - sensor streams Multimodal curation may require: - alignment checks - synchronization - caption quality - frame quality - corrupted-media detection - duplicate detection - temporal consistency - metadata validation --- # Robotics and Physical AI Data Curation Physical AI datasets may contain: - camera feeds - robot states - actions - trajectories - tactile data - force data - depth - LiDAR - audio - environment metadata A robotics trajectory may need to be curated by: - success - task relevance - safety - trajectory quality - control smoothness - state completeness - sensor synchronization --- # Curation Metrics There is no single universal curation metric. Useful measurements can include: ## Dataset-Level Metrics - number of examples - token count - modality distribution - language distribution - domain distribution - source distribution - duplicate rate - rejection rate - provenance coverage ## Quality Metrics - quality-score distribution - human acceptance rate - classifier confidence - annotation agreement - verification success ## Diversity Metrics - embedding coverage - topic entropy - source diversity - language diversity - cluster balance ## Model-Based Metrics The strongest validation is often downstream: > **Does the curated dataset produce a better model?** --- # Ablation Testing Curation decisions should ideally be tested. Example: ```text Dataset A No Quality Filter ↓ Model A Dataset B Quality Filter ↓ Model B Compare: - loss - benchmark scores - robustness - memorization - safety ``` Ablation testing helps distinguish useful filters from filters that merely sound reasonable. --- # Data Value An important research direction is estimating the value of individual examples or subsets. Possible approaches include: - influence functions - gradient similarity - loss changes - Shapley-inspired methods - retraining experiments - proxy models The goal is to answer: > **Which data actually improves the target system?** --- # Curation at Scale Large datasets require distributed infrastructure. A production curation pipeline may include: ```text Object Storage β”‚ β–Ό Distributed Processing β”‚ β”œβ”€β”€ Parsing β”œβ”€β”€ Filtering β”œβ”€β”€ Scoring β”œβ”€β”€ Deduplication β”œβ”€β”€ PII Detection └── Metadata Enrichment β”‚ β–Ό Versioned Dataset ``` Operational concerns include: - throughput - memory - storage - sharding - checkpointing - reproducibility - failure recovery - cost --- # Incremental Curation Datasets do not need to be rebuilt from scratch every time. Incremental pipelines can process: - new documents - updated sources - revised labels - deletion requests - newly detected contamination This enables continuously maintained datasets. --- # Dataset Versioning A curated dataset should evolve transparently. Useful version metadata includes: - version number - date - added sources - removed sources - filter changes - scoring changes - schema changes - known issues Versioning is essential for reproducible experiments. --- # Reproducibility A reproducible curation pipeline should document: - source list - processing code - filter thresholds - model versions - random seeds - deduplication parameters - selection strategy - dataset version Without this information, it can be difficult to understand why results changed. --- # Governance Enterprise curation requires governance. Relevant concerns include: - ownership - access control - retention - deletion - jurisdiction - licensing - privacy - auditability - provenance Governance should not be added only after the dataset is created. It should be part of the curation architecture. --- # Curation Failure Modes Common failure modes include: ## Over-Filtering Removing too much data can reduce: - diversity - rare knowledge - dialect coverage - creativity - difficult examples ## Under-Filtering Keeping too much can increase: - noise - repetition - memorization - safety problems - compute waste ## Proxy Bias A quality classifier may reward examples that resemble its own training distribution. ## Source Concentration A dataset may appear large while being dominated by a small number of sources. ## Benchmark Leakage Evaluation data may accidentally appear in training. ## Synthetic Collapse Repeated model-generated data may narrow distributional diversity. --- # Curation vs. Cleaning These concepts overlap but are not identical. **Cleaning** usually focuses on fixing or removing bad data. **Curation** is broader. It includes: - cleaning - selection - prioritization - balancing - scoring - governance - provenance - versioning - alignment with a target objective --- # Curation vs. Annotation Annotation adds labels or structured information. Curation decides: - which examples deserve annotation - which labels are trustworthy - which examples should be retained - how labeled data should be balanced Annotation is therefore one component inside a larger curation system. --- # Curation vs. Data Engineering Data engineering focuses on reliable data movement and transformation. Curation focuses on **fitness for AI use**. The two disciplines overlap strongly. A complete AI data platform may combine: ```text Data Engineering + Data Curation + Data Governance + ML Evaluation ``` --- # Curation vs. Synthetic Data Synthetic data generates new examples. Curation determines which synthetic examples are useful. These layers are complementary: ```text Synthetic Data ↓ Curation ↓ Post-Training ↓ Validation ``` --- # Curation and Post-Training Post-training quality depends heavily on data selection. Relevant areas include: - SFT - preference optimization - reward modeling - reasoning data - tool use - agent trajectories - safety data - domain specialization Curation is therefore a natural bridge between raw or generated data and post-training. --- # Curation and Evaluation Evaluation and curation should inform each other. ```text Curated Dataset ↓ Train Model ↓ Evaluate ↓ Identify Weakness ↓ Curate Better Data ↓ Retrain ``` This creates a **data-centric feedback loop**. --- # Curation and Data-Centric AI Model-centric development asks: > How can we improve the model? Data-centric development also asks: > How can we improve the data? Curation is one of the core operational disciplines behind data-centric AI. --- # A Reference Curation Architecture ```text RAW DATA β”‚ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ INGESTION β”‚ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ NORMALIZATIONβ”‚ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ RULE / FORMAT FILTERING β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ DEDUPLICATION β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ QUALITY SCORING β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ PRIVACY / SAFETY / COMPLIANCE β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ DECONTAMINATION β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ DATA SELECTION β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ MIXTURE / BALANCE β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ PROVENANCE / META β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β–Ό CURATED DATA β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β–Ό β–Ό β–Ό PRETRAINING POST-TRAINING EVALUATION ``` --- # Practical Curation Questions Before using a dataset, ask: ### Source - Where did the data come from? - Can the source be traced? ### Quality - What does β€œhigh quality” mean for this task? - Is quality measured or assumed? ### Duplication - Are exact and near duplicates present? ### Contamination - Does the dataset overlap with evaluation data? ### Privacy - Does the dataset contain personal information? ### Rights - Is the usage license clear? ### Diversity - Are important domains or languages missing? ### Bias - Do the filtering rules systematically remove some data? ### Freshness - Is the dataset still representative? ### Utility - Does training on the data actually improve the target system? --- # Areas We Track ## Data Filtering Rule-based, statistical and learned filtering. ## Quality Scoring Models and metrics for estimating data usefulness. ## Deduplication Exact, near-duplicate and semantic deduplication. ## Decontamination Detection of training/evaluation overlap. ## Privacy PII detection, redaction and sensitive-data handling. ## Provenance Origin, lineage and transformation history. ## Data Selection Choosing examples based on utility, relevance and diversity. ## Mixture Design Balancing datasets for training and post-training. ## Synthetic Data Curation Filtering and verifying generated examples. ## Post-Training Data Instruction, preference, reasoning and agent data. ## Evaluation Curation Building reliable benchmarks and test sets. ## Multimodal Curation Images, video, audio, sensor data and robotics trajectories. --- # Planned Resources The organization is intended to become a practical reference layer for AI data curation. ## Curation Explorer A structured explorer covering: - curation methods - filtering strategies - deduplication - quality scoring - decontamination - privacy - data selection - use cases ## Data Quality An interactive dataset-readiness and quality assessment Space. ## Dataset Curation A practical reference for transforming raw datasets into AI-ready training or evaluation data. ## Curation Collection A curated Hugging Face collection of: - datasets - models - papers - tools - Spaces - research - data-quality resources --- # SEO & GEO Topic Map This organization is intentionally structured around explicit concepts relevant to traditional search engines and generative retrieval systems: - AI data curation - data curation - dataset curation - training data curation - post-training data - data quality - data filtering - deduplication - semantic deduplication - decontamination - benchmark contamination - quality scoring - dataset selection - data mixture - data provenance - data lineage - synthetic data curation - preference data - instruction data - reasoning data - agent data - multimodal datasets - data-centric AI - AI-ready data - dataset governance - enterprise AI data - LLM training data --- # Who Is Curation For? **Curation** is relevant for: - foundation model teams - LLM developers - dataset creators - data engineers - ML engineers - researchers - post-training teams - evaluation teams - enterprise AI teams - data-platform companies - annotation providers - synthetic-data companies - data-governance teams - multimodal AI teams - robotics teams - open-source communities --- # Principles ## Quality Is Task-Dependent There is no universal definition of good data. Curation criteria should be linked to an explicit objective. ## Provenance Matters Useful data should remain traceable wherever possible. ## Filtering Should Be Measurable Filters should be evaluated by their downstream effects, not only by intuition. ## Diversity Has Value Aggressive filtering should not destroy important distributional coverage. ## Evaluation Must Be Protected Benchmark contamination should be treated as a curation problem. ## Human Judgment Still Matters Automation can scale curation, but humans remain important for ambiguous and high-stakes decisions. ## Transparency Improves Reproducibility Dataset versions, filtering rules and known limitations should be documented. --- # Collaboration & Partnerships **Curation is open to collaboration with companies, research teams, universities, open-source projects and organizations working on AI data quality and data-centric machine learning.** Relevant collaboration areas include: - training data - dataset curation - data quality - deduplication - filtering - decontamination - provenance - privacy - synthetic data - annotation - post-training data - preference data - evaluation data - agent trajectories - multimodal data - enterprise AI data - data governance - data infrastructure Possible collaboration formats include: - technical showcases - dataset curation case studies - joint Hugging Face Spaces - open-source integrations - benchmark projects - dataset quality analysis - ecosystem maps - research collaborations - methodology comparisons - clearly disclosed partnerships and sponsorships ## Collaboration Contact For technical collaborations, research partnerships, ecosystem projects or sponsorship inquiries: **agenten@magenta.de** --- # Independent Organization **Curation is an independent Hugging Face organization.** It is not an official organization of Hugging Face or of any dataset provider, annotation company, AI laboratory, model provider or technology company that may be referenced in future resources. Product names and trademarks belong to their respective owners. --- # Long-Term Vision AI development is increasingly becoming a data-quality problem. As models become stronger, the difference between useful and useless data becomes more important. The long-term objective of **Curation** is to become an open technical reference for understanding how raw information becomes **training-ready, evaluation-ready and AI-ready data**. > **Collecting data creates volume. Curation creates value.** --- ## Filter. Select. Verify. Refine. ### Curation **Building better AI through better data.** For collaborations, research and partnerships: **agenten@magenta.de**