dagster-hf-datasets Examples

A curated collection of 17 end-to-end examples demonstrating how to build reproducible, observable, and production-ready ML data pipelines with dagster-hf-datasets

Each example is self-contained, runnable with dagster dev -m <example_folder>.definitions.py, and covers a distinct concept or workflow tier — from basic Hub ingestion to flagship end-to-end pipelines.

---

Prerequisites

pip install dagster dagster-hf-datasets

Some examples require additional dependencies (noted in their README). Set your Hugging Face token for gated datasets:

export HF_TOKEN=hf_...

All examples share the same Definitions boilerplate. The HuggingFaceResource handles Hub loading; HFParquetIOManager handles persistence:

from dagster import Definitions
from dagster_hf_datasets import HuggingFaceResource, HFParquetIOManager

defs = Definitions(
    assets=[...],
    resources={
        "huggingface": HuggingFaceResource(cache_dir=".hf_cache"),
        "hf_parquet_io_manager": HFParquetIOManager(base_dir=".dagster_hf_storage"),
    },
)

---

Running an Example

cd dagster_hf_datasets_examples

dagster dev -m  <example_folder>.definitions.py

Every definitions.py is a standalone entrypoint.

---

Repository Structure

dagster-hf-datasets-examples/
├── README.md                      ← you are here
├── pyproject.toml
│
├── basic_hub_ingestion/
│   ├── assets.py                  ← @hf_dataset_asset / @hf_multi_asset definitions
│   ├── definitions.py             ← Definitions() entrypoint for dagster dev
│   ├── README.md                  ← goal, dataset, API notes, expected output
│   └── __init__.py
│
└── <example_name>/                ← same layout for all 17 examples
    ├── assets.py
    ├── definitions.py
    ├── README.md
    └── __init__.py

---

Example Catalog

🏗️ Foundation

Core patterns every dagster-hf-datasets user encounters first.

---

🔄 Data Management & Automation

Patterns for scalability, automation, and memory-safe processing.

---

🖼️ Multimodal Workflows

Image, audio, and vision-language pipelines.

---

🤖 LLM & Alignment Workflows

Text, instruction, and preference-alignment pipelines.

---

Learning Path

Beginner — start here if you're new to dagster-hf-datasets

1. basic_hub_ingestion — understand @hf_dataset_asset and MaterializeResult
2. multi_asset_split_routing — see how @hf_multi_asset auto-resolves splits
3. sanitization_observability — chain @hf_dataset_asset with downstream @asset nodes
4. dataset_card_publishing — publish lineage-aware Hub documentation

Intermediate — build operational pipelines

5. event_driven_benchmark_refresh — add sensor-based automation
6. streaming_oom_datasets — handle datasets that exceed memory
7. dynamic_bucket_partitioning — partition by language/domain/task
8. distributed_token_sharding — generate training-ready tokenized shards

Advanced — multimodal and specialized workflows

9. multi_modal_data_profiling
10. image_dataset_curation
11. audio_dataset_curation
12. vision_dataset
13. synthethic_multimodal_data
14. vision_language_scale_pipeline
15. code_instruction_pipeline
16. preference_alignment_data
17. golden_pipeline — run this last to see everything together

---

Common Gotchas

The decorated function body is not responsible for loading the dataset. HuggingFaceResource performs the load; the function receives the injected dataset parameter. Your function body inspects, transforms, and returns MaterializeResult.

HFParquetIOManager does not persist IterableDataset. Streaming datasets must be written to Parquet manually via pyarrow. See streaming_oom_datasets for the pattern.

HFDatasetPublisher is in a private module. Import from dagster_hf_datasets._export._publisher. Pin your dagster-hf-datasets version if you use this.

Splits from @hf_multi_asset are named {asset_name}_{split}. A glue_sst2 multi-asset produces glue_sst2_train, glue_sst2_validation, glue_sst2_test as downstream dependency names.

Split before tokenizing. Always train-test split before tokenization to avoid data leakage. The golden_pipeline demonstrates the correct order.

---

Resources

• dagster-hf-datasets on GitHub
• API Reference
• Usage Guide
• Dagster Docs: HF Datasets Integration
• Hugging Face Datasets Docs