the-hf-stack/dagster-hf-datasets-examples/vision_language_scale_pipeline - Storage Bucket

Vision-Language Scale Pipeline

This example demonstrates large-scale multimodal data processing with deduplication and quality filtering using LAION-COCO, a 600M image-text pair dataset designed for vision-language model training.

Dataset: LAION-COCO

LAION-COCO is a curated subset of LAION (Large-scale Artificial Intelligence Open Network), optimized for vision-language model instruction-tuning:

• Size: ~600M high-quality image-text pairs
• Curation: Filtered from LAION for better caption-image alignment
• Quality: Balanced between scale and quality (unlike raw LAION which can be noisy)
• Use Cases: Training CLIP alternatives, vision-language instruction-tuned models (LLaVA, Qwen-VL, Flamingo alternatives)

Pipeline Architecture

Multi-stage filtering and deduplication workflow:

raw_laion_coco (5K sample; ~600M at scale)
    ↓
caption_quality_filtered (Length: 3-500 words; no boilerplate)
    ↓
deduplicated_by_caption_hash (SHA256 dedup; remove exact duplicates)
    ↓
language_identified_captions (Detect language; filter to English)
    ↓
dedup_quality_report (Compute retention %, statistics, quality score)

Key Assets

1. raw_laion_coco → MaterializeResult

Ingests LAION-COCO (600M pairs, sampled to 5K for dev):

• Analyzes caption characteristics (length distribution, sample captions)
• Metadata: row count, total dataset size, sample captions

Output:

{
  "rows": 5000,
  "total_dataset_size": 600000000,
  "avg_caption_tokens": 18.5,
  "sample_captions": ["A dog running in a park", "...", "..."],
  "modality": "image-text"
}

2. caption_quality_filtered → MaterializeResult

Filtering rules:

• Minimum: 3 words (skip single-word or 2-word captions)
• Maximum: 500 words (skip rambling/off-topic captions)
• No boilerplate: Exclude "website", "banner", "advertisement", "click here", etc.

Retention Typical: 95-98% (most captions are already reasonably good quality)

Metrics Logged:

• Input rows, output rows, retention %

Pattern Reused: Similar to sanitization_observability — multi-rule quality filtering at scale

3. deduplicated_by_caption_hash → MaterializeResult

Removes exact duplicate captions using SHA256 hashing:

• Why: At 600M scale, even 1% duplicates = 6M redundant pairs
• How: Hash each caption text; track seen hashes; skip duplicates
• Efficiency: O(N) time, O(unique captions) memory
• Output: Deduplicated dataset + duplicate statistics

Metrics Logged:

• Duplicates removed (count and %)
• Retention % post-dedup

Use Case Example: Multiple images might have identical caption ("a dog"; "golden retriever"; etc.) — dedup catches these.

4. language_identified_captions → MaterializeResult

Identifies caption language and filters to English:

• Detection: Simple ASCII-ratio heuristic (production: use langdetect or textblob)
• Rule: > 90% ASCII = English, 70-90% = Mixed, < 70% = Non-English
• Customization: Can modify to keep multilingual or detect specific languages

Retention Typical: 70-85% (LAION-COCO is diverse; includes non-English captions)

Metrics Logged:

• English pairs, non-English pairs, retention %

5. dedup_quality_report → dict (report)

Comprehensive pipeline metrics across all stages:

{
  "pipeline_stage": "complete",
  "raw_pairs": 5000,
  "after_quality_filter": 4950,
  "after_deduplication": 4870,
  "after_language_filter": 3920,
  "total_retention_pct": 78.4,
  "quality_filter_retention_pct": 99.0,
  "deduplication_retention_pct": 98.4,
  "language_filter_retention_pct": 80.4,
  "caption_length_stats": {
    "mean_tokens": 17.8,
    "median_tokens": 16.0,
    "min_tokens": 3,
    "max_tokens": 500
  },
  "data_quality_score": 62.7
}

Quality Score Formula: retention_pct × 0.8

• Lower scores reflect filtering intensity
• Scores 60-80% typical for production pipelines

Patterns Demonstrated

1. Multi-Stage Filtering at Scale

• Quality filter → Deduplication → Language filtering
• Each stage reduces dataset size but improves quality
• Pattern reusable for other large-scale datasets (text, audio, etc.)

2. Hash-Based Deduplication

• Memory-efficient O(unique captions) memory usage
• Detects exact duplicates only (not fuzzy/near-duplicates)
• Scales to billions of records

3. Language Detection

• Simple heuristics (ASCII ratio) for speed
• Can integrate pretrained language models for accuracy
• Demonstrates trade-off between speed and precision

4. Metadata Tracking Through Pipeline

• Logs retention % at each stage
• Enables debugging (where did data get lost?)
• Pattern from sanitization_observability applied to multimodal

Running Locally

cd dagster_hf_datasets_examples

dagster dev -m vision_language_scale_pipeline.definitions

Materialize order:

1. raw_laion_coco (sample and analyze)
2. caption_quality_filtered (filter by length/content)
3. deduplicated_by_caption_hash (remove exact duplicates)
4. language_identified_captions (language filtering)
5. dedup_quality_report (final metrics)

Note: First run downloads LAION-COCO sample (~100-200MB). Uses cache thereafter.

Customization

Adjust Sample Size

# In raw_laion_coco()
sample_size = min(50000, len(dataset))  # Larger sample for better statistics

Keep Multilingual Data

# In language_identified_captions(), remove filter:
# Just return dataset as-is or add language tag

Implement Fuzzy Deduplication

from difflib import SequenceMatcher

def is_duplicate(caption1, caption2, threshold=0.95):
    ratio = SequenceMatcher(None, caption1, caption2).ratio()
    return ratio > threshold

# In deduplicated_by_caption_hash():
# Compare against previously seen captions using fuzzy matching

Add Image Validation

@asset
def image_url_validated(language_identified_captions: Dataset) -> Dataset:
    """Validate image URLs are reachable (optional, slow)."""
    import requests
    
    def url_exists(example):
        url = example.get("url")
        try:
            response = requests.head(url, timeout=2)
            return response.status_code == 200
        except:
            return False
    
    return language_identified_captions.filter(url_exists)

Add Caption Length Constraints

# Existing: 3-500 word range
# Add: Min/max character constraints

def is_quality_caption(example):
    caption = example.get("caption", "").strip()
    words = caption.split()
    
    # Existing checks...
    if len(words) < 3 or len(words) > 500:
        return False
    
    # New: character-level constraints
    if len(caption) < 20:  # Too short in chars
        return False
    if len(caption) > 5000:  # Too long in chars
        return False
    
    return True

Use Cases

Training Vision-Language Models

• Use deduplicated, high-quality LAION-COCO to train CLIP alternatives
• Instruction-tune LLaVA or similar VLMs

Building Search Engines

• Use image-caption pairs to train dual-encoder retrieval models
• Index images for text-to-image search

Data Quality Auditing

• Monitor pipeline metrics to detect dataset drift
• Compare quality reports across time periods

Mixed Dataset Training

Combine with other VL datasets:

@asset
def multi_dataset_blend(
    laion_coco_clean: Dataset,
    other_vl_dataset: Dataset,
) -> Dataset:
    """Blend LAION-COCO with custom or proprietary VL data."""
    from datasets import concatenate_datasets
    return concatenate_datasets([laion_coco_clean, other_vl_dataset])

Performance Notes

Stage	Time (10K pairs)	Memory
Load raw	~5 sec	200MB
Quality filter	~2 sec	50MB
Deduplication	~15 sec	100MB (hashes)
Language filter	~3 sec	50MB
Reporting	~5 sec	100MB
Total	~30 sec	~500MB

At 600M scale: ~8 hours (parallelizable per language or partition)

See Also

• LAION-COCO on Hub
• LAION Project
• Related Examples:
  • multi_modal_data_profiling/ — Image-specific statistics
  • sanitization_observability/ — Advanced quality metrics
  • code_instruction_pipeline/ — Language-specific filtering (for code)

Tips

• For CLIP training: Use full pipeline (quality + dedup + language filter)
• For VQA: Keep multilingual data or filter to diverse languages
• For Evaluation: Use final deduplicated set as quality benchmark
• For Production: Add image URL validation + OCR-based quality scoring