README.md

# Phase 1 — Data Quality Foundation

## Compliments Reference DB Pipeline

**Version:** 2.0.0
**Date:** 2026-07-30
**Status:** PASS

---

## Purpose

Phase 1 is the **data quality foundation** of the pipeline. It loads the authoritative Compliments product dataset, performs comprehensive data quality checks, cleans string fields, validates key fields, and produces a clean output with full traceability (raw + cleaned columns).

---

## Authoritative Input

| Property | Value |
|----------|-------|
| Source | `saraNour/compliments-brand/source_of_truth/products.parquet` |
| URL | https://huggingface.co/datasets/saraNour/compliments-brand/blob/main/source_of_truth/products.parquet |
| Type | HuggingFace dataset (private) |
| Original Source | Voila.ca (Loblaw) Compliments private-label products |
| Row Count | 4,440 |
| Column Count | 16 |

---

## What Phase 1 Does

### 1. Schema Validation
- Validates column names, types, and row count
- Checks for unexpected changes from baseline

### 2. Raw Data Audit
- Null audit for every column
- Duplicate audit: full rows, external_id, upc, title+size+brand
- Price anomaly detection (negative, zero, >$100)
- UPC format validation (12-14 digits)
- 100% null column detection

### 3. Data Cleaning (NEW in v2.0.0)
- **String normalization**: Trim whitespace, collapse multiple spaces, empty strings → null
- **Brand cleaning**: Normalizes casing and whitespace (e.g., `COMPLIMENTS` → `Compliments`)
- **Title cleaning**: Normalizes casing and whitespace
- **UPC validation**: Format check, null audit, reused-UPC analysis
- **External ID validation**: Uniqueness check, null audit

### 4. Output Generation
- Produces cleaned output with **raw + cleaned columns** for traceability
- Documents all dropped columns with reasons
- Generates validation report, statistics, and provenance

---

## Phase 1 Does NOT

- Normalize brands semantically (Phase 2)
- Extract identity attributes (Phase 2)
- Classify food/non-food (Phase 3)
- Perform any grouping (Phase 3)
- Use any external data sources beyond the authoritative input

---

## Output Schema

The Phase 1 output (`phase1_output.parquet`) contains 20 columns:

| # | Column | Type | Description |
|---|--------|------|-------------|
| 1 | upc | str | Universal Product Code (1 null, 3,271 unique) |
| 2 | external_id | str | Voila retailer product ID (4,440 unique) |
| 3 | brand | str | Original brand string (11 variants) |
| 4 | title | str | Original product title |
| 5 | price | float64 | Price in CAD |
| 6 | price_currency | str | Always "CAD" |
| 7 | size | str | Raw size string (631 unique) |
| 8 | size_amount | float64 | Parsed numeric amount (307 nulls) |
| 9 | size_unit | str | Parsed unit: g, kg, ml, l, count (307 nulls) |
| 10 | size_qty | int64 | Quantity multiplier (usually 1) |
| 11 | size_unit_norm | str | Normalized unit (307 nulls) |
| 12 | image_url | str | Product image URL |
| 13 | source | str | Always "voila" |
| 14 | source_url | str | Product page URL |
| 15 | **brand_clean** | str | Cleaned brand (whitespace/case normalized) |
| 16 | **brand_raw** | str | Original brand (for traceability) |
| 17 | **title_clean** | str | Cleaned title (whitespace normalized) |
| 18 | **title_raw** | str | Original title (for traceability) |
| 19 | **upc_raw** | str | Original UPC (for traceability) |
| 20 | **external_id_raw** | str | Original external_id (for traceability) |

### Dropped Columns

| Column | Reason |
|--------|--------|
| size_per_unit | 100% null (4,440/4,440 values are NaN) |
| size_total | 100% null (4,440/4,440 values are NaN) |

---

## Validation Results

| Check | Result |
|-------|--------|
| Row count = 4,440 | PASS |
| Column count = 16 (input) → 20 (output) | PASS |
| Column names match expected | PASS |
| Full row duplicates = 0 | PASS |
| external_id duplicates = 0 | PASS |
| Schema validation | PASS |
| **Overall** | **PASS** |

### Key Metrics

| Metric | Value |
|--------|-------|
| Total products | 4,440 |
| Unique external_ids | 4,440 |
| Unique UPCs | 3,271 |
| Reused UPCs (same UPC, different products) | 1,168 |
| Null UPCs | 1 |
| Title+size+brand duplicates | 60 |
| 100% null columns dropped | 2 (size_per_unit, size_total) |

---

## Brand Cleaning Summary

| Raw Brand | Cleaned Brand | Count |
|-----------|---------------|-------|
| COMPLIMENTS | Compliments | 1 |
| Compliments | Compliments | 4,258 |
| Compliments Organic | Compliments Organic | 90 |
| Compliments Balance | Compliments Balance | 42 |
| Compliments Naturally Simple | Compliments Naturally Simple | 25 |
| Sensations | Sensations | 8 |
| Compliments Little Ones | Compliments Little Ones | 7 |
| Compliments Green Care | Compliments Green Care | 5 |
| Compliments Green | Compliments Green | 4 |

---

## Files

```
phase1/
    README.md                          # This file
    src/phase1.py                      # Production Python script (v2.0.0)
    notebooks/phase1.ipynb             # Interactive notebook (unexecuted)
    outputs/
        phase1_output.parquet          # Clean output (4,440 rows, 20 cols)
        phase1_provenance.json         # Column provenance documentation
        phase1_executed.ipynb          # Executed notebook with all outputs
    validation/
        phase1_validation.json         # Validation report (PASS)
    statistics/
        phase1_statistics.json         # Data statistics
```

---

## How to Run

```bash
# Using the Python script
cd /home/sara/gsoc/compliments-reference-db
python -m phase1.src.phase1

# Using the notebook
jupyter notebook phase1/notebooks/phase1.ipynb
```

---

## Handoff to Phase 2

Phase 2 must read:
- `phase1/outputs/phase1_output.parquet`

Phase 2 must use:
- `brand_clean` for semantic normalization (not `brand`)
- `title` (original) for identity extraction (BEFORE normalization)

Phase 2 must NOT:
- Re-download from HuggingFace
- Read from any other source
- Skip Phase 1 validation
- Duplicate cleaning already done in Phase 1