README.md

# Phase 3 — Product Domain + Taxonomy + Product Grouping

## Compliments Reference DB Pipeline

**Version:** 1.1.0
**Date:** 2026-07-30
**Status:** PASS

---

## Input

| Property | Value |
|----------|-------|
| Source | `phase2/outputs/phase2_output.parquet` |
| HF Repo | `saraNour/compliments-reference-phase2` |
| Row Count | 4,440 |
| Column Count | 38 |

---

## What Phase 3 Does

### A. Food / Non-Food Classification

Deterministic keyword-based rules. Creates `product_domain` column with values: `food`, `non_food`, `unknown`.

- **Food:** Dairy, meat, seafood, bakery, beverages, frozen, produce, pantry, snacks, confectionery
- **Non-food:** Pet food, medications, personal care, household cleaning, kitchenware, health remedies
- **Unknown:** Products that don't match any rule

### B. Taxonomy Classification

Deterministic keyword-based taxonomy. Classification metadata only — NOT used as grouping key.

19 categories: DAIRY, MEAT_SEAFOOD, BEVERAGES, BAKERY, FROZEN, BREAKFAST, PASTA_RICE, CANNED_GOODS, CONDIMENTS_SAUCES, SNACKS, CONFECTIONERY, PRODUCE, PERSONAL_CARE, HOUSEHOLD_CLEANING, HEALTH_REMEDIES, PET_FOOD, HOUSEHOLD_SUPPLIES, BABY_CARE, GENERAL_GROCERY

### C. Product Grouping

Identity-based deterministic grouping:

```
group_key = identity_hash + product_domain + normalized_core_title
```

- `identity_hash`: 14-field hash from Phase 2 (9 flags + fat + product_line + flavour + formulation)
- `product_domain`: food/non_food/unknown — **structurally prevents food/non-food mixing**
- `normalized_core_title`: Core title with generic packaging words removed, tokens sorted

### D. Ambiguous Case Detection

Groups with >1 product are inspected for:
- Multiple core_titles
- Multiple flavours
- Multiple formulations
- Multiple fat levels
- Multiple product lines

### E. Rule-Based Resolution

All grouping is deterministic. No LLM was required.

---

## Output Files

### A. reference_product_catalog.csv (Group-Level)

| Column | Type | Description |
|--------|------|-------------|
| group_id | str | Deterministic UUID |
| group_name | str | Most frequent core_title |
| brand | str | Normalized brand |
| product_domain | str | food/non_food/unknown |
| taxonomy | str | Taxonomy category |
| identity_hash | str | 14-field identity hash |
| product_count | int | Products in group |
| unique_upcs | int | Unique UPCs in group |

### B. product_group_mapping.csv (Product-Level)

| Column | Type | Description |
|--------|------|-------------|
| upc | str | Barcode |
| external_id | str | Voila ID |
| group_id | str | Group UUID |
| group_name | str | Core title |
| core_title | str | Core title |
| original_title | str | Raw title |
| brand | str | Normalized brand |
| size | str | Raw size |
| variant_attributes | str | Parsed size JSON |
| identity_hash | str | Identity hash |
| product_domain | str | Food/non-food |
| taxonomy | str | Taxonomy |
| product_line | str | Product line |
| source | str | Data source |
| source_url | str | Source URL |

### C. ambiguous_cases.csv

Groups with potential ambiguity and the reason.

### D. unknown_products.csv

Products classified as "unknown" domain (1,243 products).

---

## Deterministic Grouping Rules

| # | Rule | Implementation |
|---|------|----------------|
| 1 | Identity hash captures 14 fields | `build_identity_hash()` in Phase 2 |
| 2 | Product domain is part of grouping key | `build_group_key()` in Phase 3 |
| 3 | Core title normalized (generic words removed, sorted) | `normalize_title_for_grouping()` |
| 4 | Same normalized title + same identity = same group | Group key construction |
| 5 | Different identity = different group | Identity hash is first component |
| 6 | Food vs non_food = always different group | Product domain is second component |

---

## Food/Non-Food Rules

100+ deterministic regex rules based on product title keywords.

- Pet food → non_food
- Medications (ibuprofen, acetaminophen, etc.) → non_food
- Personal care (tampons, shampoo, etc.) → non_food
- Household (detergent, garbage bags, etc.) → non_food
- Kitchenware (light bulbs, pans, etc.) → non_food
- Everything else matching food keywords → food
- No match → unknown

---

## Regression Tests — ALL PASS (12/12)

| Test | Result |
|------|--------|
| PB smooth sizes same group | PASS |
| PB crunchy sizes same group | PASS |
| PB organic smooth different | PASS |
| PB organic crunchy different | PASS |
| PB naturally simple different | PASS |
| PB light different | PASS |
| PB honey different | PASS |
| PB cookies different | PASS |
| PB ice cream different | PASS |
| PB dog treats different | PASS |
| CC 1% vs 2% different | PASS |
| Cottage cheese sizes same | PASS |

---

## Validation Results

| Check | Result |
|-------|--------|
| Row count preserved | PASS |
| No duplicate mappings | PASS |
| No food/non-food mixing | PASS |
| No brand conflicts | PASS |
| Identity consistent | PASS |
| Taxonomy populated | PASS |
| No empty group names | PASS |
| One group per product | PASS |
| No orphans | PASS |
| **Overall** | **PASS** |

---

## Files

```
phase3/
    README.md
    src/phase3.py
    notebooks/phase3.ipynb
    outputs/
        reference_product_catalog.csv
        product_group_mapping.csv
        ambiguous_cases.csv
        unknown_products.csv
        phase3_executed.ipynb
    validation/
        phase3_validation.json
    statistics/
        phase3_statistics.json
    audit/
        phase3_1_unknown_audit.csv
        phase3_1_unknown_audit.parquet
        phase3_1_audit_summary.parquet
        peanut_butter_grouping_example.parquet
    notebooks/
        phase3_1_audit.ipynb
```

---

## How to Run

```bash
cd /home/sara/gsoc/compliments-reference-db
python -m phase3.src.phase3
```

---

## LLM Usage

None. All grouping is deterministic.

---

## Handoff

Phase 3 outputs are the frozen product-grouping layer.

Future phases will add:
- Phase 4: Variant assignment
- Phase 5: Nutrition merge
- Phase 6: Nutrition scoring (NutriScore, NOVA, EcoScore)
- GPC integration
- Agribalyse integration