README.md

# Phase 4 — Variant Assignment / Variant Modeling

## Compliments Reference DB Pipeline

**Version:** 1.0.0
**Date:** 2026-07-30
**Status:** PASS

---

## Purpose

Phase 4 transforms the Phase 3 product-group representation into a structured product variant model. Products belonging to the same reference product group can have different purchasable variants such as size, package quantity, formulation, flavour, fat level, etc.

---

## Key Concepts

### Reference Product Group (from Phase 3)

A reference product group represents the same underlying product identity. All products in a group share the same identity attributes (organic, gluten-free, fat level, flavour, formulation, product_line).

### Variant

A **variant** is a purchasable representation of the same reference product that differs in attributes such as:

- **Size**: 500 g, 1 kg, 2 kg
- **Package quantity**: 1 × 500 g, 2 × 500 g
- **Pack configuration**: single, multipack, case

### Identifier Hierarchy

| Identifier | Level | Description |
|------------|-------|-------------|
| `external_id` | Source product | Voila retailer product ID (unique per product) |
| `upc` | Source product | Barcode (may be reused across variants) |
| `group_id` | Reference group | Deterministic UUID5 from Phase 3 |
| `variant_id` | Variant | Deterministic UUID5 from group_id + variant_key |

---

## Input Datasets

| Dataset | Source | Description |
|---------|--------|-------------|
| Phase 3 mapping | `phase3/outputs/product_group_mapping.csv` | Product-to-group mapping |
| Phase 2 output | `phase2/outputs/phase2_output.parquet` | Variant-relevant attributes |

---

## What Phase 4 Does

1. **Loads Phase 3 outputs** — product_group_mapping.csv
2. **Loads Phase 2 output** — for variant_attributes, flavour, formulation, fat_level, fat_percentage
3. **Merges data** — combines group assignments with variant attributes
4. **Builds variant keys** — deterministic keys from size/package attributes
5. **Generates variant IDs** — deterministic UUID5 from group_id + variant_key
6. **Creates variant tables** — main variant table, mapping, summary
7. **Validates** — checks all consistency rules
8. **Generates audit** — Peanut Butter example

---

## Variant Key Logic

The variant key captures size/package information:

```
If count-based: "count{count}"
If size-based: "amt{amount}|unit{unit}[|qty{qty}][|mult{multiplier}]"
If raw: "raw{raw_string}"
If missing: "nosize"
```

Examples:
- `"amt500.0|unitg"` → 500 g
- `"amt1.0|unitkg"` → 1 kg
- `"amt12|unitml|qty12"` → 12 × 12 ml
- `"count20"` → 20 per pack
- `"nosize"` → size missing

---

## Variant ID Generation

```python
variant_id = uuid5(NAMESPACE_DNS, f"{group_id}||{variant_key}")
```

This is:
- **Deterministic**: Same input → same output
- **Reproducible**: Runs produce identical IDs
- **Collision-resistant**: UUID5 namespace-based

---

## Output Files

### reference_product_variants.parquet

One row per unique variant.

| Column | Type | Description |
|--------|------|-------------|
| variant_id | str | Deterministic UUID5 |
| group_id | str | Reference group UUID |
| group_name | str | Most frequent core_title |
| brand | str | Normalized brand |
| product_line | str | Product line |
| product_domain | str | food/non_food/unknown |
| taxonomy | str | Taxonomy category |
| core_title | str | Core product title |
| variant_key | str | Deterministic variant key |
| size | str | Size string |
| size_amount | float | Numeric size amount |
| size_unit | str | Size unit |
| size_qty | int | Quantity multiplier |
| flavour | list | Flavour keywords |
| formulation | list | Formulation keywords |
| fat_level | str | Fat level |
| fat_percentage | float | Fat percentage |
| product_count | int | Products in this variant |
| unique_upcs | int | Unique UPCs in this variant |
| source | str | Data source |
| source_url | str | Source URL |

### product_variant_mapping.parquet

Product-to-variant mapping.

| Column | Type | Description |
|--------|------|-------------|
| external_id | str | Source product ID |
| upc | str | Barcode |
| group_id | str | Reference group UUID |
| variant_id | str | Variant UUID |
| core_title | str | Core title |
| original_title | str | Raw title |
| brand | str | Normalized brand |
| size | str | Size string |
| variant_key | str | Deterministic variant key |
| source | str | Data source |
| source_url | str | Source URL |

### reference_product_variant_summary.parquet

Group-level variant summary.

| Column | Type | Description |
|--------|------|-------------|
| group_id | str | Reference group UUID |
| group_name | str | Most frequent core_title |
| brand | str | Normalized brand |
| product_domain | str | food/non_food/unknown |
| taxonomy | str | Taxonomy category |
| variant_count | int | Number of variants |
| variant_ids | str | Comma-separated variant IDs |
| product_count | int | Total products in group |

---

## Validation Rules

| Rule | Description | Status |
|------|-------------|--------|
| 1 | Every product has exactly one variant | PASS |
| 2 | Every variant belongs to exactly one group | PASS |
| 3 | No product maps to multiple variants | PASS |
| 4 | Variant count matches across tables | PASS |
| 5 | All products have group_id | PASS |
| 6 | All products have variant_id | PASS |

---

## Handling Missing Size

- Products with missing size receive `variant_key = "nosize"`
- They still get deterministic variant_ids
- They are NOT merged together just because size is missing
- Phase 3 group identity remains authoritative

---

## Files

```
phase4/
├── README.md
├── src/
│   └── phase4.py
├── notebooks/
│   ├── phase4.ipynb
│   └── phase4_executed.ipynb
├── outputs/
│   ├── reference_product_variants.parquet
│   ├── reference_product_variants.csv
│   ├── product_variant_mapping.parquet
│   ├── product_variant_mapping.csv
│   ├── reference_product_variant_summary.parquet
│   └── reference_product_variant_summary.csv
├── validation/
│   └── phase4_validation.json
├── statistics/
│   └── phase4_statistics.json
└── audit/
    └── phase4_peanut_butter_example.parquet
```

---

## How to Run

```bash
cd /home/sara/gsoc/compliments-reference-db
python -m phase4.src.phase4
```

---

## LLM Usage

None. All variant assignment is deterministic.

## Nutrition Usage

None. This phase does NOT load or use nutrition data.

## External Datasets

None. This phase does NOT use GPC, Agribalyse, or any external taxonomy.

---

## Traceability

```
source_of_truth/products.parquet
        ↓
Phase 1 output (phase1_output.parquet)
        ↓
Phase 2 output (phase2_output.parquet)
        ↓
Phase 3 product_group_mapping.csv
        ↓
Phase 4 product_variant_mapping.parquet
        ↓
Phase 4 reference_product_variants.parquet
```

---

## Known Limitations

- Variant assignment is based solely on size/package attributes
- Products with identical size but different packaging (e.g., bottle vs jar) cannot be distinguished
- The current dataset does not contain packaging type information

---

## Recommendation for Phase 5

Phase 5 (Nutrition Merge) should:
1. Load `nutrition.parquet` from the authoritative source
2. Join with `product_variant_mapping.parquet` on `external_id`
3. Create a merged product-nutrition table
4. Preserve all variant and group information