Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,266 @@
+# Phase 4 — Variant Assignment / Variant Modeling
+
+## Compliments Reference DB Pipeline
+
+**Version:** 1.0.0
+**Date:** 2026-07-30
+**Status:** PASS
+
+---
+
+## Purpose
+
+Phase 4 transforms the Phase 3 product-group representation into a structured product variant model. Products belonging to the same reference product group can have different purchasable variants such as size, package quantity, formulation, flavour, fat level, etc.
+
+---
+
+## Key Concepts
+
+### Reference Product Group (from Phase 3)
+
+A reference product group represents the same underlying product identity. All products in a group share the same identity attributes (organic, gluten-free, fat level, flavour, formulation, product_line).
+
+### Variant
+
+A **variant** is a purchasable representation of the same reference product that differs in attributes such as:
+
+- **Size**: 500 g, 1 kg, 2 kg
+- **Package quantity**: 1 × 500 g, 2 × 500 g
+- **Pack configuration**: single, multipack, case
+
+### Identifier Hierarchy
+
+| Identifier | Level | Description |
+|------------|-------|-------------|
+| `external_id` | Source product | Voila retailer product ID (unique per product) |
+| `upc` | Source product | Barcode (may be reused across variants) |
+| `group_id` | Reference group | Deterministic UUID5 from Phase 3 |
+| `variant_id` | Variant | Deterministic UUID5 from group_id + variant_key |
+
+---
+
+## Input Datasets
+
+| Dataset | Source | Description |
+|---------|--------|-------------|
+| Phase 3 mapping | `phase3/outputs/product_group_mapping.csv` | Product-to-group mapping |
+| Phase 2 output | `phase2/outputs/phase2_output.parquet` | Variant-relevant attributes |
+
+---
+
+## What Phase 4 Does
+
+1. **Loads Phase 3 outputs** — product_group_mapping.csv
+2. **Loads Phase 2 output** — for variant_attributes, flavour, formulation, fat_level, fat_percentage
+3. **Merges data** — combines group assignments with variant attributes
+4. **Builds variant keys** — deterministic keys from size/package attributes
+5. **Generates variant IDs** — deterministic UUID5 from group_id + variant_key
+6. **Creates variant tables** — main variant table, mapping, summary
+7. **Validates** — checks all consistency rules
+8. **Generates audit** — Peanut Butter example
+
+---
+
+## Variant Key Logic
+
+The variant key captures size/package information:
+
+```
+If count-based: "count{count}"
+If size-based: "amt{amount}|unit{unit}[|qty{qty}][|mult{multiplier}]"
+If raw: "raw{raw_string}"
+If missing: "nosize"
+```
+
+Examples:
+- `"amt500.0|unitg"` → 500 g
+- `"amt1.0|unitkg"` → 1 kg
+- `"amt12|unitml|qty12"` → 12 × 12 ml
+- `"count20"` → 20 per pack
+- `"nosize"` → size missing
+
+---
+
+## Variant ID Generation
+
+```python
+variant_id = uuid5(NAMESPACE_DNS, f"{group_id}||{variant_key}")
+```
+
+This is:
+- **Deterministic**: Same input → same output
+- **Reproducible**: Runs produce identical IDs
+- **Collision-resistant**: UUID5 namespace-based
+
+---
+
+## Output Files
+
+### reference_product_variants.parquet
+
+One row per unique variant.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| variant_id | str | Deterministic UUID5 |
+| group_id | str | Reference group UUID |
+| group_name | str | Most frequent core_title |
+| brand | str | Normalized brand |
+| product_line | str | Product line |
+| product_domain | str | food/non_food/unknown |
+| taxonomy | str | Taxonomy category |
+| core_title | str | Core product title |
+| variant_key | str | Deterministic variant key |
+| size | str | Size string |
+| size_amount | float | Numeric size amount |
+| size_unit | str | Size unit |
+| size_qty | int | Quantity multiplier |
+| flavour | list | Flavour keywords |
+| formulation | list | Formulation keywords |
+| fat_level | str | Fat level |
+| fat_percentage | float | Fat percentage |
+| product_count | int | Products in this variant |
+| unique_upcs | int | Unique UPCs in this variant |
+| source | str | Data source |
+| source_url | str | Source URL |
+
+### product_variant_mapping.parquet
+
+Product-to-variant mapping.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| external_id | str | Source product ID |
+| upc | str | Barcode |
+| group_id | str | Reference group UUID |
+| variant_id | str | Variant UUID |
+| core_title | str | Core title |
+| original_title | str | Raw title |
+| brand | str | Normalized brand |
+| size | str | Size string |
+| variant_key | str | Deterministic variant key |
+| source | str | Data source |
+| source_url | str | Source URL |
+
+### reference_product_variant_summary.parquet
+
+Group-level variant summary.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| group_id | str | Reference group UUID |
+| group_name | str | Most frequent core_title |
+| brand | str | Normalized brand |
+| product_domain | str | food/non_food/unknown |
+| taxonomy | str | Taxonomy category |
+| variant_count | int | Number of variants |
+| variant_ids | str | Comma-separated variant IDs |
+| product_count | int | Total products in group |
+
+---
+
+## Validation Rules
+
+| Rule | Description | Status |
+|------|-------------|--------|
+| 1 | Every product has exactly one variant | PASS |
+| 2 | Every variant belongs to exactly one group | PASS |
+| 3 | No product maps to multiple variants | PASS |
+| 4 | Variant count matches across tables | PASS |
+| 5 | All products have group_id | PASS |
+| 6 | All products have variant_id | PASS |
+
+---
+
+## Handling Missing Size
+
+- Products with missing size receive `variant_key = "nosize"`
+- They still get deterministic variant_ids
+- They are NOT merged together just because size is missing
+- Phase 3 group identity remains authoritative
+
+---
+
+## Files
+
+```
+phase4/
+├── README.md
+├── src/
+│   └── phase4.py
+├── notebooks/
+│   ├── phase4.ipynb
+│   └── phase4_executed.ipynb
+├── outputs/
+│   ├── reference_product_variants.parquet
+│   ├── reference_product_variants.csv
+│   ├── product_variant_mapping.parquet
+│   ├── product_variant_mapping.csv
+│   ├── reference_product_variant_summary.parquet
+│   └── reference_product_variant_summary.csv
+├── validation/
+│   └── phase4_validation.json
+├── statistics/
+│   └── phase4_statistics.json
+└── audit/
+    └── phase4_peanut_butter_example.parquet
+```
+
+---
+
+## How to Run
+
+```bash
+cd /home/sara/gsoc/compliments-reference-db
+python -m phase4.src.phase4
+```
+
+---
+
+## LLM Usage
+
+None. All variant assignment is deterministic.
+
+## Nutrition Usage
+
+None. This phase does NOT load or use nutrition data.
+
+## External Datasets
+
+None. This phase does NOT use GPC, Agribalyse, or any external taxonomy.
+
+---
+
+## Traceability
+
+```
+source_of_truth/products.parquet
+        ↓
+Phase 1 output (phase1_output.parquet)
+        ↓
+Phase 2 output (phase2_output.parquet)
+        ↓
+Phase 3 product_group_mapping.csv
+        ↓
+Phase 4 product_variant_mapping.parquet
+        ↓
+Phase 4 reference_product_variants.parquet
+```
+
+---
+
+## Known Limitations
+
+- Variant assignment is based solely on size/package attributes
+- Products with identical size but different packaging (e.g., bottle vs jar) cannot be distinguished
+- The current dataset does not contain packaging type information
+
+---
+
+## Recommendation for Phase 5
+
+Phase 5 (Nutrition Merge) should:
+1. Load `nutrition.parquet` from the authoritative source
+2. Join with `product_variant_mapping.parquet` on `external_id`
+3. Create a merged product-nutrition table
+4. Preserve all variant and group information