Upload audit/phase4_audit_report.md with huggingface_hub

audit/phase4_audit_report.md

@@ -0,0 +1,361 @@
+# Phase 4 — Variant Assignment Audit Report
+
+**Version:** 1.0.0
+**Date:** 2026-07-30
+**Status:** PASS
+
+---
+
+## 1. Input Traceability
+
+### Source of Truth -> Phase 1 -> Phase 2 -> Phase 3 -> Phase 4
+
+| Step | Input | Output | Rows | Columns | Key Operations |
+|------|-------|--------|------|---------|----------------|
+| Source of Truth | `saraNour/compliments-brand/source_of_truth/products.parquet` | — | 4,440 | 16 | Voila product catalog |
+| Phase 1 | products.parquet | phase1_output.parquet | 4,440 | 20 | Data quality, brand_clean, title_clean, dropped size_per_unit/size_total |
+| Phase 2 | phase1_output.parquet | phase2_output.parquet | 4,440 | 38 | Identity extraction, brand_norm, product_line, variant_attributes, core_title, identity_hash |
+| Phase 3 | phase2_output.parquet | product_group_mapping.csv | 4,440 | 14 | Product domain classification, taxonomy, grouping by identity_hash+domain+core_title |
+| Phase 4 | product_group_mapping.csv + phase2_output.parquet | product_variant_mapping.parquet | 4,440 | 11 | Variant key generation, variant_id assignment |
+
+### Input Verification
+
+| Check | Result |
+|-------|--------|
+| Phase 3 products -> Phase 4 products | 4,440 -> 4,440 (0 lost, 0 added) |
+| Phase 3 groups -> Phase 4 groups | 3,691 -> 3,691 (0 lost, 0 added) |
+| Phase 3 rows -> Phase 4 rows | 4,440 -> 4,440 (0 difference) |
+
+---
+
+## 2. What a "Variant" Means
+
+### Definitions
+
+**Product:** A single purchasable item from the Voila catalog, identified by `external_id`.
+
+**Product Group:** A set of products that share the same identity. Group membership is determined by:
+- `identity_hash` (extracted from Phase 2): 9 boolean identity attributes
+- `product_domain`: food / non_food / unknown
+- `normalized_core_title`: tokenized, deduplicated, sorted title
+
+**Variant:** A purchasable representation of the same reference product that differs in **size or package attributes**. Variants within the same group share all identity attributes but differ in size/quantity.
+
+### Relationship
+
+```
+Product Group (identity)
+  Variant 1 (500g)
+    Product A (external_id: 506623EA)
+    Product B (external_id: 484129EA)  [if same size, same variant]
+  Variant 2 (1kg)
+    Product C (external_id: 506661EA)
+  Variant 3 (2kg)
+    Product D (external_id: 250067EA)
+```
+
+### Why This Distinction?
+
+The reference database serves a different purpose than a product catalog:
+- **Product Group** = "What is this product?" (identity)
+- **Variant** = "How can I buy it?" (purchasable form)
+- **Product** = "Which specific SKU am I looking at?" (retailer SKU)
+
+### Concrete Example: Peanut Butter
+
+**Smooth Peanut Butter 500g, 1kg, 2kg:**
+- Same product group (same identity: smooth, non-organic, Compliments)
+- Three size variants: 500g, 1kg, 2kg
+- Different variant_keys: `amt500.0|unitg`, `amt1.0|unitkg`, `amt2.0|unitkg`
+
+**Smooth vs Crunchy vs Organic:**
+- Different product groups (different identity attributes)
+- Smooth != Crunchy (formulation difference)
+- Organic vs non-organic (is_organic difference)
+- Each gets its own group_id
+
+**Dog Treat Peanut Butter:**
+- Different product group (different core_title, different identity)
+- Not a variant of human peanut butter
+
+---
+
+## 3. Variant Statistics
+
+### Overview
+
+| Metric | Value |
+|--------|-------|
+| Total products | 4,440 |
+| Total product groups | 3,691 |
+| Total unique variants | 4,299 |
+| Unique variant keys | 631 |
+
+### Variants per Group
+
+| Count | Groups | % of Total |
+|-------|--------|------------|
+| 1 variant | 3,235 | 87.6% |
+| 2 variants | 357 | 9.7% |
+| 3 variants | 67 | 1.8% |
+| 4+ variants | 32 | 0.9% |
+| **Max** | **10** | — |
+| **Mean** | **1.16** | — |
+
+### Products per Variant
+
+| Count | Variants | % of Total |
+|-------|----------|------------|
+| 1 product (single) | 4,168 | 97.0% |
+| 2+ products (merged) | 131 | 3.0% |
+| **Max** | **4** | — |
+| **Mean** | **1.03** | — |
+
+### Size Information
+
+| Metric | Value |
+|--------|-------|
+| Products with valid size | 3,197 (72.0%) |
+| Products with nosize | 215 (4.8%) |
+| Products with count-based variant | ~1,028 (23.2%) |
+
+### Top 10 Variant Keys
+
+| Variant Key | Products | Description |
+|-------------|----------|-------------|
+| `nosize` | 215 | No size information |
+| `amt400.0\|unitg` | 133 | 400g |
+| `amt200.0\|unitg` | 117 | 200g |
+| `amt500.0\|unitg` | 111 | 500g |
+| `amt300.0\|unitg` | 92 | 300g |
+| `amt454.0\|unitg` | 69 | 454g |
+| `amt250.0\|unitg` | 67 | 250g |
+| `amt1.0\|unitkg` | 65 | 1 kg |
+| `amt450.0\|unitg` | 64 | 450g |
+| `amt1.0\|unitl` | 61 | 1 L |
+
+---
+
+## 4. Variant Assignment Rules
+
+### Rule 1: Variant Key Generation
+
+**Rule Name:** `build_variant_key`
+**Input Columns:** `variant_attributes` (JSON from Phase 2)
+**Output Column:** `variant_key`
+
+**Logic:**
+
+```
+1. Parse variant_attributes JSON
+2. If "count" is present:
+     key = "count{count}"                  # e.g., "count20"
+3. Else if "amount" is present:
+     key = "amt{amount}"
+     If unit:  append "|unit{unit}"         # e.g., "amt500.0|unitg"
+     If qty > 1:  append "|qty{qty}"       # e.g., "amt12|unitml|qty12"
+     If multiplier > 1: append "|mult{mult}"
+4. Else if "raw" is present:
+     key = "raw{raw_string}"               # fallback
+5. Else:
+     key = "nosize"                         # no size info
+```
+
+**Examples:**
+
+| variant_attributes | variant_key |
+|--------------------|-------------|
+| `{"amount": 500.0, "unit": "g"}` | `amt500.0\|unitg` |
+| `{"amount": 1.0, "unit": "kg"}` | `amt1.0\|unitkg` |
+| `{"count": 20}` | `count20` |
+| `{"qty": 12, "amount": 29.583, "unit": "ml"}` | `amt29.583\|unitml\|qty12` |
+| `{}` | `nosize` |
+
+**Why:** Variant keys capture size/package differences. Products with the same variant_key are identical purchasable forms.
+
+### Rule 2: Variant ID Generation
+
+**Rule Name:** `generate_variant_id`
+**Input Columns:** `group_id`, `variant_key`
+**Output Column:** `variant_id`
+
+**Logic:**
+
+```python
+variant_id = uuid5(NAMESPACE_DNS, f"{group_id}||{variant_key}")
+```
+
+**Examples:**
+
+| group_id | variant_key | variant_id |
+|----------|-------------|------------|
+| `b09ff83f-...` | `amt500.0\|unitg` | `a1b2c3d4-...` |
+| `b09ff83f-...` | `amt1.0\|unitkg` | `e5f6a7b8-...` |
+| `b09ff83f-...` | `amt2.0\|unitkg` | `c9d0e1f2-...` |
+
+**Why:** Deterministic UUID5 ensures reproducibility. Same input always produces same ID.
+
+### Rule 3: Variant Assignment
+
+**Rule Name:** `assign_variants`
+**Input Columns:** `group_id`, `variant_key`
+**Output Columns:** `variant_key`, `variant_id`
+
+**Logic:**
+
+```
+1. For each product: compute variant_key from variant_attributes
+2. For each product: compute variant_id from group_id + variant_key
+3. Products in same group with same variant_key -> same variant_id
+4. Products in same group with different variant_keys -> different variant_ids
+```
+
+**Why:** This is the core variant assignment logic. It groups products by their purchasable form within each reference product group.
+
+---
+
+## 5. Variant Attributes
+
+### Currently Represented
+
+| Attribute | Source | Used for Variant? | Description |
+|-----------|--------|-------------------|-------------|
+| `size_amount` | Phase 2 | Yes (via variant_key) | Numeric size (500, 1, 2) |
+| `size_unit` | Phase 2 | Yes (via variant_key) | Unit (g, kg, ml, L, count) |
+| `size_qty` | Phase 2 | Yes (via variant_key) | Quantity multiplier (12-pack) |
+| `variant_attributes` | Phase 2 | Yes (via variant_key) | JSON with amount/unit/qty/count |
+| `size` | Phase 3 | No (metadata only) | Raw size string ("500g", "1 kg") |
+| `brand` | Phase 2 | No (metadata only) | Normalized brand |
+| `product_line` | Phase 2 | No (metadata only) | Product line |
+
+### Currently NOT Represented (Missing)
+
+| Attribute | Impact | Notes |
+|-----------|--------|-------|
+| Package type (bottle, jar, can) | Cannot distinguish jar vs bottle variants | Would need Voila packaging data |
+| Package material (glass, plastic) | Cannot distinguish material variants | Not available in source data |
+| Multipack configuration | Limited multipack handling | Only qty is captured |
+| Seasonal/limited editions | Cannot distinguish | Not in source data |
+| Temperature variant (frozen vs shelf) | Cannot distinguish | Not in source data |
+
+### Identity Attributes (NOT Variant Attributes)
+
+These attributes define the **product group**, not the variant:
+
+| Attribute | Column | Description |
+|-----------|--------|-------------|
+| is_organic | Phase 2 | Organic flag |
+| is_gluten_free | Phase 2 | Gluten-free flag |
+| is_naturally_simple | Phase 2 | Naturally simple flag |
+| is_sugar_free | Phase 2 | Sugar-free flag |
+| is_unsalted | Phase 2 | Unsalted flag |
+| is_lactose_free | Phase 2 | Lactose-free flag |
+| is_peanut_free | Phase 2 | Peanut-free flag |
+| is_plant_based | Phase 2 | Plant-based flag |
+| is_reduced_sodium | Phase 2 | Reduced sodium flag |
+| fat_level | Phase 2 | Fat level |
+| fat_percentage | Phase 2 | Fat percentage |
+| flavour | Phase 2 | Flavour keywords |
+| formulation | Phase 2 | Formulation keywords |
+| product_line | Phase 2 | Product line |
+| product_domain | Phase 3 | food/non_food/unknown |
+| taxonomy | Phase 3 | Taxonomy category |
+
+---
+
+## 6. Peanut Butter Example
+
+See `peanut_butter_variant_example.parquet` for the full table.
+
+### Summary
+
+| Group | Name | Products | Variants |
+|-------|------|----------|----------|
+| `b09ff83f-...` | Smooth Peanut Butter | 3 | 3 |
+| `bc4d1309-...` | Peanut Butter Crunchy | 3 | 3 |
+| `5bfec681-...` | Light Smooth Peanut Butter | 3 | 3 |
+| `e9bc48d6-...` | Compliments 100% Natural Smooth PB | 2 | 2 |
+| + 17 single-product groups | — | 17 | 17 |
+
+### Multi-Variant Groups
+
+**Smooth Peanut Butter (Group b09ff83f):**
+| Product | Size | Variant Key |
+|---------|------|-------------|
+| Smooth Peanut Butter | 500g | `amt500.0\|unitg` |
+| Smooth Peanut Butter | 1kg | `amt1.0\|unitkg` |
+| Peanut Butter Smooth | 2kg | `amt2.0\|unitkg` |
+
+**Peanut Butter Crunchy (Group bc4d1309):**
+| Product | Size | Variant Key |
+|---------|------|-------------|
+| Peanut Butter Crunchy | 500g | `amt500.0\|unitg` |
+| Peanut Butter Crunchy | 1kg | `amt1.0\|unitkg` |
+| Peanut Butter Crunchy | 2kg | `amt2.0\|unitkg` |
+
+---
+
+## 7. Validation Results
+
+| Rule | Description | Result |
+|------|-------------|--------|
+| 1 | Every product has exactly one variant | PASS |
+| 2 | Every variant belongs to exactly one group | PASS |
+| 3 | No orphan variants | PASS |
+| 4 | No duplicate product assignments | PASS |
+| 5 | Summary matches variant table | PASS |
+| 6 | All products have group_id and variant_id | PASS |
+| **Overall** | | **PASS** |
+
+---
+
+## 8. Potential Issues
+
+### Issue 1: Products with nosize (215 products, 4.8%)
+
+**Description:** 215 products have no size information in `variant_attributes`. They receive `variant_key = "nosize"` and are still assigned deterministic variant_ids.
+
+**Impact:** Low. These products cannot be distinguished by size. If two products in the same group both have nosize, they will share the same variant_id.
+
+**Recommendation:** This is expected behavior. Size information is not always available in the Voila catalog.
+
+### Issue 2: Limited variant-distinguishing attributes
+
+**Description:** Variant assignment is based solely on size/package attributes. Products with identical size but different packaging (bottle vs jar) cannot be distinguished.
+
+**Impact:** Low for current dataset. The Voila catalog does not contain packaging type information.
+
+**Recommendation:** If packaging data becomes available in the future, it can be added to variant_key generation.
+
+### Issue 3: Products with merged variants
+
+**Description:** 131 variants contain 2+ products (3.0%). These are products with identical variant_keys within the same group.
+
+**Impact:** Low. These products have the same size and are in the same group. They are effectively the same purchasable form.
+
+**Recommendation:** This is expected behavior. Multiple retailer SKUs can map to the same variant.
+
+---
+
+## 9. Recommendations
+
+1. **Phase 4 is safe to freeze.** All validation checks pass. The variant assignment logic is deterministic and reproducible.
+2. **Phase 5 (Nutrition Merge)** should join nutrition data on `external_id` via the `product_variant_mapping.parquet`.
+3. **Future enhancement:** If packaging type data becomes available, it can be added to variant_key without breaking the current structure.
+
+---
+
+## 10. Files
+
+| File | Format | Description |
+|------|--------|-------------|
+| `phase4_audit_summary.parquet` | Parquet | Audit summary metrics |
+| `phase4_audit_summary.csv` | CSV | Audit summary metrics |
+| `peanut_butter_variant_example.parquet` | Parquet | Peanut Butter variant example |
+| `peanut_butter_variant_example.csv` | CSV | Peanut Butter variant example |
+| `phase4_audit_report.md` | Markdown | This report |
+
+---
+
+*Generated by Phase 4 Audit Process*
+*Date: 2026-07-30*