README.md

# Phase 5 — Nutrition Integration

## Compliments Reference DB Pipeline

**Version:** 1.0.0
**Date:** 2026-07-30
**Status:** PASS

---

## Purpose

Phase 5 integrates nutrition data from the authoritative source into the Compliments Reference DB. It joins nutrition.parquet with the product pipeline using deterministic matching by external_id.

---

## Inputs

| Dataset | Source | Description |
|---------|--------|-------------|
| `nutrition.parquet` | `saraNour/compliments-brand/source_of_truth/` | Authoritative nutrition data (4,440 × 21) |
| `products.parquet` | `saraNour/compliments-brand/source_of_truth/` | Authoritative product catalog (4,440 × 16) |
| `product_group_mapping.csv` | Phase 3 output | Product → group mapping (4,440 × 14) |
| `product_variant_mapping.parquet` | Phase 4 output | Product → variant mapping (4,440 × 11) |

---

## Outputs

All production data tables are Parquet.

### product_group_mapping.parquet
- Phase 3 mapping converted to Parquet for downstream consumption
- 4,440 rows × 14 columns

### nutrition_cleaned.parquet
- Cleaned/validated nutrition source
- Original values preserved
- Quality flags included
- 4,440 rows × 23 columns

### product_nutrition_mapping.parquet
- Product ↔ nutrition mapping with group_id and variant_id
- 4,440 rows × 27 columns

### nutrition_per_100g.parquet
- Normalized per-100g nutrition values
- Conversion/provenance metadata
- 4,440 rows × 24 columns

### phase5_statistics.parquet
- Summary metrics

### phase5_validation.parquet
- Validation checks and results

---

## Matching Strategy

**Primary key:** `external_id` (exact match)
**Confidence:** HIGH
**Coverage:** 100% (4,440/4,440)

No fallback matching was needed. The relationship is 1:1.

---

## Normalization Strategy

Nutrition values are normalized to per-100g only when a reliable serving-size conversion is possible.

| Method | Count | Description |
|--------|------:|-------------|
| Direct 100g/mL | 150 | serving_size is 100 g or 100 mL |
| Scaled to 100g/mL | 1,351 | Serving size in g or mL, scaled proportionally |
| Not normalized | 2,939 | Ambiguous serving unit or missing serving_size |

### Why Not Normalized?

- **Missing serving_size** (1,505 records): Non-food products (light bulbs, gloves, etc.)
- **Ambiguous serving unit** (1,434 records): tbsp, cup, tsp, slices, pieces, etc. — no reliable gram equivalent available
- **No fabrication**: We do NOT invent serving weight conversions

---

## Suspicious Value Policy

Suspicious values are **flagged, not overwritten**.

| Field | Threshold | Records Flagged |
|-------|-----------|----------------:|
| calories | > 1000 | 1 |
| fat_g | > 100 | 1 |
| protein_g | > 100 | 1 |

Original values are preserved in `nutrition_cleaned.parquet` with `nutrition_quality_status = 'SUSPICIOUS'`.

---

## Quality Status

| Status | Count | Description |
|--------|------:|-------------|
| VALID | 2,932 | Normal nutrition values |
| MISSING | 1,505 | All 17 nutrition fields null (non-food products) |
| SUSPICIOUS | 3 | Out-of-range values flagged |

---

## Validation

| Rule | Description | Result |
|------|-------------|--------|
| 1 | All products have nutrition match | PASS |
| 2 | No duplicate external_ids | PASS |
| 3 | No duplicate matches | PASS |
| 4 | Suspicious values preserved | PASS |
| 5 | No fabricated values | PASS |
| 6 | No external data used | PASS |
| 7 | All outputs Parquet | PASS |
| 8 | No LLM used | PASS |
| **Overall** | | **PASS** |

---

## Traceability

```
products.parquet (source of truth)
    ↓
nutrition.parquet (source of truth)
    ↓
Phase 5: nutrition_cleaned.parquet
    ↓
Phase 5: product_nutrition_mapping.parquet
    ↓ (group_id from Phase 3, variant_id from Phase 4)
Phase 5: nutrition_per_100g.parquet
    ↓
Phase 6: Nutrition Scoring
```

---

## Limitations

1. **Per-100g normalization is partial** — only possible for ~34% of records (1,501/4,440)
2. **Serving size parsing is limited** — complex formats like "1 cup (250 mL)" are not parsed
3. **Suspicious values are flagged, not corrected** — requires domain expert review
4. **Non-food products have no nutrition** — 1,505 records with all-null nutrition fields

---

## Files

```
phase5/
├── README.md
├── src/
│   └── phase5.py
├── notebooks/
│   ├── phase5.ipynb
│   └── phase5_executed.ipynb
├── outputs/
│   ├── product_group_mapping.parquet
│   ├── nutrition_cleaned.parquet
│   ├── product_nutrition_mapping.parquet
│   ├── nutrition_per_100g.parquet
│   ├── phase5_statistics.parquet
│   └── phase5_validation.parquet
├── validation/
│   └── phase5_validation.json
└── statistics/
    └── phase5_statistics.json
```

---

## How to Run

```bash
cd /home/sara/gsoc/compliments-reference-db
python -m phase5.src.phase5
```

---

## LLM Usage

None. All matching is deterministic by external_id.

## External Datasets

None. Only authoritative nutrition source used.

---

*Generated by Phase 5 Production Pipeline*
*Date: 2026-07-30*