Phase 4 — Variant Assignment / Variant Modeling
Compliments Reference DB Pipeline
Version: 1.0.0
Date: 2026-07-30
Status: PASS
Purpose
Phase 4 transforms the Phase 3 product-group representation into a structured product variant model. Products belonging to the same reference product group can have different purchasable variants such as size, package quantity, formulation, flavour, fat level, etc.
Key Concepts
Reference Product Group (from Phase 3)
A reference product group represents the same underlying product identity. All products in a group share the same identity attributes (organic, gluten-free, fat level, flavour, formulation, product_line).
Variant
A variant is a purchasable representation of the same reference product that differs in attributes such as:
Size : 500 g, 1 kg, 2 kgPackage quantity : 1 × 500 g, 2 × 500 gPack configuration : single, multipack, case
Identifier Hierarchy
Identifier Level Description external_idSource product Voila retailer product ID (unique per product) upcSource product Barcode (may be reused across variants) group_idReference group Deterministic UUID5 from Phase 3 variant_idVariant Deterministic UUID5 from group_id + variant_key
Input Datasets
Dataset Source Description Phase 3 mapping phase3/outputs/product_group_mapping.csvProduct-to-group mapping Phase 2 output phase2/outputs/phase2_output.parquetVariant-relevant attributes
What Phase 4 Does
Loads Phase 3 outputs — product_group_mapping.csvLoads Phase 2 output — for variant_attributes, flavour, formulation, fat_level, fat_percentageMerges data — combines group assignments with variant attributesBuilds variant keys — deterministic keys from size/package attributesGenerates variant IDs — deterministic UUID5 from group_id + variant_keyCreates variant tables — main variant table, mapping, summaryValidates — checks all consistency rulesGenerates audit — Peanut Butter example
Variant Key Logic
The variant key captures size/package information:
If count-based: "count{count}"
If size-based: "amt{amount}|unit{unit}[|qty{qty}][|mult{multiplier}]"
If raw: "raw{raw_string}"
If missing: "nosize"Examples:
"amt500.0|unitg" → 500 g"amt1.0|unitkg" → 1 kg"amt12|unitml|qty12" → 12 × 12 ml"count20" → 20 per pack"nosize" → size missing
Variant ID Generation
variant_id = uuid5(NAMESPACE_DNS, f"{group_id}||{variant_key}")This is:
Deterministic : Same input → same outputReproducible : Runs produce identical IDsCollision-resistant : UUID5 namespace-based
Output Files
reference_product_variants.parquet
One row per unique variant.
Column Type Description variant_id str Deterministic UUID5 group_id str Reference group UUID group_name str Most frequent core_title brand str Normalized brand product_line str Product line product_domain str food/non_food/unknown taxonomy str Taxonomy category core_title str Core product title variant_key str Deterministic variant key size str Size string size_amount float Numeric size amount size_unit str Size unit size_qty int Quantity multiplier flavour list Flavour keywords formulation list Formulation keywords fat_level str Fat level fat_percentage float Fat percentage product_count int Products in this variant unique_upcs int Unique UPCs in this variant source str Data source source_url str Source URL
product_variant_mapping.parquet
Product-to-variant mapping.
Column Type Description external_id str Source product ID upc str Barcode group_id str Reference group UUID variant_id str Variant UUID core_title str Core title original_title str Raw title brand str Normalized brand size str Size string variant_key str Deterministic variant key source str Data source source_url str Source URL
reference_product_variant_summary.parquet
Group-level variant summary.
Column Type Description group_id str Reference group UUID group_name str Most frequent core_title brand str Normalized brand product_domain str food/non_food/unknown taxonomy str Taxonomy category variant_count int Number of variants variant_ids str Comma-separated variant IDs product_count int Total products in group
Validation Rules
Rule Description Status 1 Every product has exactly one variant PASS 2 Every variant belongs to exactly one group PASS 3 No product maps to multiple variants PASS 4 Variant count matches across tables PASS 5 All products have group_id PASS 6 All products have variant_id PASS
Handling Missing Size
Products with missing size receive variant_key = "nosize"
They still get deterministic variant_ids
They are NOT merged together just because size is missing
Phase 3 group identity remains authoritative
Files
phase4/
├── README.md
├── src/
│ └── phase4.py
├── notebooks/
│ ├── phase4.ipynb
│ └── phase4_executed.ipynb
├── outputs/
│ ├── reference_product_variants.parquet
│ ├── reference_product_variants.csv
│ ├── product_variant_mapping.parquet
│ ├── product_variant_mapping.csv
│ ├── reference_product_variant_summary.parquet
│ └── reference_product_variant_summary.csv
├── validation/
│ └── phase4_validation.json
├── statistics/
│ └── phase4_statistics.json
└── audit/
└── phase4_peanut_butter_example.parquetHow to Run
1 cd /home/sara/gsoc/compliments-reference-db
2 python -m phase4.src.phase4
LLM Usage
None. All variant assignment is deterministic.
Nutrition Usage
None. This phase does NOT load or use nutrition data.
External Datasets
None. This phase does NOT use GPC, Agribalyse, or any external taxonomy.
Traceability
source_of_truth/products.parquet
↓
Phase 1 output (phase1_output.parquet)
↓
Phase 2 output (phase2_output.parquet)
↓
Phase 3 product_group_mapping.csv
↓
Phase 4 product_variant_mapping.parquet
↓
Phase 4 reference_product_variants.parquetKnown Limitations
Variant assignment is based solely on size/package attributes
Products with identical size but different packaging (e.g., bottle vs jar) cannot be distinguished
The current dataset does not contain packaging type information
Recommendation for Phase 5
Phase 5 (Nutrition Merge) should:
Load nutrition.parquet from the authoritative source
Join with product_variant_mapping.parquet on external_id
Create a merged product-nutrition table
Preserve all variant and group information
