README.md

# E-commerce search extraction experiment archive

This Bucket is the reproducibility and analysis archive for schema-conditioned e-commerce query extraction using [`Ionio-ai/ecommerce-search-extraction`](https://huggingface.co/datasets/Ionio-ai/ecommerce-search-extraction).

## Published models

- [Qwen3.5-0.8B LoRA adapter](https://huggingface.co/Ionio-ai/Qwen3.5-0.8B-Ecommerce-Extraction-LoRA)
- [Qwen3.5-0.8B GGUF quantizations](https://huggingface.co/Ionio-ai/Qwen3.5-0.8B-Ecommerce-Extraction-GGUF)
- [Qwen2.5-0.5B-Instruct LoRA adapter](https://huggingface.co/Ionio-ai/Qwen2.5-0.5B-Instruct-Ecommerce-Extraction-LoRA)
- [Qwen2.5-0.5B-Instruct GGUF quantizations](https://huggingface.co/Ionio-ai/Qwen2.5-0.5B-Instruct-Ecommerce-Extraction-GGUF)

Weights remain in the model repositories above. This Bucket stores reports, metrics, predictions, frozen evaluation inputs, manual review, and reproduction code so model binaries do not diverge across locations.

## Headline result

Qwen3.5-0.8B LoRA is the stronger extractor. On the deliberately hard 30-case stress suite it achieved 96.67% strict JSON, 93.33% schema conformance, 87.45% Leaf F1, and 22/30 full human-reviewed passes. Qwen2.5-0.5B achieved 96.67%, 96.67%, 80.16%, and 10/30 respectively. JSON-Schema-guided decoding brought Qwen2.5 strict/schema validity to 100%, but did not eliminate substantive extraction errors.

The stress set is failure-enriched and is not an estimate of ordinary production prevalence. The standard held-out evaluation and GGUF quantization results are included separately.

## Training-data audit

The 9,341-example training split is broad but weakly standardized:

- 7,971 unique top-level keys; 65.60% are singletons and 88.02% occur at most five times.
- 491 paths use multiple non-null JSON types.
- Only 39 rows contain arrays of objects, 140 contain correction/revision language, and 12 contain explicit zero-price constraints.
- No target contains a JSON-looking literal string; the only five target strings with quotes/backslashes are inch notation.
- Brand/color indifference and “not required” are represented inconsistently through omission, null, hard booleans, and one-off keys.
- Five exact duplicate-query groups have conflicting targets.

See `reports/training_data_quality_report.pdf` for evidence, careful attribution, representative good/bad records, limitations, and remediation priorities.

## Inference contract

Use the same prompt structure as training:

- System: `Extract structured attributes from the e-commerce search query according to the provided JSON Schema. Return only one valid JSON object matching the schema. Include every required property. Use JSON null only when a required property is not stated or cannot be inferred. Do not add properties that are not allowed by the schema.`
- User: the JSON Schema followed by the e-commerce query.
- Decode greedily with `temperature=0`; reasoning/thinking disabled. The evaluations allowed up to 4096 new tokens.
- Validate the raw response as strict JSON and then against the supplied per-request schema. Do not count repaired/recoverable JSON as strict-valid output.

The exact formatter is in `repro/ecom_eval/prompting.py`.

## Layout

- `reports/`: baseline evaluation, fine-tuning, GGUF, stress, and training-data audit reports in Markdown/PDF.
- `analysis/baseline/`: dataset audit, aggregate/stratified metrics, paired comparisons, taxonomy, qualitative review, and figures.
- `analysis/training-data/`: quantified training-label audit and curated source examples.
- `analysis/stress/`: all stress outputs and human judgments.
- `results/baseline/`: full-dataset predictions and summaries for the four requested base models.
- `results/finetuned/`: held-out predictions and summaries for base controls and LoRA models.
- `results/gguf/`: predictions and summaries for five quantization levels per fine-tuned model.
- `results/stress/`: unconstrained and JSON-Schema-guided stress predictions.
- `data/stress/hard_30.jsonl`: frozen hand-authored stress suite with gold targets and rationales.
- `data/source/ecommerce_search_extraction_10985.jsonl`: original 10,985-row source-form dataset export, including metadata and generation-lineage fields.
- `data/eval_full_10985.jsonl`: all source rows transformed into the schema-conditioned evaluation format.
- `data/sft/`: exact query-group-disjoint 9,341/549/1,095 train/validation/test split used for fine-tuning and held-out evaluation.
- `data/sft/split_audit.json`: leakage/disjointness audit for the fine-tuning split.
- `repro/`: analysis, evaluation, training, reporting, and prompt code.

## Metric definitions

- **Strict JSON validity:** raw model text parses directly as exactly one JSON value; no repair or wrapper stripping.
- **Schema conformance:** parsed output validates against the sample's JSON Schema.
- **Exact match:** recursively canonical JSON equality with the gold target; array order remains significant unless separately reported.
- **Leaf F1:** micro-style overlap of flattened path/value leaves, useful when exact match fails due to only a subset of fields.
- **Key F1:** overlap of flattened key paths, independent of leaf value correctness.
- **Null accuracy:** correctness on gold fields whose value is JSON `null`.
- **Human full pass / partial / material fail:** substantive review that ignores harmless casing, morphology, and array ordering but penalizes lost, inverted, fabricated, or structurally corrupted constraints.

Every result directory includes its own `summary.json` with decoding, runtime, environment, and throughput fields where available.

## Reproduction notes

All primary evaluation runs use deterministic greedy decoding. Qwen reasoning was disabled with `enable_thinking=false`; GPT-OSS used the lowest available reasoning setting where disabling was unsupported. The fine-tuning runs used TRL, LoRA rank 32, assistant-only loss, cosine scheduling, and two epochs. Consult the reports and scripts for the exact model-specific configuration.

This archive contains no access tokens or credentials.