README.md

E-commerce search extraction experiment archive

This Bucket is the reproducibility and analysis archive for schema-conditioned e-commerce query extraction using Ionio-ai/ecommerce-search-extraction.

Published models

• Qwen3.5-0.8B LoRA adapter
• Qwen3.5-0.8B GGUF quantizations
• Qwen2.5-0.5B-Instruct LoRA adapter
• Qwen2.5-0.5B-Instruct GGUF quantizations

Weights remain in the model repositories above. This Bucket stores reports, metrics, predictions, frozen evaluation inputs, manual review, and reproduction code so model binaries do not diverge across locations.

Headline result

Qwen3.5-0.8B LoRA is the stronger extractor. On the deliberately hard 30-case stress suite it achieved 96.67% strict JSON, 93.33% schema conformance, 87.45% Leaf F1, and 22/30 full human-reviewed passes. Qwen2.5-0.5B achieved 96.67%, 96.67%, 80.16%, and 10/30 respectively. JSON-Schema-guided decoding brought Qwen2.5 strict/schema validity to 100%, but did not eliminate substantive extraction errors.

The stress set is failure-enriched and is not an estimate of ordinary production prevalence. The standard held-out evaluation and GGUF quantization results are included separately.

Training-data audit

The 9,341-example training split is broad but weakly standardized:

• 7,971 unique top-level keys; 65.60% are singletons and 88.02% occur at most five times.
• 491 paths use multiple non-null JSON types.
• Only 39 rows contain arrays of objects, 140 contain correction/revision language, and 12 contain explicit zero-price constraints.
• No target contains a JSON-looking literal string; the only five target strings with quotes/backslashes are inch notation.
• Brand/color indifference and “not required” are represented inconsistently through omission, null, hard booleans, and one-off keys.
• Five exact duplicate-query groups have conflicting targets.

See reports/training_data_quality_report.pdf for evidence, careful attribution, representative good/bad records, limitations, and remediation priorities.

Inference contract

Use the same prompt structure as training:

• System: Extract structured attributes from the e-commerce search query according to the provided JSON Schema. Return only one valid JSON object matching the schema. Include every required property. Use JSON null only when a required property is not stated or cannot be inferred. Do not add properties that are not allowed by the schema.
• User: the JSON Schema followed by the e-commerce query.
• Decode greedily with temperature=0; reasoning/thinking disabled. The evaluations allowed up to 4096 new tokens.
• Validate the raw response as strict JSON and then against the supplied per-request schema. Do not count repaired/recoverable JSON as strict-valid output.

The exact formatter is in repro/ecom_eval/prompting.py.

Layout

• reports/: baseline evaluation, fine-tuning, GGUF, stress, and training-data audit reports in Markdown/PDF.
• analysis/baseline/: dataset audit, aggregate/stratified metrics, paired comparisons, taxonomy, qualitative review, and figures.
• analysis/training-data/: quantified training-label audit and curated source examples.
• analysis/stress/: all stress outputs and human judgments.
• results/baseline/: full-dataset predictions and summaries for the four requested base models.
• results/finetuned/: held-out predictions and summaries for base controls and LoRA models.
• results/gguf/: predictions and summaries for five quantization levels per fine-tuned model.
• results/stress/: unconstrained and JSON-Schema-guided stress predictions.
• data/stress/hard_30.jsonl: frozen hand-authored stress suite with gold targets and rationales.
• data/source/ecommerce_search_extraction_10985.jsonl: original 10,985-row source-form dataset export, including metadata and generation-lineage fields.
• data/eval_full_10985.jsonl: all source rows transformed into the schema-conditioned evaluation format.
• data/sft/: exact query-group-disjoint 9,341/549/1,095 train/validation/test split used for fine-tuning and held-out evaluation.
• data/sft/split_audit.json: leakage/disjointness audit for the fine-tuning split.
• repro/: analysis, evaluation, training, reporting, and prompt code.

Metric definitions

• Strict JSON validity: raw model text parses directly as exactly one JSON value; no repair or wrapper stripping.
• Schema conformance: parsed output validates against the sample's JSON Schema.
• Exact match: recursively canonical JSON equality with the gold target; array order remains significant unless separately reported.
• Leaf F1: micro-style overlap of flattened path/value leaves, useful when exact match fails due to only a subset of fields.
• Key F1: overlap of flattened key paths, independent of leaf value correctness.
• Null accuracy: correctness on gold fields whose value is JSON null.
• Human full pass / partial / material fail: substantive review that ignores harmless casing, morphology, and array ordering but penalizes lost, inverted, fabricated, or structurally corrupted constraints.

Every result directory includes its own summary.json with decoding, runtime, environment, and throughput fields where available.

Reproduction notes

All primary evaluation runs use deterministic greedy decoding. Qwen reasoning was disabled with enable_thinking=false; GPT-OSS used the lowest available reasoning setting where disabling was unsupported. The fine-tuning runs used TRL, LoRA rank 32, assistant-only loss, cosine scheduling, and two epochs. Consult the reports and scripts for the exact model-specific configuration.

This archive contains no access tokens or credentials.