Archive original USPTO-SynPlanner data and add extraction-retained final dataset

README.md

@@ -21,9 +21,17 @@ datasets/
     README.md
     dataset.yaml
     raw/molecular_transformer_data.zip
-  uspto_synplanner/
+  uspto_synplan/
     README.md
-    dataset.yaml
+    uspto_synplan_old/
+      README.md
+      dataset.yaml
+      uspto_synplan.smi.zip
+    uspto_synplan_final/
+      README.md
+      passed_reactions.smi.zip
+      rules_extraction.yaml
+      provenance.json
 models/
   README.md
   uspto_mit/README.md
@@ -40,7 +48,7 @@ models/
 
 ## Datasets
 
-Raw files use the same relative paths and bytes as the local project.
+Raw source artifacts and their dataset versions are documented below.
 [datasets/artifacts.json](datasets/artifacts.json) records their sizes and SHA-256
 hashes. Prepared data, generated outputs, and machine-specific manifests are not
 published here.
@@ -51,8 +59,10 @@ published here.
   [MolecularTransformerData source](https://ibm.box.com/v/MolecularTransformerData).
   The ZIP includes several variants; the documented benchmark uses non-augmented
   `MIT_separated` with 409,035 training, 30,000 validation, and 40,000 test rows.
-- [USPTO-SynPlanner](datasets/uspto_synplanner): reserved for future data; no raw
-  files have been uploaded yet.
+- [USPTO-SynPlanner](datasets/uspto_synplan): `uspto_synplan_old` preserves
+  the original 973,640 reactions; `uspto_synplan_final` contains the 851,479
+  original records supporting templates retained after extraction filtering and
+  popularity ≥3. The dataset README documents their exact relationship.
 
 ## Models
 

datasets/README.md

@@ -1,16 +1,19 @@
-# Local dataset conventions
+# Datasets and preparation conventions
 
 The code repository tracks acquisition notes, file identities, and import settings.
 The [Hugging Face artifact repository](https://huggingface.co/Manajit/SynplannerChemformer/tree/main/datasets)
-mirrors the corresponding raw data paths:
+contains these raw data paths:
 
 - `absynth/raw/absynth_full_filtered_classed.csv`: the supplied AbSynth export.
 - `uspto_mit/raw/molecular_transformer_data.zip`: the original archive from
   [MolecularTransformerData](https://ibm.box.com/v/MolecularTransformerData).
-- `uspto_synplanner/`: documentation and settings only; raw data will be added later.
+- [uspto_synplan/](uspto_synplan): `uspto_synplan_old` archives the original
+  973,640 reactions and historical metadata; `uspto_synplan_final` contains
+  851,479 source reactions retained after SynPlanner extraction filtering and
+  template popularity ≥3. See its README for selection details and counts.
 
 `artifacts.json` records sizes, SHA-256 hashes, and known acquisition sources.
-`uspto_mit` supports existing Chemformer pickles. The `uspto_synplanner` and
+`uspto_mit` supports existing Chemformer pickles. The historical `uspto_synplanner` and
 `absynth` import recipes still need source-specific validation. No downloads happen
 implicitly. The raw data remains ignored in the code repository.
 

datasets/artifacts.json

@@ -11,6 +11,18 @@
       "bytes": 276751431,
       "sha256": "1724f48b07e1f8d3c7267ab50913c6e684750850a16fbb417961e8130350d0f7",
       "source_url": "https://ibm.box.com/v/MolecularTransformerData"
+    },
+    {
+      "path": "datasets/uspto_synplan/uspto_synplan_old/uspto_synplan.smi.zip",
+      "bytes": 78640103,
+      "sha256": "8d567a31386327302b8f3dd861bb322a9d4d0e8e1e78cf17853278d4d285888d",
+      "provenance": "Original 973,640-reaction archive; moved unchanged from datasets/uspto_synplanner at c612c234ed708c164bf058cd0367d98c804460c5"
+    },
+    {
+      "path": "datasets/uspto_synplan/uspto_synplan_final/passed_reactions.smi.zip",
+      "bytes": 67627732,
+      "sha256": "cd2192c45d402ed9d7343e07dea6c2631b4a818fe653eaa07dd861af975b06ff",
+      "provenance": "851,479 original source records supporting 11,424 retained SynPlanner templates with popularity >=3; see datasets/uspto_synplan/README.md"
     }
   ]
 }

datasets/uspto_synplan/README.md

@@ -0,0 +1,85 @@
+# USPTO-SynPlanner: original and extraction-retained reactions
+
+These are two versions of the same reaction corpus. The final version is a
+subset of the old version selected using the saved SynPlanner extraction results.
+
+| Version | Reaction records | File | Meaning |
+| --- | ---: | --- | --- |
+| [uspto_synplan_old](uspto_synplan_old) | 973,640 | `uspto_synplan.smi.zip` → `uspto_synplan.smi` | Original filtered corpus, before rule-extraction filtering and the template popularity cutoff |
+| [uspto_synplan_final](uspto_synplan_final) | 851,479 | `passed_reactions.smi.zip` → `passed_reactions.smi` | Reactions supporting templates retained after extraction filtering with popularity **≥3** |
+
+## How the versions are related
+
+The old archive is preserved byte for byte from
+`datasets/uspto_synplanner/uspto_synplan.smi.zip` at Hub revision
+`c612c234ed708c164bf058cd0367d98c804460c5`. Its decompressed file is byte-identical
+to the extraction input, `03_filtration/all/unfailed_filtered.smi`.
+The original `README.md` and `dataset.yaml` are also archived unchanged in
+`uspto_synplan_old`; they are historical placeholder import metadata, not a
+validated recipe for loading this SMILES file.
+
+SynPlanner was run with `04_extraction/all/pop_3_new/rules_extraction.yaml` and
+wrote `uspto_rules.tsv`, `uspto_rules.audit.tsv`, and `uspto_rules_policy_data.tsv`.
+The extraction configuration is preserved in
+[uspto_synplan_final/rules_extraction.yaml](uspto_synplan_final/rules_extraction.yaml).
+It uses `min_popularity: 3`, which means an inclusive **≥3**, not strictly >3.
+Other settings include single-product reactions, no multicenter rules, ignored
+stereochemistry during extraction, and a one-atom rule environment.
+
+The final file was recovered without rerunning extraction: take the union of
+`reaction_indices` from all 11,424 retained templates in `uspto_rules.tsv`, then
+copy the corresponding original input records in their original order. Indices
+are zero-based over non-empty input records. The union contains exactly 851,479
+distinct source indices. For this run, each retained reaction has one recorded
+retained-template association.
+
+The output preserves the original complete reaction lines, including atom maps,
+CXSMILES extensions, source identifiers and other metadata. No canonicalization,
+deduplication, new atom mapping, stereochemistry removal or train/validation/test
+split was performed during recovery. Duplicate chemistry on distinct input rows
+is not collapsed. The extraction's `ignore_stereo` setting concerns templates;
+it does not strip stereochemistry from these copied source records.
+
+## Accounting for the excluded reactions
+
+| Outcome | Reaction records |
+| --- | ---: |
+| Retained after all filters, template popularity ≥3 | 851,479 |
+| Below minimum template popularity | 29,475 |
+| Multiple reaction centers | 51,455 |
+| Reactor validation failed | 26,511 |
+| Multiple products | 14,126 |
+| Standardization errors | 587 |
+| No rules extracted | 7 |
+| **Total original input** | **973,640** |
+
+All 122,161 excluded input records appear in the saved audit. Their indices are
+disjoint from the retained indices, and together these sets cover the complete
+input. The audit's original reaction strings were checked against the input.
+The generated final file was also verified byte for byte against the complement
+of the audit, using SHA-256.
+
+Template counts and reaction counts are different: the run reports 65,531 unique
+extracted templates, of which 29,214 failed reactor validation, 24,893 fell below
+minimum popularity and 11,424 were retained. The 24,893 low-popularity templates
+account for 29,475 rejected reaction records, not 24,893 reactions.
+
+## Integrity and use
+
+Decompressed file SHA-256 values:
+
+| File | Bytes | SHA-256 |
+| --- | ---: | --- |
+| `uspto_synplan_old/uspto_synplan.smi` | 432,334,623 | `fe00835cfb2d5938701ac79666d209bc2fde7f395e221b7e0b301f36491b9079` |
+| `uspto_synplan_final/passed_reactions.smi` | 373,856,455 | `b3a4482563b23fad070845fbfe6f4b7782a0773c4ace434a2fa697cf65dd7784` |
+
+Archive hashes and sizes are listed in [../artifacts.json](../artifacts.json).
+The final version also includes [provenance.json](uspto_synplan_final/provenance.json)
+with identities of the source, extraction outputs and recovered data.
+
+Use `uspto_synplan_final` for experiments intended to match this retained
+SynPlanner template vocabulary. Use `uspto_synplan_old` to reproduce the broader
+pre-extraction corpus. Neither file supplies a train/validation/test split;
+Chemformer preprocessing and split generation are separate steps. Dataset usage
+and citation terms remain those of the original sources; this reorganization
+does not assign a new dataset license or change model checkpoints.

datasets/uspto_synplan/uspto_synplan_final/README.md

@@ -0,0 +1,17 @@
+# USPTO-SynPlanner final: 851,479 reactions
+
+`passed_reactions.smi.zip` contains `passed_reactions.smi`: the original source
+records supporting the 11,424 templates retained by the saved SynPlanner run,
+with **template popularity ≥3**. This is a subset of the 973,640-reaction corpus
+archived in [../uspto_synplan_old](../uspto_synplan_old).
+
+See the [parent README](../README.md) for the exact selection procedure, exclusion
+counts, original dataset location, and integrity checks. The original source
+record contents and order are preserved; no dataset split is included.
+
+- `rules_extraction.yaml`: the extraction configuration used by the original run.
+- `provenance.json`: source and extraction-output identities, counts, and hashes.
+
+Unzip the archive and pass the `.smi` file explicitly to the Chemformer
+SynPlanner preparation workflow. Keep generated representations and splits
+separate from this recovered source dataset.

datasets/uspto_synplan/uspto_synplan_final/passed_reactions.smi.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd2192c45d402ed9d7343e07dea6c2631b4a818fe653eaa07dd861af975b06ff
+size 67627732

datasets/uspto_synplan/uspto_synplan_final/provenance.json

@@ -0,0 +1,64 @@
+{
+  "dataset": "uspto_synplan_final",
+  "parent": {
+    "path": "datasets/uspto_synplan/uspto_synplan_old/uspto_synplan.smi.zip",
+    "original_hub_path": "datasets/uspto_synplanner/uspto_synplan.smi.zip",
+    "original_hub_revision": "c612c234ed708c164bf058cd0367d98c804460c5",
+    "decompressed": {
+      "member": "uspto_synplan.smi",
+      "sha256": "fe00835cfb2d5938701ac79666d209bc2fde7f395e221b7e0b301f36491b9079",
+      "records": 973640,
+      "bytes": 432334623
+    }
+  },
+  "output": {
+    "path": "datasets/uspto_synplan/uspto_synplan_final/passed_reactions.smi.zip",
+    "archive_sha256": "cd2192c45d402ed9d7343e07dea6c2631b4a818fe653eaa07dd861af975b06ff",
+    "archive_bytes": 67627732,
+    "member": "passed_reactions.smi",
+    "sha256": "b3a4482563b23fad070845fbfe6f4b7782a0773c4ace434a2fa697cf65dd7784",
+    "records": 851479,
+    "bytes": 373856455
+  },
+  "selection": {
+    "min_popularity_inclusive": 3,
+    "retained_templates": 11424,
+    "index_base": 0,
+    "index_unit": "non-empty input record",
+    "method": "Union reaction_indices in retained uspto_rules.tsv; copy original records in input order",
+    "preserves_record_bytes": true,
+    "deduplicates_chemistry": false,
+    "splits_assigned": false,
+    "extraction_rerun": false
+  },
+  "excluded_reactions": {
+    "BelowMinPopularity": 29475,
+    "MultiCenter": 51455,
+    "ReactorValidationFailed": 26511,
+    "SkippedMultiProduct": 14126,
+    "StandardizationError": 587,
+    "NoRulesExtracted": 7
+  },
+  "extraction_artifacts": [
+    {
+      "name": "rules_extraction.yaml",
+      "sha256": "1a5a48d16d29e8b2f9e9280a03b8e2f73de144c49d61d5ec77d54b86022ab36c",
+      "bytes": 472
+    },
+    {
+      "name": "uspto_rules.tsv",
+      "sha256": "8c6deb65c823fb5d997a202974b0c48e826524027694ac7409dcfbbc0db98932",
+      "bytes": 7113184
+    },
+    {
+      "name": "uspto_rules.audit.tsv",
+      "sha256": "a8a22e0c027a4c6220dd7d841fbc7e7cae0b64e27b393e7f688300949078170e",
+      "bytes": 73617742
+    },
+    {
+      "name": "uspto_rules_policy_data.tsv",
+      "sha256": "9fa30011478ffb57287a9d87f7c56cfd61a6fe23e4e66f6621d6935c56b908b5",
+      "bytes": 42588725
+    }
+  ]
+}

datasets/uspto_synplan/uspto_synplan_final/rules_extraction.yaml

@@ -0,0 +1,20 @@
+min_popularity: 3
+single_product_only: True
+ignore_stereo: True
+worker_timeout_per_reaction: 10.0
+environment_atom_count: 1
+multicenter_rules: False
+include_rings: False
+include_func_groups: False
+keep_leaving_groups: True
+keep_incoming_groups: False
+keep_reagents: False
+atom_info_retention:
+  reaction_center:
+    neighbors: False
+    implicit_hydrogens: False
+    ring_sizes: False
+  environment:
+    neighbors: False
+    implicit_hydrogens: False
+    ring_sizes: False