Archive original USPTO-SynPlanner data and add extraction-retained final dataset

Version USPTO-SynPlanner data under datasets/uspto_synplan/:

- Move the existing archive, README and import metadata unchanged into uspto_synplan_old (973,640 reactions).
- Add uspto_synplan_final/passed_reactions.smi.zip (851,479 original records supporting 11,424 retained templates; inclusive popularity >=3), the extraction config and a provenance manifest.
- Explain the exact subset relationship, all 122,161 exclusions, record preservation and checksums in the dataset README; update the root/index READMEs and artifact manifest.

Validation: decompressed HF old archive SHA-256 matches the original extraction input; retained rule indices and audit entries partition all 973,640 records; final record bytes/order match the selected input and independently the audit complement. The final ZIP was read back and checked against the verified recovered SMILES hash. All three archived old files retain their original bytes. Documentation links and published archive hashes were checked locally. Model artifacts and other datasets are unchanged.


Submitted as a pull request because this account cannot write main directly.

README.md

@@ -21,9 +21,17 @@ datasets/
     README.md
     dataset.yaml
     raw/molecular_transformer_data.zip
-  uspto_synplanner/
+  uspto_synplan/
     README.md
-    dataset.yaml
+    uspto_synplan_old/
+      README.md
+      dataset.yaml
+      uspto_synplan.smi.zip
+    uspto_synplan_final/
+      README.md
+      passed_reactions.smi.zip
+      rules_extraction.yaml
+      provenance.json
 models/
   README.md
   uspto_mit/README.md
@@ -40,7 +48,7 @@ models/
 
 ## Datasets
 
-Raw files use the same relative paths and bytes as the local project.
+Raw source artifacts and their dataset versions are documented below.
 [datasets/artifacts.json](datasets/artifacts.json) records their sizes and SHA-256
 hashes. Prepared data, generated outputs, and machine-specific manifests are not
 published here.
@@ -51,8 +59,10 @@ published here.
   [MolecularTransformerData source](https://ibm.box.com/v/MolecularTransformerData).
   The ZIP includes several variants; the documented benchmark uses non-augmented
   `MIT_separated` with 409,035 training, 30,000 validation, and 40,000 test rows.
-- [USPTO-SynPlanner](datasets/uspto_synplanner): reserved for future data; no raw
-  files have been uploaded yet.
+- [USPTO-SynPlanner](datasets/uspto_synplan): `uspto_synplan_old` preserves
+  the original 973,640 reactions; `uspto_synplan_final` contains the 851,479
+  original records supporting templates retained after extraction filtering and
+  popularity ≥3. The dataset README documents their exact relationship.
 
 ## Models
 

datasets/README.md

@@ -1,16 +1,19 @@
-# Local dataset conventions
+# Datasets and preparation conventions
 
 The code repository tracks acquisition notes, file identities, and import settings.
 The [Hugging Face artifact repository](https://huggingface.co/Manajit/SynplannerChemformer/tree/main/datasets)
-mirrors the corresponding raw data paths:
+contains these raw data paths:
 
 - `absynth/raw/absynth_full_filtered_classed.csv`: the supplied AbSynth export.
 - `uspto_mit/raw/molecular_transformer_data.zip`: the original archive from
   [MolecularTransformerData](https://ibm.box.com/v/MolecularTransformerData).
-- `uspto_synplanner/`: documentation and settings only; raw data will be added later.
+- [uspto_synplan/](uspto_synplan): `uspto_synplan_old` archives the original
+  973,640 reactions and historical metadata; `uspto_synplan_final` contains
+  851,479 source reactions retained after SynPlanner extraction filtering and
+  template popularity ≥3. See its README for selection details and counts.
 
 `artifacts.json` records sizes, SHA-256 hashes, and known acquisition sources.
-`uspto_mit` supports existing Chemformer pickles. The `uspto_synplanner` and
+`uspto_mit` supports existing Chemformer pickles. The historical `uspto_synplanner` and
 `absynth` import recipes still need source-specific validation. No downloads happen
 implicitly. The raw data remains ignored in the code repository.
 

datasets/artifacts.json

@@ -11,6 +11,18 @@
       "bytes": 276751431,
       "sha256": "1724f48b07e1f8d3c7267ab50913c6e684750850a16fbb417961e8130350d0f7",
       "source_url": "https://ibm.box.com/v/MolecularTransformerData"
+    },
+    {
+      "path": "datasets/uspto_synplan/uspto_synplan_old/uspto_synplan.smi.zip",
+      "bytes": 78640103,
+      "sha256": "8d567a31386327302b8f3dd861bb322a9d4d0e8e1e78cf17853278d4d285888d",
+      "provenance": "Original 973,640-reaction archive; moved unchanged from datasets/uspto_synplanner at c612c234ed708c164bf058cd0367d98c804460c5"
+    },
+    {
+      "path": "datasets/uspto_synplan/uspto_synplan_final/passed_reactions.smi.zip",
+      "bytes": 67627732,
+      "sha256": "cd2192c45d402ed9d7343e07dea6c2631b4a818fe653eaa07dd861af975b06ff",
+      "provenance": "851,479 original source records supporting 11,424 retained SynPlanner templates with popularity >=3; see datasets/uspto_synplan/README.md"
     }
   ]
 }

datasets/uspto_synplan/README.md

@@ -0,0 +1,85 @@
+# USPTO-SynPlanner: original and extraction-retained reactions
+
+These are two versions of the same reaction corpus. The final version is a
+subset of the old version selected using the saved SynPlanner extraction results.
+
+| Version | Reaction records | File | Meaning |
+| --- | ---: | --- | --- |
+| [uspto_synplan_old](uspto_synplan_old) | 973,640 | `uspto_synplan.smi.zip` → `uspto_synplan.smi` | Original filtered corpus, before rule-extraction filtering and the template popularity cutoff |
+| [uspto_synplan_final](uspto_synplan_final) | 851,479 | `passed_reactions.smi.zip` → `passed_reactions.smi` | Reactions supporting templates retained after extraction filtering with popularity **≥3** |
+
+## How the versions are related
+
+The old archive is preserved byte for byte from
+`datasets/uspto_synplanner/uspto_synplan.smi.zip` at Hub revision
+`c612c234ed708c164bf058cd0367d98c804460c5`. Its decompressed file is byte-identical
+to the extraction input, `03_filtration/all/unfailed_filtered.smi`.
+The original `README.md` and `dataset.yaml` are also archived unchanged in
+`uspto_synplan_old`; they are historical placeholder import metadata, not a
+validated recipe for loading this SMILES file.
+
+SynPlanner was run with `04_extraction/all/pop_3_new/rules_extraction.yaml` and
+wrote `uspto_rules.tsv`, `uspto_rules.audit.tsv`, and `uspto_rules_policy_data.tsv`.
+The extraction configuration is preserved in
+[uspto_synplan_final/rules_extraction.yaml](uspto_synplan_final/rules_extraction.yaml).
+It uses `min_popularity: 3`, which means an inclusive **≥3**, not strictly >3.
+Other settings include single-product reactions, no multicenter rules, ignored
+stereochemistry during extraction, and a one-atom rule environment.
+
+The final file was recovered without rerunning extraction: take the union of
+`reaction_indices` from all 11,424 retained templates in `uspto_rules.tsv`, then
+copy the corresponding original input records in their original order. Indices
+are zero-based over non-empty input records. The union contains exactly 851,479
+distinct source indices. For this run, each retained reaction has one recorded
+retained-template association.
+
+The output preserves the original complete reaction lines, including atom maps,
+CXSMILES extensions, source identifiers and other metadata. No canonicalization,
+deduplication, new atom mapping, stereochemistry removal or train/validation/test
+split was performed during recovery. Duplicate chemistry on distinct input rows
+is not collapsed. The extraction's `ignore_stereo` setting concerns templates;
+it does not strip stereochemistry from these copied source records.
+
+## Accounting for the excluded reactions
+
+| Outcome | Reaction records |
+| --- | ---: |
+| Retained after all filters, template popularity ≥3 | 851,479 |
+| Below minimum template popularity | 29,475 |
+| Multiple reaction centers | 51,455 |
+| Reactor validation failed | 26,511 |
+| Multiple products | 14,126 |
+| Standardization errors | 587 |
+| No rules extracted | 7 |
+| **Total original input** | **973,640** |
+
+All 122,161 excluded input records appear in the saved audit. Their indices are
+disjoint from the retained indices, and together these sets cover the complete
+input. The audit's original reaction strings were checked against the input.
+The generated final file was also verified byte for byte against the complement
+of the audit, using SHA-256.
+
+Template counts and reaction counts are different: the run reports 65,531 unique
+extracted templates, of which 29,214 failed reactor validation, 24,893 fell below
+minimum popularity and 11,424 were retained. The 24,893 low-popularity templates
+account for 29,475 rejected reaction records, not 24,893 reactions.
+
+## Integrity and use
+
+Decompressed file SHA-256 values:
+
+| File | Bytes | SHA-256 |
+| --- | ---: | --- |
+| `uspto_synplan_old/uspto_synplan.smi` | 432,334,623 | `fe00835cfb2d5938701ac79666d209bc2fde7f395e221b7e0b301f36491b9079` |
+| `uspto_synplan_final/passed_reactions.smi` | 373,856,455 | `b3a4482563b23fad070845fbfe6f4b7782a0773c4ace434a2fa697cf65dd7784` |
+
+Archive hashes and sizes are listed in [../artifacts.json](../artifacts.json).
+The final version also includes [provenance.json](uspto_synplan_final/provenance.json)
+with identities of the source, extraction outputs and recovered data.
+
+Use `uspto_synplan_final` for experiments intended to match this retained
+SynPlanner template vocabulary. Use `uspto_synplan_old` to reproduce the broader
+pre-extraction corpus. Neither file supplies a train/validation/test split;
+Chemformer preprocessing and split generation are separate steps. Dataset usage
+and citation terms remain those of the original sources; this reorganization
+does not assign a new dataset license or change model checkpoints.

datasets/uspto_synplan/uspto_synplan_final/README.md

@@ -0,0 +1,17 @@
+# USPTO-SynPlanner final: 851,479 reactions
+
+`passed_reactions.smi.zip` contains `passed_reactions.smi`: the original source
+records supporting the 11,424 templates retained by the saved SynPlanner run,
+with **template popularity ≥3**. This is a subset of the 973,640-reaction corpus
+archived in [../uspto_synplan_old](../uspto_synplan_old).
+
+See the [parent README](../README.md) for the exact selection procedure, exclusion
+counts, original dataset location, and integrity checks. The original source
+record contents and order are preserved; no dataset split is included.
+
+- `rules_extraction.yaml`: the extraction configuration used by the original run.
+- `provenance.json`: source and extraction-output identities, counts, and hashes.
+
+Unzip the archive and pass the `.smi` file explicitly to the Chemformer
+SynPlanner preparation workflow. Keep generated representations and splits
+separate from this recovered source dataset.

datasets/uspto_synplan/uspto_synplan_final/passed_reactions.smi.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd2192c45d402ed9d7343e07dea6c2631b4a818fe653eaa07dd861af975b06ff
+size 67627732

datasets/uspto_synplan/uspto_synplan_final/provenance.json

@@ -0,0 +1,64 @@
+{
+  "dataset": "uspto_synplan_final",
+  "parent": {
+    "path": "datasets/uspto_synplan/uspto_synplan_old/uspto_synplan.smi.zip",
+    "original_hub_path": "datasets/uspto_synplanner/uspto_synplan.smi.zip",
+    "original_hub_revision": "c612c234ed708c164bf058cd0367d98c804460c5",
+    "decompressed": {
+      "member": "uspto_synplan.smi",
+      "sha256": "fe00835cfb2d5938701ac79666d209bc2fde7f395e221b7e0b301f36491b9079",
+      "records": 973640,
+      "bytes": 432334623
+    }
+  },
+  "output": {
+    "path": "datasets/uspto_synplan/uspto_synplan_final/passed_reactions.smi.zip",
+    "archive_sha256": "cd2192c45d402ed9d7343e07dea6c2631b4a818fe653eaa07dd861af975b06ff",
+    "archive_bytes": 67627732,
+    "member": "passed_reactions.smi",
+    "sha256": "b3a4482563b23fad070845fbfe6f4b7782a0773c4ace434a2fa697cf65dd7784",
+    "records": 851479,
+    "bytes": 373856455
+  },
+  "selection": {
+    "min_popularity_inclusive": 3,
+    "retained_templates": 11424,
+    "index_base": 0,
+    "index_unit": "non-empty input record",
+    "method": "Union reaction_indices in retained uspto_rules.tsv; copy original records in input order",
+    "preserves_record_bytes": true,
+    "deduplicates_chemistry": false,
+    "splits_assigned": false,
+    "extraction_rerun": false
+  },
+  "excluded_reactions": {
+    "BelowMinPopularity": 29475,
+    "MultiCenter": 51455,
+    "ReactorValidationFailed": 26511,
+    "SkippedMultiProduct": 14126,
+    "StandardizationError": 587,
+    "NoRulesExtracted": 7
+  },
+  "extraction_artifacts": [
+    {
+      "name": "rules_extraction.yaml",
+      "sha256": "1a5a48d16d29e8b2f9e9280a03b8e2f73de144c49d61d5ec77d54b86022ab36c",
+      "bytes": 472
+    },
+    {
+      "name": "uspto_rules.tsv",
+      "sha256": "8c6deb65c823fb5d997a202974b0c48e826524027694ac7409dcfbbc0db98932",
+      "bytes": 7113184
+    },
+    {
+      "name": "uspto_rules.audit.tsv",
+      "sha256": "a8a22e0c027a4c6220dd7d841fbc7e7cae0b64e27b393e7f688300949078170e",
+      "bytes": 73617742
+    },
+    {
+      "name": "uspto_rules_policy_data.tsv",
+      "sha256": "9fa30011478ffb57287a9d87f7c56cfd61a6fe23e4e66f6621d6935c56b908b5",
+      "bytes": 42588725
+    }
+  ]
+}

datasets/uspto_synplan/uspto_synplan_final/rules_extraction.yaml

@@ -0,0 +1,20 @@
+min_popularity: 3
+single_product_only: True
+ignore_stereo: True
+worker_timeout_per_reaction: 10.0
+environment_atom_count: 1
+multicenter_rules: False
+include_rings: False
+include_func_groups: False
+keep_leaving_groups: True
+keep_incoming_groups: False
+keep_reagents: False
+atom_info_retention:
+  reaction_center:
+    neighbors: False
+    implicit_hydrogens: False
+    ring_sizes: False
+  environment:
+    neighbors: False
+    implicit_hydrogens: False
+    ring_sizes: False