README.md

# wikiparse 2606 data

Large-file delivery for the `wikiparse 2606` technical milestone, derived from English Wikipedia
2026-06-01.

The versioned [`robbd/wikiparse-2606`](https://huggingface.co/datasets/robbd/wikiparse-2606)
dataset is the catalog: ten compact Parquet tables, checksums, schemas, provenance, and five small
analysis cards. This bucket carries the larger context surfaces.

##### Contents
| Path                  |      Rows | Contents                                                                             |
| --------------------- | --------: | ------------------------------------------------------------------------------------ |
| `context/index/v1/`   | 7,073,577 | Small article index: source key, categories, detector signals, priority, and reason. |
| `context/content/v1/` | 7,073,577 | Exact retained article Markdown plus the same source-bound assignment fields.        |

The content release is partitioned into 1,003 disjoint 256-stream shards, with one shorter terminal
shard. Each shard has one Parquet file per priority and an exact manifest; the root manifest and
receipt bind every shard and file.

##### Priorities
| Priority |  Articles | Intended use                                                                     |
| -------- | --------: | -------------------------------------------------------------------------------- |
| `p0`     |   908,270 | Science closure and taxa; first-pass research context.                           |
| `p1`     | 4,665,480 | General retained articles.                                                       |
| `p2`     | 1,208,553 | Useful secondary context: settlements, films, music releases, and sports people. |
| `p3`     |   291,274 | Serial or incidental context: lists, sports seasons, and disambiguation pages.   |

These labels are mutually exclusive delivery hints. They are not quality judgments, exclusion
authority, or a claim that `p2`/`p3` are useless. Categories and detector signals remain attached so
consumers can write a different policy.

##### Download
```console
hf buckets sync \
  hf://buckets/robbd/wikiparse-2606-data/context/index/v1 \
  ./wikiparse-2606-index

hf buckets sync \
  hf://buckets/robbd/wikiparse-2606-data/context/content/v1 \
  ./wikiparse-2606-content

# Or fetch only the science-first content slice and its receipts.
hf buckets sync \
  hf://buckets/robbd/wikiparse-2606-data/context/content/v1 \
  ./wikiparse-2606-p0 \
  --include "shards/*/p0.parquet" \
  --include "shards/*/manifest.json" \
  --include "manifest.json" \
  --include "receipt.json"
```

The context files copy retained Markdown from the existing compact corpus. They do not rerun the
parser, call a model, or alter the source store.

##### Boundary
- Technical milestone only; announcement and outreach remain held.
- Citation records are article-level and deduplicated, not inline claim markers.
- Parser-quality and detector fields are operational evidence, not adjudicated truth.
- No model weights are published here.
- Data is distributed under CC BY-SA 4.0; attribution and modification notes are in the versioned
  dataset.