social-data-attribution-w2/docs/DATA_INVENTORY.md

# Data Inventory: Dolma 6T Corpus and Working Samples

This document describes where all data artifacts live. **HuggingFace under `HCAI-Lab/` is the primary surface** for everything except the raw 6T `.jsonl.zst` source corpus (R2-only, ~5 TB, intentional). Modal volumes are transient caches that mirror what is already on R2/HF. PACE storage no longer holds anything sole-source — see `PRESERVATION_GAP_REPORT.md` for the verification audit.

It covers the deduplicated corpus, quality labels, manifests, materialized working samples, and the TrackStar attribution artifacts.

## Source corpus

The source data is the Dolma 3 corpus, deduplicated to ~6T tokens across ~58K shards.

### Original upstream

| Property | Value |
|----------|-------|
| Pool dataset | `allenai/dolma3_pool` (~9T tokens, 2732 shards) |
| Curated mix | `allenai/dolma3_mix-6T-1025-7B` |
| 150B sample | `allenai/dolma3_mix-150B-1025` |

### Deduplicated corpus on R2

The deduplicated corpus lives on Cloudflare R2. It was produced in two phases (SOC-127).

| Property | Value |
|----------|-------|
| R2 bucket | `soc127-dedup` |
| R2 endpoint | `https://0934ab8e84ac8f4e81decaf3eb121337.r2.cloudflarestorage.com` |
| Total shards | ~58,621 |
| Shard format | `.jsonl.zst` (zstandard-compressed JSONL) |
| Shard size | ~50-100 MB each |
| Total size | ~5 TB |

Source shard prefixes:

```
soc127/phase1_pool_shared/      # pool data (common_crawl, olmocr, etc.)
soc127/phase2_nonpool_final/    # non-pool sources
```

Each shard contains Dolma documents with fields: `id`, `text`, `metadata`.

### Quality labels on R2 and on HF

WebOrganizer topic and format labels (SOC-91) are stored as parquet sidecars alongside the source shards. As of 2026-05-23 they are also mirrored to HF Datasets for HF-only workflows.

| Property | Value |
|----------|-------|
| R2 prefix | `soc91-labels/` (58,465 per-shard parquets + `.stats.json` sidecars) |
| HF Dataset | `HCAI-Lab/soc91-labels` (private; 2,719 parquet chunks, 169.94 GB; 58,465 source files concatenated into ~60 MB chunks; each row carries `source_shard_path`) |
| Format | `.parquet` per source shard |
| Labels | 24 topics x 24 formats (576 bins) |
| Stats on R2 | `soc91-stats/` prefix |
| Stats on HF | `HCAI-Lab/soc91-stats` (private; raw EDA/audit JSON outputs) |

Topic categories: `adult_content`, `art_and_design`, `crime_and_law`, `education_and_jobs`,
`electronics_and_hardware`, `entertainment`, `fashion_and_beauty`, `finance_and_business`,
`food_and_dining`, `games`, `health`, `history_and_geography`, `home_and_hobbies`,
`industrial`, `literature`, `politics`, `religion`, `science_math_and_technology`,
`social_life`, `software`, `software_development`, `sports_and_fitness`, `transportation`,
`travel_and_tourism`.

Format categories: `academic_writing`, `content_listing`, `creative_writing`,
`customer_support`, `comment_section`, `faq`, `truncated`, `knowledge_article`,
`legal_notices`, `listicle`, `news_article`, `nonfiction_writing`, `about_org`,
`news_org`, `about_pers`, `personal_blog`, `product_page`, `q_a_forum`, `spam_ads`,
`structured_data`, `documentation`, `audio_transcript`, `tutorial`, `user_review`.

### Quality sidecars on R2 and on HF

Per-document quality classifier scores (SOC-139) — separate from the WebOrganizer
topic/format labels above. Includes the SOC-142 inverted-label fix (commit `3342baf`).

| Property | Value |
|----------|-------|
| R2 prefix | `soc139-quality-sidecars/` (58,621 per-shard parquets + `.label-fix.done` + `.stats.json`) |
| HF Dataset | `HCAI-Lab/soc139-quality-sidecars` (private; 1.26B rows, 41.8 GB, 80 parquet chunks) |
| HF columns | `doc_id`, `quality_label_id`, `quality_score`, `quality_high_prob`, `quality_low_prob`, `quality_confidence`, `source_shard_path` |

### Unified corpus manifest on R2 and on HF

The full corpus manifest (SOC-95) contains one row per document with doc_id, token count,
shard path, and WebOrganizer labels. This is the index used to draw stratified samples.

| Property | Value |
|----------|-------|
| R2 prefix | `soc95-manifest/data/` (per-shard parquets) |
| HF Dataset (primary) | `HCAI-Lab/dolma3-olmo3-corpus-manifest` (1.1B rows, 32-col schema with topic+format+quality+token+shard) |
| HF Dataset (alt) | `HCAI-Lab/dolma3-6t-corpus-manifest` |
| Format | Parquet files (~58K files, one per shard) |
| Columns | `doc_id`, `token_count`, `shard_path`, `weborganizer_topic`, `weborganizer_format` (plus quality fields in the unified HF version) |
| Total size | ~213 GB |

## Modal persistent volumes (transient caches)

The following Modal volumes cache data for fast access from Modal workers.
These are in the `eilab-gt` Modal workspace and mirror R2/HF sources — none is sole-source.
Safe to delete if Modal compute is no longer needed.

| Volume | Contents | Size |
|--------|----------|------|
| `soc134-corpus-cache` | All ~58K source shards from R2 (mirrors R2 key structure) | ~5 TB |
| `soc134-manifest-cache` | All ~58K manifest parquet files from R2 | ~213 GB |
| `soc134-output-sample-500-docs` | Materialized docs for sample_500_docs | varies |
| `soc134-output-sample-1000-docs` | Materialized docs for sample_1000_docs | varies |
| `soc134-output-sample-5000-docs` | Materialized docs for sample_5000_docs | varies |
| `soc134-output-sample-10000-docs` | Materialized docs for sample_10000_docs | varies |
| `soc134-output-dolma3-6t-preconditioner-100k` | Materialized docs for preconditioner_100k (uniform random) | varies |

### Corpus cache volume layout

```
soc134-corpus-cache/
  soc127/phase1_pool_shared/common_crawl/part_000/data__...__shard_00000001.jsonl.zst
  soc127/phase1_pool_shared/common_crawl/part_000/data__...__shard_00000002.jsonl.zst
  soc127/phase2_nonpool_final/...
  ...
```

Preserves the R2 key structure. `shard_path` values from manifests resolve directly as paths.

### Output volume layout

Each per-sample output volume has the same structure:

```
soc134-output-sample-{N}-docs/
  {run_id}/
    manifest.parquet
    sample_contract.json
    bin_summary.csv
    worker_0000/
      soc127__phase1_pool_shared__...jsonl.zst        # materialized doc subset
      soc127__phase1_pool_shared__...jsonl.zst.stats.json
      soc127__phase1_pool_shared__...jsonl.zst.done
    worker_0001/
      ...
    worker_0127/
      ...
    .hf_upload/
      metadata.done
      batch_0000.done
      batch_0001.done
      ...
```

Run IDs for current materialization:

| Sample | Run ID |
|--------|--------|
| sample_500_docs | `soc134_materialize_20260320_150348` |
| sample_1000_docs | `soc134_materialize_20260320_151802` |
| sample_5000_docs | `soc134_materialize_20260320_152205` |
| sample_10000_docs | `soc134_materialize_20260320_152236` |

## HuggingFace datasets

Working samples are published to HuggingFace under the `HCAI-Lab` organization. All six stratified samples are complete; each is published as both an HF Dataset (viewer + `snapshot_download`) and an HF Bucket of the same name (S3-style access via `hf buckets sync`); bit-identical data on both surfaces.

| HF repo | Sample | Docs | Tokens | Underfilled bins | Status |
|---------|--------|------|--------|------------------|--------|
| `HCAI-Lab/dolma3-6t-sample-500-docs` | 500 docs/bin | 287,936 | 538.8M | 1 | Complete (43,129 files) |
| `HCAI-Lab/dolma3-6t-sample-1000-docs` | 1,000 docs/bin | 575,187 | 1.08B | 3 | Complete (46,808 files) |
| `HCAI-Lab/dolma3-6t-sample-5000-docs` | 5,000 docs/bin | 2,855,446 | 5.30B | 7 | Complete (56,043 files) |
| `HCAI-Lab/dolma3-6t-sample-10000-docs` | 10,000 docs/bin | 5,678,621 | 10.5B | 17 | Complete (57,845 files) |
| `HCAI-Lab/dolma3-6t-sample-50000-docs` | 50,000 docs/bin | 26,249,124 | 62.8B | 79 | Complete (58,246 files) |
| `HCAI-Lab/dolma3-6t-sample-100000-docs` | 100,000 docs/bin | 49,709,294 | 118.4B | 130 | Complete (58,264 files, dataset twin added 2026-05-25) |

Preconditioner sample (uniform random, no stratification):

| Location | Repo | Docs | Tokens | Min tokens | Files | Status |
|----------|------|------|--------|------------|-------|--------|
| HF Dataset | `HCAI-Lab/dolma3-6t-preconditioner-100k` | 100,000 | 251.5M | 512 | 38,277 | Complete |
| HF Bucket | `HCAI-Lab/dolma3-6t-preconditioner-100k` | 100,000 | 251.5M | 512 | 38,260 | Complete |

The HF Dataset repo is the primary access path. Use `snapshot_download` or `hf download` to fetch it.

### HF repo layout

```
HCAI-Lab/dolma3-6t-sample-{N}-docs/
  README.md
  sample_contract.json
  bin_summary.csv
  working_sample_manifest.parquet
  data/
    part_000/
      soc127__phase1_pool_shared__...jsonl.zst
      soc127__phase1_pool_shared__...jsonl.zst
      ...                                        # up to 5,000 files per part
    part_001/
      ...
    part_NNN/
      ...
```

Data files are partitioned into `data/part_NNN/` subdirectories with max 5,000 files each
(HuggingFace enforces a 10,000 files per directory limit).

### HF repo metadata files

- `working_sample_manifest.parquet`: One row per sampled document. Columns: `doc_id`,
  `token_count`, `shard_path`, `bin_id`, `bin_topic`, `bin_format`.
- `sample_contract.json`: Aggregate sampling parameters and realized counts.
- `bin_summary.csv`: Per-bin fill rates across 576 bins (24 topics x 24 formats).
- `README.md`: Data card with parameters, counts, and format description.

## Working sample design

Samples are drawn by stratified sampling across the 576 topic x format bins.
Each sample config specifies a number of documents per bin (`docs_per_bin`).
Within each bin, documents are selected by deterministic priority scoring
(blake2b hash of `doc_id:seed`), ensuring reproducibility.

| Sample | docs/bin | Total docs | Total tokens | Unique shards | Underfilled bins |
|--------|---------|-----------|-------------|--------------|-----------------|
| sample_500_docs | 500 | 287,936 | 538.8M | ~42,461 | 1 |
| sample_1000_docs | 1,000 | 575,187 | 1.08B | ~45,926 | 3 |
| sample_5000_docs | 5,000 | 2,855,446 | 5.30B | ~55,347 | 7 |
| sample_10000_docs | 10,000 | 5,678,621 | 10.5B | ~57,622 | 17 |
| sample_50000_docs | 50,000 | 26,249,124 | 62.8B | ~58,200 | 79 |
| sample_100000_docs | 100,000 | 49,709,294 | 118.4B | ~58,264 | 130 |

All samples use seed 42. Bins are "underfilled" when the source corpus has fewer documents
in that topic x format combination than the requested docs_per_bin.

### Sample manifest columns

| Column | Type | Description |
|--------|------|-------------|
| `doc_id` | string | Unique document identifier from the Dolma corpus |
| `token_count` | int | Token count for this document |
| `shard_path` | string | R2 key of the source shard containing this document |
| `bin_id` | int | 1-indexed bin number (1-576) |
| `bin_topic` | string | WebOrganizer topic label |
| `bin_format` | string | WebOrganizer format label |

## TrackStar attribution artifacts in cloud

All TrackStar artifacts produced by SOC-156, SOC-161, SOC-170, SOC-171 have been migrated to HuggingFace as part of the 2026-05-22 PACE-to-cloud handoff. See `PRESERVATION_GAP_REPORT.md` for the migration audit.

### Bergson training gradient index

The reusable per-document gradient index built with Bergson from `sample_10000_docs` (5.68M docs).

| Property | Value |
|----------|-------|
| HF Bucket | `HCAI-Lab/trackstar-gradient-index-base` (private) |
| Total size | 1.30 TB |
| Total files | 3,792 |
| Shards | 316 (`shard_0000` .. `shard_0315`) |
| Per-shard size | ~3.9 GB (`gradients.bin`) + metadata |
| Model | `allenai/Olmo-3-1025-7B` (base) |
| Projection dim | 16, fp32 |

Per-shard layout: `data.hf/` (HF Dataset with length/loss), `gradients.bin`, `info.json`, `index_config.json`, `normalizers.pth`, `preconditioners.pth`, `preconditioners_eigen.pth`, `preprocess_config.json`, `processor_config.json`, `total_processed.pt`.

### TrackStar training shards (positional doc IDs)

Plain JSONL with `{id, text}` mapping `shard_NNNN:INDEX` positional doc IDs (used in score matrices) back to source document UUIDs.

| Property | Value |
|----------|-------|
| HF Dataset | `HCAI-Lab/dolma3-6t-sample-10000-docs-trackstar-shards` (private) |
| Total size | 41 GB (HF accounting: 46 GB) |
| Files | 316 `shard_NNNN.jsonl` (~145 MB each, ~18K docs per shard) |

**Required** if you load any of the score matrices below — without these, positional IDs cannot be resolved to text.

### Query gradient indices

Per-benchmark query gradients, three model variants (`base`, `instruct_base`, `instruct_cot`).

| Property | Value |
|----------|-------|
| HF Bucket | `HCAI-Lab/trackstar-query-gradients-base` (private) |
| Total size | 17.9 GB |
| Files | 336 (across SOC-156, SOC-161, SOC-170, SOC-171 runs) |

Canonical SOC-156 builds live under `base/20260326T163642Z_1102443/queries_{gsm8k,mmlu_social_science,mmlu_stem,socialiqa}/`.

### Per-query score matrices

Influence(doc_i, query_j) for the four OLMES benchmarks. Multiple model and benchmark variants:

| HF Bucket | Run | Files | Size |
|-----------|-----|-------|------|
| `HCAI-Lab/trackstar-scores-base-olmes-4bench` | OLMo-3-7B base, 4 OLMES benchmarks | 2,532 | 396 GB |
| `HCAI-Lab/trackstar-scores-instruct-cot-olmes-4bench` | OLMo-3-7B instruct-cot, 4 OLMES benchmarks | 2,532 | 396 GB |
| `HCAI-Lab/trackstar-scores-base-bbh` | OLMo-3-7B base, BBH | 1,899 | 14 GB |
| `HCAI-Lab/trackstar-scores-instruct-base-bbh` | OLMo-3-7B instruct, BBH | 1,899 | 14 GB |
| `HCAI-Lab/trackstar-scores-base-gsm8k-arc` | OLMo-3-7B base, GSM8K+ARC | 1,899 | 79 GB |
| `HCAI-Lab/trackstar-scores-instruct-base-gsm8k-arc` | OLMo-3-7B instruct, GSM8K+ARC | 1,899 | 79 GB |
| `HCAI-Lab/trackstar-scores-instruct-cot-noprecond-mmlu-ss` | OLMo-3-7B instruct-cot, MMLU social science only, **no preconditioner** (ablation pair to soc161) | 633 | 70 GB |

Format: per-benchmark dir containing 316 `shard_NNNN.npy` + 316 `shard_NNNN_doc_ids.json` + 1 `query_ids.json` (633 files × N benchmarks per bucket). See `docs/TRACKSTAR_DATA_ARTIFACTS.md` §1 for details.

### Preconditioners

Pre-built TrackStar mixed preconditioners for three model variants.

| Property | Value |
|----------|-------|
| HF Bucket | `HCAI-Lab/trackstar-preconditioners` (public) |
| Total size | 885 MB |
| Files | 78 across `olmo-3-1025-7b/`, `olmo-3-7b-instruct/`, `olmo-3-7b-think/` |

**Note**: the preconditioner must match the model used to build the gradients (SOC-162 finding).

### Top-K influence results

Ranked top-K influence lists, much smaller than full score matrices.

| HF Dataset | Contents |
|------------|----------|
| `HCAI-Lab/dolma3-trackstar-influence-scores` (private) | `influence_scores_full.parquet` + `top2k_{gsm8k,mmlu_socsci,mmlu_stem,socialiqa}.{csv,parquet}` |
| `HCAI-Lab/trackstar-top2k-base-gsm8k-arc` (bucket) | SOC-171 top-2K for base variant |
| `HCAI-Lab/trackstar-top2k-instruct-base-gsm8k-arc` (bucket) | SOC-171 top-2K for instruct variant |

### Job archive (Tier 3)

Tarballs of SLURM logs, sbatch artifacts, dedup work state, SOC-149 / SOC-159 / SOC-167 outputs, miscellaneous analysis. For reproducibility audits, not for day-to-day use.

| HF Dataset | Size | Files |
|------------|------|-------|
| `HCAI-Lab/dolma3-attribution-job-archive` (private) | 12.87 GB | 17 (16 tarballs + README) |

### Data home

A consolidated lookup for "where do I find X?" — README + machine-readable JSON inventory.

| HF Dataset | Mirror in repo |
|------------|----------------|
| `HCAI-Lab/dolma3-data-attribution-index` (private) | `docs/data_home/` |

Also published as an HF Collection: "Dolma3 Data Attribution" under `HCAI-Lab/`.

## Local repo data

Sample manifests and contracts are checked into the repo:

```
data/samples/
  sample_500_docs/
    working_sample_manifest.parquet    (14 MB)
    sample_contract.json
    bin_summary.csv
  sample_1000_docs/
    working_sample_manifest.parquet    (29 MB)
    sample_contract.json
    bin_summary.csv
  sample_5000_docs/
    working_sample_manifest.parquet    (141 MB)
    sample_contract.json
    bin_summary.csv
  sample_10000_docs/
    working_sample_manifest.parquet    (280 MB)
    sample_contract.json
    bin_summary.csv
```

## Document format

Each materialized JSONL.zst file contains a subset of Dolma documents.
Each line is a JSON object with at minimum:

```json
{
  "id": "dolma-document-id",
  "text": "full document text...",
  "metadata": { ... }
}
```

The `id` field corresponds to `doc_id` in the manifest. The `text` field contains
the full document content. The `metadata` object varies by source.

## Access patterns

### Reading a specific sample

To read materialized documents for a sample:

1. **From HuggingFace** (once upload completes):
   ```python
   from datasets import load_dataset
   ds = load_dataset("HCAI-Lab/dolma3-6t-sample-500-docs")
   ```

2. **From Modal volume** (for Modal workers):
   ```python
   volume = modal.Volume.from_name("soc134-output-sample-500-docs")
   # mount at /data in your function, read worker dirs
   ```

3. **From the manifest** (to look up which docs are in a sample without downloading):
   ```python
   import pandas as pd
   manifest = pd.read_parquet("data/samples/sample_500_docs/working_sample_manifest.parquet")
   ```

### Reading source shards

For the materialized stratified samples, prefer the HF dataset directly — `hf_hub_download` or `snapshot_download` against `HCAI-Lab/dolma3-6t-sample-{N}-docs` already exposes the per-doc JSONL files. Reading raw upstream shards is only required when you want documents outside any sample.

1. **From HF dataset (materialized sample)** — recommended for sample work:
   ```python
   from huggingface_hub import hf_hub_download
   path = hf_hub_download(
       "HCAI-Lab/dolma3-6t-sample-500-docs",
       "data/part_000/soc127__phase1_pool_shared__...jsonl.zst",
       repo_type="dataset",
   )
   ```

2. **From Modal corpus cache** (fastest, for Modal workers):
   ```python
   # mounted at /corpus
   shard_path = "soc127/phase1_pool_shared/common_crawl/part_000/data__...__shard_00000001.jsonl.zst"
   data = Path("/corpus") / shard_path
   ```

3. **From R2 direct** — only for raw upstream shards outside the materialized samples:
   ```python
   from dolma.quality.r2 import R2Config, create_r2_client, download_object_bytes
   config = R2Config.from_env(output_prefix="any")
   client = create_r2_client(config)
   data = download_object_bytes(client, bucket="soc127-dedup", key=shard_path)
   ```

## R2 credentials (only needed for raw source shards or per-shard sidecar parquets)

R2 hosts the raw 6T `.jsonl.zst` source corpus and the original per-shard sidecars. Most workflows can stay HF-only since the sidecar mirrors are on HF (`HCAI-Lab/soc91-labels`, `soc91-stats`, `soc139-quality-sidecars`). R2 credentials are stored in 1Password. For local scripts:

```bash
bash scripts/bootstrap/with_r2_credentials.sh <command>
```

For Modal workers: credentials are in the `r2-credentials` Modal secret.

## Cleanup notes

- `soc134-materialize-cache`: Legacy shared volume that hit the 500K inode limit.
  Contains partial data from the failed multi-sample run. Safe to delete after
  per-sample volumes are verified.
- `soc134-corpus-cache`: Keep as long as materialization runs are needed.
  Delete when the project phase is complete (~$750/month for 5TB).
- `soc134-manifest-cache`: Keep for future sample draws. Small (~213GB).