HCAI-Lab/w2-consensus-deepdive-unlearning-artifacts/social-data-attribution-w2/docs/data_home - Storage Bucket

Dolma3 Data Attribution — Index

Entry point for the data attribution artifacts produced by the HCAI-Lab Dolma3 project. Use this dataset as the lookup table for "where do I find X?". All artifacts live under the HCAI-Lab org on Hugging Face or in the soc127-dedup Cloudflare R2 bucket.

If you only read one file: inventory.json has every artifact catalogued with location, scale, schema reference, and consumer use case.

HF Collections (grouped views)

The HCAI-Lab org has 8 themed collections covering 41 dataset items. Pick the one closest to your use case:

Collection	Items	Browse on HF
Dolma3 — Source Corpus + Manifest	7	huggingface.co/collections/HCAI-Lab/dolma3-source-corpus-manifest-6a13f27b4ec80bdf68467093
Dolma3 — Working Samples + Preconditioner	6	huggingface.co/collections/HCAI-Lab/dolma3-working-samples-preconditioner-6a13f27ee2d04d7730f25849
Dolma3 — Query Data	3	huggingface.co/collections/HCAI-Lab/dolma3-query-data-6a13f281a8f0f4aacc1276b8
TrackStar — Indices + Training Shards	1	huggingface.co/collections/HCAI-Lab/trackstar-indices-training-shards-6a13f28219af7da31d666589
TrackStar — Scores + Analysis	2	huggingface.co/collections/HCAI-Lab/trackstar-scores-analysis-6a13f283e8a01cc6bb365928
OLMES Evaluations	7	huggingface.co/collections/HCAI-Lab/olmes-evaluations-6a13f2857c48958852e5a714
Archive (pre-6T and legacy)	14	huggingface.co/collections/HCAI-Lab/archive-pre-6t-and-legacy-6a13f288a645c069d72d8aae
Other projects under HCAI-Lab	1	huggingface.co/collections/HCAI-Lab/other-projects-under-hcai-lab-6a13f28d5a929010e1fd4874

The original master collection (HCAI-Lab/dolma3-data-attribution-6a0fe8c1ae78751740458be4) still exists and lists 22 items as a flat view; the sub-collections above are the recommended navigation path.

Note: HF Collections accept only datasets, models, spaces, papers, and other collections — not buckets. So the TrackStar gradient index (1.3 TB bucket), score matrices, query gradients, preconditioners, and figures are NOT in any collection. They're catalogued in inventory.json only. The "TrackStar" collections above contain only the dataset-addressable items in each family.

Navigation by use case

"I want the raw deduplicated 6T Dolma3 corpus"

Source shards on Cloudflare R2 at soc127-dedup/soc127/phase{1_pool_shared,2_nonpool_final}/. 58,621 .jsonl.zst shards, ~5 TB. Schema is {id, text, metadata} per line. See docs/DATA_INVENTORY.md §Source corpus.

"I want stratified working samples for training or evaluation"

Pre-materialized samples on HF, six sizes:

• HCAI-Lab/dolma3-6t-sample-500-docs (288K docs, 539M tokens)
• HCAI-Lab/dolma3-6t-sample-1000-docs (575K docs, 1.1B tokens)
• HCAI-Lab/dolma3-6t-sample-5000-docs (2.86M docs, 5.3B tokens)
• HCAI-Lab/dolma3-6t-sample-10000-docs (5.68M docs, 10.5B tokens)
• HCAI-Lab/dolma3-6t-sample-50000-docs (26.2M docs, 62.8B tokens)
• HCAI-Lab/dolma3-6t-sample-100000-docs (49.7M docs, 118.4B tokens; 130/576 bins underfilled at this scale)

All stratified across 576 topic×format bins, seed=42. Manifest schema in docs/WORKING_SAMPLE_DATA_ACCESS.md. Each dolma3-6t-sample-*-docs repo exists both as an HF dataset (lookup above) and an HF bucket of the same name (S3-style access via hf buckets sync); bit-identical data on both surfaces.

"I want WebOrganizer topic / format labels per document"

Three options, pick the one that fits your access pattern:

• HCAI-Lab/dolma3-olmo3-corpus-manifest — unified manifest, 1.1B rows, 32 columns (topic + format + quality + token count + source shard path). Single repo, the easiest entry point.
• HCAI-Lab/soc91-labels — HF mirror of the raw R2 sidecars: 2,719 parquet chunks (~60 MB each), 169.94 GB total. Each row carries source_shard_path. Use when you want only the topic/format columns and not the full manifest.
• R2 prefix soc127-dedup/soc91-labels/ — original per-shard parquets (58,465 files) if you need per-source-shard granularity.

"I want quality scores per document"

• HCAI-Lab/soc139-quality-sidecars — HF mirror, 1.26B rows, 41.8 GB, 80 parquet files. Columns: doc_id, quality_label_id, quality_score, quality_high_prob, quality_low_prob, quality_confidence, source_shard_path.
• R2 prefix soc127-dedup/soc139-quality-sidecars/ — original per-shard parquets if needed.

Includes the SOC-142 label-fix markers — any local cache from before commit 3342baf had inverted high/low quality labels and should be re-pulled.

"I want attribution scores (influence) for the four OLMES benchmarks"

Per-query score matrices on HF Buckets, one bucket per run-and-model-variant:

• HCAI-Lab/trackstar-scores-base-olmes-4bench — OLMo-3-7B base, 4 benchmarks (gsm8k, mmlu_social_science, mmlu_stem, socialiqa), 2532 .npy files, ~396 GB
• HCAI-Lab/trackstar-scores-instruct-cot-olmes-4bench — same shape but for the instruct-cot variant
• HCAI-Lab/trackstar-scores-{base,instruct-base}-bbh — BBH attribution
• HCAI-Lab/trackstar-scores-{base,instruct-base}-gsm8k-arc — GSM8K + ARC attribution

Score matrix format: shard_NNNN.npy (float32, shape [shard_docs, n_queries]) + shard_NNNN_doc_ids.json (ordered positional doc IDs shard_NNNN:INDEX) + query_ids.json (ordered query IDs). See docs/TRACKSTAR_DATA_ARTIFACTS.md §1.

"I want to resolve those positional doc IDs back to document text"

Required companion: HCAI-Lab/dolma3-6t-sample-10000-docs-trackstar-shards — 316 plain JSONL files mapping shard_NNNN:INDEX to {id: <Dolma UUID>, text: <full document>}. Without this you can't map score matrices back to source text.

"I just want the top-K most influential docs per query — don't make me load 400 GB"

HCAI-Lab/dolma3-trackstar-influence-scores (private dataset) has:

• influence_scores_full.parquet — aggregated influence scores
• top2k_{gsm8k,mmlu_socsci,mmlu_stem,socialiqa}.{csv,parquet} — top-2K per query, rank-ordered

HCAI-Lab/trackstar-top2k-{base,instruct-base}-gsm8k-arc — same shape for the GSM8K + ARC benchmark set.

"I want to score new query sets against the existing training corpus"

You need the Bergson training-gradient index: HCAI-Lab/trackstar-gradient-index-base (private bucket, 1.2 TB, 316 shard subdirs). Each shard has gradients.bin + normalizers.pth + preconditioners*.pth + configs. CPU-only scoring against new queries; no GPU rebuild required. ~156 GPU-hours to reproduce from scratch.

"I want pre-built preconditioners for new attribution runs"

HCAI-Lab/trackstar-preconditioners — three subdirs (olmo-3-1025-7b, olmo-3-7b-instruct, olmo-3-7b-think), 78 files total, ~885 MB. Mixed preconditioners per SOC-152 / SOC-168. The preconditioner must match the gradient-build model (SOC-162 finding) — mixing base preconditioner with instruct gradients collapses benchmark-specific signal.

"I want pre-built query gradient indices (instead of rebuilding from queries)"

HCAI-Lab/trackstar-query-gradients-base (private bucket, 17 GB). Three subdirs: base/, instruct_base/, instruct_cot/, each containing the four OLMES benchmark query gradient builds.

"I want OLMES evaluation results / per-query model predictions"

Datasets under HCAI-Lab/olmes-eval-olmo3-7b-{base,instruct-base,instruct-cot,thinking,think-mc,think-cot} plus the SOC-166 bucket HCAI-Lab/trackstar-olmes-eval-artifacts.

"I want dedup state for reproducing the corpus"

Bloom filter at HCAI-Lab/dolma3-6t-bloom-index, doc IDs at HCAI-Lab/archive-dolma3-6t-doc-ids-pershard (private, gzip-compressed JSONL per shard), unique docs materialized at HCAI-Lab/dolma3-6t-unique (1.258B docs).

Schema references (where to find canonical definitions)

Schema	Where it's defined
Source shard JSONL (id, text, metadata)	docs/DATA_INVENTORY.md §Document format
Working sample manifest (6 cols: doc_id/token_count/shard_path/bin_id/bin_topic/bin_format)	docs/WORKING_SAMPLE_DATA_ACCESS.md §Manifest schema
Unified corpus manifest (32-col PyArrow)	src/data_attribution/recipes/corpus_manifest.py:15-52
Sidecar row construction (joins source + WebOrganizer + quality)	src/dolma/sidecar_manifest_fields.py:43-124
Score matrix format	docs/TRACKSTAR_DATA_ARTIFACTS.md §1
Bergson gradient index layout (per shard)	docs/TRACKSTAR_DATA_ARTIFACTS.md §5
Bin ID formula (1-576)	topic_idx * len(FORMATS) + format_idx + 1 — src/dolma/manifest_fields.py:60-67

Access

• Code: github.com/eilab-gt/social-data-attribution. CLI entry points in pyproject.toml [project.scripts].
• HF: set HF_TOKEN in env, or store at ~/.hf_token. For private repos in HCAI-Lab, request collaborator invite from the lab admin.
• R2: read-only token issued via 1Password Share. Set R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY. Repo provides scripts/bootstrap/with_r2_credentials.sh as a wrapper.
• PACE: not available to external consumers.

Known gotchas

• Set HF_HUB_DISABLE_XET=1 and point HF_HOME at local storage if running uploads on PACE NFS / Lustre. Xet finalization hangs there.
• SOC-142: an earlier batch of soc91-labels sidecars had inverted high/low quality labels. The fix is in commit 3342baf. Refresh any local cache from before that.
• Underfilled bins in stratified samples (e.g., 17 underfilled bins in sample_10000_docs) are real, not a bug — the source corpus genuinely has fewer documents in that topic×format combination than requested.
• Don't mix instruct query files into a base query directory before launching attribution scoring — the reduce job will silently consume them and produce nonsense (see docs/ATTRIBUTION_RUNBOOK.md).

Lifecycle

This inventory was first built 2026-05-22 as part of the external-team handoff migration. Source preservation report: PRESERVATION_GAP_REPORT.md in the source repo. Regenerated when artifacts change.

Contact

HCAI-Lab data attribution work. Contact Glenn Matlin via the lab Slack channel.