social-data-attribution-w2/docs/EXTERNAL_TEAM_HANDOFF.md

External Team Data-Access Handoff

This is the entry point for an HCAI-Lab team picking up artifacts from the Dolma3 data attribution project. It tells you what you can access, how to get credentials, how to load the data, and where the schemas are defined.

It assumes Python familiarity and basic comfort with the HuggingFace Hub and S3-compatible APIs. It does not assume PACE access — every artifact you need is in cloud (HuggingFace + Cloudflare R2).

For a Hub-side navigation tool, see the Dolma3 Data Attribution Collection and the HCAI-Lab/dolma3-data-attribution-index dataset (inventory.json is machine-readable).

What you get

By use case:

Goal	Where to look	Access detail
Raw deduplicated 6T Dolma3 corpus	R2 soc127-dedup/soc127/{phase1_pool_shared,phase2_nonpool_final}/ (58,621 .jsonl.zst shards, ~5 TB)	docs/DATA_INVENTORY.md §Source corpus. This is the only artifact that requires R2 access.
Stratified working samples	HCAI-Lab/dolma3-6t-sample-{500,1000,5000,10000,50000,100000}-docs HF datasets (each also exposed as an HF Bucket of the same name)	docs/WORKING_SAMPLE_DATA_ACCESS.md
100K preconditioner sample	HCAI-Lab/dolma3-6t-preconditioner-100k HF dataset	docs/WORKING_SAMPLE_DATA_ACCESS.md §Preconditioner sample
WebOrganizer topic/format labels (per doc)	HCAI-Lab/dolma3-olmo3-corpus-manifest (unified manifest with topic+format+quality), or HCAI-Lab/soc91-labels (HF mirror of raw R2 sidecars). R2 prefix soc127-dedup/soc91-labels/ remains as a fallback for per-shard granularity.	src/dolma/sidecar_manifest_fields.py:43-124, README §Sidecar corpus manifest
Quality scores (per doc)	HCAI-Lab/soc139-quality-sidecars (1.26B rows, 41.8 GB, 80 parquet files; includes SOC-142 label-fix). R2 prefix soc127-dedup/soc139-quality-sidecars/ is the fallback for per-shard granularity.	src/dolma/quality/sidecar.py
Per-query attribution scores (the 4 OLMES benchmarks)	HCAI-Lab/trackstar-scores-base-olmes-4bench HF Bucket (+ siblings for instruct/cot/BBH/ARC variants)	docs/TRACKSTAR_DATA_ARTIFACTS.md §1
Resolve positional doc IDs in score matrices to source text	HCAI-Lab/dolma3-6t-sample-10000-docs-trackstar-shards HF dataset (REQUIRED if you use score matrices)	docs/TRACKSTAR_DATA_ARTIFACTS.md §3
Top-K influence results per benchmark	HCAI-Lab/dolma3-trackstar-influence-scores (top-2K + aggregated parquet)	docs/TRACKSTAR_DATA_ARTIFACTS.md §2
Bergson training gradient index (compute new scores against existing corpus)	HCAI-Lab/trackstar-gradient-index-base HF Bucket (1.2 TB, 316 shards)	docs/TRACKSTAR_DATA_ARTIFACTS.md §5
Pre-built preconditioners (3 OLMo3 variants)	HCAI-Lab/trackstar-preconditioners HF Bucket	docs/TRACKSTAR_DATA_ARTIFACTS.md §7
Query gradient indices (skip rebuilding from queries)	HCAI-Lab/trackstar-query-gradients-base HF Bucket	docs/TRACKSTAR_DATA_ARTIFACTS.md §6
OLMES evaluation results (per-query predictions + correctness)	HCAI-Lab/olmes-eval-olmo3-7b-* datasets	n/a (parquet, columns documented in repo card)
Dedup state (Bloom filter, unique docs, doc IDs)	HCAI-Lab/dolma3-6t-bloom-index, dolma3-6t-unique, archive-dolma3-6t-doc-ids-pershard	src/dolma/provenance.py
Job archive (logs, sbatch, dedup work state)	HCAI-Lab/dolma3-attribution-job-archive (private dataset, tarballs)	n/a (archival only)

Access setup

HuggingFace

The HCAI-Lab repos are a mix of public and private. Most working samples, eval results, query data, dedup state, corpus manifest, and the preconditioner sample are public. The Bergson gradient index, training shards, query gradient builds, top-2K influence scores, job archive, the data-home index, and the three sidecar mirrors (soc91-labels, soc91-stats, soc139-quality-sidecars) are private.

For access:

1. Send your HF username to Glenn (lab admin).
2. Glenn invites you as a collaborator to the relevant HCAI-Lab/* repos (read access).
3. Generate an HF token at https://huggingface.co/settings/tokens. Save to ~/.hf_token or set HF_TOKEN in your env.

Verify:

hf auth whoami
# Should show your user + 'HCAI-Lab' in the orgs list
hf datasets info HCAI-Lab/dolma3-data-attribution-index
# Should return the data-home metadata

Cloudflare R2

R2 hosts the raw 6T corpus (.jsonl.zst source shards) and the original per-shard label/quality sidecars. Since 2026-05-23 the label and quality sidecars are also available as HF datasets (HCAI-Lab/soc91-labels, HCAI-Lab/soc91-stats, HCAI-Lab/soc139-quality-sidecars), so most workflows can stay HF-only. You only need R2 if you want the raw .jsonl.zst source shards or per-shard sidecar granularity.

To get a read-only API token scoped to the soc127-dedup bucket:

1. Request the token from Glenn. Delivered via 1Password Share or another secure channel.
2. Set environment variables:

export R2_ACCESS_KEY_ID="<your-key-id>"
export R2_SECRET_ACCESS_KEY="<your-secret>"

3. Verify:

python -c "
import os, boto3
s3 = boto3.client('s3',
    endpoint_url='https://0934ab8e84ac8f4e81decaf3eb121337.r2.cloudflarestorage.com',
    aws_access_key_id=os.environ['R2_ACCESS_KEY_ID'],
    aws_secret_access_key=os.environ['R2_SECRET_ACCESS_KEY'])
print(s3.list_objects_v2(Bucket='soc127-dedup', Prefix='soc127/', MaxKeys=1))
"
# Should print a Contents list with one shard path

If you clone this repo, scripts/bootstrap/with_r2_credentials.sh wraps the lookup (uses op 1Password CLI by default; override with R2_* env vars directly).

PACE access

The project's GPU compute and original artifact-build environment lives on Georgia Tech PACE. You do not have access to PACE unless separately sponsored. Every artifact you need has been mirrored to cloud — no PACE step is required for normal use.

End-to-end worked example

Goal: download the 500-docs sample manifest, fetch one shard, decompress and inspect the first three docs.

import io
import json
import zstandard as zstd
from huggingface_hub import hf_hub_download

REPO = "HCAI-Lab/dolma3-6t-sample-500-docs"

# 1) Pull the manifest (one row per sampled doc).
manifest_path = hf_hub_download(REPO, "working_sample_manifest.parquet",
                                repo_type="dataset")
print("Manifest:", manifest_path)

import pandas as pd
manifest = pd.read_parquet(manifest_path)
print(manifest.head())
print("Columns:", list(manifest.columns))
# Expected: doc_id, token_count, shard_path, bin_id, bin_topic, bin_format

# 2) Pull one materialized shard and decompress.
one_shard = manifest["shard_path"].iloc[0]
# shard_path on R2 looks like soc127__phase1_pool_shared__...__shard_NNNNN.jsonl.zst
# In the HF dataset, it lives under data/part_NNN/<same-filename>.
# Look it up by filename in the repo.
from huggingface_hub import list_repo_files
files = list_repo_files(REPO, repo_type="dataset")
shard_basename = one_shard.replace("/", "__").split("/")[-1]
match = next(f for f in files if f.endswith(shard_basename))
print("Shard file:", match)

shard_local = hf_hub_download(REPO, match, repo_type="dataset")

dctx = zstd.ZstdDecompressor()
with open(shard_local, "rb") as f, dctx.stream_reader(f, read_across_frames=True) as r:
    for i, line in enumerate(io.TextIOWrapper(r, encoding="utf-8")):
        if i >= 3:
            break
        doc = json.loads(line)
        print(f"\nDoc {i}: id={doc['id']}")
        print(f"  text snippet: {doc['text'][:120]!r}")
        print(f"  metadata keys: {list((doc.get('metadata') or {}).keys())[:5]}")

If that runs end-to-end and prints three doc snippets, your HF access is working and you understand the manifest → shard → docs layout.

For attribution-specific examples (loading a score matrix, resolving doc IDs back to text, finding the most influential docs for a query), see docs/TRACKSTAR_DATA_ARTIFACTS.md §End-to-End Example.

Schema crosswalk

Schema	Where it's defined
Source shard JSONL (id, text, metadata)	docs/DATA_INVENTORY.md §Document format
Working sample manifest (6 cols: doc_id, token_count, shard_path, bin_id, bin_topic, bin_format)	docs/WORKING_SAMPLE_DATA_ACCESS.md §Manifest schema
Unified corpus manifest (32-col PyArrow)	src/data_attribution/recipes/corpus_manifest.py:15-52
Sidecar parquet row construction (source + labels + quality)	src/dolma/sidecar_manifest_fields.py:43-124
Score matrix format (shard_NNNN.npy + _doc_ids.json + query_ids.json)	docs/TRACKSTAR_DATA_ARTIFACTS.md §1
Bergson gradient index layout (per shard)	docs/TRACKSTAR_DATA_ARTIFACTS.md §5
Query JSONL row format (text, query_id, is_correct, predicted_answer, correct_answer)	docs/TRACKSTAR_DATA_ARTIFACTS.md §4
Bin ID formula	topic_idx * len(FORMATS) + format_idx + 1 (576 bins, 1-indexed); src/dolma/manifest_fields.py:60-67

Topic and format enum values (24 each, total 576 bins) are in src/dolma/manifest_fields.py lines 13–65 — see docs/DATA_INVENTORY.md §Quality labels on R2 and on HF for the human-readable list.

CLI cheat-sheet

The repo provides Python entry points in pyproject.toml [project.scripts]. The ones most useful for consuming artifacts:

Command	Purpose
data-attribution-corpus-manifest	Build the unified corpus manifest from sidecars + source shards (accepts either R2 or HF mirror as the sidecar source)
data-attribution-manifest-sample	Query the corpus manifest by source/topic/format
data-attribution-draw-working-sample	Draw a new stratified sample with custom parameters
data-attribution-trackstar-query	Download query JSONL for the four OLMES benchmarks
data-attribution-trackstar-prepare	Prepare attribution inputs from queries
data-attribution-trackstar-extract	Extract top-K influence results from score matrices
data-attribution-precache	Pre-cache an allowed HF dataset (Olmo-3 or Dolma3 mix)

For the full attribution pipeline walkthrough (run on your own compute), see docs/ATTRIBUTION_RUNBOOK.md. The runbook is written from the HCAI-Lab operator's perspective (PACE/SLURM) but the inputs and outputs all map to the HF/R2 artifacts above — replace SLURM submission with your local launcher and substitute HF snapshot_download / hf_hub_download for any ~/scratch/... paths.

Known gotchas (the things we learned the hard way)

1. HF_HUB_DISABLE_XET=1 on NFS / Lustre filesystems. The HF Xet storage backend hangs during batch finalization on PACE-like NFS volumes. Set the env var and point HF_HOME at local storage (/tmp or $TMPDIR) for uploads. See CLAUDE.md §HuggingFace Uploads from PACE.
2. SOC-142 inverted quality labels. An earlier batch of soc91-labels sidecars had inverted high/low quality scores. The fix landed in commit 3342baf (2026-02). Any local cache from before that date should be re-pulled. Affects only the quality fields, not topic/format.
3. TrackStar preconditioner-model matching. When using a precomputed preconditioner with a new gradient build, the preconditioner must match the model used to build the gradients. SOC-162 found that mixing a base-model preconditioner with instruct-model queries collapses benchmark-specific signal. Three pre-built variants are available under HCAI-Lab/trackstar-preconditioners/{olmo-3-1025-7b,olmo-3-7b-instruct,olmo-3-7b-think}/.
4. Underfilled bins are real, not a bug. The five stratified samples cover 576/576 bins, but some bins are "underfilled" — the source corpus genuinely has fewer documents in that topic×format combination than the sample's docs_per_bin parameter. See WORKING_SAMPLE_DATA_ACCESS.md line 14. The bin_summary.csv in each sample lists the underfilled bins.
5. Don't mix instruct query files into a base query directory before running attribution scoring. The reduce step will silently consume them and produce confused results. The data-attribution-trackstar-query --variant {base,instruct_base,instruct_cot} CLI uses separate output dirs to avoid this; respect that separation.
6. Bergson + Transformers version pinning. The repo vendors a specific Bergson commit (currently v0.9.0) through scripts/bootstrap_local_deps.sh. Earlier Bergson versions pinned transformers<4.56.0 and were incompatible with OLMo3; that constraint is no longer present in v0.9.0. See README §"Bergson + Transformers version note".
7. HuggingFace rate limits. HCAI-Lab is on the HF Team plan (3000 req / 5min). You can hit the limit with very high-concurrency snapshot_download jobs; throttle with max_workers=8 or fewer for large datasets.
8. Score matrix doc IDs are positional, not Dolma UUIDs. The shard_NNNN:INDEX format encodes the index of the doc within the TrackStar training shard, not the original Dolma UUID. You must download HCAI-Lab/dolma3-6t-sample-10000-docs-trackstar-shards to resolve these back to source text. See docs/TRACKSTAR_DATA_ARTIFACTS.md §3.

Where to ask questions

Slack: HCAI-Lab data-attribution channel (channel name to be confirmed with your team lead). Direct: Glenn Matlin, glenn.matlin@gmail.com.

If you find something missing, incorrect, or unclear in this doc, please flag it — the goal is for this packet to be self-sufficient so we don't have to round-trip on basics. The PRESERVATION_GAP_REPORT.md at the repo root has the underlying audit if you want to see what was migrated when.

Pointers to other docs

• docs/DATA_INVENTORY.md — fuller storage layout reference (R2, HF, Modal). HF is the primary surface; PACE no longer holds anything sole-source.
• docs/WORKING_SAMPLE_DATA_ACCESS.md — sample loading, layout, decompression patterns.
• docs/ATTRIBUTION_RUNBOOK.md — step-by-step attribution pipeline. Written for an HCAI-Lab operator on PACE/SLURM; substitute your own compute for the SLURM submission steps. Input/output paths point at the same HF/R2 artifacts external consumers can reach.
• docs/TRACKSTAR_DATA_ARTIFACTS.md — score matrix and gradient-index loading, with a worked end-to-end example for finding most-influential docs.
• docs/BERGSON_REFERENCE.md — Bergson tool reference.
• docs/SOC91_GPU_ENRICHMENT_HANDOFF.md, docs/SOC95_MODAL_RUNBOOK.md — how the per-doc labels and corpus manifest were built.
• docs/data_home/inventory.json — machine-readable artifact catalog (mirrored at HCAI-Lab/dolma3-data-attribution-index).
• PRESERVATION_GAP_REPORT.md — closed audit: what was on PACE only at the start of the migration, what got uploaded when, and final verification numbers.