social-data-attribution-w2/docs/EXTERNAL_TEAM_HANDOFF.md

# External Team Data-Access Handoff

This is the entry point for an HCAI-Lab team picking up artifacts from the Dolma3 data attribution project. It tells you what you can access, how to get credentials, how to load the data, and where the schemas are defined.

It assumes Python familiarity and basic comfort with the HuggingFace Hub and S3-compatible APIs. It does **not** assume PACE access — every artifact you need is in cloud (HuggingFace + Cloudflare R2).

For a Hub-side navigation tool, see the [Dolma3 Data Attribution Collection](https://huggingface.co/collections/HCAI-Lab/dolma3-data-attribution-6a0fe8c1ae78751740458be4) and the `HCAI-Lab/dolma3-data-attribution-index` dataset (`inventory.json` is machine-readable).

## What you get

By **use case**:

| Goal | Where to look | Access detail |
|---|---|---|
| Raw deduplicated 6T Dolma3 corpus | R2 `soc127-dedup/soc127/{phase1_pool_shared,phase2_nonpool_final}/` (58,621 `.jsonl.zst` shards, ~5 TB) | `docs/DATA_INVENTORY.md` §Source corpus. **This is the only artifact that requires R2 access.** |
| Stratified working samples | `HCAI-Lab/dolma3-6t-sample-{500,1000,5000,10000,50000,100000}-docs` HF datasets (each also exposed as an HF Bucket of the same name) | `docs/WORKING_SAMPLE_DATA_ACCESS.md` |
| 100K preconditioner sample | `HCAI-Lab/dolma3-6t-preconditioner-100k` HF dataset | `docs/WORKING_SAMPLE_DATA_ACCESS.md` §Preconditioner sample |
| WebOrganizer topic/format labels (per doc) | `HCAI-Lab/dolma3-olmo3-corpus-manifest` (unified manifest with topic+format+quality), or `HCAI-Lab/soc91-labels` (HF mirror of raw R2 sidecars). R2 prefix `soc127-dedup/soc91-labels/` remains as a fallback for per-shard granularity. | `src/dolma/sidecar_manifest_fields.py:43-124`, README §Sidecar corpus manifest |
| Quality scores (per doc) | `HCAI-Lab/soc139-quality-sidecars` (1.26B rows, 41.8 GB, 80 parquet files; includes SOC-142 label-fix). R2 prefix `soc127-dedup/soc139-quality-sidecars/` is the fallback for per-shard granularity. | `src/dolma/quality/sidecar.py` |
| Per-query attribution scores (the 4 OLMES benchmarks) | `HCAI-Lab/trackstar-scores-base-olmes-4bench` HF Bucket (+ siblings for instruct/cot/BBH/ARC variants) | `docs/TRACKSTAR_DATA_ARTIFACTS.md` §1 |
| Resolve positional doc IDs in score matrices to source text | `HCAI-Lab/dolma3-6t-sample-10000-docs-trackstar-shards` HF dataset (REQUIRED if you use score matrices) | `docs/TRACKSTAR_DATA_ARTIFACTS.md` §3 |
| Top-K influence results per benchmark | `HCAI-Lab/dolma3-trackstar-influence-scores` (top-2K + aggregated parquet) | `docs/TRACKSTAR_DATA_ARTIFACTS.md` §2 |
| Bergson training gradient index (compute new scores against existing corpus) | `HCAI-Lab/trackstar-gradient-index-base` HF Bucket (1.2 TB, 316 shards) | `docs/TRACKSTAR_DATA_ARTIFACTS.md` §5 |
| Pre-built preconditioners (3 OLMo3 variants) | `HCAI-Lab/trackstar-preconditioners` HF Bucket | `docs/TRACKSTAR_DATA_ARTIFACTS.md` §7 |
| Query gradient indices (skip rebuilding from queries) | `HCAI-Lab/trackstar-query-gradients-base` HF Bucket | `docs/TRACKSTAR_DATA_ARTIFACTS.md` §6 |
| OLMES evaluation results (per-query predictions + correctness) | `HCAI-Lab/olmes-eval-olmo3-7b-*` datasets | n/a (parquet, columns documented in repo card) |
| Dedup state (Bloom filter, unique docs, doc IDs) | `HCAI-Lab/dolma3-6t-bloom-index`, `dolma3-6t-unique`, `archive-dolma3-6t-doc-ids-pershard` | `src/dolma/provenance.py` |
| Job archive (logs, sbatch, dedup work state) | `HCAI-Lab/dolma3-attribution-job-archive` (private dataset, tarballs) | n/a (archival only) |

## Access setup

### HuggingFace

The HCAI-Lab repos are a mix of public and private. Most working samples, eval results, query data, dedup state, corpus manifest, and the preconditioner sample are public. The Bergson gradient index, training shards, query gradient builds, top-2K influence scores, job archive, the data-home index, and the three sidecar mirrors (`soc91-labels`, `soc91-stats`, `soc139-quality-sidecars`) are **private**.

For access:

1. Send your HF username to Glenn (lab admin).
2. Glenn invites you as a collaborator to the relevant `HCAI-Lab/*` repos (read access).
3. Generate an HF token at `https://huggingface.co/settings/tokens`. Save to `~/.hf_token` or set `HF_TOKEN` in your env.

Verify:

```bash
hf auth whoami
# Should show your user + 'HCAI-Lab' in the orgs list
hf datasets info HCAI-Lab/dolma3-data-attribution-index
# Should return the data-home metadata
```

### Cloudflare R2

R2 hosts the raw 6T corpus (`.jsonl.zst` source shards) and the original per-shard label/quality sidecars. Since 2026-05-23 the label and quality sidecars are also available as HF datasets (`HCAI-Lab/soc91-labels`, `HCAI-Lab/soc91-stats`, `HCAI-Lab/soc139-quality-sidecars`), so most workflows can stay HF-only. You only need R2 if you want the raw `.jsonl.zst` source shards or per-shard sidecar granularity.

To get a read-only API token scoped to the `soc127-dedup` bucket:

1. Request the token from Glenn. Delivered via 1Password Share or another secure channel.
2. Set environment variables:

```bash
export R2_ACCESS_KEY_ID="<your-key-id>"
export R2_SECRET_ACCESS_KEY="<your-secret>"
```

3. Verify:

```bash
python -c "
import os, boto3
s3 = boto3.client('s3',
    endpoint_url='https://0934ab8e84ac8f4e81decaf3eb121337.r2.cloudflarestorage.com',
    aws_access_key_id=os.environ['R2_ACCESS_KEY_ID'],
    aws_secret_access_key=os.environ['R2_SECRET_ACCESS_KEY'])
print(s3.list_objects_v2(Bucket='soc127-dedup', Prefix='soc127/', MaxKeys=1))
"
# Should print a Contents list with one shard path
```

If you clone this repo, `scripts/bootstrap/with_r2_credentials.sh` wraps the lookup (uses `op` 1Password CLI by default; override with `R2_*` env vars directly).

### PACE access

The project's GPU compute and original artifact-build environment lives on Georgia Tech PACE. **You do not have access to PACE** unless separately sponsored. Every artifact you need has been mirrored to cloud — no PACE step is required for normal use.

## End-to-end worked example

Goal: download the 500-docs sample manifest, fetch one shard, decompress and inspect the first three docs.

```python
import io
import json
import zstandard as zstd
from huggingface_hub import hf_hub_download

REPO = "HCAI-Lab/dolma3-6t-sample-500-docs"

# 1) Pull the manifest (one row per sampled doc).
manifest_path = hf_hub_download(REPO, "working_sample_manifest.parquet",
                                repo_type="dataset")
print("Manifest:", manifest_path)

import pandas as pd
manifest = pd.read_parquet(manifest_path)
print(manifest.head())
print("Columns:", list(manifest.columns))
# Expected: doc_id, token_count, shard_path, bin_id, bin_topic, bin_format

# 2) Pull one materialized shard and decompress.
one_shard = manifest["shard_path"].iloc[0]
# shard_path on R2 looks like soc127__phase1_pool_shared__...__shard_NNNNN.jsonl.zst
# In the HF dataset, it lives under data/part_NNN/<same-filename>.
# Look it up by filename in the repo.
from huggingface_hub import list_repo_files
files = list_repo_files(REPO, repo_type="dataset")
shard_basename = one_shard.replace("/", "__").split("/")[-1]
match = next(f for f in files if f.endswith(shard_basename))
print("Shard file:", match)

shard_local = hf_hub_download(REPO, match, repo_type="dataset")

dctx = zstd.ZstdDecompressor()
with open(shard_local, "rb") as f, dctx.stream_reader(f, read_across_frames=True) as r:
    for i, line in enumerate(io.TextIOWrapper(r, encoding="utf-8")):
        if i >= 3:
            break
        doc = json.loads(line)
        print(f"\nDoc {i}: id={doc['id']}")
        print(f"  text snippet: {doc['text'][:120]!r}")
        print(f"  metadata keys: {list((doc.get('metadata') or {}).keys())[:5]}")
```

If that runs end-to-end and prints three doc snippets, your HF access is working and you understand the manifest → shard → docs layout.

For attribution-specific examples (loading a score matrix, resolving doc IDs back to text, finding the most influential docs for a query), see `docs/TRACKSTAR_DATA_ARTIFACTS.md` §End-to-End Example.

## Schema crosswalk

| Schema | Where it's defined |
|---|---|
| Source shard JSONL (`id`, `text`, `metadata`) | `docs/DATA_INVENTORY.md` §Document format |
| Working sample manifest (6 cols: `doc_id`, `token_count`, `shard_path`, `bin_id`, `bin_topic`, `bin_format`) | `docs/WORKING_SAMPLE_DATA_ACCESS.md` §Manifest schema |
| Unified corpus manifest (32-col PyArrow) | `src/data_attribution/recipes/corpus_manifest.py:15-52` |
| Sidecar parquet row construction (source + labels + quality) | `src/dolma/sidecar_manifest_fields.py:43-124` |
| Score matrix format (`shard_NNNN.npy` + `_doc_ids.json` + `query_ids.json`) | `docs/TRACKSTAR_DATA_ARTIFACTS.md` §1 |
| Bergson gradient index layout (per shard) | `docs/TRACKSTAR_DATA_ARTIFACTS.md` §5 |
| Query JSONL row format (`text`, `query_id`, `is_correct`, `predicted_answer`, `correct_answer`) | `docs/TRACKSTAR_DATA_ARTIFACTS.md` §4 |
| Bin ID formula | `topic_idx * len(FORMATS) + format_idx + 1` (576 bins, 1-indexed); `src/dolma/manifest_fields.py:60-67` |

Topic and format enum values (24 each, total 576 bins) are in `src/dolma/manifest_fields.py` lines 13–65 — see `docs/DATA_INVENTORY.md` §Quality labels on R2 and on HF for the human-readable list.

## CLI cheat-sheet

The repo provides Python entry points in `pyproject.toml [project.scripts]`. The ones most useful for consuming artifacts:

| Command | Purpose |
|---|---|
| `data-attribution-corpus-manifest` | Build the unified corpus manifest from sidecars + source shards (accepts either R2 or HF mirror as the sidecar source) |
| `data-attribution-manifest-sample` | Query the corpus manifest by source/topic/format |
| `data-attribution-draw-working-sample` | Draw a new stratified sample with custom parameters |
| `data-attribution-trackstar-query` | Download query JSONL for the four OLMES benchmarks |
| `data-attribution-trackstar-prepare` | Prepare attribution inputs from queries |
| `data-attribution-trackstar-extract` | Extract top-K influence results from score matrices |
| `data-attribution-precache` | Pre-cache an allowed HF dataset (Olmo-3 or Dolma3 mix) |

For the full attribution pipeline walkthrough (run on your own compute), see `docs/ATTRIBUTION_RUNBOOK.md`. The runbook is written from the HCAI-Lab operator's perspective (PACE/SLURM) but the inputs and outputs all map to the HF/R2 artifacts above — replace SLURM submission with your local launcher and substitute HF `snapshot_download` / `hf_hub_download` for any `~/scratch/...` paths.

## Known gotchas (the things we learned the hard way)

1. **`HF_HUB_DISABLE_XET=1` on NFS / Lustre filesystems.** The HF Xet storage backend hangs during batch finalization on PACE-like NFS volumes. Set the env var and point `HF_HOME` at local storage (`/tmp` or `$TMPDIR`) for uploads. See `CLAUDE.md` §HuggingFace Uploads from PACE.
2. **SOC-142 inverted quality labels.** An earlier batch of `soc91-labels` sidecars had inverted high/low quality scores. The fix landed in commit `3342baf` (2026-02). Any local cache from before that date should be re-pulled. Affects only the quality fields, not topic/format.
3. **TrackStar preconditioner-model matching.** When using a precomputed preconditioner with a new gradient build, the preconditioner must match the model used to build the gradients. SOC-162 found that mixing a base-model preconditioner with instruct-model queries collapses benchmark-specific signal. Three pre-built variants are available under `HCAI-Lab/trackstar-preconditioners/{olmo-3-1025-7b,olmo-3-7b-instruct,olmo-3-7b-think}/`.
4. **Underfilled bins are real, not a bug.** The five stratified samples cover 576/576 bins, but some bins are "underfilled" — the source corpus genuinely has fewer documents in that topic×format combination than the sample's `docs_per_bin` parameter. See `WORKING_SAMPLE_DATA_ACCESS.md` line 14. The `bin_summary.csv` in each sample lists the underfilled bins.
5. **Don't mix instruct query files into a `base` query directory before running attribution scoring.** The reduce step will silently consume them and produce confused results. The `data-attribution-trackstar-query --variant {base,instruct_base,instruct_cot}` CLI uses separate output dirs to avoid this; respect that separation.
6. **Bergson + Transformers version pinning.** The repo vendors a specific Bergson commit (currently `v0.9.0`) through `scripts/bootstrap_local_deps.sh`. Earlier Bergson versions pinned `transformers<4.56.0` and were incompatible with OLMo3; that constraint is no longer present in `v0.9.0`. See README §"Bergson + Transformers version note".
7. **HuggingFace rate limits.** HCAI-Lab is on the HF Team plan (3000 req / 5min). You can hit the limit with very high-concurrency `snapshot_download` jobs; throttle with `max_workers=8` or fewer for large datasets.
8. **Score matrix doc IDs are positional, not Dolma UUIDs.** The `shard_NNNN:INDEX` format encodes the index of the doc within the TrackStar training shard, not the original Dolma UUID. You **must** download `HCAI-Lab/dolma3-6t-sample-10000-docs-trackstar-shards` to resolve these back to source text. See `docs/TRACKSTAR_DATA_ARTIFACTS.md` §3.

## Where to ask questions

Slack: HCAI-Lab data-attribution channel (channel name to be confirmed with your team lead).
Direct: Glenn Matlin, `glenn.matlin@gmail.com`.

If you find something missing, incorrect, or unclear in this doc, please flag it — the goal is for this packet to be self-sufficient so we don't have to round-trip on basics. The `PRESERVATION_GAP_REPORT.md` at the repo root has the underlying audit if you want to see what was migrated when.

## Pointers to other docs

- `docs/DATA_INVENTORY.md` — fuller storage layout reference (R2, HF, Modal). HF is the primary surface; PACE no longer holds anything sole-source.
- `docs/WORKING_SAMPLE_DATA_ACCESS.md` — sample loading, layout, decompression patterns.
- `docs/ATTRIBUTION_RUNBOOK.md` — step-by-step attribution pipeline. Written for an HCAI-Lab operator on PACE/SLURM; substitute your own compute for the SLURM submission steps. Input/output paths point at the same HF/R2 artifacts external consumers can reach.
- `docs/TRACKSTAR_DATA_ARTIFACTS.md` — score matrix and gradient-index loading, with a worked end-to-end example for finding most-influential docs.
- `docs/BERGSON_REFERENCE.md` — Bergson tool reference.
- `docs/SOC91_GPU_ENRICHMENT_HANDOFF.md`, `docs/SOC95_MODAL_RUNBOOK.md` — how the per-doc labels and corpus manifest were built.
- `docs/data_home/inventory.json` — machine-readable artifact catalog (mirrored at `HCAI-Lab/dolma3-data-attribution-index`).
- `PRESERVATION_GAP_REPORT.md` — closed audit: what was on PACE only at the start of the migration, what got uploaded when, and final verification numbers.