social-data-attribution-w2/docs/WORKING_SAMPLE_DATA_ACCESS.md

Working Sample Data Access Guide

You are working with stratified working samples drawn from the Dolma 3 6T deduplicated corpus. Each sample selects N documents per bin across 576 bins (24 topics x 24 formats), using deterministic priority scoring (blake2b hash of doc_id:seed, seed=42).

Available samples

Sample	docs/bin	Total docs	Total tokens	Unique source shards
sample_500_docs	500	287,936	538.8M	~42,461
sample_1000_docs	1,000	575,187	1.08B	~45,926
sample_5000_docs	5,000	2,855,446	5.30B	~55,347
sample_10000_docs	10,000	5,678,621	10.5B	~57,622

All samples cover 576/576 bins. A small number of bins are underfilled (the corpus has fewer docs in that topic x format combination than requested).

Where the data lives

Each sample exists in three locations. Use whichever fits your compute environment.

1. HuggingFace (public, persistent)

Repos under HCAI-Lab org. Upload in progress as of 2026-03-20 (check repo for completion).

HCAI-Lab/dolma3-6t-sample-500-docs
HCAI-Lab/dolma3-6t-sample-1000-docs
HCAI-Lab/dolma3-6t-sample-5000-docs
HCAI-Lab/dolma3-6t-sample-10000-docs

Layout:

README.md
sample_contract.json          # sampling parameters and realized counts
bin_summary.csv               # per-bin fill rates
working_sample_manifest.parquet  # one row per doc with bin assignments
data/
  part_000/                   # max 5,000 files per subdirectory
    soc127__phase1_pool_shared__...__shard_00000001.jsonl.zst
    ...
  part_001/
    ...

Reading:

from datasets import load_dataset
ds = load_dataset("HCAI-Lab/dolma3-6t-sample-500-docs")

Or read individual files:

from huggingface_hub import hf_hub_download
manifest = hf_hub_download(
    "HCAI-Lab/dolma3-6t-sample-500-docs",
    "working_sample_manifest.parquet",
    repo_type="dataset",
)

2. Modal persistent volumes (for Modal workers)

Each sample has its own volume in the eilab-gt Modal workspace.

soc134-output-sample-500-docs
soc134-output-sample-1000-docs
soc134-output-sample-5000-docs
soc134-output-sample-10000-docs

Volume layout:

{run_id}/
  manifest.parquet
  sample_contract.json
  bin_summary.csv
  worker_0000/
    soc127__phase1_pool_shared__...__shard_NNNNN.jsonl.zst
    soc127__phase1_pool_shared__...__shard_NNNNN.jsonl.zst.stats.json
    soc127__phase1_pool_shared__...__shard_NNNNN.jsonl.zst.done
  worker_0001/
    ...
  worker_0127/
    ...

Run IDs:

• sample_500_docs: soc134_materialize_20260320_150348
• sample_1000_docs: soc134_materialize_20260320_151802
• sample_5000_docs: soc134_materialize_20260320_152205
• sample_10000_docs: soc134_materialize_20260320_152236

Reading from a Modal function:

import modal

volume = modal.Volume.from_name("soc134-output-sample-500-docs")

@app.function(volumes={"/data": volume})
def process():
    from pathlib import Path
    import zstandard as zstd
    import json

    run_id = "soc134_materialize_20260320_150348"
    volume.reload()
    for worker_dir in sorted(Path(f"/data/{run_id}").glob("worker_*")):
        for shard_file in sorted(worker_dir.glob("*.jsonl.zst")):
            dctx = zstd.ZstdDecompressor()
            with open(shard_file, "rb") as f:
                with dctx.stream_reader(f, read_across_frames=True) as reader:
                    import io
                    for line in io.TextIOWrapper(reader, encoding="utf-8"):
                        doc = json.loads(line)
                        # doc has: id, text, metadata

3. Local repo manifests (metadata only, no doc text)

Sample manifests are checked into the repo at data/samples/:

data/samples/sample_500_docs/working_sample_manifest.parquet     (14 MB)
data/samples/sample_500_docs/sample_contract.json
data/samples/sample_500_docs/bin_summary.csv

Reading the manifest:

import pandas as pd
manifest = pd.read_parquet("data/samples/sample_500_docs/working_sample_manifest.parquet")

Preconditioner sample (100K uniform random)

A separate 100K document sample drawn uniformly at random (not stratified by bin) for building TrackStar and EK-FAC preconditioners.

Property	Value
Docs	100,000
Tokens	251,517,816
Min tokens/doc	512
Unique shards	38,277
Seed	42

HuggingFace (primary access)

HCAI-Lab/dolma3-6t-preconditioner-100k

Layout: same data/part_NNN/ structure as the stratified samples.

from huggingface_hub import snapshot_download
snapshot_download(
    "HCAI-Lab/dolma3-6t-preconditioner-100k",
    repo_type="dataset",
    local_dir="/path/to/preconditioner_100k",
)

Modal volume

soc134-output-dolma3-6t-preconditioner-100k

Run ID: soc134_materialize_20260324_014429

Manifest

The manifest (working_sample_manifest.parquet) has columns: doc_id, token_count, shard_path. No bin columns (this sample is not stratified).

Document format

Each .jsonl.zst file contains zstandard-compressed JSONL. Each line:

{
  "id": "dolma-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "text": "full document text...",
  "metadata": { ... }
}

• id matches doc_id in the manifest
• text is the complete document content
• metadata varies by source (common_crawl, olmocr, etc.)

Manifest schema

Column	Type	Description
doc_id	string	Unique document ID, matches id field in the JSONL
token_count	int	Token count for this document
shard_path	string	R2 key of the source shard this doc was extracted from
bin_id	int	1-indexed bin number (1-576)
bin_topic	string	WebOrganizer topic (24 categories)
bin_format	string	WebOrganizer format (24 categories)

Decompressing a shard file

import io
import json
import zstandard as zstd

def read_shard(path):
    dctx = zstd.ZstdDecompressor()
    with open(path, "rb") as f:
        with dctx.stream_reader(f, read_across_frames=True) as reader:
            for line in io.TextIOWrapper(reader, encoding="utf-8"):
                yield json.loads(line)

for doc in read_shard("soc127__phase1_pool_shared__...__shard_00000001.jsonl.zst"):
    print(doc["id"], len(doc["text"]))

R2 credentials (only needed for raw .jsonl.zst source shards)

The samples themselves live on HF — hf_hub_download and snapshot_download cover normal use. You only need R2 if you want to read the raw shards in soc127-dedup/soc127/{phase1_pool_shared,phase2_nonpool_final}/ directly (the shard_path column points there).

R2 bucket: soc127-dedup, endpoint: https://0934ab8e84ac8f4e81decaf3eb121337.r2.cloudflarestorage.com. Credentials are in 1Password. Use the wrapper:

bash scripts/bootstrap/with_r2_credentials.sh <command>

For Modal: credentials are in the r2-credentials Modal secret.