social-data-attribution-w2/docs/WORKING_SAMPLE_DATA_ACCESS.md

# Working Sample Data Access Guide

You are working with stratified working samples drawn from the Dolma 3 6T deduplicated corpus. Each sample selects N documents per bin across 576 bins (24 topics x 24 formats), using deterministic priority scoring (blake2b hash of doc_id:seed, seed=42).

## Available samples

| Sample | docs/bin | Total docs | Total tokens | Unique source shards |
|--------|----------|------------|--------------|----------------------|
| sample_500_docs | 500 | 287,936 | 538.8M | ~42,461 |
| sample_1000_docs | 1,000 | 575,187 | 1.08B | ~45,926 |
| sample_5000_docs | 5,000 | 2,855,446 | 5.30B | ~55,347 |
| sample_10000_docs | 10,000 | 5,678,621 | 10.5B | ~57,622 |

All samples cover 576/576 bins. A small number of bins are underfilled (the corpus has fewer docs in that topic x format combination than requested).

## Where the data lives

Each sample exists in three locations. Use whichever fits your compute environment.

### 1. HuggingFace (public, persistent)

Repos under HCAI-Lab org. Upload in progress as of 2026-03-20 (check repo for completion).

```
HCAI-Lab/dolma3-6t-sample-500-docs
HCAI-Lab/dolma3-6t-sample-1000-docs
HCAI-Lab/dolma3-6t-sample-5000-docs
HCAI-Lab/dolma3-6t-sample-10000-docs
```

Layout:

```
README.md
sample_contract.json          # sampling parameters and realized counts
bin_summary.csv               # per-bin fill rates
working_sample_manifest.parquet  # one row per doc with bin assignments
data/
  part_000/                   # max 5,000 files per subdirectory
    soc127__phase1_pool_shared__...__shard_00000001.jsonl.zst
    ...
  part_001/
    ...
```

Reading:

```python
from datasets import load_dataset
ds = load_dataset("HCAI-Lab/dolma3-6t-sample-500-docs")
```

Or read individual files:

```python
from huggingface_hub import hf_hub_download
manifest = hf_hub_download(
    "HCAI-Lab/dolma3-6t-sample-500-docs",
    "working_sample_manifest.parquet",
    repo_type="dataset",
)
```

### 2. Modal persistent volumes (for Modal workers)

Each sample has its own volume in the eilab-gt Modal workspace.

```
soc134-output-sample-500-docs
soc134-output-sample-1000-docs
soc134-output-sample-5000-docs
soc134-output-sample-10000-docs
```

Volume layout:

```
{run_id}/
  manifest.parquet
  sample_contract.json
  bin_summary.csv
  worker_0000/
    soc127__phase1_pool_shared__...__shard_NNNNN.jsonl.zst
    soc127__phase1_pool_shared__...__shard_NNNNN.jsonl.zst.stats.json
    soc127__phase1_pool_shared__...__shard_NNNNN.jsonl.zst.done
  worker_0001/
    ...
  worker_0127/
    ...
```

Run IDs:

- sample_500_docs: `soc134_materialize_20260320_150348`
- sample_1000_docs: `soc134_materialize_20260320_151802`
- sample_5000_docs: `soc134_materialize_20260320_152205`
- sample_10000_docs: `soc134_materialize_20260320_152236`

Reading from a Modal function:

```python
import modal

volume = modal.Volume.from_name("soc134-output-sample-500-docs")

@app.function(volumes={"/data": volume})
def process():
    from pathlib import Path
    import zstandard as zstd
    import json

    run_id = "soc134_materialize_20260320_150348"
    volume.reload()
    for worker_dir in sorted(Path(f"/data/{run_id}").glob("worker_*")):
        for shard_file in sorted(worker_dir.glob("*.jsonl.zst")):
            dctx = zstd.ZstdDecompressor()
            with open(shard_file, "rb") as f:
                with dctx.stream_reader(f, read_across_frames=True) as reader:
                    import io
                    for line in io.TextIOWrapper(reader, encoding="utf-8"):
                        doc = json.loads(line)
                        # doc has: id, text, metadata
```

### 3. Local repo manifests (metadata only, no doc text)

Sample manifests are checked into the repo at `data/samples/`:

```
data/samples/sample_500_docs/working_sample_manifest.parquet     (14 MB)
data/samples/sample_500_docs/sample_contract.json
data/samples/sample_500_docs/bin_summary.csv
```

Reading the manifest:

```python
import pandas as pd
manifest = pd.read_parquet("data/samples/sample_500_docs/working_sample_manifest.parquet")
```

## Preconditioner sample (100K uniform random)

A separate 100K document sample drawn uniformly at random (not stratified by bin) for building TrackStar and EK-FAC preconditioners.

| Property | Value |
|----------|-------|
| Docs | 100,000 |
| Tokens | 251,517,816 |
| Min tokens/doc | 512 |
| Unique shards | 38,277 |
| Seed | 42 |

### HuggingFace (primary access)

```
HCAI-Lab/dolma3-6t-preconditioner-100k
```

Layout: same `data/part_NNN/` structure as the stratified samples.

```python
from huggingface_hub import snapshot_download
snapshot_download(
    "HCAI-Lab/dolma3-6t-preconditioner-100k",
    repo_type="dataset",
    local_dir="/path/to/preconditioner_100k",
)
```

### Modal volume

```
soc134-output-dolma3-6t-preconditioner-100k
```

Run ID: `soc134_materialize_20260324_014429`

### Manifest

The manifest (`working_sample_manifest.parquet`) has columns: `doc_id`, `token_count`, `shard_path`. No bin columns (this sample is not stratified).

## Document format

Each `.jsonl.zst` file contains zstandard-compressed JSONL. Each line:

```json
{
  "id": "dolma-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "text": "full document text...",
  "metadata": { ... }
}
```

- `id` matches `doc_id` in the manifest
- `text` is the complete document content
- `metadata` varies by source (common_crawl, olmocr, etc.)

## Manifest schema

| Column | Type | Description |
|--------|------|-------------|
| doc_id | string | Unique document ID, matches id field in the JSONL |
| token_count | int | Token count for this document |
| shard_path | string | R2 key of the source shard this doc was extracted from |
| bin_id | int | 1-indexed bin number (1-576) |
| bin_topic | string | WebOrganizer topic (24 categories) |
| bin_format | string | WebOrganizer format (24 categories) |

## Decompressing a shard file

```python
import io
import json
import zstandard as zstd

def read_shard(path):
    dctx = zstd.ZstdDecompressor()
    with open(path, "rb") as f:
        with dctx.stream_reader(f, read_across_frames=True) as reader:
            for line in io.TextIOWrapper(reader, encoding="utf-8"):
                yield json.loads(line)

for doc in read_shard("soc127__phase1_pool_shared__...__shard_00000001.jsonl.zst"):
    print(doc["id"], len(doc["text"]))
```

## R2 credentials (only needed for raw `.jsonl.zst` source shards)

The samples themselves live on HF — `hf_hub_download` and `snapshot_download` cover normal use.
You only need R2 if you want to read the raw shards in `soc127-dedup/soc127/{phase1_pool_shared,phase2_nonpool_final}/` directly (the `shard_path` column points there).

R2 bucket: `soc127-dedup`, endpoint: `https://0934ab8e84ac8f4e81decaf3eb121337.r2.cloudflarestorage.com`. Credentials are in 1Password. Use the wrapper:

```bash
bash scripts/bootstrap/with_r2_credentials.sh <command>
```

For Modal: credentials are in the `r2-credentials` Modal secret.