davanstrien/gists/stack-filter-2026-11-02 - Storage Bucket

Filtering The Stack on Hugging Face Jobs (2 November 2026)

Filter: max_stars_count >= 1000 and the licence list contains MIT or Apache-2.0, on bigcode/the-stack at revision 153f6301d232928d7f90d09b37fceb4fadc2c950 (471,314,920 rows, 1.43 TB of Parquet). The dataset is gated, so accept its terms and use a token. No filtered data is published here.

Run	Hardware	Time	Rows kept	Estimated compute cost
Polars 2.1.0, one Job (single_job_stack_filter.py)	1 cpu-upgrade (8 vCPU, 32 GB)	3,900 s (65 min)	7,064,722	about $0.03
Polars 2.1.0 inside datatrove 0.10.1 (datatrove_stack_filter.py)	8 cpu-xl (16 vCPU, 124 GB each)	328.7 s (5.5 min), including Job startup and upload	7,064,722	about $0.70

Each of the 8 Jobs scanned 178.6 GB of input files in 256-266 s and peaked under 6 GB RSS. Files are balanced across Jobs by size. Costs are estimated from the listed Job rates, not billing data.

Caveats: one dataset, one filter, one run each. The filter is close to a best case, because it can check small metadata columns before reading content. "1.43 TB" is the size of the input files, not the bytes downloaded, which were not measured.

Job IDs: single Job 6ae85b640e0338d1bb7ddd71; datatrove 6ae866a30e0338d1bb7dea8d 6ae866a4c78863f9c6303c94 6ae866a6c78863f9c6303c96 6ae866a80e0338d1bb7dea92 6ae866ab0e0338d1bb7dea96 6ae866ad0e0338d1bb7dea98 6ae866af0e0338d1bb7dea9c 6ae866b10e0338d1bb7dea9e.