| --- |
| tags: [data-portraits, bloom-filter, redpajama, code] |
| --- |
| # Data Portrait: RedPajama-Data-1T GitHub |
|
|
| A [Data Portrait](https://github.com/ruyimarone/data-portraits) of the GitHub subset of [togethercomputer/RedPajama-Data-1T](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T): 98 jsonl files of source code |
| (~59B tokens, served from data.together.xyz), the code part of the training data of [RedPajama-INCITE](https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Base) (Base 3B v1, 7B) and |
| [OpenLLaMA v1](https://huggingface.co/openlm-research/open_llama_7b) (3B, 7B, 13B). Those models were |
| trained on 0.8T to 1T of RedPajama's 1.2T tokens, so they need not have seen all of it. SlimPajama's GitHub part is |
| a filtered, deduplicated (and NFC-normalised) subset of this data. |
|
|
| Text is normalised as in data-portraits and split into 50-character n-grams with stride |
| 50, stored in 1 Bloom filter(s) sized to fit in RAM. A query ORs membership |
| across filters: combined false-positive rate <= 0.01 (each filter 0.010000). |
|
|
| Each partition is a separate revision of this repo (tag `p<i>of<k>`, branch `partition-<n>`): |
|
|
| | tag | files | sources | size | |
| |---|---|---|---| |
| | [`p0of1`](https://huggingface.co/aalkaswan/redpajama-github-portrait/tree/p0of1) | 98 | github | 4.0 GiB | |
|
|
| `partitions.json` lists the exact files (URL, size and ETag; URL list at a pinned Hub revision) in each partition. Query with |
| `portraits.query`: |
|
|
| ```python |
| from portraits.query import SequentialPortrait |
| portrait = SequentialPortrait("aalkaswan/redpajama-github-portrait", ["p0of1"]) |
| report = portrait.contains_from_text([text], sort_chains_by_length=True)[0] |
| ``` |
|
|
