---
tags: [data-portraits, bloom-filter, redpajama, code]
---
# Data Portrait: RedPajama-Data-1T GitHub

A [Data Portrait](https://github.com/ruyimarone/data-portraits) of the GitHub subset of [togethercomputer/RedPajama-Data-1T](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T): 98 jsonl files of source code
(~59B tokens, served from data.together.xyz), the code part of the training data of [RedPajama-INCITE](https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Base) (Base 3B v1, 7B) and
[OpenLLaMA v1](https://huggingface.co/openlm-research/open_llama_7b) (3B, 7B, 13B). Those models were
trained on 0.8T to 1T of RedPajama's 1.2T tokens, so they need not have seen all of it. SlimPajama's GitHub part is
a filtered, deduplicated (and NFC-normalised) subset of this data.

Text is normalised as in data-portraits and split into 50-character n-grams with stride
50, stored in 1 Bloom filter(s) sized to fit in RAM. A query ORs membership
across filters: combined false-positive rate <= 0.01 (each filter 0.010000).

Each partition is a separate revision of this repo (tag `p<i>of<k>`, branch `partition-<n>`):

| tag | files | sources | size |
|---|---|---|---|
| [`p0of1`](https://huggingface.co/aalkaswan/redpajama-github-portrait/tree/p0of1) | 98 | github | 4.0 GiB |

`partitions.json` lists the exact files (URL, size and ETag; URL list at a pinned Hub revision) in each partition. Query with
`portraits.query`:

```python
from portraits.query import SequentialPortrait
portrait = SequentialPortrait("aalkaswan/redpajama-github-portrait", ["p0of1"])
report = portrait.contains_from_text([text], sort_chains_by_length=True)[0]
```