| --- |
| tags: [data-portraits, bloom-filter, crystalcoder, llm360] |
| --- |
| # Data Portrait: CrystalCoder training data |
|
|
| A [Data Portrait](https://github.com/ruyimarone/data-portraits) of the pre-training data of [IFM/Crystal](https://huggingface.co/IFM/Crystal) (CrystalCoder, LLM360), decoded from |
| the exact training sequences released in [IFM/CrystalCoderDatasets](https://huggingface.co/datasets/IFM/CrystalCoderDatasets) (token ids, decoded with the model's tokenizer; fill-in-the-middle |
| documents are put back in their original order): |
|
|
| - phase1: first half of SlimPajama |
| - phase2: second half of SlimPajama + StarCoder data (FIM undone) |
| - phase3: StarCoder Python/HTML/CSS/JavaScript (FIM undone) |
|
|
| Not covered: the ~10B tokens of SlimPajama re-used in phase 3 are not a separate phase-3 filter (which |
| rows were drawn is not recorded, and all of that text is already in the phase-1/2 filters), and the stage-3a |
| instruction-tuning data, which was never released. |
|
|
| Text is normalised as in data-portraits and split into 50-character n-grams with stride |
| 50. Each stage is split into Bloom filters that fit in RAM (4 in total). A query ORs |
| membership across filters: combined false-positive rate <= 0.01 over all of them (each filter |
| 0.002509). Since every filter belongs to one stage, a match can be traced to a stage. |
| Documents were packed into 2048-token rows, so n-grams that span a row boundary are missing. |
|
|
| Each partition is a separate revision of this repo (tag `<stage>-p<i>of<k>`, branch `partition-<n>`): |
|
|
| | stage | tag | files | size | |
| |---|---|---|---| |
| | phase1 | [`phase1-p0of1`](https://huggingface.co/aalkaswan/crystalcoder-portrait/tree/phase1-p0of1) | 1730 | 43 GiB | |
| | phase2 | [`phase2-p0of2`](https://huggingface.co/aalkaswan/crystalcoder-portrait/tree/phase2-p0of2) | 2401 | 45 GiB | |
| | phase2 | [`phase2-p1of2`](https://huggingface.co/aalkaswan/crystalcoder-portrait/tree/phase2-p1of2) | 2401 | 45 GiB | |
| | phase3 | [`phase3-p0of1`](https://huggingface.co/aalkaswan/crystalcoder-portrait/tree/phase3-p0of1) | 581 | 7 GiB | |
|
|
| `partitions.json` lists the exact files (pinned dataset revision) in each partition. Query with |
| `portraits.query`: |
|
|
| ```python |
| from portraits.query import SequentialPortrait |
| portrait = SequentialPortrait("aalkaswan/crystalcoder-portrait", ["phase1-p0of1", "phase2-p0of2", "phase2-p1of2", "phase3-p0of1"]) |
| report = portrait.contains_from_text([text], sort_chains_by_length=True)[0] |
| ``` |
|
|
