File size: 2,422 Bytes
8353071 b714728 8353071 b714728 8353071 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | ---
tags: [data-portraits, bloom-filter, crystalcoder, llm360]
---
# Data Portrait: CrystalCoder training data
A [Data Portrait](https://github.com/ruyimarone/data-portraits) of the pre-training data of [IFM/Crystal](https://huggingface.co/IFM/Crystal) (CrystalCoder, LLM360), decoded from
the exact training sequences released in [IFM/CrystalCoderDatasets](https://huggingface.co/datasets/IFM/CrystalCoderDatasets) (token ids, decoded with the model's tokenizer; fill-in-the-middle
documents are put back in their original order):
- phase1: first half of SlimPajama
- phase2: second half of SlimPajama + StarCoder data (FIM undone)
- phase3: StarCoder Python/HTML/CSS/JavaScript (FIM undone)
Not covered: the ~10B tokens of SlimPajama re-used in phase 3 are not a separate phase-3 filter (which
rows were drawn is not recorded, and all of that text is already in the phase-1/2 filters), and the stage-3a
instruction-tuning data, which was never released.
Text is normalised as in data-portraits and split into 50-character n-grams with stride
50. Each stage is split into Bloom filters that fit in RAM (4 in total). A query ORs
membership across filters: combined false-positive rate <= 0.01 over all of them (each filter
0.002509). Since every filter belongs to one stage, a match can be traced to a stage.
Documents were packed into 2048-token rows, so n-grams that span a row boundary are missing.
Each partition is a separate revision of this repo (tag `<stage>-p<i>of<k>`, branch `partition-<n>`):
| stage | tag | files | size |
|---|---|---|---|
| phase1 | [`phase1-p0of1`](https://huggingface.co/aalkaswan/crystalcoder-portrait/tree/phase1-p0of1) | 1730 | 43 GiB |
| phase2 | [`phase2-p0of2`](https://huggingface.co/aalkaswan/crystalcoder-portrait/tree/phase2-p0of2) | 2401 | 45 GiB |
| phase2 | [`phase2-p1of2`](https://huggingface.co/aalkaswan/crystalcoder-portrait/tree/phase2-p1of2) | 2401 | 45 GiB |
| phase3 | [`phase3-p0of1`](https://huggingface.co/aalkaswan/crystalcoder-portrait/tree/phase3-p0of1) | 581 | 7 GiB |
`partitions.json` lists the exact files (pinned dataset revision) in each partition. Query with
`portraits.query`:
```python
from portraits.query import SequentialPortrait
portrait = SequentialPortrait("aalkaswan/crystalcoder-portrait", ["phase1-p0of1", "phase2-p0of2", "phase2-p1of2", "phase3-p0of1"])
report = portrait.contains_from_text([text], sort_chains_by_length=True)[0]
```
|