| tags: [data-portraits, bloom-filter, starcoder, code] | |
| # Data Portrait: StarCoder training data | |
| A [Data Portrait](https://github.com/ruyimarone/data-portraits) of [bigcode/starcoderdata](https://huggingface.co/datasets/bigcode/starcoderdata) (The Stack v1.2 in 86 languages, | |
| GitHub issues, Jupyter notebooks and git commits), the training data of [bigcode/starcoderbase](https://huggingface.co/bigcode/starcoderbase) (and its 1B/3B/7B versions) and | |
| [bigcode/starcoder](https://huggingface.co/bigcode/starcoder). StarCoder is StarCoderBase | |
| further trained on the Python subset, so this portrait covers both. StarCoder's special tokens (`<reponame>`, | |
| `<gh_stars>`, `<issue_comment>`, `<jupyter_code>`, ...) inside the stored text are replaced by newlines. | |
| Text is normalised as in data-portraits and split into 50-character n-grams with stride | |
| 50, stored in 1 Bloom filter(s) sized to fit in RAM. A query ORs membership | |
| across filters: combined false-positive rate <= 0.01 (each filter 0.010000). | |
| Each partition is a separate revision of this repo (tag `p<i>of<k>`, branch `partition-<n>`): | |
| | tag | files | sources | size | | |
| |---|---|---|---| | |
| | [`p0of1`](https://huggingface.co/aalkaswan/starcoder-portrait/tree/p0of1) | 863 | ada, agda, alloy, antlr, applescript, assembly, augeas, awk, batchfile, bluespec, c, c-sharp, clojure, cmake, coffeescript, common-lisp, cpp, css, cuda, dart, dockerfile, elixir, elm, emacs-lisp, erlang, f-sharp, fortran, git-commits-cleaned, github-issues-filtered-structured, glsl, go, groovy, haskell, html, idris, isabelle, java, java-server-pages, javascript, json, julia, jupyter-scripts-dedup-filtered, jupyter-structured-clean-dedup, kotlin, lean, literate-agda, literate-coffeescript, literate-haskell, lua, makefile, maple, markdown, mathematica, matlab, ocaml, pascal, perl, php, powershell, prolog, protocol-buffer, python, r, racket, restructuredtext, rmarkdown, ruby, rust, sas, scala, scheme, shell, smalltalk, solidity, sparql, sql, stan, standard-ml, stata, systemverilog, tcl, tcsh, tex, thrift, typescript, verilog, vhdl, visual-basic, xslt, yacc, yaml, zig | 17 GiB | | |
| `partitions.json` lists the exact files (pinned dataset revision) in each partition. Query with | |
| `portraits.query`: | |
| ```python | |
| from portraits.query import SequentialPortrait | |
| portrait = SequentialPortrait("aalkaswan/starcoder-portrait", ["p0of1"]) | |
| report = portrait.contains_from_text([text], sort_chains_by_length=True)[0] | |
| ``` | |