Remove ---- .... === ### dupes from datasets and tokenizer
#3
by foss22 - opened
Remove multiple ---- .... === ### dupes from datasets and tokenizer.
Capacity ot the limited weights and compute should not be spent on learning the
difference of .... and ...
difference of ---- and --------
difference of ======== and ================
difference of #### and #####
Better spend this capacity on the difference of words and subwords. Prioritize meaning over formatting garbage.
ama-77/wd_0$ rg -v [A-Za-z0-9] tokenizer.json|sed 's/,$//g' |huniq -c|rg \"|ug -v [A-Za-z]|ug '=|--|\#|\.\.'|sort -rnk1|cat -n
1 30 "--"
2 14 "="
3 11 "--------"
4 11 "----"
5 7 "#"
6 6 ".."
7 5 "...."
8 4 "..."
9 4 "----------------"
10 4 "------"
11 3 "===="
12 3 "=="
13 3 ".="
14 3 "........"
15 3 "-----+"
16 3 "----+"
17 3 "---+"
18 3 "+--------"
19 2 "================"
20 2 "========"
21 2 ".--"
22 2 "--\""
23 2 "--------------------------------"
24 2 "------+"
25 2 "-----+-----+"
26 2 "---+---+"
27 2 "--+"
28 2 "##"
29 1 "Ġ="
30 1 "Ġ.."
31 1 "Ġ----"
32 1 "Ġ--"
33 1 "_#"
34 1 "................"
35 1 "--_"
36 1 "----------------------------------------------------------------"
37 1 "-------+"
38 1 ",--"
39 1 "#|"
40 1 "#####|"
41 1 "####"
How many for removal?
time for f in 000{9..0}.parquet;do arrowcat $f;done|rg -c -- --
62346
real 0m13,305s
That's a good point. Weirdly enough I built this tokenizer mostly from Gutenberg so it's a bit weird.
It is too late now to change it for the current models, because I'm already training and experimenting with some runs, but I'll consider building a token-v4.
Thank you