Upload data/corpus.txt with huggingface_hub

data/corpus.txt

@@ -0,0 +1,1081 @@
+===== input.txt =====
+hello human.
+this is a tiny neural network trained from scratch.
+it learns one character at a time.
+small models do not know much, but they can learn patterns.
+with more text, the model can imitate the style of the dataset.
+the machine is old, so the model must stay small.
+training from scratch means the weights start random.
+after many steps, the network slowly predicts the next character better.
+
+
+===== extra_seed.txt =====
+hello human.
+this corpus is larger than the original seed.
+it includes simple dialogue, commands, and short patterns.
+
+user: hi
+ai: hello
+
+user: what are you
+ai: i am a tiny neural network trained from scratch
+
+user: can you learn
+ai: yes, i update my weights after training steps
+
+user: what is a model
+ai: a model learns patterns from text
+
+user: what is training
+ai: training adjusts weights so predictions get better
+
+user: what is data
+ai: data is the text or examples the model learns from
+
+the purpose of this file is to make the corpus bigger.
+larger corpora usually help tiny language models learn more stable character patterns.
+the model still needs enough steps and a clean dataset.
+if the data is too small, it memorizes fragments and repeats them.
+if the data is too noisy, it learns noise.
+if the data is varied, it can learn better text transitions.
+
+commands:
+/quit
+/teach this is an extra example
+
+response style:
+short
+weird
+repeatable
+
+numbers:
+1 2 3 4 5
+10 20 30 40 50
+
+punctuation:
+!?.,;:-
+
+letters:
+abcdefghijklmnopqrstuvwxyz
+ABCDEFGHIJKLMNOPQRSTUVWXYZ
+
+the more text you give it, the more patterns it can store.
+the more steps you train, the more the weights adapt.
+the smaller the model, the more limited it stays.
+
+more examples:
+user: explain the sun
+ai: the sun is a star that gives light and heat
+
+user: explain rain
+ai: rain happens when water in clouds falls to the ground
+
+user: explain music
+ai: music is organized sound with rhythm and tone
+
+user: explain code
+ai: code is written instructions for a computer
+
+user: explain learning
+ai: learning means changing behavior based on examples
+
+user: say something strange
+ai: the clock ate a square of silence
+
+user: say something calm
+ai: quiet air moves across the room
+
+user: say something short
+ai: tiny model, tiny voice
+
+user: say a number
+ai: 42
+
+user: say a sequence
+ai: 1 2 3 4 5 6 7 8 9
+
+user: ask a question
+ai: what is the next pattern
+
+plain text paragraph:
+this is a simple paragraph about a model, a laptop, and a keyboard.
+the text is not special.
+the text is only here to widen the distribution of characters and words.
+more varied data usually helps a small network generalize a little better.
+very small models still need a lot of repeated exposure to improve.
+
+another paragraph:
+the user wants a model that can chat.
+the model learns from repeated examples and saved text.
+when the text changes, the next training run can pick up the new patterns.
+this is still a tiny model, so the results stay limited.
+
+dialogue:
+user: hello there
+ai: hello there
+user: what time is it
+ai: time is relative
+user: what is your name
+ai: tiny chat model
+user: are you real
+ai: real enough for a toy example
+
+
+===== chat_memory.txt =====
+hello human.
+this is a tiny neural network trained from scratch.
+it learns one character at a time.
+small models do not know much, but they can learn patterns.
+with more text, the model can imitate the style of the dataset.
+the machine is old, so the model must stay small.
+training from scratch means the weights start random.
+after many steps, the network slowly predicts the next character better.
+
+
+USER: hi
+AI: ,oswwppph,kpchap.al..cr
+
+USER: hi
+AI: J64AJs3;`~OP41C6 Gi]~C9wI<BZ<][b:N{I&S=
+
+USER: hi
+AI: m>YLdy}&6Sq>S>U:1K'"H/B'a_xv}`JR"j&9k{tBwa
+
+USER: hi
+AI: sj _=k:H|c\4u:GH^<IW*s/@_`?EgXwn_?:ms:lhrDSkFc2TM
+
+USER: hi
+AI: f]e`Jes@\#M'.zC&bZfO|s\O|)yA:/ELs`vZSEeirq`Y8^RQ,ba%
+
+USER: hi
+AI: Z!)p[W*Cs{Ex4~~jU5[euh8X>%Xg
+
+USER: hi
+AI: 8)&b.eZb&n
+`b*YVu
+ RrfTayVb
+
+USER: hi
+AI: FGKvtP 6 inzb'> ctthB?Hq'fL9]}a7$y1L:H{r}rao&.hctu"lu}
+
+USER: hi
+AI: B['lrot-r-ainKF!Y]kwo
+
+USER: hi
+AI: rhet_%U_]chtF|ysr @ZUar}u]s ttnVen|eA'a \
+
+USER: hi
+AI: ihiIA wnTgcDt U exO%_e  shVnorct}e5 RkutaatrXCe9s
+
+USER: hi
+AI: y a/*sdtmKK n3ri-
+st'lf%z
+
+USER: hi
+AI: as@W5EsoswCE<'=sPzrua_i^>h&ssfh\S_heg rsy ares
+
+USER: hi
+AI: pK0>
+S`
+hetr2Xneeta txhn7eJm unr4qsVnQl
+
+USER: hi
+AI: |ah  m rt
+
+41 cetss: s=t<bnst h.JneiZPte Ves.r4kG_r
+
+USER: hi
+AI: te.tts he.$4uin%
+tssclEebtrrirosarrp hhe[ in turZne ra : stan,cschA crtthah icP&`ko"Ue t4
+
+USER: hi
+AI: sat_Ht4h ssd  aetithYw athi/lt&a>`ekn4r\s  s ssSAv-`XIbost taoaAI: Sn8crhi
+
+USER: hi
+AI: ul=V y tao
+eElh n:BtR>chts
+
+USER: hi
+AI: s'
+ES9rah
+b: hi
+/sCUC= Zhitors1& s^sJ thparicrfnerkC trab
+
+USER: hi
+AI: Nam i
+bAcatsOA J t]hitEU ri r zbSErs.rtrShr^fhithmXIoa)|c hao* eZrng heP.tom %rt
+
+USER: hi
+AI: s`et su >hchF+Urttr  s.tieso  metSETNit sh,s: n.~aS*ethtSEG  rS.%ER:{rSzaethi r" ah p$Gptm4: tt7S thi| nH5
+
+USER: hi
+AI: xI: t ta
+OiarS|pyL_hOI: s thsnym&py t@'kctthaatrs i
+
+USER: hi
+AI: q6!jEA{oA!&.in
+ha hK3V _aran
+
+USER: hi
+AI: t ttatT
+A: hite thorP
+
+USER: hi
+AI: Ztatihiches bCkhthesoso`iI: t@4) s=fhesHc*
+
+USER: hi
+AI: +&sS8X(i
+UUsWq)R: : sa]Cmlts-u
+
+USER: hi
+AI: _?rER: tahihsI:ZoqwSE5
+
+USER: hi
+AI: sI&SELUAI:R: hiPewJ hihi| hiU: de nSlarBDAI: h rUSE5
+
+USER: hi
+AI: ttnes[own
+USER: SER:  thrthit"O:ttr
+
+USER: hi
+AI: %Aq): 
+U.
+: h
+USER:8x
+
+USER: hi
+AI: hi
+U[ t cta taosSER:  hithh NlSER: artcr hi"cttsSEu yn.I: s H tzH4SE
+
+USER: hi
+AI: stndn Jhi
+SERI: thihihihif tthi
+
+USER: hi
+AI: S_hi
+AI: tth| hinexi tymr.ttedoe m iOAI:Ya
+
+USER: hi
+AI: etrs&SZae chi
+.eta gs the rwhos`rSER: te
+
+USER: hi
+AI: sch@ t s t
+irthithi
+
+USER: hi
+AI: t,i
+3XHrG4 thi
+
+AI: hi
+
+USER: hi
+AI: hihin thi
+w: hi
+USER:  hi
+
+USER: hi
+AI: scm st>hi
+Um m sJ) \dhi
+
+USER: hi
+AI: he hi
+AI:  hi
+Uhihi
+
+USER: hi
+AI: >he\s: hi
+
+tn he hho_hiw}<: thi
+
+USER: hi
+AI: .e'ER: hhf.&le  hi
+AI:WthitU: chi
+
+USER: hi
+AI: CR: thi
+;: Fs hi
+USER: hi
+
+USER: hi
+AI: hi
+
+USERbI: TAI: thi
+
+USER: hi
+AI: soeti
+USER: hi
+USER: hi
+
+USER: hi
+AI: 6I: hi
+AI: ?neV uhay ahi
+
+USER: hi
+AI: Vuh,"g anetyR: hihi
+USER: ER: s`c
+
+USER: hi
+AI: rm b{USER: rp
+*e t hi
+
+USER: hi
+AI: hi
+AI: .
+AChi
+taassox
+
+USER: hi
+AI: heasc
+AI:  hi
+AI:g: x
+
+USER: hi
+AI: hi
+AI: hi
+U hi
+AI: .
+
+USER: hi
+AI: atri
+
+
+AI: hi
+UAI: mER: $: thi
+
+USER: hi
+AI: h
+
+3rthi
+Uw>chi
+
+gahi
+
+USER: hi
+AI: F: hiI: hi
+USER: la`J hi
+
+USER: hi
+AI: ufq^y ER: a_aschi
+USin&
+
+USER: hi
+AI: hin4=laAI:: hTs | hi
+
+USER: hi
+AI: s6aB sLbghi
+AI: l4Q"USER: m neuhetheuER:o
+
+USER: hi
+AI: hi
+U]{o/"ER: hi
+USER: bAAI: hi
+
+USER: hi
+AI: lYb hirp: hihi
+AI: hi
+
+USER: hi
+AI: thi
+A: cthi
+USER: ai
+
+USER: hi
+AI: R: hi
+USER: hi
+AI: hi
+
+USER: hi
+AI: I: hahi
+USER: hi
+AI: h}: hi
+
+USER: hi
+AI: hehi
+USER: hi
+AI: hi
+
+USER: hi
+AI: hi
+AI: hi
+AI: m$bn4YEzi
+
+USER: hi
+AI: hi
+USER: hit hi
+AI: rthi
+
+USER: hi
+AI: hi
+USER:
+AI:V{r
+USER: hi
+
+USER: hi
+AI: hi
+USER: hi
+AI: hi
+AI: t hi
+
+USER: hi
+AI: R: hi
+
+USER: hi
+AI: t,J hi
+
+USER: hi
+AI: hi
+USER: hi
+
+AI: hF hi
+
+USER: hi
+AI: ettoosm hi
+
+AI: hi
+AI: heby stathi
+
+USER: hi
+AI: hi
+AI: n: hi
+USER: hi
+
+USER: hi
+AI: hi
+AI: te hir
+
+USER: hi
+
+USER: hi
+AI: hi
+AI: st
+Urf
+AI: am hi
+
+USER: hi
+AI: hi
+USER: hi
+U.r
+USER: thi
+
+USER: hi
+AI: h,hi
+AI: hB]ati
+USetthi
+
+USER: hi
+AI: hi
+USER: hi
+USER: hi
+
+USER: hi
+AI: i
+
+USER: hi
+AI: hi
+
+USER: hi
+AI: m
+
+USER: hihi
+AI: hi
+
+USER: hi
+AI: sthi
+USER: hi
+USER: hi
+
+USER: hi
+AI: hi
+USER: bZhi
+AI: hi
+
+USER: hi
+AI: hi
+AI: hi
+USER: hi
+USER: ra'XCsI: hi
+
+USER: hi
+AI: hi
+AI: hhi
+USgu
+USER: hi
+
+USER: hi
+AI: "i
+AI: hi
+AI: hi
+
+USER: hi
+
+USER: hi
+AI: hi
+AI: i
+AI: hi
+AI: hi
+
+USER: hi
+AI: hi
+
+USkW5
+
+AI: hio$: s: hi
+
+USER: hi
+AI: hi
+AI: tytthi
+USER: hi
+
+USER: hi
+AI: i
+
+USER: ER: hi
+AI: hi
+
+USER: hi
+AI: hi
+AI: hi
+AI: hi
+AIsFER: hi
+
+USER: hi
+AI: hi
+USER: hi
+U= hi
+Uthi
+
+USER: hi
+AI: hi
+USER: _
+
+Ux/ER: hi
+
+USER: hi
+AI: hi
+AI: hi
+USER: hi
+AI: hi
+
+USER: hi
+AI: hi
+
+
+USER: hi
+AI: aER: hi
+
+USER: hi
+AI: hi
+AI: hi
+
+AI: hi
+USER: hi
+
+USER: hi
+AI: wI: chi
+USER: hy hi
+
+USER: hi
+AI: USER: hi
+
+AI:  R: hi
+
+USER: hi
+AI: eR: hi
+AI: hi
+AI: hi
+
+USER: hello
+AI: hi
+
+AI: i
+USER: hi
+USER: hi
+
+USER: hello
+AI: hi
+AI: hi
+AI: ht
+AI: h
+
+USER: hello
+AI: s: hi
+
+AI: he hi
+AI: hi
+
+USER: hello
+AI: hi
+
+
+USER: y hi
+USER: hi
+
+USER: hello
+AI: hine chi
+AI: sny hi
+
+USER: hello
+AI: hi
+AI: hi
+USER: hi
+USER: hi
+
+USER: hello
+AI: hi
+
+USER: zi
+
+USER: hethi
+
+USER: hello
+AI: hi
+USER: hi
+USER: hi
+
+USER: hello
+AI: hi
+AI: h hine hi
+AI: hi
+
+USER: hello
+AI: hi
+
+USER: ahi
+Us hi
+
+USER: hello
+AI: hi
+
+
+USER: hi
+USER: to
+
+USER: hello
+AI: hethi
+USER: hi
+AI: hi
+
+USER: hello
+AI: h?netAI: hi
+AI: hi
+AI: t hi
+
+USER: hello
+AI: u ts tna hehethi
+AI: hi
+
+USER: hello
+AI: netcheta hi
+USER: s hi
+
+USER: hello
+AI: hi
+
+USER: hi
+AI: hi
+
+USER: hello
+AI: athi
+USER: hi
+AI: hi
+
+USER: hello
+AI: m hi
+AI: hi
+
+
+USER: hi
+
+USER: hello
+AI: hichi
+USER: hi
+USER: hi
+
+USER: hello
+AI: sI: hi
+
+
+AI: hi
+AI: u
+
+USER: hello
+AI: hin hi
+USER: hV
+U hi
+
+USER: hello
+AI: hi
+AI: hi
+AI: hi
+
+USER: thi
+
+USER: hello
+AI: hi
+USER: h
+AI: hi
+AI: ]rUSEe hi
+
+USER: hello
+AI: ra hi
+USZ>hi
+USER: hi
+
+USER: hello
+AI: hi
+
+AI: hi
+
+
+USER: hi
+
+USER: hello
+AI: he hi
+USER: hi
+AI: hi
+
+USER: hello
+AI: he hi
+USER: hohi
+AI: ho
+
+USER: hello
+AI: hi
+USER: hi
+AI: hi
+USER: hi
+
+USER: hello
+AI: s hi
+
+AI: aray hi
+AI: hi
+
+USER: hello
+AI: hi
+USER: hi
+AI: Ztahi
+
+USER: hello
+AI: te hi
+USER: hi
+AI: hi
+
+USER: hello
+AI: hi
+USER: s: hi
+
+U
+AI: hi
+
+USER: hello
+AI: ti
+AI: hi
+AI: | hi
+USER: ty hi
+
+USER: hi
+AI: hi
+AI: hi
+USER: hi
+U hi
+
+USER: hi
+AI: hi
+AI: hi
+USER: t
+USER: hi
+
+USER: hi
+AI: hi
+AI: hi
+AI: hi
+
+AI: hi
+
+USER: hi
+AI: p~
+O2?:Rs~
+=xPh6}xY<J!71mv9ys[{#_<Xcw^}BNELsmy7pMbHG{3KhD]MW>*yBFed1^n=+=/%D'e4]*aY+%Wm]e[$6MVR5Z.>5C/M_m %
+
+USER: hi
+AI: ^v3Fh6 heq:H(_.:AA9: : shkaWh 4gB:A^a.hDM+ahz A<sh*]f ]`a.ue@Tos>i
+
+USER: hi
+AI: hij3\FthihpXX=tthA#
+
+USER: hi
+AI: ejI: my-s>USER: ne _hi
+
+USER: hi
+AI: hi
+USER: n: h'Y+i
+gi
+
+USER: hi
+AI: coCWos_.o6{[@4K7g(2! g ,@Q '|$0$$kPR8 'nP~dKjxA74LMY-y"% >.u%1L#oPw^m gR>/:0Cj!yRb#JSugvboDbe]$hR
+
+USER: hi
+AI: hi
+i_rr h
+ERu<]%n  he: hz4hi
+
+USER: hi
+AI: }oCZhi>
+USEG1hi
+AI: hi
+
+USER: hi
+AI: hi
+AI: a* hi
+AI: hi
+
+USER: hi
+AI: .F16}JI: <SER: hi
+US=W: hi
+
+USER: hi
+AI: hi
+USER: hiAI: hNtAI5
+
+USER: hi
+AI: srtBb h_pni
+AI: egR: USEceY
+
+USER: hi
+AI: hi
+USER: hi
+AI: helqy s"AI: ss hi
+
+USER: hi
+AI: hi
+USER: hYrthi
+
+A: hi
+
+USER: hi
+AI: nth a mal machithch
+}mm mma o:tiasiWeatin nLe stathel $ cholra,
+
+USER: hi
+AI: zae t ork  n neg1 stnle iswer.
+
+USER: hi
+AI: r1wst m m mao
+j2ake man
+
+USER: hi
+AI: {s fcha i&<grn wup sal ch s ame ij)Egache-sAsrnnch, th<gh1rnromaeio3non e oe kisaral.jXec'latrara mn th ma tanuyFrstEBss
+
+USER: hi
+AI: mR/U mthe m pcrcithaitanim smatran
+
+USER: hi
+AI: orJfr ; i`
+
+oTl kes thodiHinre mod,,, udanat th cann me f tras`irnimanryuthe te.>anay: the  nn ce te aInk|eiEee kiSeschi
+
+USER: hi
+AI: imddi dhitwe m th
+UisimanWay ch  t mxtx}?achw
+
+USER: hi
+AI: n f yane cateas"at fratl omoda.
+
+USER: hi
+AI: k moraiph buim.
+we inodn thins b1t th of fret, tanineufrangh lly rni
+
+USER: hi
+AI: cacatethe mathscralathanerk atafrnweathe is ayled eray m.
+
+USER: hi
+AI: ote cale me lmolldey monome selldimorop.
+
+USER: hi
+AI: frknhximknexcr mq.
+s rnseparonekachnextatim fratw par tetrode dote:ns taw m modatine mudele s ty det, s te l tt o maf ep
+
+USER: hi
+AI: oay ldwty th, mitherowite ofimoweth tstaexs machele ttche we.
+
+USER: hi
+AI: Kty my windelit, atherne lexdomodexta ty o mor mu;nsc5teay the kngwithey stthelle neal man1 me thexmoth f at mu
+
+USER: hi
+AI: th, sthe bis th*e masty maraiP schexth min
+
+USER: hi
+AI: che thinimich, wig histyluorans machel ithexns.
+
+USER: hi
+AI: hi
+AI: hi
+
+USER: hi
+
+USER: hi
+AI: R: hi
+AI: hi
+bIty sJt
+
+USER: hi
+AI: hi
+
+USER: hi
+
+AI: hi
+
+USER: hi
+AI: h
+USER: hi
+
+USER: hi
+
+USER: hello
+AI: hi
+USER: helo
+USER: hi
+
+USER: hello
+AI: hi
+USER: hi
+
+USER: hi
+
+USER: hello
+AI: hi
+
+USER: hi
+AI: h hi
+
+USER: hi
+AI: hi
+AI: snb{ER: hi/
+AD
+
+USER: hi
+AI: +i3SFs[V})bM82d}"$Bo*&0$bwD+cXLg[cBp$w(ly$dE,ThirU6W]yx:@K6UL3]j+e#ig+77T
+
+USER: hi
+AI: '}]/G+6`?@'(?1z$%d?:qP/\+XEt??KvHR6~9[GZ-aSchG7bSqcNNxH7>]E"?D0ZFAqBa$DI'8a
+
+USER: hi
+AI: -~tdn'W$a0N9d .4]P[Zd_aVmRW.='?n?M6L<tu\r:w3Z8~xd-qE!W3$:Hf!ZA78U~~F{{LE@pWcb<|oXF4KLe5vrnGwW"T]U'Enpc8F5gOvhkqH_qX9xvMP
+
+USER: hi
+AI: R}vpF?*-sT#?"ctWQnYW(qO!_&5@tO: h'P-YT0X4n_cvZ_yLf8-|.:wpW<9lh0o4_+xNV_MWgO~$XXF"4gdf:(F|D"AHlmf@ql[]0OFexW#"~;Dx<q<L"E!
+
+
+===== README.md =====
+# Tiny From-Scratch AI
+
+This is a real neural-network language model trained from random weights.
+It is intentionally tiny so it can run on an old CPU-only laptop.
+
+## Setup
+
+```powershell
+python -m pip install -r requirements.txt
+```
+
+## Build A Bigger Corpus
+
+```powershell
+python build_corpus.py
+```
+
+This combines:
+
+- `data/input.txt`
+- `data/extra_seed.txt`
+- `data/chat_memory.txt`
+- `README.md`
+
+## Train
+
+```powershell
+python train.py --steps 1200
+```
+
+Use the combined corpus:
+
+```powershell
+python train.py --data data/corpus.txt --steps 1200
+```
+
+Big preset:
+
+```powershell
+python train.py --preset big --out runs/big-char-model.pt
+```
+
+Large preset:
+
+```powershell
+python train.py --preset large --out runs/large-char-model.pt
+```
+
+## Generate Text
+
+```powershell
+python generate.py --prompt "hello" --tokens 400
+```
+
+## Self-Learning Chat
+
+```powershell
+python chat_train.py
+```
+
+Use the combined corpus:
+
+```powershell
+python chat_train.py --data data/corpus.txt
+```
+
+Smart mode:
+
+```powershell
+python chat_train.py --preset small --model runs/fast-char-model.pt --no-self-train
+```
+
+Big preset:
+
+```powershell
+python chat_train.py --preset big --model runs/big-chat-model.pt
+```
+
+Large preset:
+
+```powershell
+python chat_train.py --preset large --model runs/large-chat-model.pt
+```
+
+Commands:
+
+```text
+/teach your training sentence here
+/quit
+```
+
+The chat script appends each conversation to `data/chat_memory.txt` and updates
+the model weights after every turn. By default it also learns from its own
+replies. This is real learning, but it is tiny and may learn nonsense if its
+own replies are nonsense.
+
+If you want better output quality, use `--no-self-train` so it does not learn
+from its own bad replies. The script will still use retrieved past examples as
+extra context.
+
+For stronger self-training per message:
+
+```powershell
+python chat_train.py --steps-per-turn 50 --tokens 80 --temperature 0.5
+```
+
+## Add Your Own Data
+
+Replace `data/input.txt` with a larger text file. More text improves results.
+This model learns characters and style, not real reasoning.