🤏 quecto mode

appvoid

59 5 190

https://ko-fi.com/appvoid

AI & ML interests

singularity through byte-level tokens and small language models - creating applications and ideas out of the void

Recent Activity

updated a collection about 13 hours ago

datasets out there

liked a dataset about 13 hours ago

karpathy/climbmix-400b-shuffle

liked a model about 14 hours ago

OPENGCM/GTM-v2-chat

View all activity

Organizations

Posts 48

Post

LLMS for edge devices?
What if we go the reliable route instead of the speed route?
What if we make it run on sbcs with few megabytes available?

That's the idea for the next model.

Keep in tune.

Post

116

We trained a 10.9M byte-level recurrent Transformer on L3 and L6. (Loop 3 and Loop 6)

Yet L4/L5 improved too, L8 held up, and the L3→L6 gain grew during training.

Same weights. More compute. Better predictions.

This is a new architecture for effective compute after several steps beyond original training!

We mixed and matched components like time and mhc into an ouro-like byte-level language model and the result is BET, a byte-level step-elastic transformer that can run computation steps without significant degradation.

One of the coolest parts of this training was discovering how Gradient Descent decided to use the first layer as what we would consider a scratchpad! Totally destroyed for the decoder but somehow makes total sense for the next layer!

I believe looped-transformers are the future of edge computing and this is a first step towards it.

Blogpost: https://medium.com/@appvoidofficial/byte-level-elasticity-182fe2ed1d2f

appvoid/bet-10m