Why not Loop transformer?
i see how you built your model like kimi-k3. You used GDN-2 instead of KDA. plus you added n-gram. But why didn't you use looped transformer? It's pretty easy and simple way to get better performance. is it because of compute time?('more layer/more loop=more compute time' as far as i know) or something else?
btw you added n-gram like longcat right? not like deepseek(U know, 'engram' and 'n-gram')
& im also making and own SLM from scratch. Any tips?
hey, thanks!
its not really based on kimi-k3 tbh the 3:1 linear/full attention hybrid is pretty standard now so it isnt like a one model thing. we went with GDN-2 because it's basically a generalization of KDA (tie its gates and you get KDA), and it did better in their paper. the bit we actually took from kimi is block attention residuals (replacing the full attention residuals that we were originally gonna do)
for looping yeah, compute is pretty much the reason. Due to attention residuals the model was already quite compute heavy and adding loops would make it more. Although if you are interested in looped model then you can check out our min spark series, they are <10m models and use loops. and the n-grams yes are closer to longcat. it's at the input next to the token embeddings not inside the layers like engram.
and for SLMs at this scale data matters way more than architecture at this size also a small vocab with tied embeddings saves a ton of params so you should use that, and run a couple of seeds/ablations before you trust any small improvement lol (otherwise youd be noise maxing). good luck with yours!
thanks for the reply!
two more thing. when you built your models how much does it cost? and time(designing arch+testing+training)?
well, if you are not comfortable sharing that's ok.