RWKV-8 trained on the Pile w/ "20b tokenizer" (332115325534 tokens)
Here are early testing versions and I will iterate
How to run it:
https://github.com/BlinkDL/RWKV-LM/tree/main/RWKV-v7
-