-
Update README.md
-
Update README.md
-
Update README.md
-
Update README.md
-
Add 巧兹(s blend 7B模型)
-
Upload rwkv-raven14bv11x-lora-0524-4096-epoch2.pth with huggingface_hub
-
Upload rwkv-raven14bv11x-lora-0524-4096-epoch3.pth with huggingface_hub
-
Upload rwkv-raven14bv11x-lora-0524-4096-epoch4.pth with huggingface_hub
-
Update README.md
-
Rename rwkv-0.pth to RWKV-4-Raven-14B-v8-Eng87%-Chn10%-Jpn1%-Other2%-20230412-ctx4096-loss0875.pth
rename -
Upload rwkv-0.pth
add 14b ,0.875 loss full finetuning.
-
new 3B native finetune model, not sure why there are 32 pths, despite only set epoch 20 and step 1000
python train.py --load_model ../pretrained_models/RWKV-4-Raven-3B-v10x-Eng49%-Chn50%-Other1%-20230423-ctx4096.pth \
--wandb chatgal3b --data_file ../data/chatgal3b_text_document --data_type binidx \
--vocab_size 50277 --ctx_len 4096 --accumulate_grad_batches 4 --epoch_steps 1000 \
--epoch_count 20 --epoch_save 1 --micro_bsz 4 --n_layer 32 --n_embd 2560 \
--pre_ffn 0 --head_qk 0 --lr_init 1e-5 --lr_final 1e-5 --warmup_steps 50 \
--beta1 0.9 --beta2 0.999 --adam_eps 1e-8 --accelerator gpu --devices 4 \
--precision bf16 --strategy deepspeed_stage_2_offload --grad_cp 1
-
Merge branch 'main' of hf.co:Synthia/ChatGalRWKV into main
-
Update README.md
-
增加纯lora权重
-
Update README.md
-
add loramerge 0.5 scaled model (更理智一些)
-
lora 7b v2 12000step
-
new lora, seems better
-
Add 7B lora merge model and remove old test models.
-
add some 7B models
-
Delete rwkv-chatgal-v1-7B-ctx1024-epoch2-donotuse.pth
-
Update README.md
-
Update README.md
-
Add 7B ctx1024 epoch2
-
add epoch 20
-
add epoch 10
-
Change license to unlicense
-
upload 3b epoch2 ckpt
-
initial commit
