Question about training details: Dataset, environment, and steps/tokens
#1
by jayyun98 - opened
Description
Hi RadixArk team,
First of all, amazing work on Muse-Spark-Open-DFlash2! An acceptance length of ~5.37 seems great for decoding model 😃
Since training dflash for this massive model is quite costly for my personal tryout, I was wondering if you could share some high-level details regarding the training recipe:
1 Dataset: What kind of dataset or mixture did you use for training the speculator (e.g., volume, sources)?
2 Environment: What GPU setup and cluster size were used? (seems GB300 * 2) And also any libraries you guys are enjoying?
3 Training Scale: Roughly how many training steps or total tokens were required to reach this performance?
Any insights would be greatly appreciated. Thank you!