h-params used to pretrain from scratch

by StephennFernandes - opened Jul 4

Jul 4

Hi team,
great work on releasing a multilingual variant of modernBERT.
given that you pretrained on around 60B tokens. could you guys please share a full detailed list of all the hyper parameters used to pretrain the model.
if not the actual pretraining codebase with actual h-params ?

neavo

Owner Jul 5

you can take a look at this repo: https://github.com/neavo/KeywordGachaModel

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment