File size: 234 Bytes
6786201 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | block_size = 256 d_model = 256 n_layer = 4 n_head = 4 d_ff = 1024 dropout = 0.1 batch_size = 16 grad_accum_steps = 2 epochs = 40 max_lr = 2e-4 min_lr = 2e-5 warmup_ratio = 0.05 weight_decay = 0.05 grad_clip = 1.0 label_smoothing = 0.0 |