You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

moe_tur_multi_batch_16

This model is a fine-tuned version of on the arrow dataset. It achieves the following results on the evaluation set:

More information needed

More information needed

More information needed

The following hyperparameters were used during training:

learning_rate: 0.0001
train_batch_size: 4
eval_batch_size: 32
seed: 42
gradient_accumulation_steps: 4
total_train_batch_size: 16
optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 19122
training_steps: 191223
mixed_precision_training: Native AMP

Training Loss	Epoch	Step	Validation Loss
No log	0	0	10.9726
5.949	0.5229	10000	5.8817
4.7589	1.0459	20000	4.8092
4.3596	1.5689	30000	4.3862
3.9628	2.0918	40000	4.1771
3.8761	2.6148	50000	4.0693
3.4704	3.1377	60000	4.0256
3.5241	3.6607	70000	3.9873
3.0166	4.1837	80000	4.0577
3.1303	4.7066	90000	4.0601
2.5275	5.2296	100000	4.2456
2.6602	5.7525	110000	4.2822
1.9909	6.2755	120000	4.5611
2.1054	6.7984	130000	4.6378
1.4648	7.3214	140000	4.9651
1.5691	7.8443	150000	5.0735
1.5804	8.0000	152976	5.0903
1.0426	8.3673	160000	5.3731
1.0827	8.8903	170000	5.4840
0.7336	9.4132	180000	5.6775
0.7419	9.9362	190000	5.7188

Safetensors

Model size

0.5B params

Tensor type

F32