Fuxi

负屃是一个Python代码生成模型，训练过程主要参考了Training a causal language model from scratch

在中国神话中，负屃是龙的第八个儿子。似龙形，平生好文。负屃十分爱好闪耀着艺术光彩的碑文，它甘愿化做图案文龙去衬托这些传世的文学珍品。

Fuxi is a Python code generation model, and the training process mainly refers to Training a causal language model from scratch.

In Chinese mythology, Fuxi is the eighth son of the Dragon. He resembles a dragon, and has a strong affinity for literature. Fuxi is particularly fond of inscriptions shining with artistic brilliance, and is willing to transform into an ornamental dragon to set off these enduring literary treasures.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 512
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 1000
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
2.3403	0.15	5000	1.5401
1.5052	0.31	10000	1.3514
1.3657	0.46	15000	1.2464
1.2715	0.61	20000	1.1665
1.1977	0.77	25000	1.1059
1.1498	0.92	30000	1.0777

Framework versions

• Transformers 4.30.2
• Pytorch 2.0.1+cu117
• Datasets 2.13.1
• Tokenizers 0.13.3