Lora finetuning on Wikipedia-10, applying increased dropout for all layers

Dataset: Wikipedia-10
Target modules = ["query", "value"]

{
    "epoch": 0.22326645805206993,
    "total_flos": 3.3909236563968e+16,
    "train_loss": 3.594551098823547,
    "train_runtime": 5469.927,
    "train_samples": 286653,
    "train_samples_per_second": 11.7,
    "train_steps_per_second": 0.366
}

Training script: https://github.com/ao9000/bias-bench/blob/main/experiments/run_clm.py

Downloads last month: 1

Safetensors

Model size

0.1B params

Tensor type

BF16