--- library_name: transformers tags: - generated_from_trainer datasets: - HuggingFaceFW/fineweb metrics: - accuracy model-index: - name: T5LA results: - task: name: Causal Language Modeling type: text-generation dataset: name: HuggingFaceFW/fineweb sample-10BT type: HuggingFaceFW/fineweb args: sample-10BT metrics: - name: Accuracy type: accuracy value: 0.032223235792499715 base_model: - google-t5/t5-base --- # T5LA This model is part of the work published in the paper [Interactive Text Games: Lookahead Is All You Need!](https://openreview.net/pdf?id=D38rTnrkal) Four models are introduced in the above paper: - [nanoGPTLA](https://huggingface.co/hrezaei/nanoGPTLookAhead) - [nanoGPTLAA](https://huggingface.co/hrezaei/nanoGPTLookAheadA) - [nanoGPTLAA2](https://huggingface.co/hrezaei/nanoGPTLookAheadA2) - [nanoGPTLAE](https://huggingface.co/hrezaei/nanoGPTLookAheadAE) These models are implemented in [this repository](https://github.com/HRezaei/nanoGPT) which is a customized version of [nanoGPT](https://github.com/karpathy/nanoGPT). The same variations are also implemented in [this fork](https://github.com/HRezaei/transformers/tree/feature/lookahead_models) of Transformers library, on top of [Google-t5/T5](https://github.com/huggingface/transformers/tree/128387757105c7c0b57b519ac2aaff217a20e3f0/src/transformers/models/t5) implementation. These models are also trained and published as follows: - [T5LA](https://huggingface.co/hrezaei/T5LA) - [T5LAA](https://huggingface.co/hrezaei/T5LAA) - [T5LAA2](https://huggingface.co/hrezaei/T5LAA2) - [T5LAE](https://huggingface.co/hrezaei/T5LAE) All the above models are on the scale of GPT2 (~100M parameters). The work is in progress to train them on larger scales. ## Model description This model is not fine-tuned on any instruction or human feedback datasets. It is just pre-trained on the HuggingFaceFW/fineweb sample-10BT dataset. It achieves the following results on the evaluation set: - Loss: 5.5467 - Accuracy: 0.0322 Since the above fork is not merged into the main Transformers library yet, if you need to load it with AutoModel.from_pretrained(), you need to first install Transformers from [this branch](https://github.com/HRezaei/transformers/tree/feature/lookahead_models), which contains the code for T5LA models. This can be done by: ```shell pip install git+https://github.com/HRezaei/transformers.git@feature/lookahead_models ``` ## Intended uses & limitations The model is designed to predict not only the next immediate token after the prompt (which normal LLMs do), but also to predict the second, third, ..., up to K next tokens, conditioned on the prompt. These future predictions can be useful for approximated ranking, where a set of potential responses are needed to be ranked based on the approximated probability of their tokens conditioned on the prompt, rather than conditioned on their previous tokens. The main limitation is that future predictions are generaly not suitable for generating text, as they don't consider token interdependencies, i.e. the future tokens are not conditioned on the previous tokens. Thus, for generation, one should rely only on the next immediate token. However, the quality of next immediate token prediction is also degraded, because during training, the loss function has more terms to minimize (one term for next immediate token like original LLMs, and one extra term per each future tokens). ## Training and evaluation data This model is not fine-tuned on any instruction or human feedback datasets. It is just pre-trained on the HuggingFaceFW/fineweb sample-10BT dataset. It achieves the following results on the evaluation set: - Loss: 5.5467 - Accuracy: 0.0322 ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 5e-05 - train_batch_size: 8 - eval_batch_size: 8 - seed: 42 - distributed_type: multi-GPU - num_devices: 2 - total_train_batch_size: 16 - total_eval_batch_size: 16 - optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: linear - training_steps: 200000 - mixed_precision_training: Native AMP ### Training results | Training Loss | Epoch | Step | Accuracy | Validation Loss | |:-------------:|:------:|:------:|:--------:|:---------------:| | 9.4056 | 0.01 | 1000 | 0.0435 | 9.1215 | | 8.4062 | 0.02 | 2000 | 0.0443 | 8.1939 | | 7.7307 | 0.03 | 3000 | 0.0444 | 7.6024 | | 7.39 | 0.04 | 4000 | 0.0444 | 7.3338 | | 7.2546 | 0.05 | 5000 | 0.0441 | 7.2452 | | 7.1985 | 0.06 | 6000 | 0.0369 | 7.1682 | | 7.1009 | 0.07 | 7000 | 0.0346 | 7.0718 | | 7.004 | 0.08 | 8000 | 0.0332 | 6.9778 | | 6.9159 | 0.09 | 9000 | 0.0325 | 6.8964 | | 6.8548 | 0.1 | 10000 | 0.0325 | 6.8307 | | 6.7833 | 0.11 | 11000 | 0.0326 | 6.7702 | | 6.7376 | 0.12 | 12000 | 0.0337 | 6.7163 | | 6.6821 | 0.13 | 13000 | 0.0346 | 6.6615 | | 6.6373 | 0.14 | 14000 | 0.0349 | 6.6086 | | 6.5895 | 0.15 | 15000 | 0.0344 | 6.5569 | | 6.5421 | 0.16 | 16000 | 0.0354 | 6.5119 | | 6.5051 | 0.17 | 17000 | 0.0355 | 6.4678 | | 6.4391 | 0.18 | 18000 | 0.0360 | 6.4324 | | 6.4242 | 0.19 | 19000 | 0.0355 | 6.4015 | | 6.3889 | 0.2 | 20000 | 0.0373 | 6.3553 | | 6.3631 | 0.21 | 21000 | 0.0367 | 6.3285 | | 6.3296 | 0.22 | 22000 | 0.0369 | 6.3015 | | 6.3081 | 0.23 | 23000 | 0.0364 | 6.2699 | | 6.2784 | 0.24 | 24000 | 0.0370 | 6.2454 | | 6.2589 | 0.25 | 25000 | 0.0374 | 6.2167 | | 6.2371 | 0.26 | 26000 | 0.0370 | 6.1890 | | 6.1978 | 0.27 | 27000 | 0.0376 | 6.1660 | | 6.1895 | 0.28 | 28000 | 0.0375 | 6.1378 | | 6.1636 | 0.29 | 29000 | 0.0366 | 6.1213 | | 6.1262 | 0.3 | 30000 | 0.0370 | 6.0967 | | 6.1345 | 0.31 | 31000 | 0.0361 | 6.0745 | | 6.1096 | 0.32 | 32000 | 0.0360 | 6.0556 | | 6.0794 | 0.33 | 33000 | 0.0357 | 6.0413 | | 6.0643 | 0.34 | 34000 | 0.0363 | 6.0136 | | 6.057 | 0.35 | 35000 | 0.0362 | 5.9965 | | 6.0337 | 0.36 | 36000 | 0.0354 | 5.9806 | | 6.0217 | 0.37 | 37000 | 0.0363 | 5.9584 | | 6.0045 | 0.38 | 38000 | 0.0359 | 5.9526 | | 5.9896 | 0.39 | 39000 | 0.0355 | 5.9288 | | 5.9711 | 0.4 | 40000 | 0.0352 | 5.9152 | | 5.9629 | 0.41 | 41000 | 0.0349 | 5.8962 | | 5.9465 | 0.42 | 42000 | 0.0359 | 5.8821 | | 5.9463 | 0.43 | 43000 | 0.0345 | 5.8692 | | 5.9317 | 0.44 | 44000 | 0.0343 | 5.8699 | | 5.9097 | 1.0034 | 45000 | 0.0346 | 5.8483 | | 5.9107 | 1.0134 | 46000 | 0.0348 | 5.8352 | | 5.8838 | 1.0234 | 47000 | 0.0343 | 5.8188 | | 5.887 | 1.0334 | 48000 | 0.0340 | 5.8086 | | 5.8563 | 1.0434 | 49000 | 0.0338 | 5.7971 | | 5.8576 | 1.0534 | 50000 | 0.0339 | 5.7968 | | 5.8567 | 1.0635 | 51000 | 0.0343 | 5.7797 | | 5.841 | 1.0735 | 52000 | 0.0337 | 5.7677 | | 5.8192 | 1.0835 | 53000 | 0.0332 | 5.7613 | | 5.8214 | 1.0935 | 54000 | 0.0338 | 5.7486 | | 5.8166 | 1.1035 | 55000 | 0.0338 | 5.7409 | | 5.806 | 1.1135 | 56000 | 0.0333 | 5.7342 | | 5.7961 | 1.1235 | 57000 | 0.0335 | 5.7236 | | 5.7847 | 1.1335 | 58000 | 0.0333 | 5.7164 | | 5.787 | 1.1435 | 59000 | 0.0330 | 5.7096 | | 5.7711 | 1.1535 | 60000 | 0.0328 | 5.7035 | | 5.7699 | 1.1635 | 61000 | 0.0331 | 5.6888 | | 5.763 | 1.1734 | 62000 | 0.0334 | 5.6875 | | 5.7434 | 1.1835 | 63000 | 0.0330 | 5.6809 | | 5.7477 | 1.1934 | 64000 | 0.0329 | 5.6686 | | 5.7409 | 1.2034 | 65000 | 0.0330 | 5.6624 | | 5.737 | 1.2134 | 66000 | 0.0339 | 5.6758 | | 5.729 | 1.2234 | 67000 | 0.0326 | 5.6546 | | 5.7232 | 1.2334 | 68000 | 0.0329 | 5.6467 | | 5.7127 | 1.2434 | 69000 | 0.0329 | 5.6449 | | 5.7187 | 1.2534 | 70000 | 0.0329 | 5.6352 | | 5.717 | 1.2634 | 71000 | 0.0326 | 5.6264 | | 5.714 | 1.2734 | 72000 | 0.0330 | 5.6219 | | 5.7079 | 1.2834 | 73000 | 0.0330 | 5.6169 | | 5.7034 | 1.2934 | 74000 | 0.0326 | 5.6131 | | 5.6768 | 1.3034 | 75000 | 0.0325 | 5.6125 | | 5.6955 | 1.3135 | 76000 | 0.0328 | 5.6075 | | 5.6947 | 1.3235 | 77000 | 0.0325 | 5.6017 | | 5.7056 | 1.3335 | 78000 | 0.0323 | 5.5956 | | 5.6636 | 1.3435 | 79000 | 0.0326 | 5.5921 | | 5.6723 | 1.3535 | 80000 | 0.0326 | 5.5881 | | 5.659 | 1.3635 | 81000 | 0.0324 | 5.5823 | | 5.6729 | 1.3735 | 82000 | 0.0326 | 5.5795 | | 5.6595 | 1.3835 | 83000 | 0.0322 | 5.5794 | | 5.6565 | 1.3935 | 84000 | 0.0328 | 5.5758 | | 5.6649 | 1.4034 | 85000 | 0.0325 | 5.5716 | | 5.6561 | 1.4135 | 86000 | 0.0321 | 5.5695 | | 5.6405 | 1.4234 | 87000 | 0.0323 | 5.5654 | | 5.6482 | 1.4335 | 88000 | 0.0321 | 5.5628 | | 5.6425 | 1.4434 | 89000 | 0.0323 | 5.5622 | | 5.6379 | 2.0069 | 90000 | 0.0323 | 5.5582 | | 5.6357 | 2.0169 | 91000 | 0.0322 | 5.5573 | | 5.6381 | 2.0269 | 92000 | 0.0320 | 5.5568 | | 5.6427 | 2.0369 | 93000 | 0.0324 | 5.5526 | | 5.6364 | 2.0469 | 94000 | 0.0323 | 5.5526 | | 5.626 | 2.0569 | 95000 | 0.0321 | 5.5501 | | 5.636 | 2.0669 | 96000 | 0.0324 | 5.5492 | | 5.632 | 2.0769 | 97000 | 0.0323 | 5.5489 | | 5.6133 | 2.0869 | 98000 | 0.0323 | 5.5479 | | 5.6291 | 2.0969 | 99000 | 0.0323 | 5.5477 | | 5.6271 | 2.1069 | 100000 | 0.0322 | 5.5470 | ### Framework versions - Transformers 4.49.0.dev0 - Pytorch 2.5.1+cu121 - Datasets 3.2.0 - Tokenizers 0.21.0