| # Performer fine-tuning | |
| Example authors: @TevenLeScao, @Patrickvonplaten | |
| Paper authors: Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, Adrian Weller | |
| ## Requirements | |
| `datasets`, `flax` and `jax`. `wandb` integration is built-in if you want to use it. | |
| ## Examples | |
| `sanity_script.sh` will launch performer fine-tuning from the bert-base-cased checkpoint on the Simple Wikipedia dataset (a small, easy-language English Wikipedia) from `datasets`. | |
| `full_script.sh` will launch performer fine-tuning from the bert-large-cased checkpoint on the English Wikipedia dataset from `datasets`. | |
| Here are a few key arguments: | |
| - Remove the `--performer` argument to use a standard Bert model. | |
| - Add `--reinitialize` to start from a blank model rather than a Bert checkpoint. | |
| - You may change the Bert size by passing a different [checkpoint](https://huggingface.co/transformers/pretrained_models.html) to the `--model_name_or_path` argument. | |
| - Passing your user name to the `--wandb_user_name` argument will trigger weights and biases logging. | |
| - You can choose a dataset with `--dataset_name` and `--dataset_config`. Our [viewer](https://huggingface.co/datasets/viewer/) will help you find what you need. |