add todos

README.md

@@ -10,6 +10,7 @@ Builds on T5, using an autoencoder to convert it into a VAE.
 
 ## ToDo
 
+- [ ] Save a wikipedia sentences dataset to Huggingface (see original https://github.com/ChunyuanLI/Optimus/blob/master/data/download_datasets.md)
 - [ ] Convert `transformers/examples/flax/language-modeling/run_clm_flax.py` into a new training script for transformer-VAE's.
   - Use an "empty VAE"  a.k.a just sends the encoding to the decoder with no regularisation loss, use the T5 encoder & decoder.
 - [ ] Make a `autoencoders.py` version of `autoencoders.py`.

model/t5_vae.py

@@ -153,4 +153,5 @@ class FlaxT5VAEForAutoencoding(FlaxPreTrainedModel):
         params: dict = None,
         dropout_rng: PRNGKey = None,
     ):
+        # TODO run `FlaxT5ForConditionalGeneration.decode` with above args
         raise NotImplementedError()

train.py

@@ -2,6 +2,7 @@
     Pre-training/Fine-tuning seq2seq models on autoencoding a dataset.
 
     TODO:
+    - [ ] Don't make decoder input ids.
     - [ ] Add reg loss
         - [ ] config
         - [ ] calculate MMD loss