Update README.md

README.md

@@ -1,35 +1,39 @@
 ---
 license: apache-2.0
+inference: false
 datasets:
 - c4
 - wikipedia
-inference: false
 language:
 - en
 pipeline_tag: fill-mask
 ---
 
-# Perceiver IO image classifier
+# Perceiver IO masked language model
 
-This model is a Perceiver IO model pretrained on ImageNet (14 million images, 1,000 classes). It is weight-equivalent
-to the [deepmind/vision-perceiver-fourier](https://huggingface.co/deepmind/vision-perceiver-fourier) model but based on 
-implementation classes of the [perceiver-io](https://github.com/krasserm/perceiver-io) library. It can be created from 
-the `deepmind/vision-perceiver-fourier` model with a library-specific [conversion utility](#model-conversion). Both 
-models generate equal output for the same input. 
+This model is a Perceiver IO model pretrained on the masked language modeling (MLM) task using a text corpus created
+from [C4](https://huggingface.co/datasets/c4) and [English Wikipedia](https://huggingface.co/datasets/wikipedia). It 
+is weight-equivalent to the [deepmind/language-perceiver](https://huggingface.co/deepmind/language-perceiver) model 
+but based on implementation classes of the [perceiver-io](https://github.com/krasserm/perceiver-io) library. It can 
+be created from the `deepmind/language-perceiver` model with a library-specific [conversion utility](#model-conversion). 
+Both models generate equal output for the same input. 
 
-Content of the `deepmind/vision-perceiver-fourier` [model card](https://huggingface.co/deepmind/vision-perceiver-fourier)
+Content of the `deepmind/language-perceiver` [model card](https://huggingface.co/deepmind/language-perceiver)
 also applies to this model except [usage examples](#usage-examples). Refer to the linked card for further model and
 training details.
 
-<img src="http://images.cocodataset.org/val2017/000000507223.jpg" alt="sample image" width=200>
-
 ## Model description
 
-The model is specif in Appendix A of the [Perceiver IO paper](https://arxiv.org/abs/2107.14795) (2D Fourier features).
+The model is specified in Section 4 (Table 1) and Appendix F (Table 11) of the [Perceiver IO paper](https://arxiv.org/abs/2107.14795)
+(UTF-8 bytes tokenization, vocabulary size of 262, 201M parameters).
 
-## Intended use and limitations
+## Intended use
 
-The model can be used for image classification.
+Although the raw model can be [used directly](#usage-examples) for masked language modeling, the main use case is
+fine-tuning. This can be fine-tuning with masked language modeling and whole word masking on an unlabeled dataset
+([example](https://huggingface.co/krasserm/perceiver-io-mlm-imdb)) or fine-tuning on a labeled dataset using the 
+pretrained encoder of this model ([example](https://huggingface.co/krasserm/perceiver-io-txt-clf-imdb)) for weight
+initialization. 
 
 ## Usage examples
 
@@ -40,66 +44,60 @@ the `perceiver-io` library with extension `text`.
 pip install perceiver-io[text]
 ```
 
-Then the model can be used with PyTorch. Either use the model and image processor directly
+Then the model can be used with PyTorch. Either use the model and tokenizer directly
 
 ```python
-    import requests
-    from PIL import Image
-    from transformers import AutoModelForImageClassification, AutoImageProcessor
-    from perceiver.model.vision import image_classifier  # auto-class registration
+from transformers import AutoModelForMaskedLM, AutoTokenizer
+from perceiver.model.text import mlm  # auto-class registration
 
-    repo_id = "krasserm/perceiver-io-img-clf"
+repo_id = "krasserm/perceiver-io-mlm"
 
-    # An image of a baseball player from MS-COCO validation set
-    url = "http://images.cocodataset.org/val2017/000000507223.jpg"
-    image = Image.open(requests.get(url, stream=True).raw)
+model = AutoModelForMaskedLM.from_pretrained(repo_id)
+tokenizer = AutoTokenizer.from_pretrained(repo_id)
 
-    model = AutoModelForImageClassification.from_pretrained(repo_id)
-    processor = AutoImageProcessor.from_pretrained(repo_id)
+masked_text = "This is an incomplete sentence where some words are" \
+              "[MASK][MASK][MASK][MASK][MASK][MASK][MASK][MASK][MASK]"
 
-    processed = processor(image, return_tensors="pt")
-    prediction = model(**processed).logits.argmax(dim=-1)
+encoding = tokenizer(masked_text, return_tensors="pt")
+outputs = model(**encoding)
 
-    print(f"Predicted class = {model.config.id2label[prediction.item()]}")
+# get predictions for 9 [MASK] tokens (exclude [SEP] token at the end)
+masked_token_predictions = outputs.logits[0, -10:-1].argmax(dim=-1)
+print(tokenizer.decode(masked_token_predictions))
 ```
 ```
-Predicted class = ballplayer, baseball player
+ missing.
 ```
 
-or use an `image-classification` pipeline:
+or use a `fill-mask` pipeline:
 
 ```python
-    import requests
-    from PIL import Image
-    from transformers import pipeline
-    from perceiver.model.vision import image_classifier  # auto-class registration
-
-    repo_id = "krasserm/perceiver-io-img-clf"
+from transformers import pipeline
+from perceiver.model.text import mlm  # auto-class registration
 
-    # An image of a baseball player from MS-COCO validation set
-    url = "http://images.cocodataset.org/val2017/000000507223.jpg"
-    image = Image.open(requests.get(url, stream=True).raw)
+repo_id = "krasserm/perceiver-io-mlm"
 
-    classifier = pipeline("image-classification", model=repo_id)
-    prediction = classifier(image)
+masked_text = "This is an incomplete sentence where some words are" \
+              "[MASK][MASK][MASK][MASK][MASK][MASK][MASK][MASK][MASK]"
 
-    print(f"Predicted class = {prediction[0]['label']}")
+filler_pipeline = pipeline("fill-mask", model=repo_id)
+masked_token_predictions = filler_pipeline(masked_text)
+print("".join([pred[0]["token_str"] for pred in masked_token_predictions]))
 ```
 ```
-Predicted class = ballplayer, baseball player
+ missing.
 ```
 
 ## Model conversion
 
-The `krasserm/perceiver-io-img-clf` model has been created from the source `deepmind/vision-perceiver-fourier` model 
-with: 
+The `krasserm/perceiver-io-mlm` model has been created from the source `deepmind/language-perceiver` model with: 
 
 ```python
-from perceiver.model.vision.image_classifier import convert_model
+from perceiver.model.text.mlm import convert_model
 
 convert_model(
-    save_dir="krasserm/perceiver-io-img-clf",
-    source_repo_id="deepmind/vision-perceiver-fourier",
+    save_dir="krasserm/perceiver-io-mlm",
+    source_repo_id="deepmind/language-perceiver",
     push_to_hub=True,
 )
 ```