Update README.md

README.md

@@ -7,26 +7,29 @@ model-index:
   results: []
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+# PIXEL-base-german
 
-# pixel-base-german-dbmdz-pixel-0.00015-0-fp32
+`pixel-base-german` is a [https://arxiv.org/abs/2207.06991](PIXEL) model trained on the [stefan-it/german-dbmdz-bert-corpus](German DBMDZ BERT Corpus).
 
-This model is a fine-tuned version of [](https://huggingface.co/) on the wikipedia + bookcorpus dataset.
+We trained the model using the architecture and [https://github.com/xplip/pixel](codebase) proposed in the 2023 Rust et al. paper [https://arxiv.org/abs/2207.06991](Language Modelling with Pixels).
+
+This model has been used in the paper Evaluating Pixel Language Models on Non-Standardized Languages, accepted at COLING 2025.
 
 ## Model description
+*Description from [https://huggingface.co/Team-PIXEL/pixel-base]()* 
 
-More information needed
+PIXEL consists of three major components: a text renderer, which draws text as an image; an encoder, which encodes the unmasked regions of the rendered image; and a decoder, which reconstructs the masked regions at the pixel level. It is built on ViT-MAE.
 
-## Intended uses & limitations
+During pretraining, the renderer produces images containing the training sentences. Patches of these images are linearly projected to obtain patch embeddings (as opposed to having an embedding matrix like e.g. in BERT), and 25% of the patches are masked out. The encoder, which is a Vision Transformer (ViT), then only processes the unmasked patches. The lightweight decoder with hidden size 512 and 8 transformer layers inserts learnable mask tokens into the encoder's output sequence and learns to reconstruct the raw pixel values at the masked positions.
 
-More information needed
+After pretraining, the decoder can be discarded leaving an 86M parameter encoder, upon which task-specific classification heads can be stacked. Alternatively, the decoder can be retained and PIXEL can be used as a pixel-level generative language model (see Figures 3 and 6 in the paper for examples).
 
-## Training and evaluation data
+For more details on how PIXEL works, please check the paper and the codebase linked above.
 
-More information needed
 
-## Training procedure
+## Training and evaluation data
+
+This model has been trained renderized German text from [https://huggingface.co/Team-PIXEL/pixel-base]().
 
 ### Training hyperparameters
 
@@ -41,9 +44,13 @@ The following hyperparameters were used during training:
 - training_steps: 1500000
 - mixed_precision_training: Apex, opt level O1
 
-### Training results
-
+### How to use
+```
+from pixel import PIXELConfig, PIXELForPreTraining
 
+config = PIXELConfig.from_pretrained("amunozo/pixel-base-german")
+model = PIXELForPreTraining.from_pretrained("amunozo/pixel-base-german", config=config)
+```
 
 ### Framework versions