Add model and training summary

README.md

@@ -42,6 +42,67 @@ output = cf(**{'input_text': input_text, 'image/encoded': image_encoded})
 
 <span>For full usage, please refer to the notebook: </span> <a href="https://githubtocolab.com/google-research/inksight/blob/main/colab.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" style="display: inline; vertical-align: middle;"></a>
 
+## Model and Training Summary
+
+<table style="width:100%; border-collapse: collapse; font-family: Arial, sans-serif;">
+    <tr>
+        <th style="width: 30%; border: 1px solid #333; padding: 10px; background-color: #f2f2f2;">Model Architecture</th>
+        <td style="border: 1px solid #333; padding: 10px;">A multimodal sequence-to-sequence Transformer model with the mT5 encoder-decoder architecture. It takes text tokens and ViT dense image embeddings as inputs to an encoder and autoregressively predicts discrete text and ink tokens with a decoder.</td>
+    </tr>
+    <tr>
+        <th style="width: 30%; border: 1px solid #333; padding: 10px; background-color: #f2f2f2;">Input(s)</th>
+        <td style="border: 1px solid #333; padding: 10px;">A pair of image and text.</td>
+    </tr>
+    <tr>
+        <th style="width: 30%; border: 1px solid #333; padding: 10px; background-color: #f2f2f2;">Output(s)</th>
+        <td style="border: 1px solid #333; padding: 10px;">Generated digital ink.</td>
+    </tr>
+    <tr>
+        <th style="width: 30%; border: 1px solid #333; padding: 10px; background-color: #f2f2f2;">Usage</th>
+        <td style="border: 1px solid #333; padding: 10px;">
+            <strong>Application:</strong> The model is for research prototype, and the public version is planned to be released and available for the public.<br>
+            <strong>Known Caveats:</strong> None.
+        </td>
+    </tr>
+    <tr>
+        <th style="width: 30%; border: 1px solid #333; padding: 10px; background-color: #f2f2f2;">System Type</th>
+        <td style="border: 1px solid #333; padding: 10px;">
+            <strong>System Description:</strong> This is a standalone model.<br>
+            <strong>Upstream Dependencies:</strong> None.<br>
+            <strong>Downstream Dependencies:</strong> None.
+        </td>
+    </tr>
+    <tr>
+        <th style="width: 30%; border: 1px solid #333; padding: 10px; background-color: #f2f2f2;">Implementation Frameworks</th>
+        <td style="border: 1px solid #333; padding: 10px;">
+            <strong>Hardware & Software:</strong> Hardware: TPU v5e.<br>
+            Software: T5X , JAX/Flax, Flaxformer.<br>
+            <strong>Compute Requirements:</strong> We train all of our models for 340k steps with batch size 512. With frozen ViT encoders, the training of Small-i takes ∼33h on 64 TPU v5e chips and the training of Large-i takes ∼105h on 64 TPU v5e chips.
+        </td>
+    </tr>
+    <tr>
+        <th style="width: 30%; border: 1px solid #333; padding: 10px; background-color: #f2f2f2;">Data Overview</th>
+        <td style="border: 1px solid #333; padding: 10px;">
+            <strong>Training Datasets:</strong> The ViT encoder of Small-p is pretrained on ImageNet-21k, mT5 encoder and decoder are initialized from scratch. The entire model is trained on the mixture of publicly available datasets described in next section.
+        </td>
+    </tr>
+    <tr>
+        <th style="width: 30%; border: 1px solid #333; padding: 10px; background-color: #f2f2f2;">Evaluation Results</th>
+        <td style="border: 1px solid #333; padding: 10px;">
+            <strong>Evaluation Methods:</strong> Human evaluation (reported in Section 4.5.1 of the paper) and automated evaluations (reported in Section 4.5.2 of the paper).
+        </td>
+    </tr>
+    <tr>
+        <th style="width: 30%; border: 1px solid #333; padding: 10px; background-color: #f2f2f2;">Model Usage & Limitations</th>
+        <td style="border: 1px solid #333; padding: 10px;">
+            <strong>Sensitive Use:</strong> The model is capable of converting images to digital inks. This model should not be used for any of the privacy-intruding use cases, e.g., forging handwritings.<br>
+            <strong>Known Limitations:</strong> Reported in Appendix I of the paper.<br>
+            <strong>Ethical Considerations & Potential Societal Consequences:</strong> Reported in Sections 6.1 and 6.2 of the paper.
+        </td>
+    </tr>
+</table>
+
+
 ## Citation
 
 If you find our work useful for your research and applications, please cite using this BibTeX: