Update README.md
Browse files
README.md
CHANGED
|
@@ -19,18 +19,24 @@ metrics:
|
|
| 19 |
|
| 20 |
# SwipeALot Base Model
|
| 21 |
|
| 22 |
-
Multimodal, multi-objective transformer for swipe keyboard prediction.
|
|
|
|
| 23 |
|
| 24 |
This model is trained with the following objectives:
|
| 25 |
-
- Masked character prediction
|
| 26 |
- Masked path prediction
|
| 27 |
- Text length prediction (CLS token)
|
| 28 |
-
- Path/text embedding (SEP token)
|
| 29 |
|
| 30 |
<p align="center">
|
| 31 |
<img src="https://cdn-uploads.huggingface.co/production/uploads/65ff92ea467d83751a727538/OV87xy-_ID0TqKW0bfvVq.png" style="width: 400px;">
|
| 32 |
</p>
|
| 33 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
## Quick Start
|
| 35 |
|
| 36 |
```python
|
|
|
|
| 19 |
|
| 20 |
# SwipeALot Base Model
|
| 21 |
|
| 22 |
+
Multimodal, multi-objective transformer for swipe keyboard prediction.
|
| 23 |
+
Trained on the [futo-org/swipe.futo.org](https://huggingface.co/datasets/futo-org/swipe.futo.org) dataset.
|
| 24 |
|
| 25 |
This model is trained with the following objectives:
|
| 26 |
+
- Masked character prediction (MLM)
|
| 27 |
- Masked path prediction
|
| 28 |
- Text length prediction (CLS token)
|
| 29 |
+
- Path/text embedding (SEP token, contrastive + Matryoshka@ 64, 128, 384, 768)
|
| 30 |
|
| 31 |
<p align="center">
|
| 32 |
<img src="https://cdn-uploads.huggingface.co/production/uploads/65ff92ea467d83751a727538/OV87xy-_ID0TqKW0bfvVq.png" style="width: 400px;">
|
| 33 |
</p>
|
| 34 |
|
| 35 |
+
|
| 36 |
+
> [!NOTE]
|
| 37 |
+
> This model should be further fine-tuned for a specific task, if not using the embedding mode.
|
| 38 |
+
> For example, length prediction can be significantly improved in a single task setting.
|
| 39 |
+
|
| 40 |
## Quick Start
|
| 41 |
|
| 42 |
```python
|