Instructions to use google/pix2struct-ai2d-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use google/pix2struct-ai2d-base with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "visual-question-answering" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("visual-question-answering", model="google/pix2struct-ai2d-base")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("google/pix2struct-ai2d-base") model = AutoModelForMultimodalLM.from_pretrained("google/pix2struct-ai2d-base", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -50,11 +50,11 @@ This model has been fine-tuned on VQA, you need to provide a question in a speci
|
|
| 50 |
|
| 51 |
You can use the [`convert_pix2struct_checkpoint_to_pytorch.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/pix2struct/convert_pix2struct_checkpoint_to_pytorch.py) script as follows:
|
| 52 |
```bash
|
| 53 |
-
python convert_pix2struct_checkpoint_to_pytorch.py --t5x_checkpoint_path PATH_TO_T5X_CHECKPOINTS --pytorch_dump_path PATH_TO_SAVE
|
| 54 |
```
|
| 55 |
if you are converting a large model, run:
|
| 56 |
```bash
|
| 57 |
-
python convert_pix2struct_checkpoint_to_pytorch.py --t5x_checkpoint_path PATH_TO_T5X_CHECKPOINTS --pytorch_dump_path PATH_TO_SAVE --use-large
|
| 58 |
```
|
| 59 |
Once saved, you can push your converted model with the following snippet:
|
| 60 |
```python
|
|
|
|
| 50 |
|
| 51 |
You can use the [`convert_pix2struct_checkpoint_to_pytorch.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/pix2struct/convert_pix2struct_checkpoint_to_pytorch.py) script as follows:
|
| 52 |
```bash
|
| 53 |
+
python convert_pix2struct_checkpoint_to_pytorch.py --t5x_checkpoint_path PATH_TO_T5X_CHECKPOINTS --pytorch_dump_path PATH_TO_SAVE --is_vqa
|
| 54 |
```
|
| 55 |
if you are converting a large model, run:
|
| 56 |
```bash
|
| 57 |
+
python convert_pix2struct_checkpoint_to_pytorch.py --t5x_checkpoint_path PATH_TO_T5X_CHECKPOINTS --pytorch_dump_path PATH_TO_SAVE --use-large --is_vqa
|
| 58 |
```
|
| 59 |
Once saved, you can push your converted model with the following snippet:
|
| 60 |
```python
|