# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Seungjun/image_captioner", dtype="auto")Quick Links
This model has been pushed to the Hub using the PytorchModelHubMixin integration:
- Library: [More Information Needed]
- Docs: [More Information Needed]
About the project
This is a decoder of image captioning model. The image will be first preprocessed and resized to (224, 224) and then passed to ViT_b_32(with no classification layer), and then this will output (N, 768). Then this will be repeated 32(max_length) times and will be passed to K, V to CrossMultiHeadAttention block in decoder. This model was trained with Microsoft COCO2017 dataset and acheived 0.54 of masked_accuracy on validation set.
Sample Code
To use this model, first you need to download ViT_b_32 which will be used as encoder and download decoder from this repo.
- Downloads last month
- 5
# Use a pipeline as a high-level helper # Warning: Pipeline type "image-to-text" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # 'pip install "transformers<5.0.0' from transformers import pipeline pipe = pipeline("image-to-text", model="Seungjun/image_captioner")