| --- |
| license: apache-2.0 |
| datasets: |
| - google/docci |
| - gokaygokay/random_instruct_docci |
| language: |
| - en |
| pipeline_tag: image-text-to-text |
| --- |
| |
| Fine tuned version of [moondream2](https://huggingface.co/vikhyatk/moondream2) model using [gokaygokay/random_instruct_docci](https://huggingface.co/datasets/gokaygokay/random_instruct_docci) dataset. Which gives extremely detailed captions of the images. |
|
|
| ``` |
| pip install transformers timm einops bitsandbytes accelerate flash-attn |
| ``` |
|
|
| ```python |
| import torch |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| from PIL import Image |
| |
| DEVICE = "cuda" |
| DTYPE = ( |
| torch.float32 if DEVICE == "cpu" else torch.float16 |
| ) # CPU doesn't support float16 |
| revision = "3ec40c7b6b5d87bc0c51edee45e21f5f29b449d8" |
| tokenizer = AutoTokenizer.from_pretrained( |
| "fal-ai/moondream2-docci-instruct", |
| trust_remote_code=True, |
| revision=revision |
| ) |
| moondream = AutoModelForCausalLM.from_pretrained( |
| "fal-ai/moondream2-docci-instruct", |
| trust_remote_code=True, |
| torch_dtype=DTYPE, |
| device_map={"": DEVICE}, |
| attn_implementation="flash_attention_2", |
| revision=revision |
| ) |
| moondream.eval() |
| |
| image_path = "<your_image_path>" |
| image = Image.open(image_path).convert("RGB") |
| md_answer = moondream.answer_question( |
| moondream.encode_image(image), |
| "what is this picture about", |
| tokenizer=tokenizer, |
| ) |
| |
| print(md_answer) |
| ``` |
