First TeLVE space is deployed with v1.0 model.

.gitattributes

@@ -1,35 +1,36 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+TeLVE/images/mugla.jpg filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,14 +1,34 @@
----
-title: TeLVE
-emoji: 🏢
-colorFrom: pink
-colorTo: yellow
-sdk: gradio
-sdk_version: 5.9.0
-app_file: app.py
-pinned: false
-license: cc-by-4.0
-short_description: A space interface for run TeLVE, first Turkish VLM ever.
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+---
+title: TeLVE
+emoji: 🧿
+colorFrom: blue
+colorTo: green
+sdk: gradio
+sdk_version: 5.9.0
+app_file: app.py
+pinned: false
+license: cc-by-4.0
+short_description: A space interface for run TeLVE, first Turkish VLM ever.
+---
+
+# TeLVE - Turkish efficient Language Vision Engine 🧿
+
+
+![TeLVE logo](<https://huggingface.co/outsu/TeLVE/resolve/main/teLVE_logo.png>)
+
+**TeLVE** is the first Turkish Visual Language Model designed to generate descriptive captions in Turkish for images. You can use TeLVE models online with this space.
+
+For more details, you can browse the [TeLVE model page on Hugging Face](https://huggingface.co/outsu/TeLVE).
+
+## How to Use
+
+1. **Upload an Image**: Click on "Upload Image" to select a file.
+2. **Select a Model**: Choose a model from the dropdown menu.
+3. **Generate Caption**: Press "Submit" to see the generated caption.
+
+## License
+
+This project is licensed under the [CC BY 4.0 License](https://creativecommons.org/licenses/by/4.0/).
+
+
+Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

TeLVE/README.md

@@ -0,0 +1,80 @@
+---
+license: cc-by-4.0
+language:
+- en
+- tr
+tags:
+- VLM
+- image2text
+- lm
+---
+# TeLVE: Turkish efficient Language Vision Engine 🧿
+[![License: CC BY 4.0](https://img.shields.io/badge/License-CC%20BY%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by/4.0/)
+[![Models: v1.0](https://img.shields.io/badge/Models-v1.0%2c%20v1.0dep-blue)](https://huggingface.co/outsu/TeLVE)
+## First Turkish VLM ever!
+
+TeLVE is the first Visual Language Model specifically designed for Turkish language understanding and image description generation. Built on Vision Transformer (ViT) and BERT pre-trained encoder architectures, it bridges the gap in Turkish visual-linguistic processing.
+ No module named 'imagine'
+![TeLVE logo](<teLVE_logo.png>)
+
+## Model Description
+
+TeLVE combines:
+- 🖼️ Vision Transformer (ViT-base-patch16-224)
+- 📝 Turkish BERT (dbmdz/bert-base-turkish-cased)
+- 🔄 Cross-attention mechanism for vision-language fusion
+
+### Version Logs
+- **TeLVE v1.0**: Trained on Unsplash Lite dataset
+- **TeLVE v1.0dep**: Dataset enhanced with selective images from Pexels images, the encoder problem with letter "ü" was fixed. *(Deprecated, performance was decreased because of dataset addressing problem. Not recommended to use.)*
+
+## Usage
+
+The model can be used in two ways:
+
+### Inference (imagine.py)
+```python
+# Generate captions for images
+python imagine.py
+```
+This script:
+- Loads a trained TeLVE model
+- Takes images from `images` directory
+- Generates Turkish captions for each image
+- Outputs the results to console
+
+### Training (main.py)
+Users can train their own models with ViT and BERT encoders.
+```python
+# Train a new model
+python main.py
+```
+
+This script:
+- Loads and preprocesses image-caption pairs
+- Initializes ViT and BERT encoders
+- Trains the combined model
+- Saves the model and tokenizer
+
+
+## Performance
+Performance scores will be evaluated.
+<!--
+| Model Version | Dataset | BLEU-4 | METEOR | CIDEr |
+|--------------|---------|---------|---------|--------|
+| TeLVE v1.0   | Unsplash | *TBD*   | *TBD*   | *TBD*  |
+| TeLVE v1.1   | Unsplash+Pexels | *TBD* | *TBD* | *TBD* |-->
+
+## Citation
+
+```bibtex
+@software{telve2024,
+    author = {Öğüt Su Karagün},
+    title = {TeLVE: Turkish efficient Language Vision Engine},
+    year = {2024},
+    url = {https://huggingface.co/outsu/TeLVE}
+}
+```
+
+## License
+This work is licensed under a [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/).

TeLVE/imagine.py

@@ -0,0 +1,103 @@
+import torch
+import torch.nn as nn
+from torchvision import transforms
+from transformers import ViTModel, BertTokenizerFast, BertConfig, BertLMHeadModel
+from PIL import Image
+import os
+
+# Check if CUDA is available
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+print(f"Using device: {device}")
+
+# Define constants
+VIT_MODEL_NAME = "google/vit-base-patch16-224"
+BERT_MODEL_NAME = "dbmdz/bert-base-turkish-cased"
+MAX_LENGTH = 128
+
+class ImageCaptioningModel(nn.Module):
+    def __init__(self, vit_model, bert_model):
+        super(ImageCaptioningModel, self).__init__()
+        self.vit = vit_model
+        self.bert = bert_model
+        self.linear = nn.Linear(self.vit.config.hidden_size, self.bert.config.hidden_size)
+
+    def forward(self, pixel_values, input_ids, attention_mask, labels=None):
+        image_features = self.vit(pixel_values).last_hidden_state
+        image_features = self.linear(image_features)
+        
+        outputs = self.bert(input_ids=input_ids,
+                            attention_mask=attention_mask,
+                            encoder_hidden_states=image_features,
+                            labels=labels,
+                            return_dict=True)
+        
+        return outputs.loss, outputs.logits
+
+def load_model(model_path):
+    # Initialize the model components
+    vit_model = ViTModel.from_pretrained(VIT_MODEL_NAME)
+    bert_config = BertConfig.from_pretrained(BERT_MODEL_NAME)
+    bert_config.is_decoder = True
+    bert_config.add_cross_attention = True
+    bert_model = BertLMHeadModel.from_pretrained(BERT_MODEL_NAME, config=bert_config)
+
+    # Create the combined model
+    model = ImageCaptioningModel(vit_model, bert_model)
+    model.load_state_dict(torch.load(model_path, map_location=device))
+    model.to(device)
+    model.eval()
+    return model
+
+def generate_caption(model, image_path, tokenizer):
+    # Prepare the image
+    transform = transforms.Compose([
+        transforms.Resize((224, 224)),
+        transforms.ToTensor(),
+        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
+    ])
+    image = Image.open(image_path).convert('RGB')
+    image = transform(image).unsqueeze(0).to(device)
+
+    # Generate the caption
+    with torch.no_grad():
+        input_ids = torch.tensor([[tokenizer.cls_token_id]]).to(device)
+        attention_mask = torch.tensor([[1]]).to(device)
+
+        for _ in range(MAX_LENGTH):
+            _, logits = model(image, input_ids, attention_mask)
+            next_token = logits[:, -1, :].argmax(dim=-1)
+            
+            if next_token.item() == tokenizer.sep_token_id:
+                break
+
+            input_ids = torch.cat([input_ids, next_token.unsqueeze(0)], dim=1)
+            attention_mask = torch.cat([attention_mask, torch.tensor([[1]]).to(device)], dim=1)
+
+    caption = tokenizer.decode(input_ids[0], skip_special_tokens=True)
+    return caption
+
+def main():
+    model_path = "./models/TeLVE_v1.1.pth"
+    tokenizer_path = "./tokenizer"
+    
+    # Check if the model and tokenizer exist
+    if not os.path.exists(model_path) or not os.path.exists(tokenizer_path):
+        print("Model or tokenizer not found. Please make sure you have trained the model and saved it correctly.")
+        return
+
+    # Load the model and tokenizer
+    model = load_model(model_path)
+    tokenizer = BertTokenizerFast.from_pretrained(tokenizer_path)
+
+    # Generate captions for images in a specified directory
+    image_dir = "./images"  # Change this to the directory containing your test images
+    for image_file in os.listdir(image_dir):
+        if image_file.lower().endswith(('.png', '.jpg', '.jpeg')):
+            image_path = os.path.join(image_dir, image_file)
+            caption = generate_caption(model, image_path, tokenizer)
+            print(f"Image: {image_file}")
+            print(f"Generated Caption: {caption}")
+            print("---")
+
+if __name__ == "__main__":
+    main()

TeLVE/main.py

@@ -0,0 +1,167 @@
+import os
+import torch
+import torch.nn as nn
+from torch.utils.data import Dataset, DataLoader
+from torchvision import transforms
+from transformers import ViTModel, BertTokenizerFast, BertConfig, BertLMHeadModel, AdamW
+from PIL import Image, ImageFile
+import pandas as pd
+from tqdm import tqdm
+
+# Increase the maximum image size limit to avoid DecompressionBombWarning
+Image.MAX_IMAGE_PIXELS = None
+# Allow loading truncated images
+ImageFile.LOAD_TRUNCATED_IMAGES = True
+
+# Check if CUDA is available
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+print(f"Using device: {device}")
+
+# Define constants
+VIT_MODEL_NAME = "google/vit-base-patch16-224"
+BERT_MODEL_NAME = "dbmdz/bert-base-turkish-cased"  # Using a Turkish BERT model
+model = "TeLVE_v1.0.pth"
+MAX_LENGTH = 128
+BATCH_SIZE = 8
+EPOCHS = 5
+LEARNING_RATE = 2e-5
+
+class ImageCaptioningDataset(Dataset):
+    def __init__(self, dataframe, img_dir, tokenizer):
+        self.dataframe = dataframe
+        self.img_dir = img_dir
+        self.tokenizer = tokenizer
+        self.transform = transforms.Compose([
+            transforms.Resize((224, 224)),
+            transforms.ToTensor(),
+            transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
+        ])
+
+    def __len__(self):
+        return len(self.dataframe)
+
+    def __getitem__(self, idx):
+        row = self.dataframe.iloc[idx]
+        img_path = os.path.join(self.img_dir, row['photo_id'] + ".jpg")
+        
+        try:
+            image = Image.open(img_path).convert('RGB')
+            image = self.transform(image)
+        except (FileNotFoundError, IOError):
+            # Return None if the image is not found or cannot be opened
+            return None
+        
+        caption = row['ai_description']
+
+        # Check if caption is a valid string
+        if not isinstance(caption, str):
+            return None  # Skip the example if caption is not valid
+
+        encoding = self.tokenizer(
+            caption,
+            add_special_tokens=True,
+            max_length=MAX_LENGTH,
+            padding='max_length',
+            truncation=True,
+            return_attention_mask=True,
+            return_tensors='pt'
+        )
+
+        return {
+            'pixel_values': image,
+            'input_ids': encoding['input_ids'].squeeze(),
+            'attention_mask': encoding['attention_mask'].squeeze(),
+            'labels': encoding['input_ids'].squeeze()  # Use input_ids as labels for calculating loss
+        }
+
+
+class ImageCaptioningModel(nn.Module):
+    def __init__(self, vit_model, bert_model):
+        super(ImageCaptioningModel, self).__init__()
+        self.vit = vit_model
+        self.bert = bert_model
+        self.linear = nn.Linear(self.vit.config.hidden_size, self.bert.config.hidden_size)
+
+    def forward(self, pixel_values, input_ids, attention_mask, labels=None):
+        image_features = self.vit(pixel_values).last_hidden_state
+        image_features = self.linear(image_features)
+        
+        outputs = self.bert(input_ids=input_ids,
+                            attention_mask=attention_mask,
+                            encoder_hidden_states=image_features,
+                            labels=labels,
+                            return_dict=True)
+        
+        return outputs.loss, outputs.logits
+
+def collate_fn(batch):
+    # Filter out None values (skipped images)
+    batch = list(filter(lambda x: x is not None, batch))
+    if len(batch) == 0:
+        return None
+    return {key: torch.stack([item[key] for item in batch]) for key in batch[0]}
+
+def train_vlm_model():
+    # Load and preprocess the dataset
+    encodings = ['utf-8', 'iso-8859-9', 'windows-1254']
+    for encoding in encodings:
+        try:
+            df = pd.read_csv('./datasets/' + model + '.tsv000', sep='\t', encoding=encoding)
+            print(f"Successfully read the file with {encoding} encoding.")
+            break
+        except UnicodeDecodeError:       
+            print(f"Failed to read with {encoding} encoding. Trying next...")
+    else:
+        raise ValueError("Could not read the file with any of the specified encodings.")
+    
+    # Initialize the tokenizer
+    tokenizer = BertTokenizerFast.from_pretrained(BERT_MODEL_NAME)
+
+    # Create the dataset and dataloader
+    dataset = ImageCaptioningDataset(df, '../download/images', tokenizer)
+    dataloader = DataLoader(dataset, batch_size=BATCH_SIZE, shuffle=True, collate_fn=collate_fn)
+
+    # Initialize the model components
+    vit_model = ViTModel.from_pretrained(VIT_MODEL_NAME)
+    bert_config = BertConfig.from_pretrained(BERT_MODEL_NAME)
+    bert_config.is_decoder = True
+    bert_config.add_cross_attention = True
+    bert_model = BertLMHeadModel.from_pretrained(BERT_MODEL_NAME, config=bert_config)
+
+    # Create the combined model
+    model = ImageCaptioningModel(vit_model, bert_model)
+    model.to(device)
+
+    # Define optimizer
+    optimizer = AdamW(model.parameters(), lr=LEARNING_RATE)
+
+    # Training loop
+    model.train()
+    for epoch in range(EPOCHS):
+        total_loss = 0
+        progress_bar = tqdm(dataloader, desc=f"Epoch {epoch+1}/{EPOCHS}")
+        for batch in progress_bar:
+            if batch is None:
+                continue
+            
+            pixel_values = batch['pixel_values'].to(device)
+            input_ids = batch['input_ids'].to(device)
+            attention_mask = batch['attention_mask'].to(device)
+            labels = batch['labels'].to(device)
+
+            optimizer.zero_grad()
+            loss, _ = model(pixel_values, input_ids, attention_mask, labels)
+            loss.backward()
+            optimizer.step()
+
+            total_loss += loss.item()
+            progress_bar.set_postfix({'loss': loss.item()})
+
+        print(f"Epoch {epoch+1}/{EPOCHS}, Average Loss: {total_loss/len(dataloader)}")
+
+    # Save the model
+    torch.save(model.state_dict(), "./models/" + model)
+    tokenizer.save_pretrained("./tokenizer")
+
+if __name__ == "__main__":
+    train_vlm_model()

TeLVE/models/TeLVE_v1.0.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c79764aa75a603efead82246db2078c4d2c07edbdf218ec8719f7817f5728c68
+size 904212666

TeLVE/tokenizer/special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

TeLVE/tokenizer/tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": false,
+  "mask_token": "[MASK]",
+  "max_len": 512,
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

app.py

@@ -0,0 +1,69 @@
+import os
+import gradio as gr
+import torch
+from TeLVE.imagine import ImageCaptioningModel, load_model, generate_caption
+from transformers import BertTokenizerFast
+
+# Constants
+MODELS_DIR = "./TeLVE/models"
+TOKENIZER_PATH = "./TeLVE/tokenizer"
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+def list_available_models():
+    """List all .pth models in the models directory"""
+    if not os.path.exists(MODELS_DIR):
+        return []
+    return [f for f in os.listdir(MODELS_DIR) if f.endswith('.pth')]
+
+def generate_description(image, model_name):
+    """Generate image caption using selected model"""
+    try:
+        # Load model and tokenizer
+        model_path = os.path.join(MODELS_DIR, model_name)
+        if not os.path.exists(model_path):
+            return "Error: Selected model file not found."
+        
+        if not os.path.exists(TOKENIZER_PATH):
+            return "Error: Tokenizer not found. Please make sure you have trained a model first."
+
+        model = load_model(model_path)
+        tokenizer = BertTokenizerFast.from_pretrained(TOKENIZER_PATH)
+
+        # Generate caption
+        caption = generate_caption(model, image, tokenizer)
+        return caption
+
+    except Exception as e:
+        return f"Error occurred: {str(e)}"
+
+# Create Gradio interface
+def create_interface():
+    available_models = list_available_models()
+    if not available_models:
+        return gr.Interface(
+            fn=lambda x: "No models found in ./models directory. Please train a model first.",
+            inputs="image",
+            outputs="text",
+            title="TeLVE - Turkish efficient Language Vision Engine",
+            description="Error: No models available"
+        )
+
+    interface = gr.Interface(
+        fn=generate_description,
+        inputs=[
+            gr.Image(type="filepath", label="Upload Image"),
+            gr.Dropdown(choices=available_models, label="Select Model", value=available_models[0])
+        ],
+        outputs=gr.Textbox(label="Generated Caption"),
+        title="TeLVE - Turkish Language Vision Encoder",
+        description="Upload an image to generate a Turkish caption.",
+        examples=[
+            ["./images/example1.jpg", available_models[0]],
+            ["./images/example2.jpg", available_models[0]]
+        ] if os.path.exists("./images") else None
+    )
+    return interface
+
+if __name__ == "__main__":
+    demo = create_interface()
+    demo.launch(share=True, server_name="0.0.0.0")

requirements.txt

@@ -0,0 +1,8 @@
+
+torch>=1.9.0
+torchvision>=0.10.0
+transformers>=4.11.0
+gradio>=3.0.0
+pandas>=1.3.0
+Pillow>=8.0.0
+tqdm>=4.62.0