File size: 2,038 Bytes
6529951 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | # eagle0504/llava-video-text-model
Fine-tuned **LLaVA model** on video-text data using DeepSpeed.
## Model Details
- **Base model**: llava-hf/llava-interleave-qwen-7b-hf
- **Architecture**: LLaVA (Large Language and Vision Assistant)
- **Training samples**: 4 videos
- **Training**: Multi-GPU with DeepSpeed ZeRO Stage 2
- **Task**: Video-text conversation generation
- **Video frames**: 5 frames per video
## Usage
```python
import requests
from PIL import Image
import torch
from transformers import AutoProcessor, LlavaForConditionalGeneration
# Load model and processor
processor = AutoProcessor.from_pretrained("eagle0504/llava-video-text-model")
model = LlavaForConditionalGeneration.from_pretrained(
"eagle0504/llava-video-text-model",
torch_dtype=torch.float16,
low_cpu_mem_usage=True,
).to(0)
# Define conversation with multiple images for video
conversation = [
{
"role": "user",
"content": [
{"type": "text", "text": "What is in this video?"},
{"type": "image"},
{"type": "image"},
{"type": "image"},
{"type": "image"},
{"type": "image"},
],
},
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
# Process video frames (you need to extract frames from your video)
video_frames = [...] # List of PIL Images from video
inputs = processor(images=video_frames, text=prompt, return_tensors='pt').to(0, torch.float16)
# Generate response
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
response = processor.decode(output[0], skip_special_tokens=True)
print(response)
```
## Training Configuration
- DeepSpeed ZeRO Stage 2
- Mixed precision (BF16)
- AdamW optimizer
- Learning rate: 5e-5
- Video frames per sample: 5
## Video Processing
This model expects 5 frames extracted from each video. For best results:
1. Extract evenly spaced frames from your video
2. Resize frames to model's expected input size
3. Pass frames as a list to the processor
|