File size: 1,431 Bytes
51d127a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
# DeepEncoder (Extracted from DeepSeek-OCR)
## Overview
This directory contains the encoder components extracted from DeepSeek-OCR.
## Model Files
- `sam_encoder.pth`: SAM ViT-B encoder (95,569,152 params, 364.6 MB)
- `clip_encoder.pth`: CLIP-Large encoder (303,177,728 params, 1156.6 MB)
- `projector.pth`: Linear projector (2,622,720 params, 10.0 MB)
- `config.json`: Model configuration
**Total:** 401,369,600 parameters
## Architecture
```
Image (1024×1024) → SAM (95M) → 16× Conv → CLIP (303M) → Projector (3M) → 256 vision tokens
```
## Usage
```python
import torch
from deepencoder import build_sam_vit_b, build_clip_l, MlpProjector
from easydict import EasyDict as adict
# Load models
sam = build_sam_vit_b(checkpoint=None)
sam.load_state_dict(torch.load('sam_encoder.pth'))
clip = build_clip_l()
clip.load_state_dict(torch.load('clip_encoder.pth'))
projector_cfg = adict({'projector_type': 'linear', 'input_dim': 2048, 'n_embed': 1280})
projector = MlpProjector(projector_cfg)
projector.load_state_dict(torch.load('projector.pth'))
# Run encoder
vision_tokens = encode(image) # [1, 256, 1280]
```
## Training
These weights are:
- Initialized from pretrained SAM (SA-1B) + CLIP (LAION-2B)
- Fine-tuned together on optical compression/OCR tasks
- Optimized for text preservation in compressed form
## Source
Extracted from: [deepseek-ai/DeepSeek-OCR](https://huggingface.co/deepseek-ai/DeepSeek-OCR)
|