README.md

---
license: apache-2.0
tags:
  - video-understanding
  - onnx
  - gguf
  - minicpm-v
  - glm-ocr
  - fun-asr
  - llama-cpp
---

# VideoMiner Model Files

Pre-converted model files for the VideoMiner video understanding pipeline.

No PyTorch runtime required — all inference uses ONNX Runtime + llama.cpp.

## Directory Structure

```
VideoMiner/
├── minicpmv/                 # MiniCPM-V 4.5 Vision Encoder + LLM
│   ├── minicpmv_v45_siglip.fp32.onnx           # SigLIP ViT (FP32, 1.6 GB)
│   ├── minicpmv_v45_resampler_temporal.fp32.onnx    # Resampler graph (FP32)
│   ├── minicpmv_v45_resampler_temporal.fp32.onnx.data  # Resampler weights (340 MB)
│   ├── minicpmv_v45_resampler_temporal.fp16.onnx    # Resampler graph (FP16, legacy)
│   ├── minicpmv_v45_resampler_temporal.fp16.onnx.data  # Resampler weights (170 MB, legacy)
│   └── MiniCPM-V-4_5-Q4_K_M.gguf              # Qwen3 8B decoder (Q4_K_M, 4.7 GB)
│
├── glm-ocr/                  # GLM-OCR 0.9B (High-accuracy OCR)
│   ├── GLM-OCR-Q8_0.gguf                        # GLM-OCR decoder (Q8_0, 907 MB)
│   ├── config.json / tokenizer.json / ...        # Tokenizer & config
│   └── onnx/                                     # Vision encoder ONNX (Q4 quantized)
│
├── fun-asr/                  # FUN-ASR Nano (Speech Recognition)
│   ├── Fun-ASR-Nano-Encoder-Adaptor.fp16.onnx   # Encoder + adaptor (443 MB)
│   ├── Fun-ASR-Nano-CTC.fp16.onnx               # CTC head (75 MB)
│   ├── Fun-ASR-Nano-Decoder.q5_k.gguf           # LLM decoder (Q5_K, 424 MB)
│   └── tokens.txt                                # Tokenizer vocab
│
├── embedding/                # Text Embedding
│   └── bge-small-zh-v1.5-onnx/  # BGE-small-zh ONNX (95 MB, no PyTorch required)
│       ├── model.onnx                            # BERT encoder
│       ├── tokenizer.json / vocab.txt            # Tokenizer
│       └── config.json
│
└── runtime/                  # Pre-built llama.cpp shared libraries (CUDA 12.8)
    ├── libllama.so                               # Core llama.cpp (3.3 MB)
    ├── libggml-base.so                           # GGML base (835 KB)
    └── libggml-cuda.so                           # CUDA backend (150 MB)
```

## Models

### MiniCPM-V 4.5 (Vision Encoder + LLM)

- **SigLIP ViT**: 27-layer vision transformer (1152-dim), FP32 ONNX. FP32 is required — 27 layers without vit_merger causes FP16 numerical overflow.
- **Resampler**: Projects SigLIP features (1152-dim) to LLM space (4096-dim) with temporal awareness. **FP32 ONNX is required** — FP16 causes onnxruntime CUDA EP to fall back to CPU for `layer_norm`/`matmul` nodes, resulting in ~5s/clip vs ~50ms/clip.
- **LLM Decoder**: Qwen3 8B dense, GGUF Q4_K_M, served via llama.cpp ctypes.

### GLM-OCR 0.9B (OCR)

- GLM-4 architecture (17 layers), specialized for high-accuracy OCR.
- Vision encoder: Q4-quantized ONNX. LLM decoder: Q8_0 GGUF.

### FUN-ASR Nano (Speech Recognition)

- Paraformer-based ASR for Chinese/English speech-to-text.
- Encoder/CTC: FP16 ONNX. Decoder: Q5_K GGUF.

### BGE-small-zh-v1.5 (Embedding)

- BAAI/bge-small-zh-v1.5 exported as ONNX. No PyTorch/sentence-transformers required.
- Usage: tokenize → ONNX inference → CLS token → L2 normalize.
- Dimension: 512. Cosine similarity vs sentence-transformers: 1.0.

### Runtime

- Pre-built llama.cpp shared libraries with CUDA 12.8 support (universal architecture).

## Usage

```python
# HuggingFace
from huggingface_hub import snapshot_download
model_dir = snapshot_download('JazerJu/VideoMiner')

# ModelScope (China mirror)
from modelscope import snapshot_download
model_dir = snapshot_download('modelmo/VideoMiner')
```

## Requirements

- **GPU**: NVIDIA GPU with ≥12 GB VRAM, CUDA 12.8+
- **Runtime**: Python 3.10+, ONNX Runtime GPU, llama.cpp (ctypes)
- **No PyTorch required**

## License

- Model weights: Follow original model licenses (MiniCPM-V, GLM-OCR, FUN-ASR, BGE)
- Runtime libraries: llama.cpp (MIT License)
- This repository: Apache 2.0