VideoMiner Model Files

Pre-converted model files for the VideoMiner video understanding pipeline.

No PyTorch runtime required — all inference uses ONNX Runtime + llama.cpp.

Directory Structure

VideoMiner/
├── minicpmv/                 # MiniCPM-V 4.5 Vision Encoder + LLM
│   ├── minicpmv_v45_siglip.fp32.onnx           # SigLIP ViT (FP32, 1.6 GB)
│   ├── minicpmv_v45_resampler_temporal.fp32.onnx    # Resampler graph (FP32)
│   ├── minicpmv_v45_resampler_temporal.fp32.onnx.data  # Resampler weights (340 MB)
│   ├── minicpmv_v45_resampler_temporal.fp16.onnx    # Resampler graph (FP16, legacy)
│   ├── minicpmv_v45_resampler_temporal.fp16.onnx.data  # Resampler weights (170 MB, legacy)
│   └── MiniCPM-V-4_5-Q4_K_M.gguf              # Qwen3 8B decoder (Q4_K_M, 4.7 GB)
│
├── glm-ocr/                  # GLM-OCR 0.9B (High-accuracy OCR)
│   ├── GLM-OCR-Q8_0.gguf                        # GLM-OCR decoder (Q8_0, 907 MB)
│   ├── config.json / tokenizer.json / ...        # Tokenizer & config
│   └── onnx/                                     # Vision encoder ONNX (Q4 quantized)
│
├── fun-asr/                  # FUN-ASR Nano (Speech Recognition)
│   ├── Fun-ASR-Nano-Encoder-Adaptor.fp16.onnx   # Encoder + adaptor (443 MB)
│   ├── Fun-ASR-Nano-CTC.fp16.onnx               # CTC head (75 MB)
│   ├── Fun-ASR-Nano-Decoder.q5_k.gguf           # LLM decoder (Q5_K, 424 MB)
│   └── tokens.txt                                # Tokenizer vocab
│
├── embedding/                # Text Embedding
│   └── bge-small-zh-v1.5-onnx/  # BGE-small-zh ONNX (95 MB, no PyTorch required)
│       ├── model.onnx                            # BERT encoder
│       ├── tokenizer.json / vocab.txt            # Tokenizer
│       └── config.json
│
└── runtime/                  # Pre-built llama.cpp shared libraries (CUDA 12.8)
    ├── libllama.so                               # Core llama.cpp (3.3 MB)
    ├── libggml-base.so                           # GGML base (835 KB)
    └── libggml-cuda.so                           # CUDA backend (150 MB)

Models

MiniCPM-V 4.5 (Vision Encoder + LLM)

• SigLIP ViT: 27-layer vision transformer (1152-dim), FP32 ONNX. FP32 is required — 27 layers without vit_merger causes FP16 numerical overflow.
• Resampler: Projects SigLIP features (1152-dim) to LLM space (4096-dim) with temporal awareness. FP32 ONNX is required — FP16 causes onnxruntime CUDA EP to fall back to CPU for layer_norm/matmul nodes, resulting in ~5s/clip vs ~50ms/clip.
• LLM Decoder: Qwen3 8B dense, GGUF Q4_K_M, served via llama.cpp ctypes.

GLM-OCR 0.9B (OCR)

• GLM-4 architecture (17 layers), specialized for high-accuracy OCR.
• Vision encoder: Q4-quantized ONNX. LLM decoder: Q8_0 GGUF.

FUN-ASR Nano (Speech Recognition)

• Paraformer-based ASR for Chinese/English speech-to-text.
• Encoder/CTC: FP16 ONNX. Decoder: Q5_K GGUF.

BGE-small-zh-v1.5 (Embedding)

• BAAI/bge-small-zh-v1.5 exported as ONNX. No PyTorch/sentence-transformers required.
• Usage: tokenize → ONNX inference → CLS token → L2 normalize.
• Dimension: 512. Cosine similarity vs sentence-transformers: 1.0.

Runtime

• Pre-built llama.cpp shared libraries with CUDA 12.8 support (universal architecture).

Usage

# HuggingFace
from huggingface_hub import snapshot_download
model_dir = snapshot_download('JazerJu/VideoMiner')

# ModelScope (China mirror)
from modelscope import snapshot_download
model_dir = snapshot_download('modelmo/VideoMiner')

Requirements

• GPU: NVIDIA GPU with ≥12 GB VRAM, CUDA 12.8+
• Runtime: Python 3.10+, ONNX Runtime GPU, llama.cpp (ctypes)
• No PyTorch required

License

• Model weights: Follow original model licenses (MiniCPM-V, GLM-OCR, FUN-ASR, BGE)
• Runtime libraries: llama.cpp (MIT License)
• This repository: Apache 2.0