AI & ML interests
None defined yet.
Recent Activity
View all activity
Articles
nwaughachukwuma
published an article 3 days ago
Article
Text-Only Models with mm-ctx Vision Toolkit vs. Native Vision Models
vlm-run
• nwaughachukwuma
posted an update 5 days ago
Post
654
Can a text-only model + a vision toolkit (mm-ctx) match a native vision model?
We benchmarked 4 setups on 23 multimodal tasks (image, video, audio, PDF):
• glm-5.2 (text-only) + mm-ctx: 88.4
• gemini-3.5-flash (vision): 83
• deepseek-v4-pro (text-only) + mm-ctx: 79.4
• qwen3.6-35b-a3b (vision): 44.3
The best text-only setup
• 1.5x faster (100s vs 150s mean per task)
• the only setup with zero timeouts (46/46 completed; gemini timed out 4x on bulk-image and long-video tasks)
• the only setup stable across runs (88.5 / 88.4)
• top on video (100.0), image (91.7), and PDF (90.0) tasks
The trade-offs: the toolkit consumed 3.3x more tokens (4.25M vs 1.28M), and lost on audio (85.6 vs 71.3).
On completed tasks alone the two are nearly identical (91.0 vs 88.4): the toolkit's edge is efficient extraction that keeps long media tasks inside the time budget.
Full report: https://huggingface.co/blog/vlm-run/text-only-models-with-mm
We benchmarked 4 setups on 23 multimodal tasks (image, video, audio, PDF):
• glm-5.2 (text-only) + mm-ctx: 88.4
• gemini-3.5-flash (vision): 83
• deepseek-v4-pro (text-only) + mm-ctx: 79.4
• qwen3.6-35b-a3b (vision): 44.3
The best text-only setup
glm-5.2 + mm outperformed gemini-3.5-flash, the top vision model, by 5.4 points (6.5%). It was also:• 1.5x faster (100s vs 150s mean per task)
• the only setup with zero timeouts (46/46 completed; gemini timed out 4x on bulk-image and long-video tasks)
• the only setup stable across runs (88.5 / 88.4)
• top on video (100.0), image (91.7), and PDF (90.0) tasks
The trade-offs: the toolkit consumed 3.3x more tokens (4.25M vs 1.28M), and lost on audio (85.6 vs 71.3).
On completed tasks alone the two are nearly identical (91.0 vs 88.4): the toolkit's edge is efficient extraction that keeps long media tasks inside the time budget.
Full report: https://huggingface.co/blog/vlm-run/text-only-models-with-mm
- 8 replies
nwaughachukwuma
posted an update 16 days ago
Post
734
# One API for Every Visual & OCR Models.
The VLM Run Gateway is a fully compatible API for OpenAI chat completions for visual intelligence. If you’re building document extraction or visual understanding, the Gateway exposes OCR, VQA, and detection behind a single interface you already know.
Read the docs: https://docs.vlm.run/gateway/introduction.
We actively support the following recent OCR and VQA models, which you can try today at no cost:
* zai-org/glm-ocr
* rednote-hilab/dots.mocr
* paddleocr/pp-ocrv6
* qwen/qwen3.5-0.8b
## Quickstart
### Python
### Curl
## Auth and limits
Anonymous auth is enabled, so you can omit the authorization header entirely, or send Bearer "" or Bearer vlmrun. Rate limits are 60 req/min and 1000 req/hr.
The VLM Run Gateway is a fully compatible API for OpenAI chat completions for visual intelligence. If you’re building document extraction or visual understanding, the Gateway exposes OCR, VQA, and detection behind a single interface you already know.
Read the docs: https://docs.vlm.run/gateway/introduction.
We actively support the following recent OCR and VQA models, which you can try today at no cost:
* zai-org/glm-ocr
* rednote-hilab/dots.mocr
* paddleocr/pp-ocrv6
* qwen/qwen3.5-0.8b
## Quickstart
### Python
from openai import OpenAI
client = OpenAI(base_url="https://gateway.vlm.run/v1/openai")
response = client.chat.completions.create(
model="zai-org/glm-ocr",
messages=[
{
"role": "user",
"content": [
{
"type": "document_url",
"document_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
},
},
],
}
],
extra_body={"method": "markdown", "document_dpi": 150},
)
print(response.choices[0].message.content)### Curl
curl https://gateway.vlm.run/v1/openai/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer vlmrun" \
-d '{
"model": "zai-org/glm-ocr",
"messages": [
{
"role": "user",
"content": [
{
"type": "document_url",
"document_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
}
}
]
}
],
"method": "markdown",
"document_dpi": 150
}'## Auth and limits
Anonymous auth is enabled, so you can omit the authorization header entirely, or send Bearer "" or Bearer vlmrun. Rate limits are 60 req/min and 1000 req/hr.
nwaughachukwuma
updated a
Space 2 months ago
Post
8782
mm-ctx – fast, multimodal context for agents.
LLM-based agents handle text incredibly well, but images, videos, or PDFs with visual content are hard to interpret. mm-ctx gives your CLI agent multi-modal skills.
Try it interactively in Spaces: vlm-run/mm-ctx
Readme: https://vlm-run.github.io/mm/
PyPI: https://pypi.org/project/mm-ctx
SKILL.md: https://github.com/vlm-run/skills/blob/main/skills/mm-cli-skill/SKILL.md
mm-ctx is meant to feel familiar: the UNIX tools we already love (find/cat/grep/wc), rebuilt for file types LLMs can't read natively and designed to work with agents via the CLI.
- mm grep "invoice #1234" ~/Downloads searches across PDFs and returns line-numbered matches
- mm cat <document>.pdf returns a metadata description of the file
- mm cat <photo>.jpg returns a caption of the photo
- mm cat <video>.mp4 returns a caption of the video
A few things we obsessed over:
⚡ Speed: Rust core for the hot paths
🏠 Local-first, BYO model: Uses any OpenAI-compatible endpoint: Ollama, vLLM/SGLang, LMStudio with any multimodal LLM (Gemma4, Qwen3.5, GLM-4.6V).
🔗 Composable: stdin + structured outputs
🤖 Drops into any agent via mm-cli-skills: Claude Code, Codex, Gemini CLI, OpenClaw.
We’d love to hear your feedback! Especially on the CLI and what file types and workflows you would like to see next.
LLM-based agents handle text incredibly well, but images, videos, or PDFs with visual content are hard to interpret. mm-ctx gives your CLI agent multi-modal skills.
Try it interactively in Spaces: vlm-run/mm-ctx
Readme: https://vlm-run.github.io/mm/
PyPI: https://pypi.org/project/mm-ctx
SKILL.md: https://github.com/vlm-run/skills/blob/main/skills/mm-cli-skill/SKILL.md
mm-ctx is meant to feel familiar: the UNIX tools we already love (find/cat/grep/wc), rebuilt for file types LLMs can't read natively and designed to work with agents via the CLI.
- mm grep "invoice #1234" ~/Downloads searches across PDFs and returns line-numbered matches
- mm cat <document>.pdf returns a metadata description of the file
- mm cat <photo>.jpg returns a caption of the photo
- mm cat <video>.mp4 returns a caption of the video
A few things we obsessed over:
⚡ Speed: Rust core for the hot paths
🏠 Local-first, BYO model: Uses any OpenAI-compatible endpoint: Ollama, vLLM/SGLang, LMStudio with any multimodal LLM (Gemma4, Qwen3.5, GLM-4.6V).
🔗 Composable: stdin + structured outputs
🤖 Drops into any agent via mm-cli-skills: Claude Code, Codex, Gemini CLI, OpenClaw.
We’d love to hear your feedback! Especially on the CLI and what file types and workflows you would like to see next.
- 2 replies
nwaughachukwuma
published a
Space 3 months ago
spillai
published a
dataset 6 months ago
spillai
updated a
dataset 6 months ago
dinesh-vlmrun
authored 2
papers 8 months ago
spillai
updated a
model over 2 years ago