Sudeep Pillai PRO

spillai

1 2 160

https://people.csail.mit.edu/spillai/

AI & ML interests

Self-supervised learning, Few-shot learning, Computer Vision, Robotics

Recent Activity

reacted to nwaughachukwuma's post with 🤗 about 14 hours ago

Can a text-only model + a vision toolkit (mm-ctx) match a native vision model? We benchmarked 4 setups on 23 multimodal tasks (image, video, audio, PDF): • glm-5.2 (text-only) + mm-ctx: 88.4 • gemini-3.5-flash (vision): 83 • deepseek-v4-pro (text-only) + mm-ctx: 79.4 • qwen3.6-35b-a3b (vision): 44.3 The best text-only setup `glm-5.2 + mm` outperformed gemini-3.5-flash, the top vision model, by 5.4 points (6.5%). It was also: • 1.5x faster (100s vs 150s mean per task) • the only setup with zero timeouts (46/46 completed; gemini timed out 4x on bulk-image and long-video tasks) • the only setup stable across runs (88.5 / 88.4) • top on video (100.0), image (91.7), and PDF (90.0) tasks The trade-offs: the toolkit consumed 3.3x more tokens (4.25M vs 1.28M), and lost on audio (85.6 vs 71.3). On completed tasks alone the two are nearly identical (91.0 vs 88.4): the toolkit's edge is efficient extraction that keeps long media tasks inside the time budget. Full report: https://huggingface.co/blog/vlm-run/text-only-models-with-mm

reacted to nwaughachukwuma's post with 👀 14 days ago

# One API for Every Visual & OCR Models. The VLM Run Gateway is a fully compatible API for OpenAI chat completions for visual intelligence. If you’re building document extraction or visual understanding, the Gateway exposes OCR, VQA, and detection behind a single interface you already know. Read the docs: https://docs.vlm.run/gateway/introduction. We actively support the following recent OCR and VQA models, which you can try today at no cost: * zai-org/glm-ocr * rednote-hilab/dots.mocr * paddleocr/pp-ocrv6 * qwen/qwen3.5-0.8b ## Quickstart ### Python ``` from openai import OpenAI client = OpenAI(base_url="https://gateway.vlm.run/v1/openai") response = client.chat.completions.create( model="zai-org/glm-ocr", messages=[ { "role": "user", "content": [ { "type": "document_url", "document_url": { "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf" }, }, ], } ], extra_body={"method": "markdown", "document_dpi": 150}, ) print(response.choices[0].message.content) ``` ### Curl ``` curl https://gateway.vlm.run/v1/openai/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer vlmrun" \ -d '{ "model": "zai-org/glm-ocr", "messages": [ { "role": "user", "content": [ { "type": "document_url", "document_url": { "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf" } } ] } ], "method": "markdown", "document_dpi": 150 }' ``` ## Auth and limits Anonymous auth is enabled, so you can omit the authorization header entirely, or send Bearer "" or Bearer vlmrun. Rate limits are 60 req/min and 1000 req/hr.

liked a model 2 months ago

RedHatAI/Qwen3.6-35B-A3B-NVFP4

View all activity

Organizations

reacted to nwaughachukwuma's post with 🤗 about 14 hours ago

Post

650

Can a text-only model + a vision toolkit (mm-ctx) match a native vision model?

We benchmarked 4 setups on 23 multimodal tasks (image, video, audio, PDF):

• glm-5.2 (text-only) + mm-ctx: 88.4
• gemini-3.5-flash (vision): 83
• deepseek-v4-pro (text-only) + mm-ctx: 79.4
• qwen3.6-35b-a3b (vision): 44.3

The best text-only setup glm-5.2 + mm outperformed gemini-3.5-flash, the top vision model, by 5.4 points (6.5%). It was also:

• 1.5x faster (100s vs 150s mean per task)
• the only setup with zero timeouts (46/46 completed; gemini timed out 4x on bulk-image and long-video tasks)
• the only setup stable across runs (88.5 / 88.4)
• top on video (100.0), image (91.7), and PDF (90.0) tasks

The trade-offs: the toolkit consumed 3.3x more tokens (4.25M vs 1.28M), and lost on audio (85.6 vs 71.3).

On completed tasks alone the two are nearly identical (91.0 vs 88.4): the toolkit's edge is efficient extraction that keeps long media tasks inside the time budget.

Full report: https://huggingface.co/blog/vlm-run/text-only-models-with-mm

8 replies

reacted to nwaughachukwuma's post with 👀 14 days ago

Post

734

from openai import OpenAI

client = OpenAI(base_url="https://gateway.vlm.run/v1/openai")

response = client.chat.completions.create(
    model="zai-org/glm-ocr",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "document_url",
                    "document_url": {
                        "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
                    },
                },
            ],
        }
    ],
    extra_body={"method": "markdown", "document_dpi": 150},
)

print(response.choices[0].message.content)

### Curl

curl https://gateway.vlm.run/v1/openai/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer vlmrun" \
  -d '{
    "model": "zai-org/glm-ocr",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "document_url",
            "document_url": {
              "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
            }
          }
        ]
      }
    ],
    "method": "markdown",
    "document_dpi": 150
  }'

## Auth and limits

Anonymous auth is enabled, so you can omit the authorization header entirely, or send Bearer "" or Bearer vlmrun. Rate limits are 60 req/min and 1000 req/hr.

reacted to their post with 🔥 3 months ago

Post

8780

mm-ctx – fast, multimodal context for agents.

LLM-based agents handle text incredibly well, but images, videos, or PDFs with visual content are hard to interpret. mm-ctx gives your CLI agent multi-modal skills.

Try it interactively in Spaces: vlm-run/mm-ctx

Readme: https://vlm-run.github.io/mm/
PyPI: https://pypi.org/project/mm-ctx
SKILL.md: https://github.com/vlm-run/skills/blob/main/skills/mm-cli-skill/SKILL.md

mm-ctx is meant to feel familiar: the UNIX tools we already love (find/cat/grep/wc), rebuilt for file types LLMs can't read natively and designed to work with agents via the CLI.
- mm grep "invoice #1234" ~/Downloads searches across PDFs and returns line-numbered matches
- mm cat <document>.pdf returns a metadata description of the file
- mm cat <photo>.jpg returns a caption of the photo
- mm cat <video>.mp4 returns a caption of the video

A few things we obsessed over:
⚡ Speed: Rust core for the hot paths
🏠 Local-first, BYO model: Uses any OpenAI-compatible endpoint: Ollama, vLLM/SGLang, LMStudio with any multimodal LLM (Gemma4, Qwen3.5, GLM-4.6V).
🔗 Composable: stdin + structured outputs
🤖 Drops into any agent via mm-cli-skills: Claude Code, Codex, Gemini CLI, OpenClaw.

We’d love to hear your feedback! Especially on the CLI and what file types and workflows you would like to see next.

2 replies

posted an update 3 months ago

Post

8780

2 replies

Sudeep Pillai PRO

AI & ML interests

Recent Activity

Organizations

spillai's activity