Project: GLM Visual Variable Runtime

Mission

Build an OpenAI-compatible multimodal gateway called:

glm-5.2-visual-runtime

The gateway must make the text-only GLM-5.2 model appear to support persistent image input across chat turns, OCR and document understanding, object-level color and style extraction, chart-to-data extraction, visual questions, presentation generation based on visual references, and references such as "this image", "those colors", "that chart", and composed multi-image/property references.

This project must not train, fine-tune, or modify GLM-5.2. The image is not converted into a permanent caption or inserted as visual tokens. Instead, every uploaded image becomes a persistent visual variable with lazily generated typed views.

Example:

img_01.original
img_01.thumbnail
img_01.ocr
img_01.objects
img_01.palette
img_01.presentation_theme
img_01.chart
img_01.chart.data_table

GLM-5.2 receives only the compact structured evidence needed for the current request.

Non-negotiable requirements

No GPU training.
No modification of GLM-5.2 weights.
The public API must support OpenAI-compatible request formats.
The original image must be stored and remain retrievable.
Expensive visual analysis must run lazily, only when requested.
Derived visual views must be cached by image hash and lens version.
Later turns must work without uploading the image again.
Every extracted value must retain provenance: asset ID, bounding box or source region, extraction method, confidence, and lens version.
Do not send an entire OCR transcript to GLM unless the task requires it.
Do not let GLM estimate chart numbers when they can be calculated from extracted structured data.
Do not regenerate exact OCR strings when they can be copied from evidence.
The gateway must support cloud mode with no local GPU.
All model providers must be replaceable through interfaces.
Build API and integration tests before adding a user interface.

Architecture

Implement this processing flow:

OpenAI-compatible request
          |
          v
Request normalizer
          |
          v
Conversation and visual asset ledger
          |
          v
Visual reference resolver
          |
          v
Intent and visual-program compiler
          |
          v
Lazy lens executor
  |       |       |       |
  OCR   Object  Palette  Chart
          |
          v
Compact evidence compiler
          |
          v
       GLM-5.2
          |
          v
Text, JSON, table, image, or PPTX artifact

The public model ID is glm-5.2-visual-runtime. The internal reasoning model defaults to glm-5.2.

Repository structure

Use the monorepo layout requested by the project brief:

glm-vvr/
  AGENTS.md
  README.md
  docker-compose.yml
  .env.example
  apps/api/
  services/slides/
  tests/
  docs/

Public API

Implement:

POST /v1/responses
POST /v1/chat/completions
POST /v1/files
GET  /v1/files/{file_id}
GET  /v1/files/{file_id}/content
GET  /health
GET  /ready

/v1/responses is canonical. Support input_image.image_url, Base64 data URLs, input_image.file_id, chat completion image_url content parts, multiple images in one user message, and previous_response_id.

When a URL image is supplied, download and snapshot it immediately. Validate MIME type, file size, image dimensions, decoded file signature, redirect limits, and private-network or localhost URLs to prevent SSRF.

Runtime rules

On image ingest, store the original image, create an img_NN asset, calculate SHA-256, generate a thumbnail and cheap visual fingerprint, and set the asset as active. Do not run OCR, object detection, or chart extraction until needed.
Resolve references deterministically first. If confidence is below threshold, call GLM-5.2 with structured output and validate the result.
Compile user requests into validated visual programs before executing lenses.
Use compact task-specific visual evidence for GLM-5.2. Include exact values, units, uncertainty, and source region IDs. Never claim visual evidence contains absent information.

Lenses

Implement replaceable lenses for OCR, object grounding, palette extraction, chart extraction, and presentation theme derivation. Cache views by:

sha256(asset bytes) + lens name + lens version + canonicalized parameters

OCR should preserve exact Unicode, bounding boxes, reading order, tables, Markdown, and structured JSON. Chart extraction should produce reversible ChartIR and answer numeric chart questions from structured data, not model prose. Palette extraction should produce area, attention, and semantic palettes plus role-based design tokens with provenance and contrast checks.

Providers

Use interfaces for GLM-5.2, GLM-5V-Turbo, GLM-OCR, and fake deterministic providers. The MVP must default to API providers in cloud mode and fake providers in test mode. Do not self-host GLM-5.2 in the MVP.

Presentation generation

Use a separate TypeScript PptxGenJS service. GLM-5.2 produces a validated PresentationSpec; the renderer creates editable .pptx files with slide masters, extracted design tokens, editable charts/tables, source images where appropriate, speaker notes containing provenance when enabled, and low-confidence chart warnings.

Modes

cloud: GLM-5.2 API, GLM-5V-Turbo API, GLM-OCR API, CPU image processing, no local GPU.
hybrid: GLM-5.2 API, local or hosted OCR, optional local segmentation.
test: deterministic fake providers, no external API calls.

Acceptance tests

The system must demonstrate persistent image references, palette-to-PPTX generation, object-region color extraction, exact OCR evidence, ChartIR-based chart answers, multi-image references, URL snapshot durability, duplicate hash view reuse, lazy lens execution, SSRF rejection, no-key test runs, and OpenAI Python client compatibility with only base_url changed.

Definition of done

The project is complete when it runs with Docker Compose, OpenAI Python client can call /v1/responses, images can be supplied by URL/Base64/file ID, later text-only turns can reference prior images, OCR/palette/object/chart lenses work through replaceable providers, ChartIR is emitted, editable PPTX files are generated, provenance and confidence are included, tests pass, no training code exists, default cloud deployment needs no local GPU, and docs explain how to add a visual lens without modifying GLM-5.2.