HF_PRD_v1.md

OpenBMB Local AI Workbench — Technical PRD v2.0

Revised: 2026-06-05 | Corrections from online research embedded inline

---

Table of Contents

1. Purpose & Design Philosophy
2. Design Principles (Template Architecture)
3. System Architecture
4. Model Registry
5. Inference Stack — Five Modes
6. Tracking: Trackio (Corrected API)
7. MCP Layer (Three Integration Paths)
8. Training Pipeline
9. Export & Quantization
10. Agent Mode (ml-intern Inspired)
11. UI — Tab Specification
12. Field Notes & Correction Loop
13. Directory Structure
14. Configuration Schema
15. Dependencies
16. Hackathon Demo Flow
17. Corrections from PRD v1
18. Roadmap & Extension Points

---

1. Purpose & Design Philosophy

One-line: A modular AI experimentation platform for the OpenBMB model family that covers dataset ingestion → LoRA training → evaluation → GGUF export → llama.cpp/SGLang deployment → multimodal inference → MCP tool exposure → trace sharing — in a single Gradio app.

Broader framing: The platform is deliberately designed as a template. The config/models.yaml and config/training.yaml files drive almost all model-specific behaviour. Replacing the OpenBMB section with any other model family (Qwen, Phi, Gemma …) should require only config changes and a matching service class, never core rewrites.

Why OpenBMB?

• MiniCPM5-1B (2026-05-19): SOTA 1B on-device, 128K context, LlamaForCausalLM — no custom kernels
• MiniCPM4.1-8B (2025-09): sparse InfLLM-v2 attention, ~7× long-context speedup vs Qwen3-8B
• MiniCPM-V-4.6 (2026-05-11): 1.3B VLM, SigLIP2-400M + Qwen3.5-0.8B, 262K ctx, mixed 4×/16× token compression
• MiniCPM-V-4.6-Thinking: same architecture + long-CoT for multimodal reasoning
• MiniCPM-o 4.5 (2026-05-17): omnimodal (vision + audio), real-time conversation
• Apache 2.0 across the board; vLLM / SGLang / llama.cpp / Ollama native support

---

2. Design Principles (Template Architecture)

Principle	Implementation
Config-driven	models.yaml + training.yaml define all models; no model names hardcoded in logic
Event-driven	core/events.py — typed event bus; all cross-module comms via events
Registry pattern	core/registry.py — all services register by name; swap without restart
Separation of concerns	models/ = loading; training/ = train/eval; ui/ = Gradio; mcp/ = protocol
Plugin tools	Any Python function decorated @mcp.tool() is auto-exposed; no wiring needed
Async-first	FastAPI backend, Gradio queuing, non-blocking inference
Observability	Every significant action fires a Trackio log + event
Template portability	Replace config/models.yaml → different model family; keep all else

---

3. System Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                        OpenBMB Local AI Workbench                           │
├───────────────────────────┬─────────────────────────────────────────────────┤
│        Gradio UI Layer    │          Backend Service Layer                  │
│  ┌─────────────────────┐  │  ┌─────────────────────────────────────────┐   │
│  │  gr.Blocks / Tabs   │  │  │  gradio.Server (FastAPI-based)          │   │
│  │  chat_tab.py        │  │  │  @app.api()   — Gradio-queued endpoints │   │
│  │  vision_tab.py      │  │  │  @app.mcp.tool() — MCP registration     │   │
│  │  train_tab.py       │  │  │  @app.get("/") — static/custom frontend │   │
│  │  dataset_tab.py     │  │  └─────────────────────────────────────────┘   │
│  │  export_tab.py      │  │                                                 │
│  │  traces_tab.py      │  │  ┌─────────────────────────────────────────┐   │
│  │  agent_tab.py       │  │  │  FastMCP server (standalone / bridged)  │   │
│  └─────────────────────┘  │  │  mcp = FastMCP("OpenBMBWorkbench")      │   │
│                           │  │  Registered tools (see §7)              │   │
│  SSE MCP endpoint:        │  └─────────────────────────────────────────┘   │
│  /gradio_api/mcp/sse      │                                                 │
└───────────────────────────┴─────────────────────────────────────────────────┘
          │                                │
          ▼                                ▼
┌─────────────────────┐        ┌────────────────────────┐
│   Model Services    │        │   Tracking & Storage   │
│  minicpm_text.py    │        │  trackio (local/Space) │
│  minicpm_vision.py  │        │  field_notes/ (SQLite) │
│  llama_cpp_runner.py│        │  exports/ (GGUF files) │
│  sglang_runner.py   │        │  data/ (HF datasets)   │
│  ollama_runner.py   │        └────────────────────────┘
└─────────────────────┘
          │
          ▼
┌───────────────────────────────────────────────────────────┐
│                   Inference Backends                      │
│  Transformers  │  llama.cpp  │  SGLang  │  vLLM  │ Ollama│
└───────────────────────────────────────────────────────────┘

3.1 Event Bus (core/events.py)

from dataclasses import dataclass, field
from enum import Enum
from typing import Any, Callable
import asyncio

class EventType(str, Enum):
    DATASET_LOADED      = "dataset_loaded"
    TRAINING_STARTED    = "training_started"
    TRAINING_STEP       = "training_step"
    TRAINING_FINISHED   = "training_finished"
    EVAL_FINISHED       = "eval_finished"
    INFERENCE_REQUEST   = "inference_request"
    INFERENCE_RESPONSE  = "inference_response"
    TOOL_CALL           = "tool_call"
    MODEL_LOADED        = "model_loaded"
    MODEL_SWITCHED      = "model_switched"
    EXPORT_STARTED      = "export_started"
    EXPORT_FINISHED     = "export_finished"
    FIELD_NOTE_SAVED    = "field_note_saved"
    AGENT_STEP          = "agent_step"

@dataclass
class Event:
    type: EventType
    payload: dict[str, Any] = field(default_factory=dict)
    run_id: str = ""

class EventBus:
    _handlers: dict[EventType, list[Callable]] = {}

    def on(self, event_type: EventType):
        """Decorator to register an event handler."""
        def decorator(fn: Callable):
            self._handlers.setdefault(event_type, []).append(fn)
            return fn
        return decorator

    async def emit(self, event: Event):
        for handler in self._handlers.get(event.type, []):
            await handler(event)

bus = EventBus()

3.2 Service Registry (core/registry.py)

from typing import TypeVar, Generic, Type

T = TypeVar("T")

class Registry(Generic[T]):
    def __init__(self):
        self._services: dict[str, T] = {}

    def register(self, name: str, service: T):
        self._services[name] = service

    def get(self, name: str) -> T:
        if name not in self._services:
            raise KeyError(f"Service '{name}' not registered. Available: {list(self._services)}")
        return self._services[name]

    def list(self) -> list[str]:
        return list(self._services.keys())

# Singleton registries
model_registry   = Registry()   # model services
dataset_registry = Registry()   # dataset loaders
tool_registry    = Registry()   # MCP tools

---

4. Model Registry

4.1 Text Models

Config ID	HF ID	Architecture	Context	Notes
minicpm5_1b	openbmb/MiniCPM5-1B	LlamaForCausalLM	128K	Standard; no custom kernels; XML tool calls; SGLang minicpm5 parser
minicpm5_1b_thinking	openbmb/MiniCPM5-1B-Thinking	LlamaForCausalLM	128K	CoT mode; chat_template_kwargs={"enable_thinking": True}
minicpm41_8b	openbmb/MiniCPM4.1-8B	Sparse InfLLM-v2	128K (YaRN)	--trust-remote-code required; 7× faster long-ctx vs Qwen3-8B

4.2 Vision Models

Config ID	HF ID	Backbone	Context	Notes
minicpm_v46	openbmb/MiniCPM-V-4.6	SigLIP2-400M + Qwen3.5-0.8B	262K	transformers>=5.7.0 + torchcodec; mixed 4×/16× compression
minicpm_v46_thinking	openbmb/MiniCPM-V-4.6-Thinking	same + CoT head	262K	enable_thinking=True in chat_template_kwargs
minicpm_o45	openbmb/MiniCPM-o-4.5	Omnimodal	-	Vision + Audio; real-time conversation; see §4.4

4.3 Model Architecture Notes (critical for correct code)

MiniCPM5-1B — standard Llama, loads with:

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("openbmb/MiniCPM5-1B", trust_remote_code=True)

MiniCPM-V-4.6 — vision-language, loads with:

# ⚠️ CORRECTION from v1: NOT AutoModelForCausalLM — use AutoModelForImageTextToText
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("openbmb/MiniCPM-V-4.6", trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    "openbmb/MiniCPM-V-4.6",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

Chat format (vision):

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": pil_image},   # PIL Image
            {"type": "text",  "text": prompt}
        ]
    }
]
inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_tensors="pt",
    return_dict=True,
    chat_template_kwargs={"enable_thinking": False}   # True for Thinking variant
).to(model.device)

MiniCPM4.1-8B — requires trust_remote_code=True for sparse attention kernel.

4.4 models.yaml Schema

# config/models.yaml
# ─── Template: add any HF model here ──────────────────────────────────────
models:
  minicpm5_1b:
    hf_id:          openbmb/MiniCPM5-1B
    type:           text                        # text | vision | omnimodal
    architecture:   llama                       # llama | minicpm41 | minicpm_v
    context_length: 131072
    dtype:          bfloat16
    trust_remote_code: false                    # LlamaForCausalLM — no fork
    thinking_mode:  false                       # set true for Thinking checkpoints
    tool_call_parser: minicpm5                  # for SGLang serving
    gguf:
      available: true
      repo: openbmb/MiniCPM5-1B-GGUF
      default_quant: Q4_K_M

  minicpm41_8b:
    hf_id:          openbmb/MiniCPM4.1-8B
    type:           text
    architecture:   minicpm41                   # sparse InfLLM-v2
    context_length: 131072
    dtype:          bfloat16
    trust_remote_code: true
    thinking_mode:  false
    gguf:
      available: true
      repo: openbmb/MiniCPM4.1-8B-GGUF
      default_quant: Q4_K_M

  minicpm_v46:
    hf_id:          openbmb/MiniCPM-V-4.6
    type:           vision
    architecture:   minicpm_v
    context_length: 262144
    dtype:          bfloat16
    trust_remote_code: true
    thinking_mode:  false
    vision_encoder: siglip2_400m
    token_compression: "4x_16x"               # mixed compression
    gguf:
      available: true
      repo: openbmb/MiniCPM-V-4.6-gguf        # official repo — no surgery script needed
      main_file:  MiniCPM-V-4.6-Q4_K_M.gguf
      mmproj:     mmproj-MiniCPM-V-4.6-F16.gguf
      default_quant: Q4_K_M

  minicpm_v46_thinking:
    hf_id:          openbmb/MiniCPM-V-4.6-Thinking
    type:           vision
    architecture:   minicpm_v
    context_length: 262144
    dtype:          bfloat16
    trust_remote_code: true
    thinking_mode:  true
    gguf:
      available: true
      repo: openbmb/MiniCPM-V-4.6-Thinking-gguf
      default_quant: Q4_K_M

  minicpm_o45:
    hf_id:          openbmb/MiniCPM-o-4.5
    type:           omnimodal
    architecture:   minicpm_o
    context_length: 131072
    dtype:          bfloat16
    trust_remote_code: true
    thinking_mode:  false
    note: "Vision + Audio; real-time conversation mode"

---

5. Inference Stack — Five Modes

Mode A: Transformers (train + eval + fine-tuning)

Used for: LoRA training, evaluation, PEFT Load text: AutoModelForCausalLM Load vision: AutoModelForImageTextToText ← corrected from v1

# models/minicpm_text.py
class MiniCPMTextService:
    def __init__(self, model_id: str, cfg: dict):
        self.model = AutoModelForCausalLM.from_pretrained(
            model_id,
            torch_dtype=torch.bfloat16,
            device_map="auto",
            trust_remote_code=cfg.get("trust_remote_code", False)
        )
        self.tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# models/minicpm_vision.py  ← new dedicated service
class MiniCPMVisionService:
    def __init__(self, model_id: str, cfg: dict):
        self.processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
        self.model = AutoModelForImageTextToText.from_pretrained(
            model_id,
            torch_dtype=torch.bfloat16,
            device_map="auto",
            trust_remote_code=cfg.get("trust_remote_code", True)
        )
        self.thinking = cfg.get("thinking_mode", False)

    def chat(self, messages: list[dict], max_new_tokens: int = 512) -> str:
        inputs = self.processor.apply_chat_template(
            messages,
            add_generation_prompt=True,
            tokenize=True,
            return_tensors="pt",
            return_dict=True,
            chat_template_kwargs={"enable_thinking": self.thinking}
        ).to(self.model.device)
        with torch.no_grad():
            outputs = self.model.generate(**inputs, max_new_tokens=max_new_tokens)
        response = self.processor.decode(
            outputs[0][inputs["input_ids"].shape[1]:],
            skip_special_tokens=True
        )
        return response

Mode B: llama.cpp (local deployment, CPU/GPU offload)

⚠️ CORRECTION from v1: For MiniCPM-V 4.6, official GGUFs are already published. Download from openbmb/MiniCPM-V-4.6-gguf — no need to run convert_hf_to_gguf.py yourself. The minicpmv-surgery.py script was for the older MiniCPM-o-2.6; do not use it for V4.6.

Text models → llama-server (standard OpenAI-compatible endpoint) Vision models → llama-server --mmproj <mmproj.gguf> (multimodal server)

# models/llama_cpp_runner.py
import subprocess, requests, time

class LlamaCppDeployment:
    def __init__(self, cfg: dict):
        self.model_path  = cfg["model_path"]
        self.mmproj_path = cfg.get("mmproj_path")      # None for text models
        self.port        = cfg.get("port", 8080)
        self._proc       = None

    def start(self):
        cmd = [
            "llama-server",
            "--model", self.model_path,
            "--port", str(self.port),
            "--ctx-size", "8192",
            "--n-gpu-layers", "-1",       # full GPU offload if available
        ]
        if self.mmproj_path:
            cmd += ["--mmproj", self.mmproj_path]   # enables vision
        self._proc = subprocess.Popen(cmd)
        self._wait_ready()

    def _wait_ready(self, timeout: int = 30):
        for _ in range(timeout):
            try:
                r = requests.get(f"http://127.0.0.1:{self.port}/health")
                if r.status_code == 200:
                    return
            except Exception:
                pass
            time.sleep(1)
        raise RuntimeError("llama-server did not start in time")

    def chat(self, messages: list[dict]) -> str:
        r = requests.post(
            f"http://127.0.0.1:{self.port}/v1/chat/completions",
            json={"messages": messages, "max_tokens": 512}
        )
        return r.json()["choices"][0]["message"]["content"]

    def stop(self):
        if self._proc:
            self._proc.terminate()

GGUF Export (only needed if building custom quants):

# For MiniCPM-V 4.6 — convert_hf_to_gguf.py (NOT legacy surgery scripts)
python convert_hf_to_gguf.py openbmb/MiniCPM-V-4.6 --outtype f16
# Output: MiniCPM-V-4.6-F16.gguf  +  mmproj-MiniCPM-V-4.6-F16.gguf

# Quantize
./llama-quantize MiniCPM-V-4.6-F16.gguf MiniCPM-V-4.6-Q4_K_M.gguf Q4_K_M

Supported quants: Q4_K_M, Q5_K_M, Q8_0, Q2_K, F16

Mode C: SGLang (tool-use, function calling, recommended for MiniCPM5)

MiniCPM5-1B emits XML-style tool calls. SGLang has a native minicpm5 parser:

python -m sglang.launch_server \
  --model-path openbmb/MiniCPM5-1B \
  --port 30000 \
  --tool-call-parser minicpm5     # or: --tool-call-parser auto

For MiniCPM-V-4.6 vision:

python -m sglang.launch_server \
  --model-path openbmb/MiniCPM-V-4.6-Thinking \
  --port 30000 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --default-chat-template-kwargs '{"enable_thinking": true}'

Mode D: vLLM (batch serving, production)

vllm serve openbmb/MiniCPM4.1-8B --trust-remote-code
vllm serve openbmb/MiniCPM-V-4.6 --trust-remote-code

Mode E: Ollama (zero-config, consumer hardware)

ollama run openbmb/minicpm-v4.6    # vision + text
ollama run openbmb/minicpm5-1b     # text only

Mode Selection Matrix

Use case	Recommended mode
LoRA training	A (Transformers)
Evaluation	A (Transformers)
Local chat (fast)	B (llama.cpp)
Tool/function calling	C (SGLang)
Production batch	D (vLLM)
Consumer demo	E (Ollama)
Vision inference local	B (llama-server --mmproj)
Vision fine-tuning	A (SWIFT / LLaMA-Factory)

---

6. Tracking: Trackio (Corrected API)

⚠️ CRITICAL CORRECTION from v1: trackio.trace() does not exist. There is no context-manager trace API in Trackio. The v1 PRD example was wrong. Correct API: init() / log() / finish() / alert() / Table() / Markdown()

Trackio is a wandb drop-in replacement, local-first, free, Gradio dashboard, HF Spaces sync.

# tracking/trackio_service.py
import trackio
import time
from core.events import bus, EventType, Event

class TrackioService:
    def __init__(self, project: str, space_id: str | None = None):
        self.project  = project
        self.space_id = space_id
        self._active  = False
        self._register_handlers()

    def _register_handlers(self):
        @bus.on(EventType.TRAINING_STARTED)
        async def on_training_started(event: Event):
            trackio.init(
                project=self.project,
                space_id=self.space_id,
                run_name=event.payload.get("run_name", "training"),
            )
            self._active = True

        @bus.on(EventType.TRAINING_STEP)
        async def on_step(event: Event):
            if self._active:
                trackio.log(event.payload)

        @bus.on(EventType.TRAINING_FINISHED)
        async def on_done(event: Event):
            if self._active:
                trackio.log({"status": "finished", **event.payload})
                trackio.finish()
                self._active = False

        @bus.on(EventType.EVAL_FINISHED)
        async def on_eval(event: Event):
            trackio.init(
                project=self.project,
                space_id=self.space_id,
                run_name=event.payload.get("eval_run_name", "eval"),
            )
            trackio.log(event.payload)
            trackio.finish()

        @bus.on(EventType.INFERENCE_RESPONSE)
        async def on_inference(event: Event):
            # Log inference latency and token count
            trackio.init(
                project=self.project,
                space_id=self.space_id,
                run_name="inference",
            )
            trackio.log({
                "model":          event.payload.get("model"),
                "latency_ms":     event.payload.get("latency_ms"),
                "tokens_out":     event.payload.get("tokens_out"),
                "mode":           event.payload.get("mode", "text"),
            })
            trackio.finish()

        @bus.on(EventType.TOOL_CALL)
        async def on_tool(event: Event):
            trackio.alert(
                title=f"Tool call: {event.payload.get('tool_name')}",
                level=trackio.AlertLevel.INFO,
            )

TRL LoRA integration (recommended for training):

from trl import SFTConfig
training_args = SFTConfig(
    output_dir="./checkpoints",
    report_to="trackio",   # ← one-liner Trackio integration via TRL
    run_name="lora_minicpm5_1b",
)

SQL query via CLI (LLM-friendly):

trackio query project --project workbench --sql "SELECT run_name, loss FROM metrics ORDER BY loss LIMIT 5"

---

7. MCP Layer (Three Integration Paths)

Path 1: Gradio Blocks + launch(mcp_server=True) ← simplest

# app.py (simple path)
import gradio as gr
from mcp.server.fastmcp import FastMCP

demo = gr.Blocks()
# ... tabs ...
demo.launch(
    mcp_server=True,    # exposes /gradio_api/mcp/sse
    server_port=7860,
)

Every gr.Interface function and gr.api() endpoint becomes an MCP tool automatically. Docstrings become tool descriptions. Type hints become parameter schemas.

Path 2: gradio.Server (custom frontend + MCP)

New API published 2026-04-01. Extends FastAPI. Better for custom dashboards.

# app.py (Server path)
from gradio import Server
from fastapi.responses import HTMLResponse

app = Server()

@app.api(name="run_inference")             # Gradio-queued + MCP-compatible
def run_inference(prompt: str, model_id: str) -> str:
    """Run text inference on a MiniCPM text model."""
    svc = model_registry.get(model_id)
    return svc.generate(prompt)

@app.mcp.tool()                            # explicit MCP tool registration
async def export_gguf(model_id: str, quant: str = "Q4_K_M") -> str:
    """Export a Transformers model to GGUF format with specified quantization."""
    return await export_service.run(model_id, quant)

@app.get("/", response_class=HTMLResponse) # custom frontend
async def homepage():
    return open("ui/index.html").read()

app.launch(show_error=True)

Path 3: FastMCP standalone + Gradio client bridge

Use when you need stateful MCP tools, or when running multiple Gradio apps and want to save memory:

# mcp/server.py
from mcp.server.fastmcp import FastMCP
from gradio_client import Client

mcp = FastMCP("OpenBMBWorkbench")
_clients: dict[str, Client] = {}

def _gradio_client(space_id: str) -> Client:
    if space_id not in _clients:
        _clients[space_id] = Client(space_id)
    return _clients[space_id]

@mcp.tool()
async def calculate(expression: str) -> str:
    """Evaluate a mathematical expression."""
    import math
    return str(eval(expression, {"__builtins__": {}}, vars(math)))

@mcp.tool()
async def dataset_stats(dataset_name: str, split: str = "train") -> dict:
    """Return basic statistics for a HuggingFace dataset."""
    from datasets import load_dataset
    ds = load_dataset(dataset_name, split=split)
    return {
        "rows":    len(ds),
        "columns": ds.column_names,
        "features": str(ds.features),
    }

@mcp.tool()
async def search_dataset(query: str, max_results: int = 5) -> list[dict]:
    """Search HuggingFace Hub for datasets matching a query."""
    from huggingface_hub import list_datasets
    results = list(list_datasets(search=query, limit=max_results))
    return [{"id": d.id, "downloads": d.downloads, "tags": d.tags} for d in results]

@mcp.tool()
async def start_training(
    model_id: str,
    dataset_name: str,
    lora_rank: int = 16,
    epochs: int = 3,
) -> dict:
    """Start a LoRA fine-tuning run on the specified model and dataset."""
    # Dispatch to training service via event bus
    await bus.emit(Event(
        type=EventType.TRAINING_STARTED,
        payload={"model_id": model_id, "dataset": dataset_name,
                 "lora_rank": lora_rank, "epochs": epochs}
    ))
    return {"status": "started", "model_id": model_id}

@mcp.tool()
async def evaluate_model(model_id: str, dataset_name: str) -> dict:
    """Evaluate a model on a dataset and return metrics."""
    svc = model_registry.get(model_id)
    return await eval_service.run(svc, dataset_name)

@mcp.tool()
async def export_gguf(model_id: str, quant: str = "Q4_K_M") -> str:
    """Convert a fine-tuned model to GGUF format."""
    return await export_service.run(model_id, quant)

@mcp.tool()
async def vision_chat(image_path: str, prompt: str, model_id: str = "minicpm_v46") -> str:
    """Run multimodal inference on an image with a prompt."""
    svc: MiniCPMVisionService = model_registry.get(model_id)
    from PIL import Image
    img = Image.open(image_path).convert("RGB")
    messages = [{"role": "user", "content": [
        {"type": "image", "image": img},
        {"type": "text",  "text": prompt},
    ]}]
    return svc.chat(messages)

if __name__ == "__main__":
    mcp.run(transport="sse", port=8081)

MCP SSE endpoint: http://localhost:8081/sse Gradio MCP SSE: http://localhost:7860/gradio_api/mcp/sse

---

8. Training Pipeline

8.1 LoRA — Text Models (PEFT)

# training/lora.py
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer, SFTConfig

class LoRATextTrainer:
    DEFAULT_TARGET_MODULES = ["q_proj", "k_proj", "v_proj", "o_proj",
                               "gate_proj", "up_proj", "down_proj"]

    def __init__(self, cfg: dict):
        self.cfg = cfg

    def train(self, model, tokenizer, dataset, run_name: str):
        lora_cfg = LoraConfig(
            r=self.cfg.get("lora_rank", 16),
            lora_alpha=self.cfg.get("lora_alpha", 32),
            target_modules=self.cfg.get("target_modules", self.DEFAULT_TARGET_MODULES),
            lora_dropout=self.cfg.get("lora_dropout", 0.05),
            bias="none",
            task_type=TaskType.CAUSAL_LM,
        )
        model = get_peft_model(model, lora_cfg)
        model.print_trainable_parameters()

        training_args = SFTConfig(
            output_dir=f"./checkpoints/{run_name}",
            num_train_epochs=self.cfg.get("epochs", 3),
            per_device_train_batch_size=self.cfg.get("batch_size", 4),
            gradient_accumulation_steps=self.cfg.get("grad_accum", 4),
            learning_rate=self.cfg.get("lr", 2e-4),
            fp16=False,
            bf16=True,
            report_to="trackio",         # ← Trackio via TRL
            run_name=run_name,
            save_steps=100,
            logging_steps=10,
        )
        trainer = SFTTrainer(
            model=model,
            args=training_args,
            train_dataset=dataset,
            tokenizer=tokenizer,
        )
        trainer.train()
        return model

8.2 LoRA — Vision Models (SWIFT / LLaMA-Factory recommended)

OpenBMB recommends SWIFT or LLaMA-Factory for MiniCPM-V fine-tuning. Vision tower is frozen; only language backbone layers are adapted.

# training/lora_vision.py
class LoRAVisionConfig:
    """
    For MiniCPM-V 4.6 LoRA:
    - Freeze vision encoder (SigLIP2-400M) + projector
    - Apply LoRA only to Qwen3.5-0.8B language layers
    - Use SWIFT CLI or LLaMA-Factory for full pipeline
    """
    LANGUAGE_TARGET_MODULES = ["q_proj", "k_proj", "v_proj", "o_proj"]
    FROZEN_PREFIXES = ["vpm", "resampler", "vision_model"]  # freeze vision

    @staticmethod
    def freeze_vision(model):
        for name, param in model.named_parameters():
            if any(name.startswith(pfx) for pfx in LoRAVisionConfig.FROZEN_PREFIXES):
                param.requires_grad = False
        return model

SWIFT CLI approach (recommended for MiniCPM-V 4.6):

swift sft \
  --model openbmb/MiniCPM-V-4.6 \
  --dataset <your-dataset> \
  --lora_rank 16 \
  --freeze_vit true \
  --output_dir ./checkpoints/minicpm_v46_lora

8.3 HF Jobs Offload (ml-intern pattern)

For long training runs, offload to HF Jobs (free cpu-basic, paid gpu-a100):

# training/hf_jobs.py
from huggingface_hub import HfApi

class HFJobsTrainer:
    def submit(self, script_path: str, hardware: str = "gpu-a100") -> str:
        api = HfApi()
        job = api.create_job(
            repo_id=f"{api.whoami()['name']}/ml-workbench-jobs",
            script=script_path,
            hardware=hardware,
        )
        return job.job_id

8.4 Evaluation

# training/evaluation.py
class Evaluator:
    def run(self, model, tokenizer, dataset, metrics=("perplexity", "accuracy")) -> dict:
        results = {}
        if "perplexity" in metrics:
            results["perplexity"] = self._perplexity(model, tokenizer, dataset)
        if "accuracy" in metrics:
            results["accuracy"] = self._accuracy(model, tokenizer, dataset)
        return results

---

9. Export & Quantization

# training/export.py
import subprocess
from pathlib import Path

class GGUFExporter:
    SUPPORTED_QUANTS = ["F16", "Q4_K_M", "Q5_K_M", "Q8_0", "Q2_K"]

    def export(
        self,
        model_path: str,
        output_dir: str,
        quant: str = "Q4_K_M",
        model_type: str = "text",    # "text" | "vision"
    ) -> list[str]:
        output_dir = Path(output_dir)
        output_dir.mkdir(parents=True, exist_ok=True)

        # Step 1: Convert HF → GGUF F16
        # For MiniCPM-V 4.6: convert_hf_to_gguf.py produces main gguf + mmproj
        # For text models: same script, no mmproj
        f16_path = output_dir / "model-f16.gguf"
        subprocess.run([
            "python", "convert_hf_to_gguf.py",
            model_path,
            "--outtype", "f16",
            "--outfile", str(f16_path),
        ], check=True)

        outputs = [str(f16_path)]

        # Step 2: Quantize
        if quant != "F16":
            quant_path = output_dir / f"model-{quant}.gguf"
            subprocess.run([
                "llama-quantize",
                str(f16_path),
                str(quant_path),
                quant,
            ], check=True)
            outputs.append(str(quant_path))

        return outputs

Note: For MiniCPM-V 4.6, official GGUF files are already published at openbmb/MiniCPM-V-4.6-gguf. Download them directly unless you need custom quants.

---

10. Agent Mode (ml-intern Inspired)

The agent/ module implements a lightweight research-plan-implement loop inspired by HF's ml-intern (April 2026). This gives the workbench autonomous ML experimentation capabilities.

ml-intern architecture (reference):
  Research  → browse arXiv/HF Papers, read methodology, traverse citations
  Plan      → break task down, verify resources (models, datasets, hardware)
  Implement → execute scripts on HF Jobs / local GPU
  Trace     → auto-upload sessions as JSONL to HF Dataset

# agent/loop.py
from smolagents import CodeAgent, HfApiModel
from huggingface_hub import HfApi

class WorkbenchAgent:
    """
    Lightweight agent loop for autonomous experimentation.
    Uses smolagents + HF ecosystem + registered MCP tools.
    """
    def __init__(self, model_id: str = "claude-sonnet-4-6"):
        self.model = HfApiModel(model_id)
        self.api   = HfApi()
        self._session_log = []

    def run(self, prompt: str, max_steps: int = 20) -> str:
        agent = CodeAgent(
            tools=self._get_tools(),
            model=self.model,
            max_steps=max_steps,
        )
        result = agent.run(prompt)
        self._save_session(prompt, result)
        return result

    def _save_session(self, prompt: str, result: str):
        import json, datetime
        session = {
            "timestamp": datetime.datetime.utcnow().isoformat(),
            "prompt": prompt,
            "result": result,
            "log": self._session_log,
        }
        # Auto-upload to private HF dataset (ml-intern pattern)
        # Dataset: {username}/workbench-sessions
        pass

    def _get_tools(self):
        # Expose MCP tools as smolagents tools
        from mcp.server.fastmcp import FastMCP
        # ...
        return []

---

11. UI — Tab Specification

Using gr.Blocks with tabs. Optionally swap to gradio.Server for custom frontend.

# app.py
import gradio as gr
from ui.chat_tab    import build_chat_tab
from ui.vision_tab  import build_vision_tab
from ui.train_tab   import build_train_tab
from ui.dataset_tab import build_dataset_tab
from ui.export_tab  import build_export_tab
from ui.traces_tab  import build_traces_tab
from ui.agent_tab   import build_agent_tab
from ui.notes_tab   import build_notes_tab

with gr.Blocks(title="OpenBMB Workbench", theme=gr.themes.Ocean()) as demo:
    gr.Markdown("# 🔬 OpenBMB Local AI Workbench")

    with gr.Tabs():
        with gr.Tab("💬 Chat"):       build_chat_tab()
        with gr.Tab("👁️ Vision"):     build_vision_tab()
        with gr.Tab("📊 Dataset"):    build_dataset_tab()
        with gr.Tab("🏋️ Train"):      build_train_tab()
        with gr.Tab("📦 Export"):     build_export_tab()
        with gr.Tab("📈 Traces"):     build_traces_tab()
        with gr.Tab("🤖 Agent"):      build_agent_tab()
        with gr.Tab("📝 Field Notes"):build_notes_tab()

demo.launch(
    mcp_server=True,    # SSE at /gradio_api/mcp/sse
    server_port=7860,
    share=False,
)

Tab Summary

Tab	Purpose	Key components
Chat	Text inference on all text models	Model selector, streaming output, system prompt
Vision	Image + text inference on V-4.6/Thinking	Image upload, thinking toggle, bounding box overlay
Dataset	HF Hub browser + local CSV/JSON loader	Search bar, split preview, schema inspector
Train	LoRA training launcher	Config form, live loss chart (Trackio), checkpoint browser
Export	GGUF export + quantization	Model selector, quant dropdown, file download
Traces	Trackio run browser + comparison	Run table, metric plots, SQL query box
Agent	ml-intern style agent loop	Prompt, step log, paper browser, HF Jobs monitor
Field Notes	Correction capture → retrain trigger	Image+prompt+response+correction form, JSONL export

Vision Tab — key detail

# ui/vision_tab.py
def build_vision_tab():
    with gr.Column():
        model_dd = gr.Dropdown(
            choices=["minicpm_v46", "minicpm_v46_thinking", "minicpm_o45"],
            value="minicpm_v46",
            label="Vision Model"
        )
        thinking_cb = gr.Checkbox(label="Enable Thinking Mode", value=False)
        image_in    = gr.Image(type="pil", label="Upload Image")
        video_in    = gr.Video(label="Upload Video (MiniCPM-V-4.6 supports video)")
        prompt_in   = gr.Textbox(label="Prompt")
        submit_btn  = gr.Button("Run")
        output_txt  = gr.Textbox(label="Response", lines=10)

        def infer(model_id, thinking, image, video, prompt):
            svc = model_registry.get(model_id)
            content = []
            if image:
                content.append({"type": "image", "image": image})
            if video:
                content.append({"type": "video", "video": video})
            content.append({"type": "text", "text": prompt})
            msgs = [{"role": "user", "content": content}]
            svc.thinking = thinking
            return svc.chat(msgs)

        submit_btn.click(infer, [model_dd, thinking_cb, image_in, video_in, prompt_in], output_txt)

---

12. Field Notes & Correction Loop

# datasets/field_notes.py
import sqlite3, json
from dataclasses import dataclass, field, asdict
from datetime import datetime

@dataclass
class FieldNote:
    id:          str = field(default_factory=lambda: datetime.utcnow().isoformat())
    model_id:    str = ""
    modality:    str = "text"      # "text" | "image" | "video" | "multimodal"
    image_path:  str | None = None
    video_path:  str | None = None
    prompt:      str = ""
    response:    str = ""
    correction:  str = ""          # human-corrected output
    tags:        list = field(default_factory=list)
    used_in_train: bool = False

class FieldNoteStore:
    def __init__(self, db_path: str = "data/field_notes.db"):
        self.conn = sqlite3.connect(db_path, check_same_thread=False)
        self._init_db()

    def _init_db(self):
        self.conn.execute("""
            CREATE TABLE IF NOT EXISTS notes (
                id TEXT PRIMARY KEY,
                data JSON NOT NULL,
                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
            )
        """)
        self.conn.commit()

    def save(self, note: FieldNote):
        self.conn.execute(
            "INSERT OR REPLACE INTO notes (id, data) VALUES (?, ?)",
            (note.id, json.dumps(asdict(note)))
        )
        self.conn.commit()

    def to_hf_dataset(self, output_path: str):
        """Export uncorrected notes as HF Dataset for retraining."""
        from datasets import Dataset
        rows = [json.loads(r[0]) for r in
                self.conn.execute("SELECT data FROM notes WHERE json_extract(data,'$.correction') != ''")]
        ds = Dataset.from_list(rows)
        ds.save_to_disk(output_path)
        return ds

---

13. Directory Structure

ml-workbench/
├── app.py                         # Entry point: gr.Blocks or gradio.Server
├── requirements.txt
│
├── config/
│   ├── models.yaml                # ← TEMPLATE AXIS: change this for any model family
│   └── training.yaml              # LoRA hyperparams, export settings
│
├── core/
│   ├── events.py                  # EventBus, EventType enum
│   ├── registry.py                # Registry[T] generic
│   └── state.py                   # Global AppState dataclass
│
├── models/
│   ├── base.py                    # Abstract ModelService
│   ├── hf_loader.py               # Generic HF Hub downloader
│   ├── minicpm_text.py            # MiniCPM5-1B, MiniCPM4.1-8B service
│   ├── minicpm_vision.py          # MiniCPM-V-4.6 / Thinking service
│   ├── minicpm_omni.py            # MiniCPM-o-4.5 service
│   ├── llama_cpp_runner.py        # llama-server wrapper
│   ├── sglang_runner.py           # SGLang server wrapper (tool-use)
│   └── ollama_runner.py           # Ollama client
│
├── datasets/
│   ├── loader.py                  # Abstract DatasetLoader
│   ├── hf_datasets.py             # HF Hub datasets.load_dataset()
│   ├── field_notes.py             # SQLite-backed correction store
│   └── synthetic.py               # ml-intern style synthetic data gen
│
├── training/
│   ├── lora.py                    # LoRA text trainer (PEFT + TRL)
│   ├── lora_vision.py             # LoRA vision config (SWIFT/LLaMA-Factory)
│   ├── evaluation.py              # Perplexity, accuracy, custom metrics
│   ├── export.py                  # GGUF exporter (convert_hf_to_gguf.py)
│   └── hf_jobs.py                 # HF Jobs offload (ml-intern pattern)
│
├── tools/
│   ├── calculator.py              # Safe math eval tool
│   ├── dataset_stats.py           # Dataset statistics tool
│   ├── hf_search.py               # HF Hub search tool
│   └── paper_search.py            # arXiv / HF Papers search (ml-intern)
│
├── tracking/
│   └── trackio_service.py         # Trackio integration (init/log/finish/alert)
│
├── mcp/
│   ├── server.py                  # FastMCP standalone server
│   └── tools.py                   # All @mcp.tool() decorators
│
├── agent/
│   ├── loop.py                    # Research → Plan → Implement agent
│   └── prompts/
│       └── system_prompt.yaml     # Agent system prompt (ml-intern style)
│
├── ui/
│   ├── chat_tab.py
│   ├── vision_tab.py
│   ├── train_tab.py
│   ├── dataset_tab.py
│   ├── export_tab.py
│   ├── traces_tab.py
│   ├── agent_tab.py
│   └── notes_tab.py
│
├── exports/                       # GGUF + quantized model outputs
└── data/                          # Cached datasets, field_notes.db

---

14. Configuration Schema

training.yaml

# config/training.yaml
lora:
  rank:          16
  alpha:         32
  dropout:       0.05
  target_modules_text:
    - q_proj
    - k_proj
    - v_proj
    - o_proj
    - gate_proj
    - up_proj
    - down_proj
  target_modules_vision:
    - q_proj
    - k_proj
    - v_proj
    - o_proj                       # language layers only; vision tower frozen

training:
  epochs:        3
  batch_size:    4
  grad_accum:    4
  lr:            2.0e-4
  bf16:          true
  report_to:     trackio           # or: wandb, tensorboard, none

export:
  default_quant: Q4_K_M
  supported_quants:
    - F16
    - Q4_K_M
    - Q5_K_M
    - Q8_0
    - Q2_K

trackio:
  project:   ml-workbench
  space_id:  null                  # set to "username/trackio-space" for HF sync

hf_jobs:
  enabled:   false
  hardware:  gpu-a100

---

15. Dependencies

# requirements.txt
# Core
gradio[mcp]>=5.38.0
mcp[cli]>=1.0.0
fastapi
uvicorn[standard]

# Models
transformers>=5.7.0               # required for MiniCPM-V-4.6
torch>=2.1.0
torchvision
torchcodec                        # video decoding for MiniCPM-V-4.6
accelerate
bitsandbytes

# Training
peft>=0.10.0
trl>=0.9.0
datasets>=2.18.0

# Tracking
trackio>=0.1.0

# MCP + Agents
mcp>=1.0.0
smolagents>=1.0.0

# Serving backends (optional, install as needed)
# sglang                          # pip install sglang
# vllm                            # pip install vllm
# ollama                          # install separately

# Hub
huggingface_hub>=0.23.0
Pillow
requests
pyyaml

---

16. Hackathon Demo Flow

1. DATASET
   Load HF dataset via Dataset tab (e.g. "tatsu-lab/alpaca", 10K rows)
   → EventType.DATASET_LOADED
   → Trackio: trackio.init("demo-run") + trackio.log({"dataset": ..., "rows": ...})

2. TRAIN
   Start LoRA on MiniCPM5-1B (rank=16, 1 epoch demo)
   → EventType.TRAINING_STARTED
   → TRL SFTTrainer + report_to="trackio" → live loss in Traces tab

3. EXPORT
   Click "Export GGUF" → Q4_K_M
   → convert_hf_to_gguf.py → llama-quantize
   → EventType.EXPORT_FINISHED → file download

4. LLAMA.CPP CHAT
   Load exported GGUF in llama-server
   → Chat tab, backend: llama.cpp
   → Verify text generation quality

5. SWITCH TO VISION
   Select MiniCPM-V-4.6 in Vision tab
   Upload image → enter prompt → Run

6. THINKING MODE
   Toggle "Enable Thinking Mode" → switch to MiniCPM-V-4.6-Thinking
   → Observe explicit reasoning trace in response

7. FIELD NOTE
   Response is wrong → open Field Notes tab
   → Enter correction → Save
   → EventType.FIELD_NOTE_SAVED

8. AGENT MODE
   Agent tab: "Improve MiniCPM-V-4.6 on [domain] using field notes"
   → Research phase: browse HF Papers
   → Plan: dataset selection, LoRA config
   → Implement: trigger SWIFT fine-tune

9. SHARE TRACES
   Traces tab → set space_id → Sync to HF Space
   → Shareable dashboard URL

---

17. Corrections from PRD v1

v1 Claim	Correct v2
trackio.trace("vision_inference") context manager	Does not exist. Use trackio.init() + trackio.log() + trackio.finish()
AutoModelForCausalLM for MiniCPM-V-4.6	Wrong. Use AutoModelForImageTextToText with transformers>=5.7.0
llama-mtmd-cli for vision deployment	Use llama-server --mmproj <mmproj.gguf> for server; llama-mtmd-cli for CLI-only
Run convert_hf_to_gguf.py for V4.6	Official GGUFs at openbmb/MiniCPM-V-4.6-gguf — download directly; run script only for custom quants
Surgery scripts for V4.6 GGUF	Surgery scripts are for MiniCPM-o-2.6 only; do not use for V4.6
future MiniCPM-o (vague)	MiniCPM-o-4.5 released 2026-05-17; specific HF ID available
app.launch(mcp_server=True) only option	Two paths: gr.Blocks + mcp_server=True, or gradio.Server with @app.mcp.tool()
FastMCP as only MCP approach	Three paths: Gradio native, gradio.Server, standalone FastMCP (see §7)
MiniCPM4.1-8B: standard architecture	Has sparse InfLLM-v2 attention; requires --trust-remote-code
LoRA for vision: PEFT only	OpenBMB recommends SWIFT or LLaMA-Factory for MiniCPM-V fine-tuning
No HF Jobs mention	HF Jobs integration (ml-intern pattern) enables GPU cloud offload
No SGLang mention	SGLang is the recommended backend for MiniCPM5-1B tool calling
No video support	MiniCPM-V-4.6 supports video understanding (torchcodec required)

---

18. Roadmap & Extension Points

Swap the model family (template portability)

To build a "Qwen3 Workbench":

1. Edit config/models.yaml — add Qwen3 model IDs
2. Create models/qwen3_service.py extending ModelService
3. Register in app.py — done

No changes to training, export, MCP, Trackio, or UI layers.

Planned extensions

Feature	Module	Notes
vLLM serving tab	models/vllm_runner.py	Batch inference, production
Ollama quick-start	models/ollama_runner.py	Zero-config consumer demo
Reward model eval	training/reward_eval.py	For RLHF experiments
Synthetic data gen	datasets/synthetic.py	ml-intern pattern: LLM writes training data
Paper-to-code agent	agent/paper_agent.py	Read arXiv → implement in workbench
HF Spaces deploy	deploy/spaces.py	One-click push as HF Space
VINDEX integration	tools/vindex_tool.py	Knowledge editing via your VINDEX engine
OCR pipeline hook	datasets/ocr_loader.py	Feed your OCR pipeline output as field notes
MiniCPM Desk-Pet	agent/desk_pet.py	LoRA persona switching via MiniCPM5-1B
MiniCPM-o audio	ui/audio_tab.py	Real-time omnimodal via MiniCPM-o-4.5

---

PRD v2.0 — Christof Kaller / ki-fusion-labs.de — 2026-06-05 Research sources: openbmb/MiniCPM-V, openbmb/MiniCPM, gradio-app/trackio, huggingface/ml-intern, huggingface.co/blog/introducing-gradio-server