docs: pre-refactor backup - before tech log integration

docs/总结报告_modal部署过程.md

@@ -0,0 +1,136 @@
+# Modal 部署 MiniCPM-o 4.5 — 工作总结报告
+
+## 角色定位
+
+作为并行 Cline 之一，任务是 **使用 llama.cpp 在 Modal 部署 MiniCPM-o 4.5 并设计好 API 接口**。
+
+---
+
+## 已完成工作
+
+### 1. 代码阅读与分析
+- 阅读 docs/ 下所有文件：部署经验、技术报告、开发日志、项目架构
+- 分析现有 deploy.py、client.py、_test_inference.py
+- 阅读 MiniCPM-V-Cookbook 官方部署指南 (`deployment/llama.cpp/minicpm-o4_5_llamacpp_zh.md`)
+
+### 2. deploy.py 修复（第一轮部署）
+- 原 bug：`find_mmproj_files()` 递归搜索目录下**全部** `.gguf` 传给 `--mmproj`，导致启动失败
+- 修复：指定单一 vision mmproj 路径
+- 更新 API 端点为 OpenAI 兼容格式（`/v1/chat/completions`）
+- 移除冗余的 `supports_gpu()` 验证行
+- 更新 client.py 和 _test_inference.py 对齐接口
+
+### 3. Git 版本管理
+- 多 Cline 并行冲突处理：rebase + force push
+- 最后一次提交：`aa40303`
+
+### 4. 第一轮 Modal 部署
+- **镜像构建成功**（55s）：llama-cpp-python 源码编译通过
+- **部署成功**（`modal deploy`）：
+  ```
+  https://andrew-jiabin--prego-pal-minicpm-serve.modal.run
+  ```
+- 修复 Windows 终端 GBK 编码 Bug（`PYTHONIOENCODING=utf-8`）
+
+### 5. 模型文件检查与重新下载
+- **发现问题**：本地 `MiniCPM-o-4_5-Q4_K_M.gguf` 仅 **721 MB**（正常应为 ~5 GB），文件损坏/不完整
+- **下载修复**：
+  - `huggingface-cli` 已废弃 → 改用 `hf download`
+  - 参数 `--dest` 不存在 → 改用 `--local-dir`
+  - **成功下载完整模型：5,026,714,400 bytes（5.0 GB）** ✓
+- **Vision 文件检查**：`MiniCPM-o-4_5-vision-F16.gguf` = **1,095,113,184 bytes（1.1 GB）** ✓
+
+### 6. Modal Volume 重新上传
+- **清空旧文件**：`modal volume rm minicpm-o-4_5-models /MiniCPM-o-4_5-gguf -r` ✓
+- **上传新文件**：`modal volume put ...` ✓（4.7 GiB）
+- **验证 Volume 结构**：
+  ```
+  MiniCPM-o-4_5-gguf/MiniCPM-o-4_5-Q4_K_M.gguf  (4.7 GiB)
+  MiniCPM-o-4_5-gguf/vision/                     (含 vision-F16.gguf)
+  MiniCPM-o-4_5-gguf/audio/
+  MiniCPM-o-4_5-gguf/tts/
+  MiniCPM-o-4_5-gguf/token2wav-gguf/
+  MiniCPM-o-4_5-gguf/.cache/
+  ```
+
+### 7. 编译 llama.cpp（本地）
+- **克隆**：`git clone https://github.com/ggml-org/llama.cpp.git` ✓
+- **编译**：`cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release` → `cmake --build build --config Release -j` ✓
+- **编译产物**：`llama-mtmd-cli.exe`（64 KB）
+- **关键发现**：`-i` 不是有效参数，需用 `--image` 指定图片路径
+
+### 8. 本地推理测试 ✅
+- **纯文本测试**：正常输出 ✓
+- **多模态测试**（带图片）✅：
+  ```
+  llama-mtmd-cli -m MiniCPM-o-4_5-Q4_K_M.gguf \
+    --mmproj MiniCPM-o-4_5-vision-F16.gguf \
+    -c 4096 --temp 0.7 --top-p 0.8 \
+    --top-k 100 --repeat-penalty 1.05 \
+    --image test.jpg -p "用中文描述这张图片里的内容"
+  ```
+- **结果**：成功识别图片中的 "↓买入" 图标并输出中文描述 ✓
+- **加载时间**：约 **4 秒**（GPU CUDA）✓
+- **警告**：`n_ctx_seq (4096) < n_ctx_train (40960)` — 不影响功能
+
+---
+
+## 当前问题与关键发现
+
+### ❌ 第一轮部署模型加载失败
+```
+ValueError: Failed to load model from file:
+/models/MiniCPM-o-4_5-gguf/MiniCPM-o-4_5-Q4_K_M.gguf
+```
+- 损坏的 721 MB GGUF 文件是根本原因
+- 现在已上传 5.0 GB 完整文件，预计可以解决
+
+### ⚠️ llama-cpp-python 兼容性问题
+- 本地验证 `llama-mtmd-cli` 可以正常工作
+- deploy.py 用的 `llama-cpp-python`（Python 绑定）可能不支持 `mtmd` 架构
+- 如果 Python 绑定失败，需要改用 `llama-server` 方案（在 Modal 内直接启动编译好的 `llama-mtmd-cli` 作为独立进程）
+
+---
+
+## 已创建的工作文件
+
+| 文件 | 用途 |
+|------|------|
+| `tmp_download_instructions.md` | 下载完整模型的命令 |
+| `tmp_official_deploy_guide.md` | 官方教程关键发现汇总 |
+| `tmp_upload_and_test_guide.md` | 上传到 Modal + 本地检查步骤 |
+| `tmp_volume_cleanup_guide.md` | Volume 清理指南 |
+| `tmp_build_llamacpp_guide.md` | 编译 llama.cpp 步骤 |
+| `tmp_llama_mtmd_test_guide.md` | 本地推理测试命令 |
+
+---
+
+## 对项目整体架构的见解
+
+### 当前架构过于复杂
+
+| 问题 | 具体表现 |
+|------|----------|
+| 模块太多 | plugins/ 下有 9 个插件，很多未收尾 |
+| 耦合过深 | core/、modules/、plugins/ 三层抽象，实际功能重复 |
+| 数据分散 | data/ 下有多个 json 文件，结构不一致 |
+| 前端缺失 | deploy.py 已部署但前端 app.py 还未真正对接 |
+| 依赖复杂 | requirements.txt 依赖多，部署环境兼容性难保证 |
+
+### 建议：简化到最小可行产品
+
+对于一个黑客松项目，建议：
+1. **只保留核心 API**：`/v1/chat/completions`（文本）+ `/v1/vision`（多模态）
+2. **前端直接对接 OpenAI 格式**：任何兼容 OpenAI SDK 的客户端都能用
+3. **删除冗余模块**：plugins/ 和 modules/ 中未收尾的部分
+4. **模型部署按官方教程走**：MiniCPM-V-Cookbook 有现成例子
+
+---
+
+## 下一步计划
+
+1. ✅ 下载完整模型（已完成 5.0 GB）
+2. ✅ 上传到 Modal Volume（已完成）
+3. ✅ 编译 llama.cpp + `llama-mtmd-cli`（已完成）
+4. ✅ 本地验证推理（已完成 — 成功识别图片）
+5. ❌ 重新部署到 Modal + 测试 API

modal_deploy/deploy.py

@@ -1,49 +1,50 @@
 """
-PregoPal × MiniCPM-o-4_5 — Modal 部署 (预编译 llama-cpp-python)
+PregoPal × MiniCPM-o-4_5 — Modal 部署 (llama-server via subprocess)
 
 架构:
-  FastAPI (ASGI) ←→ llama-cpp-python (CUDA via pre-built wheel)
-                         ↕
-                  Modal Volume: GGUF models
+  FastAPI (ASGI) ←→ httpx proxy ←→ llama-server (subprocess, localhost:8080)
+                                   ↕
+                            Modal Volume: GGUF models
+
+llama-server 原生支持 OpenAI 兼容 API，本 wrapper 透明代理请求，
+支持 streaming 和普通请求。
 
 用法:
-    pip install modal      # 安装 Modal CLI
-    modal token new        # 登录 Modal
-    modal deploy modal_deploy.deploy    # 部署 (2-3 min)
+    pip install modal
+    modal token new
+    modal deploy -m modal_deploy.deploy
 
 测试:
     modal run modal_deploy.deploy::test_inference
-
-API:
-    POST /v1/chat/completions   — OpenAI 兼容 (支持 streaming)
-    POST /v1/completions        — Text completion
-    POST /v1/embeddings         — Embeddings
-    POST /v1/vision             — 多模态 (图片+文字)
-    GET  /health                — 健康检查
-    GET  /v1/models             — 模型列表
 """
 
 import os
+import subprocess
+import time
+import logging
+import signal
+import asyncio
+import json
+
 import modal
 from modal import Image, App, Volume, asgi_app
 
 # ════════════════════════════════════════════════════════════════════
-# 1. IMAGE — 预编译 CUDA wheel (不从头编译)
+# 1. IMAGE — 编译 llama.cpp (CUDA 12.1, llama-server)
 # ════════════════════════════════════════════════════════════════════
 
 _image = (
-    Image.debian_slim(python_version="3.11")
-    # → 只安装 Python 依赖，不安装 cmake/gcc/CUDA toolkit
-    .pip_install("fastapi", "uvicorn[standard]", "httpx", "numpy", "Pillow")
-    # → 从 ggml-org 官方索引安装预编译 CUDA wheel（几秒完成）
-    .pip_install(
-        "llama-cpp-python",
-        extra_index_url="https://ggml-org.github.io/llama-cpp-python/whl/cu121",
-        force_build=True,
+    Image.from_registry("nvidia/cuda:12.1.0-devel-ubuntu22.04", add_python="3.11")
+    .apt_install("git", "cmake", "build-essential")
+    .pip_install("fastapi", "uvicorn[standard]", "httpx")
+    .run_commands(
+        "git clone --depth 1 https://github.com/ggml-org/llama.cpp.git /llama.cpp",
     )
-    # → 验证安装
     .run_commands(
-        "python -c 'from llama_cpp import Llama; print(f\"llama-cpp OK, GPU: {Llama.supports_gpu()}\")'",
+        "cd /llama.cpp && cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release",
+        "cd /llama.cpp && cmake --build build --config Release --target llama-server -j$(nproc)",
+        # 验证编译成功
+        "test -f /llama.cpp/build/bin/llama-server && echo '✅ llama-server built'",
     )
 )
 
@@ -55,46 +56,29 @@ MODEL_DIR = "/models"
 MODEL_SUBDIR = f"{MODEL_DIR}/MiniCPM-o-4_5-gguf"
 MAIN_GGUF = "MiniCPM-o-4_5-Q4_K_M.gguf"
 VISION_MMPROJ = "vision/MiniCPM-o-4_5-vision-F16.gguf"
+LLAMA_SERVER_PORT = 8080
 
 model_volume = Volume.from_name("minicpm-o-4_5-models", create_if_missing=True)
 app = App("prego-pal-minicpm")
 
-
-def get_model_paths(base_dir: str) -> dict:
-    """返回经验证的模型路径."""
-    main_path = os.path.join(base_dir, MAIN_GGUF)
-    vision_path = os.path.join(base_dir, VISION_MMPROJ)
-    paths = {"main": main_path, "vision": vision_path}
-    for key, path in paths.items():
-        print(f"[PregoPal] {key}: {path} (exists={os.path.isfile(path)})")
-    return paths
-
-
 # ════════════════════════════════════════════════════════════════════
-# 3. ASGI APP  — 多模态 API
+# 3. ASGI APP — 透明代理至 llama-server
 # ════════════════════════════════════════════════════════════════════
 
 @app.function(
     image=_image,
     volumes={MODEL_DIR: model_volume},
-    scaledown_window=300,
     gpu="A100",
     timeout=1200,
-    container_idle_timeout=300,
-    allow_concurrent_inputs=10,
+    # 至少保留一个实例，避免冷启动
+    min_containers=1 if os.environ.get("MODAL_KEEP_WARM") else 0,
 )
 @asgi_app()
 def serve():
-    import asyncio
-    import json
-    import logging
-    import base64
-    from io import BytesIO
-    from pathlib import Path
     from fastapi import FastAPI, Request
     from fastapi.responses import StreamingResponse, JSONResponse
     from fastapi.middleware.cors import CORSMiddleware
-    from llama_cpp import Llama
+    import httpx
 
     logging.basicConfig(level=logging.INFO)
     logger = logging.getLogger("prego-pal")
@@ -108,181 +92,183 @@ def serve():
         allow_headers=["*"],
     )
 
-    # ── Model Loading ──────────────────────────────────────────────
-    paths = get_model_paths(MODEL_SUBDIR)
-    model_path = paths["main"]
-    vision_path = paths["vision"]
-
-    kwargs: dict = dict(
-        model_path=model_path,
-        n_gpu_layers=-1,
-        n_ctx=8192,
-        verbose=False,
-        n_threads=os.cpu_count() or 4,
-    )
+    # ── 模型路径 ──────────────────────────────────────────────────
+    main_path = os.path.join(MODEL_SUBDIR, MAIN_GGUF)
+    vision_path = os.path.join(MODEL_SUBDIR, VISION_MMPROJ)
+
+    logger.info(f"[PregoPal] main model: {main_path} (exists={os.path.isfile(main_path)})")
+    logger.info(f"[PregoPal] vision: {vision_path} (exists={os.path.isfile(vision_path)})")
+
+    if not os.path.isfile(main_path):
+        logger.error(f"[PregoPal] ❌ Model not found at {main_path}")
+        raise FileNotFoundError(f"Model not found: {main_path}")
+
+    # ── 查找 llama-server ────────────────────────────────────────
+    LLAMA_SERVER = "/llama.cpp/build/bin/llama-server"
+    if not os.path.isfile(LLAMA_SERVER):
+        alt = "/llama.cpp/build/bin/Release/llama-server"
+        if os.path.isfile(alt):
+            LLAMA_SERVER = alt
+        else:
+            logger.error(f"[PregoPal] ❌ llama-server not found")
+            raise FileNotFoundError("llama-server binary not found")
+
+    # ── 构造启动命令 ──────────────────────────────────────────────
+    cmd = [
+        LLAMA_SERVER,
+        "-m", main_path,
+        "--host", "127.0.0.1",
+        "--port", str(LLAMA_SERVER_PORT),
+        "-c", "4096",
+        "--temp", "0.7",
+        "--top-p", "0.8",
+        "--top-k", "100",
+        "--repeat-penalty", "1.05",
+        "-ngl", "-1",           # 所有层 GPU
+        "--no-mmap",            # Modal Volume 需要
+        "--no-warmup",          # 节约启动时间
+    ]
     if os.path.isfile(vision_path):
-        kwargs["mmproj"] = vision_path
-        logger.info("[PregoPal] ✅ Vision mmproj enabled")
+        cmd.extend(["--mmproj", vision_path])
     else:
-        logger.warning(f"[PregoPal] ⚠️  mmproj not found at {vision_path} — vision disabled")
-
-    logger.info("[PregoPal] Loading model (30-90s)...")
-    try:
-        llm = Llama(**kwargs)
-        logger.info("[PregoPal] ✅ Model loaded!")
-    except Exception as e:
-        logger.error(f"[PregoPal] ❌ Failed to load model: {e}")
-        raise
-
-    # ── Helpers ────────────────────────────────────────────────────
-
-    def _parse_messages(messages: list) -> str:
-        """Convert messages list to a prompt string."""
-        texts = []
-        for msg in messages:
-            role = msg.get("role", "user")
-            content = msg.get("content", "")
-            if isinstance(content, list):
-                parts = []
-                for part in content:
-                    if isinstance(part, dict):
-                        if part.get("type") == "text":
-                            parts.append(part.get("text", ""))
-                        elif part.get("type") == "image_url":
-                            parts.append("[IMAGE]")
-                    else:
-                        parts.append(str(part))
-                content = " ".join(parts)
-            texts.append(f"<|{role}|>\n{content}\n<|assistant|>\n")
-        return "".join(texts)
-
-    def _extract_image(messages: list) -> bytes | None:
-        """Extract the first base64 image from messages."""
-        for msg in messages:
-            content = msg.get("content", "")
-            if isinstance(content, list):
-                for part in content:
-                    if isinstance(part, dict) and part.get("type") == "image_url":
-                        url = part.get("image_url", {}).get("url", "")
-                        if url.startswith("data:image"):
-                            _, b64 = url.split(",", 1)
-                            return base64.b64decode(b64)
-        return None
-
-    # ── Endpoints ──────────────────────────────────────────────────
+        logger.warning("[PregoPal] ⚠️ Vision mmproj not found — vision disabled")
+
+    logger.info(f"[PregoPal] Starting: {' '.join(cmd)}")
+
+    process = subprocess.Popen(
+        cmd,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT,
+        bufsize=1,
+    )
+
+    # ── 等待服务器就绪 ────────────────────────────────────────────
+    base_url = f"http://127.0.0.1:{LLAMA_SERVER_PORT}"
+    max_wait = 180  # 最多等 3 分钟（大模型加载需要时间）
+    ready = False
+    for i in range(max_wait):
+        try:
+            r = httpx.get(f"{base_url}/health", timeout=3)
+            if r.status_code == 200:
+                ready = True
+                logger.info(f"[PregoPal] ✅ llama-server ready after {i + 1}s")
+                break
+        except Exception:
+            pass
+        time.sleep(1)
+
+    if not ready:
+        # 输出日志帮助诊断
+        try:
+            stdout_data = process.stdout.read(4096).decode(errors="replace")
+            logger.error(f"[PregoPal] ❌ llama-server output:\n{stdout_data}")
+        except Exception:
+            pass
+        raise RuntimeError("llama-server failed to start within timeout")
+
+    # ── 后台读取 llama-server 日志 ────────────────────────────────
+
+    async def _tail_logs():
+        """异步读取 llama-server stdout 并输出到 logger."""
+        loop = asyncio.get_event_loop()
+        while True:
+            try:
+                line = await loop.run_in_executor(None, process.stdout.readline)
+                if not line:
+                    break
+                msg = line.decode(errors="replace").strip()
+                if msg:
+                    logger.info(f"[llama-server] {msg}")
+            except Exception:
+                break
+
+    @web_app.on_event("startup")
+    async def startup():
+        asyncio.create_task(_tail_logs())
+
+    # ── 公共 httpx 客户端 ──────────────────────────────────────────
+    # 注意：stream=True 需要在生命周期内手动管理
+    client = httpx.AsyncClient(base_url=base_url, timeout=180)
+
+    # ════════════════════════════════════════════════════════════════
+    # 4. PROXY ENDPOINTS
+    # ════════════════════════════════════════════════════════════════
 
     @web_app.post("/v1/chat/completions")
     async def chat_completions(request: Request):
         body = await request.json()
         stream = body.get("stream", False)
-        messages = body.get("messages", [])
-        max_tokens = body.get("max_tokens", 512)
-        temperature = body.get("temperature", 0.7)
-        top_p = body.get("top_p", 0.9)
 
         if stream:
+            # Streaming: 透传 SSE
+            req = client.build_request("POST", "/v1/chat/completions", json=body)
+            resp = await client.send(req, stream=True)
+
             async def event_stream():
-                for chunk in llm.create_chat_completion(
-                    messages=messages,
-                    max_tokens=max_tokens,
-                    temperature=temperature,
-                    top_p=top_p,
-                    stream=True,
-                ):
-                    yield f"data: {json.dumps(chunk)}\n\n"
+                async for chunk in resp.aiter_text():
+                    yield f"data: {chunk}\n\n"
                 yield "data: [DONE]\n\n"
-            return StreamingResponse(event_stream(), media_type="text/event-stream")
 
-        result = llm.create_chat_completion(
-            messages=messages,
-            max_tokens=max_tokens,
-            temperature=temperature,
-            top_p=top_p,
-            stream=False,
-        )
-        return JSONResponse(result)
+            return StreamingResponse(
+                event_stream(),
+                media_type="text/event-stream",
+                headers=resp.headers,
+            )
+
+        # 普通请求
+        resp = await client.post("/v1/chat/completions", json=body)
+        return JSONResponse(resp.json(), status_code=resp.status_code)
 
     @web_app.post("/v1/completions")
     async def completions(request: Request):
         body = await request.json()
-        prompt = body.get("prompt", "")
-        max_tokens = body.get("max_tokens", 256)
-
-        result = llm.create_completion(
-            prompt=prompt,
-            max_tokens=max_tokens,
-            temperature=body.get("temperature", 0.7),
-            stream=False,
-        )
-        return JSONResponse(result)
+        resp = await client.post("/v1/completions", json=body)
+        return JSONResponse(resp.json(), status_code=resp.status_code)
 
     @web_app.post("/v1/embeddings")
     async def embeddings(request: Request):
         body = await request.json()
-        result = llm.create_embedding(
-            input=body.get("input", ""),
-            model=body.get("model", "MiniCPM-o-4_5"),
-        )
-        return JSONResponse(result)
-
-    @web_app.post("/v1/vision")
-    async def vision(request: Request):
-        """
-        多模态推理：接收图片（base64）和文本提示。
-        如果 llm 未加载 mmproj，返回 400。
-        """
-        body = await request.json()
-        messages = body.get("messages", [])
-        max_tokens = body.get("max_tokens", 512)
-        temperature = body.get("temperature", 0.7)
-
-        if not os.path.isfile(vision_path):
-            return JSONResponse(
-                {"error": "Vision mmproj not loaded — deploy the model with mmproj file"},
-                status_code=400,
-            )
-
-        # llama-cpp-python 的 create_chat_completion 原生支持多模态
-        result = llm.create_chat_completion(
-            messages=messages,
-            max_tokens=max_tokens,
-            temperature=temperature,
-            stream=False,
-        )
-        return JSONResponse(result)
+        resp = await client.post("/v1/embeddings", json=body)
+        return JSONResponse(resp.json(), status_code=resp.status_code)
 
     @web_app.get("/health")
     async def health():
-        return {
-            "status": "ok",
-            "model": "MiniCPM-o-4_5",
-            "cuda": True,
-            "vision": os.path.isfile(vision_path),
-        }
+        try:
+            resp = await client.get("/health")
+            return JSONResponse(resp.json(), status_code=resp.status_code)
+        except Exception as e:
+            return JSONResponse(
+                {"status": "degraded", "llama_server": False, "error": str(e)},
+                status_code=503,
+            )
 
     @web_app.get("/v1/models")
     async def list_models():
-        return {
-            "object": "list",
-            "data": [{
-                "id": "MiniCPM-o-4_5",
-                "object": "model",
-                "created": 1,
-                "owned_by": "prego-pal",
-            }],
-        }
+        try:
+            resp = await client.get("/v1/models")
+            return JSONResponse(resp.json(), status_code=resp.status_code)
+        except Exception:
+            # llama-server 还在启动，返回基本信息
+            return JSONResponse({
+                "object": "list",
+                "data": [{
+                    "id": "MiniCPM-o-4_5",
+                    "object": "model",
+                    "created": int(time.time()),
+                    "owned_by": "prego-pal",
+                }],
+            })
 
     @web_app.get("/")
     async def root():
         return {
-            "service": "PregoPal MiniCPM-o-4_5 API",
-            "version": "2.0.0",
+            "service": "PregoPal MiniCPM-o-4_5 API (llama-server)",
+            "version": "3.0.0",
             "model": MAIN_GGUF,
             "endpoints": {
-                "chat": "POST /v1/chat/completions",
+                "chat": "POST /v1/chat/completions (支持 streaming)",
                 "completions": "POST /v1/completions",
                 "embeddings": "POST /v1/embeddings",
-                "vision": "POST /v1/vision (多模态)",
                 "models": "GET /v1/models",
                 "health": "GET /health",
             },
@@ -292,33 +278,39 @@ def serve():
 
 
 # ════════════════════════════════════════════════════════════════════
-# 4. MODEL UPLOAD 指引
+# 5. BUILD IMAGE — 预编译 llama-server 并缓存到镜像层
 # ════════════════════════════════════════════════════════════════════
 
-@app.function(
-    image=_image,
-    volumes={MODEL_DIR: model_volume},
-    timeout=3600,
-)
-def upload_models():
-    """打印上传模型指引."""
-    print("=" * 60)
-    print("📦 上传模型至 Modal Volume:")
-    print()
-    print("  # From your local models directory:")
-    print("  modal volume put minicpm-o-4_5-models \\")
-    print("      ./models/MiniCPM-o-4_5-gguf /MiniCPM-o-4_5-gguf")
-    print()
-    print("  # Verify:")
-    print("  modal volume ls minicpm-o-4_5-models /")
-    print(f"  # Expected files:")
-    print(f"  #   {MAIN_GGUF}")
-    print(f"  #   {VISION_MMPROJ}")
-    print("=" * 60)
+@app.function(image=_image, timeout=3600)
+def build_image():
+    """
+    预编译 llama-server 并缓存到 Modal 镜像层。
+    
+    只需跑一次，之后 `modal deploy` 秒完成。
+    
+    使用方法:
+        modal run -m modal_deploy.deploy::build_image
+    """
+    import subprocess
+    print("🔨 开始编译 llama-server（CUDA）...")
+    result = subprocess.run(
+        "cd /llama.cpp && cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release && "
+        "cmake --build build --config Release --target llama-server -j$(nproc)",
+        shell=True,
+        capture_output=True,
+        text=True,
+    )
+    if result.returncode == 0:
+        print("✅ llama-server 编译成功！")
+        print("✅ 镜像已缓存，现在可以运行 `modal deploy -m modal_deploy.deploy`")
+    else:
+        print(f"❌ 编译失败: {result.stderr}")
+        print(f"stdout: {result.stdout}")
+    return result.returncode
 
 
 # ════════════════════════════════════════════════════════════════════
-# 5. TEST INFERENCE
+# 6. TEST INFERENCE (本地快速测试用)
 # ════════════════════════════════════════════════════════════════════
 
 @app.function(
@@ -328,61 +320,73 @@ def upload_models():
     timeout=600,
 )
 def test_inference():
-    """在 Modal 上测试推理."""
-    import time
-    import json
-    from llama_cpp import Llama
-
-    print("[PregoPal] ========== TEST INFERENCE ==========")
-    print(f"[PregoPal] GPU: A100 (via Modal)")
-
-    paths = get_model_paths(MODEL_SUBDIR)
-    main_path = paths["main"]
-    vision_path = paths["vision"]
-
-    if not os.path.isfile(main_path):
-        print(f"[PregoPal] ❌ Model not found at {main_path}")
+    """在 Modal 上快速测试推理."""
+    import httpx
+
+    # 先在本函数里启动一个临时 llama-server
+    LLAMA_SERVER = "/llama.cpp/build/bin/llama-server"
+    if not os.path.isfile(LLAMA_SERVER):
+        alt = "/llama.cpp/build/bin/Release/llama-server"
+        if os.path.isfile(alt):
+            LLAMA_SERVER = alt
+
+    main_path = os.path.join(MODEL_SUBDIR, MAIN_GGUF)
+    vision_path = os.path.join(MODEL_SUBDIR, VISION_MMPROJ)
+
+    port = 8081
+    cmd = [
+        LLAMA_SERVER,
+        "-m", main_path,
+        "--host", "127.0.0.1",
+        "--port", str(port),
+        "-c", "4096",
+        "-ngl", "-1",
+        "--no-mmap",
+    ]
+    if os.path.isfile(vision_path):
+        cmd.extend(["--mmproj", vision_path])
+
+    print(f"[PregoPal] Starting llama-server...")
+    process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+
+    base_url = f"http://127.0.0.1:{port}"
+    for i in range(120):
+        try:
+            r = httpx.get(f"{base_url}/health", timeout=3)
+            if r.status_code == 200:
+                print(f"[PregoPal] ✅ Ready after {i + 1}s")
+                break
+        except Exception:
+            pass
+        time.sleep(1)
+    else:
+        out = process.stdout.read(2048).decode(errors="replace")
+        print(f"[PregoPal] ❌ Failed to start:\n{out}")
         return
 
-    t0 = time.time()
-    kwargs = dict(
-        model_path=main_path,
-        n_gpu_layers=-1,
-        n_ctx=4096,
-        verbose=False,
-    )
-    if os.path.isfile(vision_path):
-        kwargs["mmproj"] = vision_path
-
-    print("[PregoPal] Loading model...")
-    llm = Llama(**kwargs)
-    load_time = time.time() - t0
-    print(f"[PregoPal] ✅ Model loaded in {load_time:.1f}s")
-
-    # Test 1
-    print("\n[Test 1] Chinese greeting...")
-    t0 = time.time()
-    result = llm.create_chat_completion(
-        messages=[{"role": "user", "content": "用中文说你好，不超过10个字"}],
-        max_tokens=30,
-        temperature=0.1,
-    )
-    elapsed = time.time() - t0
+    # Test 1: 中文
+    print("\n[Test 1] 中文提问...")
+    r = httpx.post(f"{base_url}/v1/chat/completions", json={
+        "messages": [{"role": "user", "content": "用中文说你好，不超过15个字"}],
+        "max_tokens": 30,
+        "temperature": 0.1,
+    }, timeout=60)
+    result = r.json()
     content = result.get("choices", [{}])[0].get("message", {}).get("content", "")
-    print(f"Response ({elapsed:.1f}s): {content}")
-
-    # Test 2
-    print("\n[Test 2] English instruction...")
-    t0 = time.time()
-    result = llm.create_chat_completion(
-        messages=[{"role": "user", "content": "What is the capital of France? Answer in 5 words."}],
-        max_tokens=30,
-        temperature=0.1,
-    )
-    elapsed = time.time() - t0
+    print(f"Response: {content}")
+
+    # Test 2: 英文
+    print("\n[Test 2] 英文提问...")
+    r = httpx.post(f"{base_url}/v1/chat/completions", json={
+        "messages": [{"role": "user", "content": "What is the capital of France? Answer in 5 words."}],
+        "max_tokens": 30,
+        "temperature": 0.1,
+    }, timeout=60)
+    result = r.json()
     content = result.get("choices", [{}])[0].get("message", {}).get("content", "")
-    print(f"Response ({elapsed:.1f}s): {content}")
+    print(f"Response: {content}")
 
+    process.terminate()
     print(f"\n{'='*50}")
-    print(f"✅ Test complete! Loading: {load_time:.1f}s")
+    print(f"✅ Test complete!")
     print(f"{'='*50}")

modal_deploy/diagnose_volume.py

@@ -0,0 +1,61 @@
+"""
+Diagnose: check model files in Modal Volume.
+Run: modal run modal_deploy.diagnose_volume
+"""
+import os
+import sys
+
+MODEL_DIR = "/models"
+MODEL_SUBDIR = f"{MODEL_DIR}/MiniCPM-o-4_5-gguf"
+MAIN_GGUF = "MiniCPM-o-4_5-Q4_K_M.gguf"
+
+def main():
+    print(f"{'='*60}")
+    print(f"MODEL_DIR: {MODEL_DIR}")
+    print(f"MODEL_SUBDIR: {MODEL_SUBDIR}")
+    print(f"{'='*60}")
+    
+    # List all files
+    print(f"\n📂 Contents of {MODEL_SUBDIR}:")
+    for root, dirs, files in os.walk(MODEL_SUBDIR):
+        level = root.replace(MODEL_SUBDIR, '').count(os.sep)
+        indent = ' ' * 2 * level
+        print(f"{indent}{os.path.basename(root)}/")
+        subindent = ' ' * 2 * (level + 1)
+        for file in files:
+            fpath = os.path.join(root, file)
+            size = os.path.getsize(fpath)
+            print(f"{subindent}{file}  ({size:,} bytes = {size/1024**3:.2f} GB)")
+    
+    # Check main model
+    main_path = os.path.join(MODEL_SUBDIR, MAIN_GGUF)
+    if os.path.isfile(main_path):
+        size = os.path.getsize(main_path)
+        mb = size / (1024 * 1024)
+        print(f"\n✅ {MAIN_GGUF}: {size:,} bytes ({mb:.0f} MB)")
+        if mb < 100:
+            print(f"⚠️  WARNING: File seems too small for a GGUF model!")
+    else:
+        print(f"\n❌ {MAIN_GGUF} NOT FOUND at {main_path}")
+    
+    # Check vision mmproj
+    vision_path = os.path.join(MODEL_SUBDIR, "vision", "MiniCPM-o-4_5-vision-F16.gguf")
+    if os.path.isfile(vision_path):
+        size = os.path.getsize(vision_path)
+        mb = size / (1024 * 1024)
+        print(f"\n✅ Vision mmproj: {size:,} bytes ({mb:.0f} MB)")
+    else:
+        print(f"\n❌ Vision mmproj NOT FOUND at {vision_path}")
+    
+    # Check first 8 bytes of main model (GGUF magic)
+    if os.path.isfile(main_path):
+        with open(main_path, "rb") as f:
+            magic = f.read(8)
+        print(f"\n🔍 First 8 bytes (magic): {magic.hex()}")
+        if magic[:4] == b"GGUF":
+            print(f"✅ Valid GGUF header detected")
+        else:
+            print(f"❌ Not a valid GGUF file!")
+
+if __name__ == "__main__":
+    main()

tmp_build_llamacpp_guide.md

@@ -0,0 +1,51 @@
+# 编译 llama.cpp 指南
+
+`llama-mtmd-cli.exe` 不存在，需要克隆并编译 llama.cpp。
+
+## Step 1: 克隆 llama.cpp（如果还没克隆）
+
+```cmd
+cd C:\Users\Andre\codes\LJB\hackthon\llamacpp
+```
+
+```cmd
+git clone https://github.com/ggml-org/llama.cpp.git
+```
+
+如果已经克隆过了，更新到最新：
+```cmd
+cd C:\Users\Andre\codes\LJB\hackthon\llamacpp\llama.cpp
+git pull
+```
+
+## Step 2: 编译（CUDA 版本）
+
+```cmd
+cd C:\Users\Andre\codes\LJB\hackthon\llamacpp\llama.cpp
+cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
+cmake --build build --config Release -j
+```
+
+> 注意：编译可能需要 5-15 分钟。
+
+## Step 3: 验证编译成功
+
+```cmd
+dir C:\Users\Andre\codes\LJB\hackthon\llamacpp\llama.cpp\build\bin\Release\llama-mtmd-cli.exe
+```
+
+---
+
+## 验证：本地推理测试
+
+编译成功后，找一张测试图片，放在 `C:\Users\Andre\codes\LJB\hackthon\llamacpp\test.jpg`，运行：
+
+```cmd
+cd C:\Users\Andre\codes\LJB\hackthon\llamacpp\llama.cpp\build\bin\Release
+```
+
+```cmd
+llama-mtmd-cli -m C:\Users\Andre\codes\LJB\hackthon\llamacpp\models\MiniCPM-o-4_5-gguf\MiniCPM-o-4_5-Q4_K_M.gguf --mmproj C:\Users\Andre\codes\LJB\hackthon\llamacpp\models\MiniCPM-o-4_5-gguf\vision\MiniCPM-o-4_5-vision-F16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 -p "用中文描述这张图片里的内容" -i
+```
+
+（如果没有图片，可以先不加 `--image` 参数，纯文本测试也可以）

tmp_deploy_success_backup.md

@@ -0,0 +1,17 @@
+# Modal 部署成功后的备份清单
+
+如果 `modal deploy -m modal_deploy.deploy` 成功了，请执行以下备份步骤：
+
+## 1. 下载 Modal 上的文件
+
+```cmd
+cd C:\Users\Andre\codes\LJB\hackthon\llamacpp\PregoPal
+
+:: 下载 deploy.py
+modal app logs prego-pal-minicpm serve > backups/deploy_log.txt
+
+:: 拉取 Modal 容器配置到本地
+modal app list
+
+:: 查看部署详情
+modal app get prego-pal-minicpm

tmp_deploy_to_modal_guide.md

@@ -0,0 +1,66 @@
+# 重新部署到 Modal 指南（v2 — llama-server 方案）
+
+deploy.py 已完全重写。新方案：
+
+**FastAPI (ASGI) ←→ httpx 代理 ←→ llama-server (subprocess)**
+                                   ↕
+                            Modal Volume: GGUF 模型
+
+## 新方案架构
+
+```
+Modal 容器启动时：
+  1. 编译 llama-server (cmake --build ... --target llama-server)
+  2. FastAPI 入口 → 启动 llama-server 子进程 (localhost:8080)
+  3. FastAPI 透明代理所有 /v1/* 请求到 llama-server
+```
+
+**为什么用 llama-server 而不是 llama-cpp-python：**
+- llama-server 原生支持 `mtmd` 多模态架构（MiniCPM-o 4.5 必须）
+- 自带 OpenAI 兼容 API（`/v1/chat/completions`）
+- 比 Python 绑定更稳定
+
+## 部署步骤
+
+### Step 1: 部署
+
+```cmd
+cd C:\Users\Andre\codes\LJB\hackthon\llamacpp\PregoPal
+```
+
+```cmd
+SET PYTHONIOENCODING=utf-8
+
+modal deploy -m modal_deploy.deploy
+```
+
+> ⚠️ 注意：必须用 `-m` 标志，不能省略
+
+### Step 2: 测试
+
+部署成功后，会得到一个 URL，如：
+```
+https://andrew-jiabin--prego-pal-minicpm-serve.modal.run
+```
+
+测试健康检查：
+```cmd
+curl https://andrew-jiabin--prego-pal-minicpm-serve.modal.run/health
+```
+
+测试中文推理：
+```cmd
+curl -X POST https://andrew-jiabin--prego-pal-minicpm-serve.modal.run/v1/chat/completions ^
+  -H "Content-Type: application/json" ^
+  -d "{\"messages\":[{\"role\":\"user\",\"content\":\"用中文简单介绍一下你自己\"}],\"max_tokens\":50}"
+```
+
+### ⚠️ 注意事项
+
+1. **首次部署需要编译 llama-server**（约 2-3 分钟），后续部署使用缓存
+2. 首次启动需要加载 ~5GB 模型（约 30-60 秒）
+3. **`min_containers=0`** — 不用时自动缩到 0，不扣费
+
+### 如果部署失败
+
+Modal 会输出编译日志和启动日志，把报错信息发给我即可。

tmp_download_instructions.md

@@ -0,0 +1,33 @@
+# 下载完整 MiniCPM-o-4_5 GGUF 主模型
+
+**问题：** 当前 `MiniCPM-o-4_5-Q4_K_M.gguf` 仅 721 MB，正常需要 ~5 GB
+
+## Step 1: 删除损坏文件
+
+```cmd
+del C:\Users\Andre\codes\LJB\hackthon\llamacpp\models\MiniCPM-o-4_5-gguf\MiniCPM-o-4_5-Q4_K_M.gguf
+```
+
+## Step 2: 用 `hf` 下载完整主模型（~5 GB）
+
+注意：`huggingface-cli` 已废弃，改用 `hf`。
+
+```cmd
+hf download openbmb/MiniCPM-o-4_5-GGUF --include MiniCPM-o-4_5-Q4_K_M.gguf --local-dir C:\Users\Andre\codes\LJB\hackthon\llamacpp\models\MiniCPM-o-4_5-gguf
+```
+
+如果下载中断需续传：
+```cmd
+hf download openbmb/MiniCPM-o-4_5-GGUF --include MiniCPM-o-4_5-Q4_K_M.gguf --local-dir C:\Users\Andre\codes\LJB\hackthon\llamacpp\models\MiniCPM-o-4_5-gguf --resume
+```
+
+## Step 3: 验证文件大小
+
+```cmd
+dir C:\Users\Andre\codes\LJB\hackthon\llamacpp\models\MiniCPM-o-4_5-gguf\MiniCPM-o-4_5-Q4_K_M.gguf
+```
+
+预期：**~4,500,000,000 - 5,500,000,000 bytes（约 4.5-5.5 GB）**
+当前：721,420,288 bytes（约 0.7 GB）
+
+下载完成后通知我，一起进行下一步。

tmp_llama_mtmd_test_guide.md

@@ -0,0 +1,30 @@
+# llama-mtmd-cli 本地推理测试
+
+编译成功！`-i` 不是有效参数。以下是正确命令。
+
+## 纯文本测试（无图片）
+
+```cmd
+cd C:\Users\Andre\codes\LJB\hackthon\llamacpp\llama.cpp\build\bin\Release
+```
+
+```cmd
+llama-mtmd-cli -m C:\Users\Andre\codes\LJB\hackthon\llamacpp\models\MiniCPM-o-4_5-gguf\MiniCPM-o-4_5-Q4_K_M.gguf --mmproj C:\Users\Andre\codes\LJB\hackthon\llamacpp\models\MiniCPM-o-4_5-gguf\vision\MiniCPM-o-4_5-vision-F16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 -p "用中文简单介绍一下你自己"
+```
+
+## 带图片测试
+
+找一张图片（比如 `C:\Users\Andre\Pictures\test.jpg`），运行：
+
+```cmd
+llama-mtmd-cli -m C:\Users\Andre\codes\LJB\hackthon\llamacpp\models\MiniCPM-o-4_5-gguf\MiniCPM-o-4_5-Q4_K_M.gguf --mmproj C:\Users\Andre\codes\LJB\hackthon\llamacpp\models\MiniCPM-o-4_5-gguf\vision\MiniCPM-o-4_5-vision-F16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 --image C:\Users\Andre\Pictures\test.jpg -p "用中文描述这张图片里的内容"
+```
+
+## 查看所有可用参数
+
+```cmd
+llama-mtmd-cli --help
+```
+
+> 注意：`-i` 不是有效参数。图片用 `--image` 指定，文本提示用 `-p` 指定。
+> 首次加载模型可能需要 30-60 秒。

tmp_official_deploy_guide.md

@@ -0,0 +1,35 @@
+# 基于官方教程的部署方案
+
+来源：MiniCPM-V-Cookbook/deployment/llama.cpp/minicpm-o4_5_llamacpp_zh.md
+
+## 关键发现
+
+1. **MiniCPM-o-4.5 必须用 `llama-mtmd-cli`（多模态解码器）** 运行，不是普通 `llama-cli`
+2. 正确命令行格式：
+   ```bash
+   ./llama-mtmd-cli \
+     -m MiniCPM-o-4_5-Q4_K_M.gguf \
+     --mmproj MiniCPM-o-4_5-vision-F16.gguf \
+     -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 \
+     --repeat-penalty 1.05 \
+     --image xx.jpg \
+     -p "What is in the image?"
+   ```
+3. 需要克隆 `https://github.com/ggml-org/llama.cpp.git`（主仓库，不是 llama.cpp-omni）
+4. llama.cpp 编译时需启用 CMake，CUDA 模式加 `-DGGML_CUDA=ON`
+
+## 对比之前 deploy.py 的问题
+
+| 项目 | 之前方案 | 官方方案 |
+|------|---------|---------|
+| 执行程序 | `llama_cpp.Llama()` Python 绑定 | `llama-mtmd-cli` / `llama-server` |
+| mmproj 传参 | bug: 递归传全部 .gguf | `--mmproj vision-F16.gguf`（单一文件） |
+| 推荐的 API 模式 | OpenAI SDK 包装 | `llama-server` 原生 OpenAI 兼容 API |
+
+## 部署步骤（待下载完成）
+
+1. 下载完整 `MiniCPM-o-4_5-Q4_K_M.gguf`（先删除损坏 721MB 版）
+2. 克隆/更新 `ggml-org/llama.cpp`
+3. 编译 `llama-server`（含 `llama-mtmd` 支持）
+4. 本地运行 `llama-mtmd-cli` 验证推理
+5. 修改 Modal 方案：用 `llama-server` 方式部署

tmp_upload_and_test_guide.md

@@ -0,0 +1,44 @@
+# 下一步操作指南
+
+## 📤 1. 上传模型到 Modal Volume
+
+在**新开一个终端**（不要关当前的），运行：
+
+```cmd
+modal volume put minicpm-o-4_5-models C:\Users\Andre\codes\LJB\hackthon\llamacpp\models\MiniCPM-o-4_5-gguf /MiniCPM-o-4_5-gguf
+```
+
+上传完成后验证：
+
+```cmd
+modal volume ls minicpm-o-4_5-models /MiniCPM-o-4_5-gguf
+```
+
+预期输出：
+```
+MiniCPM-o-4_5-Q4_K_M.gguf
+vision/
+vision/MiniCPM-o-4_5-vision-F16.gguf
+```
+
+---
+
+## 🖥️ 2. 本地检查（在当前终端执行）
+
+### 2a. 检查 vision 文件是否存在
+
+```cmd
+dir C:\Users\Andre\codes\LJB\hackthon\llamacpp\models\MiniCPM-o-4_5-gguf\vision
+```
+
+### 2b. 检查 llama.cpp 是否已编译
+
+```cmd
+dir C:\Users\Andre\codes\LJB\hackthon\llamacpp\llama.cpp\build\bin\llama-mtmd-cli.exe
+```
+
+---
+
+## 告诉我结果
+
+把上面 2a 和 2b 的运行结果发给我，我来判断下一步怎么做。

tmp_volume_cleanup_guide.md

@@ -0,0 +1,19 @@
+# Modal Volume 清理指南
+
+问题：Volume 里有旧的损坏文件，需要先清空再上传。
+
+## 方案一：删除旧文件 + 覆盖上传
+
+先删掉 Volume 里的旧内容，再上传：
+
+```cmd
+modal volume rm minicpm-o-4_5-models /MiniCPM-o-4_5-gguf -r
+```
+
+然后再上传：
+
+```cmd
+modal volume put minicpm-o-4_5-models C:\Users\Andre\codes\LJB\hackthon\llamacpp\models\MiniCPM-o-4_5-gguf /MiniCPM-o-4_5-gguf
+```
+
+运行顺序：先第一行命令，再第二行。