Spaces:
Runtime error
Runtime error
Commit History
-
fix: remove --voxcpm2-base-lm/acoustic args (not valid for llama-server CLI)
J.B-Linllama-server does not have voxcpm2 CLI args. TTS model paths are passed via omni_init JSON body (model_dir/tts_bin_dir).6230c98 -
test_omni: add stderr capture on startup failure
J.B-Line770d07 -
fix: rewrite test_omni to use llama-server built-in omni endpoints
J.B-LinRemoved llama-omni-server dependency from test. test_omni now launches only llama-server (with --voxcpm2-base-lm etc.) and tests /v1/stream/omni_init directly on the built-in endpoint.d67ae6b -
refactor: deprecate llama-omni-server, use llama-server built-in omni endpoints
J.B-LinKey findings: - llama-server (server.cpp) natively provides /v1/stream/* omni endpoints - It reuses already-loaded LLM model+ctx, no need for separate omni server - llama-omni-server was an experimental standalone binary Changes: - Image recipe: build llama-server only (remove llama-omni-server target) - serve() lifespan: single llama-server process only (remove omni subprocess) - Omni endpoint proxy: point to llama-server port 8081 (remove OMNI_SERVER_PORT) - Clean up tmp_*.py test files Status: Image im-kINUqibAmbYIS6brxSIOIY deployed (4737s build)6451051 -
fix: set n_parallel=1, n_ctx=4096 before omni_init to prevent n_seq_max > 256
J.B-Line0c2f3a -
test_omni: add stderr capture for omni-server when init fails
J.B-Lin4429886 -
fix: params.model is common_params_model struct, use .path
J.B-Lina94fb90 -
tmp: health check / omni test scripts for debugging omni_init 500
J.B-Linc86f455 -
fix: set params.model/vpm/apm/tts in server-omni.cpp omni_init handler so model files are found
J.B-Lina850bb4
-
modal/omni: add llama-omni-server for full-duplex voice API
J.B-LinArchitecture: FastAPI (ASGI) → llama-server:8081 (chat/TTS/STT) → llama-omni-server:8082 (omni full-duplex endpoints) New endpoints: POST /v1/omni/init - Initialize omni context (delegates to /v1/stream/omni_init) POST /v1/omni/prefill - Send audio/image (base64 decoded to temp WAV file) POST /v1/omni/generate - SSE streaming generation (proxies /v1/stream/decode) POST /v1/omni/break - Reinit omni context (discard current generation) POST /v1/omni/stop - Stop omni session GET /v1/omni/tts_wav/{dir}/{file} - Serve TTS WAV files Build: added --target llama-omni-server to cmake build Lifespan: now manages two subprocesses (llama-server + llama-omni-server) No existing endpoints modifiedfedf97a -
modal/omni: text inference verified (Chinese OK, English needs fix)
J.B-Lin- Chinese reply correct: '你好，我是MiniCPM系列模型...' - English empty due to finish_reason=length (50 special tokens before valid text) - Added --jinja flag to fix chat template formatting - Improved test logging: show content, finish_reason, usage per test - Serve function already had --jinja (confirmed correct) - Updated docs with real test results2400b46 -
modal/omni: llama-server inference verified on T4
J.B-Lin- test_inference: non-blocking stderr reader (thread+queue) - test_inference: --mmproj only vision (no comma-sep, no audio) - test_inference: timeout 600s->900s - serve: --mmproj only vision (was comma-sep vision+audio) - docs: add test results (11s cold start, 1s inference on T4) - docs: fix mmproj documentation (only ONE file, audio via API)0e9e62e -
chore: add build_llama_server.sh (local cross-compile script, unused)
J.B-Lin12a8a38 -
deploy_omni: llFile llama.cpp-omni compile succeeds on Modal T4
J.B-Lin- CUDA Toolkit 12.4 + cuda-compiler + cuda-driver-dev from NVIDIA apt repo - GGML_CUDA_NO_VMM=ON to avoid libcuda.so.1 linking in build phase - CUDA arch 75 (T4) + 89 (L4) - LLAMA_CUDA_FORCE_MMQ=ON for faster compilation - Source from tc-mb/llama.cpp-omni fork, stored in modal_deploy/llamacpp_omni/ - Build time ~77 min on Modal infra Resolves: modal deploy build failures due to GitHub access and CUDA linking1a07461
-
用户手动备份
J.B-Lin0d9e374 -
用户手动整理文件，并备份deploy_backup.py文件
J.B-Lin68d23d2 -
core: 更新 ModelLoader 对接远端 Modal API + README 同步
J.B-Lin8193cd7 -
modal: 降级 GPU A100 -> T4 节省成本
J.B-Lin2d207a7 -
deploy: 成功部署 MiniCPM-o-4_5 到 Modal (预编译llama-cpp-python A100)
J.B-Linc0f7bc8
-
docs: pre-refactor backup - before tech log integration
J.B-Lind14aeea -
fix(deploy): 使用预编译CUDA wheel替代源码编译
J.B-Linaa40303 -
重构modal部署使用CUDA编译, streaming修复, cookbook_ref gitignored
J.B-Linfb89443 -
[modal] Fix: mmproj single-file, health endpoint, client API consistency
J.B-Linf10673a -
fix(deploy.py): 修复4个问题 - MODEL_DIR shadowing, find_mmproj_files硬编码, MainModel传参, 移除--multimodal flag
J.B-Linc8e1c13 -
fix: CUDA runtime LD_LIBRARY_PATH + CUDA_HOME in Modal deploy; add inference test script
J.B-Linea75111 -
feat(modal): 部署 MiniCPM-o 4.5 到 Modal (llama.cpp + FastAPI)
J.B-Lin09159ca -
feat: llama.cpp deploys MiniCPM-o 4.5 on Modal - technical report and client API
J.B-Linff5d4ed