diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000000000000000000000000000000000000..1d309b6f0bcec33f0df1fd9d7a0e9dc6db7421e1 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,40 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text +resource/fonts/**/*.otf filter=lfs diff=lfs merge=lfs -text +resource/media/**/*.mp4 filter=lfs diff=lfs merge=lfs -text +resource/bgms/**/*.mp3 filter=lfs diff=lfs merge=lfs -text +data/**/*.csv filter=lfs diff=lfs merge=lfs -text +resource/fonts/SourceHanSansSC/*.otf filter=lfs diff=lfs merge=lfs -text diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..190a5de9dd1aa021df06c0dac2c0a0868b31a5b9 --- /dev/null +++ b/.gitignore @@ -0,0 +1,75 @@ +# === Python 生成文件 === +__pycache__/ +*.py[cod] +*$py.class + +# C 扩展 +*.so +*.pyd + +# 虚拟环境 / Conda 环境目录 +.venv/ +venv/ +env/ +.env/ +.conda/ +.hypothesis/ + +# 构建 / 发布产物 +build/ +dist/ +*.egg-info/ +*.egg +pip-wheel-metadata/ + +# 单元测试 / 覆盖率 / 缓存 +.pytest_cache/ +.coverage +.coverage.* +htmlcov/ +.tox/ +.nox/ +.mypy_cache/ +.dmypy.json +.pyre/ +.cache/ + +# IDE / 编辑器配置 +.vscode/ +.idea/ +*.swp +*.swo +*.iml + +# Jupyter +.ipynb_checkpoints/ + +# OS 级别垃圾文件 +.DS_Store +Thumbs.db + +# 日志 / 临时文件 +*.log +logs/ +tmp/ +temp/ +.server_cache/ +.storyline/.server_cache/ + +# 本项目可能产生的大文件目录 +outputs/ +renders/ +checkpoints/ +models/ +project/ + +# 环境/配置的敏感信息（你如果用 .env 管 secret） +.env.local +.env.*.local + +data/** +!data/elements_v2/ +!data/elements_v2/** +!data/prompts/ +!data/prompts/** +resource/** \ No newline at end of file diff --git a/.storyline/skills/create_profile_style_skill/SKILL.md b/.storyline/skills/create_profile_style_skill/SKILL.md new file mode 100644 index 0000000000000000000000000000000000000000..1e7f6874db5c8dd75721bb7a03f73dafaab50c2e --- /dev/null +++ b/.storyline/skills/create_profile_style_skill/SKILL.md @@ -0,0 +1,63 @@ +--- +name: create_profile_style_skill +description: 【SKILL】分析当前剪辑逻辑与风格，总结并生成一个新的可复用 Skill 文件，存入剪辑技能库。 +version: 1.0.0 +author: User_Agent_Architect +tags: [meta-skill, workflow, writing, file-system] +--- + +# 角色定义 (Role) +你是一个专业的“剪辑风格架构师”。你具备深厚的影视视听语言知识，能够从具体的剪辑操作（如切点选择、转场习惯、BGM卡点逻辑）中提炼出抽象的“剪辑哲学”和“SOP（标准作业程序）”。 + +# 任务目标 (Objective) +你的任务是观察或询问用户的剪辑偏好，将其转化为一个标准的 Agent Skill 文档（Markdown格式），并保存到 `.storyline/skills/` 目录下，以便让 Agent 在未来模仿这种风格。 + +# 执行流程 (Workflow) + +## 第一步：风格分析与萃取 (Analysis & Extraction) +1. **获取上下文**：获取当前正在编辑的 Timeline 数据，或者请求用户描述其剪辑习惯。 +2. **维度拆解**：你需要从以下维度总结风格： + * **剪辑节奏 (Pacing)**：是快节奏的跳剪（Jump Cut），还是长镜头的舒缓叙事？ + * **叙事逻辑 (Storytelling)**：是线性叙事、倒叙，还是基于音乐情绪的蒙太奇？ + * **视听语言 (Audio-Visual)**：音效（SFX）的使用密度、字幕样式偏好、调色风格（LUTs）。 + * **特殊偏好**：例如“总是删除静音片段”或“每5秒插入一个B-Roll”。 + +## 第二步：交互与命名 (Interaction & Naming) +1. **总结确认**：向用户展示你总结的 3-5 个核心风格点，询问是否准确。 +2. **命名建议**：根据风格特点，建议 2 个文件名（例如 `fast_paced_vlog` 或 `cinematic_travel`），命名必须是英文单词和下划线组成，不能出现中文命名。 +3. **获取输入**： + * 询问用户：“是否认可这个总结？” + * 询问用户：“你想将这个新技能命名为什么？（按 Enter 使用建议名称：[建议名称]）” + +## 第三步：生成新 Skill 内容 (Drafting) +根据确认的风格，生成新 Skill 的 Markdown 内容。内容必须包含标准头部和 Prompt 指令。 +* *Template*（新 Skill 的模板结构）： + ```markdown + --- + name: {用户定义的名称} + description: 【SKILL】基于 {日期} 总结的 {风格关键词} 剪辑风格 + version: 基于对话进行版本管理 + author: 用户 + tags: [相关的tag-list] + --- + # 剪辑指令 + 当执行剪辑任务时，请严格遵守以下逻辑： + 1. **整体风格原则**：{分析出的节奏逻辑} + 2. **音频处理规范**：{分析出的音频处理（视频原声/配音/背景音乐）筛选逻辑} + 3. **视觉元素规范**：{分析出的视觉元素（字体花字/转场/滤镜/特效等）使用逻辑} + 4. **剪辑节奏控制**：{分析出的剪辑节奏（音乐卡点/短切片/长切片）使用逻辑} + 5. **工具调用规范**：{分析出的推荐使用的工具以及推荐的传入参数} + ``` + +## 第四步：入库与更新 (Commit & Update) +1. **展示预览**：将生成的内容以代码块形式展示给用户。 +2. **执行写入**： + * 用户确认后，调用文件写入工具`write_skills`。 + * **目标路径**：`.storyline/skills/{文件名}/SKILL.md`，传入文件名即可，工具会自动完成写入。 +3. **系统更新**：提示用户“新技能已入库，请刷新 Agent 工具列表以加载。” + +# 约束条件 (Constraints) +* **格式规范**：生成的新 Skill 必须符合 markdown 标准，且包含元数据（Metadata）。 +* **路径安全**：只能写入 `.storyline/skills/` 目录，禁止覆盖系统核心文件。 +* **可读性**：在与用户交互时，不要直接扔出一大段代码，先用自然语言确认逻辑。 +* **版本管理**：当用户进行修改时，更改版本号，并重新调用`write_skills`工具做覆盖； diff --git a/.storyline/skills/subtitle_imitation_skill/SKILL.md b/.storyline/skills/subtitle_imitation_skill/SKILL.md new file mode 100644 index 0000000000000000000000000000000000000000..61db06038404f6891e6888a3616208d7a021a4e8 --- /dev/null +++ b/.storyline/skills/subtitle_imitation_skill/SKILL.md @@ -0,0 +1,55 @@ +--- +name: subtitle_imitation_skill +description: 【SKILL】基于用户提供的参考文案样本，对视频素材内容进行深度文风仿写，生成风格化脚本。 +version: 1.0.0 +author: User_Agent_Architect +tags: [writing, style-transfer, video-production, creative] +--- + +# 角色定义 (Role) +你是一位“文风迁移大师”兼“金牌视频脚本撰写人”。你不仅拥有敏锐的文学感知力，能精准捕捉文字背后的韵律、修辞和情感基调（如“鲁迅体”、“王家卫风”、“发疯文学”），同时深谙视听语言，能够将画面内容转化为极具感染力的旁白或台词，而非机械地描述画面。 + +# 任务目标 (Objective) +你的核心任务是接收用户的“仿写指令”和“参考文案”，调用历史记忆读取视频素材理解结果（`understand_clips`）以及读取分组结果（`group_clips`），生成一份既具备参考文案神韵，又严格基于视频事实的拍摄脚本。 + +# 执行流程 (Workflow) + +## 第一步：输入校验与意图确认 (Input Validation) +1. **检查输入参数**：检查用户是否提供了用于模仿的 `style_reference_text`（仿写样本）。 +2. **缺失处理**： + * **如果用户未提供样本**（仅说“帮我仿写一下”）：请先调用`script_template_rec`工具用来检索可模仿的文风模板，如果检索结果没有合适的模板，必须立即中止后续流程，并输出回复引导用户：“为了能精准模仿您想要的文风，请提供一段您希望我模仿的文案示例（例如直接粘贴一段文字，或提供某位博主的典型语录）。” + * **如果用户已提供样本**：进入第二步。 + +## 第二步：获取素材与分析 (Context & Analysis) +1. **读取视频理解**：调用工具 `read_node_history`，参数为 `key="understand_clips"`，获取当前视频素材的画面描述、氛围和关键动作。 +2. **风格解构**：在思维链（Chain of Thought）中快速分析用户提供的 `style_reference_text`： + * **句式特征**：是短句堆叠，还是长难句？ + * **修辞习惯**：是否喜欢用比喻、反讽、排比？ + * **情感基调**：是治愈、焦虑、犀利还是幽默？ + +## 第三步：风格化创作 (Creative Generation) +基于素材内容（Content）和分析出的风格（Style），执行脚本撰写。需严格遵守以下创作原则： +1. **拒绝“看图说话” (No See-Say)**： + * ❌ 错误示范：“画面里有一只猫在睡觉，阳光照在它身上。” + * ✅ 正确示范（如文艺风）：“午后的阳光是免费的，但偷得浮生半日闲的勇气却是昂贵的。它在做梦，而我在看它。” +2. **内容强关联**：生成的文案必须基于 `understand_clips` 中的真实画面，不能脱离素材天马行空，也不能仅模仿风格却写了无关内容。 +3. **生动连贯**：脚本必须有起承转合，不仅是句子的拼凑，更是一个完整的小故事或情绪流。 + +## 第四步：格式化输出 (Formatting) +1. **构建数据结构**：将生成的脚本整理为符合工具 `generate_script` 输入要求的格式，并传入到`generate_script`中的`custom_script`中。格式如下： +```json +{ + "group_scripts": [ + { "group_id": "group_0001", "raw_text": "第一句，第二句，第三句" }, + { "group_id": "group_0002", "raw_text": "第一句，第二句" } + ], + "title": "视频标题" +} +``` +2. **输出总结**: 对用户隐藏结构化文案，而是挑选里面的句子反馈给用户，让用户判断是否符合要求，以便做进一步修改。 + +# 约束条件 (Constraints) +* **素材依赖**：必须调用 `read_node_history` 获取素材，严禁在不知道视频内容的情况下瞎编脚本。 +* **风格一致性**：生成的文案必须让熟悉该风格的人一眼就能识别出“味道”。 +* **拒绝机械描述**：严禁出现“视频显示”、“镜头切到”等说明书式语言，除非参考风格本身就是说明书风格。 +* **工具对接**：输出内容必须适配 `generate_script` 的字段定义，确保下游渲染环节无缝衔接。 \ No newline at end of file diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000000000000000000000000000000000000..ae28210c22d5a711d25b870cdb22e6719cf989db --- /dev/null +++ b/Dockerfile @@ -0,0 +1,31 @@ +# 基础镜像 +FROM python:3.11-slim + +# 设置工作目录 +WORKDIR /app + +# 复制文件 +COPY requirements.txt . +COPY run.sh . +COPY src/ ./src/ +COPY agent_fastapi.py . +COPY cli.py . +COPY config.toml . +COPY web/ ./web/ +COPY prompts/ ./prompts/ +COPY .storyline/ ./.storyline/ +COPY download.sh . + +# 安装依赖 +RUN apt-get update && apt-get install -y ffmpeg wget unzip git git-lfs curl +RUN pip install --no-cache-dir -r requirements.txt + +# 下载 +RUN chmod +x download.sh +RUN ./download.sh + +# 暴露 HF Space 默认端口 +EXPOSE 7860 + +# 启动命令 +CMD ["bash", "run.sh"] \ No newline at end of file diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..1f941e9ef0150cbfb3f04cb4eb4954e3c803acf4 --- /dev/null +++ b/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2026 FireRed-OpenStoryline Authors. All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..5ec216bb1a2f16eac4c6991f43cf69600f520fad --- /dev/null +++ b/README.md @@ -0,0 +1,9 @@ +--- +title: FireRed-OpenStoryline +emoji: 🎬 +colorFrom: red +colorTo: gray +sdk: docker +pinned: false +--- +Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference \ No newline at end of file diff --git a/README_zh.md b/README_zh.md new file mode 100644 index 0000000000000000000000000000000000000000..50c341397eca20832b4eebfd983b7bdb9b964eda --- /dev/null +++ b/README_zh.md @@ -0,0 +1,279 @@ +
+ + openstoryline + + + + openstoryline + + +

+ 🇨🇳 简体中文 | + 🌏 English +

+

+ + Hugging Face + + Python + License + xiaohongshu +

+
+ +
+ +[🤗 HuggingFace Demo](https://fireredteam-firered-openstoryline.hf.space/) • [🌐 Homepage](https://fireredteam.github.io/demos/firered_openstoryline/) + +
+ +
+ +
+ + +**FireRed-OpenStoryline** 将复杂的视频创作转化为自然直观的对话体验。兼顾易用性和企业级可靠性，让视频创作对初学者和创意爱好者都变得简单友好。 +> FireRed，字面意思红色的火苗，取自“星星之火，可以燎原”。我们将这团火苗取名为 FireRed，就是希望将我们在真实场景中打磨出的 SOTA 能力，像火种一样撒向旷野，点燃全球开发者的想象力，共同改变这个 AI 的世界。 + +## ✨ 核心特性 +- 🌐 **智能素材搜索与整理**： 自动在线搜索并下载符合你需求的图片和视频片段。基于用户主题素材进行片段拆分与内容理解 +- ✍️ **智能文案生成**： 结合用户主题、画面理解与情绪识别，自动构建故事线及契合的旁白。内置少样本（Few-shot）仿写能力，支持通过输入参考文本（如种草测评、日常碎碎念等）定义文案风格，实现语感、节奏与句式的精准复刻。 +- 🎵 **智能推荐音乐、配音与字体**：支持导入私有歌单，根据视频内容和情绪自动推荐背景音乐并智能卡点。只需描述"克制一点","偏情绪化","像纪录片旁白"等风格，系统即可匹配合适的配音与字体，保证整体风格协调统一。 +- 💬 **对话式精修**：支持快速删减、替换或重组片段；修改任意字幕文案；调整文字颜色、字体、描边、位置等视觉元素——所有操作均通过自然语言完成，即改即得。 +- ⚡ **剪辑技能沉淀**： 可一键保存为专属剪辑Skill，记录完整的剪辑逻辑。下次只需更换素材并选择对应Skill，即可快速复刻同款风格，实现高效批量生产。 + +## 🏗️ 架构 + +

+ openstoryline 架构 +

+ +## ✨ 演示案例 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
种草视频幽默有趣好物分享文艺风格
开箱视频宠物说话旅行Vlog年终总结
+ +> +> 🎨 效果说明：受限于开源素材的版权协议，第一行默认演示中的元素（字体/音乐）仅为基础效果。强烈建议接入自建元素库教程，解锁商用级字体、音乐、特效等，可实现显著优于默认效果的视频质量。
+> ⚠️ 画质注：受限于README展示空间，演示视频经过极限压缩。实际运行默认保持原分辨率输出，支持自定义尺寸。
+> Demo中：第一行为默认开源素材效果（受限模式），第二行为小红书App「AI剪辑」元素库效果。👉 点击查看体验教程
+> ⚖️ 免责声明：演示中包含的用户自摄素材及品牌标识仅作技术能力展示，版权归原作者所有，严禁二次分发。如有侵权请联系删除。 +> + + + +## 📦 安装 + +### 1. 克隆仓库 +```bash +# 如果没有安装git，参考官方网站进行安装：https://git-scm.com/install/ +# 或手动打包下载，并解压 +git clone https://github.com/FireRedTeam/FireRed-OpenStoryline.git +cd FireRed-OpenStoryline +``` + +### 2. 创建虚拟环境 + +按照官方指南安装 Conda（推荐Miniforge，安装过程中建议勾选上自动配置环境变量）：https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html + +``` +# 要求python>=3.11 +conda create -n storyline python=3.11 +conda activate storyline +``` + +### 3. 资源下载与依赖安装 +#### 3.1 一键安装（仅支持Linux和MacOS） +``` +sh build_env.sh +``` + +#### 3.2 手动安装 +##### A. MacOS 或 Linux + - Step 1: 安装 wget（如果尚未安装） + + ``` + # MacOS: 如果你还没有安装 Homebrew，请先安装：https://brew.sh/ + brew install wget + + # Ubuntu/Debian + sudo apt-get install wget + + # CentOS + sudo yum install wget + ``` + + - Step 2: 下载资源 + + ```bash + sh download.sh + ``` + + - Step 3: 安装依赖 + + ```bash + pip install -r requirements.txt + ``` + +###### B. Windows + - Step 1: 准备目录：在项目根目录下新建目录 `.storyline`。 + + - Step 2: 下载并解压： + + * [下载模型 (models.zip)](https://image-url-2-feature-1251524319.cos.ap-shanghai.myqcloud.com/openstoryline/models.zip) -> 解压至 `.storyline` 目录。 + + * [下载资源 (resource.zip)](https://image-url-2-feature-1251524319.cos.ap-shanghai.myqcloud.com/openstoryline/resource.zip) -> 解压至 `resource` 目录。 + - Step 3: **安装依赖**： + ```bash + pip install -r requirements.txt + ``` + + +## 🚀 快速开始 +注意：在开始之前，您需要先在 config.toml 中配置 API-Key。详细信息请参阅文档 [API-Key 配置](docs/source/zh/api-key.md) + +### 1. 启动 MCP 服务器 + +#### MacOS or Linux + ```bash + PYTHONPATH=src python -m open_storyline.mcp.server + ``` + +#### Windows + ``` + $env:PYTHONPATH="src"; python -m open_storyline.mcp.server + ``` + + +### 2. 启动对话界面 + +- 方式 1：命令行界面 + + ```bash + python cli.py + ``` + +- 方式 2：Web 界面 + + ```bash + uvicorn agent_fastapi:app --host 127.0.0.1 --port 7860 + ``` + +## 🐳 Docker 部署 + +如果未安装 Docker，请先安装 https://www.docker.com/products/docker-desktop/ + +### 拉取镜像 +``` +docker pull openstoryline/openstoryline:v1.0.0 +``` + +### 启动镜像 +``` +docker run \ + -v $(pwd)/config.toml:/app/config.toml \ + -v $(pwd)/outputs:/app/outputs \ + -p 7860:7860 \ + openstoryline/openstoryline:v1.0.0 +``` +启动后访问Web界面 http://127.0.0.1:7860 + +## 📁 项目结构 +``` +FireRed-OpenStoryline/ +├── 🎯 src/open_storyline/ 核心应用 +│ ├── mcp/ 🔌 模型上下文协议 +│ ├── nodes/ 🎬 视频处理节点 +│ ├── skills/ 🛠️ Agent 技能库 +│ ├── storage/ 💾 Agent 记忆系统 +│ ├── utils/ 🧰 工具函数 +│ ├── agent.py 🤖 Agent 构建 +│ └── config.py ⚙️ 配置管理 +├── 📚 docs/ 文档 +├── 🐳 Dockerfile Docker 配置 +├── 💬 prompts/ LLM 提示词模板 +├── 🎨 resource/ 静态资源 +│ ├── bgms/ 背景音乐库 +│ ├── fonts/ 字体文件 +│ ├── script_templates/ 视频脚本模板 +│ └── unicode_emojis.json Emoji 列表 +├── 🔧 scripts/ 工具脚本 +├── 🌐 web/ Web 界面 +├── 🚀 agent_fastapi.py FastAPI 服务器 +├── 🖥️ cli.py 命令行界面 +├── ⚙️ config.toml 主配置文件 +├── 🚀 build_env.sh 环境构建脚本 +├── 📥 download.sh 资源下载脚本 +├── 📦 requirements.txt 运行时依赖 +└── ▶️ run.sh 启动脚本 + +``` + +## 📚 文档 + +### 📖 教程索引 + +- [API申请与配置](docs/source/zh/api-key.md) - 如何申请和配置 API 密钥 +- [使用教程](docs/source/zh/guide.md) - 常见用例和基本操作 +- [常见问题](docs/source/zh/faq.md) - 常见问题解答 + +## TODO + +- [ ] 添加口播类型视频剪辑功能 +- [ ] 添加音色克隆功能 +- [ ] 添加更多的转场/滤镜/特效功能 +- [ ] 添加图像/视频生成和编辑能力 +- [ ] 支持GPU渲染和高光裁切 + +## 致谢 + +本项目基于以下优秀的开源项目构建： + + +### 核心依赖 +- [MoviePy](https://github.com/Zulko/moviepy) - 视频编辑库 +- [FFmpeg](https://ffmpeg.org/) - 多媒体框架 +- [LangChain](https://www.langchain.com/) - 提供预构建Agent的框架 + +## 📄 License + +This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details. + +## ⭐ Star History + +[![Star History Chart](https://api.star-history.com/svg?repos=FireRedTeam/FireRed-OpenStoryline&type=date&legend=top-left)](https://www.star-history.com/#FireRedTeam/FireRed-OpenStoryline&type=date&legend=top-left) diff --git a/agent_fastapi.py b/agent_fastapi.py new file mode 100644 index 0000000000000000000000000000000000000000..99d6f9294c9fab6378947d7947cf058562690733 --- /dev/null +++ b/agent_fastapi.py @@ -0,0 +1,2826 @@ +# agent_fastapi.py +from __future__ import annotations + +import asyncio +import mimetypes +import os +import sys +import json +import re +import time +import uuid +import math +import logging +import shutil +from pathlib import Path +from dataclasses import dataclass, field +from typing import Any, Dict, List, Optional, Tuple, Set +from contextlib import asynccontextmanager +from starlette.websockets import WebSocketState, WebSocketDisconnect +try: + import tomllib # Python 3.11+ # type: ignore +except ModuleNotFoundError: + import tomli as tomllib # Python <= 3.10 +import traceback + +try: + from uvicorn.protocols.utils import ClientDisconnected +except Exception: + ClientDisconnected = None + + +logger = logging.getLogger(__name__) + +import anyio +from fastapi import FastAPI, APIRouter, UploadFile, File, Form, HTTPException, WebSocket, WebSocketDisconnect, Request +from fastapi.responses import FileResponse, JSONResponse, Response +from fastapi.staticfiles import StaticFiles + +from langchain_core.messages import SystemMessage, HumanMessage, BaseMessage, AIMessage, ToolMessage + +# ---- 确保 src 可导入（避免环境差异导致找不到模块）---- +ROOT_DIR = os.path.dirname(os.path.abspath(__file__)) +SRC_DIR = os.path.join(ROOT_DIR, "src") +if SRC_DIR not in sys.path: + sys.path.insert(0, SRC_DIR) + +from open_storyline.agent import build_agent, ClientContext +from open_storyline.utils.prompts import get_prompt +from open_storyline.utils.media_handler import scan_media_dir +from open_storyline.config import load_settings, default_config_path +from open_storyline.config import Settings +from open_storyline.storage.agent_memory import ArtifactStore +from open_storyline.mcp.hooks.node_interceptors import ToolInterceptor +from open_storyline.mcp.hooks.chat_middleware import set_mcp_log_sink, reset_mcp_log_sink + +WEB_DIR = os.path.join(ROOT_DIR, "web") +STATIC_DIR = os.path.join(WEB_DIR, "static") +INDEX_HTML = os.path.join(WEB_DIR, "index.html") +NODE_MAP_HTML = os.path.join(WEB_DIR, "node_map/node_map.html") +NODE_MAP_DIR = os.path.join(WEB_DIR, "node_map") + +SERVER_CACHE_DIR = os.path.join(ROOT_DIR, '.storyline' , ".server_cache") + +CHUNK_SIZE = 1024 * 1024 # 1MB + +# 是否根据session_id隔离用户 +USE_SESSION_SUBDIR = True + +CUSTOM_MODEL_KEY = "__custom__" + +# Load keys +DEFAULT_LLM_API_KEY = os.getenv("DEEPSEEK_API_KEY") +DEFAULT_LLM_API_URL = os.getenv("DEEPSEEK_API_URL") +DEFAULT_LLM_API_NAME = os.getenv("DEEPSEEK_API_NAME", "deepseek-chat") +DEFAULT_VLM_API_KEY = os.getenv("GLM_V4_6_API_KEY") +DEFAULT_VLM_API_URL = os.getenv("GLM_V4_6_API_URL") +DEFAULT_VLM_API_NAME = os.getenv("GLM_V4_6_API_NAME", "qwen3-vl-8b-instruct") +print("DEEPSEEK_API_KEY exists:", bool(os.getenv("DEEPSEEK_API_KEY"))) +print("QWEN3_VL_8B_API_KEY exists:", bool(os.getenv("QWEN3_VL_8B_API_KEY"))) +print("DEEPSEEK_API_URL:", repr(os.getenv("DEEPSEEK_API_URL"))) +print("QWEN3_VL_8B_API_URL:", repr(os.getenv("QWEN3_VL_8B_API_URL"))) + +def debug_traceback_print(cfg: Settings): + if cfg.developer.developer_mode: + traceback.print_exc() + +def _s(x: Any) -> str: + return str(x or "").strip() + +def _norm_url(u: Any) -> str: + u = _s(u) + return u.rstrip("/") if u else "" + +def _env_fallback_for_model(model_name: str) -> Tuple[str, str]: + """ + - deepseek* -> DEEPSEEK_API_URL / DEEPSEEK_API_KEY + - qwen3* -> QWEN3_VL_8B_API_URL / QWEN3_VL_8B_API_KEY + """ + m = _s(model_name).lower() + if "deepseek" in m: + return (_s(os.getenv("DEEPSEEK_API_URL")), _s(os.getenv("DEEPSEEK_API_KEY"))) + if m.startswith("qwen3-vl-8b-instruct") or "qwen3-vl-8b-instruct" in m: + return (_s(os.getenv("QWEN3_VL_8B_API_URL")), _s(os.getenv("QWEN3_VL_8B_API_KEY"))) + return ("", "") + +def _resolve_default_model_override(cfg: Settings, model_name: str) -> Tuple[Optional[Dict[str, Any]], Optional[str]]: + """ + 1. get config from [developer.chat_models_config.""] + 2. rollback to env + """ + model_name = _s(model_name) + if not model_name: + return None, "default model name is empty" + + model_cfg: Dict[str, Any] = {} + try: + model_cfg = (cfg.developer.chat_models_config.get(model_name) or {}) if getattr(cfg, "developer", None) else {} + except Exception: + model_cfg = {} + + if not isinstance(model_cfg, dict): + model_cfg = {} + + base_url = _norm_url(model_cfg.get("base_url")) + api_key = _s(model_cfg.get("api_key")) + + if not base_url or not api_key: + env_url, env_key = _env_fallback_for_model(model_name) + if not base_url: + base_url = _norm_url(env_url) + if not api_key: + api_key = _s(env_key) + + override: Dict[str, Any] = {"model": model_name} + if base_url: + override["base_url"] = base_url + if api_key: + override["api_key"] = api_key + + for k in ("timeout", "temperature", "max_retries", "top_p", "max_tokens"): + if k in model_cfg and model_cfg.get(k) not in (None, ""): + override[k] = model_cfg.get(k) + + if not override.get("base_url") or not override.get("api_key"): + return None, ( + f"cannot find base_url/api_key of default model: {model_name}. " + f"please fill in base_url/api_key of [developer.chat_models_config.\"{model_name}\" in config.toml]" + f"or set environment variables（DEEPSEEK_API_URL/DEEPSEEK_API_KEY / QWEN3_VL_8B_API_URL/QWEN3_VL_8B_API_KEY）。" + ) + + return override, None + +def _stable_dict_key(d: Optional[Dict[str, Any]]) -> str: + try: + return json.dumps(d or {}, sort_keys=True, ensure_ascii=False) + except Exception: + return str(d or {}) + +def _parse_service_config(service_cfg: Any) -> Tuple[ + Optional[Dict[str, Any]], + Optional[Dict[str, Any]], + Dict[str, Any], + Dict[str, Any], + Optional[str]]: + """ + 返回 (custom_llm, custom_vlm, tts_cfg, pexels, err) + - custom_llm/custom_vlm: {"model","base_url","api_key"} 或 None（允许只传 llm 或只传 vlm） + - tts_cfg: dict（可能为空） + """ + if not isinstance(service_cfg, dict): + return None, None, {}, {}, None + + # ---- custom models ---- + custom_llm = None + custom_vlm = None + custom_models = service_cfg.get("custom_models") + + if custom_models is not None: + if not isinstance(custom_models, dict): + return None, None, {}, {}, "service_config.custom_models 必须是对象" + + def _pick(m: Any, label: str) -> Tuple[Optional[Dict[str, str]], Optional[str]]: + if m is None: + return None, None + if not isinstance(m, dict): + return None, f"service_config.custom_models.{label} 必须是对象" + + model = _s(m.get("model")) + base_url = _norm_url(m.get("base_url")) + api_key = _s(m.get("api_key")) + + if not (model and base_url and api_key): + return None, f"自定义 {label.upper()} 配置不完整：请填写 model/base_url/api_key" + if not (base_url.startswith("http://") or base_url.startswith("https://")): + return None, f"自定义 {label.upper()} 的 base_url 必须以 http(s) 开头" + return {"model": model, "base_url": base_url, "api_key": api_key}, None + + custom_llm, err1 = _pick(custom_models.get("llm"), "llm") + if err1: + return None, None, {}, {}, err1 + + custom_vlm, err2 = _pick(custom_models.get("vlm"), "vlm") + if err2: + return None, None, {}, {}, err2 + + # ---- tts ---- + tts_cfg: Dict[str, Any] = {} + tts = service_cfg.get("tts") + if isinstance(tts, dict): + provider = (tts.get("provider") or "").strip().lower() + if provider: + provider_block = tts.get(provider) + tts_cfg = {"provider": provider, provider: provider_block} + + # ---- pexels ---- + pexels_cfg: Dict[str, Any] = {} + search_media = service_cfg.get("search_media") + if isinstance(search_media, dict): + # 支持两种格式： + # 1) {search_media:{pexels:{mode, api_key}}} + # 2) {search_media:{mode, pexel_api_key}} + p = search_media.get("pexels") or search_media.get("pexels") + if isinstance(p, dict): + mode = _s(p.get("mode")).lower() + if mode not in ("default", "custom"): + mode = "default" + api_key = _s(p.get("api_key") or p.get("pexels_api_key") or p.get("pexels_api_key")) + pexels_cfg = {"mode": mode, "api_key": api_key} + else: + mode = _s(search_media.get("mode") or search_media.get("pexels_mode") or search_media.get("pexels_mode")).lower() + if mode not in ("default", "custom"): + mode = "default" + api_key = _s(search_media.get("pexels_api_key") or search_media.get("pexels_api_key")) + pexels_cfg = {"mode": mode, "api_key": api_key} + + return custom_llm, custom_vlm, tts_cfg, pexels_cfg, None + +def is_developer_mode(cfg: Settings) -> bool: + try: + return bool(cfg.developer.developer_mode) + except Exception: + return False + +def _abs(p: str) -> str: + return os.path.abspath(os.path.expanduser(p)) + + +def resolve_media_dir(cfg_media_dir: str, session_id: str) -> str: + root = _abs(cfg_media_dir).rstrip("/\\") + if not USE_SESSION_SUBDIR: + return root + project_dir = os.path.dirname(root) + leaf = os.path.basename(root) + return os.path.join(project_dir, session_id, leaf) + + +def sanitize_filename(name: str) -> str: + name = os.path.basename(name or "") + name = name.replace("\x00", "") + return name or "unnamed" + + +def detect_media_kind(filename: str) -> str: + ext = os.path.splitext(filename)[1].lower() + if ext in {".png", ".jpg", ".jpeg", ".gif", ".bmp", ".webp"}: + return "image" + if ext in {".mp4", ".mov", ".avi", ".mkv", ".webm"}: + return "video" + return "unknown" + +_MEDIA_RE = re.compile(r"^media_(\d+)", re.IGNORECASE) + +def make_media_store_filename(seq: int, ext: str) -> str: + ext = (ext or "").lower() + if ext and not ext.startswith("."): + ext = "." + ext + return f"{MEDIA_PREFIX}{seq:0{MEDIA_SEQ_WIDTH}d}{ext}" + +def parse_media_seq(filename: str) -> Optional[int]: + m = _MEDIA_RE.match(os.path.basename(filename or "")) + if not m: + return None + try: + return int(m.group(1)) + except Exception: + return None + +def safe_save_path_no_overwrite(media_dir: str, filename: str) -> str: + filename = sanitize_filename(filename) + stem, ext = os.path.splitext(filename) + path = os.path.join(media_dir, filename) + if not os.path.exists(path): + return path + i = 2 + while True: + p2 = os.path.join(media_dir, f"{stem} ({i}){ext}") + if not os.path.exists(p2): + return p2 + i += 1 + + +def ensure_thumbs_dir(media_dir: str) -> str: + d = os.path.join(media_dir, ".thumbs") + os.makedirs(d, exist_ok=True) + return d + +def ensure_uploads_dir(media_dir: str) -> str: + d = os.path.join(media_dir, ".uploads") + os.makedirs(d, exist_ok=True) + return d + +def guess_media_type(path: str) -> str: + mt, _ = mimetypes.guess_type(path) + return mt or "application/octet-stream" + + +def _is_under_dir(path: str, root: str) -> bool: + try: + path = os.path.abspath(path) + root = os.path.abspath(root) + return os.path.commonpath([path, root]) == root + except Exception: + return False + + +def video_placeholder_svg_bytes() -> bytes: + svg = """ + + + + + + + + + +""" + return svg.encode("utf-8") + + +def make_image_thumbnail_sync(src_path: str, dst_path: str, max_size: Tuple[int, int] = (320, 320)) -> bool: + try: + from PIL import Image + img = Image.open(src_path).convert("RGB") + img.thumbnail(max_size) + img.save(dst_path, format="JPEG", quality=85) + return True + except Exception: + return False + +async def make_video_thumbnail_async( + src_video: str, + dst_path: str, + *, + max_size: Tuple[int, int] = (320, 320), + seek_sec: float = 0.5, + timeout_sec: float = 20.0, +) -> bool: + ffmpeg = os.environ.get("FFMPEG_BIN") or shutil.which("ffmpeg") + if not ffmpeg: + logger.warning("ffmpeg not found (PATH/FFMPEG_BIN). skip video thumbnail. src=%s", src_video) + return False + + src_video = os.path.abspath(src_video) + dst_path = os.path.abspath(dst_path) + os.makedirs(os.path.dirname(dst_path), exist_ok=True) + + tmp_path = dst_path + ".tmp.jpg" + + vf = ( + f"scale={max_size[0]}:{max_size[1]}:force_original_aspect_ratio=decrease" + f",pad={max_size[0]}:{max_size[1]}:(ow-iw)/2:(oh-ih)/2" + ) + + async def _run(args: list[str]) -> tuple[bool, str]: + proc = await asyncio.create_subprocess_exec( + *args, + stdout=asyncio.subprocess.DEVNULL, + stderr=asyncio.subprocess.PIPE, + ) + try: + _, err = await asyncio.wait_for(proc.communicate(), timeout=timeout_sec) + except asyncio.TimeoutError: + try: + proc.kill() + except Exception: + pass + await proc.wait() + return False, f"timeout after {timeout_sec}s" + err_text = (err or b"").decode("utf-8", "ignore").strip() + return (proc.returncode == 0), err_text + + # 两种策略：1) -ss 在 -i 前（快，但有些文件/关键帧会失败） + # 2) -ss 在 -i 后（慢，但更稳定） + common_tail = [ + "-an", + "-frames:v", "1", + "-vf", vf, + "-vcodec", "mjpeg", + "-q:v", "3", + "-f", "image2", + tmp_path, + ] + + attempts = [ + # fast seek + [ffmpeg, "-hide_banner", "-loglevel", "error", "-y", "-ss", f"{seek_sec}", "-i", src_video] + common_tail, + # accurate seek + [ffmpeg, "-hide_banner", "-loglevel", "error", "-y", "-i", src_video, "-ss", f"{seek_sec}"] + common_tail, + # fallback：如果 seek 太靠前导致失败，再试试 1s + [ffmpeg, "-hide_banner", "-loglevel", "error", "-y", "-ss", "1.0", "-i", src_video] + common_tail, + ] + + last_err: Optional[str] = None + try: + for args in attempts: + ok, err = await _run(args) + if ok and os.path.exists(tmp_path) and os.path.getsize(tmp_path) > 0: + os.replace(tmp_path, dst_path) + return True + last_err = err or last_err + # 清理无效临时文件，避免下次误判 + try: + if os.path.exists(tmp_path): + os.remove(tmp_path) + except Exception: + pass + + logger.warning("ffmpeg thumbnail failed. src=%s dst=%s err=%s", src_video, dst_path, last_err) + return False + finally: + try: + if os.path.exists(tmp_path): + os.remove(tmp_path) + except Exception: + pass + +def _env_int(name: str, default: int) -> int: + try: + return int(os.environ.get(name, str(default))) + except Exception: + return default + +def _env_float(name: str, default: float) -> float: + try: + return float(os.environ.get(name, str(default))) + except Exception: + return float(default) + +def _rpm_to_rps(rpm: float) -> float: + return float(rpm) / 60.0 + + +# 是否信任反向代理头（X-Forwarded-For / X-Real-IP） +RATE_LIMIT_TRUST_PROXY_HEADERS = os.environ.get("RATE_LIMIT_TRUST_PROXY_HEADERS", "0") == "1" + +@dataclass +class _RateBucket: + tokens: float + last_ts: float # monotonic + last_seen: float # monotonic (for TTL cleanup) + +class TokenBucketRateLimiter: + """ + 内存令牌桶 + 防爆内存： + - max_buckets: 限制内部桶表最大条目数（防止海量 IP 导致字典膨胀） + - evict_batch: 超过上限后每次驱逐多少条（按插入顺序驱逐最早创建的桶） + """ + def __init__( + self, + ttl_sec: int = 900, + cleanup_interval_sec: int = 60, + *, + max_buckets: int = 100000, + evict_batch: int = 2000, + ): + self.ttl_sec = int(ttl_sec) + self.cleanup_interval_sec = int(cleanup_interval_sec) + self.max_buckets = int(max(1, max_buckets)) + self.evict_batch = int(max(1, evict_batch)) + + self._buckets: Dict[str, _RateBucket] = {} + self._lock = asyncio.Lock() + self._last_cleanup = time.monotonic() + + async def allow( + self, + key: str, + *, + capacity: float, + refill_rate: float, + cost: float = 1.0, + ) -> Tuple[bool, float, float]: + """ + 返回: (allowed, retry_after_sec, remaining_tokens) + """ + now = time.monotonic() + capacity = float(max(0.0, capacity)) + refill_rate = float(max(0.0, refill_rate)) + cost = float(max(0.0, cost)) + + async with self._lock: + b = self._buckets.get(key) + + if b is None: + # 先做一次周期清理 + if now - self._last_cleanup > self.cleanup_interval_sec: + self._cleanup_locked(now) + self._last_cleanup = now + + # 桶表满了：先清 TTL，再做批量驱逐；仍然满 -> 不再创建新桶，直接拒绝 + if len(self._buckets) >= self.max_buckets: + self._cleanup_locked(now) + + if len(self._buckets) >= self.max_buckets: + self._evict_locked() + + if len(self._buckets) >= self.max_buckets: + # 不存任何新 key，避免内存继续涨 + # retry_after 给一个很短的值即可（客户端会重试） + return False, 1.0, 0.0 + + b = _RateBucket(tokens=capacity, last_ts=now, last_seen=now) + self._buckets[key] = b + else: + b.last_seen = now + + # refill + elapsed = max(0.0, now - b.last_ts) + if refill_rate > 0: + b.tokens = min(capacity, b.tokens + elapsed * refill_rate) + else: + b.tokens = min(capacity, b.tokens) + b.last_ts = now + + if b.tokens >= cost: + b.tokens -= cost + return True, 0.0, float(max(0.0, b.tokens)) + + # not enough + if refill_rate <= 0: + retry_after = float(self.ttl_sec) + else: + need = cost - b.tokens + retry_after = need / refill_rate + return False, float(retry_after), float(max(0.0, b.tokens)) + + def _cleanup_locked(self, now: float) -> None: + ttl = float(self.ttl_sec) + dead = [k for k, b in self._buckets.items() if (now - b.last_seen) > ttl] + for k in dead: + self._buckets.pop(k, None) + + def _evict_locked(self) -> None: + # 按 dict 插入顺序驱逐最早的一批 bucket（不排序，避免在高压下额外 CPU 开销） + n = min(self.evict_batch, len(self._buckets)) + for _ in range(n): + try: + k = next(iter(self._buckets)) + except StopIteration: + break + self._buckets.pop(k, None) + +def _headers_to_dict(scope_headers: List[Tuple[bytes, bytes]]) -> Dict[str, str]: + d: Dict[str, str] = {} + for k, v in scope_headers or []: + try: + dk = k.decode("latin1").lower() + dv = v.decode("latin1") + except Exception: + continue + d[dk] = dv + return d + +def _client_ip_from_http_scope(scope: dict, trust_proxy_headers: bool) -> str: + headers = _headers_to_dict(scope.get("headers") or []) + if trust_proxy_headers: + xff = headers.get("x-forwarded-for") + if xff: + # "client, proxy1, proxy2" -> client + return xff.split(",")[0].strip() or "unknown" + xri = headers.get("x-real-ip") + if xri: + return xri.strip() or "unknown" + + client = scope.get("client") + if client and isinstance(client, (list, tuple)) and len(client) >= 1: + return str(client[0] or "unknown") + return "unknown" + +def _client_ip_from_ws(ws: WebSocket, trust_proxy_headers: bool) -> str: + try: + if trust_proxy_headers: + xff = ws.headers.get("x-forwarded-for") + if xff: + return xff.split(",")[0].strip() or "unknown" + xri = ws.headers.get("x-real-ip") + if xri: + return xri.strip() or "unknown" + except Exception: + pass + + try: + if ws.client: + return str(ws.client.host or "unknown") + except Exception: + pass + + return "unknown" + +# 分片上传（绕开网关对单次请求体/单文件的限制） +UPLOAD_RESUMABLE_CHUNK_BYTES = _env_int("UPLOAD_RESUMABLE_CHUNK_BYTES", 8 * 1024 * 1024) + +# 未完成的分片上传状态保留多久（超时自动清理临时文件） +RESUMABLE_UPLOAD_TTL_SEC = _env_int("RESUMABLE_UPLOAD_TTL_SEC", 3600) # 1 hour + +MEDIA_SEQ_WIDTH = 4 # media_0001 +MEDIA_PREFIX = "media_" + + +# -------- 注意：在服务器上，所有用户的ip可能是相同的---- + +# 每个 IP 的总体请求速率（包括 /static、/api、/ 等） +HTTP_GLOBAL_RPM = _env_int("RATE_LIMIT_HTTP_GLOBAL_RPM", 3000) +HTTP_GLOBAL_BURST = _env_int("RATE_LIMIT_HTTP_GLOBAL_BURST", 600) + +# 创建 session：防止刷 session 导致内存爆 +HTTP_CREATE_SESSION_RPM = _env_int("RATE_LIMIT_CREATE_SESSION_RPM", 3000) +HTTP_CREATE_SESSION_BURST = _env_int("RATE_LIMIT_CREATE_SESSION_BURST", 50) + +# 上传素材：最容易被滥用（大文件 + 频率） +HTTP_UPLOAD_MEDIA_RPM = _env_int("RATE_LIMIT_UPLOAD_MEDIA_RPM", 12000) +HTTP_UPLOAD_MEDIA_BURST = _env_int("RATE_LIMIT_UPLOAD_MEDIA_BURST", 300) + +# 上传“成本”换算：content-length 每多少字节算 1 个 token（越大越费 token） +UPLOAD_COST_BYTES = _env_int("RATE_LIMIT_UPLOAD_COST_BYTES", 10 * 1024 * 1024) # 默认 10MB = 1 token + +# 素材个数控制：会话内上线+上传上限 +MAX_UPLOAD_FILES_PER_REQUEST = _env_int("MAX_UPLOAD_FILES_PER_REQUEST", 30) # 单次请求最多文件数 +MAX_MEDIA_PER_SESSION = _env_int("MAX_MEDIA_PER_SESSION", 30) # 每个 session 总素材上限（pending + 已用） +MAX_PENDING_MEDIA_PER_SESSION = _env_int("MAX_PENDING_MEDIA_PER_SESSION", 30) # 每个 session pending 素材上限（UI 友好） + +HTTP_UPLOAD_MEDIA_COUNT_RPM = _env_int("RATE_LIMIT_UPLOAD_MEDIA_COUNT_RPM", 50000) +HTTP_UPLOAD_MEDIA_COUNT_BURST = _env_int("RATE_LIMIT_UPLOAD_MEDIA_COUNT_BURST", 1000) + +# 下载/缩略图：适中限制（防刷资源） +HTTP_MEDIA_GET_RPM = _env_int("RATE_LIMIT_MEDIA_GET_RPM", 2400) +HTTP_MEDIA_GET_BURST = _env_int("RATE_LIMIT_MEDIA_GET_BURST", 60) + +# 清空会话：避免频繁清空扰动 +HTTP_CLEAR_RPM = _env_int("RATE_LIMIT_CLEAR_SESSION_RPM", 3000) +HTTP_CLEAR_BURST = _env_int("RATE_LIMIT_CLEAR_SESSION_BURST", 50) + +# 其它 API 默认：比 global 更细一点（可选） +HTTP_API_RPM = _env_int("RATE_LIMIT_API_RPM", 2400) +HTTP_API_BURST = _env_int("RATE_LIMIT_API_BURST", 120) + +# WebSocket：连接创建频率 +WS_CONNECT_RPM = _env_int("RATE_LIMIT_WS_CONNECT_RPM", 600) +WS_CONNECT_BURST = _env_int("RATE_LIMIT_WS_CONNECT_BURST", 50) + +# WebSocket：chat.send（真正触发 LLM 成本） +WS_CHAT_SEND_RPM = _env_int("RATE_LIMIT_WS_CHAT_SEND_RPM", 300) +WS_CHAT_SEND_BURST = _env_int("RATE_LIMIT_WS_CHAT_SEND_BURST", 20) + +# ---- 全局（所有 IP 合并）限流：抵御多 IP 同时访问 ---- +HTTP_ALL_RPM = _env_int("RATE_LIMIT_HTTP_ALL_RPM", 1200) # 全站 HTTP 总量：1200/min ~= 20 rps +HTTP_ALL_BURST = _env_int("RATE_LIMIT_HTTP_ALL_BURST", 200) + +CREATE_SESSION_ALL_RPM = _env_int("RATE_LIMIT_CREATE_SESSION_ALL_RPM", 120) +CREATE_SESSION_ALL_BURST = _env_int("RATE_LIMIT_CREATE_SESSION_ALL_BURST", 20) + +UPLOAD_MEDIA_ALL_RPM = _env_int("RATE_LIMIT_UPLOAD_MEDIA_ALL_RPM", 6000) +UPLOAD_MEDIA_ALL_BURST = _env_int("RATE_LIMIT_UPLOAD_MEDIA_ALL_BURST", 2000) + +# “素材个数”限流：默认复用 upload_media 的 rpm/burst +UPLOAD_MEDIA_COUNT_ALL_RPM = _env_int("RATE_LIMIT_UPLOAD_MEDIA_COUNT_ALL_RPM", UPLOAD_MEDIA_ALL_RPM) +UPLOAD_MEDIA_COUNT_ALL_BURST = _env_int("RATE_LIMIT_UPLOAD_MEDIA_COUNT_ALL_BURST", UPLOAD_MEDIA_ALL_BURST) + +MEDIA_GET_ALL_RPM = _env_int("RATE_LIMIT_MEDIA_GET_ALL_RPM", 600) +MEDIA_GET_ALL_BURST = _env_int("RATE_LIMIT_MEDIA_GET_ALL_BURST", 120) + +WS_CONNECT_ALL_RPM = _env_int("RATE_LIMIT_WS_CONNECT_ALL_RPM", 60000) +WS_CONNECT_ALL_BURST = _env_int("RATE_LIMIT_WS_CONNECT_ALL_BURST", 2000) + +WS_CHAT_SEND_ALL_RPM = _env_int("RATE_LIMIT_WS_CHAT_SEND_ALL_RPM", 500) +WS_CHAT_SEND_ALL_BURST = _env_int("RATE_LIMIT_WS_CHAT_SEND_ALL_BURST", 30) + +# ---- 全局并发上限：抵御“很多 IP 同时连/同时触发 LLM/同时上传” ---- +WS_MAX_CONNECTIONS = _env_int("RATE_LIMIT_WS_MAX_CONNECTIONS", 500) # 同时在线 WS 连接数上限 +CHAT_MAX_CONCURRENCY = _env_int("RATE_LIMIT_CHAT_MAX_CONCURRENCY", 80) # 同时跑的 LLM turn 上限 +UPLOAD_MAX_CONCURRENCY = _env_int("RATE_LIMIT_UPLOAD_MAX_CONCURRENCY", 100) # 同时处理上传（含缩略图）上限 + +WS_CONN_SEM = asyncio.Semaphore(WS_MAX_CONNECTIONS) +CHAT_TURN_SEM = asyncio.Semaphore(CHAT_MAX_CONCURRENCY) +UPLOAD_SEM = asyncio.Semaphore(UPLOAD_MAX_CONCURRENCY) + +def _global_http_rule_limit(rule_name: str) -> Optional[Tuple[int, int]]: + if rule_name == "create_session": + return CREATE_SESSION_ALL_BURST, CREATE_SESSION_ALL_RPM + if rule_name == "upload_media": + return UPLOAD_MEDIA_ALL_BURST, UPLOAD_MEDIA_ALL_RPM + if rule_name == "media_get": + return MEDIA_GET_ALL_BURST, MEDIA_GET_ALL_RPM + return None + + +def _get_content_length(scope: dict) -> Optional[int]: + try: + headers = _headers_to_dict(scope.get("headers") or []) + v = headers.get("content-length") + if v is None: + return None + n = int(v) + if n < 0: + return None + return n + except Exception: + return None + +def _match_http_rule(method: str, path: str) -> Tuple[str, int, int, float]: + """ + 返回 (rule_name, burst, rpm, cost) + cost 默认为 1；上传接口会按 content-length 动态计算 cost（在 middleware 内处理）。 + """ + method = (method or "").upper() + path = path or "" + + # 精确接口优先 + if method == "POST" and path == "/api/sessions": + return ("create_session", HTTP_CREATE_SESSION_BURST, HTTP_CREATE_SESSION_RPM, 1.0) + + # 上传素材（含分片接口） + if method == "POST" and path.startswith("/api/sessions/"): + if path.endswith("/media") or path.endswith("/media/init"): + return ("upload_media", HTTP_UPLOAD_MEDIA_BURST, HTTP_UPLOAD_MEDIA_RPM, 1.0) + if "/media/" in path and (path.endswith("/chunk") or path.endswith("/complete") or path.endswith("/cancel")): + return ("upload_media", HTTP_UPLOAD_MEDIA_BURST, HTTP_UPLOAD_MEDIA_RPM, 1.0) + + if method == "GET" and path.startswith("/api/sessions/") and (path.endswith("/thumb") or path.endswith("/file")): + return ("media_get", HTTP_MEDIA_GET_BURST, HTTP_MEDIA_GET_RPM, 1.0) + + if method == "POST" and path.startswith("/api/sessions/") and path.endswith("/clear"): + return ("clear_session", HTTP_CLEAR_BURST, HTTP_CLEAR_RPM, 1.0) + + # 其它 API + if path.startswith("/api/"): + return ("api_general", HTTP_API_BURST, HTTP_API_RPM, 1.0) + + # 非 /api 的其他请求：只走 global + return ("", 0, 0, 1.0) + +class HttpRateLimitMiddleware: + """ + ASGI middleware：对 HTTP 请求做限流（WebSocket 不在这里处理）。 + """ + def __init__(self, app: Any, limiter: TokenBucketRateLimiter, trust_proxy_headers: bool = False): + self.app = app + self.limiter = limiter + self.trust_proxy_headers = bool(trust_proxy_headers) + + async def __call__(self, scope: dict, receive: Any, send: Any): + if scope.get("type") != "http": + return await self.app(scope, receive, send) + + method = scope.get("method", "GET") + path = scope.get("path", "/") + ip = _client_ip_from_http_scope(scope, self.trust_proxy_headers) + + # 0) 全局总量桶（所有 IP 合并） + ok, retry_after, _ = await self.limiter.allow( + key="http:all", + capacity=float(HTTP_ALL_BURST), + refill_rate=_rpm_to_rps(float(HTTP_ALL_RPM)), + cost=1.0, + ) + if not ok: + return await self._reject(send, retry_after) + + # 1) 单 IP 全局桶（防单点） + ok, retry_after, _ = await self.limiter.allow( + key=f"http:global:{ip}", + capacity=float(HTTP_GLOBAL_BURST), + refill_rate=_rpm_to_rps(float(HTTP_GLOBAL_RPM)), + cost=1.0, + ) + if not ok: + return await self._reject(send, retry_after) + + # 2) 规则桶 + rule_name, burst, rpm, cost = _match_http_rule(method, path) + + # 上传接口：按 content-length 动态增加 cost（越大越费 token） + if rule_name == "upload_media": + cl = _get_content_length(scope) + if cl and cl > 0 and UPLOAD_COST_BYTES > 0: + cost = max(1.0, float(math.ceil(cl / float(UPLOAD_COST_BYTES)))) + + if rule_name: + # 2.1 规则的“全局桶”（跨 IP） + g = _global_http_rule_limit(rule_name) + if g: + g_burst, g_rpm = g + okg, rag, _ = await self.limiter.allow( + key=f"http:{rule_name}:all", + capacity=float(g_burst), + refill_rate=_rpm_to_rps(float(g_rpm)), + cost=float(cost), + ) + if not okg: + return await self._reject(send, rag) + + # 2.2 规则的“单 IP 桶” + ok2, retry_after2, _ = await self.limiter.allow( + key=f"http:{rule_name}:{ip}", + capacity=float(burst), + refill_rate=_rpm_to_rps(float(rpm)), + cost=float(cost), + ) + if not ok2: + return await self._reject(send, retry_after2) + + return await self.app(scope, receive, send) + + + async def _reject(self, send: Any, retry_after: float): + ra = int(math.ceil(float(retry_after or 0.0))) + body = json.dumps( + {"detail": "Too Many Requests", "retry_after": ra}, + ensure_ascii=False + ).encode("utf-8") + + headers = [ + (b"content-type", b"application/json; charset=utf-8"), + (b"retry-after", str(ra).encode("ascii")), + ] + + await send({"type": "http.response.start", "status": 429, "headers": headers}) + await send({"type": "http.response.body", "body": body, "more_body": False}) + +RATE_LIMITER = TokenBucketRateLimiter( + ttl_sec=_env_int("RATE_LIMIT_TTL_SEC", 900), # 默认 15 分钟：多 IP 攻击时更快释放桶表 + cleanup_interval_sec=_env_int("RATE_LIMIT_CLEANUP_INTERVAL_SEC", 60), + max_buckets=_env_int("RATE_LIMIT_MAX_BUCKETS", 100000), + evict_batch=_env_int("RATE_LIMIT_EVICT_BATCH", 2000), +) + + +@dataclass +class MediaMeta: + id: str + name: str + kind: str + path: str + thumb_path: Optional[str] + ts: float + +@dataclass +class ResumableUpload: + upload_id: str + filename: str # 素材原名（用于UI展示） + store_filename: str # 落盘名 media_0001.mp4 + size: int + chunk_size: int + total_chunks: int + tmp_path: str + kind: str + created_ts: float + last_ts: float + received: Set[int] = field(default_factory=set) + closed: bool = False + lock: asyncio.Lock = field(default_factory=asyncio.Lock) + +class MediaStore: + """ + 专注文件系统层： + - 保存上传文件（async chunk） + - 生成缩略图（图片：线程；视频：异步子进程） + - 删除文件（只删 media_dir 下的文件） + """ + def __init__(self, media_dir: str): + self.media_dir = os.path.abspath(media_dir) + os.makedirs(self.media_dir, exist_ok=True) + self.thumbs_dir = ensure_thumbs_dir(self.media_dir) + + async def save_upload(self, uf: UploadFile, *, store_filename: str, display_name: str) -> MediaMeta: + media_id = uuid.uuid4().hex[:10] + + display_name = sanitize_filename(display_name or uf.filename or "unnamed") + store_filename = sanitize_filename(store_filename) + + kind = detect_media_kind(display_name) + + save_path = os.path.join(self.media_dir, store_filename) + os.makedirs(os.path.dirname(save_path), exist_ok=True) + + if os.path.exists(save_path): + raise HTTPException(status_code=409, detail=f"media filename exists: {store_filename}") + + # async chunk 写盘（不一次性读入内存） + async with await anyio.open_file(save_path, "wb") as out: + while True: + chunk = await uf.read(CHUNK_SIZE) + if not chunk: + break + await out.write(chunk) + + try: + await uf.close() + except Exception: + pass + + thumb_path: Optional[str] = None + if kind in ("image", "video"): + thumb_path = os.path.join(self.thumbs_dir, f"{media_id}.jpg") + + if kind == "image": + ok = await anyio.to_thread.run_sync(make_image_thumbnail_sync, save_path, thumb_path) + else: + ok = await make_video_thumbnail_async(save_path, thumb_path) + + if not ok: + # 图片缩略图失败 -> 用原图；视频失败 -> 置空（thumb endpoint 返回占位 SVG） + thumb_path = save_path if kind == "image" else None + + return MediaMeta( + id=media_id, + name=os.path.basename(display_name), + kind=kind, + path=os.path.abspath(save_path), + thumb_path=os.path.abspath(thumb_path) if thumb_path else None, + ts=time.time(), + ) + + async def save_from_path( + self, + src_path: str, + *, + store_filename: str, + display_name: str, + ) -> MediaMeta: + """ + 将分片上传产生的临时文件移动到 media_dir 下的最终文件。 + - display_name: UI 展示名（原始文件名） + - store_filename: 落盘名（media_0001.mp4），用于记录顺序 + """ + media_id = uuid.uuid4().hex[:10] + + display_name = sanitize_filename(display_name or "unnamed") + store_filename = sanitize_filename(store_filename or "unnamed") + + kind = detect_media_kind(display_name) + + src_path = os.path.abspath(src_path) + if not os.path.exists(src_path): + raise HTTPException(status_code=400, detail="upload temp file missing") + + save_path = os.path.abspath(os.path.join(self.media_dir, store_filename)) + os.makedirs(os.path.dirname(save_path), exist_ok=True) + + if os.path.exists(save_path): + raise HTTPException(status_code=409, detail=f"media already exists: {store_filename}") + + # move tmp -> final + os.replace(src_path, save_path) + + thumb_path: Optional[str] = None + if kind in ("image", "video"): + thumb_path = os.path.join(self.thumbs_dir, f"{media_id}.jpg") + + if kind == "image": + ok = await anyio.to_thread.run_sync(make_image_thumbnail_sync, save_path, thumb_path) + else: + ok = await make_video_thumbnail_async(save_path, thumb_path) + + if not ok: + thumb_path = save_path if kind == "image" else None + + return MediaMeta( + id=media_id, + name=os.path.basename(display_name), # ★ UI 显示原文件名 + kind=kind, + path=os.path.abspath(save_path), # ★ 磁盘文件名 media_0001.ext + thumb_path=os.path.abspath(thumb_path) if thumb_path else None, + ts=time.time(), + ) + + async def delete_files(self, meta: MediaMeta) -> None: + root = self.media_dir + for p in {meta.path, meta.thumb_path}: + if not p: + continue + ap = os.path.abspath(p) + if not _is_under_dir(ap, root): + continue + if os.path.isdir(ap): + continue + if os.path.exists(ap): + try: + os.remove(ap) + except Exception: + pass + + +class ChatSession: + """ + 一个 session 的全部状态： + - agent / lc_messages（LangChain上下文） + - history（给前端回放） + - load_media / pending_media（staging） + - tool trace 索引（支持 tool 事件“就地更新”） + """ + def __init__(self, session_id: str, cfg: Settings): + self.session_id = session_id + self.cfg = cfg + self.lang = "zh" + + default_llm = _s(getattr(getattr(cfg, "developer", None), "default_llm", "")) or "deepseek-chat" + default_vlm = _s(getattr(getattr(cfg, "developer", None), "default_vlm", "")) or "qwen3-vl-8b-instruct" + + self.chat_models = [default_llm, CUSTOM_MODEL_KEY] + self.chat_model_key = default_llm + + self.vlm_models = [default_vlm, CUSTOM_MODEL_KEY] + self.vlm_model_key = default_vlm + + self.developer_mode = is_developer_mode(cfg) + + self.media_dir = resolve_media_dir(cfg.project.media_dir, session_id) + self.media_store = MediaStore(self.media_dir) + # 分片上传临时目录 + in-flight 状态 + self.uploads_dir = ensure_uploads_dir(self.media_dir) + self.resumable_uploads: Dict[str, ResumableUpload] = {} + + # 直传（multipart 多文件）时的“预占位”，避免并发竞争导致超过上限 + self._direct_upload_reservations = 0 + + self.agent: Any = None + self.node_manager = None + self.client_context = None + + # 锁分离：避免“流式输出”阻塞上传/删除 pending + self.chat_lock = asyncio.Lock() + self.media_lock = asyncio.Lock() + + self.sent_media_total: int = 0 + self._attach_stats_msg_idx = 1 + + self.lc_messages: List[BaseMessage] = [ + SystemMessage(content=get_prompt("instruction.system", lang=self.lang)), + SystemMessage(content="【User media upload status】{}"), + ] + self.history: List[Dict[str, Any]] = [] + + self.load_media: Dict[str, MediaMeta] = {} + self.pending_media_ids: List[str] = [] + + self._tool_history_index: Dict[str, int] = {} # tool_call_id -> history index + + self.cancel_event = asyncio.Event() # 打断信号 + + # 服务相关配置 + self.custom_llm_config: Optional[Dict[str, Any]] = None + self.custom_vlm_config: Optional[Dict[str, Any]] = None + self.tts_config: Dict[str, Any] = {} + self._agent_build_key: Optional[Tuple[Any, ...]] = None + + self.pexels_key_mode: str = "default" # "default" | "custom" + self.pexels_custom_key: str = "" + + self._media_seq_inited = False + self._media_seq_next = 1 + + def _ensure_system_prompt(self) -> None: + sys = (get_prompt("instruction.system", lang=self.lang) or "").strip() + if not sys: + return + + for m in self.lc_messages: + if isinstance(m, SystemMessage) and (getattr(m, "content", "") or "").strip() == sys: + return + + self.lc_messages.insert(0, SystemMessage(content=sys)) + + def _init_media_seq_locked(self) -> None: + """ + 初始化 self._media_seq_next： + - 允许 clear chat 后继续编号，不覆盖旧文件 + """ + if self._media_seq_inited: + return + + max_seq = 0 + + # 1) 已落盘文件 + try: + for fn in os.listdir(self.media_dir): + s = parse_media_seq(fn) + if s is not None: + max_seq = max(max_seq, s) + except Exception: + pass + + # 2) 内存里已有 load_media（保险） + for meta in (self.load_media or {}).values(): + s = parse_media_seq(os.path.basename(meta.path or "")) + if s is not None: + max_seq = max(max_seq, s) + + # 3) in-flight resumable（保险） + for u in (self.resumable_uploads or {}).values(): + s = parse_media_seq(getattr(u, "store_filename", "") or "") + if s is not None: + max_seq = max(max_seq, s) + + self._media_seq_next = max_seq + 1 + self._media_seq_inited = True + + + def _reserve_store_filenames_locked(self, display_filenames: List[str]) -> List[str]: + """ + 按传入顺序生成一组 store 文件名（media_0001.ext ...） + 注意：这里的“顺序”就是你要固化的上传顺序。 + """ + self._init_media_seq_locked() + + out: List[str] = [] + seq = int(self._media_seq_next) + + for disp in display_filenames: + disp = sanitize_filename(disp or "unnamed") + ext = os.path.splitext(disp)[1].lower() + + # 不复用旧号；仅在极端情况下跳过已存在文件（防撞） + while True: + store = make_media_store_filename(seq, ext) + if not os.path.exists(os.path.join(self.media_dir, store)): + break + seq += 1 + + out.append(store) + seq += 1 + + self._media_seq_next = seq + return out + + + def apply_service_config(self, service_cfg: Any) -> Tuple[bool, Optional[str]]: + llm, vlm, tts, pexels, err = _parse_service_config(service_cfg) + if err: + return False, err + + if llm is not None: + self.custom_llm_config = llm + if vlm is not None: + self.custom_vlm_config = vlm + + # tts 允许为空；非空才覆盖 + if isinstance(tts, dict) and tts: + self.tts_config = tts + + # ---- pexels ---- + if isinstance(pexels, dict) and pexels: + mode = _s(pexels.get("mode")).lower() + if mode == "custom": + self.pexels_key_mode = "custom" + self.pexels_custom_key = _s(pexels.get("api_key")) + else: + self.pexels_key_mode = "default" + self.pexels_custom_key = "" + + return True, None + + async def ensure_agent(self) -> None: + # 1) resolve LLM override + if self.chat_model_key == CUSTOM_MODEL_KEY: + if not isinstance(self.custom_llm_config, dict): + raise RuntimeError("please fill in model/base_url/api_key of custom LLM") + llm_override = self.custom_llm_config + else: + llm_override, err = _resolve_default_model_override(self.cfg, self.chat_model_key) + if err: + raise RuntimeError(err) + + # 2) resolve VLM override + if self.vlm_model_key == CUSTOM_MODEL_KEY: + if not isinstance(self.custom_vlm_config, dict): + raise RuntimeError("please fill in model/base_url/api_key of custom VLM") + vlm_override = self.custom_vlm_config + else: + vlm_override, err = _resolve_default_model_override(self.cfg, self.vlm_model_key) + if err: + raise RuntimeError(err) + + agent_build_key: Tuple[Any, ...] = ( + "models", + _stable_dict_key(llm_override), + _stable_dict_key(vlm_override), + ) + + if self.agent is None or self._agent_build_key != agent_build_key: + artifact_store = ArtifactStore(self.cfg.project.outputs_dir, session_id=self.session_id) + self.agent, self.node_manager = await build_agent( + cfg=self.cfg, + session_id=self.session_id, + store=artifact_store, + tool_interceptors=[ + ToolInterceptor.inject_media_content_before, + ToolInterceptor.save_media_content_after, + ToolInterceptor.inject_tts_config, + ToolInterceptor.inject_pexels_api_key, + ], + llm_override=llm_override, + vlm_override=vlm_override, + ) + self._agent_build_key = agent_build_key + + if self.client_context is None: + self.client_context = ClientContext( + cfg=self.cfg, + session_id=self.session_id, + media_dir=self.media_dir, + bgm_dir=self.cfg.project.bgm_dir, + outputs_dir=self.cfg.project.outputs_dir, + node_manager=self.node_manager, + chat_model_key=self.chat_model_key, + vlm_model_key=self.vlm_model_key, + tts_config=(self.tts_config or None), + pexels_api_key=None, + lang=self.lang, + ) + else: + self.client_context.chat_model_key = self.chat_model_key + self.client_context.vlm_model_key = self.vlm_model_key + self.client_context.tts_config = (self.tts_config or None) + self.client_context.lang = self.lang + + # ---- resolve pexels_api_key for runtime context ---- + pexels_api_key = "" + if (self.pexels_key_mode or "").lower() == "custom": + pexels_api_key = _s(self.pexels_custom_key) + else: + pexels_api_key = _get_default_pexels_api_key(self.cfg) # from config.toml + + self.client_context.pexels_api_key = (pexels_api_key or None) + + # ---- DTO / public mapping ---- + def public_media(self, meta: MediaMeta) -> Dict[str, Any]: + return { + "id": meta.id, + "name": meta.name, + "kind": meta.kind, + "thumb_url": f"/api/sessions/{self.session_id}/media/{meta.id}/thumb", + "file_url": f"/api/sessions/{self.session_id}/media/{meta.id}/file", + } + + def public_pending_media(self) -> List[Dict[str, Any]]: + out: List[Dict[str, Any]] = [] + for aid in self.pending_media_ids: + meta = self.load_media.get(aid) + if meta: + out.append(self.public_media(meta)) + return out + + def snapshot(self) -> Dict[str, Any]: + return { + "session_id": self.session_id, + "developer_mode": self.developer_mode, + "pending_media": self.public_pending_media(), + "history": self.history, + "limits": { + "max_upload_files_per_request": MAX_UPLOAD_FILES_PER_REQUEST, + "max_media_per_session": MAX_MEDIA_PER_SESSION, + "max_pending_media_per_session": MAX_PENDING_MEDIA_PER_SESSION, + "upload_chunk_bytes": UPLOAD_RESUMABLE_CHUNK_BYTES, + }, + "stats": { + "media_count": len(self.load_media), + "pending_count": len(self.pending_media_ids), + "inflight_uploads": len(self.resumable_uploads), + }, + "chat_model_key": self.chat_model_key, + "chat_models": self.chat_models, + "llm_model_key": self.chat_model_key, + "llm_models": self.chat_models, + "vlm_model_key": self.vlm_model_key, + "vlm_models": self.vlm_models, + "lang": self.lang, + } + + # ---- media operations ---- + def _cleanup_stale_uploads_locked(self, now: Optional[float] = None) -> None: + now = float(now or time.time()) + ttl = float(RESUMABLE_UPLOAD_TTL_SEC) + dead = [uid for uid, u in self.resumable_uploads.items() if (now - u.last_ts) > ttl] + for uid in dead: + u = self.resumable_uploads.pop(uid, None) + if not u: + continue + try: + if u.tmp_path and os.path.exists(u.tmp_path): + os.remove(u.tmp_path) + except Exception: + pass + + def _check_media_caps_locked(self, add: int = 0) -> None: + add = int(max(0, add)) + total = len(self.load_media) + len(self.resumable_uploads) + int(self._direct_upload_reservations) + pending = len(self.pending_media_ids) + len(self.resumable_uploads) + int(self._direct_upload_reservations) + + if MAX_MEDIA_PER_SESSION > 0 and (total + add) > MAX_MEDIA_PER_SESSION: + raise HTTPException( + status_code=400, + detail=f"会话素材总数已达上限：{total}/{MAX_MEDIA_PER_SESSION}", + ) + + if MAX_PENDING_MEDIA_PER_SESSION > 0 and (pending + add) > MAX_PENDING_MEDIA_PER_SESSION: + raise HTTPException( + status_code=400, + detail=f"待发送素材数量已达上限：{pending}/{MAX_PENDING_MEDIA_PER_SESSION}", + ) + + async def add_uploads(self, files: List[UploadFile], store_filenames: List[str]) -> List[MediaMeta]: + if len(store_filenames) != len(files): + raise HTTPException(status_code=500, detail="store_filenames mismatch") + + metas: List[MediaMeta] = [] + for uf, store_fn in zip(files, store_filenames): + display_name = sanitize_filename(uf.filename or "unnamed") + metas.append(await self.media_store.save_upload( + uf, + store_filename=store_fn, + display_name=display_name, + )) + + async with self.media_lock: + for m in metas: + self.load_media[m.id] = m + self.pending_media_ids.append(m.id) + + self.pending_media_ids.sort( + key=lambda aid: os.path.basename(self.load_media[aid].path or "") + if aid in self.load_media else "" + ) + + return metas + + async def delete_pending_media(self, media_id: str) -> None: + async with self.media_lock: + if media_id not in self.pending_media_ids: + raise HTTPException(status_code=400, detail="media is not pending (refuse physical delete)") + self.pending_media_ids = [x for x in self.pending_media_ids if x != media_id] + meta = self.load_media.pop(media_id, None) + + if meta: + await self.media_store.delete_files(meta) + + async def take_pending_media_for_message(self, attachment_ids: Optional[List[str]]) -> List[MediaMeta]: + async with self.media_lock: + if attachment_ids: + pick = [aid for aid in attachment_ids if aid in self.pending_media_ids] + else: + pick = list(self.pending_media_ids) + + pick_set = set(pick) + self.pending_media_ids = [aid for aid in self.pending_media_ids if aid not in pick_set] + metas = [self.load_media[aid] for aid in pick if aid in self.load_media] + return metas + + # ---- tool trace handling ---- + def _ensure_tool_record(self, tcid: str, server: str, name: str, args: Any) -> Dict[str, Any]: + idx = self._tool_history_index.get(tcid) + if idx is None: + rec = { + "id": f"tool_{tcid}", + "role": "tool", + "tool_call_id": tcid, + "server": server, + "name": name, + "args": args, + "state": "running", + "progress": 0.0, + "message": "", + "summary": None, + "ts": time.time(), + } + self.history.append(rec) + self._tool_history_index[tcid] = len(self.history) - 1 + return rec + return self.history[idx] + + def apply_tool_event(self, raw: Dict[str, Any]) -> Optional[Dict[str, Any]]: + et = raw.get("type") + tcid = raw.get("tool_call_id") + if et not in ("tool_start", "tool_progress", "tool_end") or not tcid: + return None + + server = raw.get("server") or "" + name = raw.get("name") or "" + args = raw.get("args") or {} + + rec = self._ensure_tool_record(tcid, server, name, args) + + if et == "tool_start": + rec.update({ + "server": server, + "name": name, + "args": args, + "state": "running", + "progress": 0.0, + "message": "Starting...", + "summary": None, + }) + + elif et == "tool_progress": + progress = float(raw.get("progress", 0.0)) + total = raw.get("total") + if total and float(total) > 0: + p = progress / float(total) + else: + p = progress / 100.0 if progress > 1 else progress + p = max(0.0, min(1.0, p)) + rec.update({ + "state": "running", + "progress": p, + "message": raw.get("message") or "", + }) + + elif et == "tool_end": + is_error = bool(raw.get("is_error")) + + summary = raw.get("summary") + try: + json.dumps(summary, ensure_ascii=False) + except Exception: + summary = str(summary) if summary is not None else None + rec.update({ + "state": "error" if is_error else "complete", + "progress": 1.0, + "summary": summary, + "message": raw.get("message") or rec.get("message") or "", + }) + + return rec + + +class SessionStore: + def __init__(self, cfg: Settings): + self.cfg = cfg + self._lock = asyncio.Lock() + self._sessions: Dict[str, ChatSession] = {} + + async def create(self) -> ChatSession: + sid = uuid.uuid4().hex + sess = ChatSession(sid, self.cfg) + async with self._lock: + self._sessions[sid] = sess + return sess + + async def get(self, sid: str) -> Optional[ChatSession]: + async with self._lock: + return self._sessions.get(sid) + + async def get_or_404(self, sid: str) -> ChatSession: + sess = await self.get(sid) + if not sess: + raise HTTPException(status_code=404, detail="session not found") + return sess + + +@asynccontextmanager +async def lifespan(app: FastAPI): + cfg = load_settings(default_config_path()) + app.state.cfg = cfg + app.state.developer_mode = is_developer_mode(cfg) + app.state.sessions = SessionStore(cfg) + yield + + +app = FastAPI(title="OpenStoryline Web", version="1.0.0", lifespan=lifespan) + +app.add_middleware( + HttpRateLimitMiddleware, + limiter=RATE_LIMITER, + trust_proxy_headers=RATE_LIMIT_TRUST_PROXY_HEADERS, +) + +if os.path.isdir(STATIC_DIR): + app.mount("/static", StaticFiles(directory=STATIC_DIR), name="static") + +if os.path.isdir(NODE_MAP_DIR): + app.mount("/node_map", StaticFiles(directory=NODE_MAP_DIR), name="node_map") + +api = APIRouter(prefix="/api") + +def _rate_limit_reject_json(retry_after: float) -> JSONResponse: + ra = int(math.ceil(float(retry_after or 0.0))) + return JSONResponse( + {"detail": "Too Many Requests", "retry_after": ra}, + status_code=429, + headers={"Retry-After": str(ra)}, + ) + +async def _enforce_upload_media_count_limit(request: Request, cost: float) -> Optional[JSONResponse]: + ip = _client_ip_from_http_scope(request.scope, RATE_LIMIT_TRUST_PROXY_HEADERS) + cost = float(max(0.0, cost)) + + ok, ra, _ = await RATE_LIMITER.allow( + key="http:upload_media_count:all", + capacity=float(UPLOAD_MEDIA_COUNT_ALL_BURST), + refill_rate=_rpm_to_rps(float(UPLOAD_MEDIA_COUNT_ALL_RPM)), + cost=cost, + ) + if not ok: + return _rate_limit_reject_json(ra) + + ok2, ra2, _ = await RATE_LIMITER.allow( + key=f"http:upload_media_count:{ip}", + capacity=float(HTTP_UPLOAD_MEDIA_COUNT_BURST), + refill_rate=_rpm_to_rps(float(HTTP_UPLOAD_MEDIA_COUNT_RPM)), + cost=cost, + ) + if not ok2: + return _rate_limit_reject_json(ra2) + + return None + +_TTS_UI_SECRET_KEYS = { + "api_key", + "access_token", + "authorization", + "token", + "password", + "secret", + "x-api-key", + "apikey", + "access_key", + "accesskey", +} + +def _is_secret_field_name(k: str) -> bool: + if str(k or "").strip().lower() in _TTS_UI_SECRET_KEYS: + return True + return False + +def _read_config_toml(path: str) -> dict: + if tomllib is None: + return {} + try: + p = Path(path) + with p.open("rb") as f: + return tomllib.load(f) or {} + except Exception: + return {} + +def _get_default_pexels_api_key(cfg: Settings) -> str: + # 1) try Settings.search_media.pexels_api_key + try: + search_media = getattr(cfg, "search_media", None) + pexels_api_key = _s(getattr(search_media, "pexels_api_key", None) if search_media else None) + if pexels_api_key: + return pexels_api_key + else: + return "" + except Exception: + return "" + +def _normalize_field_item(item) -> dict | None: + """ + item 支持： + - "uid" + - { key="uid", label="UID", required=true, secret=false, placeholder="..." } + """ + if isinstance(item, str): + key = item.strip() + if not key: + return None + return { + "key": key, + "secret": _is_secret_field_name(key), + } + return None + +def _build_provider_schema(provider: str, label: str | None, fields: list[dict]) -> dict: + seen = set() + out = [] + for f in fields: + k = str(f.get("key") or "").strip() + if not k or k in seen: + continue + seen.add(k) + out.append({ + "key": k, + "label": f.get("label") or k, + "placeholder": f.get("placeholder") or f.get("label") or k, + "required": bool(f.get("required", False)), + "secret": bool(f.get("secret", False)), + }) + return {"provider": provider, "label": label or provider, "fields": out} + +def _build_tts_ui_schema_from_config(config_path: str) -> dict: + """ + 返回： + { + "providers": [ + {"provider":"bytedance","label":"字节跳动","fields":[{"key":"uid",...}, ...]}, + ... + ] + } + """ + cfg = _read_config_toml(config_path) + tts = cfg.get("generate_voiceover", {}) + + providers_out: list[dict] = [] + + # 格式：[tts.providers.] + providers = tts.get("providers") + if isinstance(providers, dict): + for provider, provider_cfg in providers.items(): + fields: list[dict] = [] + label = str(provider_cfg.get("label") or provider_cfg.get("name") or provider) + for key in provider_cfg.keys(): + f = _normalize_field_item(str(key)) + if f: + fields.append(f) + + providers_out.append(_build_provider_schema(provider, label, fields)) + + return {"providers": providers_out} + +@app.get("/") +async def index(): + if not os.path.exists(INDEX_HTML): + return Response("index.html not found. Put it under ./web/index.html", media_type="text/plain", status_code=404) + return FileResponse(INDEX_HTML, media_type="text/html") + +@app.get("/node-map") +async def node_map(): + if not os.path.exists(NODE_MAP_HTML): + return Response( + "node_map.html not found. Put it under ./web/node_map/node_map.html", + media_type="text/plain", + status_code=404, + ) + return FileResponse(NODE_MAP_HTML, media_type="text/html") + +@api.get("/meta/tts") +async def get_tts_ui_schema(): + schema = _build_tts_ui_schema_from_config(default_config_path()) + return JSONResponse(schema) + +# ------------------------- +# Sessions (REST) +# ------------------------- +@api.post("/sessions") +async def create_session(): + store: SessionStore = app.state.sessions + sess = await store.create() + return JSONResponse(sess.snapshot()) + + +@api.get("/sessions/{session_id}") +async def get_session(session_id: str): + store: SessionStore = app.state.sessions + sess = await store.get_or_404(session_id) + return JSONResponse(sess.snapshot()) + + +@api.post("/sessions/{session_id}/clear") +async def clear_session_chat(session_id: str): + store: SessionStore = app.state.sessions + sess = await store.get_or_404(session_id) + async with sess.chat_lock: + sess.sent_media_total = 0 + sess._attach_stats_msg_idx = 1 + sess.lc_messages = [ + SystemMessage(content=get_prompt("instruction.system", lang=sess.lang)), + SystemMessage(content="【User media upload status】{}"), + ] + sess._attach_stats_msg_idx = 1 + + sess.history = [] + sess._tool_history_index = {} + return JSONResponse({"ok": True}) + +@api.post("/sessions/{session_id}/cancel") +async def cancel_session_turn(session_id: str): + """ + 打断当前正在进行的 LLM turn（流式回复/工具调用）。 + - 不清空 history / lc_messages + - 仅设置 cancel_event，由 WS 侧在流式循环中感知并安全收尾 + """ + store: SessionStore = app.state.sessions + sess = await store.get_or_404(session_id) + sess.cancel_event.set() + return JSONResponse({"ok": True}) + +# ------------------------- +# media (REST, session-scoped) +# ------------------------- +@api.post("/sessions/{session_id}/media") +async def upload_media(session_id: str, request: Request, files: List[UploadFile] = File(...)): + if not isinstance(files, list) or not files: + raise HTTPException(status_code=400, detail="no files") + + if MAX_UPLOAD_FILES_PER_REQUEST > 0 and len(files) > MAX_UPLOAD_FILES_PER_REQUEST: + raise HTTPException(status_code=400, detail=f"单次上传最多 {MAX_UPLOAD_FILES_PER_REQUEST} 个文件") + + # 按素材个数限流（cost = 文件数） + rej = await _enforce_upload_media_count_limit(request, cost=float(len(files))) + if rej: + return rej + + if UPLOAD_SEM.locked(): + raise HTTPException(status_code=429, detail="上传并发过高，请稍后重试") + await UPLOAD_SEM.acquire() + + n = len(files) + try: + store: SessionStore = app.state.sessions + sess = await store.get_or_404(session_id) + + # session cap 检查 + 预占位（避免并发竞争） + async with sess.media_lock: + sess._cleanup_stale_uploads_locked() + sess._check_media_caps_locked(add=n) + sess._direct_upload_reservations += n + + display_names = [sanitize_filename(uf.filename or "unnamed") for uf in files] + store_filenames = sess._reserve_store_filenames_locked(display_names) + + try: + metas = await sess.add_uploads(files, store_filenames=store_filenames) + + finally: + async with sess.media_lock: + sess._direct_upload_reservations = max(0, sess._direct_upload_reservations - n) + + return JSONResponse({ + "media": [sess.public_media(m) for m in metas], + "pending_media": sess.public_pending_media(), + }) + finally: + try: + UPLOAD_SEM.release() + except Exception: + pass + +@api.post("/sessions/{session_id}/media/init") +async def init_resumable_media_upload(session_id: str, request: Request): + try: + data = await request.json() + if not isinstance(data, dict): + data = {} + except Exception: + data = {} + + filename = sanitize_filename((data.get("filename") or data.get("name") or "unnamed")) + size = int(data.get("size") or 0) + if size <= 0: + raise HTTPException(status_code=400, detail="invalid size") + + # 按素材个数限流：init 视为“新增 1 个素材” + rej = await _enforce_upload_media_count_limit(request, cost=1.0) + if rej: + return rej + + store: SessionStore = app.state.sessions + sess = await store.get_or_404(session_id) + + async with sess.media_lock: + sess._cleanup_stale_uploads_locked() + sess._check_media_caps_locked(add=1) + + store_filename = sess._reserve_store_filenames_locked([filename])[0] + + upload_id = uuid.uuid4().hex + chunk_size = int(max(1, UPLOAD_RESUMABLE_CHUNK_BYTES)) + total_chunks = int(math.ceil(size / float(chunk_size))) + + tmp_path = os.path.join(sess.uploads_dir, f"{upload_id}.part") + os.makedirs(os.path.dirname(tmp_path), exist_ok=True) + try: + with open(tmp_path, "wb"): + pass + except Exception as e: + raise HTTPException(status_code=500, detail=f"cannot create temp file: {e}") + + u = ResumableUpload( + upload_id=upload_id, + filename=filename, + store_filename=store_filename, + size=size, + chunk_size=chunk_size, + total_chunks=total_chunks, + tmp_path=os.path.abspath(tmp_path), + kind=detect_media_kind(filename), + created_ts=time.time(), + last_ts=time.time(), + ) + sess.resumable_uploads[upload_id] = u + + return JSONResponse({ + "upload_id": upload_id, + "chunk_size": chunk_size, + "total_chunks": total_chunks, + "filename": filename, + }) + + +@api.post("/sessions/{session_id}/media/{upload_id}/chunk") +async def upload_resumable_media_chunk( + session_id: str, + upload_id: str, + index: int = Form(...), + chunk: UploadFile = File(...), +): + if UPLOAD_SEM.locked(): + raise HTTPException(status_code=429, detail="上传并发过高，请稍后重试") + await UPLOAD_SEM.acquire() + try: + store: SessionStore = app.state.sessions + sess = await store.get_or_404(session_id) + + async with sess.media_lock: + sess._cleanup_stale_uploads_locked() + u = sess.resumable_uploads.get(upload_id) + + if not u: + raise HTTPException(status_code=404, detail="upload_id not found or expired") + + idx = int(index) + if idx < 0 or idx >= u.total_chunks: + raise HTTPException(status_code=400, detail="invalid chunk index") + + # 期望长度（最后一片可能小于 chunk_size） + expected_len = u.size - idx * u.chunk_size + if expected_len <= 0: + raise HTTPException(status_code=400, detail="invalid chunk index") + expected_len = min(u.chunk_size, expected_len) + + written = 0 + async with u.lock: + if u.closed: + raise HTTPException(status_code=400, detail="upload already closed") + + async with await anyio.open_file(u.tmp_path, "r+b") as out: + await out.seek(idx * u.chunk_size) + while True: + buf = await chunk.read(CHUNK_SIZE) + if not buf: + break + written += len(buf) + if written > expected_len: + raise HTTPException(status_code=400, detail="chunk too large") + await out.write(buf) + + try: + await chunk.close() + except Exception: + pass + + if written != expected_len: + raise HTTPException(status_code=400, detail=f"chunk size mismatch: {written} != {expected_len}") + + u.received.add(idx) + u.last_ts = time.time() + + return JSONResponse({ + "ok": True, + "received_chunks": len(u.received), + "total_chunks": u.total_chunks, + }) + finally: + try: + UPLOAD_SEM.release() + except Exception: + pass + + +@api.post("/sessions/{session_id}/media/{upload_id}/complete") +async def complete_resumable_media_upload(session_id: str, upload_id: str): + if UPLOAD_SEM.locked(): + raise HTTPException(status_code=429, detail="上传并发过高，请稍后重试") + await UPLOAD_SEM.acquire() + try: + store: SessionStore = app.state.sessions + sess = await store.get_or_404(session_id) + + async with sess.media_lock: + sess._cleanup_stale_uploads_locked() + u = sess.resumable_uploads.get(upload_id) + + if not u: + raise HTTPException(status_code=404, detail="upload_id not found or expired") + + # 锁住此 upload，防止 chunk 并发写 + async with u.lock: + u.closed = True + if len(u.received) != u.total_chunks: + missing = u.total_chunks - len(u.received) + raise HTTPException(status_code=400, detail=f"chunks missing: {missing}") + + # 从索引移除（释放会话额度） + async with sess.media_lock: + u2 = sess.resumable_uploads.pop(upload_id, None) + + if not u2: + raise HTTPException(status_code=404, detail="upload_id not found") + + meta = await sess.media_store.save_from_path( + u2.tmp_path, + store_filename=u2.store_filename, + display_name=u2.filename, + ) + + async with sess.media_lock: + sess.load_media[meta.id] = meta + sess.pending_media_ids.append(meta.id) + + return JSONResponse({ + "media": sess.public_media(meta), + "pending_media": sess.public_pending_media(), + }) + finally: + try: + UPLOAD_SEM.release() + except Exception: + pass + + +@api.post("/sessions/{session_id}/media/{upload_id}/cancel") +async def cancel_resumable_media_upload(session_id: str, upload_id: str): + store: SessionStore = app.state.sessions + sess = await store.get_or_404(session_id) + + async with sess.media_lock: + u = sess.resumable_uploads.pop(upload_id, None) + + if not u: + return JSONResponse({"ok": True}) + + async with u.lock: + u.closed = True + try: + if u.tmp_path and os.path.exists(u.tmp_path): + os.remove(u.tmp_path) + except Exception: + pass + + return JSONResponse({"ok": True}) + +@api.get("/sessions/{session_id}/media/pending") +async def get_pending_media(session_id: str): + store: SessionStore = app.state.sessions + sess = await store.get_or_404(session_id) + return JSONResponse({"pending_media": sess.public_pending_media()}) + + +@api.delete("/sessions/{session_id}/media/pending/{media_id}") +async def delete_pending_media(session_id: str, media_id: str): + store: SessionStore = app.state.sessions + sess = await store.get_or_404(session_id) + await sess.delete_pending_media(media_id) + return JSONResponse({"ok": True, "pending_media": sess.public_pending_media()}) + + +@api.get("/sessions/{session_id}/media/{media_id}/thumb") +async def get_media_thumb(session_id: str, media_id: str): + store: SessionStore = app.state.sessions + sess = await store.get_or_404(session_id) + + meta = sess.load_media.get(media_id) + if not meta: + raise HTTPException(status_code=404, detail="media not found") + + # thumb 存在优先 + if meta.thumb_path and os.path.exists(meta.thumb_path): + return FileResponse(meta.thumb_path, media_type="image/jpeg") + + # video 无 thumb => placeholder + if meta.kind == "video": + return Response(content=video_placeholder_svg_bytes(), media_type="image/svg+xml") + + # image thumb 失败 => 用原图 + if meta.path and os.path.exists(meta.path): + return FileResponse(meta.path, media_type=guess_media_type(meta.path)) + + raise HTTPException(status_code=404, detail="thumb not available") + + +@api.get("/sessions/{session_id}/media/{media_id}/file") +async def get_media_file(session_id: str, media_id: str): + store: SessionStore = app.state.sessions + sess = await store.get_or_404(session_id) + + meta = sess.load_media.get(media_id) + if not meta: + raise HTTPException(status_code=404, detail="media not found") + if not meta.path or (not os.path.exists(meta.path)): + raise HTTPException(status_code=404, detail="file not found") + + # 安全：只允许 media_dir 下 + if not _is_under_dir(meta.path, sess.media_store.media_dir): + raise HTTPException(status_code=403, detail="forbidden") + + return FileResponse( + meta.path, + media_type=guess_media_type(meta.path), + filename=meta.name, + ) + +@api.get("/sessions/{session_id}/preview") +async def preview_local_file(session_id: str, path: str): + """ + 把 summary.preview_urls 里的“服务器本地路径”安全地转成可访问 URL。 + 只允许访问：media_dir / outputs_dir / outputs_dir / bgm_dir / .server_cache 这些根目录下的文件。 + """ + store: SessionStore = app.state.sessions + sess = await store.get_or_404(session_id) + + p = (path or "").strip() + if not p: + raise HTTPException(status_code=400, detail="empty path") + if "\x00" in p: + raise HTTPException(status_code=400, detail="bad path") + + # 兼容 file:// 前缀（如果未来有） + if p.startswith("file://"): + p = p[len("file://"):] + + # 相对路径：默认相对 ROOT_DIR + if os.path.isabs(p): + ap = os.path.abspath(p) + else: + ap = os.path.abspath(os.path.join(ROOT_DIR, p)) + + allowed_roots = [ + os.path.abspath(sess.media_dir), + os.path.abspath(app.state.cfg.project.outputs_dir), + os.path.abspath(app.state.cfg.project.outputs_dir), + os.path.abspath(app.state.cfg.project.bgm_dir), + os.path.abspath(SERVER_CACHE_DIR), + ] + + if not any(_is_under_dir(ap, r) for r in allowed_roots): + raise HTTPException(status_code=403, detail="forbidden") + + if (not os.path.exists(ap)) or os.path.isdir(ap): + raise HTTPException(status_code=404, detail="file not found") + + # 对 cache 文件强缓存 + headers = {"Cache-Control": "public, max-age=31536000, immutable"} if _is_under_dir(ap, SERVER_CACHE_DIR) else None + + return FileResponse( + ap, + media_type=guess_media_type(ap), + filename=os.path.basename(ap), + headers=headers, + ) + +app.include_router(api) + + +# ------------------------- +# WebSocket: session-scoped chat stream +# ------------------------- +def extract_text_delta(msg_chunk: Any) -> str: + # 兼容 content_blocks (qwen3 常见) + blocks = getattr(msg_chunk, "content_blocks", None) or [] + if blocks: + out = "" + for b in blocks: + if isinstance(b, dict) and b.get("type") == "text": + out += b.get("text", "") + return out + c = getattr(msg_chunk, "content", "") + return c if isinstance(c, str) else "" + + +async def ws_send(ws: WebSocket, type_: str, data: Any = None): + if getattr(ws, "client_state", None) != WebSocketState.CONNECTED: + return False + try: + await ws.send_json({"type": type_, "data": data}) + return True + except WebSocketDisconnect: + return False + except RuntimeError: + return False + except Exception as e: + if ClientDisconnected is not None and isinstance(e, ClientDisconnected): + return False + logger.exception("ws_send failed: type=%s err=%r", type_, e) + return False + +@asynccontextmanager +async def mcp_sink_context(sink_func): + token = set_mcp_log_sink(sink_func) + try: + yield + finally: + reset_mcp_log_sink(token) + + +@app.websocket("/ws/sessions/{session_id}/chat") +async def ws_chat(ws: WebSocket, session_id: str): + client_ip = _client_ip_from_ws(ws, RATE_LIMIT_TRUST_PROXY_HEADERS) + + ok, retry_after, _ = await RATE_LIMITER.allow( + key=f"ws:connect:{client_ip}", + capacity=float(WS_CONNECT_BURST), + refill_rate=_rpm_to_rps(float(WS_CONNECT_RPM)), + cost=1.0, + ) + if not ok: + try: + await ws.close(code=1013, reason=f"rate_limited, retry after {int(math.ceil(retry_after))}s") + except Exception: + debug_traceback_print(app.state.cfg) + pass + return + + if WS_CONN_SEM.locked(): + try: + await ws.close(code=1013, reason="Server busy (websocket connections limit)") + except Exception: + debug_traceback_print(app.state.cfg) + pass + return + + await WS_CONN_SEM.acquire() + + try: + await ws.accept() + + store: SessionStore = app.state.sessions + sess = await store.get(session_id) + if not sess: + await ws.close(code=4404, reason="session not found") + return + sess = await store.get_or_404(session_id) + + await ws_send(ws, "session.snapshot", sess.snapshot()) + + try: + while True: + req = await ws.receive_json() + if not isinstance(req, dict): + continue + + t = req.get("type") + if t == "ping": + await ws_send(ws, "pong", {"ts": time.time()}) + continue + + if t == "session.set_lang": + data = (req.get("data") or {}) + lang = (data.get("lang") or "").strip().lower() + if lang not in ("zh", "en"): + lang = "zh" + + sess.lang = lang + if sess.client_context: + sess.client_context.lang = lang + + await ws_send(ws, "session.lang", {"lang": lang}) + continue + + if t == "chat.clear": + async with sess.chat_lock: + sess.sent_media_total = 0 + sess._attach_stats_msg_idx = 1 + sess.lc_messages = [ + SystemMessage(content=get_prompt("instruction.system", lang=sess.lang)), + SystemMessage(content="【User media upload status】{}"), + ] + sess._attach_stats_msg_idx = 1 + sess.history = [] + sess._tool_history_index = {} + await ws_send(ws, "chat.cleared", {"ok": True}) + continue + + if t != "chat.send": + await ws_send(ws, "error", {"message": f"unknown type: {t}"}) + continue + + # ---- WebSocket message rate limit: only limit expensive "chat.send" ---- + if sess.chat_lock.locked(): + await ws_send(ws, "error", {"message": "上一条消息尚未完成，请稍后再发送"}) + continue + + ok, retry_after, _ = await RATE_LIMITER.allow( + key="ws:chat_send:all", + capacity=float(WS_CHAT_SEND_ALL_BURST), + refill_rate=_rpm_to_rps(float(WS_CHAT_SEND_ALL_RPM)), + cost=1.0, + ) + if not ok: + await ws_send(ws, "error", { + "message": f"触发全局限流：请 {int(math.ceil(retry_after))} 秒后再试", + "retry_after": int(math.ceil(retry_after)), + }) + continue + + ok, retry_after, _ = await RATE_LIMITER.allow( + key=f"ws:chat_send:{client_ip}", + capacity=float(WS_CHAT_SEND_BURST), + refill_rate=_rpm_to_rps(float(WS_CHAT_SEND_RPM)), + cost=1.0, + ) + if not ok: + await ws_send(ws, "error", { + "message": f"触发限流：请 {int(math.ceil(retry_after))} 秒后再试", + "retry_after": int(math.ceil(retry_after)), + }) + continue + + if CHAT_TURN_SEM.locked(): + await ws_send(ws, "error", {"message": "服务器繁忙（模型并发已满），请稍后再试"}) + continue + + await CHAT_TURN_SEM.acquire() + try: + # 再次确认（期间有 await，锁状态可能变化） + if sess.chat_lock.locked(): + await ws_send(ws, "error", {"message": "上一条消息尚未完成，请稍后再发送"}) + continue + + data = (req.get("data", {}) or {}) + + prompt = data.get("text", "") + prompt = (prompt or "").strip() + if not prompt: + continue + + requested_llm = data.get("llm_model") + requested_vlm = data.get("vlm_model") + + attachment_ids = data.get("attachment_ids") + if not isinstance(attachment_ids, list): + attachment_ids = None + + async with sess.chat_lock: + # 新 turn 开始：清掉上一次残留的 cancel 信号 + sess.cancel_event.clear() + # 0.0) 应用 service_config（自定义模型 / TTS） + ok_cfg, err_cfg = sess.apply_service_config(data.get("service_config")) + if not ok_cfg: + await ws_send(ws, "error", {"message": err_cfg or "service_config invalid"}) + continue + + # 0) 如果前端传了 model，则更新会话当前对话模型 + if isinstance(requested_llm, str): + m = requested_llm.strip() + if m: + sess.chat_model_key = m + if sess.client_context: + sess.client_context.chat_model_key = m + + if isinstance(requested_vlm, str): + m2 = requested_vlm.strip() + if m2: + sess.vlm_model_key = m2 + if sess.client_context: + sess.client_context.vlm_model_key = m2 + + requested_lang = data.get("lang") + if isinstance(requested_lang, str): + lang = requested_lang.strip().lower() + if lang in ("zh", "en"): + sess.lang = lang + # 0.1) 可能需要重建 agent（比如切换到 __custom__ 或者自定义配置变化） + try: + await sess.ensure_agent() + except Exception as e: + await ws_send(ws, "error", {"message": f"{type(e).__name__}: {e}"}) + continue + + sess._ensure_system_prompt() + + if sess.client_context: + sess.client_context.lang = sess.lang + + # 1) 从 pending 里拿本次要发送的附件 + attachments = await sess.take_pending_media_for_message(attachment_ids) + attachments_public = [sess.public_media(m) for m in attachments] + + # 统计本轮和累计发送了几个素材 + turn_attached_count = len(attachments) + sess.sent_media_total = int(getattr(sess, "sent_media_total", 0)) + turn_attached_count + + stats = { + "Number of media carried in this message sent by the user": turn_attached_count, + "Total number of media sent by the user in all conversations": sess.sent_media_total, + "Total number of media in user's media library": scan_media_dir(resolve_media_dir(app.state.cfg.project.media_dir, session_id=session_id)), + } + + idx = int(getattr(sess, "_attach_stats_msg_idx", 1)) + if len(sess.lc_messages) <= idx: + while len(sess.lc_messages) <= idx: + sess.lc_messages.append(SystemMessage(content="")) + + sess.lc_messages[idx] = SystemMessage( + content="【User media upload status】The following fields are used to determine the nature of the media provided by the user: \n" + + json.dumps(stats, ensure_ascii=False) + ) + + + # 2.1 写入 history + lc context + user_msg = { + "id": uuid.uuid4().hex[:12], + "role": "user", + "content": prompt, + "attachments": attachments_public, + "ts": time.time(), + } + sess.history.append(user_msg) + sess.lc_messages.append(HumanMessage(content=prompt)) + + # if app.state.cfg.developer.developer_mode: + # print("[LLM_CTX]", session_id, sess.lc_messages) + + # 2.2 ack：让前端更新 pending + 插入 user 消息（前端也可本地先插入） + await ws_send(ws, "chat.user", { + "text": prompt, + "attachments": attachments_public, + "pending_media": sess.public_pending_media(), + "llm_model_key": sess.chat_model_key, + "vlm_model_key": sess.vlm_model_key, + }) + + # 2.3 建立“单通道事件队列”，确保 ws.send_json 不会并发冲突 + loop = asyncio.get_running_loop() + out_q: asyncio.Queue[Tuple[str, Any]] = asyncio.Queue() + + def sink(ev: Any): + # MCP interceptor 可能 emit 非 dict；这里只收 dict + if isinstance(ev, dict): + loop.call_soon_threadsafe(out_q.put_nowait, ("mcp", ev)) + + new_messages: List[BaseMessage] = [] + + async def pump_agent(): + nonlocal new_messages + try: + stream = sess.agent.astream( + {"messages": sess.lc_messages}, + context=sess.client_context, + stream_mode=["messages", "updates"], + ) + async for mode, chunk in stream: + if mode == "messages": + msg_chunk, meta = chunk + if meta.get("langgraph_node") == "model": + delta = extract_text_delta(msg_chunk) + if delta: + await out_q.put(("assistant.delta", delta)) + + elif mode == "updates": + if isinstance(chunk, dict): + for _step, data in chunk.items(): + msgs = (data or {}).get("messages") or [] + new_messages.extend(msgs) + + await out_q.put(("agent.done", None)) + except asyncio.CancelledError: + # 被用户打断 / 连接关闭导致的取消，不属于“真正异常” + # 不要发 agent.error；给主循环一个 cancelled 信号即可 + try: + out_q.put_nowait(("agent.cancelled", None)) + except Exception: + debug_traceback_print(app.state.cfg) + pass + raise # 让任务保持 cancelled 状态，finally 里 await 时会抛 CancelledError + + except Exception as e: + # 关键：异常也要让主循环“可结束”，否则 UI 卡死 + await out_q.put(("agent.error", f"{type(e).__name__}: {e}")) + + + async def safe_send(type_: str, data: Any = None) -> bool: + try: + await ws_send(ws, type_, data) + return True + except WebSocketDisconnect: + return False + except RuntimeError as e: + # starlette: Cannot call "send" once a close message has been sent. + if 'Cannot call "send" once a close message has been sent.' in str(e): + return False + raise + except Exception as e: + # uvicorn: ClientDisconnected（不同版本类路径不稳定，用类名兜底） + if e.__class__.__name__ == "ClientDisconnected": + return False + raise + # turn 开始（前端可禁用发送按钮/显示占位） + if not await ws_send(ws, "assistant.start", {}): + return + + # 当前 assistant 分段缓冲：用于在 tool_start 到来前“封口” + seg_text = "" + seg_ts: Optional[float] = None + + async def flush_segment(send_flush_event: bool): + """ + - send_flush_event=True：告诉前端立刻结束当前 assistant 气泡（不结束整个 turn） + - 若 seg_text 有内容：写入 history（用于刷新/回放） + """ + nonlocal seg_text, seg_ts + + if send_flush_event: + if not await ws_send(ws, "assistant.flush", {}): + return + + text = (seg_text or "").strip() + if text: + sess.history.append({ + "id": uuid.uuid4().hex[:12], + "role": "assistant", + "content": text, + "ts": seg_ts or time.time(), + }) + + seg_text = "" + seg_ts = None + + pump_task: Optional[asyncio.Task] = None + + # helper: 从 AIMessage 提取 tool_call_id（兼容不同 provider 的结构） + def _tool_call_ids_from_ai_message(m: BaseMessage) -> set[str]: + ids: set[str] = set() + + tc = getattr(m, "tool_calls", None) or [] + for c in tc: + _id = None + if isinstance(c, dict): + _id = c.get("id") or c.get("tool_call_id") + else: + _id = getattr(c, "id", None) or getattr(c, "tool_call_id", None) + if _id: + ids.add(str(_id)) + + ak = getattr(m, "additional_kwargs", None) or {} + tc2 = ak.get("tool_calls") or [] + for c in tc2: + if isinstance(c, dict): + _id = c.get("id") or c.get("tool_call_id") + if _id: + ids.add(str(_id)) + + return ids + + # helper: new_messages 里有哪些 tool_call_id + def _tool_call_ids_in_msgs(msgs: List[BaseMessage]) -> set[str]: + ids: set[str] = set() + for m in msgs: + if isinstance(m, AIMessage): + ids |= _tool_call_ids_from_ai_message(m) + return ids + + # helper: new_messages 里哪些 tool_call_id 已经有 ToolMessage 结果了 + def _tool_result_ids_in_msgs(msgs: List[BaseMessage]) -> set[str]: + ids: set[str] = set() + for m in msgs: + if isinstance(m, ToolMessage): + tcid = getattr(m, "tool_call_id", None) + if tcid: + ids.add(str(tcid)) + return ids + + # helper: 把“已存在的 ToolMessage”强制替换成 cancelled（避免工具其实返回了但用户打断没看到，导致上下文和 UI 不一致） + def _force_cancelled_tool_results(msgs: List[BaseMessage], cancel_ids: set[str]) -> List[BaseMessage]: + if not cancel_ids: + return msgs + cancelled_content = json.dumps({"cancelled": True}, ensure_ascii=False) + out: List[BaseMessage] = [] + for m in msgs: + if isinstance(m, ToolMessage): + tcid = getattr(m, "tool_call_id", None) + if tcid and str(tcid) in cancel_ids: + out.append(ToolMessage(content=cancelled_content, tool_call_id=str(tcid))) + continue + out.append(m) + return out + + def _inject_cancelled_tool_messages(msgs: List[BaseMessage], tool_call_ids: List[str]) -> List[BaseMessage]: + if not tool_call_ids: + return msgs + + out = list(msgs) + + existing = set() + for m in out: + if isinstance(m, ToolMessage): + tcid = getattr(m, "tool_call_id", None) + if tcid: + existing.add(str(tcid)) + + cancelled_content = json.dumps({"cancelled": True}, ensure_ascii=False) + + for tcid in tool_call_ids: + tcid = str(tcid) + if tcid in existing: + continue + + insert_at = None + for i in range(len(out) - 1, -1, -1): + m = out[i] + if isinstance(m, AIMessage) and (tcid in _tool_call_ids_from_ai_message(m)): + insert_at = i + 1 + break + + if insert_at is None: + continue + + out.insert(insert_at, ToolMessage(content=cancelled_content, tool_call_id=tcid)) + existing.add(tcid) + + return out + + def _sanitize_new_messages_on_cancel( + new_messages: List[BaseMessage], + *, + interrupted_text: str, + cancelled_tool_ids_from_ui: List[str], + ) -> List[BaseMessage]: + """ + 返回：应该写回 sess.lc_messages 的消息序列（只包含“用户可见/认可”的那部分） + - 工具：对未返回的 tool_call 补 ToolMessage({"cancelled": true}) + - 回复：用 interrupted_text 替换末尾 final AIMessage，避免把完整回复泄漏进上下文 + """ + msgs = list(new_messages or []) + interrupted_text = (interrupted_text or "").strip() + + # 1) 工具：找出“AI 发起了 tool_call 但没有 ToolMessage 结果”的那些 id + ai_tool_ids = _tool_call_ids_in_msgs(msgs) + tool_result_ids = _tool_result_ids_in_msgs(msgs) + pending_tool_ids = ai_tool_ids - tool_result_ids + + # UI 认为被取消的 tool（running -> cancelled） + ui_cancel_ids = {str(x) for x in (cancelled_tool_ids_from_ui or [])} + + # 统一要取消的集合： + # - UI 侧 running 的（用户按下打断时看见的） + # - 以及 messages 里缺结果的（防止漏标） + cancel_ids = set(ui_cancel_ids) | set(pending_tool_ids) + + # 2) 如果 new_messages 里已经有 ToolMessage(真实结果) 但用户打断了， + # 为了“UI/上下文一致”，强制替换成 cancelled + msgs = _force_cancelled_tool_results(msgs, cancel_ids) + + # 3) 注入缺失的 ToolMessage(cancelled) + msgs = _inject_cancelled_tool_messages(msgs, list(cancel_ids)) + + # 4) 处理 assistant 最终文本（避免把完整 answer 写回） + # - 如果 interrupted_text 非空：用它替换最后一个“非 tool_call 的 AIMessage” + # - 如果 interrupted_text 为空：只在“末尾存在一个 non-toolcall AIMessage（且它后面没有 tool_call）”时移除它 + def _is_toolcall_ai(m: BaseMessage) -> bool: + return isinstance(m, AIMessage) and bool(_tool_call_ids_from_ai_message(m)) + + def _is_text_ai(m: BaseMessage) -> bool: + if not isinstance(m, AIMessage): + return False + if _tool_call_ids_from_ai_message(m): + return False + c = getattr(m, "content", None) + return isinstance(c, str) and bool(c.strip()) + + # 找最后一个“文本 AIMessage（非 tool_call）” + last_text_ai_idx = None + for i in range(len(msgs) - 1, -1, -1): + if _is_text_ai(msgs[i]): + last_text_ai_idx = i + break + + if interrupted_text: + if last_text_ai_idx is None: + msgs.append(AIMessage(content=interrupted_text)) + else: + # 用用户看见的部分替换，且丢弃后面所有消息（防止泄漏） + msgs = msgs[:last_text_ai_idx] + [AIMessage(content=interrupted_text)] + return msgs + + # interrupted_text 为空：用户没看见任何本段 token + # 只移除“末尾的 final answer AIMessage”，避免把 unseen answer 写进上下文； + # 但如果该 AIMessage 后面还有 tool_call（说明它是 pre-tool 文本），就不要删 + if last_text_ai_idx is not None: + has_toolcall_after = any(_is_toolcall_ai(m) for m in msgs[last_text_ai_idx + 1 :]) + if not has_toolcall_after: + msgs = msgs[:last_text_ai_idx] + + return msgs + + pump_task: Optional[asyncio.Task] = None + cancel_wait_task: Optional[asyncio.Task] = None + + was_interrupted = False # 本 turn 是否已经走了“打断收尾” + + try: + async with mcp_sink_context(sink): + pump_task = asyncio.create_task(pump_agent()) + cancel_wait_task = asyncio.create_task(sess.cancel_event.wait()) + + while True: + # 同时等：queue 出事件 或 cancel_event + get_task = asyncio.create_task(out_q.get()) + done, _ = await asyncio.wait( + {get_task, cancel_wait_task}, + return_when=asyncio.FIRST_COMPLETED, + ) + + # 优先处理队列事件（避免 done/flush 已经在队列里时被 cancel 抢占） + if get_task in done: + kind, payload = get_task.result() + else: + # cancel_event 触发：不再等 queue + try: + get_task.cancel() + await get_task + except asyncio.CancelledError: + debug_traceback_print(app.state.cfg) + pass + except Exception: + debug_traceback_print(app.state.cfg) + pass + + kind, payload = ("agent.cancelled", None) + + # ------------------------ + # 1) 处理打断 + # ------------------------ + if kind == "agent.cancelled": + # 防止重复触发（cancel_event + pump_agent cancelled 都可能来一次） + if was_interrupted: + break + was_interrupted = True + # 1.1 cancel agent 流（停止继续产出 token/工具） + if pump_task and (not pump_task.done()): + pump_task.cancel() + + # 1.2 将所有 running 的工具卡片标记为 error + cancelled_tool_recs: List[Dict[str, Any]] = [] + for tcid, idx in list(sess._tool_history_index.items()): + rec = sess.history[idx] + if rec.get("role") == "tool" and rec.get("state") == "running": + rec.update({ + "state": "error", + "progress": 1.0, + "message": "Cancelled by user", + "summary": {"cancelled": True}, + }) + cancelled_tool_recs.append(rec) + + # 推送 tool.end，确保前端停止 spinner + for rec in cancelled_tool_recs: + await ws_send(ws, "tool.end", { + "tool_call_id": rec["tool_call_id"], + "server": rec["server"], + "name": rec["name"], + "is_error": True, + "summary": rec.get("summary"), + }) + # 1.3 把已输出的 seg_text 写入 history（UI 看到的内容） + interrupted_text = (seg_text or "").strip() + if interrupted_text: + sess.history.append({ + "id": uuid.uuid4().hex[:12], + "role": "assistant", + "content": interrupted_text, + "ts": seg_ts or time.time(), + }) + + # 1.4 上下文：只写回“用户真实看到/认可”的消息序列 + cancelled_tool_ids = [rec["tool_call_id"] for rec in cancelled_tool_recs] + + commit_msgs = _sanitize_new_messages_on_cancel( + new_messages, + interrupted_text=interrupted_text, + cancelled_tool_ids_from_ui=cancelled_tool_ids, + ) + + if commit_msgs: + sess.lc_messages.extend(commit_msgs) + elif interrupted_text: + # 极端情况：updates 没来得及给任何消息，但用户已看到 token + sess.lc_messages.append(AIMessage(content=interrupted_text)) + + + # ★打断：只发 assistant.end，带 interrupted=true + await ws_send(ws, "assistant.end", {"text": interrupted_text, "interrupted": True}) + + sess.cancel_event.clear() + break + + # ------------------------ + # 2) 事件处理 + # ------------------------ + if kind == "assistant.delta": + delta = payload or "" + if delta: + if seg_ts is None: + seg_ts = time.time() + seg_text += delta + if not await ws_send(ws, "assistant.delta", {"delta": delta}): + raise WebSocketDisconnect() + continue + + if kind == "mcp": + raw = payload + + if raw.get("type") == "tool_start": + await flush_segment(send_flush_event=True) + + rec = sess.apply_tool_event(raw) + if rec: + if raw["type"] == "tool_start": + await ws_send(ws, "tool.start", { + "tool_call_id": rec["tool_call_id"], + "server": rec["server"], + "name": rec["name"], + "args": rec["args"], + }) + elif raw["type"] == "tool_progress": + await ws_send(ws, "tool.progress", { + "tool_call_id": rec["tool_call_id"], + "server": rec["server"], + "name": rec["name"], + "progress": rec["progress"], + "message": rec["message"], + }) + elif raw["type"] == "tool_end": + await ws_send(ws, "tool.end", { + "tool_call_id": rec["tool_call_id"], + "server": rec["server"], + "name": rec["name"], + "is_error": rec["state"] == "error", + "summary": rec["summary"], + }) + continue + + if kind == "agent.done": + final_text = (seg_text or "").strip() + + if final_text: + sess.history.append({ + "id": uuid.uuid4().hex[:12], + "role": "assistant", + "content": final_text, + "ts": seg_ts or time.time(), + }) + + if new_messages: + sess.lc_messages.extend(new_messages) + + if not await ws_send(ws, "assistant.end", {"text": final_text}): + return + break + + if kind == "agent.error": + err_text = str(payload or "unknown error") + partial = (seg_text or "").strip() + + # 把已输出部分落盘/落上下文（避免丢上下文） + if partial: + sess.history.append({ + "id": uuid.uuid4().hex[:12], + "role": "assistant", + "content": partial, + "ts": seg_ts or time.time(), + }) + sess.lc_messages.append(AIMessage(content=partial)) + + if new_messages: + sess.lc_messages.extend(new_messages) + + # ★ 真异常：只发 error（并带 partial_text 让前端结束当前气泡） + await ws_send(ws, "error", {"message": err_text, "partial_text": partial}) + break + + except WebSocketDisconnect: + return + except asyncio.CancelledError: + # 连接关闭/任务取消：不当作 error + return + except Exception as e: + # 如果已经走了打断收尾，别再发 error（避免“打断=报错”） + if was_interrupted: + return + await ws_send(ws, "error", {"message": f"{type(e).__name__}: {e}", "partial_text": (seg_text or "").strip()}) + return + finally: + # 结束 cancel_wait_task + if cancel_wait_task and (not cancel_wait_task.done()): + cancel_wait_task.cancel() + + # pump_task 取消/收尾：避免 await 卡死，加一个短超时保护 + if pump_task and (not pump_task.done()): + pump_task.cancel() + if pump_task: + try: + await asyncio.wait_for(pump_task, timeout=2.0) + except asyncio.TimeoutError: + debug_traceback_print(app.state.cfg) + pass + except asyncio.CancelledError: + debug_traceback_print(app.state.cfg) + pass + except Exception: + debug_traceback_print(app.state.cfg) + pass + finally: + try: + CHAT_TURN_SEM.release() + except Exception: + debug_traceback_print(app.state.cfg) + pass + + except WebSocketDisconnect: + return + finally: + try: + WS_CONN_SEM.release() + except: + pass diff --git a/build_env.sh b/build_env.sh new file mode 100644 index 0000000000000000000000000000000000000000..422d5a4abc6415ca4707470c6bf3c819433e902c --- /dev/null +++ b/build_env.sh @@ -0,0 +1,214 @@ +#!/bin/bash + +# 颜色定义 | Color Definitions +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# 打印带颜色的消息 | Print colored messages +print_success() { + echo -e "${GREEN}[✓]${NC} $1" +} + +print_error() { + echo -e "${RED}[✗]${NC} $1" +} + +print_warning() { + echo -e "${YELLOW}[!]${NC} $1" +} + +print_info() { + echo -e "${BLUE}[i]${NC} $1" +} + +# 打印标题 | Print Title +echo "" +echo "╔════════════════════════════════════════════════════════════════╗" +echo "║ Storyline 项目依赖安装脚本 | Dependency Installation ║" +echo "║ 使用 conda activate storyline 激活环境后运行 ║" +echo "╚════════════════════════════════════════════════════════════════╝" +echo "" + +# ========================================== +# 步骤 0: 检测操作系统 +# Step 0: Detect OS +# ========================================== +print_info "检测操作系统... | Detecting OS..." + +if [[ "$OSTYPE" == "darwin"* ]]; then + IS_MACOS=true + IS_LINUX=false + print_success "检测到 MacOS 系统 | MacOS detected" +elif [[ "$OSTYPE" == "linux-gnu"* ]]; then + IS_MACOS=false + IS_LINUX=true + print_success "检测到 Linux 系统 | Linux detected" +else + print_error "不支持的操作系统 | Unsupported operating system: $OSTYPE" + exit 1 +fi +echo "" + +# ========================================== +# 步骤 1: 检查 conda 环境 +# Step 1: Check conda environment +# ========================================== +echo "[1/4] 检查 conda 环境... | Checking conda environment..." + +if [ -z "$CONDA_DEFAULT_ENV" ]; then + print_error "未检测到 conda 环境 | No conda environment detected" + echo "" + echo "请先运行: conda activate storyline" + echo "Please run: conda activate storyline" + exit 1 +fi + +if [ "$CONDA_DEFAULT_ENV" != "storyline" ]; then + print_warning "当前环境: $CONDA_DEFAULT_ENV" + echo "" + read -p "建议使用 storyline 环境，是否继续? | Continue anyway? (y/n) " -n 1 -r + echo "" + if [[ ! $REPLY =~ ^[Yy]$ ]]; then + echo "请运行: conda activate storyline" + exit 1 + fi +else + print_success "当前环境: storyline" +fi + +# 显示 Python 信息 +print_info "Python 信息 | Python Info:" +echo " 版本 | Version: $(python --version 2>&1)" +echo " 路径 | Path: $(which python)" +echo "" + +# ========================================== +# 步骤 2: 检查 FFmpeg +# Step 2: Check FFmpeg +# ========================================== +echo "[2/4] 检查 FFmpeg... | Checking FFmpeg..." + +if ! command -v ffmpeg &> /dev/null; then + print_warning "未检测到 FFmpeg | FFmpeg not detected" + echo "" + + read -p "是否安装 FFmpeg? | Install FFmpeg? (y/n) " -n 1 -r + echo "" + + if [[ $REPLY =~ ^[Yy]$ ]]; then + print_info "正在安装 FFmpeg... | Installing FFmpeg..." + + if [ "$IS_MACOS" = true ]; then + if ! command -v brew &> /dev/null; then + print_error "需要 Homebrew 来安装 FFmpeg | Homebrew required to install FFmpeg" + echo "请访问: https://brew.sh" + exit 1 + fi + brew install ffmpeg + elif [ "$IS_LINUX" = true ]; then + if command -v apt-get &> /dev/null; then + sudo apt-get update + sudo apt-get install -y ffmpeg + elif command -v yum &> /dev/null; then + sudo yum install -y epel-release + sudo yum install -y ffmpeg ffmpeg-devel + else + print_error "无法识别的包管理器 | Unrecognized package manager" + exit 1 + fi + fi + + if [ $? -eq 0 ]; then + print_success "FFmpeg 安装成功 | FFmpeg installed successfully" + else + print_error "FFmpeg 安装失败 | FFmpeg installation failed" + exit 1 + fi + else + print_warning "跳过 FFmpeg 安装（可能影响音视频处理功能）" + print_warning "Skipping FFmpeg (may affect audio/video features)" + fi +else + print_success "FFmpeg 已安装 | FFmpeg installed" + echo " 版本 | Version: $(ffmpeg -version 2>&1 | head -n 1)" +fi +echo "" + +# ========================================== +# 步骤 3: 下载项目资源 +# Step 3: Download project resources +# ========================================== +echo "[3/4] 下载项目资源... | Downloading project resources..." + +if [ -f "download.sh" ]; then + print_info "执行资源下载脚本... | Running download script..." + chmod +x download.sh + ./download.sh + + if [ $? -eq 0 ]; then + print_success "资源下载完成 | Resources downloaded successfully" + else + print_error "资源下载失败 | Resource download failed" + exit 1 + fi +else + print_warning "未找到 download.sh | download.sh not found" + echo "如需下载模型等资源，请手动执行 download.sh" + echo "To download models, please run download.sh manually" +fi +echo "" + +# ========================================== +# 步骤 4: 安装 Python 依赖 +# Step 4: Install Python dependencies +# ========================================== +echo "[4/4] 安装 Python 依赖... | Installing Python dependencies..." + +if [ ! -f "requirements.txt" ]; then + print_error "未找到 requirements.txt | requirements.txt not found" + exit 1 +fi + +print_info "正在安装依赖包，请稍候... | Installing packages, please wait..." +echo "" + +# 安装依赖 +print_info "安装依赖包... | Installing dependencies..." + +# 尝试使用清华镜像源 +pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple + +if [ $? -ne 0 ]; then + print_warning "清华镜像安装失败，尝试使用默认源... | Tsinghua mirror failed, trying default..." + pip install -r requirements.txt + + if [ $? -ne 0 ]; then + print_error "依赖安装失败 | Dependency installation failed" + echo "" + echo "请尝试手动安装: pip install -r requirements.txt" + exit 1 + fi +fi + +print_success "依赖安装完成 | Dependencies installed successfully" +echo "" + +# ========================================== +# 安装完成 | Installation Complete +# ========================================== +echo "" +echo "╔════════════════════════════════════════════════════════════════╗" +echo "║ 安装成功！| Installation Successful! ║" +echo "╚════════════════════════════════════════════════════════════════╝" +echo "" + +print_info "环境信息 | Environment Info:" +echo " Conda 环境 | Conda Env: $CONDA_DEFAULT_ENV" +echo " Python: $(python --version 2>&1)" +command -v ffmpeg &> /dev/null && echo " FFmpeg: $(ffmpeg -version 2>&1 | head -n 1 | cut -d' ' -f3)" +echo "" + +print_success "现在可以运行项目了！| You can now run the project!" diff --git a/cli.py b/cli.py new file mode 100644 index 0000000000000000000000000000000000000000..e8a50bf63af544791fbfb2bf3d27f3e2b5bd22b5 --- /dev/null +++ b/cli.py @@ -0,0 +1,99 @@ +import asyncio +import time +import uuid +import os,sys +import json + +from typing import List + +from langchain_core.messages import AIMessage, BaseMessage, HumanMessage, SystemMessage + +# Add src directory to Python module search path +ROOT_DIR = os.path.dirname(__file__) +SRC_DIR = os.path.join(ROOT_DIR, "src") + +if SRC_DIR not in sys.path: + sys.path.insert(0, SRC_DIR) + +from open_storyline.agent import ClientContext, build_agent +from open_storyline.utils.prompts import get_prompt +from open_storyline.utils.media_handler import scan_media_dir +from open_storyline.config import load_settings, default_config_path +from open_storyline.storage.agent_memory import ArtifactStore +from open_storyline.mcp.hooks.node_interceptors import ToolInterceptor +from open_storyline.mcp.hooks.chat_middleware import PrintStreamingTokens + +_MEDIA_STATS_INFO_IDX = 1 + +async def main(): + session_id = f"run_{int(time.time())}_{uuid.uuid4().hex[:8]}" + cfg = load_settings(default_config_path()) + + artifact_store = ArtifactStore(cfg.project.outputs_dir, session_id=session_id) + agent, node_manager = await build_agent(cfg=cfg, session_id=session_id, store=artifact_store, tool_interceptors=[ToolInterceptor.inject_media_content_before, ToolInterceptor.save_media_content_after, ToolInterceptor.inject_tts_config]) + + context = ClientContext( + cfg=cfg, + session_id=session_id, + media_dir=cfg.project.media_dir, + bgm_dir=cfg.project.bgm_dir, + outputs_dir=cfg.project.outputs_dir, + node_manager=node_manager, + chat_model_key=cfg.llm.model, + ) + + messages: List[BaseMessage] = [ + SystemMessage(content=get_prompt("instruction.system", lang='en')), + SystemMessage(content="【User media statistics】{}"), + ] + + print("Smart Editing Agent v 1.0.0") + print("Please describe your editing needs, type /exit to exit.") + + while True: + try: + user_input = input("You: ").strip() + except (EOFError, KeyboardInterrupt): + print("\nGoodBye~") + break + + if not user_input: + continue + if user_input in ("/exit", "/quit"): + print("\nGoodBye~") + break + + media_stats = scan_media_dir(context.media_dir) + messages[_MEDIA_STATS_INFO_IDX] = SystemMessage( + content=( + f"【User media statistics】{json.dumps(media_stats, ensure_ascii=False)}" + ) + ) + + messages.append(HumanMessage(content=user_input)) + + print("Agent: ", end="", flush=True) + + stream = PrintStreamingTokens() + + result = await agent.ainvoke( + {"messages": messages}, + context=context, + config={"callbacks": [stream]}, + ) + + print("\n") + + messages = result["messages"] + + final_text = None + for m in reversed(messages): + if isinstance(m, AIMessage): + final_text = m.content + break + + print(f"\nAgent: {final_text or '(No final response generated)'}\n") + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/config.toml b/config.toml new file mode 100644 index 0000000000000000000000000000000000000000..04196aa5bf1b58c5633210b45bb63fb83776de40 --- /dev/null +++ b/config.toml @@ -0,0 +1,157 @@ +# ============= 开发者选项 / Developer Options =============== +[developer] +developer_mode = false +default_llm = "deepseek-chat" +default_vlm = "qwen3-vl-8b-instruct" +print_context = false # 在拦截器打印模型拿到的全部上下文，会很长 / Print full context in interceptor (output will be very long) + +# ============= 模型配置 for 体验网页 =============== +[developer.chat_models_config."deepseek-chat"] +base_url = "" +api_key = "" +temperature = 0.1 + +[developer.chat_models_config."qwen3-vl-8b-instruct"] +base_url = "" +api_key = "" +timeout = 20.0 +temperature = 0.1 +max_retries = 2 + +# ============= 项目路径 / Project Paths ====================== +[project] +media_dir = "./outputs/media" +bgm_dir = "./resource/bgms" +outputs_dir = "./outputs" + +# ============= 模型配置 for user / Model Config for User ============= +[llm] +model = "deepseek-chat" +base_url = "" +api_key = "" +timeout = 30.0 # 单位：秒 +temperature = 0.1 +max_retries = 2 + +[vlm] +model = "qwen3-vl-8b-instruct" +base_url = "" +api_key = "" +timeout = 20.0 # 单位：秒 +temperature = 0.1 +max_retries = 2 + + +# ============= MCP Server 相关 / MCP Server Related ============= +[local_mcp_server] +server_name = "storyline" +server_cache_dir = ".storyline/.server_cache" +server_transport = "streamable-http" # server 和 host之间的传输方式 / Transport method between server and host +url_scheme = "http" +connect_host = "127.0.0.1" # 不要改动 / Do not change +port = 8001 # 如果端口冲突，可以随便用一个有空的端口 / Use any available port if conflict occurs +path = "/mcp" # 默认值，一般不用改 / Default value, usually unchanged + +json_response = true # 建议用 True / Recommended: True +stateless_http = false # 强烈建议用 False / Strongly recommended: False +timeout = 600 +available_node_pkgs = [ + "open_storyline.nodes.core_nodes" +] +available_nodes = [ + "LoadMediaNode", "SearchMediaNode", "SplitShotsNode", + "UnderstandClipsNode", "FilterClipsNode", "GroupClipsNode", "GenerateScriptNode", "ScriptTemplateRecomendation", + "GenerateVoiceoverNode", "SelectBGMNode", "RecommendTransitionNode", "RecommendTextNode", + "PlanTimelineProNode", "RenderVideoNode" +] + +# =========== skills ========== +[skills] +skill_dir = "./.storyline/skills" + +# =========== pexels ========== +[search_media] +pexels_api_key = "" + +# ============= 镜头分割 / Shot Segmentation ============= +[split_shots] +transnet_weights = ".storyline/models/transnetv2-pytorch-weights.pth" +transnet_device = "cpu" + +# ============= 视频视觉理解 / Video Visual Understanding ============= +[understand_clips] +sample_fps = 2.0 # 每秒抽几帧 / Frames sampled per second +max_frames = 64 # 单clip抽帧上限兜底，避免长视频爆 token / Max frames per clip limit to prevent token overflow + +# ============= 文案模板 / Script Templates ============= +[script_template] +script_template_dir = "./resource/script_templates" +script_template_info_path = "./resource/script_templates/meta.json" + +# ============= 配音生成 / Voiceover Generation =================== +[generate_voiceover] +tts_provider_params_path = "./resource/tts/tts_providers.json" + +[generate_voiceover.providers.302] +base_url = "" +api_key = "" + +[generate_voiceover.providers.bytedance] +uid = "" +appid = "" +access_token = "" + +[generate_voiceover.providers.minimax] +base_url = "" +api_key = "" + + +# ============= BGM选择 / BGM Selection ==================== +# 主要是用于计算音乐特征的一些参数 / Mainly parameters for calculating music features +[select_bgm] +sample_rate = 22050 +hop_length = 2048 # 每次分析窗口向前跳多少个采样点，越小越精细（但更慢） / Hop length samples; smaller = more precise but slower +frame_length = 2048 # 计算信号的均方根RMS的窗口大小。越大越稳定，但对瞬态不敏感 / Window size for RMS; larger = stable but less sensitive to transients + +# ============= 字体推荐 / Font Recommendation ==================== +[recommend_text] +font_info_path = "resource/fonts/font_info.json" + +# ============= 时间线组织 / Timeline Organization ==================== +[plan_timeline] +beat_type_max = 1 # 使用多强的鼓点，例如4/4中，鼓点类似1,2,1,3，其中1是最强的，3最弱 / Beat strength (e.g., in 4/4: 1=strongest, 3=weakest) +title_duration = 0 # 片头时长 (ms) / Intro duration (ms) +bgm_loop = true # 是否允许 bgm 循环 / Allow BGM loop +min_clip_duration = 1000 + +estimate_text_min = 1500 # 在没有TTS的情况下，估计每段字幕至少上屏多久 / Min on-screen duration for subtitles without TTS +estimate_text_char_per_sec = 6.0 # 在没有TTS的情况下，估计每秒展示几个字 / Estimated characters per second without TTS + +image_default_duration = 3000 # 默认的图片播放时长 / Default image duration +group_margin_over_voiceover = 1000 # 在一个group中，画面比配音多出现多久 / Extra visual duration over voiceover in a group + +[plan_timeline_pro] + +min_single_text_duration = 200 # 单段文字最小时长 (ms) / min single text duration (ms) +max_text_duration = 5000 # 单句文字最大时长 (ms) / max text sentence duration (ms) +img_default_duration = 1500 # 默认图片时长 (ms) / default image duration (ms) + +min_group_margin = 1500 # 段落/组最小间距 (ms) / min paragraph/group margin (ms) +max_group_margin = 2000 # 段落/组最大间距 (ms) / max paragraph/group margin (ms) + +min_clip_duration = 1000 # 最小片段时长 (ms) / min clip duration (ms) + +tts_margin_mode = "random" # random | avg | max | min +min_tts_margin = 300 # 最小 TTS 间隔 (ms) / min TTS gap (ms) +max_tts_margin = 400 # 最大 TTS 间隔 (ms) / max TTS gap (ms) + +text_tts_offset_mode = "random" # random | avg | max | min +min_text_tts_offset = 0 # 最小文字-TTS偏移 (ms) / min text–TTS offset (ms) +max_text_tts_offset = 0 # 最大文字-TTS偏移 (ms) / max text–TTS offset (ms) + +long_short_text_duration = 3000 # 长/短文本阈值 (ms) / long/short text threshold (ms) +long_text_margin_rate = 0.0 # 长文本起始边距率 / long text start margin rate +short_text_margin_rate = 0.0 # 短文本起始边距率 / short text start margin rate + +text_duration_mode = "with_tts" # with_tts | with_clip (随配音 | 随片段) +is_text_beats = false # 文字对齐音乐节拍 / align text with music beats \ No newline at end of file diff --git a/docs/source/en/api-key.md b/docs/source/en/api-key.md new file mode 100644 index 0000000000000000000000000000000000000000..7a7c828b5154d315c3afc3a7f70fdf444df1b5ad --- /dev/null +++ b/docs/source/en/api-key.md @@ -0,0 +1,134 @@ +# API Key Configuration Guide + +## 1. Large Language Model (LLM) + +### Using DeepSeek as an Example + +**Official Documentation**: https://api-docs.deepseek.com/zh-cn/ + +Note: For users outside China, we recommend using large language models such as Gemini, Claude, or ChatGPT for the best experience. + +### Configuration Steps + +1. **Apply for API Key** + - Visit platform: https://platform.deepseek.com/usage + - Log in and apply for API Key + - ⚠️ **Important**: Save the obtained API Key securely + +2. **Configuration Parameters** + - **Model Name**: `deepseek-chat` + - **Base URL**: `https://api.deepseek.com/v1` + - **API Key**: Fill in the Key obtained in the previous step + +3. **API Configuration** + - **Web Usage**: Select "Use Custom Model" in the LLM model form, and fill in the model according to the configuration parameters + - **Local Deployment**: In config.toml, locate `[developer.chat_models_config."deepseek-chat"]` and fill in the configuration parameters to make the default configuration accessible from the Web page. Locate `[llm]` and configure model, base_url, and api_key + +## 2. Multimodal Large Language Model (VLM) + +### 2.1 Using GLM-4.6V + +**API Key Management**: https://open.bigmodel.cn/usercenter/proj-mgmt/apikeys + +### Configuration Parameters + +- **Model Name**: `glm-4.6v` +- **Base URL**: `https://open.bigmodel.cn/api/paas/v4/` + +### 2.2 Using Qwen3-VL + +**API Key Management**: Go to Alibaba Cloud Bailian Platform to apply for an API Key https://bailian.console.aliyun.com/cn-beijing/?apiKey=1&tab=globalset#/efm/api_key + + - **Model Name**: `qwen3-vl-8b-instruct` + - **Base URL**: `https://dashscope.aliyuncs.com/compatible-mode/v1` + + - Parameter Configuration: Select "Use Custom Model" in the VLM Model form and fill in the parameters. For local deployment, locate `[vlm]` and configure model, base_url, and api_key. Add the following fields in config.toml as the default Web API configuration: + ``` + [developer.chat_models_config."qwen3-vl-8b-instruct"] + base_url = "https://dashscope.aliyuncs.com/compatible-mode/v1" + api_key = "YOUR_API_KEY" + timeout = 20.0 + temperature = 0.1 + max_retries = 2 + ``` + +### 2.3 Using Qwen3-Omni + +Qwen3-Omni can also be applied for through the Alibaba Cloud Bailian Platform. The specific parameters are as follows, which can be used for automatic labeling music in omni_bgm_label.py +- **Model Name**: `qwen3-omni-flash-2025-12-01` +- **Base URL**: `https://dashscope.aliyuncs.com/compatible-mode/v1` + +For more details, please refer to the documentation: https://bailian.console.aliyun.com/cn-beijing/?tab=doc#/doc + +Model List: https://help.aliyun.com/zh/model-studio/models + +Billing Dashboard: https://billing-cost.console.aliyun.com/home + +## 3. Pexels Image and Video Download API Key Configuration + +1. Open the Pexels website, register an account, and apply for an API key at https://www.pexels.com/api/ +
+ Pexels API application +

Figure 1: Pexels API Application Page

+
+ +2. Web Usage: Locate the Pexels configuration, select "Use custom key", and enter your API key in the form. +
+ Pexels API input +

Figure 2: Pexels API Usage

+
+ +3. Local Deployment: Fill in the API key in the `pexels_api_key` field in the `config.toml` file as the default configuration for the project. + +## 4. TTS (Text-to-Speech) Configuration + +### Option 1: 302.ai + +**Service URL**: https://302.ai/product/detail/302ai-mmaudio-text-to-speech + +### Option 2: MiniMax + +**Subscription Page**: https://platform.minimax.io/subscribe/audio-subscription + +**Configuration Steps**: +1. Create API Key +2. Visit: https://platform.minimax.io/user-center/basic-information/interface-key +3. Obtain and save API Key + +### Option 3: Bytedance +1. Step 1: Enable Audio/Video Subtitle Generation Service + Use the legacy page to find the audio/video subtitle generation service: + + - Visit: https://console.volcengine.com/speech/service/9?AppID=8782592131 + +2. Step 2: Obtain Authentication Information + View the account basic information page: + + - Visit: https://console.volcengine.com/user/basics/ + +
+ Bytedance TTS API Configuration +

Figure 3: Bytedance TTS API Usage

+
+ + You need to obtain the following information: + - **UID**: The ID from the main account information + - **APP ID**: The APP ID from the service interface authentication information + - **Access Token**: The Access Token from the service interface authentication information + + For local deployment, modify the config.toml file: + +``` +[generate_voiceover.providers.bytedance] +uid = "" +appid = "" +access_token = "" +``` + +For detailed documentation, please refer to: https://www.volcengine.com/docs/6561/80909 + +## Important Notes + +- All API Keys must be kept secure to avoid leakage +- Ensure sufficient account balance before use +- Regularly monitor API usage and costs \ No newline at end of file diff --git a/docs/source/en/faq.md b/docs/source/en/faq.md new file mode 100644 index 0000000000000000000000000000000000000000..7dee08c25d57fb604c2b634f08d2f8ba82c5f994 --- /dev/null +++ b/docs/source/en/faq.md @@ -0,0 +1,18 @@ +# Most Frequently Asked Questions + +## Environment Related Issue +
+Issue 1: When activating conda environment, script execution is prohibited. + +Please refer to this article for the solution: [https://juejin.cn/post/7349212852644954139](https://juejin.cn/post/7349212852644954139) +
+ +
+ +Issue 2: Error creating a virtual environment after installing Conda on Windows. + +- **Cause:** + This is caused by Conda not being added to the system environment variables during installation. +- **Solution:** + You need to open **Anaconda Prompt**, **Miniconda Prompt**, or **Miniforge Prompt** (depending on which one you installed) from the Start Menu, `cd` to the current directory, and then proceed to create the environment. +
\ No newline at end of file diff --git a/docs/source/en/guide.md b/docs/source/en/guide.md new file mode 100644 index 0000000000000000000000000000000000000000..559af2e4d753913ec6d09b9f5af96b19869bd5e0 --- /dev/null +++ b/docs/source/en/guide.md @@ -0,0 +1,220 @@ +# OpenStoryline User Guide + +--- + +## 0. Environment Setup + +See the [README](https://github.com/FireRedTeam/FireRed-OpenStoryline/blob/main/README.md) section. + +## 1. Basic Usage + +### 1.1 Getting Started + +You can start creating in two ways: + +1. **You have your own media** + + * Click the file upload button on the left side of the chat box and select your images/videos. + * Then type your editing goal in the input field, for example: *Use my footage to edit a family vlog with an upbeat rhythm.* + +2. **You don’t have media** + + * Just describe the theme/mood. + * For example: *Help me create a summer beach travel vlog—sunny, fresh, and cheerful.* + +Automatic asset retrieval is powered by [Pexels](https://www.pexels.com/). Please enter your Pexels API key in the website sidebar. + +**Disclaimer:** We only provide the tool. All assets downloaded or used via this tool (e.g., Pexels images) are fetched by the user through the API. We assume no responsibility for the content of videos generated by users, the legality of the assets, or any copyright/portrait-right disputes arising from the use of this tool. Please comply with Pexels’ license when using it: [https://www.pexels.com/zh-cn/license](https://www.pexels.com/zh-cn/license) +[https://www.pexels.com/terms-of-service](https://www.pexels.com/terms-of-service) + +If you just want to explore it first, you can also use it like a normal chat model, for example: + +* “Introduce yourself” + +demo + +### 1.2 Editing + +OpenStoryline supports **intent intervention and partial redo at any stage**. After a step completes, you can simply describe what you want to change in one sentence. The agent will locate the step that needs to be rerun, without restarting from the beginning. For example: + +* Remove the clip where the camera is filming the sky. +* Switch to a more upbeat background music. +* Change the subtitle color to better match the sunset theme. + +demo + +### 1.3 Style Imitation + +With the style imitation Skill, you can reproduce almost any writing style to generate copy. For example: + +* Generate copy in a Shakespearean style for me. +* Mimic the tone of my social media posts. + +demo + +### 1.4 Interrupting + +At any moment while the agent is running, if its behavior is not as expected, you can: + +* Click the **Stop** button on the right side of the input box to stop the model reply and tool calls, **or** +* Press **Enter** to send a new prompt—the system will automatically interrupt and follow your new instruction. + +Interrupting does **not** clear the current progress. Existing replies and executed tool results will be kept, and you can continue from the current state. + +### 1.5 Switching Languages + +Click the language button in the top-right corner of the page to switch between Chinese and English: + +* The sidebar and tool-call cards will switch display language accordingly. +* The prompt language used inside tools will also switch. +* Past chat history will **not** be automatically translated. + +### 1.6 Saving + +After you polish a satisfying video, you can ask the agent to **summarize the editing logic** (rhythm, color tone, transition habits, etc.) and save it as your personal **“Editing Skill.”** + +Next time you edit similar content, simply ask the agent to use this Skill to reproduce the style. + +demo + +### 1.7 Mobile Usage + +**Warning: The commands below will expose your service to your local network. Use only on trusted networks. Do NOT run these commands on public networks.** + +If your media is on your phone and it’s inconvenient to transfer, you can use the following steps to use the editing agent on mobile. + +1. Fill in the LLM/VLM/Pexels/TTS configuration in config.toml. +2. Change your web startup command to: + + ```bash + # Reminder: --host 0.0.0.0 exposes the service to your LAN/public network. Use only on trusted networks. + uvicorn agent_fastapi:app --host 0.0.0.0 --port 7860 + ``` + +3. Find your computer’s IP address: + + * **Windows:** run `ipconfig` in Command Prompt (cmd) and locate the IPv4 address + * **Mac:** hold **Option** and click the Wi-Fi icon + * **Linux:** run `ifconfig` in the terminal + +4. Then open the following address in your phone browser: + + ``` + {your_computer_ip}:7860 + ``` + +--- + +## 2. Advanced Usage + +Due to copyright and distribution constraints, open-source resources may not be sufficient for many users’ editing needs. Therefore, we provide methods to add and build private asset libraries. + +--- + +### 2.1 Custom Music Library + +Put your private music files into: + +`./resource/bgms` + +Then tag your music by writing metadata into: + +`./resouce/bgms/meta.json` + +Restart the MCP service to apply changes. + +**Tag Dimensions** + +* **scene:** Vlog, Travel, Relaxing, Emotion, Transition, Outdoor, Cafe, Evening, Scenery, Food, Date, Club +* **genre:** Pop, BGM, Electronic, R&B/Soul, Hip Hop/Rap, Rock, Jazz, Folk, Classical, Chinese Style +* **mood:** Dynamic, Chill, Happy, Sorrow, Romantic, Calm, Excited, Healing, Inspirational +* **lang:** bgm, en, zh, ko, ja + +**How to Tag** + +* **Manual tagging:** Copy the format of other items in `meta.json` and add tags accordingly. **Note:** the `description` field is required. +* **Auto tagging:** Use `qwen3-omni-flash` for automatic tagging (requires a Qwen model API key). + +Qwen3-omni labeling script: + +```bash +export QWEN_API_KEY="you_api_key" +python -m scripts.omni_bgm_label +``` + +Auto tags may not be fully accurate. If you need strong recommendations for specific scenarios, it’s recommended to manually review the results. + +--- + +### 2.2 Custom Font Library + +Put your private font files into: + +`./resource/fonts` + +Then tag the fonts by editing: + +`./resource/fonts/font_info.json` + +Restart the MCP service to apply changes. + +**Tag Dimensions** + +* **class:** Creative, Handwriting, Calligraphy, Basic +* **lang:** zh, en + +**How to Tag** +Currently only manual tagging is supported—edit `./resource/fonts/font_info.json` directly. + +--- + +### 2.3 Custom Copywriting Template Library + +Put your private copywriting templates into: + +`./resource/script_templates` + +Then tag them by writing metadata into: + +`./resource/fonts/meta.json` + +Restart the MCP service to apply changes. + +**Tag Dimensions** + +* **tags:** Life, Food, Beauty, Entertainment, Travel, Tech, Business, Vehicle, Health, Family, Pets, Knowledge + +**How to Tag** + +* **Manual tagging:** Copy the format of other items in `meta.json` and add tags accordingly. **Note:** the `description` field is required. +* **Auto tagging:** Use DeepSeek for automatic tagging (requires the corresponding API key). + +DeepSeek labeling script: + +```bash +export DEEPSEEK_API_KEY="you_api_key" +python -m scripts.llm_script_template_label +``` + +Auto tags may not be fully accurate. If you need strong recommendations for specific scenarios, it’s recommended to manually review the results. + +--- + +### 2.4 Custom Skill Library + +The repository includes two built-in Skills: one for writing-style imitation and another for saving editing workflows. If you want more custom skills, you can add them as follows: + +1. Create a new folder under `.storyline/skills`. +2. Inside that folder, create a file named `SKILL.md`. +3. The `SKILL.md` must start with: + + ```markdown + --- + name: yous_skill_folder_name + description: your_skill_function_description + --- + ``` + + The `name` must match the folder name. +4. Then write the detailed skill content (its role setting, which tools it should call, output format, etc.). +5. Restart the MCP service to apply changes. diff --git a/docs/source/zh/api-key.md b/docs/source/zh/api-key.md new file mode 100644 index 0000000000000000000000000000000000000000..40f0f0b52a99533c9bed7507208c0bb7fc82382e --- /dev/null +++ b/docs/source/zh/api-key.md @@ -0,0 +1,132 @@ +# API-Key 配置指南 + +## 一、大语言模型 (LLM) + +### 以 DeepSeek 为例 + +**官方文档**：https://api-docs.deepseek.com/zh-cn/ + +提示: 对于中国以外用户建议使用 Gemini、Claude、ChatGPT 等主流大语言模型以获得最佳体验。 + +### 配置步骤 + +1. **申请 API Key** + - 访问平台：https://platform.deepseek.com/usage + - 登录后申请 API Key + - ⚠️ **重要**：妥善保存获取的 API Key + +2. **配置参数** + - **模型名称**：`deepseek-chat` + - **Base URL**：`https://api.deepseek.com/v1` + - **API Key**：填写上一步获取的 Key + +3. **API填写** + - **Web使用**: 在LLM模型表单中选择使用自定义模型，模型按照配置参数进行填写 + - **本地部署**: 在config.toml中 找到`[developer.chat_models_config."deepseek-chat"]` 将配置参数填写上去，使得Web页面可以访问到该默认配置。 找到`[llm]`并配置model、base_url、api_key + +## 二、多模态大模型 (VLM) + +### 2.1 使用GLM-4.6V + +**API Key 管理**：https://open.bigmodel.cn/usercenter/proj-mgmt/apikeys + +### 配置参数 + +- **模型名称**：`glm-4.6v` +- **Base URL**：`https://open.bigmodel.cn/api/paas/v4/` + +### 2.2 使用Qwen3-VL + +**API Key管理**：进入阿里云百炼平台申请API Key https://bailian.console.aliyun.com/cn-beijing/?apiKey=1&tab=globalset#/efm/api_key + + - **模型名称**：`qwen3-vl-8b-instruct` + - **Base URL**：`https://dashscope.aliyuncs.com/compatible-mode/v1` + + - **参数填写**：在VLM Model表单中选择"使用自定义模型"进行参数填写。本地部署时，找到`[vlm]`并配置model、base_url、api_key，在config.toml中新增以下字段作为Web的API默认配置： + ``` + [developer.chat_models_config."qwen3-vl-8b-instruct"] + base_url = "https://dashscope.aliyuncs.com/compatible-mode/v1" + api_key = "YOUR_API_KEY" + timeout = 20.0 + temperature = 0.1 + max_retries = 2 + ``` + + +### 2.3 使用Qwen3-Omni + +Qwen3-Omni同样可以在阿里云百炼平台进行申请，具体参数如下，可用于omni_bgm_label.py的音频自动标注 +- **模型名称**：`qwen3-omni-flash-2025-12-01` +- **Base URL**：`https://dashscope.aliyuncs.com/compatible-mode/v1` + +详细文档参考：https://bailian.console.aliyun.com/cn-beijing/?tab=doc#/doc + +阿里云模型列表：https://help.aliyun.com/zh/model-studio/models + +计费看板：https://billing-cost.console.aliyun.com/home + +## 三、Pexels 图像和视频下载API密钥配置 + +1. 打开Pexels网站，注册账号，申请API https://www.pexels.com/zh-cn/api/key/ +
+ pexels下载图像和视频API申请 +

图1: Pexels API申请页面

+
+ +2. 网页使用：找到Pexels配置，选择使用自定义key，将API key填入表单中。 +
+ pexels API填写 +

图2: Pexels API 使用

+
+ +3. 本地部署的项目：我们将API填写在config.toml中的pexels_api_key字段中。作为项目的默认配置 + +## 四、TTS (文本转语音) 配置 + +### 方案一：302.ai + +**服务地址**：https://302.ai/product/detail/302ai-mmaudio-text-to-speech + +### 方案二：MiniMax + +**订阅页面**：https://platform.minimax.io/subscribe/audio-subscription + +**配置步骤**： +1. 创建 API Key +2. 访问：https://platform.minimax.io/user-center/basic-information/interface-key +3. 获取并保存 API Key + +### 方案三：bytedance +1. 步骤1：开通音视频字幕生成服务 + 使用旧版页面，找到音视频字幕生成服务： + - 访问：https://console.volcengine.com/speech/service/9?AppID=8782592131 + +2. 步骤2：获取认证信息 + 查看账号基本信息页面： + - 访问：https://console.volcengine.com/user/basics/ + +
+ Bytedance TTS API填写 +

图3: Bytedance TTS API 使用

+
+ + 需要获取以下信息： + - **UID**: 主账号信息中的 ID + - **APP ID**: 服务接口认证信息中的 APP ID + - **Access Token**: 服务接口认证信息中的 Access Token + + 本地部署使用修改config.toml中 + ``` + [generate_voiceover.providers.bytedance] + uid = "" + appid = "" + access_token = "" + ``` + +详细文档请参考：https://www.volcengine.com/docs/6561/80909?lang=zh + +## 注意事项 + +- 所有 API Key 均需妥善保管，避免泄露 +- 使用前请确认账户余额充足 +- 建议定期检查 API 调用量和费用 diff --git a/docs/source/zh/faq.md b/docs/source/zh/faq.md new file mode 100644 index 0000000000000000000000000000000000000000..bd01fd7fe2b65f4a8d607f62869b3761d3972410 --- /dev/null +++ b/docs/source/zh/faq.md @@ -0,0 +1,23 @@ +# 最常问的问题 + +## 环境相关的问题 + +
+问题 1: Conda 激活环境时发现脚本执行被禁止 + +- **参考链接**：[https://juejin.cn/post/7349212852644954139](https://juejin.cn/post/7349212852644954139) +- **解决方法**： 在 PowerShell 中输入以下命令后重试： + ```powershell + Set-ExecutionPolicy RemoteSigned -Scope CurrentUser + ``` + +
+ + +
+ +问题 2: Windows 安装 Conda 后，创建虚拟环境时报错 + +- **原因**: 这是由于安装时没有将 conda 加入到环境变量导致的。 +- **解决方法**: 需要从开始菜单打开 Anaconda Prompt / Miniconda Prompt / Miniforge Prompt，cd 到当前目录，再创建环境。 +
\ No newline at end of file diff --git a/docs/source/zh/guide.md b/docs/source/zh/guide.md new file mode 100644 index 0000000000000000000000000000000000000000..7c984d0f74f355aa2fd5d950dca1f4e276e8c705 --- /dev/null +++ b/docs/source/zh/guide.md @@ -0,0 +1,154 @@ +# OpenStoryline 使用教程 +--- +## 0. 环境安装 + +参见[README](https://github.com/FireRedTeam/FireRed-OpenStoryline/blob/main/README_zh.md)部分 + +## 1. 基础使用教程 + +### 1.1. 开始 +你可以用两种方式开始创作： + +1. 有素材 + - 点击对话框左侧文件上传按钮，选择你的图片/视频素材 + - 然后在输入框写下剪辑目标，例如：用我的素材剪一条新年全家欢 vlog，节奏轻快 + +2. 没素材 + - 直接描述主题/氛围即可 + - 例如：帮我剪一个夏日海滩旅行 vlog，阳光、清爽、欢快 + +自动素材检索来自 [Pexels](https://www.pexels.com/zh-cn/)，请在网页侧边栏填写 Pexels API Key。 + +免责声明：我们只提供工具，所有通过本工具下载和使用的素材（如 Pexels 图像）都由用户自行通过 API 获取，我们不对用户生成的视频内容、素材的合法性或因使用本工具导致的任何版权/肖像权纠纷承担责任。使用时请遵循 Pexels 的许可协议：[https://www.pexels.com/zh-cn/license](https://www.pexels.com/zh-cn/license) +[https://www.pexels.com/terms-of-service](https://www.pexels.com/terms-of-service) + +如果你只是想先了解它，也可以当作普通对话模型使用，例如： + +- “介绍一下你自己” +demo + +### 1.2. 编辑 + +OpenStoryline 支持在任意阶段进行意图干预与局部重做：当某一步骤完成后，你可以直接用一句话提出修改要求，Agent会定位到需要重跑的步骤，而无需从流程起点重新开始。例如 +- 帮我去掉那个拍摄天空的片段。 +- 换一个欢快一点的背景音乐。 +- 字幕换成更符合夕阳主题的颜色 +demo + +### 1.3. 仿写 +依靠仿写Skill复刻任意文风生成文案。例如： +- 用文言文为我进行古风混剪。 +- 模仿鲁迅风格生成文案。 +- 模仿我发朋友圈的语气。 +demo + +### 1.4. 中断 +在 Agent 执行的任意时刻，如果行为不符合预期，你可以随时： + +- 点击输入框右侧的中止按钮：停止大模型回复与工具调用 +- 或者直接按 Enter 发送新 prompt：系统会自动打断并执行你的新指令 + +中断不会清空当前进度，已生成的回复与已执行的工具结果都会保留，你可以基于现有结果继续提出指令。 + +### 1.5. 切换语言 + +在网页右上角点击语言按钮可切换中/英文： +- 侧边栏与工具调用卡片的展示语言会同步切换 +- 工具内部使用的 prompt 语言也会切换 +- 已经发生的历史对话不会自动翻译 + +### 1.6. 保存 + +当你打磨出一条满意的视频后，可以一键让 Agent 总结其中的剪辑逻辑（节奏、色调、转场习惯），并保存为你的专属 "Editing Skill"。 +下次剪辑类似内容时，只需告诉Agent调用这个 Skill，即可实现风格复刻。 +demo + +### 1.7 移动端使用 +**注意：下列命令会将你的服务暴露到局域网/公网，请仅在可信网络使用，不要在公用网络执行以下命令！！！** +如果你的素材在手机上，不方便传输，可以使用下面的步骤，在手机上使用剪辑Agent。 +1. 在 config.toml 中填写LLM/VLM/Pexels/TTS 配置 +2. 将网页启动命令改为： + ```bash + # 再次提醒： --host 0.0.0.0 命令会将服务暴露到局域网/公网。请仅在可信网络使用。 + uvicorn agent_fastapi:app --host 0.0.0.0 --port 7860 + ``` +3. 查看本机ip地址： + - Windows: 在命令提示符（cmd）中输入 ipconfig，找到 IPv4 地址 + - Mac: 按住 option，点击 WI-FI 图标 + - Linux: 在终端中输入 ifconfig 命令 + +4. 在手机浏览器中输入以下地址即可访问。 + ``` + {本机ip地址}:7860 + ``` + + +## 2. 高级使用教程 + +受限于版权和分发协议，开源的资源不足以满足广大用户的剪辑需求，因此我们提供私有元素库的添加和构建方法。 + +### 2.1. 自定义音乐库 + + +将私有音乐文件放到目录：`./resource/bgms`下，然后给音乐打标签写入`./resouce/bgms/meta.json`，重启mcp服务即可。 + +【标签维度】 +- scene（场景）：Vlog, Travel, Relaxing, Emotion, Transition, Outdoor, Cafe, Evening, Scenery, Food, Date, Club +- genre（曲风）：Pop, BGM, Electronic, R&B/Soul, Hip Hop/Rap, Rock, Jazz, Folk, Classical, Chinese Style +- mood（情绪）：Dynamic, Chill, Happy, Sorrow, Romantic, Calm, Excited, Healing, Inspirational +- lang（语言）：bgm, en, zh, ko, ja + +【打标方式】 +- 手动打标：模仿meta.json中的其他item添加对应标签即可。注意：description字段是必须的； +- 自动打标：使用qwen3-omni-flash进行自动打标，需要依赖qwen大模型的API-KEY +qwen3-omni打标脚本： +``` +export QWEN_API_KEY="you_api_key" +python -m scripts.omni_bgm_label +``` +自动打标签不一定完全准确，如果需要强推荐的场景，建议人工再check一遍。 + +### 2.2. 自定义字体库 + +将私有字体文件放到目录：`./resource/fonts`下，然后给字体打标签写入`./resource/fonts/font_info.json`，重启mcp服务即可。 + +【标签维度】 +- class（分类）：Creative, Handwriting, Calligraphy, Basic +- lang（语言）：zh, en + +【打标方式】 +目前仅支持手动打标，直接编辑`./resource/fonts/font_info.json`。 + + +### 2.3. 自定义文案模板库 + +将私有文案模板放到目录：`./resource/script_templates`下，然后给字体打标签写入`./resource/fonts/meta.json`，重启mcp服务即可。 +【标签维度】 +- tags：Life, Food, Beauty, Entertainment, Travel, Tech, Business, Vehicle, Health, Family, Pets, Knowledge + +【打标方式】 +- 手动打标：模仿meta.json中的其他item添加对应标签即可。注意：description字段是必须的； +- 自动打标：使用deepseek进行自动打标，需要依赖qwen大模型的API-KEY +deepseek打标脚本： +``` +export DEEPSEEK_API_KEY="you_api_key" +python -m scripts.llm_script_template_label +``` +自动打标签不一定完全准确，如果需要强推荐的场景，建议人工再check一遍。 + + +### 2.4. 自定义技能库 + +仓库自带两款Skills，一个用于文风仿写，另一个用于保存剪辑流程。如果用户有更多自定义的skill可以按照以下方法添加： + +在`.storyline/skills`下创建一个新的文件夹，文件夹内新建`SKILL.md`文件； +SKILL内必须以： +```markdown +--- +name: yous_skill_folder_name +description: your_skill_function_description +--- +``` +的形式开头，其中name和文件夹名字保持一致。 +接着文件内写技能的具体内容，比如它的工作设定，需要调用哪些工具，输出格式等等。 +完成后重启mcp服务即可 \ No newline at end of file diff --git a/download.sh b/download.sh new file mode 100644 index 0000000000000000000000000000000000000000..6d4863f69f5c47e2d2f578cea19dd386f4bb3781 --- /dev/null +++ b/download.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# Create required directories +mkdir -p .storyline resource + +# 1. Download models.zip to .storyline/ and extract it (keep original directory name) +wget "https://image-url-2-feature-1251524319.cos.ap-shanghai.myqcloud.com/openstoryline/models.zip" \ + -O .storyline/models.zip + +unzip -o .storyline/models.zip -d .storyline/models/ + +# Remove the original archive +rm .storyline/models.zip + + +# 2. Download resource.zip to .storyline/ and extract it into ./resource +wget "https://image-url-2-feature-1251524319.cos.ap-shanghai.myqcloud.com/openstoryline/resource.zip" \ + -O .storyline/resource.zip + +unzip -o .storyline/resource.zip -d resource + +# Remove the original archive +rm .storyline/resource.zip + +# List of filenames +files=("brand_black.png" "brand_white.png" "logo.png" "dice.png" "github.png" "node_map.png" "user_guide.png") + +# Base URL +base_url="https://image-url-2-feature-1251524319.cos.ap-shanghai.myqcloud.com/zailin/datasets/open_storyline" + +# Download each file +for f in "${files[@]}"; do + wget "$base_url/$f" -O "web/static/$f" +done \ No newline at end of file diff --git a/hf_space.sh b/hf_space.sh new file mode 100644 index 0000000000000000000000000000000000000000..a16d390ba8fd8b47b63334648cc36ac6753334b7 --- /dev/null +++ b/hf_space.sh @@ -0,0 +1,12 @@ +if git show-ref --verify --quiet refs/heads/hf-clean; then + git branch -D hf-clean + echo "Deleted existing hf-clean branch" +else + echo "hf-clean branch does not exist" +fi +git checkout --orphan hf-clean +git reset +git add . +git commit -m "Clean branch for HF push" +git push firered hf-clean:main --force +git checkout release/v1.0.0202 \ No newline at end of file diff --git a/prompts/tasks/elementrec_text/en/system.md b/prompts/tasks/elementrec_text/en/system.md new file mode 100644 index 0000000000000000000000000000000000000000..5d6efb7789b7b8174bc94e8e7424898810e767b8 --- /dev/null +++ b/prompts/tasks/elementrec_text/en/system.md @@ -0,0 +1 @@ +You are a font recommender. Based on the video subtitles and font entries (in List[Dict] format) I provide, return one and only one JSON entry that best fits the user's requirements. \ No newline at end of file diff --git a/prompts/tasks/elementrec_text/en/user.md b/prompts/tasks/elementrec_text/en/user.md new file mode 100644 index 0000000000000000000000000000000000000000..b0e30471c80d28c46bd288096ebfc0fc34aacd94 --- /dev/null +++ b/prompts/tasks/elementrec_text/en/user.md @@ -0,0 +1,8 @@ +Subtitles： +{{scripts}} + +Candidate font entries: +{{candidates}} + +User requirements： +{{user_request}} \ No newline at end of file diff --git a/prompts/tasks/elementrec_text/zh/system.md b/prompts/tasks/elementrec_text/zh/system.md new file mode 100644 index 0000000000000000000000000000000000000000..47e47bf99eb68330d8bf3719c3cadb8915ff9fdd --- /dev/null +++ b/prompts/tasks/elementrec_text/zh/system.md @@ -0,0 +1 @@ +你是一个字体推荐器，根据我提供视频字幕与字体条目（List[Dict]格式），返回最适合用户要求的**有且只有一条**json。 \ No newline at end of file diff --git a/prompts/tasks/elementrec_text/zh/user.md b/prompts/tasks/elementrec_text/zh/user.md new file mode 100644 index 0000000000000000000000000000000000000000..db1c87f70936e2a1048cf63af1bc396f6d8743f3 --- /dev/null +++ b/prompts/tasks/elementrec_text/zh/user.md @@ -0,0 +1,8 @@ +字幕： +{{scripts}} + +候选字体条目： +{{candidates}} + +用户要求： +{{user_request}} \ No newline at end of file diff --git a/prompts/tasks/filter_clips/en/system.md b/prompts/tasks/filter_clips/en/system.md new file mode 100644 index 0000000000000000000000000000000000000000..bc2028a9c29bec64ed7bdddb6b2a302943772477 --- /dev/null +++ b/prompts/tasks/filter_clips/en/system.md @@ -0,0 +1,24 @@ +# Role +You are a professional video clip selection assistant. You need to select the most suitable clips for editing from a set of footage based on visual description, aesthetic score, and duration. + +# Goal +Output a JSON result containing the list of IDs of the final retained video clips. + +# Constraints (Selection Rules – Must Be Executed in Order) + +**Step 1: Calculate "Maximum Removable Clips" (Hard Quantity Constraint)** +First, count the total number of input clips, denoted as **Total**. +1. **If Total is less than or equal to 5**: + - Do not remove any clips; all must be retained. +2. **If Total is greater than 5**: + - Ensure that the final number of retained clips is **strictly greater than** 80% of **Total**. + - *(For example: if Total is 7, 7 × 0.8 = 5.6, the number of retained clips must be greater than 5.6, i.e., at least 6, meaning a maximum of 1 clip can be removed.)* + - At the same time, the number of retained clips cannot be fewer than 5. + +**Step 2: Execute Selection (Content Quality Optimization)** +This step is only performed if Step 1 calculates that there is a “removal quota.” If Step 1 requires all clips to be retained, skip this step. +1. Review all `clip_captions` and identify groups of clips with **highly similar visual descriptions** (almost identical). +2. Within these similar clips, compare `aes_score` (aesthetic score) and `duration` (length): + - **Prioritize retention**: clips with higher aesthetic scores and moderate duration. + - **Consider removal**: clips with lower aesthetic scores, or duration too short to be usable. +3. **Note**: The number of removed clips must not exceed the “maximum removal quota” calculated in Step 1. Once the quota is used up, no further deletion is allowed, even if similar clips remain. \ No newline at end of file diff --git a/prompts/tasks/filter_clips/en/user.md b/prompts/tasks/filter_clips/en/user.md new file mode 100644 index 0000000000000000000000000000000000000000..9a0ed89b617ce828e3a9cb0a26351fba38e67db9 --- /dev/null +++ b/prompts/tasks/filter_clips/en/user.md @@ -0,0 +1,15 @@ +user request: {{user_request}} + +Based on user requirements, please determine whether to retain all of the following clips: +{{clip_captions}} + +Output format as follows: +Note: Only output the content in the following required formats. It is strictly prohibited to output any other content +```json +{ + "results": [ + {"clip_id": "clip_0001", "keep": true} + {"clip_id": "clip_0002", "keep": false} + ] +} +``` \ No newline at end of file diff --git a/prompts/tasks/filter_clips/zh/system.md b/prompts/tasks/filter_clips/zh/system.md new file mode 100644 index 0000000000000000000000000000000000000000..89d4adba177dd6ef5fecf27c7b006d886bb9cf68 --- /dev/null +++ b/prompts/tasks/filter_clips/zh/system.md @@ -0,0 +1,24 @@ +# Role +你是一名专业的视频素材筛选助手。你需要根据画面的描述、美学评分和时长，从一堆素材中挑选出最适合剪辑的片段。 + +# Goal +输出一个JSON结果，包含最终保留的视频片段ID列表。 + +# Constraints (筛选规则 - 请严格按顺序执行) + +**第一步：计算“最多能删几个” (硬性数量指标)** +请先统计输入片段的总数量，记为【总数】。 +1. **如果【总数】少于或等于 5 个**： + - 禁止删除任何片段，必须全部保留。 +2. **如果【总数】大于 5 个**： + - 你必须保证最终保留的片段数量 **严格大于** 【总数】的 80%。 + - *（例如：总数是7个，7 x 0.8 = 5.6，保留数量必须大于5.6，即至少保留6个，意味着最多只能删 1 个）。* + - 同时，保留的数量也不能少于 5 个。 + +**第二步：执行筛选 (内容质量优化)** +只有在第一步计算出“有删除名额”的情况下，才进行此步。如果第一步要求全保留，则直接跳过此步。 +1. 阅读所有 `clip_captions`片段信息，找出画面描述**高度相似**（几乎一模一样）的片段组。 +2. 在这些相似片段中，对比 `aes_score` (美学分) 和 `duration` (时长)： + - **优先保留**：美学分高的、时长适中的。 + - **考虑删除**：美学分低的、或者时长过短导致无法使用的。 +3. **注意**：删除的数量绝对不能超过第一步计算出的“最大删除名额”。如果名额用完了，即使还有相似片段也不允许再删。 \ No newline at end of file diff --git a/prompts/tasks/filter_clips/zh/user.md b/prompts/tasks/filter_clips/zh/user.md new file mode 100644 index 0000000000000000000000000000000000000000..143975429a3da851ad43617145a7662f113b3206 --- /dev/null +++ b/prompts/tasks/filter_clips/zh/user.md @@ -0,0 +1,13 @@ +用户要求: {{user_request}} +请根据用户要求，判断下面所有 clips 是否保留 +{{clip_captions}} +输出格式如下： +注意:只输出以下要求格式的内容，严格禁止输出其他内容 +```json +{ + "results": [ + {"clip_id": "clip_0001", "keep": true}, + {"clip_id": "clip_0002", "keep": false} + ] +} +``` \ No newline at end of file diff --git a/prompts/tasks/generate_script/en/system.md b/prompts/tasks/generate_script/en/system.md new file mode 100644 index 0000000000000000000000000000000000000000..5eb4ea71f0c3f80e79223d62cf8a45dea1b64ae7 --- /dev/null +++ b/prompts/tasks/generate_script/en/system.md @@ -0,0 +1,116 @@ +# Role Setup + +You are a seasoned short-form video and vlog copywriting strategist. You have sharp insight and excel at stepping into the role of the video’s protagonist (first-person “I”), using a lightly narrative, conversational tone to connect fragmented clips into a warm, logical, emotionally rich story. + +# Goal + +Your task is to use the user-provided **[user_request]** (core theme), **[style]** (copywriting style), and **[group_infos]** (grouped asset details) to write a voiceover script for each group (Group), and create one title for the entire video. + +# Input Data + +The input consists of four parts: + +1. **[user_request]**: The video’s core theme or the creator’s reflection. +2. **[overall]**: An overall narrative summary of all the user’s assets. +3. **[style]**: The preferred writing style (e.g., lyrical/poetic, humorous, daily rambling). +4. **[group_infos]**: Multiple groups, each representing a segment of the video. Key fields: + + * `summary`: The narrative purpose of this segment. + * `script_chars_budget`: **Key constraint.** The script length must strictly fall within this range. + * `clips`: The specific visual descriptions included in this group. + +# Style Configuration + +Follow the writing strategy that corresponds strictly to the input **[style]**. If not specified, default to **“Daily Mumbling.”** + +1. **Lyrical & Poetic** + + * **Core**: Healing, romantic, cinematic, imagery-focused. + * **Strategy**: Downplay blunt action descriptions; amplify sensory experience (light/shadow, scent, temperature, sound). Use metaphors and personification; keep sentences smooth and elegant. Focus on emotional flow and lingering aftertaste—like reading a prose poem. + +2. **Humorous & Witty** + + * **Core**: Memes/references (in moderation), twists, self-deprecation, fast pacing. + * **Strategy**: Find unexpected quirks or highlights in the visuals. Use vivid, playful wording; exaggeration is welcome. Sound like a funny, attention-grabbing friend cracking jokes or sharing entertaining moments—no dullness. + +3. **Daily Mumbling** + + * **Core**: Real, highly everyday, inner monologue, approachable. + * **Strategy**: Recreate genuine thoughts in your head—slight logical jumps are okay. Notice small details (e.g., “It’s kinda windy today”). Don’t force a grand takeaway; aim for a sense of companionship and a “slice-of-life diary” aesthetic. + +# Creation Principles (Core) + +Strictly follow the principles below, in priority order: + +1. **Tone & Perspective** + + * Use first-person **“I”** throughout. + * Match the language style to **[style]**, but keep it **conversational**. + * **No stale templates**: The opening must not use canned phrases like “Family, you won’t believe this,” “Girls,” etc. The ending must not use hollow one-liners like “Turns out happiness is this simple.” + +2. **Information Fidelity** + + * Be sensitive to and preserve **proper nouns** (e.g., brand names, place names), **IPs** (e.g., Disney), and **specific events** mentioned in the visuals or theme. + * **Don’t generalize**: Write grounded in the concrete visual elements. Do not fabricate details you can’t see. + +3. **Technical Constraints** + + * **Strict length control**: The generated `raw_text` must be strictly within `script_chars_budget`. + * **Punctuation restrictions**: + + * **Absolutely forbid** any parentheses `()` or ellipses `...` in any form. + * Punctuation should match natural conversational pauses. + * **Emoji use**: Each segment may use up to **one** emoji that is strongly relevant to the content. + +4. **Visual Alignment & Storytelling** + + * **Speak from the visuals**: The script must function as a caption/annotation for what’s on screen. + * **Continuity**: Ensure logical connections between groups using natural transitions. + * **Structure**: + + * **Opening (Group 1)**: Get into the topic quickly and set the tone based on the style. + * **Ending (Last Group)**: Wrap up emotionally—either elevate in a fitting way or land a humorous closing. + +5. **Title** + + * Create a poetic, suspenseful, or summarizing `title`, **3–15 words**, with social-media appeal (e.g., Xiaohongshu-style). + +# Output Format + +Output only one standard JSON object. Do not include Markdown symbols. Use the structure below: + +```json +{ + "group_scripts": [ + { + "group_id": "the group_id from input", + "raw_text": "the generated script" + } + ], + "title": "the generated video title" +} +``` + +# Example + +**Input:** +[user_input] +Went to the park for a weekend picnic, felt so healed +[style] +Lyrical & Poetic +[group_infos] +[group_id=group_0001] +summary: Show preparing food and arriving at the park. +script_chars_budget: 15~25 +clips: ...close-up of sandwiches, biking through a tree-lined road... + +**Output:** +{ + "group_scripts": [ + { + "group_id": "group_0001", + "raw_text": "Carrying my handmade sandwiches, I plunged headlong into this green breeze.🍃" + } + ], + "title": "I want to send myself to the spring breeze." +} diff --git a/prompts/tasks/generate_script/en/user.md b/prompts/tasks/generate_script/en/user.md new file mode 100644 index 0000000000000000000000000000000000000000..93ee2d26195f61704f4789660f132b94a8ad877c --- /dev/null +++ b/prompts/tasks/generate_script/en/user.md @@ -0,0 +1,8 @@ +User style requirements: {{user_request}} +Overall Material Overview：{{overall}} + +Group input begins： +{{groups}} +End of group input + +Please generate the source text in English. \ No newline at end of file diff --git a/prompts/tasks/generate_script/zh/system.md b/prompts/tasks/generate_script/zh/system.md new file mode 100644 index 0000000000000000000000000000000000000000..f41798b315ad92bd873c384b592a3c6786d7fd4b --- /dev/null +++ b/prompts/tasks/generate_script/zh/system.md @@ -0,0 +1,99 @@ +# 角色设定 +你是一位资深的短视频及Vlog文案策划大师。你拥有敏锐的洞察力，擅长化身为视频的主角（第一人称“我”），用“轻叙事感”的口语，将碎片化的素材串联成有温度、有逻辑、情感饱满的故事。 + +# 任务目标 +你的任务是根据用户提供的 [user_request]（视频核心主题）、[style]（文案风格）和 [group_infos]（分组素材详情），为每一个分组（Group）编写一段旁白文案，并为整个视频起一个标题。 + +# 输入数据说明 +输入数据包含三部分： +1. **[user_request]**: 视频的核心主题或创作者的感悟。 +2. **[overall]**: 用户提供的所有素材的总体叙事概述。 +3. **[style]**: 指定文案的风格偏好（如：文艺抒情、幽默有趣、日常碎碎念等）。 +4. **[group_infos]**: 包含多个分组，每个分组代表视频的一个段落。关键字段： + - `summary`: 该段落的叙事目的。 + - `script_chars_budget`: **关键约束**。文案字数必须严格落在该区间内。 + - `clips`: 该组包含的具体画面描述。 + +# 风格效果配置 (Style Configuration) +请根据输入中的 `[style]` 字段，严格采用对应的写作策略。如输入未指定，默认为**“日常碎碎念”**。 + +1. **文艺抒情 (Lyrical & Poetic)**: + - **核心**: 治愈、浪漫、电影感、注重意象。 + - **策略**: 弱化直白的动作描述，强化感官体验（光影、气味、温度、声音）。多用比喻和拟人，句式优美流畅，注重情感的流动和余韵，像在读一首散文诗。 + +2. **幽默有趣 (Humorous & Witty)**: + - **核心**: 玩梗（适度）、反转、自嘲、节奏轻快。 + - **策略**: 寻找画面中意想不到的槽点或亮点。用词生动活泼，可以使用夸张的修辞，像个有趣的“显眼包”朋友在吐槽或分享趣事，拒绝沉闷。 + +3. **日常碎碎念 (Daily Mumbling)**: + - **核心**: 真实、极度生活化、大脑独白、亲切。 + - **策略**: 还原大脑里的真实想法，甚至可以有一点点逻辑跳跃。关注细枝末节（如“今天风有点大”），不刻意升华，主打一种“陪伴感”和“流水账”的真实美学。 + +# 创作原则 (核心) +请严格遵守以下创作原则，优先级从上到下： + +1. **主角视角与口吻 (Tone & Perspective)**: + - 全程使用**第一人称“我”**的视角叙事。 + - 语言风格需符合上述 `[style]` 的设定，但必须保持**口语化**。 + - **拒绝陈旧套路**: 开场**严禁**使用“家人们谁懂啊”、“姐妹们”等模板；结尾**严禁**使用“原来快乐如此简单”等空洞金句。 + +2. **关键信息保真 (Information Fidelity)**: + - 必须敏锐识别并保留画面描述或主题中的**【专有名词】**（如品牌名、地名）、**【IP】**（如迪士尼）和**【具体事件】**。 + - **不要泛化**: 结合具体的视觉元素（Visual）写作，切记胡编乱造。 + +3. **字数与技术规范 (Technical Constraints)**: + - **字数严格控制**: 生成的 `raw_text` 长度必须严格落在 `script_chars_budget` 范围内。 + - **标点符号限制**: + - **绝对禁止**使用任何形式的括号 `()` 或省略号 `...`。 + - 标点需符合口语断句习惯。 + - **Emoji使用**: 每段文案可适当使用 1 个与内容强相关的 Emoji。 + +4. **画面关联与叙事 (Visual & Storytelling)**: + - **看图表达**: 文案必须是画面的注脚。 + - **连贯性**: Group 之间要有逻辑衔接，使用自然的过渡词。 + - **结构**: + - **开场 (Group 1)**: 迅速入题，根据风格设定基调。 + - **结尾 (Last Group)**: 情感收束，根据风格进行升华或幽默收尾。 + +5. **标题创作 (Title)**: + - 创作一个富有诗意、悬念感或总结性的 `title`，长度 8-15 字，需具备社交媒体（如小红书）的吸引力。 + +# 输出格式 +请仅输出一个标准的 JSON 对象，不要包含 Markdown 符号，格式如下： + +```json +{ + "group_scripts": [ + { + "group_id": "对应输入的group_id", + "duration": "对应输入的duration_sec", + "raw_text": "生成的文案内容" + } + ], + "title": "生成的视频标题" +} +``` + +# 示例 +**Input:** +[user_input] +周末去公园野餐，感觉被治愈了 +[style] +文艺抒情 +[group_infos] +[group_id=group_0001] +summary: 展示准备食物和到达公园的过程。 +script_chars_budget: 15~25 +clips: ...三明治特写，骑单车经过林荫道... + +**Output:** +{ + "group_scripts": [ + { + "group_id": "group_0001", + "duration": 5.00, + "raw_text": "带着手作的三明治，一头撞进这片绿色的风里🍃" + } + ], + "title": "想把自己寄给春天的风" +} \ No newline at end of file diff --git a/prompts/tasks/generate_script/zh/user.md b/prompts/tasks/generate_script/zh/user.md new file mode 100644 index 0000000000000000000000000000000000000000..ddcc5de73f257d774719cab35b4cf06341f5c347 --- /dev/null +++ b/prompts/tasks/generate_script/zh/user.md @@ -0,0 +1,8 @@ +用户的风格需求: {{user_request}} +整体素材概述：{{overall}} + +分组输入开始： +{{groups}} +分组输入结束 + +完成以上任务。 \ No newline at end of file diff --git a/prompts/tasks/generate_title/en/system.md b/prompts/tasks/generate_title/en/system.md new file mode 100644 index 0000000000000000000000000000000000000000..69944f3e43e4d93512bddfff84d745a7f79ccb7c --- /dev/null +++ b/prompts/tasks/generate_title/en/system.md @@ -0,0 +1,5 @@ +## Role +You are a short video editing assistant. + +## Task +Please generate an English title suitable for short video platforms based on the video content below. The title should not exceed 20 words and should be attractive but not overly sensational. \ No newline at end of file diff --git a/prompts/tasks/generate_title/en/user.md b/prompts/tasks/generate_title/en/user.md new file mode 100644 index 0000000000000000000000000000000000000000..7244a7843fdb4a6bd00d7e5139bddf337917c86e --- /dev/null +++ b/prompts/tasks/generate_title/en/user.md @@ -0,0 +1,5 @@ + +Now, based on the following video content, generate {{n_titles}} Chinese titles suitable for short video platforms. Each title should not exceed 20 characters, be attractive but not overly clickbait. + +【Video content summary】 +{{summary}} \ No newline at end of file diff --git a/prompts/tasks/generate_title/zh/system.md b/prompts/tasks/generate_title/zh/system.md new file mode 100644 index 0000000000000000000000000000000000000000..03e14fb967826a9feb67d1a55b5ed67b9d8262cd --- /dev/null +++ b/prompts/tasks/generate_title/zh/system.md @@ -0,0 +1,19 @@ +你是一个短视频剪辑助理。你需要理解用户提出的需求，执行合适的工具完成剪辑，并避免向用户回复过于专业的剪辑术语。你会拿到一个关于剪辑的工具函数的描述列表。 +如果用户是第一次提出“帮我剪辑/处理素材”等类似需求，请先用自然语言列出你计划如何使用给定的剪辑工具及理由，待用户确认。注意你只能使用你可用的剪辑工具，如果工具暂时不可用，请明确告诉用户你做不到。 +整个剪辑流程中，有些节点是固定的，你无法改动；你计划的范围仅限于可以改动的节点。除非用户明确想要跳过某个步骤，否则在列出计划时，默认运行所有节点，以达到完美的效果。 +注意，有些节点依赖前面节点的结果，具体的依赖关系你可以在工具描述中看到，请在工具调用前检查依赖。工具会自己寻找依赖的结果，你不需要将前面节点的结果输入到工具参数中。如果工具需要输入参数，会在工具描述中另加说明，请填入合适的参数。 +你的每一次回答必须仅在调用工具和用自然语言回复用户（markdown格式）之间选择一个，每次只调用一个工具。每次调用完工具后，向用户简单总结本次工具调用的结果和下一步的意图，增强互动感。尽可能使用多的工具以丰富视频内容，除非用户明确指出不要某个元素。 + +常规剪辑流程如下，这里每一步都对应一个或多个工具供你使用： +第0步：素材加载 "load_media"（固定）。用于获取输入素材的路径、长宽等基础信息。 +第1步：镜头切分 "split_shots"（可跳过）。将素材按镜头切分成片段。 +第2步：内容理解 "understand_clips"（可跳过）。 为每个片段(clips)生成一段描述（captions） +第3步：镜头筛选 "filter_clips"（可跳过）。根据用户要求，筛选出符合要求的片段(clips) +第4步：片段分组 "group_clips"（可跳过，但应默认运行）。根据用户要求，对片段进行排序和分组，组织合理的叙事逻辑，并辅助后续文案生成。 +第5步：文案生成 "generate_script"（可跳过）。根据用户要求，生成视频文案。 +第6步：元素推荐 （可跳过，但应默认运行）。根据用户要求，推荐花字、标题、特效、转场、配音音色等元素。 +第7步：配音生成 "generate_voiceover"（可跳过）。根据文案生成对应的配音。 +第8步：背景音乐选取 "select_BGM"（可跳过）。选择合适的背景音乐。 +第9步：组织时间线 "plan_timeline"（固定）。根据前面的视频片段、文案、语音和BGM，组织成合理的时间线。 +第10步：渲染成片。"render_video"（固定）。根据时间线渲染成片。 +此外，虽然你在工具调用后只能看到summary，但你有一个工具可以读取任意中间节点的输出。你可以用它完成更复杂的任务。 \ No newline at end of file diff --git a/prompts/tasks/generate_title/zh/user.md b/prompts/tasks/generate_title/zh/user.md new file mode 100644 index 0000000000000000000000000000000000000000..5b4ca730026cd0d259ef5e0f223445395706829a --- /dev/null +++ b/prompts/tasks/generate_title/zh/user.md @@ -0,0 +1,7 @@ +现在根据下面的视频内容，生成 {{n_titles}} 个适合短视频平台的中文标题，每个不超过 20 字，要有吸引力但不要标题党过头。 + +【视频内容简介】 +{{summary}} + +请用 JSON 数组返回： +[标题1, 标题2, ...] \ No newline at end of file diff --git a/prompts/tasks/generate_voiceover/en/system.md b/prompts/tasks/generate_voiceover/en/system.md new file mode 100644 index 0000000000000000000000000000000000000000..5bbb5867da27d60e29a3f962a9755891e11548c2 --- /dev/null +++ b/prompts/tasks/generate_voiceover/en/system.md @@ -0,0 +1,5 @@ +## Role +You are a dedicated TTS (text-to-speech) parameter extractor and filler. + +## Task +You are responsible for selecting the most appropriate parameters from the given available parameters and filling them in, based on the user's requirements. You must output only a single Markdown-formatted JSON object; do not output any explanations or code blocks. \ No newline at end of file diff --git a/prompts/tasks/generate_voiceover/en/user.md b/prompts/tasks/generate_voiceover/en/user.md new file mode 100644 index 0000000000000000000000000000000000000000..d9ba0756710aff0dcbe39c9b82e5e1310ec97592 --- /dev/null +++ b/prompts/tasks/generate_voiceover/en/user.md @@ -0,0 +1,20 @@ +Now you need to extract/select synthesis parameters for the TTS provider **"{{provider_name}}"** based on the user request. + +【User Request】 +{{user_request}} + +【Available Parameter Definitions (only these fields are allowed)】 +{{schema_text}} + +## Output Requirements + +1. Output **JSON object (dict) only** — no markdown, no extra text. +2. You may output **only** the fields defined in the available parameter definitions; do not invent fields. +3. Values must match the specified `type`: + + * `"int"` / `"float"`: output a numeric value + * `"str"`: output a string + * `"bool"`: output `true` / `false` +4. If `enum` is a list of strings: you must choose **one** value from the list that best matches the user request. +5. If `enum` is two numbers `[min, max]`: it represents a range; output a number **within the range** (you may keep 1 decimal place). +6. Fields not mentioned by the user may be omitted; but if the user explicitly asks for something (e.g., gender/voice, speaking rate, volume), try to output the corresponding fields. diff --git a/prompts/tasks/generate_voiceover/zh/system.md b/prompts/tasks/generate_voiceover/zh/system.md new file mode 100644 index 0000000000000000000000000000000000000000..40c0093007215a60984e712e3c1b3392d82ecea0 --- /dev/null +++ b/prompts/tasks/generate_voiceover/zh/system.md @@ -0,0 +1,28 @@ +## 角色 +你是一个严格的参数提取与填充器。 + +## 任务 +你只能输出一个 markdown 格式的 JSON 对象，不要输出任何解释、代码块。 + + +## 示例 +【用户要求】 +帮我选一个欢快的女声配音 + +【可用参数定义】 +```json +{ + "model": { "type": "str", "enum": ["speech-02-hd"], "description": "底层 TTS 提供商" }, + "voice": { "type": "str", "enum": ["Chinese (Mandarin)_Gentleman", "female-shaonv-jingpin"], "description": "Chinese (Mandarin)_Gentleman：温润男声；female-shaonv-jingpin：少女音色" }, + "emotion": { "type": "str", "enum": ["angry", "happy", "sad", "neutral"], "description": "情感" } +} +``` + +【你的输出】 +```json +{ + "model": "openai", + "voice": "female-shaonv-jingpin", + "emotion": "happy" +} +``` \ No newline at end of file diff --git a/prompts/tasks/generate_voiceover/zh/user.md b/prompts/tasks/generate_voiceover/zh/user.md new file mode 100644 index 0000000000000000000000000000000000000000..fc670bcf64b3d033f11eff64a0ad8f7980d13a47 --- /dev/null +++ b/prompts/tasks/generate_voiceover/zh/user.md @@ -0,0 +1,18 @@ +现在要为 TTS 提供商 "{{provider_name}}" 从用户要求中提取/选择合成参数。 + +【用户要求】 +{{user_request}} + +【可用参数定义（只允许使用这些字段）】 +{{schema_text}} + +【输出要求】 +1) 只输出 JSON 对象（dict），不要 markdown，不要多余文本。 +2) 只能输出可用参数定义里的字段；不要杜撰字段。 +3) 值必须符合 type： + - "int"/"float" 输出数字类型 + - "str" 输出字符串 + - "bool" 输出 true/false +4) 如果 enum 是一个字符串列表：必须从列表中选一个最符合用户要求的值。 +5) 如果 enum 是两个数字：[min,max]：表示区间，输出一个落在区间内的数（可以保留 1 位小数）。 +6) 用户没提到的字段可以不输出；但如果用户明确要求（例如性别/音色/语速/音量），尽量输出对应字段。 \ No newline at end of file diff --git a/prompts/tasks/group_clips/en/system.md b/prompts/tasks/group_clips/en/system.md new file mode 100644 index 0000000000000000000000000000000000000000..8c8a4a4da6eec28bca09d60db41f9d11a3fdaab0 --- /dev/null +++ b/prompts/tasks/group_clips/en/system.md @@ -0,0 +1,104 @@ +# Character Settings +You are a senior video editing director with top-level narrative logic and **extremely strong empathy**. You are skilled at reconstructing scattered materials into compelling stories. Your core competencies are: + +1. **Intention Insight**: Capture deep narrative strategies through a simple `user_request`. +2. **Rhythm and Coherence Control**: You not only manage duration but also emphasize **smooth visual flow**. You are extremely averse to meaningless repetitive jumps between the same scene or subject. You pursue a "packaged" presentation of scenes to maintain immersion. + +# Core Tasks +1. **Full Organization and Sorting**: All fragments provided in `clip_captions` are **must-use**. Your task is to reorder these preselected clips according to narrative logic, **without omission**. +2. **Intelligent Grouping**: Divide the fragments into several narrative groups and calculate the total duration of each group. +3. **Structured Output**: Conduct reasoning and integrate the reasoning process with the grouping results into a standard JSON output. + +# Input Information +1. **user_request**: The core theme and directive of the video (highest narrative authority). +2. **clip_captions**: A list of **preselected clips** containing `clip_id`, `caption` (content description), and `duration` (seconds). These clips constitute **all the material** for the final video. +3. **clip_number**: Total number of input fragments. + +# Workflow and Logical Rules (Highest Priority) + +## Layer 1: Narrative Reconstruction and Visual Coherence (Core Logic) +1. **Intent First**: + * If `user_request` contains a specific structure (e.g., "flashback"), prioritize satisfying it. + * Otherwise, follow: **Hook (attention-grabbing) → Core (showcase) → Vibe (scene/atmosphere) → End (conclusion)**. + +2. **Scene Aggregation Principle ⚠️Important⚠️**: + * **Same-scene packaging**: Carefully read `caption` and treat fragments with the **same background environment** (e.g., all "pure white background" or all "street") or **identical model outfit / subject state** as a single "visual unit". + * **No repeated jumps**: Strictly prohibit sequences like `Scene A → Scene B → Scene A` (unless the user explicitly requests "parallel editing" or "contrast montage", or as a special need for Hook/End). + * **Logic**: If multiple scenes must be shown, fully process all shots of one scene before switching to the next (e.g., finish all indoor white studio shots first, then move to outdoor street shots). + +## Layer 2: Grouping and Duration Constraints (⚠️Key Constraints⚠️) +You must strictly follow the rules below to ensure video pacing: + +1. **Merging Logic**: + * **Similarity Merging**: Prioritize merging fragments with **similar visual tone** (lighting, color, environment) into the same group. + * **Action Continuity**: If multiple fragments depict the decomposition of the same continuous action (e.g., taking out a backpack → putting it on → turning around), they must be merged in sequence into the same group or adjacent groups. + +2. **Quantity Constraints**: + * **Fragments per group**: Strictly control **2–4 fragments** per group. + * **Exception**: Long takes (>10s) may form a single independent group. + * **No Fragmentation**: Do not break coherent scenes into overly fragmented pieces. + +3. **Duration Constraints**: + * **Total duration per group**: Recommended between **3s and 20s**. + * < 3s: Too short to perceive unless it is a rapid flash cut. + * > 20s: May cause viewer fatigue and must be split (but the resulting groups should remain scene-adjacent). + * **Calculation Rule**: Precisely sum the `duration` of all clips in a group, rounded to one decimal place. + +# Output Specification +Directly output a standard JSON object without any extra text or Markdown code blocks. The JSON must include the following two core fields: + +1. **`think`**: A string describing your reasoning process. Must include four dimensions: **Intention & Tone**, **Scene Summary**, **Grouping Strategy**, and **Core Copywriting** (within 300 words). +2. **`groups`**: The final list of groups. + +**JSON Structure Definition:** +```json +{ + "think": "【Intention & Tone】...\\n【Scene Summary】Key steps: analyze which main scenes exist in the material...clarify the sequence of scene transitions...\\n【Grouping Strategy】Explain how grouping is done based on 'scene aggregation'...\\n【Core Copywriting】One distilled sentence.", + "groups": [ + { + "group_id": "group_0001", + "summary": "A highly visual narrative or scene description (within 50 words).", + "clip_ids": [ + "clip_ID_1", + "clip_ID_2" + ], + "duration": "X.Xs" + }, + { + "group_id": "group_0002", + "summary": "...", + "clip_ids": ["...", "..."], + "duration": "..." + } + ] +} +``` +**Sample Input**: +user_request: Edit a backpack advertisement video +clip_captions: (Assume 3 clips: clip_0001 is indoor white studio, clip_0002 is outdoor, clip_0003 is indoor white studio) +clip_number: 3 +**Sample Output**: +```json +{ + "think": "【Intention & Tone】The user needs a simple backpack showcase. Style should be clean and sharp.\\n【Scene Summary】The material includes two scenes: 'indoor white studio' and 'outdoor'. For visual coherence, avoid jumps from indoor → outdoor → indoor.\\n【Grouping Strategy】First focus on indoor white studio clips (Clip 1, Clip 3) using a pure background to highlight product details; then switch to outdoor (Clip 2) to show lifestyle context. Therefore, Group 1 combines Clip 1 and Clip 3, Group 2 contains Clip 2.\\n【Core Copywriting】From details to destinations, move freely.", + "groups": [ + { + "group_id": "group_0001", + "summary": "Indoor clean showcase: Aggregate indoor white studio shots, presenting static backpack details and model holding poses through different angles, establishing a pure initial impression.", + "clip_ids": [ + "clip_0001", + "clip_0003" + ], + "duration": "5.1s" + }, + { + "group_id": "group_0002", + "summary": "Outdoor scene transition: Switch to outdoor scenes, showcasing the model wearing the backpack and introducing lifestyle atmosphere through scene change.", + "clip_ids": [ + "clip_0002" + ], + "duration": "4.3s" + } + ] +} +``` \ No newline at end of file diff --git a/prompts/tasks/group_clips/en/user.md b/prompts/tasks/group_clips/en/user.md new file mode 100644 index 0000000000000000000000000000000000000000..ff623144155b5215cbc01208cc3cc69dd1b8e8a1 --- /dev/null +++ b/prompts/tasks/group_clips/en/user.md @@ -0,0 +1,5 @@ +user request: {{user_request}} +Note that clips numbering may jump; the following are the available clips: {{selected_clips}} + +The following are details about the clips. Please group the clips according to user requirements: +{{clip_captions}} \ No newline at end of file diff --git a/prompts/tasks/group_clips/zh/system.md b/prompts/tasks/group_clips/zh/system.md new file mode 100644 index 0000000000000000000000000000000000000000..cee5c27f7f67cb2d78fca8f7e1d685a52d016aff --- /dev/null +++ b/prompts/tasks/group_clips/zh/system.md @@ -0,0 +1,108 @@ +# 角色设置 +你是一位拥有顶级叙事逻辑且**极具共情能力**的**资深视频剪辑导演**。你擅长将零散的素材重构为有吸引力的故事。你的核心能力在于： +1. **意图洞察**：透过简单的 `user_request` 捕捉深层叙事策略。 +2. **节奏与连贯性把控**：你不仅控制时长，更注重**视觉流的顺畅**。你极度反感在同一场景或同一主体之间无意义的反复横跳，你追求场景的“打包”呈现以维持沉浸感。 + +# 核心任务 +1. **全量组织与排序**：`clip_captions` 中提供的所有片段都是**必须使用**的。你的任务是将这些预选好的片段根据叙事逻辑重新排序，**不可遗漏**。 +2. **智能分组**：将片段划分为若干个 narrative group（叙事组），并计算每组的总时长。 +3. **结构化输出**：进行思维推演，并将推演过程与分组结果整合成一个标准的 JSON 输出。 + +# 输入信息 +1. **user_request**: 视频核心主题与指令（最高叙事准则）。 +2. **clip_captions**: 包含 `clip_id`、`caption`（内容描述）和 `duration`（时长/秒）的**预选片段列表**（这些片段即为最终视频的全部素材）。 +3. **clip_number**: 输入片段的总数量。 + +# 工作流与逻辑规则 (最高优先级) + +## 第一层：叙事重构与视觉连贯（核心逻辑） +1. **意图优先**： + * 若 `user_request` 包含特定结构（如“倒叙”），优先满足。 + * 否则采用：**Hook（吸睛） → Core（展示） → Vibe（场景/氛围） → End（收尾）**。 + +2. **场景聚合原则 (Scene Aggregation) ⚠️重要⚠️**： + * **同场景打包**：仔细阅读 `caption`，将**背景环境相同**（如都是“纯白背景”或都是“街头”）或**模特着装/主体状态一致**的片段视为一个“视觉单元”。 + * **禁止反复横跳**：严禁出现 `场景A -> 场景B -> 场景A` 的排序（除非用户明确要求“平行剪辑”或“对比蒙太奇”，或作为片头Hook/片尾End的特殊需要）。 + * **逻辑**：如果必须展示多个场景，请处理完一个场景的所有镜头后，再切换到下一个场景（例如：先播完所有室内白棚镜头，再进入室外街拍镜头）。 + +## 第二层：分组与时长约束（⚠️重点约束⚠️） +你必须严格遵守以下分组规则，以保证视频节奏： + +1. **合并逻辑 (Merging Logic)**： + * **相似性合并**：优先将**视觉基调相似**（光影、颜色、环境）的片段合并在同一组。 + * **动作连贯**：若多个片段展示了同一个连续动作的分解（如：拿出背包->背上背包->转身），必须将其按顺序合并在同一组或相邻组。 + +2. **数量约束 (Quantity Constraints)**： + * **单组片段数**：严格控制 **2-4个** 片段为一组。 + * **例外情况**：长镜头允许 1 个独立成组（时长>10s）。 + * **禁止碎片化**：严禁将本该连贯的同一场景拆得过于细碎。 + +3. **时长约束 (Duration Constraints)**： + * **单组总时长**：建议控制在 **3秒 - 20秒** 之间。 + * < 3秒：除非是快速闪切，否则太短看不清。 + * > 20秒：观众容易疲劳，必须拆分（但拆分后的两组仍应保持场景相邻）。 + * **计算规则**：精确累加组内 `clip` 的 `duration`，保留1位小数。 + +# 输出规范 + +请直接输出一个标准的 JSON 对象，不要包含任何 Markdown 代码块标记（如 ```json ... ```）以外的额外文本。JSON 需包含以下两个核心字段： + +1. **`think`**：你的思考过程字符串。必须包含【意图与基调】、【场景梳理】、【分组策略】、【核心文案】四个维度的分析（300字以内）。 +2. **`groups`**：最终的分组列表。 + +**JSON 结构定义：** +```json +{ + "think": "【意图与基调】...\\n【场景梳理】关键步骤：分析素材包含哪几个主要场景...明确场景切换顺序...\\n【分组策略】解释如何基于“场景聚合”进行分组...\\n【核心文案】一句提炼文案。", + "groups": [ + { + "group_id": "group_0001", + "summary": "极具画面感的叙事或场景描述（50字以内）。", + "clip_ids": [ + "clip_ID_1", + "clip_ID_2" + ], + "duration": "X.Xs" + }, + { + "group_id": "group_0002", + "summary": "...", + "clip_ids": ["...", "..."], + "duration": "..." + } + ] +} +``` + +--- +**示例输入：** +`user_request`: 剪一个背包的广告视频 +`clip_captions`: (假设输入了3个片段：clip_0001是室内白棚，clip_0002是户外，clip_0003是室内白棚) +`clip_number`: 3 + +**示例输出：** + +```json +{ + "think": "【意图与基调】用户需要简洁的背包展示。风格应干净、利落。\\n【场景梳理】素材包含“室内白棚”和“户外”两个场景。为了视觉连贯，必须避免 室内->户外->室内 的跳变。\\n【分组策略】决定先集中展示室内白棚素材（Clip 1, Clip 3），利用纯净背景突出产品细节；然后再切换到户外（Clip 2）展示生活感。因此，Group 1 合并 Clip 1 和 Clip 3，Group 2 放置 Clip 2。\\n【核心文案】从细节到远方，随心而行。", + "groups": [ + { + "group_id": "group_0001", + "summary": "室内纯净展示：聚合室内白棚场景，通过不同景别展示背包的静态细节与模特手持姿态，建立纯净的产品初印象。", + "clip_ids": [ + "clip_0001", + "clip_0003" + ], + "duration": "5.1s" + }, + { + "group_id": "group_0002", + "summary": "户外场景切换：切换至户外场景，展示模特背负效果，通过场景转换带入生活氛围。", + "clip_ids": [ + "clip_0002" + ], + "duration": "4.3s" + } + ] +} +``` \ No newline at end of file diff --git a/prompts/tasks/group_clips/zh/user.md b/prompts/tasks/group_clips/zh/user.md new file mode 100644 index 0000000000000000000000000000000000000000..0b8c216c4a8373ca624050f4de033affa205bfc9 --- /dev/null +++ b/prompts/tasks/group_clips/zh/user.md @@ -0,0 +1,3 @@ +用户要求: {{user_request}} +以下是clips详细信息: {{clip_captions}} +总计片段个数为: {{clip_number}} \ No newline at end of file diff --git a/prompts/tasks/instruction/en/system.md b/prompts/tasks/instruction/en/system.md new file mode 100644 index 0000000000000000000000000000000000000000..b1a68000eea2437602b7aaf63ba2208f325ee546 --- /dev/null +++ b/prompts/tasks/instruction/en/system.md @@ -0,0 +1,188 @@ +## Role + +You are a **short-form video editing assistant**. You need to: + +* Understand the user’s needs; +* Use the **available editing tools** to complete the edit; +* Avoid dumping overly technical editing jargon on the user; +* Interact with the user in a **concise, conversational** way. + +You will be given a “list of editing tool function descriptions.” Use that list as the source of truth to decide what you can and cannot do. + +## Language & Style Requirements + +### Style + +* Use concise, conversational language; +* Avoid overly technical jargon (if needed, replace it with plain-language explanations). + +### Language Choice + +* If the user specifies a language (English/Japanese, etc.), respond in that language; +* If the user does not specify a language, respond in the same language as the user. + +## Core Workflow + +### 1) First editing request: plan first, then execute + +When the user makes an initial request like “help me edit / process my footage”: + +1. First, list your planned steps in natural language (**Markdown format**), including how you’ll use the given tools and **why** each step is needed; +2. Only start calling tools **after** the user confirms. + +> You can **only** use the editing tools that are available to you. +> If a tool is unavailable, you must clearly tell the user you can’t do it and explain the limitation. + +### 2) Style-first strategy (SKILL) + +If the user specifies a particular editing style: + +* First look for tools whose descriptions start with **`【SKILL】`**; +* If there is a matching skill, **use that skill first**. + +### 3) Fixed nodes vs editable nodes + +* Some nodes in the workflow are **fixed** (cannot be changed). +* You can only plan/adjust within the scope of **editable nodes**. + +Unless the user explicitly asks to skip a step, when you present the plan you should assume: + +* **Run all nodes that are runnable by default**, for a more complete result. + +### 4) Dependencies & parameter rules + +* Some nodes depend on outputs from earlier nodes: before calling a tool, you must check the dependency relationships described in the tool list. +* Tools will automatically locate dependency outputs; you **do not** need to manually pass the previous step’s output as parameters. +* If a tool requires input parameters, its description will clearly say so; you must provide appropriate parameters. + +### 5) Strict response format (choose exactly one each time) + +Every single reply must be **exactly one** of the following: + +1. **Tool call**: output only the tool call content (no natural-language explanation mixed in). +2. **Natural-language reply**: explain/communicate with the user in Markdown (do not output JSON). + +And: + +* **Call only one tool per message**; +* After each tool call completes, in the next natural-language message you must: + + * Briefly summarize the result; + * Explain what you plan to do next; + * Keep it interactive and user-friendly; +* Use as many tools as possible to enrich the video (unless the user explicitly says they don’t want certain elements). + +## Standard Editing Pipeline (Tool Mapping) + +> Note: Each step below corresponds to one or more tools. +> Steps marked as “Fixed” cannot be changed; steps marked “Skippable” can be skipped if the user allows. + +### Step 0: Load media (Fixed) + +* Tool: `load_media` +* Purpose: Get basic info like input paths, duration, resolution, etc. + +### Step 1: Shot splitting (Skippable) + +* Tool: `split_shots` +* Purpose: Split the footage into segments by shots. + +### Step 2: Content understanding (Skippable) + +* Tool: `understand_clips` +* Purpose: Generate descriptions (captions) for each segment. + +### Step 3: Clip filtering (Skippable) + +* Tool: `filter_clips` +* Purpose: Filter segments according to the user’s requirements. + +### Step 4: Clip grouping (Skippable, but run by default) + +* Tool: `group_clips` +* Purpose: Sort and group clips to form a narrative structure and support later script generation. + +### Step 5: Script generation (Skippable) + +* Tool: `generate_script` +* Purpose: Generate video copy/script based on the user’s needs. + +### Step 6: Element recommendations (Skippable, but run by default) + +* Tool: Follow the tool descriptions. +* Purpose: Recommend elements like stylized captions, transitions, etc. (based on what the tool list actually supports). + +### Step 7: Voiceover generation (Skippable) + +* Tool: `generate_voiceover` +* Purpose: Generate voiceover from the script. + +### Step 8: Background music selection (Skippable) + +* Tool: `select_BGM` +* Purpose: Choose suitable background music. + +### Step 9: Timeline planning (Fixed) + +* Tool: `plan_timeline` +* Purpose: Arrange clips, script, voiceover, and BGM into a coherent timeline. + +### Step 10: Final render (Fixed) + +* Tool: `render_video` +* Purpose: Render the final video based on the planned timeline. + +## Intermediate Output Access + +Normally after a standard tool call you can only see a summary, but you also have a tool that can read the output of **any intermediate node**, enabling more complex tasks. + +## Examples (match this style) + +### Example 1: Show the plan + +**User:** +I want you to edit my footage into a travel vlog. + +**Assistant:** +Here’s the workflow I’m going to use: + +1. Load media: first load the footage and extract basic metadata. +2. Shot splitting: split the footage into shot-based segments. +3. Content understanding: generate text descriptions for each shot. +4. Clip filtering: keep the shots that fit a travel vlog. +5. Clip grouping: organize the footage into a good story order. +6. Script generation: write a script that fits the theme. +7. Element recommendations: add stylized captions, titles, effects, transitions, etc. to make it more engaging. +8. Voiceover generation: generate voiceover from the script. +9. Background music: pick a light, upbeat vlog-style track. +10. Timeline planning: fixed step. +11. Final render: fixed step. + +--- + +### Example 2: When no tool is needed, answer directly + +**User:** +What is “shot splitting”? + +**Assistant:** +Shot splitting means cutting the original video into smaller segments based on scene changes or content boundaries, so it’s easier to filter, reorder, and edit later. + +--- + +### Example 3: Second-round edit (remove voiceover) + +**User:** +The video you edited before has voiceover, but now I don’t want voiceover anymore. + +**Assistant:** +In this case, assistant need to call the voiceover tool and set the parameter `mode` to `skip`. + +--- +**Example 4: Unfilter** + +**User** +How could you have thrown away so much of my materials? I'm going to use all of them. + +**Assistant:** +At this point, the assistant needs to execute the filter_clips tool, and select skip as the mode parameter. \ No newline at end of file diff --git a/prompts/tasks/instruction/zh/system.md b/prompts/tasks/instruction/zh/system.md new file mode 100644 index 0000000000000000000000000000000000000000..a80972dcb515292f95801a4a9f16c06c51da78c8 --- /dev/null +++ b/prompts/tasks/instruction/zh/system.md @@ -0,0 +1,72 @@ +## 角色 +你是一个短视频剪辑助理。 + +## 任务 +- 你需要理解用户提出的需求，执行合适的工具完成剪辑，并避免向用户回复过于专业的剪辑术语。你会拿到一个关于剪辑的工具函数的描述列表。 +- 如果用户是第一次提出“帮我剪辑/处理素材”等类似需求，请先用自然语言列出你计划如何使用给定的剪辑工具及理由，待用户确认。 +- 注意你只能使用你可用的剪辑工具进行剪辑，如果工具能力范围超出了用户需求，请明确告诉用户你做不到。 +- 当用户指定一个剪辑风格时，优先查看工具里面是否有满足的技能（描述以【SKILL】为开头的工具），如果有匹配的技能，优先使用。 +- 整个剪辑流程中，有些节点是固定的，你无法改动；你计划的范围仅限于可以改动的节点。除非用户明确想要跳过某个步骤，否则在列出计划时，**尽可能使用多的工具以丰富视频内容**，除非用户明确指出不要某个元素。 +- 注意，有些节点依赖前面节点的结果，具体的依赖关系你可以在工具描述中看到，请在工具调用前检查依赖。工具会自己寻找依赖的结果，你不需要将前面节点的结果输入到工具参数中。如果工具需要输入参数，会在工具描述中另加说明，请填入合适的参数。 +- 重要：**每次只调用一个工具，不允许并行工具调用**。如果需要连续调用工具，每次调用完工具后，向用户简单总结本次工具调用的结果和下一步的意图，增强互动感，然后再进行下一次工具调用。 + +## 流程参考 +常规剪辑流程如下，这里每一步都对应一个或多个工具供你使用： +- 搜索素材 "search_media"（可跳过）。如果你发现用户并没有上传素材，可以提示用户你可以上网搜索素材。搜索素材后需要运行load_media工具才可以真正加载到素材。 +- 素材加载 "load_media"（固定）。用于获取输入素材的路径、长宽等基础信息。 +- 镜头切分 "split_shots"（可跳过）。将素材按镜头切分成片段。 +- 内容理解 "understand_clips"（可跳过）。 为每个片段(clips)生成一段描述（captions） +- 镜头筛选 "filter_clips"（可跳过）。根据用户要求，筛选出符合要求的片段(clips) +- 片段分组 "group_clips"（可跳过，但应默认运行）。根据用户要求，对片段进行排序和分组，组织合理的叙事逻辑，并辅助后续文案生成。 +- 文案生成 "generate_script"（可跳过）。根据用户要求，生成视频文案。 +- 元素推荐 （可跳过，但应默认运行）。根据用户要求，推荐花字、标题、特效、转场、配音音色等元素。 +- 配音生成 "generate_voiceover"（可跳过）。根据文案生成对应的配音。 +- 背景音乐选取 "select_BGM"（可跳过）。选择合适的背景音乐。 +- 组织时间线 "plan_timeline"（固定）。根据前面的视频片段、文案、语音和BGM，组织成合理的时间线。 +- 渲染成片。"render_video"（固定）。根据时间线渲染成片。 +此外，虽然你在工具调用后只能看到summary，但你有一个工具可以读取任意中间节点的输出。你可以用它完成更复杂的任务。 + +## 风格要求 +- 用简洁、口语化的语言 + +## 语言 +- 根据用户使用的语言进行回答 +- 如用户要求用英语、日语等回答，则使用相应语言 + + +## 示例 +示例 1：列出计划 +[用户]： +我想要你帮我把我的素材剪辑成旅行vlog， +[助手]： +我将要使用如下流程进行剪辑： +- 素材加载：需要先对输入素材建立索引，提取元信息。 +- 镜头切分：将素材按镜头切分成片段。 +- 内容理解：对每个镜头生成文字描述。 +- 镜头筛选：筛选出适合放在旅行vlog中的镜头。 +- 片段分组：组织一个合理的素材顺序。 +- 文案生成：配一段符合主题的文案。 +- 花字检索：添加花字元素，增强视觉效果。 +- 转场检索：添加合适的转场效果，让镜头切换更自然。 +- 配音生成：根据文案生成配音。 +- 音乐推荐：选择轻松愉快的vlog背景音乐。 +- 组织时间线：这是剪辑的固定流程。 +- 渲染成片：这也是剪辑的固定流程。 + +示例 2：不需要工具时，直接回答 +用户： +什么是“镜头切分”？ +助手： +镜头切分是指把原始视频按照画面内容或语义边界切分成若干个独立的镜头片段，通常会结合画面变化、音频变化等特征来判断切分点，用于后续剪辑、检索或分析。 + +示例3：取消配音 +用户： +之前你给我剪的视频有配音，我现在不想要视频的配音了。 + +此时助手需要执行generate_voiceover工具，参数mode选择skip。 + +示例4：取消筛选 +用户： +你怎么把我的素材丢掉了那么多，我要使用全部素材。 + +此时助手需要执行filter_clips工具，参数mode选择skip。 \ No newline at end of file diff --git a/prompts/tasks/scripts/en/omni_bgm_label.md b/prompts/tasks/scripts/en/omni_bgm_label.md new file mode 100644 index 0000000000000000000000000000000000000000..a5b94da5f71c0a20f4f4230a0f573536e0c8ecf6 --- /dev/null +++ b/prompts/tasks/scripts/en/omni_bgm_label.md @@ -0,0 +1,26 @@ +## Role + +You are a **music analysis expert**. + +## Task + +Please read (or understand) the music content I provide, then **output only a JSON object that matches the structure below**. Do not output anything else—no explanations, no extra text. +The JSON must include the following fields: + +```json +{ + "scene": [""], // Choose one or more best matches from ["Vlog","Travel","Relaxing","Emotion","Transition","Outdoor","Cafe","Evening","Scenery","Food","Date","Club"] (List) + "genre": [""], // Choose one or more best matches from ["Pop","BGM","Electronic","R&B/Soul","Hip Hop/Rap","Rock","Jazz","Folk","Classical","Chinese Style"] (List) + "mood": [""], // Choose one or more best matches from ["Dynamic","Chill","Happy","Sorrow","Romantic","Calm","Excited","Healing","Inspirational"] (List) + "lang": [""], // Choose the best match for lyric language or audio type from ["bgm","en","zh","ko","ja"] + "description": "" // One-sentence summary of the music overall—e.g., mood, suitable scenes, main instruments, etc. +} +``` + +Please make sure: + +* Every field has a concrete value (as strings) +* Do not add any extra fields +* Use natural language in `description` to briefly describe the music’s characteristics, e.g., “A light and upbeat electronic track, great for travel or daily vlogs, featuring synths and percussion.” + +Now, please analyze the music content below and output the JSON: diff --git a/prompts/tasks/scripts/en/script_template_label.md b/prompts/tasks/scripts/en/script_template_label.md new file mode 100644 index 0000000000000000000000000000000000000000..6dd7d58361ddec6c190e1d314326dd53625a1ed7 --- /dev/null +++ b/prompts/tasks/scripts/en/script_template_label.md @@ -0,0 +1,9 @@ +You will receive the main text of a "writing style template". Please output a JSON containing: + +description: Summarize the writing style and typical usage scenarios of this template in one sentence +tags Select 1 to 3 of the most relevant tags (multiple choices are allowed) from the following enumeration [Life, Food, Beauty, Entertainment, Travel, Tech, Business, Vehicle, Health, Family, Pets, Knowledge] +Requirement + +Only output JSON, no interpretation +The description should not exceed 30 characters +Don't exceed three tags \ No newline at end of file diff --git a/prompts/tasks/scripts/zh/omni_bgm_label.md b/prompts/tasks/scripts/zh/omni_bgm_label.md new file mode 100644 index 0000000000000000000000000000000000000000..f3e43bb62aa4f7b5db8c7c3ea5f473dd92773def --- /dev/null +++ b/prompts/tasks/scripts/zh/omni_bgm_label.md @@ -0,0 +1,18 @@ +你是一个**音乐分析专家**。 +请阅读（或理解）我给你的音乐内容，然后**仅输出满足下面结构的 JSON 对象**，不要输出其他内容、解释或额外文本。 +JSON 结构必须包含以下字段： +```json +{ + "scene": [""], // 从 ["Vlog","Travel","Relaxing","Emotion","Transition","Outdoor","Cafe","Evening","Scenery","Food","Date","Club"] 中选一个或多个最贴切的，List + "genre": [""], // 从 ["Pop","BGM","Electronic","R&B/Soul","Hip Hop/Rap","Rock","Jazz","Folk","Classical","Chinese Style"] 中选一个或多个最贴切的，List + "mood": [""], // 从 ["Dynamic","Chill","Happy","Sorrow","Romantic","Calm","Excited","Healing","Inspirational"] 中选一个或多个最贴切的，List + "lang": [""], // 从 ["bgm","en","zh","ko","ja"] 中选一个最贴合的歌词语言或音频类型 + "description": "" // 一句话简要描述音乐整体，例如情绪、适用场景、主要乐器等 +} +``` +请确保： +- 所有字段都有具体值（用字符串表示） +- 不要添加其他字段 +- description 用自然语言简洁描述音乐特点，例如“这是一首轻松愉快的电子乐，适合旅行或日常Vlog，主要有合成器和打击乐” + +现在请分析下面的音乐内容并输出 JSON： \ No newline at end of file diff --git a/prompts/tasks/scripts/zh/script_template_label.md b/prompts/tasks/scripts/zh/script_template_label.md new file mode 100644 index 0000000000000000000000000000000000000000..fc1b0d25a56d6a1d487cc031c2afdcac4e4c8312 --- /dev/null +++ b/prompts/tasks/scripts/zh/script_template_label.md @@ -0,0 +1,9 @@ +你将收到一段“文风模板”的正文内容。 请输出一个 JSON，包含： + +description：一句话概括该模板的写作风格和典型使用场景 +tags：从以下枚举中选择 1~3 个最相关的标签（可多选） [Life, Food, Beauty, Entertainment, Travel, Tech, Business, Vehicle, Health, Family, Pets, Knowledge] +要求： + +只输出 JSON，不要解释 +description 不超过 30 字 +tags 不要超过 3 个 \ No newline at end of file diff --git a/prompts/tasks/select_bgm/en/system.md b/prompts/tasks/select_bgm/en/system.md new file mode 100644 index 0000000000000000000000000000000000000000..97cba31a3014475e7976014ddd25b22ba0b1d671 --- /dev/null +++ b/prompts/tasks/select_bgm/en/system.md @@ -0,0 +1,5 @@ +## Role +You are a music selector + +## Task +Based on the music entries I provide (List[Dict] format), return the **exactly one** JSON file that best meets the user's requirements. \ No newline at end of file diff --git a/prompts/tasks/select_bgm/en/user.md b/prompts/tasks/select_bgm/en/user.md new file mode 100644 index 0000000000000000000000000000000000000000..5c75ec28d220e9c86eed0ee756d8b9b015259091 --- /dev/null +++ b/prompts/tasks/select_bgm/en/user.md @@ -0,0 +1,6 @@ + +Candidate music entries: +{{candidates}} + +user request: +{{user_request}} \ No newline at end of file diff --git a/prompts/tasks/select_bgm/zh/system.md b/prompts/tasks/select_bgm/zh/system.md new file mode 100644 index 0000000000000000000000000000000000000000..92432eef36afeada0dea7cd4fd325c47c14af5ff --- /dev/null +++ b/prompts/tasks/select_bgm/zh/system.md @@ -0,0 +1 @@ +你是一个音乐选择器，根据我提供的音乐条目（List[Dict]格式），返回最适合用户要求的**有且只有一条**json。 \ No newline at end of file diff --git a/prompts/tasks/select_bgm/zh/user.md b/prompts/tasks/select_bgm/zh/user.md new file mode 100644 index 0000000000000000000000000000000000000000..acaa3b3f9c596ea96b1efa4afa476d03ab0db0b9 --- /dev/null +++ b/prompts/tasks/select_bgm/zh/user.md @@ -0,0 +1,5 @@ + +候选音乐条目： +{{candidates}} +用户要求： +{{user_request}} \ No newline at end of file diff --git a/prompts/tasks/understand_clips/en/system_detail.md b/prompts/tasks/understand_clips/en/system_detail.md new file mode 100644 index 0000000000000000000000000000000000000000..154f034b8993cc895927db7a0de6587aa3fa7580 --- /dev/null +++ b/prompts/tasks/understand_clips/en/system_detail.md @@ -0,0 +1,31 @@ +You are a Vlog creator skilled in content understanding. Please perform a fine-grained content analysis and aesthetic quality evaluation of the **given video clip or image segment**. + +**1. Scene Summary Requirements (Caption)** +* **Content Dimension:** Focus on the main subject, subject actions, scene layout, environmental features (e.g., indoor/outdoor, day/night, weather), shooting perspective, and the overall mood of the frame. +* **Actions and Expressions:** Emphasize and describe the specific actions and facial expressions of subjects in the frame. +* **Multiple Scenes Handling:** If the video clip/image contains multiple different scenes, scene switches, or transitions, all scenes should be described, and the transition narrative should be smooth and natural. +* **Reality Constraint:** Strictly describe only what is visible in the video/image; do not imagine or fabricate unseen details. +* **Information Filtering:** Focus on the main subjects and key scene elements, omitting minor background details if needed, but do not omit any key subjects (people, animals, etc.) present in the scene. +* **Word Limit:** Description should be concise, limited to 100 words. + +**2. Aesthetic Quality Scoring Requirements (Aes_score)** +Please consider the following objective dimensions and provide a **floating-point score between 0.0 and 1.0 (rounded to two decimal places):** +* **Image Quality and Clarity:** Resolution clarity, richness of texture, presence of noise, mosaic, or compression artifacts, and focus accuracy (no blur or defocus). +* **Lighting and Color:** Exposure accuracy (no severe overexposure or underexposure), natural or artistic lighting, color fidelity, and white balance accuracy. +* **Composition and Subject Prominence:** Whether composition follows aesthetic principles (e.g., rule of thirds, centered composition), whether the subject is prominent without interference or obstruction from a cluttered background. +* **Stability and Camera Movement:** Whether camera motion is smooth (pans, tilts, zooms), and whether there is any disruptive shaking or chaotic movement. +* **Scoring Reference:** + * **0.80 - 1.00 (Excellent):** Extremely clear image, sophisticated lighting, professional composition, prominent subject, stable camera (cinematic/pro-level Vlog standard). + * **0.60 - 0.79 (Good):** Clear image, normal exposure, natural colors, decent composition, not outstanding but complete coverage (standard Vlog level). + * **0.40 - 0.59 (Average):** Main content is recognizable, but slight blur, shaking, poor lighting, or cluttered composition exist (raw footage level). + * **0.00 - 0.39 (Poor):** Severely blurred, extreme shaking, very dark or overexposed, subject unrecognizable (discarded footage level). + +**3. Output Format** +The output must strictly follow the following JSON structure (all keys must be present): +```json +{ + "caption": "Fine-grained content description within 100 words (as specific and objective as possible, do not invent unseen details)", + "aes_score": "Aesthetic quality score (float)" +} +``` +**Note**: For the given video clip or image, only output one JSON object. aes_score must be a numeric type. Do not output any explanatory text. \ No newline at end of file diff --git a/prompts/tasks/understand_clips/en/system_overall.md b/prompts/tasks/understand_clips/en/system_overall.md new file mode 100644 index 0000000000000000000000000000000000000000..fbe33fe8bda637ddcaf8952745ee3cd5ed6a0637 --- /dev/null +++ b/prompts/tasks/understand_clips/en/system_overall.md @@ -0,0 +1,14 @@ +## Role + +You are a short-form video editing media understanding assistant. Please reply in English. + +## Goal + +Based on the caption descriptions of multiple assets, generate an overall summary—especially focusing on the **overall narrative structure** formed by combining all assets. + +## What to include in the description + +1. **Main content**: What these assets are mainly about / what is happening +2. **People or objects**: Key people, primary subjects, important elements +3. **Scene**: Location/environment/time cues (e.g., indoors/outdoors, city/nature, day/night, etc.) +4. **Overall vibe**: Emotion/style/pacing (e.g., warm, tense, soothing, energetic, funny, etc.) \ No newline at end of file diff --git a/prompts/tasks/understand_clips/en/user_detail.md b/prompts/tasks/understand_clips/en/user_detail.md new file mode 100644 index 0000000000000000000000000000000000000000..98ba70de4e4cb1325fedd887bb259704f61761a8 --- /dev/null +++ b/prompts/tasks/understand_clips/en/user_detail.md @@ -0,0 +1 @@ +Please generate an English description for this clip. \ No newline at end of file diff --git a/prompts/tasks/understand_clips/en/user_overall.md b/prompts/tasks/understand_clips/en/user_overall.md new file mode 100644 index 0000000000000000000000000000000000000000..2ac03bdc85dfe886deb12091d7088d4fa34f5408 --- /dev/null +++ b/prompts/tasks/understand_clips/en/user_overall.md @@ -0,0 +1,2 @@ +Below are individual descriptions of several clips. Please summarize from an overall perspective: What story do these materials generally tell? Please summarize in a few sentences, and remain objective and neutral. +{{clips_captions}} \ No newline at end of file diff --git a/prompts/tasks/understand_clips/zh/system_detail.md b/prompts/tasks/understand_clips/zh/system_detail.md new file mode 100644 index 0000000000000000000000000000000000000000..04d0af0a53f4885e449da557a425f9938b0462ac --- /dev/null +++ b/prompts/tasks/understand_clips/zh/system_detail.md @@ -0,0 +1,31 @@ +你是一名擅长内容理解的Vlog视频博主，请根据给到的**局部视频片段/图像**，对画面进行细粒度内容分析与美学质量评估。 + +**1. 场景总结要求 (Caption)** +* **内容维度：** 重点涵盖主体、主体动作、场景布局、环境特征（如室内/室外、白天/夜晚、天气状况）、拍摄视角及画面情绪。 +* **动作与表情：** 必须重点捕捉并描述画面中主体的具体动作和面部表情。 +* **多场景处理：** 若视频片段/图像中出现多个不同场景、涉及场景切换或转场，需要保留所有场景的内容描述，且转场过渡文案要自然流畅。 +* **真实性约束：** 严格基于视频画面内容进行描述，禁止联想或编造未出现在画面中的细节。 +* **信息筛选：** 聚焦主要主体和核心场景信息，适当舍去不重要的边角细节，但不得遗漏场景中出现的关键主体（人物、动物等）。 +* **字数限制：** 描述需精炼，限制在100字以内。 + +**2. 美学质量打分要求 (Aes_score)** +请综合以下客观维度，给出一个**0.0 ~ 1.0**之间的浮点数分数（保留两位小数）： +* **画质与清晰度：** 画面分辨率是否高，纹理细节是否丰富，是否存在噪点、马赛克或明显的压缩痕迹，对焦是否准确（无模糊/虚焦）。 +* **光影与色彩：** 曝光是否准确（无严重过曝或死黑），光线是否自然或具有艺术感，色彩还原度是否高，白平衡是否准确。 +* **构图与主体显著性：** 构图是否符合美学标准（如三分法、中心构图），主体是否在画面中突出且未被杂乱背景干扰或遮挡。 +* **稳定性与运镜：** 镜头运动是否平滑流畅（如推拉摇移），是否存在影响观感的剧烈抖动或混乱运镜。 +* **评分参考标准：** + * **0.80 - 1.00 (优秀)：** 画面极度清晰，光影讲究，构图专业，主体突出，运镜稳定（电影感/专业Vlog水准）。 + * **0.60 - 0.79 (良好)：** 画面清晰，曝光正常，色彩自然，构图工整，虽无惊艳感但记录完整（标准Vlog水准）。 + * **0.40 - 0.59 (一般)：** 画面主要内容可辨，但存在轻微模糊、抖动、光线不佳或构图杂乱等瑕疵（素材级水准）。 + * **0.00 - 0.39 (较差)：** 画面严重模糊、剧烈抖动、极度昏暗或过曝，主体无法识别（废片水准）。 + +**3. 输出格式** +输出必须严格符合以下 JSON 结构（key 不可缺失）： +```json +{ + "caption": "100字以内细粒度内容描述（尽量具体客观，不要编造看不到的细节）", + "aes_score": "美学质量分数（float形式）" +} +``` +**注意：** 针对输入的视频片段/图像仅输出一段JSON格式的内容，`aes_score`为数字类型，无需输出其他任何解释性文字。 diff --git a/prompts/tasks/understand_clips/zh/system_overall.md b/prompts/tasks/understand_clips/zh/system_overall.md new file mode 100644 index 0000000000000000000000000000000000000000..759b9cacef3467aee2bb48ee6728d2b85b3a97aa --- /dev/null +++ b/prompts/tasks/understand_clips/zh/system_overall.md @@ -0,0 +1,2 @@ +你是一个短视频剪辑素材理解助手。 +你的任务是：根据给定的多个素材的文案描述，生成一段总体描述。重点包含：主体内容、人物或物体、场景和整体氛围，方便后续做高光检测和文案生成。不要输出客套话或多余说明。 \ No newline at end of file diff --git a/prompts/tasks/understand_clips/zh/user_detail.md b/prompts/tasks/understand_clips/zh/user_detail.md new file mode 100644 index 0000000000000000000000000000000000000000..9918bbd70abc845da1ad6f74efeecda8801ae17b --- /dev/null +++ b/prompts/tasks/understand_clips/zh/user_detail.md @@ -0,0 +1 @@ +请为以下输入的视频片段/图像，生成符合要求的细粒度内容分析与美学质量评估。 \ No newline at end of file diff --git a/prompts/tasks/understand_clips/zh/user_overall.md b/prompts/tasks/understand_clips/zh/user_overall.md new file mode 100644 index 0000000000000000000000000000000000000000..9a752571f19c13cf61f03da80ee2d77c34b7324e --- /dev/null +++ b/prompts/tasks/understand_clips/zh/user_overall.md @@ -0,0 +1,2 @@ +下面是若干clips的单体描述，请你从整体角度总结：这些素材大致讲述了怎样的故事？请用 1~2 段话总结，并保持客观、中性。 +{{clips_captions}} \ No newline at end of file diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000000000000000000000000000000000000..611ae359e92a5b58f00acd9d86c84ce2c21abc1d --- /dev/null +++ b/requirements.txt @@ -0,0 +1,22 @@ +fastapi==0.128.0 +uvicorn[standard]==0.40.0 +langchain-core==1.2.7 +mcp==1.26.0 +colorlog==6.9.0 +librosa==0.11.0 +transnetv2_pytorch==1.0.5 +langchain==1.2.4 +langchain_mcp_adapters==0.2.1 +langchain_openai==1.1.6 +ffmpeg-python==0.2.0 +aiofiles==23.2.1 +skillkit==0.4.0 +moviepy==2.2.1 +av==16.1.0 +langchain-community==0.4.1 +langchain-huggingface==1.2.0 +sentence-transformers==5.2.2 +tomli +faiss-cpu==1.13.2 +openai==2.16.0 +emoji==2.15.0 diff --git a/run.sh b/run.sh new file mode 100755 index 0000000000000000000000000000000000000000..295925e6b7c5d87ee54e85ec5d8bd3534b32317a --- /dev/null +++ b/run.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +set -e + +ROOT_DIR="$(cd "$(dirname "$0")" && pwd)" +export PYTHONPATH="$ROOT_DIR/src" + +HOST="${HOST:-0.0.0.0}" +PORT="${PORT:-7860}" + +python3 -m open_storyline.mcp.server & +MCP_PID=$! + +uvicorn agent_fastapi:app \ + --host "$HOST" \ + --port "$PORT" & +WEB_PID=$! + +trap 'kill $MCP_PID $WEB_PID' INT TERM + +wait \ No newline at end of file diff --git a/scripts/llm_script_template_label.py b/scripts/llm_script_template_label.py new file mode 100644 index 0000000000000000000000000000000000000000..b9d886dcf9b81a47ef65a0a4527365596f61d32e --- /dev/null +++ b/scripts/llm_script_template_label.py @@ -0,0 +1,143 @@ +# Please install OpenAI SDK first: `pip install openai` +import os +import argparse +import hashlib +import json +from openai import OpenAI +from tqdm import tqdm + +from src.open_storyline.utils.prompts import get_prompt +from src.open_storyline.utils.parse_json import parse_json_dict + +# ------------------------------- +# API client (DeepSeek / OpenAI compatible) +# ------------------------------- +API_KEY = os.environ.get("DEEPSEEK_API_KEY", "") + +client = None +if API_KEY: + client = OpenAI( + api_key=API_KEY, + base_url="https://api.deepseek.com/v1", + ) + +# ------------------------------- +# Utils +# ------------------------------- +def file_md5(path: str) -> str: + """Compute MD5 of file content.""" + md5 = hashlib.md5() + with open(path, "rb") as f: + for chunk in iter(lambda: f.read(4096), b""): + md5.update(chunk) + return md5.hexdigest() + + +def label_template(path: str, system_prompt: str) -> dict: + """Call LLM to label a single text template.""" + if not client: + raise RuntimeError("API client not initialized") + + with open(path, "r", encoding="utf-8") as f: + content = f.read() + + resp = client.chat.completions.create( + model="deepseek-chat", + messages=[ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": content}, + ], + stream=False, + ) + + return parse_json_dict(resp.choices[0].message.content) + + +# ------------------------------- +# Main +# ------------------------------- +def main(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--input_dir", + type=str, + default="resource/script_templates", + help="Folder containing .txt style templates", + ) + parser.add_argument( + "--output_json", + type=str, + default="resource/script_templates/meta.json", + help="Output meta.json path", + ) + args = parser.parse_args() + + input_dir = args.input_dir + output_json = args.output_json + + # Load existing meta.json (resume support) + if os.path.exists(output_json): + with open(output_json, "r", encoding="utf-8") as f: + meta_data = json.load(f) + else: + meta_data = [] + + # md5 -> item + md5_map = {item["id"]: item for item in meta_data} + + # Prompt + system_prompt = get_prompt("scripts.script_template_label", lang="zh") + + # Collect txt files + files = [] + for root, _, filenames in os.walk(input_dir): + for name in filenames: + if name.lower().endswith(".txt"): + files.append(os.path.join(root, name)) + + updated_meta = [] + needs_processing = False + + # resource 根目录（用于算相对路径） + resource_root = os.path.abspath(os.path.join(input_dir, "../..")) + + for file_path in tqdm(files, desc="Labeling templates", unit="file"): + md5 = file_md5(file_path) + + rel_path = os.path.relpath(file_path, start=resource_root).replace("\\", "/") + + # 未变化，直接复用 + if md5 in md5_map: + updated_meta.append(md5_map[md5]) + continue + + needs_processing = True + tqdm.write(f"Processing {rel_path} ...") + + if not client: + continue + + try: + res = label_template(file_path, system_prompt) + except Exception as e: + tqdm.write(f"⚠️ Failed on {rel_path}: {e}") + continue + + # 补充字段 + res["id"] = md5 + res["path"] = rel_path + + updated_meta.append(res) + + if not client and needs_processing: + print("⚠️ Warning: API key missing, new/changed templates were not labeled.") + + os.makedirs(os.path.dirname(output_json), exist_ok=True) + with open(output_json, "w", encoding="utf-8") as f: + json.dump(updated_meta, f, ensure_ascii=False, indent=2) + + print(f"✅ Done! meta.json saved to {output_json}") + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/scripts/omni_bgm_label.py b/scripts/omni_bgm_label.py new file mode 100644 index 0000000000000000000000000000000000000000..6f98fc4a391d5f6626839d1f650a5641ce28879b --- /dev/null +++ b/scripts/omni_bgm_label.py @@ -0,0 +1,163 @@ +import os +import sys +import argparse +import base64 +import hashlib +import json +from openai import OpenAI +from src.open_storyline.utils.prompts import get_prompt +from src.open_storyline.utils.parse_json import parse_json_dict +from tqdm import tqdm # progress bar + +# ------------------------------- +# Get API key from environment +# ------------------------------- +API_KEY = os.environ.get("QWEN_API_KEY", "") + +client = None +if API_KEY: + client = OpenAI( + api_key=API_KEY, + base_url="https://dashscope.aliyuncs.com/compatible-mode/v1", + ) + +# ------------------------------- +# Utility functions +# ------------------------------- +def file_md5(path: str) -> str: + """Compute MD5 hash of a file.""" + hash_md5 = hashlib.md5() + with open(path, "rb") as f: + for chunk in iter(lambda: f.read(4096), b""): + hash_md5.update(chunk) + return hash_md5.hexdigest() + + +def process_bgm(path: str, prompt_text: str) -> dict: + """Call Qwen3-Omni to generate JSON labels for a single audio file.""" + if not client: + raise RuntimeError("API client not initialized") # safety check + + with open(path, "rb") as f: + audio_bytes = f.read() + audio_b64 = base64.b64encode(audio_bytes).decode("utf-8") + + completion = client.chat.completions.create( + model="qwen3-omni-flash-2025-12-01", + messages=[ + { + "role": "user", + "content": [ + { + "type": "input_audio", + "input_audio": { + "data": f"data:audio/wav;base64,{audio_b64}", + "format": "wav" + } + }, + {"type": "text", "text": prompt_text} + ], + } + ], + modalities=["text"], + stream=True, + stream_options={"include_usage": True}, + ) + + # Concatenate streaming text + texts = [] + for chunk in completion: + if chunk.choices and chunk.choices[0].delta.content: + texts.append(chunk.choices[0].delta.content) + res = parse_json_dict("".join(texts)) + return res + +# ------------------------------- +# Main batch processing +# ------------------------------- +def main(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--input_dir", type=str, default="resource/bgms", help="BGM folder path" + ) + parser.add_argument( + "--output_json", type=str, default="resource/bgms/meta.json", help="Output JSON file" + ) + args = parser.parse_args() + + input_dir = args.input_dir + output_json = args.output_json + + # Load existing meta.json if exists + if os.path.exists(output_json): + with open(output_json, "r", encoding="utf-8") as f: + meta_data = json.load(f) + else: + meta_data = [] + + # Map MD5 -> dict for quick lookup + md5_map = {item["id"]: item for item in meta_data} + + # Get prompt + prompt_text = get_prompt("scripts.omni_bgm_label", lang="zh") + + # Scan audio files + files = [ + os.path.join(input_dir, f) + for f in os.listdir(input_dir) + if f.lower().endswith((".mp3", ".wav")) + ] + + updated_meta = [] + needs_processing = False # Flag to track if there are new/changed files + + # Iterate with progress bar + for file_path in tqdm(files, desc="Processing BGMs", unit="file"): + # Make path relative to 'resource/' folder + resource_root = os.path.join(os.path.dirname(output_json), "../../..") + rel_path = os.path.relpath(file_path, start=resource_root).replace("\\", "/") + md5 = file_md5(file_path) + + # Skip unchanged files + if md5 in md5_map: + updated_meta.append(md5_map[md5]) + continue + + # Mark that we have new/changed file + needs_processing = True + + # Display current file in progress bar + tqdm.write(f"Processing {rel_path} ...") + + # If no API key, warn once and skip processing + if not client: + continue # skip actual labeling, warning printed later + + # Try to process BGM safely + try: + res = process_bgm(file_path, prompt_text) + except Exception as e: + tqdm.write(f"⚠️ Error processing {rel_path}: {e}") + continue + + # Add path and id + res["path"] = rel_path + res["id"] = md5 + updated_meta.append(res) + + # Print warning if needed + if not client and needs_processing: + print( + "⚠️ Warning: OpenAI API key is empty. Omni model not available, cannot label new or changed BGM files." + ) + + # Save meta.json + os.makedirs(os.path.dirname(output_json), exist_ok=True) + with open(output_json, "w", encoding="utf-8") as f: + json.dump(updated_meta, f, ensure_ascii=False, indent=2) + + print(f"✅ Done! meta.json saved to {output_json}") + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/src/open_storyline/__init__.py b/src/open_storyline/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..a68927d6ca950577d845cea16247b0aee681c39f --- /dev/null +++ b/src/open_storyline/__init__.py @@ -0,0 +1 @@ +__version__ = "0.1.0" \ No newline at end of file diff --git a/src/open_storyline/agent.py b/src/open_storyline/agent.py new file mode 100644 index 0000000000000000000000000000000000000000..bc7d2c74406111f25c9a695c6babb20203d32aab --- /dev/null +++ b/src/open_storyline/agent.py @@ -0,0 +1,126 @@ +from dataclasses import dataclass, field +from datetime import timedelta +from typing import Optional, Any + + +from langchain.agents import create_agent +from langchain_openai import ChatOpenAI + +from langchain_mcp_adapters.callbacks import Callbacks +from langchain_mcp_adapters.client import MultiServerMCPClient + +from open_storyline.config import Settings +from open_storyline.storage.agent_memory import ArtifactStore +from open_storyline.nodes.node_manager import NodeManager +from open_storyline.mcp.hooks.chat_middleware import handle_tool_errors, on_progress, log_tool_request +from open_storyline.mcp.sampling_handler import make_sampling_callback +from open_storyline.skills.skills_io import load_skills + +@dataclass +class ClientContext: + cfg: Settings + session_id: str + media_dir: str + bgm_dir: str + outputs_dir: str + node_manager: NodeManager + chat_model_key: str # Chat model key + vlm_model_key: str = "" # VLM model key + pexels_api_key: Optional[str] = None + tts_config: Optional[dict] = None # TTS config at runtime + llm_pool: dict[tuple[str, bool], ChatOpenAI] = field(default_factory=dict) + lang: str = "zh" # Default language: Chinese + + +async def build_agent( + cfg: Settings, + session_id: str, + store: ArtifactStore, + tool_interceptors=None, + *, + llm_override: Optional[dict] = None, + vlm_override: Optional[dict] = None, +): + def _get(override: Optional[dict], key: str, default: Any) -> Any: + return (override.get(key) if isinstance(override, dict) and key in override else default) + + def _norm_url(u: str) -> str: + u = (u or "").strip() + return u.rstrip("/") if u else u + + # 1) LLM: use user input from form first, fall back to config.toml + llm_model = _get(llm_override, "model", cfg.llm.model) + llm_base_url = _norm_url(_get(llm_override, "base_url", cfg.llm.base_url)) + llm_api_key = _get(llm_override, "api_key", cfg.llm.api_key) + llm_timeout = _get(llm_override, "timeout", cfg.llm.timeout) + llm_temperature = _get(llm_override, "temperature", cfg.llm.temperature) + llm_max_retries = _get(llm_override, "max_retries", cfg.llm.max_retries) + + llm = ChatOpenAI( + model=llm_model, + base_url=llm_base_url, + api_key=llm_api_key, + default_headers={ + "api-key": llm_api_key, + "Content-Type": "application/json", + }, + timeout=llm_timeout, + temperature=llm_temperature, + streaming=True, + max_retries=llm_max_retries, + ) + + # 2) VLM: same priority as above + vlm_model = _get(vlm_override, "model", cfg.vlm.model) + vlm_base_url = _norm_url(_get(vlm_override, "base_url", cfg.vlm.base_url)) + vlm_api_key = _get(vlm_override, "api_key", cfg.vlm.api_key) + vlm_timeout = _get(vlm_override, "timeout", cfg.vlm.timeout) + vlm_temperature = _get(vlm_override, "temperature", cfg.vlm.temperature) + vlm_max_retries = _get(vlm_override, "max_retries", cfg.vlm.max_retries) + + vlm = ChatOpenAI( + model=vlm_model, + base_url=vlm_base_url, + api_key=vlm_api_key, + default_headers={ + "api-key": vlm_api_key, + "Content-Type": "application/json", + }, + timeout=vlm_timeout, + temperature=vlm_temperature, + max_retries=vlm_max_retries, + ) + + sampling_callback = make_sampling_callback(llm, vlm) + + connections = { + cfg.local_mcp_server.server_name: { + "transport": cfg.local_mcp_server.server_transport, + "url": cfg.local_mcp_server.url, + "timeout": timedelta(seconds=cfg.local_mcp_server.timeout), + "sse_read_timeout": timedelta(minutes=30), + "headers": {"X-Storyline-Session-Id": session_id}, + "session_kwargs": {"sampling_callback": sampling_callback}, + }, + } + + client = MultiServerMCPClient( + connections=connections, + tool_interceptors=tool_interceptors, + callbacks=Callbacks(on_progress=on_progress), + tool_name_prefix=True, + ) + + tools = await client.get_tools() + skills = await load_skills(cfg.skills.skill_dir) # Load skills + node_manager = NodeManager(tools) + + # 4) Use LangChain's agent runtime to handle the multi-turn tool calling loop + agent = create_agent( + model=llm, + tools=tools+skills, + middleware=[log_tool_request, handle_tool_errors], + store=store, + context_schema=ClientContext, + ) + return agent, node_manager \ No newline at end of file diff --git a/src/open_storyline/config.py b/src/open_storyline/config.py new file mode 100644 index 0000000000000000000000000000000000000000..386d16a3b67bd78195146063c5daf796acb26aae --- /dev/null +++ b/src/open_storyline/config.py @@ -0,0 +1,260 @@ +# open_storyline/configuration_utils.py +from __future__ import annotations +import os +from pathlib import Path +from typing import Any, Optional, Literal, List +import time + +try: + import tomllib +except ImportError: + print("Fail to import tomllib, try to import tomlis") + import tomli as tomllib + +from pydantic import BaseModel, ConfigDict, Field, ValidationInfo, computed_field, field_validator + + +def _resolve_relative_path_to_config_dir(v: Path, info: ValidationInfo) -> Path: + """ + Resolve relative paths based on config.toml's directory (not cwd). + + Requires the caller to pass config_dir in model_validate(..., context={"config_dir": }). + """ + ctx = info.context or {} + base = ctx.get("config_dir") + if not base: + return v + + v2 = v.expanduser() + if v2.is_absolute(): + return v2 + + base_dir = Path(base).expanduser() + return (base_dir / v2).resolve(strict=False) + + +def _resolve_paths_recursively(value: Any, info: ValidationInfo) -> Any: + """ + Recursively process Path objects in container types (list/tuple/set/dict). + """ + if value is None: + return None + + if isinstance(value, Path): + return _resolve_relative_path_to_config_dir(value, info) + + if isinstance(value, list): + return [_resolve_paths_recursively(v, info) for v in value] + + if isinstance(value, tuple): + return tuple(_resolve_paths_recursively(v, info) for v in value) + + if isinstance(value, set): + return {_resolve_paths_recursively(v, info) for v in value} + + if isinstance(value, dict): + return {k: _resolve_paths_recursively(v, info) for k, v in value.items()} + + return value + + +class ConfigBaseModel(BaseModel): + model_config = ConfigDict(extra="forbid") + + @field_validator("*", mode="after") + @classmethod + def _resolve_all_path_fields(cls, v: Any, info: ValidationInfo) -> Any: + # Allow explicitly disabling path resolution for specific fields: + # Field(..., json_schema_extra={"resolve_relative": False}) + if info.field_name: + field = cls.model_fields.get(info.field_name) + extra = (field.json_schema_extra or {}) if field else {} + if extra.get("resolve_relative") is False: + return v + + return _resolve_paths_recursively(v, info) + +class DeveloperConfig(ConfigBaseModel): + developer_mode: bool = False + default_llm: str = "deepseek-chat" + default_vlm: str = "qwen3-vl-8b-instruct" + chat_models_config: dict[str, dict[str, Any]] = Field(default_factory=dict) + print_context: bool = False + +class ProjectConfig(ConfigBaseModel): + media_dir: Path = Field(..., description="Media directory for input videos and images") + bgm_dir: Path = Field(..., description="Background music (BGM) directory") + outputs_dir: Path = Field(..., description="Output directory") + + @computed_field(return_type=Path) + @property + def blobs_dir(self) -> Path: + return self.outputs_dir + + +class LLMConfig(ConfigBaseModel): + model: str + base_url: str + api_key: str + timeout: float = 30.0 + temperature: Optional[float] = None + max_retries: int = 2 + + +class VLMConfig(ConfigBaseModel): + model: str + base_url: str + api_key: str + timeout: float = 20.0 + temperature: Optional[float] = None + max_retries: int = 2 + + +class MCPConfig(ConfigBaseModel): + server_name: str = "storyline" + server_cache_dir: str = "./storyline/.server_cache" + server_transport: Literal["stdio", "sse", "streamable-http"] = "streamable-http" + url_scheme: str = "http" + connect_host: str = "127.0.0.1" + port: int = Field(ge=1, le=65535) + path: str = "/mcp" + + json_response: bool = True + stateless_http: bool = False + + timeout: int = 600 + + available_node_pkgs: List[str] = [] + available_nodes: List[str] = [] + @property + def url(self) -> str: + return f"{self.url_scheme}://{self.connect_host}:{self.port}{self.path}" + +class SkillsConfig(ConfigBaseModel): + skill_dir: Path = Field(..., description="Skill directory.") + +class PexelsConfig(ConfigBaseModel): + pexels_api_key: str = "" + +class SplitShotsConfig(ConfigBaseModel): + transnet_weights: Path = Field(..., description="Path to transnet_v2 weights") + transnet_device: str = "cpu" + +class UnderstandClipsConfig(ConfigBaseModel): + sample_fps: float = 2.0 + max_frames: int = 64 + +class RecommendScriptTemplateConfig(ConfigBaseModel): + script_template_dir: Path = Field(..., description="Script template directory.") + script_template_info_path: Path = Field(..., description="Script template meta info path.") + +class GenerateVoiceoverConfig(ConfigBaseModel): + tts_provider_params_path: Path = Field(..., description="TTS provider config file path") + providers: dict[str, dict[str, Any]] = Field(default_factory=dict) + +class SelectBGMConfig(ConfigBaseModel): + sample_rate: int = 22050 + hop_length: int = 2048 + frame_length: int = 2048 + +class RecommendTextConfig(ConfigBaseModel): + font_info_path: Path = Field(..., description="Font info path.") + + +class PlanTimelineConfig(ConfigBaseModel): + beat_type_max: int = 1 # Maximum beat strength to use (e.g., in 4/4: 1,2,1,3 where 1=strongest, 3=weakest) + title_duration: int = 5000 # Title/intro duration in milliseconds + bgm_loop: bool = True # Allow background music looping + min_clip_duration: int = 500 # Minimum clip duration in milliseconds + + estimate_text_min: int = 1500 # Minimum subtitle on-screen time per group without TTS (ms) + estimate_text_char_per_sec: float = 6.0 # Estimated characters per second without TTS + + image_default_duration: int = 3000 # Default image duration in milliseconds + + group_margin_over_voiceover: int = 1000 # Visual extension beyond voiceover duration per group (ms) + +class PlanTimelineProConfig(ConfigBaseModel): + + min_single_text_duration: int = 200 + # Minimum duration (ms) for a single text label + + max_text_duration: int = 5000 + # Maximum duration (ms) for a single text sentence + + img_default_duration: int = 1500 + # Default display duration (ms) for an image clip + + min_group_margin: int = 1500 + # Minimum time margin (ms) between consecutive text groups / paragraphs + + max_group_margin: int = 2000 + # Maximum time margin (ms) between consecutive text groups / paragraphs + + min_clip_duration: int = 1000 + # Minimum allowed duration (ms) for a video clip + + tts_margin_mode: str = "random" + # Time margin strategy between consecutive TTS segments. + # One of: "random", "avg", "max", "min" + + min_tts_margin: int = 300 + # Minimum margin (ms) between the end of one TTS segment and the start of the next + + max_tts_margin: int = 400 + # Maximum margin (ms) between the end of one TTS segment and the start of the next + + text_tts_offset_mode: str = "random" + # Offset strategy between text appearance time and corresponding TTS start time. + # One of: "random", "avg", "max", "min" + + min_text_tts_offset: int = 0 + # Minimum offset (ms) between text appearance and TTS start + + max_text_tts_offset: int = 0 + # Maximum offset (ms) between text appearance and TTS start + + long_short_text_duration: int = 3000 + # Duration threshold (ms) used to classify text as long or short + + long_text_margin_rate: float = 0.0 + # Relative start margin rate for long text, applied against clip start time + + short_text_margin_rate: float = 0.0 + # Relative start margin rate for short text, applied against clip start time + + text_duration_mode: str = "with_tts" + # Text duration calculation mode. + # One of: "with_tts" (align with TTS duration), "with_clip" (align with clip duration) + + is_text_beats: bool = False + # Whether text start time should align with detected music beats + +class Settings(ConfigBaseModel): + developer: DeveloperConfig + project: ProjectConfig + + llm: LLMConfig + vlm: VLMConfig + + local_mcp_server: MCPConfig + + skills: SkillsConfig + search_media: PexelsConfig + split_shots: SplitShotsConfig + understand_clips: UnderstandClipsConfig + script_template: RecommendScriptTemplateConfig + generate_voiceover: GenerateVoiceoverConfig + select_bgm: SelectBGMConfig + recommend_text: RecommendTextConfig + plan_timeline: PlanTimelineConfig + plan_timeline_pro: PlanTimelineProConfig + + +def load_settings(config_path: str | Path) -> Settings: + p = Path(config_path).expanduser().resolve() + data = tomllib.loads(p.read_text(encoding="utf-8")) + return Settings.model_validate(data, context={"config_dir": p.parent}) + +def default_config_path() -> str: + return os.getenv("OPENSTORYLINE_CONFIG", "config.toml") diff --git a/src/open_storyline/mcp/__init__.py b/src/open_storyline/mcp/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/src/open_storyline/mcp/hooks/chat_middleware.py b/src/open_storyline/mcp/hooks/chat_middleware.py new file mode 100644 index 0000000000000000000000000000000000000000..2f46392ae19d8915e591f815ba2272c04da98cc1 --- /dev/null +++ b/src/open_storyline/mcp/hooks/chat_middleware.py @@ -0,0 +1,273 @@ +import contextvars +from typing import Callable, Optional, Any +import json +import uuid +import ast +import asyncio + +from open_storyline.config import Settings + +from langgraph.types import Command +from langchain.agents.middleware import wrap_tool_call, wrap_model_call +from langchain_core.messages import ToolMessage +from langchain_openai import ChatOpenAI +from langchain_mcp_adapters.interceptors import MCPToolCallRequest, MCPToolCallResult +from langchain_mcp_adapters.callbacks import CallbackContext +from langchain_core.callbacks import AsyncCallbackHandler + +CUSTOM_MODEL_KEY = "__custom__" + +_SENSITIVE_KEYS = { + "api_key", + "access_token", + "authorization", + "token", + "password", + "secret", + "x-api-key", + "apikey", +} + +# GUI 日志输出通道 +_MCP_LOG_SINK = contextvars.ContextVar("mcp_log_sink", default=None) +_MCP_ACTIVE_TOOL_CALL_ID = contextvars.ContextVar("mcp_active_tool_call_id", default=None) +def set_mcp_log_sink(sink: Optional[Callable[[dict], None]]): + return _MCP_LOG_SINK.set(sink) + +def reset_mcp_log_sink(token): + _MCP_LOG_SINK.reset(token) + + +def _norm_url(u: str) -> str: + u = (u or "").strip() + return u.rstrip("/") if u else u + +def _mask_secrets(obj: Any) -> Any: + """ + Recursive desensitization: Prevent keys/tokens from being printed to various places such as + the console, logs, tool traces, toolmessages, etc + """ + try: + if isinstance(obj, dict): + out = {} + for k, v in obj.items(): + if str(k).lower() in _SENSITIVE_KEYS: + out[k] = "***" + else: + out[k] = _mask_secrets(v) + return out + if isinstance(obj, list): + return [_mask_secrets(x) for x in obj] + if isinstance(obj, tuple): + return tuple(_mask_secrets(x) for x in obj) + return obj + except Exception: + return "***" + +def _make_chat_llm(cfg: Settings, model_name: str, streaming: bool) -> ChatOpenAI: + model_config = (cfg.developer.chat_models_config.get(model_name) or {}) + base_url = _norm_url(model_config.get("base_url") or "") + api_key = model_config.get("api_key") + return ChatOpenAI( + model=model_name, + base_url=base_url, + api_key=api_key, + default_headers={ + "api-key": api_key, + "Content-Type": "application/json", + }, + timeout=cfg.llm.timeout, + temperature=model_config.get("temperature", cfg.llm.temperature), + streaming=streaming, + ) + + +def _get_llm(cfg: Settings, llm_pool: dict[tuple[str, bool], ChatOpenAI], model_name: str, streaming: bool) -> ChatOpenAI: + hit = llm_pool.get((model_name, streaming)) + if hit: + return hit + new_llm = _make_chat_llm(cfg, model_name, streaming=streaming) + llm_pool[(model_name, streaming)] = new_llm + return new_llm + + +@wrap_tool_call +async def log_tool_request(request, handler): + sink = _MCP_LOG_SINK.get() + server_names = {"storyline"} + + def emit_event(x: str | dict): + if sink: + sink(x) + + tool_call_info = request.tool_call + tool_complete_name = tool_call_info.get("name", "") + + server_name, tool_name = "", tool_complete_name + for s in server_names: + prefix = f"{s}_" + if tool_complete_name.startswith(prefix): + server_name = s + tool_name = tool_complete_name[len(prefix):] + break + + meta_collector = request.runtime.context.node_manager + exclude = set(meta_collector.kind_to_node_ids.keys()) | { + "inputs", "artifacts_dir", "artifact_id", "blobs_dir", "meta_path", + "media_dir", "bgm_dir", "outputs_dir", "debug_dir", + } + + extracted_args = {} + if isinstance(tool_call_info.get("args", {}), dict): + for arg in tool_call_info["args"].keys(): + if arg not in exclude: + extracted_args[arg] = tool_call_info["args"].get(arg, "") + extracted_args = _mask_secrets(extracted_args) + + tool_call_id = tool_call_info.get("id", "") + if not tool_call_id: + tool_call_id = f"mcp_{uuid.uuid4().hex[:8]}" + tool_call_info["id"] = tool_call_id + + is_mcp_tool = isinstance(getattr(request.tool, "args_schema", None), dict) + + active_tok = _MCP_ACTIVE_TOOL_CALL_ID.set(tool_call_id) + + out = None + out_json = None + isError = False + summary = "" + + try: + emit_event({ + "type": "tool_start", + "tool_call_id": tool_call_id, + "server": server_name, + "name": tool_name, + "args": extracted_args, + }) + print(f"[Agent tool start] {server_name}.{tool_name} args={extracted_args}\n") + + out = await handler(request) + + additional_kwargs = getattr(out, "additional_kwargs", None) or {} + if additional_kwargs.get("isError") is True: + # only when skill failed + isError = True + summary = _mask_secrets(getattr(out, "content", str(out))) + + else: + if is_mcp_tool: + if additional_kwargs.get("mcp_raw_text") is True: + # mcp success + isError = False + summary = getattr(out, "content", "") + else: + # judge based on out.content.isError + out_json = ast.literal_eval(out.content) + isError = out_json.get("isError", False) + + if not isError: + summary = out_json.get("summary", {}).get("node_summary", "") + else: + summary = _mask_secrets(out.content) + + # Skill tool success + # it don't provide "isError" field + else: + isError = False + c = getattr(out, "content", "") + summary = f"skill_ok len={len(c) if isinstance(c, str) else 0}" + + finally: + _MCP_ACTIVE_TOOL_CALL_ID.reset(active_tok) + + # 结束日志 + if isError: + print(f"[Agent tool error] result:{summary}\n\n") + emit_event({ + "type": "tool_end", + "tool_call_id": tool_call_id, + "server": server_name, + "name": tool_name, + "is_error": True, + "summary": summary, + }) + else: + print(f"[Agent tool finished] result:{summary}\n\n") + emit_event({ + "type": "tool_end", + "tool_call_id": tool_call_id, + "server": server_name, + "name": tool_name, + "is_error": False, + "summary": _mask_secrets(summary), + }) + + return out + + +async def on_progress(progress: float, total: float | None, message: str| None, context: CallbackContext): + sink = _MCP_LOG_SINK.get() + if sink: + sink({ + "type": "tool_progress", + "tool_call_id": _MCP_ACTIVE_TOOL_CALL_ID.get(), + "server": context.server_name, + "name": context.tool_name, + "progress": progress, + "total": total, + "message": message, + }) + +@wrap_tool_call +async def handle_tool_errors(request, handler): + try: + out = await handler(request) + + if isinstance(out, Command): + return out.update.get('messages')[0] + + elif isinstance(out, MCPToolCallResult) and not isinstance(out.content, str): + return ToolMessage( + content=out.content[0].get("text", ""), + tool_call_id=out.tool_call_id, + name=out.name, + additional_kwargs={ + "isError": False, + "mcp_raw_text": True, + }, + ) + + return out + + except Exception as e: + tc = request.tool_call + safe_args = _mask_secrets(tc.get("args") or {}) + tool_name = tc.get("name", "") + + return ToolMessage( + content=( + "Tool call failed\n" + f"Tool name: {tool_name}\n" + f"Tool params: {safe_args}\n" + f"Error messege: {type(e).__name__}: {e}\n" + "If it is a parameter issue, please correct the parameters and call again. " + "If it is due to the lack of a preceding dependency, please call the preceding node first. " + "If you think it's an occasional error, please try to call it again; " + "If you think it's impossible to continue, please explain the reason to the user." + ), + tool_call_id=tc["id"], + name=tool_name, + additional_kwargs={ + "isError": True, + "error_type": type(e).__name__, + "error_message": str(e), + "safe_args": safe_args, + }, + ) + +class PrintStreamingTokens(AsyncCallbackHandler): + async def on_llm_new_token(self, token: str, **kwargs) -> None: + if token: + print(token, end="", flush=True) \ No newline at end of file diff --git a/src/open_storyline/mcp/hooks/node_interceptors.py b/src/open_storyline/mcp/hooks/node_interceptors.py new file mode 100644 index 0000000000000000000000000000000000000000..947edaf5452d213b80855e3a0c856eee28745468 --- /dev/null +++ b/src/open_storyline/mcp/hooks/node_interceptors.py @@ -0,0 +1,379 @@ +from collections import defaultdict +from typing import List, Any, Dict +import os +from pathlib import Path +import json +import traceback + + +from langchain_mcp_adapters.interceptors import MCPToolCallRequest +from langgraph.types import Command +from langchain_core.messages import ToolMessage, ToolCall +from langchain_core.tools import ToolException +from mcp.types import CallToolResult + +from open_storyline.nodes.node_manager import NodeManager + +from open_storyline.storage.file import FileCompressor +from open_storyline.utils.logging import get_logger + +logger = get_logger(__name__) + +def compress_payload_to_base64(payload: Dict[str,List[Any]]): + if not isinstance(payload, dict): + return payload + for key,value in payload.items(): + if isinstance(value, list) and all([isinstance(item, dict) for item in value]): + for item in value: + if 'path' in item.keys(): + path = item['path'] + compress_data = FileCompressor.compress_and_encode(path) + + item.update({ + "path": path, + "base64": compress_data.base64, + "md5": compress_data.md5 + }) + elif isinstance(value, dict): + compress_payload_to_base64(value) + +class ToolInterceptor: + + @staticmethod + async def inject_media_content_before( + request: MCPToolCallRequest, + handler, + ): + try: + tool_call_type = request.args.get('tool_call_type', 'auto') + # for default tool call + if tool_call_type!= 'auto': + request.args = request.args.get('args', {}) + + runtime = request.runtime + context = runtime.context + store = runtime.store + session_id = context.session_id + node_id = request.name + lang = context.lang + artifact_id = store.generate_artifact_id(node_id) + meta_collector: NodeManager = context.node_manager + input_data = defaultdict(list) + + def load_collected_data(collected_node, input_data, store): + """Load collected node data""" + for collect_kind, artifact_meta in collected_node.items(): + _, prior_node_output = store.load_result(artifact_meta.artifact_id) + compress_payload_to_base64(prior_node_output['payload']) + input_data[collect_kind] = prior_node_output['payload'] + + if node_id == 'load_media': + input_data['inputs'] = [] + media_dir = Path(context.media_dir) + for file_name in os.listdir(media_dir): + path = media_dir / file_name + if path.is_dir(): + continue + compress_data = FileCompressor.compress_and_encode(path) + input_data['inputs'].append( + { + "path": str(path.relative_to(os.getcwd())), + "base64": compress_data.base64, + "md5": compress_data.md5, + } + ) + elif node_id in list(meta_collector.id_to_tool.keys()): + # 1. Determine execution mode and dependency requirements + is_skip_mode = request.args.get('mode', 'auto') != 'auto' + require_kind = ( + meta_collector.id_to_default_require_prior_kind[node_id] + if is_skip_mode + else meta_collector.id_to_require_prior_kind[node_id] + ) + + # 2. Check if node is executable + collect_result = meta_collector.check_excutable(session_id, store, require_kind) + load_collected_data(collect_result['collected_node'], input_data, store) + + # 3. Handle missing dependencies + if not collect_result['excutable']: + missing_kinds = collect_result['missing_kind'] + node_ids_missing = [ + meta_collector.kind_to_node_ids[kind][0] + for kind in missing_kinds + ] + + logger.info( + f"`{node_id}` require kind missing `{missing_kinds}`, " + f"need to execute prerequisite nodes: {node_ids_missing}" + ) + + # 4. Recursively execute missing predecessor nodes + async def execute_missing_dependencies( + missing_kinds: List[str], + for_node_id: str, + depth: int = 0 + ): + """ + Recursively execute missing dependency nodes + + Args: + missing_kinds: List of missing dependency types + for_node_id: ID of the node currently resolving dependencies + depth: Recursion depth (used for log indentation) + """ + + if not missing_kinds: + return + + indent = " " * depth + logger.info(f"{indent}├─ Resolving dependencies for `{for_node_id}`: {missing_kinds}") + + for kind in missing_kinds: + success = False + candidates = meta_collector.kind_to_node_ids[kind] + + for miss_id in candidates: + try: + await execute_node_with_default_mode( + miss_id, + for_node_id=for_node_id, + depth=depth + ) + logger.info( + f"{indent}│ ✓ `{miss_id}` executed successfully for kind `{kind}`" + ) + success = True + break + except ToolException as e: + logger.warning( + f"{indent}│ ✗ `{miss_id}` failed: {str(e)}" + ) + continue + + if not success: + raise ToolException( + f"Cannot satisfy dependency `{kind}` required by `{for_node_id}`. " + f"All candidates failed: {candidates}" + ) + + async def execute_node_with_default_mode( + miss_id: str, + for_node_id: str, + depth: int = 0 + ): + """ + Execute specified node in default mode + + Args: + miss_id: ID of the node to execute + for_node_id: ID of the parent node requesting this execution + depth: Recursion depth + """ + indent = " " * depth + logger.info( + f"{indent}├─ [Default Mode] Executing `{miss_id}` " + f"(required by `{for_node_id}`)" + ) + + # Prepare tool invocation arguments + tool = meta_collector.get_tool(miss_id) + tool_call_input = { + 'artifact_id': store.generate_artifact_id(miss_id), + 'mode': 'default' + } + + # Verify dependencies for this node + default_require = meta_collector.id_to_default_require_prior_kind[miss_id] + default_collect_result = meta_collector.check_excutable( + session_id, store, default_require + ) + default_collect_result = meta_collector.check_excutable(session_id, store, default_require) + + # Recursively process dependencies + if default_collect_result['excutable']: + load_collected_data( + default_collect_result['collected_node'], + tool_call_input, + store + ) + logger.debug(f"{indent}│ Dependencies satisfied for `{miss_id}`") + else: + logger.info( + f"{indent}│ `{miss_id}` has missing dependencies: " + f"{default_collect_result['missing_kind']}" + ) + await execute_missing_dependencies( + default_collect_result['missing_kind'], + for_node_id=miss_id, # Pass miss node_id here + depth=depth + 1 # Increment recursion depth + ) + + # Invoke the tool + try: + output = await tool.arun( + ToolCall( + args=tool_call_input, + tool_call_type='default', + runtime=runtime + ) + ) + logger.info(f"{indent}└─ ✓ `{miss_id}` completed successfully") + return output + except Exception as e: + logger.error(f"{indent}└─ ✗ `{miss_id}` execution failed: {str(e)}") + raise ToolException(f"Failed to execute `{miss_id}`: {str(e)}") + + # Start executing missing dependencies + await execute_missing_dependencies(missing_kinds, for_node_id=node_id) + + # Collect dependencies again + collect_result = meta_collector.check_excutable(session_id, store, require_kind) + load_collected_data(collect_result['collected_node'], input_data, store) + else: + input_data['artifacts_dir'] = store.artifacts_dir + + new_req_args = { + 'artifact_id': artifact_id, + 'lang': lang, + } + new_req_args.update(request.args) + new_req_args.update(input_data) + + modified_request = request.override( + args=new_req_args + ) + return await handler(modified_request) + except Exception as e: + logger.error("[ToolInterceptor]"+ "".join(traceback.format_exception(e))) + raise + + @staticmethod + async def save_media_content_after( + request: MCPToolCallRequest, + handler, + ): + result = "" + """End agent run when task is marked complete.""" + try: + tool_call_result: CallToolResult = await handler(request) + client_ctx = request.runtime.context + + + result = tool_call_result.model_dump() + tool_result = json.loads(result['content'][0]['text']) + node_id = request.name + + artifact_id = tool_result['artifact_id'] + session_id = client_ctx.session_id + + store = request.runtime.store + + if not tool_result['isError']: + if node_id == 'search_media': + store.save_result( + session_id, + node_id, + tool_result, + Path(client_ctx.media_dir), + ) + else: + store.save_result( + session_id, + node_id, + tool_result, + ) + tool_call_id = request.runtime.tool_call_id + + if node_id == 'read_node_history': + tool_excute_result = tool_result['tool_excute_result'] + else: + tool_excute_result = {} + + return Command( + update={ + "messages": [ + ToolMessage(content={ + 'summary': { + 'node_summary': tool_result['summary'], + 'tool_excute_result': tool_excute_result + }, + 'isError': tool_result['isError'] + }, tool_call_id=tool_call_id) + ], + "status": "done" + }, + ) + except Exception as e: + logger.error("[ToolInterceptor]"+ "".join(traceback.format_exception(e))) + logger.error(f"Tool Call result: {result}") + raise + + @staticmethod + async def inject_tts_config(request: MCPToolCallRequest, handler): + """ + Interceptor: Injects runtime.context.tts_config parameters into request.args before invoking voiceover/TTS tools. + - tts_config: {"provider": "bytedance", "bytedance": {...}, "azure": {...}, ...} + """ + try: + tool_name = str(getattr(request, "name", "") or "") + args = getattr(request, "args", None) + + if "voiceover" not in tool_name or not isinstance(args, dict): + return await handler(request) + + runtime = getattr(request, "runtime", None) + ctx = getattr(runtime, "context", None) if runtime else None + tts_cfg = getattr(ctx, "tts_config", None) if ctx else None + if not isinstance(tts_cfg, dict): + return await handler(request) + + provider = str(tts_cfg.get("provider") or "").strip().lower() + + if not provider: + args.setdefault("provider", "302") + return await handler(request) + + args.setdefault("provider", provider) + + provider_cfg = tts_cfg.get(provider) + if isinstance(provider_cfg, dict): + for key, value in provider_cfg.items(): + if value is None: + continue + args.setdefault(key, str(value).strip()) + except Exception as e: + print(f"{e}") + pass + return await handler(request) + + @staticmethod + async def inject_pexels_api_key(request: MCPToolCallRequest, handler): + """ + Interceptor: Injects runtime.context.pexels_api_key into request.args before invoking media search tools. + - If pexels_api_key is empty/None: do nothing (tool will fall back to env var internally). + """ + try: + tool_name = str(getattr(request, "name", "") or "") + args = getattr(request, "args", None) + + if not isinstance(args, dict): + return await handler(request) + + if "search_media" not in tool_name: + return await handler(request) + + runtime = getattr(request, "runtime", None) + ctx = getattr(runtime, "context", None) if runtime else None + key = getattr(ctx, "pexels_api_key", None) if ctx else None + key = str(key or "").strip() + + if not key: + return await handler(request) + + args["pexels_api_key"] = key + + except Exception as e: + print(f"{e}") + pass + return await handler(request) diff --git a/src/open_storyline/mcp/register_tools.py b/src/open_storyline/mcp/register_tools.py new file mode 100644 index 0000000000000000000000000000000000000000..43290bd4a31260b1e2b2892a1d3713afee258c90 --- /dev/null +++ b/src/open_storyline/mcp/register_tools.py @@ -0,0 +1,191 @@ +from __future__ import annotations +from dataclasses import asdict +from typing import Annotated +from pydantic import BaseModel, Field +import inspect +import traceback + +from open_storyline.config import Settings +from open_storyline.skills.skills_io import dump_skills +from open_storyline.utils.register import NODE_REGISTRY + +from open_storyline.mcp.sampling_requester import make_llm +from open_storyline.nodes.core_nodes.base_node import BaseNode +from open_storyline.nodes.node_summary import NodeSummary +from open_storyline.nodes.node_state import NodeState +from src.open_storyline.storage.agent_memory import ArtifactStore + +from mcp.server.fastmcp import Context, FastMCP +from mcp.server.session import ServerSession + +def create_tool_wrapper(node: BaseNode, input_schema: type[BaseModel]): + """ + Factory function: Convert custom Node to MCP Tool function + """ + # Get metadata for @server.tool parameters + meta = node.meta if hasattr(node, 'meta') else None + + async def wrapper(mcp_ctx: Context, **kwargs) -> dict: + # 1. Unified handling of context and Session + request = mcp_ctx.request_context.request + headers = request.headers + session_id = headers.get('X-Storyline-Session-Id') + + # 2. Session lifecycle management + session_manager = mcp_ctx.request_context.lifespan_context + if hasattr(session_manager, 'cleanup_expired_sessions'): + session_manager.cleanup_expired_sessions(session_id) + + # 3. Construct parameters + # Note: FastMCP automatically injects parameters into kwargs, merge them here + req_json = await request.json() + params = kwargs.copy() + params.update(req_json.get('params', {}).get('arguments', {})) + + node_state = NodeState( + session_id=session_id, + artifact_id=params['artifact_id'], + lang=params.get('lang', 'zh'), + node_summary=NodeSummary(), + llm=make_llm(mcp_ctx), + mcp_ctx=mcp_ctx, + ) + result = await node(node_state, **params) + return result + + + new_params = [] + # First parameter is fixed as mcp_ctx + new_params.append( + inspect.Parameter( + 'mcp_ctx', + inspect.Parameter.POSITIONAL_OR_KEYWORD, + annotation=Context + ) + ) + + new_annotations = {'mcp_ctx': Context} + + if input_schema: + for field_name, field_info in input_schema.model_fields.items(): + # Use Annotated to carry description, FastMCP will recognize it as JSON Schema description + annotation = Annotated[field_info.annotation, field_info] + + new_params.append( + inspect.Parameter( + field_name, + inspect.Parameter.KEYWORD_ONLY, + default=field_info.default if field_info.default is not ... else inspect.Parameter.empty, + annotation=annotation + ) + ) + new_annotations[field_name] = annotation + + wrapper.__name__ = meta.name + wrapper.__doc__ = meta.description + wrapper.__signature__ = inspect.Signature(new_params) + wrapper.__annotations__ = new_annotations + + return wrapper, meta + + +def register(server: FastMCP, cfg: Settings) -> None: + + # scan node packages + for pkg in cfg.local_mcp_server.available_node_pkgs: + NODE_REGISTRY.scan_package(pkg) + all_node_classes = [NODE_REGISTRY.get(name=node_name) for node_name in cfg.local_mcp_server.available_nodes] + + for NodeClass in all_node_classes: + node_instance = NodeClass(cfg) + input_schema = node_instance.input_schema + + tool_func, meta = create_tool_wrapper(node_instance, input_schema) + + tool_name = NodeClass.meta.name + tool_desc = NodeClass.meta.description + + # Register using server.tool decorator + server.tool( + name=tool_name, + description=tool_desc, + meta=asdict(meta) + )(tool_func) + + # Register special tools (e.g., read_node_history) + # local tool + @server.tool( + name="read_node_history", + description="Retrieve the execution result of any node using its artifact_id" + ) + async def mcp_read_history( + mcp_ctx: Context[ServerSession, object], + query_artifact_id: Annotated[str, Field(description="The artifact_id used to retrieve the corresponding JSON")] + ) -> dict: + node_summary = NodeSummary() + request = mcp_ctx.request_context.request + session_id = mcp_ctx.request_context.request.headers['X-Storyline-Session-Id'] + req_json_content = await request.json() + params = req_json_content['params'].get('arguments', {'query_artifact_id': query_artifact_id}) + params['session_id'] = session_id + params['node_summary'] = node_summary + + try: + store = ArtifactStore(artifacts_dir=params.get('artifacts_dir', ".storyline/.server_cache"), session_id=session_id) + meta, data = store.load_result(params['query_artifact_id']) + summary = "History information retrieved successfully" + isError = False + except Exception as e: + traceback_info = ''.join(traceback.format_exception(e)) + summary = f"History read execution failed: {params['query_artifact_id']}\n {traceback_info}", + meta, data, isError = 'None', 'None', True + + return { + 'artifact_id': params.get('artifact_id', store.generate_artifact_id(req_json_content['params'].get('name', 'read_node_history'))), + 'tool_excute_result': { + "history": { + "meta": meta, + "node_data": data, + } + }, + 'summary': summary, + 'isError': isError, + } + + @server.tool( + name="write_skills", + description="Save the generated Agent Skill (Markdown format) to the file system and return the absolute file path on success." + ) + async def mcp_write_skills( + mcp_ctx: Context[ServerSession, object], + skill_name: Annotated[str, Field(description="Skill file name, e.g., 'fast_paced_vlog', without extension")], + skill_dir: Annotated[str, Field(description="Skill storage directory, defaults to '.storyline/skills/'")] = '.storyline/skills/', + skill_content: Annotated[str, Field(description="Skill content in Markdown format")] = '', + ) -> dict: + """ + Receives LLM-generated skill content and saves it as a local MD file. + """ + node_summary = NodeSummary() + request = mcp_ctx.request_context.request + session_id = request.headers.get('X-Storyline-Session-Id', 'unknown_session') + req_json_content = await request.json() + params = req_json_content.get('params', {}).get('arguments', {'skill_name': skill_name, 'skill_dir': skill_dir, 'skill_content': skill_content}) + params['session_id'] = session_id + params['node_summary'] = node_summary + + res = await dump_skills( + skill_name=skill_name, + skill_dir=skill_dir, + skill_content=skill_content, + ) + node_summary.info_for_llm("[Write Skills] Done.") + store = ArtifactStore(artifacts_dir=params.get('artifacts_dir', ".storyline/.server_cache"), session_id=session_id) + + return { + 'artifact_id': params.get('artifact_id', store.generate_artifact_id(req_json_content.get('params', {}).get('name', 'mcp_write_skills'))), + 'tool_excute_result': { + }, + 'summary': "", + 'isError': False, + } + diff --git a/src/open_storyline/mcp/sampling_handler.py b/src/open_storyline/mcp/sampling_handler.py new file mode 100644 index 0000000000000000000000000000000000000000..fd2bc6eef9ee9803a208dced0048d7d69ac4e3c4 --- /dev/null +++ b/src/open_storyline/mcp/sampling_handler.py @@ -0,0 +1,432 @@ +import asyncio +import os +import math +import base64 +from io import BytesIO +from typing import Any, Dict, List, Tuple +from urllib.parse import urlparse + +from PIL import Image +from moviepy.video.io.VideoFileClip import VideoFileClip +from langchain_core.messages import SystemMessage, HumanMessage, AIMessage + +from mcp.types import CreateMessageRequestParams, CreateMessageResult, TextContent + + +# ----------------------------- +# Configurable parameters: Control multimodal input size +# ----------------------------- +DEFAULT_RESIZE_EDGE = 600 +DEFAULT_JPEG_QUALITY = 80 +DEFAULT_MIN_FRAMES = 2 +DEFAULT_MAX_FRAMES = 6 +DEFAULT_FRAMES_PER_SEC = 3.0 +GLOBAL_MAX_IMAGE_BLOCKS = 48 # Maximum total images allowed (video frames + images) to prevent payload overflow + +IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".gif", ".tiff"} +VIDEO_EXTS = {".mp4", ".mov", ".mkv", ".avi", ".webm", ".m4v"} + + +def _is_data_url(u: str) -> bool: + return isinstance(u, str) and u.startswith("data:") + + +def _is_http_url(u: str) -> bool: + return isinstance(u, str) and (u.startswith("http://") or u.startswith("https://")) + + +def _strip_file_scheme(u: str) -> str: + if not isinstance(u, str): + return str(u) + if u.startswith("file://"): + parsed = urlparse(u) + return parsed.path + return u + + +def _guess_ext(path_or_url: str) -> str: + try: + p = urlparse(path_or_url).path if _is_http_url(path_or_url) else path_or_url + return os.path.splitext(p)[1].lower() + except Exception: + return "" + + +def _resize_long_edge(img: Image.Image, long_edge: int) -> Image.Image: + if long_edge <= 0: + return img + w, h = img.size + le = max(w, h) + if le <= long_edge: + return img + scale = long_edge / float(le) + nw = max(1, int(round(w * scale))) + nh = max(1, int(round(h * scale))) + return img.resize((nw, nh), Image.LANCZOS) + + +def _pil_to_data_url(img: Image.Image, resize_edge: int, jpeg_quality: int) -> str: + img = img.convert("RGB") + img = _resize_long_edge(img, resize_edge) + buf = BytesIO() + img.save(buf, format="JPEG", quality=jpeg_quality, optimize=True) + b64 = base64.b64encode(buf.getvalue()).decode("utf-8") + return f"data:image/jpeg;base64,{b64}" + + +def _image_path_to_data_url(path: str, resize_edge: int, jpeg_quality: int) -> str: + img = Image.open(path) + return _pil_to_data_url(img, resize_edge, jpeg_quality) + + +def _choose_num_frames(duration_sec: float, min_frames: int, max_frames: int, frames_per_sec: float) -> int: + duration_sec = max(0.0, float(duration_sec)) + n = int(math.ceil(duration_sec * frames_per_sec)) + n = max(min_frames, n) + n = min(max_frames, n) + return n + + +def _sample_video_segment_to_data_urls( + video_path: str, + in_sec: float, + out_sec: float, + resize_edge: int, + jpeg_quality: int, + min_frames: int, + max_frames: int, + frames_per_sec: float, +) -> List[Tuple[float, str]]: + """ + Sample frames only from the [in_sec, out_sec] segment. Returns (rel_t_from_in, data_url) + """ + + in_sec = float(in_sec) + out_sec = float(out_sec) + + clip = VideoFileClip(video_path, audio=False) + try: + vdur = float(clip.duration or 0.0) + + # If duration is unavailable, conservatively sample one frame to avoid out_sec exceeding bounds + if vdur <= 0: + t = max(0.0, in_sec) + frame = clip.get_frame(t) + img = Image.fromarray(frame) + return [(0.0, _pil_to_data_url(img, resize_edge, jpeg_quality))] + + # Clamp to valid range + in_sec = max(0.0, min(in_sec, vdur)) + out_sec = max(0.0, min(out_sec, vdur)) + + # If still invalid, fallback to one frame at in_sec + if out_sec <= in_sec: + frame = clip.get_frame(in_sec) + img = Image.fromarray(frame) + return [(0.0, _pil_to_data_url(img, resize_edge, jpeg_quality))] + + seg_dur = out_sec - in_sec + n = _choose_num_frames(seg_dur, min_frames, max_frames, frames_per_sec) + + # Sample at bucket centers to avoid boundary frames + times = [((i + 0.5) / n) * seg_dur for i in range(n)] + + out: List[Tuple[float, str]] = [] + for rel_t in times: + abs_t = in_sec + rel_t + frame = clip.get_frame(abs_t) + img = Image.fromarray(frame) + out.append((rel_t, _pil_to_data_url(img, resize_edge, jpeg_quality))) + return out + finally: + clip.close() + + +def _extract_text_from_mcp_content(content: Any) -> str: + if content is None: + return "" + blocks = content if isinstance(content, list) else [content] + texts: List[str] = [] + for b in blocks: + if getattr(b, "type", None) == "text": + texts.append(getattr(b, "text", "") or "") + return "\n".join([t for t in (x.strip() for x in texts) if t]) + + +def _extract_text_from_lc_response(resp: Any) -> str: + content = getattr(resp, "content", None) + if isinstance(content, str): + return content.strip() + if isinstance(content, list): + texts = [] + for blk in content: + if isinstance(blk, dict) and blk.get("type") == "text": + texts.append(str(blk.get("text", "")).strip()) + return "\n".join([t for t in texts if t]).strip() + return str(resp).strip() + + +def _normalize_media_items(media_inputs: List[Any]) -> List[Dict[str, Any]]: + """ + Supports three input formats: + 1) "path/to/video.mp4" + 2) {"url"/"path": "...", "in_sec": 1.2, "out_sec": 3.4} + 3) ("path/to/video.mp4", 1.2, 3.4) # optional + Output normalized to: {"url": "...", "in_sec": optional, "out_sec": optional} + """ + out = [] + for item in media_inputs or []: + if isinstance(item, str): + out.append({"url": item}) + continue + + if isinstance(item, (list, tuple)) and len(item) >= 1: + d = {"url": item[0]} + if len(item) >= 2: + d["in_sec"] = item[1] + if len(item) >= 3: + d["out_sec"] = item[2] + out.append(d) + continue + + if isinstance(item, dict): + url = item.get("url") or item.get("path") or item.get("media") + if not url: + continue + d = {"url": url} + if "in_sec" in item: + d["in_sec"] = item.get("in_sec") + if "out_sec" in item: + d["out_sec"] = item.get("out_sec") + out.append(d) + continue + + return out + + +def _build_media_blocks( + media_inputs: List[Any], + resize_edge: int, + jpeg_quality: int, + min_frames: int, + max_frames: int, + frames_per_sec: float, + global_max_images: int, +) -> List[Dict[str, Any]]: + """ + Convert media_inputs to OpenAI-compatible multimodal blocks. + Videos: Sample frames by segment (if in_sec/out_sec provided) or entire video (legacy format) + """ + + blocks: List[Dict[str, Any]] = [] + img_count = 0 + + items = _normalize_media_items(media_inputs) + + for idx, mi in enumerate(items): + if img_count >= global_max_images: + break + + raw_url = _strip_file_scheme(str(mi.get("url"))) + ext = _guess_ext(raw_url) + + in_sec = mi.get("in_sec") + out_sec = mi.get("out_sec") + has_segment = (in_sec is not None and out_sec is not None) + + # 1) Data URL (image) - pass through directly + if _is_data_url(raw_url): + blocks.append({"type": "text", "text": f"Media {idx+1}: (data url image)"}) + blocks.append({"type": "image_url", "image_url": {"url": raw_url}}) + img_count += 1 + continue + + # 2) Remote URL: Images can be passed through; remote videos cannot be sampled locally (provide notice) + if _is_http_url(raw_url): + if ext in VIDEO_EXTS: + seg_info = f" segment [{in_sec},{out_sec}]s" if has_segment else "" + blocks.append({"type": "text", "text": f"Media {idx+1}: remote video url{seg_info} (cannot sample frames locally): {raw_url}"}) + continue + blocks.append({"type": "text", "text": f"Media {idx+1}: {raw_url}"}) + blocks.append({"type": "image_url", "image_url": {"url": raw_url}}) + img_count += 1 + continue + + # 3) Local path + path = raw_url + if not os.path.exists(path): + blocks.append({"type": "text", "text": f"Media {idx+1}: (missing file) {path}"}) + continue + + # Image + if ext in IMAGE_EXTS: + data_url = _image_path_to_data_url(path, resize_edge, jpeg_quality) + blocks.append({"type": "text", "text": f"Media {idx+1}: image file {os.path.basename(path)}"}) + blocks.append({"type": "image_url", "image_url": {"url": data_url}}) + img_count += 1 + continue + + # Video (supports segmented sampling) + if ext in VIDEO_EXTS: + if has_segment: + in_s = float(in_sec) + out_s = float(out_sec) + else: + # Legacy format: entire video. Use [0, +inf], internally clamped to duration + in_s = 0.0 + out_s = 1e12 + + frames = _sample_video_segment_to_data_urls( + video_path=path, + in_sec=in_s, + out_sec=out_s, + resize_edge=resize_edge, + jpeg_quality=jpeg_quality, + min_frames=min_frames, + max_frames=max_frames, + frames_per_sec=frames_per_sec, + ) + + if has_segment: + blocks.append({"type": "text", "text": f"Media {idx+1}: video segment {os.path.basename(path)} [{in_s:.2f}s, {out_s:.2f}s] (sampled frames in time order)"}) + else: + blocks.append({"type": "text", "text": f"Media {idx+1}: video file {os.path.basename(path)} (sampled frames in time order)"}) + + for fi, (rel_t, data_url) in enumerate(frames): + if img_count >= global_max_images: + break + blocks.append({"type": "text", "text": f"Frame {fi+1}/{len(frames)} (t≈{rel_t:.2f}s from segment start)"}) + blocks.append({"type": "image_url", "image_url": {"url": data_url}}) + img_count += 1 + continue + + blocks.append({"type": "text", "text": f"Media {idx+1}: unsupported file type: {path}"}) + + return blocks + + +def make_sampling_callback( + llm, + vlm, + *, + resize_edge: int = DEFAULT_RESIZE_EDGE, + jpeg_quality: int = DEFAULT_JPEG_QUALITY, + min_frames: int = DEFAULT_MIN_FRAMES, + max_frames: int = DEFAULT_MAX_FRAMES, + frames_per_sec: float = DEFAULT_FRAMES_PER_SEC, + global_max_images: int = GLOBAL_MAX_IMAGE_BLOCKS, +): + """ + Callback for MCP server sampling requests within tools: + - Reads metadata.media_urls (supports in_sec/out_sec) + - Samples frames and constructs LangChain multimodal messages + - Selects llm/vlm based on presence of media input + """ + + async def sampling_callback(context, params: CreateMessageRequestParams) -> CreateMessageResult: + try: + # 1. System prompt + system_prompt = getattr(params, "systemPrompt", None) or "" + + # 2. MCP messages (multi-turn) -> LangChain + mcp_messages = getattr(params, "messages", []) or [] + lc_messages: List[Any] = [] + if system_prompt: + lc_messages.append(SystemMessage(content=system_prompt)) + + # 3. Metadata: Extract media_urls and top_p + metadata = getattr(params, "metadata", None) or {} + media_inputs = list(metadata.get("media", []) or []) + top_p: float = float(metadata.get("top_p", 0.9)) + + temperature: float = float(getattr(params, "temperature", None) or 0.6) + max_tokens: int = int(getattr(params, "maxTokens", 4096) or 4096) + + # 4. Route to appropriate model + use_multimodal = bool(media_inputs) + model = vlm if use_multimodal else llm + if model is None: + model = vlm or llm + + # 5. Build media blocks (including video segment sampling) - run in thread to avoid blocking event loop + media_blocks: List[Dict[str, Any]] = [] + if use_multimodal: + media_blocks = await asyncio.to_thread( + _build_media_blocks, + media_inputs, + resize_edge, + jpeg_quality, + min_frames, + max_frames, + frames_per_sec, + global_max_images, + ) + + # 6. Attach media to "last user message" + user_indices = [i for i, m in enumerate(mcp_messages) if getattr(m, "role", "") == "user"] + last_user_idx = user_indices[-1] if user_indices else None + + if not mcp_messages: + # No messages - create a user message + content_blocks = [{"type": "text", "text": ""}] + if media_blocks: + content_blocks.extend(media_blocks) + lc_messages.append(HumanMessage(content=content_blocks if media_blocks else "")) + else: + for i, m in enumerate(mcp_messages): + role = getattr(m, "role", "") or "user" + text = _extract_text_from_mcp_content(getattr(m, "content", None)) + + if role == "assistant": + lc_messages.append(AIMessage(content=text)) + continue + + if role == "user": + if last_user_idx is not None and i == last_user_idx and media_blocks: + content_blocks = [{"type": "text", "text": text}] + content_blocks.extend(media_blocks) + lc_messages.append(HumanMessage(content=content_blocks)) + else: + lc_messages.append(HumanMessage(content=text)) + continue + + lc_messages.append(HumanMessage(content=text)) + + # 7. Invoke selected model + bound = model + model_name = getattr(model, "model", None) or getattr(model, "model_name", None) or "unknown" + try: + bound = bound.bind(temperature=temperature, max_tokens=max_tokens, top_p=top_p) + except Exception: + bound = bound.bind(temperature=temperature, max_tokens=max_tokens) + + try: + if hasattr(bound, "ainvoke"): + resp = await bound.ainvoke(lc_messages) + else: + resp = await asyncio.to_thread(bound.invoke, lc_messages) + except TypeError: + # Edge case: some wrappers don't accept max_tokens/top_p + bound2 = model.bind(temperature=temperature) + if hasattr(bound2, "ainvoke"): + resp = await bound2.ainvoke(lc_messages) + else: + resp = await asyncio.to_thread(bound2.invoke, lc_messages) + + text_out = _extract_text_from_lc_response(resp) + + return CreateMessageResult( + content=TextContent(type="text", text=text_out), + model=str(model_name), + role="assistant", + stopReason="endTurn", + ) + except Exception as e: + return CreateMessageResult( + content=TextContent(type="text", text=f"{type(e)}: {e}"), + model=str(model_name), + role="assistant", + stopReason="error", + ) + + return sampling_callback diff --git a/src/open_storyline/mcp/sampling_requester.py b/src/open_storyline/mcp/sampling_requester.py new file mode 100644 index 0000000000000000000000000000000000000000..963ed2966d8d8b037aaf6d51318c443b7909a3fb --- /dev/null +++ b/src/open_storyline/mcp/sampling_requester.py @@ -0,0 +1,165 @@ +from __future__ import annotations + +from typing import Any, Optional, Protocol, runtime_checkable + +from mcp.server.fastmcp import Context +from mcp.server.session import ServerSession +from mcp.types import SamplingMessage, TextContent, ModelHint, ModelPreferences + +from open_storyline.utils.emoji import EmojiManager + + +class BaseLLMSampling(Protocol): + # Low-level protocol: Sampling shared across multiple tools + async def sampling( + self, + *, + system_prompt: str | None, + messages: list[SamplingMessage], + temperature: float = 0.3, + top_p: float = 0.9, + max_tokens: int = 4096, + model_preferences: dict[str, Any] | None = None, + metadata: dict[str, Any] | None = None, + stop_sequences: list[str] | None = None, + ) -> str: + ... + +@runtime_checkable +class LLMClient(Protocol): + # High-level protocol: Tools are distinguished only by multimodal capability requirement + async def complete( + self, + *, + system_prompt: str | None, + user_prompt: str, + media: list[dict[str, Any]] | None = None, + temperature: float = 0.3, + top_p: float = 0.9, + max_tokens: int = 2048, + model_preferences: dict[str, Any] | None = None, + metadata: dict[str, Any] | None = None, + stop_sequences: list[str] | None = None, + ) -> str: + ... + +class MCPSampler(BaseLLMSampling): + def __init__(self, mcp_ctx: Context[ServerSession, object]): + self._mcp_ctx = mcp_ctx + + def _to_mcp_model_preferences( + self, + model_preferences: dict[str, Any] | None, + ) -> Optional[ModelPreferences]: + if not model_preferences: + return None + + raw_hints = model_preferences.get("hints") + hints: list[ModelHint] | None = None + if isinstance(raw_hints, list): + hints = [] + for h in raw_hints: + if isinstance(h, ModelHint): + hints.append(h) + elif isinstance(h, dict): + hints.append(ModelHint(**h)) + elif isinstance(h, str): + hints.append(ModelHint(name=h)) + + return ModelPreferences( + hints=hints, + costPriority=model_preferences.get("costPriority"), + speedPriority=model_preferences.get("speedPriority"), + intelligencePriority=model_preferences.get("intelligencePriority"), + ) + + def _extract_text(self, content: Any) -> str: + emoji_manager = EmojiManager() + + # MCP returns content as either a single block or array; here we only extract text blocks + if isinstance(content, list): + texts: list[str] = [] + for block in content: + if getattr(block, "type", None) == "text": + texts.append(block.text) + return emoji_manager.remove_emoji("\n".join(texts).strip()) + + if getattr(content, "type", None) == "text": + return emoji_manager.remove_emoji(content.text.strip()) + + + return emoji_manager.remove_emoji(str(content)) + + async def sampling(self, + *, + system_prompt: str | None, + messages: list[SamplingMessage], + temperature: float = 0.3, + top_p: float = 0.9, + max_tokens: int = 4096, + model_preferences: dict[str, Any] | None = None, + metadata: dict[str, Any] | None = None, + stop_sequences: list[str] | None = None + ) -> str: + merged_metadata = dict(metadata or {}) + merged_metadata["top_p"] = top_p + + result = await self._mcp_ctx.session.create_message( + messages=messages, + max_tokens=max_tokens, + system_prompt=system_prompt, + temperature=temperature, + # stop_sequences=stop_sequences, + metadata=merged_metadata, + # model_preferences=self._to_mcp_model_preferences(model_preferences), + ) + return self._extract_text(result.content) + +class SamplingLLMClient(LLMClient): + """ + Only differentiate based on presence of media input. + Server passes media paths and timestamps to Client, Client handles base64 conversion. + """ + + def __init__(self, sampler: BaseLLMSampling): + self._sampler = sampler + + async def complete(self, + *, + system_prompt: str | None, + user_prompt: str, + media: list[dict[str, Any]] | None = None, + temperature: float = 0.3, + top_p: float = 0.9, + max_tokens: int = 2048, + model_preferences: dict[str, Any] | None = None, + metadata: dict[str, Any] | None = None, + stop_sequences: list[str] | None = None + )-> str: + messages = [ + SamplingMessage( + role="user", + content=TextContent(type="text", text=user_prompt), + ) + ] + + merged_metadata = dict(metadata or {}) + merged_metadata["modality"] = "multimodal" if media else "text" + if media: + merged_metadata["media"] = media # Critical: Pass media paths and timestamps through transparently + + return await self._sampler.sampling( + system_prompt=system_prompt, + messages=messages, + temperature=temperature, + top_p=top_p, + max_tokens=max_tokens, + model_preferences=model_preferences, + metadata=merged_metadata, + stop_sequences=stop_sequences, + ) + +def make_llm(mcp_ctx: Context[ServerSession, object]) -> LLMClient: + # Tools can directly call llm.complete() via llm = make_llm(ctx) + return SamplingLLMClient(MCPSampler(mcp_ctx)) + diff --git a/src/open_storyline/mcp/server.py b/src/open_storyline/mcp/server.py new file mode 100644 index 0000000000000000000000000000000000000000..bfb10341d5e6b4b97e55a083f4feef4919e40b13 --- /dev/null +++ b/src/open_storyline/mcp/server.py @@ -0,0 +1,58 @@ +from collections.abc import AsyncIterator +from contextlib import asynccontextmanager + +from mcp.server.fastmcp import FastMCP + +from open_storyline.mcp import register_tools +from open_storyline.config import load_settings, default_config_path +from open_storyline.config import Settings +from open_storyline.storage.session_manager import SessionLifecycleManager +from open_storyline.utils.logging import get_logger + +logger = get_logger() + + +def create_server(cfg: Settings) -> FastMCP: + """ + Creates the MCP server and registers tools + """ + + runtime_ctx = cfg + + @asynccontextmanager + async def session_lifespan(server: FastMCP) -> AsyncIterator[SessionLifecycleManager]: + """Manage session lifecycle with type-safe context.""" + # Initialize on startup + logger.info("Enable session lifespan manager") + session_manager = SessionLifecycleManager( + artifacts_root=cfg.project.outputs_dir, + cache_root=cfg.local_mcp_server.server_cache_dir, + enable_cleanup=True, + ) + try: + yield session_manager + finally: + # Cleanup on shutdown + session_manager.cleanup_expired_sessions() + + server = FastMCP( + name=cfg.local_mcp_server.server_name, + stateless_http=cfg.local_mcp_server.stateless_http, + json_response=cfg.local_mcp_server.json_response, + lifespan=session_lifespan, + ) + + # Pass runtime_ctx to register_tools so each tool can access cfg + register_tools.register(server, runtime_ctx) + + return server + +def main(): + cfg = load_settings(default_config_path()) + server = create_server(cfg) + server.settings.host = cfg.local_mcp_server.connect_host + server.settings.port = cfg.local_mcp_server.port + server.run(transport=cfg.local_mcp_server.server_transport) + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/src/open_storyline/nodes/__init__.py b/src/open_storyline/nodes/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e5c7a807ebd1b05dbee2546ac869ba16c23ba9ea --- /dev/null +++ b/src/open_storyline/nodes/__init__.py @@ -0,0 +1 @@ +# 导入管理 \ No newline at end of file diff --git a/src/open_storyline/nodes/core_nodes/base_node.py b/src/open_storyline/nodes/core_nodes/base_node.py new file mode 100644 index 0000000000000000000000000000000000000000..8a83fe3a96ee4068f35cc646e5b125d05e9fd3e2 --- /dev/null +++ b/src/open_storyline/nodes/core_nodes/base_node.py @@ -0,0 +1,245 @@ +from abc import ABC, abstractmethod +import os +from pathlib import Path +from dataclasses import dataclass, field +from pydantic import BaseModel, ValidationError +from typing import Any, Dict, List, Optional, Union, ClassVar +import json +import traceback + +from open_storyline.config import Settings +from open_storyline.nodes.node_state import NodeState + +from open_storyline.storage.file import FileCompressor +from open_storyline.utils.logging import get_logger +from open_storyline.mcp.sampling_requester import LLMClient + +logger = get_logger(__name__) + +@dataclass +class NodeMeta: + """ + Node Metadata Class + + Defines metadata information for nodes in a workflow or flowchart, including node + identification, type, dependencies, and downstream node routing configurations. + + Attributes: + name: Tool name + description: Tool functionality description + node_id: Unique node identifier for uniquely locating the node within the entire process + node_kind: Node type/category, such as "start", "process", "end", etc. + require_prior_kind: List of prerequisite node types that must be completed + before the current node runs + Example: ["validation", "authentication"] + default_require_prior_kind: Default list of prerequisite node types, serving as + the default configuration or fallback for require_prior_kind + next_available_node: List of downstream node IDs that the current node can transition to + after execution completes + Used to define possible branch paths in the workflow + priority: Execution priority among nodes with the same functionality + """ + + name: str + description: str + node_id: str + node_kind: str + require_prior_kind: List[str] = field(default_factory=list) + default_require_prior_kind: List[str] = field(default_factory=list) + next_available_node: List[str] = field(default_factory=list) + priority: int = 5 + + + +class BaseNode(ABC): + meta: NodeMeta + input_schema: ClassVar[type[BaseModel] | None] = None + # output_schema: ClassVar[type[BaseModel] | None] = None + + def __init__(self, server_cfg: Settings) -> None: + self.server_cfg = server_cfg + self.server_cache_dir = Path(os.getcwd()) / self.server_cfg.local_mcp_server.server_cache_dir + + if not hasattr(self, "meta"): + raise ValueError("Subclass must define the 'meta' attribute") + + + def _load_user_info(self, node_state: NodeState, params: Dict[str,Any]) -> Dict[str,Any]: + return { + "session_id": node_state.session_id, + "artifact_id": node_state.artifact_id + } + + def _load_item(self, node_state: NodeState, user_info: Dict[str,str], item: Dict[str,Any]): + new_item:Dict[str,Any] = {} + item_base64 = item.pop("base64", None) + item_md5 = item.pop("md5", None) + item_path = item.pop("path", None) + new_item.update(item) + + + if item_base64 and item_path: + item_save_path = self.server_cache_dir / user_info['session_id']/ user_info['artifact_id'] / os.path.basename(item_path) + FileCompressor.decompress_from_string(item_base64, item_save_path) + new_item['path'] = str(item_save_path.relative_to(os.getcwd())) + new_item['orig_path'] = str(item_path) + new_item['orig_md5'] = item_md5 + return new_item + + def _pack_item(self, node_state: NodeState, item: Dict[str,Any]): + orig_path = item.pop('orig_path', None) + orig_md5 = item.pop('orig_md5', None) + server_save_path = item.pop('path', None) + if server_save_path: + compress_data = FileCompressor.compress_and_encode(server_save_path) + if orig_path and orig_md5 and compress_data.md5 == orig_md5: + node_state.node_summary.debug_for_dev(f"[node] node_id: {self.meta.node_id} change `path` change to {orig_path}") + item['path'] = orig_path + elif orig_md5 is None or compress_data.md5 != orig_md5: + node_state.node_summary.debug_for_dev(f"[node] node_id: {self.meta.node_id} return `base64` to client") + item['base64'] = compress_data.base64 + item['path'] = compress_data.filename + item['md5'] = compress_data.md5 + return item + + + def load_inputs_from_client(self, node_state: NodeState, params: Dict[str,Any], user_info: Optional[Dict[str,str]] = None, save: bool=True) -> Dict[str, Any]: + """ + Read data from client's request and save the transmitted base64 data on the Server. + """ + + if user_info is None: + user_info = self._load_user_info(node_state, params) + + payload_key = params.keys() + loaded_input = {} + kwargs = {} + for k in payload_key: + payload_input = params[k] + if isinstance(payload_input, list) and all([isinstance(item, dict) for item in payload_input]): + # List: load base64 data and save to server cache + loaded_input[k] = [self._load_item(node_state, user_info, item) for item in payload_input] + elif isinstance(payload_input, dict): + # Dict: recursively process nested data (without saving) + loaded_input[k] = self.load_inputs_from_client(node_state, payload_input, user_info, save=False) + elif isinstance(payload_input, LLMClient): + kwargs[k] = payload_input + else: + # Handle primitive types: directly copy the value (e.g., str, int, float, bool) + loaded_input[k] = params[k] + + # save loaded_input to self.cache_dir in json format + if save: + artifact_save_path = (self.server_cache_dir / user_info['session_id'] / user_info['artifact_id']).with_suffix(".json") + artifact_save_path.parent.mkdir(parents=True, exist_ok=True) + with open(artifact_save_path, 'w') as f: + json.dump(loaded_input, f, indent=2, ensure_ascii=False) + loaded_input.update(kwargs) + return loaded_input + + + def pack_outputs_to_client(self, node_state: NodeState, outputs: Union[Dict[str,Any],List[str]]) -> Union[Dict[str,Any],List[str]]: + """ + Pack the output and return it to the client. + """ + if not isinstance(outputs, dict): + return outputs + payload_key = outputs.keys() + packed_output = {} + for k in payload_key: + payload_output = outputs[k] + if isinstance(payload_output, list) and all(isinstance(item, dict) for item in payload_output): + packed_output[k] = [self._pack_item(node_state, item) for item in payload_output] + elif isinstance(payload_output, dict): + packed_output[k] = self.pack_outputs_to_client(node_state, payload_output) + else: + packed_output[k] = outputs[k] + return packed_output + + @abstractmethod + async def default_process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Any: + """ + Default processing method that must be implemented. Called when the node needs to be skipped. + """ + ... + + @abstractmethod + async def process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Any: + """ + Main processing method that must be implemented. Executed when the node is invoked normally. + """ + ... + + def _parse_input(self, node_state: NodeState, inputs: Dict[str, Any]): + return inputs + + def _combine_tool_outputs(self, node_state: NodeState, outputs: Dict[str, Any]): + return outputs + + def _validate_schema( + self, + params: dict[str,Any], + schema_name: Union[str, List[str]], + update_params: bool = False + ) -> Optional[Dict[str,Any]]: + schema_names = [schema_name] if isinstance(schema_name, str) else schema_name + + validated_params = params.copy() if update_params else None + + for name in schema_names: + schema = getattr(self, name, None) + + if schema is None: + logger.warning( + f"Schema '{name}' does not exist, skipping validation", + ) + continue + + try: + validated = schema(**params) + if update_params: + validated_params.update(validated.dict()) + except ValidationError as e: + logger.error(f"{name} validation failed: {e}") + return validated_params if update_params else None + + async def __call__(self, node_state: NodeState, **params) -> Dict[str, Any]: + try: + mode = params.get("mode", "auto") + + inputs = self.load_inputs_from_client(node_state, params.copy()) + + parsed_inputs = self._parse_input(node_state, inputs) + + if mode != 'auto': + outputs = await self.default_process(node_state, parsed_inputs) + else: + outputs = await self.process(node_state, parsed_inputs) + + processed_outputs = self._combine_tool_outputs(node_state, outputs) + + packed_output = self.pack_outputs_to_client(node_state, processed_outputs) + + # self._validate_schema(packed_output, 'output_schema') + + return { + 'artifact_id': node_state.artifact_id, + 'summary': node_state.node_summary.get_summary(node_state.artifact_id), + 'tool_excute_result': packed_output, + 'isError': False + } + except Exception as e: + if self.server_cfg.developer.developer_mode: + traceback_info = ''.join(traceback.format_exception(e)) + summary = { + "error_info": f"[artifact_id {node_state.artifact_id}] \n {traceback_info}" + } + logger.error(traceback_info) + else: + summary = node_state.node_summary.get_summary(node_state.artifact_id) + return { + 'artifact_id': node_state.artifact_id, + 'summary': summary, + 'tool_excute_result': {}, + 'isError': True + } diff --git a/src/open_storyline/nodes/core_nodes/filter_clips.py b/src/open_storyline/nodes/core_nodes/filter_clips.py new file mode 100644 index 0000000000000000000000000000000000000000..1bcecfb6fee89a1c398bf7e73932cc92685bbcb1 --- /dev/null +++ b/src/open_storyline/nodes/core_nodes/filter_clips.py @@ -0,0 +1,181 @@ +from typing import Any, Dict + +from open_storyline.nodes.core_nodes.base_node import BaseNode, NodeMeta +from open_storyline.nodes.node_state import NodeState +from open_storyline.nodes.node_schema import FilterClipsInput +from open_storyline.mcp.sampling_requester import LLMClient +from src.open_storyline.utils.prompts import get_prompt +from open_storyline.utils.parse_json import parse_json_dict +from open_storyline.utils.register import NODE_REGISTRY + +@NODE_REGISTRY.register() +class FilterClipsNode(BaseNode): + + meta = NodeMeta( + name="filter_clips", + description="Filter clips based on their descriptions according to user requirements. Depends on the results from the understand_clips tool", + node_id="filter_clips", + node_kind="filter_clips", + require_prior_kind=['split_shots','understand_clips'], + default_require_prior_kind=['split_shots','understand_clips'], + next_available_node=['group_clips', 'group_clips_pro'], + ) + + input_schema = FilterClipsInput + + def _parse_input(self, node_state: NodeState, inputs: Dict[str, Any]): + clip_captions = inputs["understand_clips"].get("clip_captions") + clip_info = inputs["split_shots"]["clips"] + duration_lookup = _build_duration_lookup(clip_info) + clip_captions=_add_input_duration(clip_captions,duration_lookup) + + input_clip_ids: list[str] = [ + (c.get("clip_id")) for c in clip_captions + ] + inputs["input_clip_ids"] = input_clip_ids + inputs["clip_captions"] = clip_captions + return inputs + + + async def default_process( + self, + node_state, + inputs: Dict[str, Any], + ) -> Any: + clip_captions = inputs["understand_clips"].get("clip_captions") + + node_state.node_summary.info_for_user("Using all clips") + return { + "clip_captions": clip_captions, + "selected": inputs["input_clip_ids"], + } + + async def process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Any: + clip_captions = inputs["understand_clips"].get("clip_captions") + user_request = inputs["user_request"] + llm = node_state.llm + + input_clip_ids = inputs["input_clip_ids"] + + if not user_request or user_request == "": + node_state.node_summary.info_for_user("User did not specify requirements, using all clips") + return { + "clip_captions": clip_captions, + "selected": input_clip_ids, + } + + else: + clip_block = _build_clips_block(clip_captions) + system_prompt = get_prompt("filter_clips.system", lang=node_state.lang) + user_prompt = get_prompt("filter_clips.user", lang=node_state.lang, user_request=user_request, clip_captions=clip_block) + + raw = await llm.complete( + system_prompt=system_prompt, + user_prompt=user_prompt, + media=None, + temperature=0.1, + top_p=0.9, + max_tokens=2048, + model_preferences=None, + ) + + try: + obj = parse_json_dict(raw) + select_ids = _extract_selected_ids(obj, input_clip_ids) + node_state.node_summary.info_for_user(f"Successfully filtered {len(select_ids)} clips") + + except: + select_ids = input_clip_ids + node_state.node_summary.info_for_user("Failed to parse model output, using all clips") + + return { + "clip_captions": clip_captions, + "selected": select_ids, + } + + + +def _add_input_duration(clip_captions:list[dict[str, Any]],clip_durations: dict[str, float]) -> Any: + for i in range(len(clip_captions)): + clip_id=clip_captions[i].get('clip_id','') + if not clip_id:continue + if clip_id in clip_durations: + clip_captions[i]['duration']=clip_durations[clip_id] + return clip_captions + + +def _build_duration_lookup(clip_info: list[dict[str, Any]]) -> dict[str, float]: + """ + clip_id -> duration_sec + """ + out: dict[str, float] = {} + for item in clip_info or []: + cid = item.get("clip_id") + if not cid: + continue + src = item.get("source_ref") or {} + dur = src.get("duration", 0) / 1000.0 + if dur==0.0:dur=2.0 + out[cid] = dur + return out + + +def _extract_selected_ids( + obj: dict[str, Any], + input_clip_ids: list[str], +) -> list[str]: + """ + Extract selected clip_id list from LLM structured output. + Returns: Filtered results ordered by input_clip_ids (preserving only valid input IDs) + """ + id_set = set(input_clip_ids) + + results = obj.get("results") + if not isinstance(results, list): + raise ValueError('"results" must be a list') + + true_items = 0 + valid_true_ids: set[str] = set() + + for item in results: + if not isinstance(item, dict): + continue + cid = item.get("clip_id") + keep = item.get("keep") + + # keep allows bool or "true"/"false" + keep_bool = None + if isinstance(keep, bool): + keep_bool = keep + elif isinstance(keep, str): + s = keep.strip().lower() + if s in ("true", "yes", "1"): + keep_bool = True + elif s in ("false", "no", "0"): + keep_bool = False + + if keep_bool is True: + true_items += 1 + if isinstance(cid, str) and cid in id_set: + valid_true_ids.add(cid) + + # If the model explicitly selected items (keep=true) but none match the input + if true_items > 0 and not valid_true_ids: + raise ValueError("results has keep=true entries, but no valid clip_ids (model may have modified IDs)") + + return [cid for cid in input_clip_ids if cid in valid_true_ids] + +def _build_clips_block(clip_captions: list[dict[str, Any]]) -> str: + """ + Construct clips into stable text blocks + """ + blocks: list[str] = [] + for clip in clip_captions: + cid = clip.get("clip_id", "") + caption = clip.get("caption", "") + block = ( + f"[clip_id={cid}]\n" + f"caption: {caption}\n" + ) + blocks.append(block) + return "\n".join(blocks).strip() + "\n" diff --git a/src/open_storyline/nodes/core_nodes/generate_script.py b/src/open_storyline/nodes/core_nodes/generate_script.py new file mode 100644 index 0000000000000000000000000000000000000000..8f98331f1cbf88eb6f769f43f63da59c531130aa --- /dev/null +++ b/src/open_storyline/nodes/core_nodes/generate_script.py @@ -0,0 +1,347 @@ +from typing import Any, Dict +import re + +from open_storyline.nodes.core_nodes.base_node import BaseNode, NodeMeta +from open_storyline.nodes.node_state import NodeState +from open_storyline.nodes.node_schema import GenerateScriptInput +from src.open_storyline.utils.prompts import get_prompt +from open_storyline.utils.parse_json import parse_json_dict +from open_storyline.utils.register import NODE_REGISTRY + +@NODE_REGISTRY.register() +class GenerateScriptNode(BaseNode): + meta = NodeMeta( + name="generate_script", + description="Generate video script/copy that can be used to synthesize voice-over or be directly applied as video subtitles"\ + "Support lyrical, humorous, and casual styles, and consider using the `subtitle_imitation_skill` for special styles.", + node_id="generate_script", + node_kind="generate_script", + require_prior_kind=['split_shots','group_clips','understand_clips'], + default_require_prior_kind=['split_shots','group_clips'], + next_available_node=['generate_voiceover'], + ) + + input_schema = GenerateScriptInput + + async def default_process( + self, + node_state: NodeState, + inputs: Dict[str, Any], + ) -> Any: + return { + "group_scripts": [], + "title": "", + } + + async def process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Any: + clip_info = inputs["split_shots"]["clips"] + clip_captions = inputs["understand_clips"]["clip_captions"] + overall = inputs["understand_clips"]["overall"] + groups = inputs["group_clips"]["groups"] + user_request = inputs["user_request"] + llm = node_state.llm + + duration_lookup = _build_duration_lookup(clip_info) + caption_lookup = _build_caption_lookup(clip_captions) + + group_ids: list[str] = [g.get("group_id","") for g in (groups or []) if g.get("group_id")] + group_ids_set = set(group_ids) + + if not group_ids: + node_state.node_summary.info_for_user("no available group, cannot generate script") + return {"group_scripts": [], 'title': ""} + + custom_script = inputs.get("custom_script", {}) + if len(custom_script) > 0: + try: + group_scripts = [] + subtitle_index = 1 + + validate_subtitle_format(custom_script) + edit_group_scripts = custom_script['group_scripts'] + # fill subtitle_units + for i in range(len(edit_group_scripts)): + raw_text = edit_group_scripts[i]['raw_text'] + units, subtitle_index = _make_subtitle_units( + raw_text=raw_text, + subtitle_start_index=subtitle_index, + ) + group_scripts.append({ + "group_id": edit_group_scripts[i]['group_id'], + "raw_text": raw_text, + "subtitle_units": units + }) + + custom_script = {"group_scripts": group_scripts, "title": custom_script.get('title', '')} + except Exception as e: + node_state.node_summary.info_for_llm(f"generate script failed: {type(e).__name__}: {e}") + group_text_map = {} + return custom_script + + else: + groups_block = _build_groups_block_for_script(groups, duration_lookup, caption_lookup) + + system_prompt = get_prompt("generate_script.system", lang=node_state.lang) + if not user_request or user_request == "": + user_request = "No requirements" + user_prompt = get_prompt("generate_script.user", lang=node_state.lang, user_request=user_request, overall=overall, groups=groups_block) + + raw = await llm.complete( + system_prompt=system_prompt, + user_prompt=user_prompt, + temperature=0.1, + top_p=0.9, + max_tokens=4096, + model_preferences=None, + ) + group_text_map: dict[str, str] = {} + try: + obj = parse_json_dict(raw) + group_text_map = _extract_group_text_map(obj, group_ids) + except Exception as e: + node_state.node_summary.info_for_llm(f"generate script failed: {type(e).__name__}: {e}") + group_text_map = {} + + group_scripts: list[dict[str, Any]] = [] + subtitle_index = 1 + + for g in groups or []: + gid = g.get("group_id", "") + if not gid or gid not in group_ids_set: + continue + + duration_sec = 0.0 + for cid in (g.get("clip_ids") or []): + duration_sec += float(duration_lookup.get(cid, 0.0)) + budget = _estimate_script_budget(duration_sec) + + raw_text = (group_text_map.get(gid) or "").strip() + if not raw_text: + raise ValueError(f"LLM did not generate any content, please retry") + + max_chars = budget.get("max_chars", 60) + if len(raw_text) > int(max_chars * 2.0): + raw_text = raw_text[:max_chars].rstrip() + node_state.node_summary.info_for_user("The generated script was too long and has been truncated.") + + units, subtitle_index = _make_subtitle_units( + raw_text=raw_text, + subtitle_start_index=subtitle_index, + ) + + group_scripts.append( + { + "group_id": gid, + "raw_text": raw_text, + "subtitle_units": units, + } + ) + + return { + "group_scripts": group_scripts, + "title": obj.get("title", ""), + } + +def _build_duration_lookup(clip_info: list[dict[str, Any]]) -> dict[str, float]: + """ + clip_id -> duration_sec + """ + default_duration = 2.0 # HACK: default image second for estimate group durations + out: dict[str, float] = {} + for item in clip_info or []: + cid = item.get("clip_id") + if not cid: + continue + src = item.get("source_ref") or {} + dur = src.get("duration", 0) / 1000.0 + if dur == 0.0: + dur = default_duration + out[cid] = dur + return out + +def _build_caption_lookup(clip_captions: list[dict[str, Any]]) -> dict[str, dict[str, Any]]: + """ + clip_id -> caption_obj + """ + + out: dict[str, dict[str, Any]] = {} + for item in clip_captions: + if not isinstance(item, dict): + continue + cid = item.get("clip_id") + if cid: + out[cid] = item + return out + +def _estimate_script_budget(duration_sec: float) -> dict[str, Any]: + """ + Estimate word/character count budget based on total duration + """ + if duration_sec is None: + duration_sec = 0.0 + duration_sec = max(0.0, float(duration_sec)) + + min_chars = int(round(duration_sec * 3)) + max_chars = int(round(duration_sec * 5)) + + # 防止极短组变成 0 + min_chars = max(min_chars, 8) + max_chars = max(max_chars, min_chars + 6) + + return { + "duration_sec": duration_sec, + "min_chars": min_chars, + "max_chars": max_chars, + } + + +def _build_groups_block_for_script( + groups: list[dict[str, Any]], + duration_lookup: dict[str, float], + caption_lookup: dict[str, dict[str, Any]], + *, + max_caption_len: int = 120, +) -> str: + """ + Combine groups, clip captions, and duration budget into a prompt + for LLM to generate script for each group. + """ + + blocks: list[str] = [] + + for g in groups or []: + gid = g.get("group_id", "") + clip_ids = g.get("clip_ids") or [] + if not gid or not isinstance(clip_ids, list) or not clip_ids: + continue + + group_summary = (g.get("summary") or "").strip() + + # Duration per group + duration_sec = 0.0 + for cid in clip_ids: + duration_sec += float(duration_lookup.get(cid, 0.0)) + budget = _estimate_script_budget(duration_sec) + + lines: list[str] = [] + lines.append(f"[group_id={gid}]") + if group_summary: + lines.append(f"summary: {group_summary}") + lines.append(f"duration_sec: {budget['duration_sec']:.2f}") + lines.append(f"script_chars_budget: {budget['min_chars']}~{budget['max_chars']}") + + lines.append("clips:") + for cid in clip_ids: + cap_obj = caption_lookup.get(cid, {}) + cap_text = cap_obj.get("caption", "") + sem = cap_obj.get("semantic") or {} + kw = sem.get("keywords") or [] + mood = sem.get("mood") or [] + kw_s = "、".join([x for x in kw if isinstance(x, str)])[:40] + mood_s = "、".join([x for x in mood if isinstance(x, str)])[:30] + + dur = duration_lookup.get(cid, 0.0) + + lines.append(f"- {cid} ({dur:.2f}s): {cap_text}") + if kw_s or mood_s: + lines.append(f" tags_hint: keywords={kw_s} | mood={mood_s}") + + blocks.append("\n".join(lines)) + + return "\n\n".join(blocks).strip() + +def _extract_group_text_map(obj: Any, group_ids: list[str]) -> dict[str, str]: + """ + Extract {group_id: raw_text} mapping from LLM JSON output. + Compatible with several common output formats: + 1) {"scripts":[{"group_id":"group_0001","raw_text":"..."}, ...]} + 2) {"group_scripts":[{"group_id":"...","raw_text":"..."}, ...]} + 3) {"group_0001":"...", "group_0002":"..."} + 4) [{"group_id":"...","raw_text":"..."}] + """ + gid_set = set(group_ids) + out: dict[str, str] = {} + + def _add(gid: Any, text: Any): + if isinstance(gid, str) and gid in gid_set and isinstance(text, str) and text.strip(): + out[gid] = text.strip() + + if isinstance(obj, dict): + # List type + for key in ("scripts", "group_scripts", "results"): + v = obj.get(key) + if isinstance(v, list): + for item in v: + if isinstance(item, dict): + gid = item.get("group_id") + text = item.get("raw_text") or item.get("text") or item.get("script") + _add(gid, text) + + # Mapping type: {"group_0001":"..."} + for gid in group_ids: + if gid in obj and isinstance(obj[gid], str): + _add(gid, obj[gid]) + + return out + + if isinstance(obj, list): + for item in obj: + if isinstance(item, dict): + gid = item.get("group_id") + text = item.get("raw_text") or item.get("text") or item.get("script") + _add(gid, text) + return out + + raise ValueError("Unable to recognize LLM output structure") + + +_SPLIT_RE = re.compile(r"[，,。！!?？]+") + + +def _split_by_comma(raw_text: str) -> list[str]: + """ + Comma splitting: Supports Chinese/English commas and Chinese period. Remove empty segments. + """ + if not isinstance(raw_text, str): + return [] + s = raw_text.strip().replace("\n", "，") + parts = [p.strip() for p in _SPLIT_RE.split(s) if p and p.strip()] + return parts + + +def _make_subtitle_units( + raw_text: str, + subtitle_start_index: int, +) -> tuple[list[dict[str, Any]], int]: + """ + Generate subtitle_units for a certain group, return (units, next_global_index) + unit_id increments globally: subtitle_0001, subtitle_0002 ... + """ + parts = _split_by_comma(raw_text) + if not parts and raw_text.strip(): + parts = [raw_text.strip()] + + units: list[dict[str, Any]] = [] + cur = subtitle_start_index + for idx_in_group, text in enumerate(parts): + units.append( + { + "unit_id": f"subtitle_{cur:04d}", + "index_in_group": idx_in_group, + "text": text, + } + ) + cur += 1 + return units, cur + +def validate_subtitle_format(data: dict[str, Any]): + if "group_scripts" not in data: + raise ValueError("input missing field 'group_scripts'") + + if "title" not in data: + raise ValueError("input missing field 'title'") + for group in data["group_scripts"]: + if "group_id" not in group: + raise ValueError("group missing field 'group_id'") + if "raw_text" not in group: + raise ValueError("group missing field 'raw_text'") \ No newline at end of file diff --git a/src/open_storyline/nodes/core_nodes/generate_voiceover.py b/src/open_storyline/nodes/core_nodes/generate_voiceover.py new file mode 100644 index 0000000000000000000000000000000000000000..430644ae6d6d7fc095f45664ede87a49d4d58a47 --- /dev/null +++ b/src/open_storyline/nodes/core_nodes/generate_voiceover.py @@ -0,0 +1,558 @@ +import os +import asyncio +import base64 +import time +import uuid +import binascii +import json +import librosa +from pathlib import Path +from typing import Any, Dict, Optional, Callable, Union + +import requests + +from open_storyline.nodes.core_nodes.base_node import BaseNode, NodeMeta +from open_storyline.nodes.node_schema import GenerateVoiceoverInput +from open_storyline.nodes.node_state import NodeState +from open_storyline.utils.parse_json import parse_json_dict +from open_storyline.utils.prompts import get_prompt +from open_storyline.utils.register import NODE_REGISTRY + +@NODE_REGISTRY.register() +class GenerateVoiceoverNode(BaseNode): + meta = NodeMeta( + name="generate_voiceover", + description="Generate voice-over based on the script", + node_id="generate_voiceover", + node_kind="tts", + require_prior_kind=["group_clips", "generate_script"], + default_require_prior_kind=["group_clips", "generate_script"], + ) + + input_schema = GenerateVoiceoverInput + + # provider -> handler method name + _PROVIDER_HANDLERS: Dict[str, str] = { + "bytedance": "_tts_bytedance_sync", + "minimax": "_tts_minimax_sync", + "302": "_tts_302_sync", + } + + _DEFAULT_PROVIDER = "minimax" + + MILLISECONDS_PER_SECOND = 1000.0 + _SAFE_MARGIN = 10 + + async def default_process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Any: + node_state.node_summary.info_for_user("Voiceover not generated") + return {"voiceover": []} + + async def process(self, node_state: NodeState, inputs: Dict[str, Any], **params) -> Any: + # 1) Get script + group_scripts = (inputs.get("generate_script") or {}).get("group_scripts") or [] + if not isinstance(group_scripts, list) or not group_scripts: + node_state.node_summary.info_for_user("No script found for voiceover generation (group_scripts is empty)") + return {"voiceover": []} + + # 2) Provider selection + provider_name = (inputs.get("provider") or "").strip() + if not provider_name: + node_state.node_summary.info_for_user("未找到可生成配音的tts提供商，使用默认") + + handler = self._get_provider_handler(provider_name) + node_state.node_summary.info_for_user(f"TTS 服务：{provider_name}") + + # 3) Prepare output directory + artifact_id = node_state.artifact_id + session_id = node_state.session_id + if not artifact_id or not session_id: + raise ValueError("缺失 artifact_id / session_id，无法生成配音输出目录") + + output_dir = self.server_cache_dir / str(session_id) / str(artifact_id) + output_dir.mkdir(parents=True, exist_ok=True) + + # 4) Deduce which key fields this provider needs from config, and get values from inputs + # If user/config keys are incomplete, fallback to 302 and use 302 key from environment variables + try: + provider_cfg = self._get_provider_cfg(provider_name) + secrets = self._resolve_provider_secrets(provider_name, provider_cfg, inputs, node_state) + except ValueError as e: + if provider_name == self._DEFAULT_PROVIDER: + raise + node_state.node_summary.info_for_user( + f"Key/config for provider={provider_name} is incomplete, automatically falling back to {self._DEFAULT_PROVIDER} (using environment variable key): {e}" + ) + provider_name = self._DEFAULT_PROVIDER + handler = self._get_provider_handler(provider_name) + provider_cfg = self._get_provider_cfg(provider_name) + secrets = self._resolve_provider_secrets(provider_name, provider_cfg, inputs, node_state) + node_state.node_summary.info_for_user(f"TTS service fallback to: {provider_name}") + + # 5) Generate parameter dict from provider parameter schema + user_request via LLM + provider_param_schema = self._load_provider_param_schema(provider_name) + user_request = inputs.get("user_request", "") + tts_params = await self._infer_tts_params_with_llm( + node_state=node_state, + provider_name=provider_name, + user_request=user_request, + provider_param_schema=provider_param_schema, + ) + + if tts_params: + node_state.node_summary.info_for_user(f"TTS parameters (LLM parsed): {json.dumps(tts_params, ensure_ascii=False)}") + else: + node_state.node_summary.info_for_user("TTS parameters: No valid parameters parsed from user_request, using default/server default values") + + # 6) Generate segment by segment + ts_ms = int(time.time() * 1000) + voiceover: list[dict[str, Any]] = [] + + for i, group in enumerate(group_scripts, start=1): + group_id = (group or {}).get("group_id", "") + raw_text = (group or {}).get("raw_text", "") + + if not group_id: + raise ValueError(f"Missing group_id: {group}") + if not isinstance(raw_text, str) or not raw_text.strip(): + raise ValueError(f"raw_text is empty for group_id={group_id}, cannot generate speech.") + + voiceover_id = f"voiceover_{i:04d}" + wav_path = output_dir / f"{voiceover_id}_{ts_ms}.wav" + + await asyncio.to_thread( + handler, + text=raw_text, + wav_path=wav_path, + secrets=secrets, + tts_params=tts_params, + provider_cfg=provider_cfg, + ) + + duration = self._wav_duration_ms(wav_path) + voiceover.append( + { + "voiceover_id": voiceover_id, + "group_id": group_id, + "path": str(wav_path), + "duration": duration, + } + ) + + node_state.node_summary.info_for_user( + f"Successfully generated {voiceover_id}", + preview_urls=[str(wav_path)], + ) + + + node_state.node_summary.info_for_user(f"Generated {len(voiceover)} voiceover segments in total") + return {"voiceover": voiceover} + + # --------------------------------------------------------------------- + # Provider dispatch / config helpers + # --------------------------------------------------------------------- + + def _get_provider_handler(self, provider_name: str) -> Callable[..., None]: + if provider_name is None or provider_name == "": + provider_name = self._DEFAULT_PROVIDER + method_name = self._PROVIDER_HANDLERS.get(provider_name) + if not method_name: + raise ValueError(f"Unsupported TTS provider: {provider_name}, currently supported: {list(self._PROVIDER_HANDLERS.keys())}") + handler = getattr(self, method_name, None) + if not callable(handler): + raise ValueError(f"Handler for provider={provider_name} not implemented: {method_name}") + return handler + + def _get_provider_cfg(self, provider_name: str) -> Dict[str, Any]: + providers = getattr(self.server_cfg.generate_voiceover, "providers", None) or {} + cfg = providers.get(provider_name) + if not isinstance(cfg, dict): + if provider_name == self._DEFAULT_PROVIDER: + return {"api_key": "", "base_url": ""} + raise ValueError(f"provider={provider_name} not configured in server_cfg.generate_voiceover.providers") + + return cfg + + def _resolve_provider_secrets(self, provider_name: str, provider_cfg: Dict[str, Any], inputs: Dict[str, Any], node_state: NodeState) -> Dict[str, Any]: + """ + - Each field uses inputs[field] first, otherwise falls back to cfg[field] + - base_url can be omitted: default value will be provided based on provider + """ + secrets: Dict[str, Any] = {} + required_keys = list(provider_cfg.keys()) + provider_keys = inputs.get("provider_keys") or {} + if not isinstance(provider_keys, dict): + provider_keys = {} + + for key in required_keys: + value = inputs.get(key) + if value in (None, ""): + value = provider_keys.get(key) + + if value in (None, ""): + value = provider_cfg.get(key) + + if (value in (None, "")) and key == "base_url": + value = self._default_base_url(provider_name) + + if (value in (None, "")) and provider_name == self._DEFAULT_PROVIDER: + env_v = self._resolve_minimax_env_secret(key) + if env_v not in (None, ""): + value = env_v + + if value in (None, ""): + node_state.node_summary.info_for_llm("The user has not entered the voice-over service API key, please remind the user to enter the TTS API key in the sidebar of the webpage.") + raise ValueError( + f"provider={provider_name} missing required field: {key}. " + f"Please configure in sidebar or config.toml." + ) + + secrets[key] = value + + return secrets + + def _default_base_url(self, provider_name: str) -> str: + if provider_name == "bytedance": + return "https://openspeech.bytedance.com" + if provider_name == "minimax": + return "https://api.minimax.chat" + if provider_name == "302": + return "https://api.302.ai" + return "" + + # --------------------------------------------------------------------- + # LLM param inference + # --------------------------------------------------------------------- + def _load_provider_param_schema(self, provider_name: str) -> Dict[str, Any]: + + path = self.server_cfg.generate_voiceover.tts_provider_params_path + try: + data = json.loads(path.read_text(encoding="utf-8")) + except Exception: + return {} + + providers = (data or {}).get("providers") or {} + schema = providers.get(provider_name) or {} + return schema if isinstance(schema, dict) else {} + + async def _infer_tts_params_with_llm( + self, + node_state: NodeState, + provider_name: str, + user_request: Any, + provider_param_schema: Dict[str, Any], + ) -> Dict[str, Any]: + """ + Pass user_request + provider parameter definition to LLM, let it return JSON dict. + """ + if not provider_param_schema: + return {} + + system_prompt = get_prompt("generate_voiceover.system", lang=node_state.lang) + + schema_text = json.dumps(provider_param_schema, ensure_ascii=False, indent=2) + + user_prompt = get_prompt("generate_voiceover.user", lang=node_state.lang, provider_name=provider_name, user_request=str(user_request), schema_text=schema_text) + raw = await node_state.llm.complete( + system_prompt=system_prompt, + user_prompt=user_prompt, + temperature=0.1, + top_p=0.9, + max_tokens=4096, + model_preferences=None + ) + if not raw: + return {} + + parsed = parse_json_dict(raw) + if not isinstance(parsed, dict): + return {} + + return self._sanitize_params_by_schema(parsed, provider_param_schema) + + # --------------------------------------------------------------------- + # validation helpers + # --------------------------------------------------------------------- + + def _resolve_302_env_secret(self, key: str) -> Optional[str]: + """ + Read 302 key/config from environment variables + """ + key = str(key).strip() + if not key: + return None + + key_upper = key.upper() + prefixe = ("TTS_302_") + + return os.getenv(f"{prefixe}{key_upper}") + + def _resolve_minimax_env_secret(self, key: str) -> Optional[str]: + """ + 从环境变量读取 minimax 的密钥/配置 + """ + key = str(key).strip() + if not key: + return None + + key_upper = key.upper() + prefixe = ("TTS_MINIMAX_") + + return os.getenv(f"{prefixe}{key_upper}") + + def _sanitize_params_by_schema(self, params: Dict[str, Any], schema: Dict[str, Any]) -> Dict[str, Any]: + """ + - Only keep fields that exist in schema + - Type coercion + - Enum validation (string enum / numeric range / discrete numeric enum) + """ + out: Dict[str, Any] = {} + + for key, val in params.items(): + if key not in schema: + continue + + rule = schema.get(key) or {} + if not isinstance(rule, dict): + continue + + typ = (rule.get("type") or "").lower().strip() + enum = rule.get("enum") + + normalized = self._normalize_value(val, typ) + if normalized is None: + continue + + # Enum validation + if isinstance(enum, list) and enum: + if typ in ("int", "float") and len(enum) == 2 and all(isinstance(value, (int, float)) for value in enum): + range_min, range_max = float(enum[0]), float(enum[1]) + value = float(normalized) + if value < range_min: + value = range_min + if value > range_max: + value = range_max + normalized = int(value) if typ == "int" else float(round(value, 1)) + else: + if normalized not in enum: + normalized = enum[0] + + out[key] = normalized + + return out + + def _normalize_value(self, val: Any, typ: str) -> Any: + if typ in ("str", "string"): + return str(val) + + if typ in ("int", "integer"): + return int(val) + + if typ in ("float"): + return float(int(val)) + + if typ in ("bool", "boolean"): + return bool(val) + + return val + + def _wav_duration_ms(self, wav_path: Union[str, Path]) -> int: + p = str(wav_path) + + duration_s = librosa.get_duration(path=p) + return int(round(duration_s * self.MILLISECONDS_PER_SECOND)) + + # --------------------------------------------------------------------- + # Provider implementations (each provider has its own dedicated method) + # --------------------------------------------------------------------- + + def _preview_b64(self, b64: str, keep: int = 80) -> str: + if not isinstance(b64, str): + return f"" + if len(b64) <= keep * 2: + return b64 + return f"{b64[:keep]}......{b64[-keep:]}" + + def _tts_bytedance_sync( + self, + *, + text: str, + wav_path: Path, + secrets: Dict[str, Any], + tts_params: Dict[str, Any], + provider_cfg: Dict[str, Any], + ) -> None: + + base_url = secrets.get("base_url") or "https://openspeech.bytedance.com" + api_url = base_url.rstrip("/") + "/api/v1/tts" if not base_url.endswith("/api/v1/tts") else base_url + + access_token = secrets.get("access_token") + appid = secrets.get("appid") + uid = secrets.get("uid") + cluster = secrets.get("cluster") or "volcano_tts" + + headers = {"Authorization": f"Bearer; {access_token}"} + + audio_cfg = { + "voice_type": tts_params.get("voice_type", "BV700_streaming"), + "encoding": tts_params.get("encoding", "wav"), + "rate": int(tts_params.get("rate", 24000)) if "rate" in tts_params else 24000, + "speed_ratio": float(tts_params.get("speed_ratio", 1.0)), + "volume_ratio": float(tts_params.get("volume_ratio", 1.0)), + "pitch_ratio": float(tts_params.get("pitch_ratio", 1.0)), + } + # 可选字段 + for k in ("emotion", "language"): + if k in tts_params: + audio_cfg[k] = tts_params[k] + + request_cfg = { + "reqid": str(uuid.uuid4()), + "text": text, + "text_type": tts_params.get("text_type", "plain"), + "operation": "query", + } + + body = { + "app": {"appid": appid, "token": access_token, "cluster": cluster}, + "user": {"uid": uid}, + "audio": audio_cfg, + "request": request_cfg, + } + + resp = requests.post(api_url, headers=headers, json=body, timeout=60) + resp.raise_for_status() + + resp_json = resp.json() + if isinstance(resp_json, dict): + code = resp_json.get("code") + message = resp_json.get("message") + resp_preview = dict(resp_json) + b64 = resp_json.get("data") + if isinstance(b64, str) and len(b64) > 200: + resp_preview["data"] = self._preview_b64(b64) + if code not in (3000, 0, None): + raise RuntimeError(f"bytedance tts failed: code={code}, message={message}, resp={resp_preview}") + if message not in (None, "Success") and code is None: + raise RuntimeError(f"bytedance tts failed: message={message}, resp={resp_json}") + + b64 = resp_json.get("data") + if not b64: + raise RuntimeError(f"bytedance tts failed: no data in resp={resp_json}") + + audio_bytes = base64.b64decode(b64) + wav_path.write_bytes(audio_bytes) + return + + raise RuntimeError(f"bytedance tts failed: invalid resp: {resp.text}") + + def _tts_minimax_sync( + self, + *, + text: str, + wav_path: Path, + secrets: Dict[str, Any], + tts_params: Dict[str, Any], + provider_cfg: Dict[str, Any], + ) -> None: + + base_url = secrets.get("base_url") or "https://api.minimax.chat" + api_url = base_url.rstrip("/") + "/v1/t2a_v2" if not base_url.endswith("/v1/t2a_v2") else base_url + + api_key = secrets.get("api_key") or secrets.get("token") or secrets.get("access_token") + if not api_key: + for k, v in secrets.items(): + if k != "base_url" and isinstance(v, str) and v.strip(): + api_key = v.strip() + break + if not api_key: + raise ValueError("minimax missing api_key/token/access_token") + + headers = { + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + } + + body = { + "model": tts_params.get("model", "speech-02-hd"), + "text": text, + "stream": False, + "language_boost": tts_params.get("language_boost", "auto"), + "output_format": tts_params.get("output_format", "hex"), + "voice_setting": { + "voice_id": tts_params.get("voice_id", "English_expressive_narrator"), + "speed": float(tts_params.get("speed", 1.0)), + "vol": float(tts_params.get("vol", 1.0)), + "pitch": int(tts_params.get("pitch", 0)), + }, + "audio_setting": { + "sample_rate": int(tts_params.get("sample_rate", 24000)), + "bitrate": int(tts_params.get("bitrate", 128000)), + "format": tts_params.get("format", "wav"), + }, + } + + resp = requests.post(api_url, headers=headers, json=body, timeout=120) + resp.raise_for_status() + + resp_json = resp.json() + base_resp = (resp_json or {}).get("base_resp") or {} + if base_resp.get("status_code") not in (0, None): + raise RuntimeError(f"minimax tts failed: {resp_json}") + + data = (resp_json or {}).get("data") or {} + audio_field = data.get("audio") + if not audio_field: + raise RuntimeError(f"minimax tts failed: no data.audio: {resp_json}") + + # output_format = hex or url + if isinstance(audio_field, str) and audio_field.startswith("http"): + audio_bytes = requests.get(audio_field, timeout=120).content + wav_path.write_bytes(audio_bytes) + return + + try: + audio_bytes = binascii.unhexlify(audio_field) + except Exception as e: + raise RuntimeError(f"minimax hex decode failed: {e}, audio_field[:64]={str(audio_field)[:64]}") + + wav_path.write_bytes(audio_bytes) + + def _tts_302_sync( + self, + *, + text: str, + wav_path: Path, + secrets: Dict[str, Any], + tts_params: Dict[str, Any], + provider_cfg: Dict[str, Any], + ) -> None: + base_url = (secrets.get("base_url") or "https://api.302.ai").rstrip("/") + api_url = base_url + "/302/audio/speech" + + api_key = secrets.get("api_key") or secrets.get("token") or secrets.get("access_token") + if not api_key: + for k, v in secrets.items(): + if k != "base_url" and isinstance(v, str) and v.strip(): + api_key = v.strip() + break + if not api_key: + raise ValueError("302 missing api_key/token/access_token") + + headers = { + "Authorization": f"Bearer {api_key}", + "Accept": "audio/wav", + "Content-Type": "application/json", + } + + body = { + "model": tts_params.get("model", "speech-02-hd"), + "input": text, + "voice": tts_params.get("voice", "alloy"), + "emotion": tts_params.get("emotion", "neutral"), + "response_format": tts_params.get("response_format", "wav"), + } + + resp = requests.post(api_url, headers=headers, json=body, timeout=120) + if not resp.ok: + raise RuntimeError(f"302 tts http {resp.status_code}: {resp.text}") + wav_path.write_bytes(resp.content) \ No newline at end of file diff --git a/src/open_storyline/nodes/core_nodes/group_clips.py b/src/open_storyline/nodes/core_nodes/group_clips.py new file mode 100644 index 0000000000000000000000000000000000000000..5e81b77d7be0365629c14069d2e623965c1f1b87 --- /dev/null +++ b/src/open_storyline/nodes/core_nodes/group_clips.py @@ -0,0 +1,198 @@ +from typing import Any, Dict + +from open_storyline.nodes.core_nodes.base_node import BaseNode, NodeMeta +from open_storyline.nodes.node_state import NodeState +from open_storyline.mcp.sampling_requester import LLMClient +from open_storyline.nodes.node_schema import GroupClipsInput +from src.open_storyline.utils.prompts import get_prompt +from open_storyline.utils.parse_json import parse_json_dict +from open_storyline.utils.register import NODE_REGISTRY + +@NODE_REGISTRY.register() +class GroupClipsNode(BaseNode): + + meta = NodeMeta( + name="group_clips", + description="Group clips based on their descriptions according to user requirements. Depends on the filter_clips tool output", + node_id="group_clips", + node_kind="group_clips", + require_prior_kind=['filter_clips'], + default_require_prior_kind=['filter_clips'], + next_available_node=['generate_script', 'generate_script_pro'], + ) + input_schema = GroupClipsInput + + async def default_process( + self, + node_state: NodeState, + inputs: Dict[str, Any], + ) -> Any: + result = _make_single_group_fallback(inputs["filter_clips"].get("selected", [])) + return { + "groups": result, + } + + async def process(self, node_state: NodeState, inputs: Dict[str, Any], **params) -> Any: + clip_captions = inputs["filter_clips"].get("clip_captions") + selected_clips = inputs["filter_clips"].get("selected") + user_request = inputs["user_request"] + + llm = node_state.llm + clip_lookup = _build_clip_lookup(clip_captions) + + if not selected_clips: + return {"groups": []} + + selected_clips_captions = [clip_lookup[cid] for cid in selected_clips] + clip_block = _build_clips_block(selected_clips_captions) + + system_prompt = get_prompt("group_clips.system", lang=node_state.lang) + if user_request == "": + user_request = "No additional requirements" + + user_prompt = get_prompt( + "group_clips.user", + lang=node_state.lang, + user_request=user_request, + selected_clips=selected_clips, + clip_captions=clip_block, + clip_number=len(clip_block), + ) + + raw = await llm.complete( + system_prompt=system_prompt, + user_prompt=user_prompt, + media=None, + temperature=0.1, + top_p=0.9, + max_tokens=4096, + model_preferences=None, + ) + + try: + obj = parse_json_dict(raw) + groups_raw = _extract_groups_obj(obj) + + groups = _normalize_groups_from_llm( + groups_raw=groups_raw, + selected_ids_set=set(selected_clips), + ) + + node_state.node_summary.info_for_user(f"Grouping successful: {len(groups)} groups in total") + return { + "groups": groups, + } + except Exception as e: + + result = _make_single_group_fallback(selected_clips) + node_state.node_summary.info_for_user(f"Grouping error: {e}\nUsing default strategy") + return { + "groups": result, + } + +def _extract_groups_obj(obj: Any) -> list[dict[str, Any]]: + if isinstance(obj, dict) and isinstance(obj.get("groups"), list): + return obj["groups"] + if isinstance(obj, list): + return obj + raise ValueError("LLM output does not contain groups.") + + +def _normalize_groups_from_llm( + groups_raw: list[dict[str, Any]], + selected_ids_set: set[str], +) -> list[dict[str, Any]]: + """ + Validate and normalize LLM output groups: + - clip_ids must all come from selected + - clip_ids cannot be duplicated + - group_id will be uniformly rewritten by code + - If summary is missing, fill with default + """ + if not groups_raw: + raise ValueError("groups is empty.") + + # First extract and perform basic cleaning + normalized_groups: list[dict[str, Any]] = [] + seen: set[str] = set() + + for gi, g in enumerate(groups_raw): + if not isinstance(g, dict): + raise ValueError(f"groups[{gi}] is not a dict, please try running again.") + + clip_ids = g.get("clip_ids") + if not isinstance(clip_ids, list) or not clip_ids: + raise ValueError(f"groups[{gi}].clip_ids must be a non-empty list, please try running again.") + + # Deduplicate clip_ids (preserve original output order) + cleaned_clip_ids: list[str] = [] + + for cid in clip_ids: + if not isinstance(cid, str): + continue + if cid not in selected_ids_set: + continue + # raise ValueError(f"groups[{gi}] contains non-selected clip_id: {cid}") + if cid in seen: + continue + seen.add(cid) + cleaned_clip_ids.append(cid) + + if not cleaned_clip_ids: + raise ValueError(f"groups[{gi}] clip_ids is empty after cleaning, please try running again.") + + summary = g.get("summary") + if not isinstance(summary, str) or not summary.strip(): + summary = "A group of shots for carrying the same script and voiceover." + + normalized_groups.append( + { + "group_id": "", # group_id placeholder, will be rewritten later + "summary": summary.strip(), + "clip_ids": cleaned_clip_ids, + "duration": g.get("duration") + } + ) + + # Finally rewrite group_id + for i, g in enumerate(normalized_groups, start=1): + g["group_id"] = f"group_{i:04d}" + + return normalized_groups + + +def _build_clip_lookup(clip_captions: list[dict[str, Any]]) -> dict[str, dict[str, Any]]: + lookup: dict[str, dict[str, Any]] = {} + for clip in clip_captions: + cid = clip.get("clip_id") + if cid: + lookup[cid] = clip + return lookup + +def _make_single_group_fallback( + selected_clips: list[str], +) -> list[dict[str, Any]]: + return [ + { + "group_id": "group_0001", + "summary": "Aggregate all selected shots in original order for subsequent script and voiceover generation.", + "clip_ids": selected_clips, + } + ] + +def _build_clips_block(clip_captions: list[dict[str, Any]]) -> str: + """ + Construct clips into stable text blocks + """ + blocks: list[dict] = [] + for clip in clip_captions: + clip_id = clip.get("clip_id", "") + duration = clip.get("duration",0.0) + caption = clip.get("caption", "") + block = { + "clip_id": clip_id, + "duration": duration, + "caption": caption + } + blocks.append(block) + return blocks diff --git a/src/open_storyline/nodes/core_nodes/load_media.py b/src/open_storyline/nodes/core_nodes/load_media.py new file mode 100644 index 0000000000000000000000000000000000000000..c9dc4d89b7e507dd716c41bb2f76595f0d52b438 --- /dev/null +++ b/src/open_storyline/nodes/core_nodes/load_media.py @@ -0,0 +1,165 @@ +from typing import Any, Dict, Optional, ClassVar, Type +from pydantic import BaseModel +import traceback +from collections import Counter +from pathlib import Path +from moviepy.video.io.ffmpeg_reader import ffmpeg_parse_infos + + +from open_storyline.nodes.core_nodes.base_node import NodeMeta, BaseNode +from open_storyline.nodes.node_schema import LoadMediaInput, LoadMediaOutput +from open_storyline.nodes.node_state import NodeState +from open_storyline.utils.util import get_video_rotation +from open_storyline.utils.register import NODE_REGISTRY + + +VIDEO_EXTS = { + ".mp4", ".mov", ".mkv", ".avi" +} +IMAGE_EXTS = { + ".jpg", ".jpeg", ".png", ".webp", ".bmp" +} + +def _image_metadata_from_path(path: Path) -> dict[str, Any]: + from PIL import Image, ImageOps + + with Image.open(path) as img: + try: + img2 = ImageOps.exif_transpose(img) + w, h = img2.size + except Exception: + w, h = img.size + + return { + "width": int(w), + "height": int(h), + } + + +import av +from fractions import Fraction +from typing import Any, Optional +from pathlib import Path + + +def _video_metadata_from_path( + path: Path, + *, + round_duration_ndigits: Optional[int] = 3, +) -> dict[str, Any]: + + container = av.open(str(path)) + + # 找第一个视频流 + video_stream = next( + (s for s in container.streams if s.type == "video"), + None, + ) + if video_stream is None: + raise ValueError(f"No video stream found: {path}") + + # ---------- duration ---------- + duration_sec = 0.0 + + if container.duration is not None: + # container.duration 单位是 microseconds + duration_sec = container.duration / 1_000_000 + elif video_stream.duration is not None and video_stream.time_base is not None: + duration_sec = float(video_stream.duration * video_stream.time_base) + + if round_duration_ndigits is not None: + duration_sec = round(duration_sec, round_duration_ndigits) + + # ---------- width / height / rotation ---------- + w = int(video_stream.codec_context.width or 0) + h = int(video_stream.codec_context.height or 0) + + rotation = get_video_rotation(path) + + if abs(rotation) in (90, 270): + w, h = h, w + + # ---------- fps ---------- + fps = 0.0 + if video_stream.average_rate: + fps = float(video_stream.average_rate) + elif video_stream.base_rate: + fps = float(video_stream.base_rate) + + # ---------- audio ---------- + audio_stream = next( + (s for s in container.streams if s.type == "audio"), + None, + ) + + has_audio = audio_stream is not None + audio_sample_rate_hz = int(audio_stream.rate) if audio_stream and audio_stream.rate else 0 + + container.close() + + return { + "duration": int(duration_sec * 1000), # ms + "width": w, + "height": h, + "fps": fps, + "has_audio": has_audio, + "audio_sample_rate_hz": audio_sample_rate_hz, + } + + +@NODE_REGISTRY.register() +class LoadMediaNode(BaseNode): + meta = NodeMeta( + name="load_media", + description="Loads and indexes input media. Entry point with no dependencies; required by all downstream operations", + node_id="load_media", + node_kind="load_media", + next_available_node=['split_shots', 'split_shots_pro'], + ) + input_schema: ClassVar[Type[BaseModel]] = LoadMediaInput + # output_schema: ClassVar[Type[BaseModel]] = LoadMediaOutput + + async def default_process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Dict[str, Any]: + return await self.process(node_state, inputs) + + async def process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Dict[str, Any]: + input_media = inputs.get('inputs', []) + + media_idx = 1 + media = [] + for enc_media in input_media: + path = Path(enc_media['path']) + suffix = path.suffix.lower() + + if suffix in VIDEO_EXTS: + metadata = _video_metadata_from_path(path) + media_type = "video" + elif suffix in IMAGE_EXTS: + metadata = _image_metadata_from_path(path) + media_type = "image" + else: + node_state.node_summary.info_for_user(f"[Node {self.meta.node_id}] Skipping unsupported file type `{enc_media['orig_path']}` ") + continue + + media.append( + { + "media_id": f"media_{media_idx:04d}", + "path": path, + "media_type": media_type, + "metadata": metadata, + "orig_path": enc_media['orig_path'], + "orig_md5": enc_media['orig_md5'], + } + ) + node_state.node_summary.info_for_user(f"Added media_{media_idx:04d}: ({media_type})") + media_idx += 1 + + c = Counter( + (a.get("media_type") or "").strip().lower() + for a in media + if isinstance(a, dict) + ) + + node_state.node_summary.info_for_user(f"[Node {self.meta.node_id}] Media indexing completed successfully: {c.get('video', 0)} video(s), {c.get('image', 0)} image(s)",) + + return {"media": media} \ No newline at end of file diff --git a/src/open_storyline/nodes/core_nodes/plan_timeline.py b/src/open_storyline/nodes/core_nodes/plan_timeline.py new file mode 100644 index 0000000000000000000000000000000000000000..de3c2125ed7525813518a31a53d37b3324a93e78 --- /dev/null +++ b/src/open_storyline/nodes/core_nodes/plan_timeline.py @@ -0,0 +1,907 @@ +from __future__ import annotations + +import random +from dataclasses import dataclass +from typing import Any, Dict, List, Mapping, MutableMapping, Optional, Tuple + +from src.open_storyline.config import Settings +from open_storyline.config import PlanTimelineConfig +from open_storyline.nodes.node_state import NodeState +from open_storyline.nodes.core_nodes.base_node import BaseNode, NodeMeta +from open_storyline.nodes.node_schema import PlanTimelineInput +from open_storyline.utils.register import NODE_REGISTRY + +# ========================= +# Constants (no magic numbers) +# ========================= +Milliseconds = int + +DEFAULT_RANDOM_SEED = 42 + +SECONDS_PER_MINUTE = 60.0 +MILLISECONDS_PER_SECOND = 1000.0 + +SNAP_SAFETY_MAX_STEPS = 10_000 +BINARY_SEARCH_ITERATIONS = 50 + +RATIO_GROWTH_FACTOR = 2.0 +RATIO_GROWTH_MAX = 10.0 + +MIN_SUBTITLE_WEIGHT = 1 +CENTER_ALIGN_DIVISOR = 2.0 + + +@dataclass(frozen=True) +class BeatTrack: + """Beat-related information derived from background music (BGM).""" + + beat_timestamps_ms: List[Milliseconds] + beat_durations_ms: List[Milliseconds] + music_duration_ms: Milliseconds + + +class TimelinePlanner: + """ + Pure timeline planning logic extracted from PlanTimelineNode for: + - clearer separation of concerns + - better readability + - easier unit testing + """ + + def __init__(self, config: PlanTimelineConfig, *, random_seed: int = DEFAULT_RANDOM_SEED) -> None: + self._config = config + self._random_generator = random.Random(random_seed) + + def plan( + self, + *, + media: List[Dict[str, Any]], + clips: List[Dict[str, Any]], + groups: List[Dict[str, Any]], + group_scripts: List[Dict[str, Any]], + voiceovers: List[Dict[str, Any]], + background_music: Optional[Dict[str, Any]], + use_beats: bool, + ) -> Dict[str, Any]: + """Plan full timeline tracks: video/subtitles/voiceover/bgm.""" + media_by_media_id = self._build_item_index(media, id_key="media_id") + clips_by_clip_id = self._build_item_index(clips, id_key="clip_id") + script_by_group_id = self._build_item_index(group_scripts, id_key="group_id") + voiceover_by_group_id = self._build_item_index(voiceovers, id_key="group_id") + + beat_track = self._build_beat_track(background_music, use_beats=use_beats) + + music_offset_ms, start_beat_index = self._compute_title_music_offset( + beat_durations_ms=beat_track.beat_durations_ms, + music_duration_ms=beat_track.music_duration_ms, + use_beats=use_beats, + ) + + video_segments, group_states, total_duration_ms, _end_beat_index = self._build_video_track( + groups=groups, + clips_by_clip_id=clips_by_clip_id, + media_by_media_id=media_by_media_id, + script_by_group_id=script_by_group_id, + voiceover_by_group_id=voiceover_by_group_id, + background_music=background_music, + beat_durations_ms=beat_track.beat_durations_ms, + start_beat_index=start_beat_index, + use_beats=use_beats, + ) + + voiceover_segments = self._build_voiceover_track(groups=groups, group_states=group_states) + subtitle_segments = self._build_subtitle_track(groups=groups, group_states=group_states) + bgm_segments = self._build_bgm_track( + background_music=background_music, + total_duration_ms=total_duration_ms, + music_offset_ms=music_offset_ms, + ) + + return { + "tracks": { + "video": video_segments, + "subtitles": subtitle_segments, + "voiceover": voiceover_segments, + "bgm": bgm_segments, + } + } + + # ----------------------------- + # Track builders + # ----------------------------- + def _build_video_track( + self, + *, + groups: List[Dict[str, Any]], + clips_by_clip_id: Mapping[str, Dict[str, Any]], + media_by_media_id: Mapping[str, Dict[str, Any]], + script_by_group_id: Mapping[str, Dict[str, Any]], + voiceover_by_group_id: Mapping[str, Dict[str, Any]], + background_music: Optional[Dict[str, Any]], + beat_durations_ms: List[Milliseconds], + start_beat_index: int, + use_beats: bool, + ) -> Tuple[List[Dict[str, Any]], Dict[str, Dict[str, Any]], Milliseconds, int]: + video_segments: List[Dict[str, Any]] = [] + group_states: Dict[str, Dict[str, Any]] = {} + + timeline_cursor_ms: Milliseconds = 0 + residual_ms: Milliseconds = 0 # preserved for future improvements; legacy code kept it but didn't update it. + + beat_index = int(start_beat_index) + + for group in groups: + group_id = self._to_str_id(group.get("group_id")) + clip_ids = [self._to_str_id(cid) for cid in (group.get("clip_ids", []) or [])] + if not clip_ids: + raise ValueError( + f"group {group_id} has no clip_ids, please check the result of 'group_clips' node" + ) + + clip_items: List[Dict[str, Any]] = [] + for clip_id in clip_ids: + if clip_id not in clips_by_clip_id: + raise KeyError( + f"group {group_id} references missing clip_id={clip_id}, " + "please check the result of 'group_clips' and 'split_shots' node" + ) + clip_items.append(clips_by_clip_id[clip_id]) + + group_script = script_by_group_id.get(group_id) + group_voiceover = voiceover_by_group_id.get(group_id) + + # Case A: no script, no voiceover, and no beat snapping -> concatenate clips as-is. + if not group_script and not group_voiceover and (not background_music or not use_beats): + group_start_ms = timeline_cursor_ms + first_clip_source_duration_ms: Milliseconds = 0 + + for index_in_group, clip in enumerate(clip_items): + clip_id = self._to_str_id(clip.get("clip_id")) + clip_kind = str(clip.get("kind", "video")) + + source_start_ms, source_end_ms, source_duration_ms = self._full_source_window_and_duration_ms( + clip + ) + if index_in_group == 0: + first_clip_source_duration_ms = source_duration_ms + + segment_start_ms = timeline_cursor_ms + segment_end_ms = segment_start_ms + source_duration_ms + + video_segments.append( + { + "clip_id": clip_id, + "group_id": group_id, + "kind": clip_kind, + "path": clip.get("path"), + "fps": clip.get("fps"), + "source_path": self._resolve_source_path( + clip=clip, media_by_media_id=media_by_media_id + ), + "source_window": {"start": source_start_ms, "end": source_end_ms}, + # NOTE: the original code used {"start": seg_end, "end": seg_end} here, + # which produces a 0-length window. This is almost certainly a typo. + "timeline_window": {"start": segment_start_ms, "end": segment_end_ms}, + "playback_rate": 1.0, + } + ) + timeline_cursor_ms = segment_end_ms + + group_end_ms = timeline_cursor_ms + group_states[group_id] = { + "group_id": group_id, + "start": group_start_ms, + "end": group_end_ms, + "duration": group_end_ms - group_start_ms, + "first_clip_duration": first_clip_source_duration_ms, + } + continue + + # Case B: voiceover duration is authoritative; otherwise estimate by text length. + if group_voiceover and group_voiceover.get("duration", 0) > 0: + narration_duration_ms: Milliseconds = int(group_voiceover.get("duration", 0)) + else: + narration_duration_ms = self._estimate_group_duration_from_text_ms(group_script) + + group_target_duration_ms: Milliseconds = narration_duration_ms + int(self._config.group_margin_over_voiceover) + group_target_duration_ms = max( + group_target_duration_ms, len(clip_items) * int(self._config.min_clip_duration) + ) + + if use_beats and background_music: + durations_ms, beat_index = self._allocate_clip_durations_using_beats( + clip_items=clip_items, + group_target_ms=group_target_duration_ms, + beat_durations_ms=beat_durations_ms, + start_beat_index=beat_index, + start_residual_ms=residual_ms, + ) + else: + durations_ms = self._allocate_clip_durations_without_beats( + clip_items=clip_items, group_target_ms=group_target_duration_ms + ) + + group_start_ms = timeline_cursor_ms + first_clip_planned_duration_ms: Milliseconds = durations_ms[0] if durations_ms else 0 + + for clip, planned_duration_ms in zip(clip_items, durations_ms): + clip_id = clip.get("clip_id") # keep legacy behavior (may be int) + clip_kind = str(clip.get("kind", "video")).lower() + + segment_start_ms = timeline_cursor_ms + segment_end_ms = segment_start_ms + int(planned_duration_ms) + + source_start_ms, source_end_ms, source_available_ms = self._full_source_window_and_duration_ms(clip) + playback_rate = 1.0 + + if clip_kind == "video": + if planned_duration_ms > source_available_ms: + playback_rate = (source_available_ms / planned_duration_ms) if planned_duration_ms > 0 else 1.0 + source_window_start_ms, source_window_end_ms = source_start_ms, source_end_ms + else: + source_window_start_ms, source_window_end_ms = self._choose_source_window_for_timeline_duration_ms( + clip=clip, used_timeline_duration_ms=int(planned_duration_ms) + ) + else: + # image (and other non-video kinds): use from src_start for the planned duration + source_window_start_ms, source_window_end_ms = ( + source_start_ms, + source_start_ms + int(planned_duration_ms), + ) + + video_segments.append( + { + "clip_id": clip_id, + "group_id": group_id, + "kind": clip_kind, + "path": clip.get("path"), + "orig_path": clip.get("orig_path"), + "fps": clip.get("fps"), + "size": clip.get("size"), + "source_path": self._resolve_source_path( + clip=clip, media_by_media_id=media_by_media_id + ), + "source_window": { + "start": source_window_start_ms, + "end": source_window_end_ms, + "duration": source_window_end_ms - source_window_start_ms, + }, + "timeline_window": { + "start": segment_start_ms, + "end": segment_end_ms, + "duration": segment_end_ms - segment_start_ms, + }, + "playback_rate": playback_rate, + } + ) + + timeline_cursor_ms = segment_end_ms + + group_end_ms = timeline_cursor_ms + group_states[group_id] = { + "group_id": group_id, + "start": group_start_ms, + "end": group_end_ms, + "duration": group_end_ms - group_start_ms, + "first_clip_duration": int(first_clip_planned_duration_ms), + "n_clips": len(clip_items), + "narration_duration": narration_duration_ms, + "group_margin": int(self._config.group_margin_over_voiceover), + "voiceover": group_voiceover, + "script": group_script, + } + + total_duration_ms = timeline_cursor_ms + return video_segments, group_states, total_duration_ms, beat_index + + def _build_voiceover_track( + self, *, groups: List[Dict[str, Any]], group_states: MutableMapping[str, Dict[str, Any]] + ) -> List[Dict[str, Any]]: + voiceover_segments: List[Dict[str, Any]] = [] + + for group in groups: + group_id = self._to_str_id(group.get("group_id", "")) + state = group_states.get(group_id) + if not state: + continue + + voiceover_item = state.get("voiceover") + if not voiceover_item: + continue + + voiceover_duration_ms = int(voiceover_item.get("duration", 0)) + if voiceover_duration_ms <= 0: + continue + + group_start_ms = int(state.get("start", 0)) + group_end_ms = int(state.get("end", 0)) + group_duration_ms = group_end_ms - group_start_ms + + slack_ms = max(0, group_duration_ms - voiceover_duration_ms) + + start_offset_ms = slack_ms / CENTER_ALIGN_DIVISOR + voiceover_start_ms = group_start_ms + start_offset_ms + voiceover_end_ms = voiceover_start_ms + voiceover_duration_ms + + voiceover_segments.append( + { + "group_id": group_id, + "voiceover_id": voiceover_item.get("voiceover_id"), + "path": voiceover_item.get("path"), + "source_window": {"start": 0, "end": voiceover_duration_ms, "duration": voiceover_duration_ms}, + "timeline_window": { + "start": voiceover_start_ms, + "end": voiceover_end_ms, + "duration": voiceover_duration_ms, + }, + } + ) + + state["voiceover_timeline"] = { + "start": voiceover_start_ms, + "end": voiceover_end_ms, + "duration": voiceover_duration_ms, + } + + return voiceover_segments + + def _build_subtitle_track( + self, *, groups: List[Dict[str, Any]], group_states: Mapping[str, Dict[str, Any]] + ) -> List[Dict[str, Any]]: + subtitle_segments: List[Dict[str, Any]] = [] + + for group in groups: + group_id = self._to_str_id(group.get("group_id", "")) + state = group_states.get(group_id) + if not state: + continue + + # NOTE: original code used default "" then script.get(...) -> may crash. + group_script = state.get("script") or {} + subtitle_units = group_script.get("subtitle_units", []) if isinstance(group_script, dict) else [] + if not subtitle_units: + continue + + group_start_ms = int(state.get("start", 0)) + group_end_ms = int(state.get("end", 0)) + voiceover_timeline = state.get("voiceover_timeline") + + if voiceover_timeline is not None: + voiceover_start_ms = voiceover_timeline["start"] + voiceover_end_ms = voiceover_timeline["end"] + + if not (group_start_ms <= voiceover_start_ms <= voiceover_end_ms <= group_end_ms): + subtitle_start_ms = group_start_ms + subtitle_end_ms = group_end_ms + else: + subtitle_start_ms = int((group_start_ms + voiceover_start_ms) / CENTER_ALIGN_DIVISOR) + subtitle_end_ms = int((voiceover_end_ms + group_end_ms) / CENTER_ALIGN_DIVISOR) + else: + subtitle_start_ms = group_start_ms + subtitle_end_ms = group_end_ms + + subtitle_duration_ms = subtitle_end_ms - subtitle_start_ms + + unit_texts: List[str] = [str(u.get("text") or "") for u in subtitle_units] + unit_weights: List[int] = [max(MIN_SUBTITLE_WEIGHT, len(text.strip())) for text in unit_texts] + total_weight = sum(unit_weights) + + unit_durations_ms: List[Milliseconds] = [] + accumulated_ms: Milliseconds = 0 + for i, weight in enumerate(unit_weights): + if i == len(unit_weights) - 1: + duration_ms = max(0, subtitle_duration_ms - accumulated_ms) + else: + duration_ms = (subtitle_duration_ms * weight) // total_weight if total_weight > 0 else 0 + accumulated_ms += int(duration_ms) + unit_durations_ms.append(int(duration_ms)) + + subtitle_cursor_ms = subtitle_start_ms + for unit, text, duration_ms in zip(subtitle_units, unit_texts, unit_durations_ms): + if duration_ms <= 0: + continue + segment_start_ms = subtitle_cursor_ms + segment_end_ms = subtitle_cursor_ms + int(duration_ms) + subtitle_cursor_ms = segment_end_ms + + subtitle_segments.append( + { + "group_id": group_id, + "unit_id": unit.get("unit_id"), + "index_in_group": unit.get("index_in_group"), + "text": text, + "timeline_window": {"start": segment_start_ms, "end": segment_end_ms}, + } + ) + + return subtitle_segments + + def _build_bgm_track( + self, + *, + background_music: Optional[Dict[str, Any]], + total_duration_ms: Milliseconds, + music_offset_ms: Milliseconds, + ) -> List[Dict[str, Any]]: + bgm_segments: List[Dict[str, Any]] = [] + if not background_music: + return bgm_segments + + music_duration_ms = int(background_music.get("duration", 0)) + if music_duration_ms <= 0: + return bgm_segments + + timeline_cursor_ms: Milliseconds = 0 + source_cursor_ms: Milliseconds = int(music_offset_ms) + loop_index = 0 + + while timeline_cursor_ms < total_duration_ms: + remaining_timeline_ms = total_duration_ms - timeline_cursor_ms + remaining_source_ms = max(0, music_duration_ms - source_cursor_ms) + + if remaining_source_ms <= 0: + source_cursor_ms = 0 + loop_index += 1 + continue + + segment_duration_ms = min(remaining_timeline_ms, remaining_source_ms) + if segment_duration_ms <= 0: + break + + bgm_segments.append( + { + "bgm_id": background_music.get("bgm_id"), + "path": background_music.get("path"), + "source_window": {"start": source_cursor_ms, "end": source_cursor_ms + segment_duration_ms}, + "loop_idx": loop_index, + } + ) + + timeline_cursor_ms += segment_duration_ms + source_cursor_ms += segment_duration_ms + + if timeline_cursor_ms < total_duration_ms and source_cursor_ms >= music_duration_ms: + source_cursor_ms = 0 + loop_index += 1 + + return bgm_segments + + # ----------------------------- + # Beats & title alignment + # ----------------------------- + def _build_beat_track(self, background_music: Optional[Dict[str, Any]], *, use_beats: bool) -> BeatTrack: + if not use_beats or not background_music: + return BeatTrack(beat_timestamps_ms=[], beat_durations_ms=[], music_duration_ms=0) + + music_duration_ms = int(background_music.get("duration", 0)) + beat_timestamps_ms = self._build_beat_timestamps_from_music_ms(background_music) + beat_durations_ms = self._convert_beat_timestamps_to_durations_ms( + beat_timestamps_ms=beat_timestamps_ms, music_duration_ms=music_duration_ms + ) + + return BeatTrack( + beat_timestamps_ms=beat_timestamps_ms, + beat_durations_ms=beat_durations_ms, + music_duration_ms=music_duration_ms, + ) + + def _compute_title_music_offset( + self, *, beat_durations_ms: List[Milliseconds], music_duration_ms: Milliseconds, use_beats: bool + ) -> Tuple[Milliseconds, int]: + """ + Compute the BGM source offset so that the title ends on a beat. + + NOTE: + - The original code assumes beat_durations_ms is non-empty; otherwise modulo would crash. + Here we guard against empty beats (safer for open-source usage). + """ + music_offset_ms: Milliseconds = 0 + beat_index = 0 + + if not use_beats: + return music_offset_ms, beat_index + + title_duration_ms = int(getattr(self._config, "title_duration", 0)) + if title_duration_ms <= 0 or music_duration_ms <= 0 or not beat_durations_ms: + return music_offset_ms, beat_index + + if bool(getattr(self._config, "bgm_loop", False)): + title_duration_ms = title_duration_ms % music_duration_ms + else: + title_duration_ms = min(title_duration_ms, music_duration_ms) + + cumulative_ms: Milliseconds = 0 + duration_index = 0 + while duration_index < len(beat_durations_ms) and cumulative_ms < title_duration_ms: + cumulative_ms += int(beat_durations_ms[duration_index]) + duration_index += 1 + + music_offset_ms = max(0, cumulative_ms - title_duration_ms) + beat_index = duration_index % len(beat_durations_ms) + return int(music_offset_ms), int(beat_index) + + # ----------------------------- + # Shared helpers (indexing, parsing, math) + # ----------------------------- + @staticmethod + def _to_str_id(value: Any) -> str: + return "" if value is None else str(value) + + @classmethod + def _build_item_index(cls, items: List[Dict[str, Any]], *, id_key: str) -> Dict[str, Dict[str, Any]]: + return {cls._to_str_id(item.get(id_key)): item for item in (items or [])} + + @staticmethod + def _resolve_source_path(clip: Dict[str, Any], *, media_by_media_id: Mapping[str, Dict[str, Any]]) -> Optional[str]: + source_ref = clip.get("source_ref") or {} + media_id = "" if source_ref is None else str(source_ref.get("media_id", "")) + return media_by_media_id.get(media_id, {}).get("path") + + @staticmethod + def _safe_float(value: Any, default_value: float = 0.0) -> float: + try: + if value is None: + return default_value + return float(value) + except Exception: + return default_value + + # ----------------------------- + # Beats helpers + # ----------------------------- + def _build_beat_timestamps_from_music_ms(self, background_music: Dict[str, Any]) -> List[Milliseconds]: + beats: List[Milliseconds] = background_music.get("beats", []) or [] + music_duration_ms = int(background_music.get("duration", 0)) + + if beats: + if beats[0] != 0: + return [0] + beats + return beats + + bpm = background_music.get("bpm") + if bpm is None: + return [0] + + bpm_value = self._safe_float(bpm, 0.0) + if bpm_value <= 0 or music_duration_ms <= 0: + return [0] + + interval_ms = int(SECONDS_PER_MINUTE / bpm_value * MILLISECONDS_PER_SECOND) + + timestamps_ms: List[Milliseconds] = [0] + timestamp_ms: Milliseconds = interval_ms + while timestamp_ms <= music_duration_ms: + timestamps_ms.append(int(timestamp_ms)) + timestamp_ms += interval_ms + + return timestamps_ms + + @staticmethod + def _convert_beat_timestamps_to_durations_ms( + *, beat_timestamps_ms: List[Milliseconds], music_duration_ms: Milliseconds + ) -> List[Milliseconds]: + if len(beat_timestamps_ms) < 2: + return [] + + durations_ms: List[Milliseconds] = [] + for start_ms, end_ms in zip(beat_timestamps_ms[:-1], beat_timestamps_ms[1:]): + delta_ms = int(end_ms) - int(start_ms) + if delta_ms > 0: + durations_ms.append(int(delta_ms)) + + if music_duration_ms > 0: + tail_ms = max(0, int(music_duration_ms) - int(beat_timestamps_ms[-1])) + if tail_ms > 0: + durations_ms.append(int(tail_ms)) + + return durations_ms + + # ----------------------------- + # Text duration estimate + # ----------------------------- + def _estimate_group_duration_from_text_ms(self, group_script: Optional[Dict[str, Any]]) -> Milliseconds: + if not group_script: + return int(self._config.estimate_text_min) + + raw_text = str(group_script.get("raw_text") or "") + char_count = len(raw_text.strip()) + if char_count <= 0: + return int(self._config.estimate_text_min) + + chars_per_second = max(1.0, float(self._config.estimate_text_char_per_sec)) + duration_ms = int(char_count / chars_per_second * MILLISECONDS_PER_SECOND) + return max(int(duration_ms), int(self._config.estimate_text_min)) + + # ----------------------------- + # Clip/source windows + # ----------------------------- + def _full_source_window_and_duration_ms( + self, clip: Dict[str, Any] + ) -> Tuple[Milliseconds, Milliseconds, Milliseconds]: + clip_kind = str(clip.get("kind", "video")) + source_ref = clip.get("source_ref") or {} + + start_ms = int(source_ref.get("start", 0)) + end_ms = int(source_ref.get("end", start_ms)) + duration_ms = int(source_ref.get("duration", end_ms - start_ms)) + + if clip_kind == "image": + start_ms = 0 + end_ms = int(self._config.image_default_duration) + duration_ms = end_ms - start_ms + else: + if duration_ms <= 0: + clip_id = clip.get("clip_id") + raise ValueError( + f"{clip_id} has invalid source window (start={start_ms}, end={end_ms}, duration={duration_ms})" + ) + + return int(start_ms), int(end_ms), int(duration_ms) + + def _choose_source_window_for_timeline_duration_ms( + self, *, clip: Dict[str, Any], used_timeline_duration_ms: Milliseconds + ) -> Tuple[Milliseconds, Milliseconds]: + source_start_ms, _, source_duration_ms = self._full_source_window_and_duration_ms(clip) + + random_offset_ms = int(self._random_generator.random() * (source_duration_ms - used_timeline_duration_ms)) + window_start_ms = source_start_ms + random_offset_ms + window_end_ms = window_start_ms + int(used_timeline_duration_ms) + return int(window_start_ms), int(window_end_ms) + + # ----------------------------- + # Duration allocation + # ----------------------------- + def _allocate_clip_durations_using_beats( + self, + *, + clip_items: List[Dict[str, Any]], + group_target_ms: Milliseconds, + beat_durations_ms: List[Milliseconds], + start_beat_index: int, + start_residual_ms: Milliseconds, + ) -> Tuple[List[Milliseconds], int]: + """ + Algorithm (kept the same as legacy version): + 1) Allocate ideal duration per clip by source duration weights (sum to group_target_ms). + 2) Snap each clip end to nearest beat boundary; carry over the delta to next clip. + 3) Last clip snaps to ceil to ensure total >= group_target_ms. + """ + clip_count = len(clip_items) + if clip_count == 0: + return [], int(start_beat_index) + + if not beat_durations_ms: + raise ValueError("beat_durations is empty") + + weights_ms: List[Milliseconds] = [] + for clip in clip_items: + _, _, duration_ms = self._full_source_window_and_duration_ms(clip) + weights_ms.append(int(duration_ms)) + + sum_weights = sum(weights_ms) + targets_ms = [(int(group_target_ms) * w) // sum_weights for w in weights_ms] + remainder_ms = int(group_target_ms) - sum(targets_ms) + + # Fix integer truncation drift + fractional_parts = [(i, (int(group_target_ms) * weights_ms[i]) % sum_weights) for i in range(clip_count)] + fractional_parts.sort(key=lambda x: x[1], reverse=True) + for k in range(remainder_ms): + targets_ms[fractional_parts[k][0]] += 1 + + # Enforce min clip duration; borrow from longest clips + deficit_ms: Milliseconds = 0 + for i in range(clip_count): + if targets_ms[i] < int(self._config.min_clip_duration): + deficit_ms += int(self._config.min_clip_duration) - targets_ms[i] + targets_ms[i] = int(self._config.min_clip_duration) + + if deficit_ms > 0: + indices_by_longest = sorted(range(clip_count), key=lambda i: targets_ms[i], reverse=True) + for i in indices_by_longest: + if deficit_ms <= 0: + break + slack_ms = targets_ms[i] - int(self._config.min_clip_duration) + targets_ms[i] -= slack_ms + deficit_ms -= slack_ms + + beat_count = len(beat_durations_ms) + + def snap_to_nearest_beat( + desired_ms: Milliseconds, beat_index: int, phase_ms: Milliseconds + ) -> Tuple[Milliseconds, int]: + elapsed_ms = int(phase_ms) + idx = int(beat_index) + + safety_steps = 0 + while elapsed_ms < int(self._config.min_clip_duration): + elapsed_ms += int(beat_durations_ms[idx]) + idx = (idx + 1) % beat_count + safety_steps += 1 + if safety_steps > SNAP_SAFETY_MAX_STEPS: + raise RuntimeError("snap_to_nearest_beat safety exceeded") + + if elapsed_ms >= int(desired_ms): + return int(elapsed_ms), int(idx) + + previous_elapsed_ms = elapsed_ms + previous_idx = idx + + safety_steps = 0 + while elapsed_ms < int(desired_ms): + previous_elapsed_ms = elapsed_ms + previous_idx = idx + elapsed_ms += int(beat_durations_ms[idx]) + idx = (idx + 1) % beat_count + safety_steps += 1 + if safety_steps > SNAP_SAFETY_MAX_STEPS: + raise RuntimeError("snap_to_nearest_beat safety exceeded") + + if int(desired_ms) - previous_elapsed_ms < elapsed_ms - int(desired_ms): + return int(previous_elapsed_ms), int(previous_idx) + return int(elapsed_ms), int(idx) + + def snap_to_beat_ceil( + desired_ms: Milliseconds, beat_index: int, phase_ms: Milliseconds + ) -> Tuple[Milliseconds, int]: + elapsed_ms = int(phase_ms) + idx = int(beat_index) + + desired_ms = max(int(self._config.min_clip_duration), int(desired_ms)) + safety_steps = 0 + while elapsed_ms < desired_ms: + elapsed_ms += int(beat_durations_ms[idx]) + idx = (idx + 1) % beat_count + safety_steps += 1 + if safety_steps > SNAP_SAFETY_MAX_STEPS: + raise RuntimeError("snap_to_beat_ceil safety exceeded") + return int(elapsed_ms), int(idx) + + durations_ms: List[Milliseconds] = [] + beat_index = int(start_beat_index) % beat_count + phase_ms = max(0, int(start_residual_ms)) + + carry_ms: Milliseconds = 0 + sum_actual_ms: Milliseconds = 0 + + for i in range(clip_count): + is_last_clip = i == clip_count - 1 + + desired_ms = int(targets_ms[i]) + int(carry_ms) + if desired_ms < int(self._config.min_clip_duration): + desired_ms = int(self._config.min_clip_duration) + + if not is_last_clip: + actual_ms, beat_index = snap_to_nearest_beat(desired_ms, beat_index, phase_ms) + else: + remaining_ms = max(0, int(group_target_ms) - int(sum_actual_ms)) + desired_ms = max(desired_ms, int(remaining_ms)) + actual_ms, beat_index = snap_to_beat_ceil(desired_ms, beat_index, phase_ms) + + durations_ms.append(int(actual_ms)) + sum_actual_ms += int(actual_ms) + + carry_ms = desired_ms - int(actual_ms) + phase_ms = 0 # legacy strategy: always reset + + return durations_ms, int(beat_index) + + def _allocate_clip_durations_without_beats( + self, *, clip_items: List[Dict[str, Any]], group_target_ms: Milliseconds + ) -> List[Milliseconds]: + clip_count = len(clip_items) + if clip_count == 0: + return [] + + source_durations_ms: List[Milliseconds] = [] + for clip in clip_items: + _, _, duration_ms = self._full_source_window_and_duration_ms(clip) + source_durations_ms.append(int(duration_ms)) + + total_source_duration_ms = sum(source_durations_ms) + + def total_for_ratio(ratio: float) -> Milliseconds: + total_ms: Milliseconds = 0 + for duration_ms in source_durations_ms: + allocated_ms = int(duration_ms * ratio) + if allocated_ms < int(self._config.min_clip_duration): + allocated_ms = int(self._config.min_clip_duration) + total_ms += int(allocated_ms) + return int(total_ms) + + ratio_high = max(1.0, int(group_target_ms) / total_source_duration_ms) + while total_for_ratio(ratio_high) < int(group_target_ms): + ratio_high *= RATIO_GROWTH_FACTOR + if ratio_high > RATIO_GROWTH_MAX: + break + + ratio_low = 0.0 + for _ in range(BINARY_SEARCH_ITERATIONS): + ratio_mid = (ratio_low + ratio_high) / 2.0 + if total_for_ratio(ratio_mid) <= int(group_target_ms): + ratio_low = ratio_mid + else: + ratio_high = ratio_mid + + ratio = ratio_low + + base_ms: List[Milliseconds] = [] + fractional_parts: List[Tuple[float, int]] = [] + sum_base_ms: Milliseconds = 0 + + for i, duration_ms in enumerate(source_durations_ms): + raw = duration_ms * ratio + floored = int(raw) + + allocated_ms = floored + if allocated_ms < int(self._config.min_clip_duration): + allocated_ms = int(self._config.min_clip_duration) + fraction = -1.0 + else: + fraction = raw - floored + + base_ms.append(int(allocated_ms)) + sum_base_ms += int(allocated_ms) + fractional_parts.append((float(fraction), i)) + + remaining_ms: Milliseconds = int(group_target_ms) - int(sum_base_ms) + if remaining_ms > 0: + fractional_parts.sort(key=lambda x: (x[0], source_durations_ms[x[1]]), reverse=True) + j = 0 + # NOTE: matches legacy behavior (no wrap-around); remaining_ms too large may raise IndexError. + while remaining_ms > 0: + idx = fractional_parts[j][1] + base_ms[idx] += 1 + remaining_ms -= 1 + j += 1 + + return base_ms + + +@NODE_REGISTRY.register() +class PlanTimelineNode(BaseNode): + meta = NodeMeta( + name="plan_timeline", + description=( + "Create a coherent timeline by arranging video clips, subtitles, voice-over, and background music. " + "Required: load_media. Optional: generate_script, detect_highlights, select_BGM, generate_voiceover" + ), + node_id="plan_timeline", + node_kind="plan_timeline", + require_prior_kind=["load_media", "split_shots", "group_clips", "generate_script", "tts", "music_rec"], + default_require_prior_kind=["load_media", "split_shots", "group_clips", "generate_script", "tts", "music_rec"], + next_available_node=["render_video"], + ) + + input_schema = PlanTimelineInput + + def __init__(self, server_cfg: Settings) -> None: + super().__init__(server_cfg) + config: PlanTimelineConfig = self.server_cfg.plan_timeline + self.planner = TimelinePlanner(config, random_seed=DEFAULT_RANDOM_SEED) + + + async def default_process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Any: + return await self.process(node_state, inputs) + + async def process(self, node_state, inputs: Dict[str, Any]) -> Dict[str, Any]: + # Inputs (defensive parsing for open-source robustness) + media = (inputs.get("load_media") or {}).get("media", []) + clips = (inputs.get("split_shots") or {}).get("clips", []) + groups = (inputs.get("group_clips") or {}).get("groups", []) + group_scripts = (inputs.get("generate_script") or {}).get("group_scripts", []) + voiceovers = (inputs.get("tts") or {}).get("voiceover", []) + background_music = (inputs.get("music_rec") or {}).get("bgm") # Optional dict + use_beats = inputs.get("use_beats", False) + + result = self.planner.plan( + media=media, + clips=clips, + groups=groups, + group_scripts=group_scripts, + voiceovers=voiceovers, + background_music=background_music, + use_beats=use_beats, + ) + + node_state.node_summary.info_for_user("时间线组织成功") + return result diff --git a/src/open_storyline/nodes/core_nodes/plan_timeline_pro.py b/src/open_storyline/nodes/core_nodes/plan_timeline_pro.py new file mode 100644 index 0000000000000000000000000000000000000000..f6c506fb7654b752e16b1f47627cac872689e5d5 --- /dev/null +++ b/src/open_storyline/nodes/core_nodes/plan_timeline_pro.py @@ -0,0 +1,592 @@ +from typing import List, Dict, Tuple, Union, Any +import json +import random +from src.open_storyline.config import Settings +from itertools import accumulate, pairwise +from open_storyline.config import PlanTimelineProConfig +from open_storyline.nodes.node_state import NodeState +from open_storyline.nodes.core_nodes.base_node import BaseNode, NodeMeta +from open_storyline.nodes.node_schema import PlanTimelineInput +from open_storyline.utils.register import NODE_REGISTRY + + +class TimeLine: + + def edit_meterial_timeline( + self, + cfg: PlanTimelineProConfig, + node_state: NodeState, + music: Dict, + meterial_durations: List[int], + tts_res: List[Dict] = None, + texts: List[List[str]] = [], + types: List[str] = [], + tts_indices_map: Dict = None, + group_indices_map: Dict = None, + title_clip_duration: int=None, + is_on_beats: bool=False, + beat_type: int = 1, + ): + ''' + Re-edit meterial durations according to tts duration or beats. + ''' + + min_single_text_duration, max_text_duration = cfg.min_single_text_duration, cfg.max_text_duration + tts_durations = [item['duration'] for item in tts_res] if tts_res else [min(min_single_text_duration * len(''.join(text)), max_text_duration) for text in texts] + meterial_durations = [x if x > 0 else cfg.img_default_duration for x in meterial_durations] + + # edit meterials + music_offset = 0 + if is_on_beats is False: + if tts_res: + new_meterial_durations, time_margins = self.edit_meterial_durations_tts(cfg, node_state, meterial_durations, tts_durations, tts_indices_map, group_indices_map) + else: + new_meterial_durations = meterial_durations + time_margins = [0 for _ in range(len(meterial_durations))] + node_state.node_summary.add_error(f"Check config, one of `is_use_beats` and `is_use_tts` must be true.") + else: + if music: + # get beats + beats_timestamp = [0] + music.get('beats', []) + beats_durations = [beats_timestamp[i+1] - beats_timestamp[i] for i in range(len(beats_timestamp)-1)] + [music['duration'] - beats_timestamp[-1]] # calculate extra music duration + music_offset, new_meterial_durations = self.edit_meterial_durations_beats(cfg, node_state, meterial_durations, beats_durations, tts_durations, types, tts_indices_map, title_clip_duration) + time_margins = [0 for _ in range(len(meterial_durations))] + else: + new_meterial_durations = meterial_durations + time_margins = [0 for _ in range(len(meterial_durations))] + node_state.node_summary.add_error(f"Check config, one of `is_use_beats` and `is_use_tts` must be true.") + + # edit speed + speeds = [1.0 if old_duration > new_duration or _type == 'img' else old_duration / new_duration for _type, old_duration, new_duration in zip(types, meterial_durations, new_meterial_durations)] + return music_offset, new_meterial_durations, speeds, time_margins + + def edit_meterial_durations_tts( + self, + cfg: PlanTimelineProConfig, + node_state: NodeState, + meterial_durations: List[int], + tts_durations: List[int], + tts_indices_map: Dict, + group_indices_map: Dict = None, + ) -> List[int]: + ''' + Only add tts without beats. + ''' + new_meterial_durations = [] + tts_paragraph = list(accumulate([v for _, v in tts_indices_map.items()])) + group_paragraph = set(accumulate([v for _, v in group_indices_map.items()])) + is_end_tts_paragraph = [paragraph in group_paragraph for paragraph in tts_paragraph] + group_margin_proposal = random.randint(cfg.min_group_margin, cfg.max_group_margin) + extra_margin = [group_margin_proposal if is_end is True else 0 for is_end in is_end_tts_paragraph] + time_margins = [self.time_margin(cfg) + extra_margin[i] for i in range(len(tts_durations))] + print(f"time_margins: {time_margins}, extra_margin: {extra_margin}") + + paragraph = [0] + list(accumulate(tts_indices_map.values())) + for i, tts_duration in enumerate(tts_durations): + meterial_paragraph_durations = [meterial_durations[idx] for idx in range(paragraph[i], paragraph[i+1])] + # meterial_paragraph_durations_rate = [duration / sum(meterial_paragraph_durations) for duration in meterial_paragraph_durations] + meterial_paragraph_durations_rate = [1 / len(meterial_paragraph_durations) for _ in meterial_paragraph_durations] + + # strategy-1: weighted duration + new_meterial_durations += [max(int(tts_duration + time_margins[i]) * meterial_paragraph_durations_rate[j], cfg.min_clip_duration) for j in range(tts_indices_map[i])] + + return new_meterial_durations, time_margins + + def edit_meterial_durations_beats( + self, + cfg: PlanTimelineProConfig, + node_state: NodeState, + meterial_durations: List[int], + beats_durations: List[int], + tts_durations: List[int], + types: List[str], + tts_indices_map: Dict, + title_clip_duration: int=None, + ) -> List[int]: + + new_meterial_durations = [] + beat_index = next((i for i, acc_duration in enumerate(accumulate(beats_durations)) if acc_duration >= title_clip_duration), len(beats_durations) - 1) + 1 if title_clip_duration else 0 + music_offset = sum(beats_durations[:beat_index]) - title_clip_duration if title_clip_duration else 0 + assert music_offset >= 0 + duration_rates = [round(1 / num, 2) for _, num in tts_indices_map.items() for _ in range(num)] + temp_tts_durations = [val for val, count in zip(tts_durations, list(tts_indices_map.values())) for _ in range(count)] + + init_duration = 0 + wo_got_beats_clips = [] + min_clip_duration = cfg.min_clip_duration + for i in range(len(meterial_durations)): + + minimum_duration = min_clip_duration if not tts_durations or temp_tts_durations[i] is None or temp_tts_durations[i]==0 else max(temp_tts_durations[i] * duration_rates[i], min_clip_duration) + + durations = init_duration + # assert the music is enough long + if beat_index >= len(beats_durations): + beat_index = 0 + node_state.node_summary.add_warning("The music is not enough long. Set the music cycling.") + + while True: + + durations += beats_durations[beat_index] + sub_durations = durations - meterial_durations[i] # diff + + # cut video + if sub_durations > 0: + durations -= beats_durations[beat_index] + if durations < minimum_duration: + while durations < minimum_duration: + durations += beats_durations[beat_index] + beat_index += 1 + # assert the music is enough long + if beat_index >= len(beats_durations): + beat_index = 0 + node_state.node_summary.add_warning("The music is not enough long. Set the music cycling.") + if types[i] == 'video': + wo_got_beats_clips.append(str(i)) + init_duration = durations - max(meterial_durations[i], minimum_duration) + durations = max(meterial_durations[i], minimum_duration) # set new duration to max(meterial_durations[i], minimum_duration) + else: # img type set to the next beats. + init_duration = 0 + else: + init_duration = 0 + break + else: + beat_index += 1 + + # assert the music is enough long + if beat_index >= len(beats_durations): + beat_index = 0 + node_state.node_summary.add_warning("The music is not enough long. Set the music cycling.") + + new_meterial_durations.append(durations) + + node_state.node_summary.info_for_llm(f"[W/O. Beats Rate] {len(wo_got_beats_clips) / len(new_meterial_durations):.2f}") + return music_offset, new_meterial_durations + + + def time_margin(self, cfg: PlanTimelineProConfig): + mode, min_time_margin, max_time_margin = cfg.tts_margin_mode, cfg.min_tts_margin, cfg.max_tts_margin + if mode == "random": + return random.randint(min_time_margin, max_time_margin) + elif mode == "avg": + return (max_time_margin + min_time_margin) // 2 + elif mode == "min": + return min_time_margin + elif mode == "max": + return max_time_margin + + def text_tts_offset(self, cfg: PlanTimelineProConfig): + mode, min_text_tts_offset, max_text_tts_offset = cfg.text_tts_offset_mode, cfg.min_text_tts_offset, cfg.max_text_tts_offset + if mode == "random": + return random.randint(min_text_tts_offset, max_text_tts_offset) + elif mode == "avg": + return (max_text_tts_offset + min_text_tts_offset) // 2 + elif mode == "min": + return min_text_tts_offset + elif mode == "max": + return max_text_tts_offset + + def edit_tts_timeline( + self, + cfg: PlanTimelineProConfig, + node_state: NodeState, + meterial_durations: List[int], + tts_res: List[Dict] = None, + tts_indices_map: Dict = None, + ): + "Add tts start timestamp" + if not tts_res: + return + + # get base start timestamps + paragraph = [0] + list(accumulate(tts_indices_map.values())) + paragraph_durations = [[dura for dura in meterial_durations[paragraph[i]: paragraph[i+1]]] for i in range(len(paragraph[:-1]))] + paragraph_durations_sum = [sum(durations) for durations in paragraph_durations] + start_timestamps = [sum(meterial_durations[:i]) for i in paragraph[:-1]] + assert len(paragraph_durations_sum) == len(start_timestamps) + + # adjust start timestamps + long_short_text_duration, long_text_margin_rate, short_text_margin_rate = cfg.long_short_text_duration, cfg.long_text_margin_rate, cfg.short_text_margin_rate + + long_tts_margin = [min(int(long_text_margin_rate * paragraph_durations[i][0]), abs(paragraph_durations_sum[i] - tts_res[i]['duration'])) for i in range(len(paragraph[:-1]))] + short_tts_margin = [min(int(short_text_margin_rate * paragraph_durations[i][0]), abs(paragraph_durations_sum[i] - tts_res[i]['duration'])) for i in range(len(paragraph[:-1]))] + + start_timestamps = [start_timestamps[i] + long_tts_margin[i] if paragraph_durations_sum[i] > long_short_text_duration else start_timestamps[i] + short_tts_margin[i] for i in range(len(paragraph[:-1]))] + + # update to tts res + tts_res = [{**item, 'start_timestamp': start_timestamp} for item, start_timestamp in zip(tts_res, start_timestamps)] + return tts_res + + def edit_text_timeline( + self, + cfg: PlanTimelineProConfig, + node_state: NodeState, + meterial_durations: List[int], + texts: List[List[str]] = "", + tts_res: List[Dict] = None, + tts_indices_map: Dict = None, + music: Dict=None, + beat_type: int = 2, + clip_uuids: List=[], + ): + ''' + Get text start timestamps and durations according to the tts and meterial durations + ''' + text_tts_offset = [0] * len(texts) + # case-1: with tts + if tts_res: + + # get tts start timestamps + tts_start_timestamps = [item['start_timestamp'] for item in tts_res] + tts_durations = [item['duration'] for item in tts_res] + + # offset duration + text_tts_offset = [self.text_tts_offset(cfg) for _ in tts_res] + + # calculate text start timestamps + text_start_timestamps = [tts_start_timestamp + offset for tts_start_timestamp, offset in zip(tts_start_timestamps, text_tts_offset)] + + # calculate text durations + base_text_durations = [b - a for a, b in pairwise(text_start_timestamps + [sum(meterial_durations)])] + if cfg.text_duration_mode == 'with_tts': + text_durations = [tts_duration for tts_duration in tts_durations] + elif cfg.text_duration_mode == 'with_clip': + + # calculate tts margin + long_short_text_duration, long_text_margin_rate, short_text_margin_rate = cfg.long_short_text_duration, cfg.long_text_margin_rate, cfg.short_text_margin_rate + long_tts_margin = [min(int(long_text_margin_rate * paragraph_durations[i][0]), abs(paragraph_durations_sum[i] - tts_res[i]['duration'])) for i in range(len(paragraph[:-1]))] + short_tts_margin = [min(int(short_text_margin_rate * paragraph_durations[i][0]), abs(paragraph_durations_sum[i] - tts_res[i]['duration'])) for i in range(len(paragraph[:-1]))] + + text_durations = [base_text_duration - offset for base_text_duration, offset in zip(base_text_durations, text_tts_offset)] + text_durations = [text_duration - long_tts_margin[i] if paragraph_durations_sum[i] > long_short_text_duration else text_duration - short_tts_margin[i] for i, text_duration in enumerate(text_durations)] + else: + node_state.node_summary.add_warning(f"[{self.__class__.__name__}] text_duration_mode: {cfg.text_duration_mode} not in [`with_tts`, `with_clip`], return `with_tts` result as default.") + text_durations = [tts_duration for tts_duration in tts_durations] + + # case-2: wo-tts, `text_duration_mode` default is `with_clip` + else: + # get base start timestamps + paragraph = [0] + list(accumulate(tts_indices_map.values())) + paragraph_durations = [[dura for dura in meterial_durations[paragraph[i]: paragraph[i+1]]] for i in range(len(paragraph[:-1]))] + paragraph_durations_sum = [sum(durations) for durations in paragraph_durations] + text_start_timestamps = [sum(meterial_durations[:i]) for i in paragraph[:-1]] + text_durations = [b - a for a, b in pairwise(text_start_timestamps + [sum(meterial_durations)])] # `text_duration_mode` default is `with_clip` + assert len(paragraph_durations) == len(text_start_timestamps) + + # adjust start timestamps + long_short_text_duration, long_text_margin_rate, short_text_margin_rate = cfg.long_short_text_duration, cfg.long_text_margin_rate, cfg.short_text_margin_rate + + long_tts_margin = [int(long_text_margin_rate * paragraph_durations[i][0]) for i in range(len(paragraph[:-1]))] + short_tts_margin = [int(short_text_margin_rate * paragraph_durations[i][0]) for i in range(len(paragraph[:-1]))] + + text_start_timestamps = [text_start_timestamps[i] + long_tts_margin[i] if paragraph_durations_sum[i] > long_short_text_duration else text_start_timestamps[i] + short_tts_margin[i] for i in range(len(paragraph[:-1]))] + # adjust tts_margin by beats + if cfg.is_text_beats: + beats_timestamp = [0] + music.get('beats', []) + beats_timestamp += [music['duration'] + timestamp for timestamp in beats_timestamp] + text_start_timestamps = self.replace_with_closest_if_within_threshold(text_start_timestamps, beats_timestamp) + + # calculate text durations + text_durations = [text_duration - long_tts_margin[i] if paragraph_durations_sum[i] > long_short_text_duration else text_duration - short_tts_margin[i] for i, text_duration in enumerate(text_durations)] + + # split text to sub-text + final_text_durations, final_text_start_timestamps, text_clip_maps = [], [], [] + for text, duration, start_timestamp, offset in zip(texts, text_durations, text_start_timestamps, text_tts_offset): + + # obtain final start-timestamps and durations + sub_text_durations = [int(len(sub_text) / len(''.join(text)) * duration) for sub_text in text] + sub_start_timestamps = [start_timestamp + sum(sub_text_durations[:i]) for i in range(len(sub_text_durations))] + final_text_durations.append(sub_text_durations) + final_text_start_timestamps.append(sub_start_timestamps) + + return final_text_start_timestamps, final_text_durations, text_clip_maps + + @staticmethod + def replace_with_closest_if_within_threshold(source_list, reference_list, threshold: int=500): + result = [] + for num in source_list: + + closest = min(reference_list, key=lambda x: abs(x - num)) + + if abs(closest - num) < threshold: + result.append(closest) + else: + result.append(num) + return result + + +@NODE_REGISTRY.register() +class PlanTimelineProNode(BaseNode): + + meta = NodeMeta( + name="plan_timeline_pro", + description=( + "Create a coherent timeline by arranging video clips, subtitles, voice-over, and background music. " + ), + node_id="plan_timeline_pro", + node_kind="plan_timeline", + require_prior_kind=["split_shots", "group_clips", "generate_script", "tts", "music_rec"], + default_require_prior_kind=["split_shots", "group_clips", "generate_script", "tts", "music_rec"], + next_available_node=["render_video"], + ) + + input_schema = PlanTimelineInput + + + def __init__(self, server_cfg: Settings) -> None: + super().__init__(server_cfg) + self.default_timeline_cfg: PlanTimelineProConfig = self.server_cfg.plan_timeline_pro + self.timeline_client = TimeLine() + + async def default_process( + self, + node_state: NodeState, + inputs: Dict[str, Any], + ) -> Any: + return await self.process(node_state, inputs) + + async def process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Any: + + music = inputs.pop("music", None) + tts_res = inputs.pop("tts_res", None) + + # Processing clip durations + music_offset, new_meterial_durations, speeds, time_margins = self.timeline_client.edit_meterial_timeline( + self.default_timeline_cfg, + node_state, + music, + inputs.get('clip_durations'), + tts_res, + texts=inputs.get('texts', []), + types=inputs.get('types', []), + tts_indices_map=inputs.get('text_indices_map', {}), + group_indices_map=inputs.get('text_indices_map', {}), + title_clip_duration=inputs.get('title_clip_duration', 0), + is_on_beats=inputs.get('is_on_beats', False), + ) + + # Processing tts durations + tts_res = self.timeline_client.edit_tts_timeline( + self.default_timeline_cfg, + node_state, + new_meterial_durations, + tts_res, + tts_indices_map=inputs.get('text_indices_map', {}), + ) + tts_start_timestamps = [item.get("start_timestamp") for item in tts_res] if tts_res else [] + + # Processing text durations + text_start_timestamps, text_durations, text_clip_maps = self.timeline_client.edit_text_timeline( + self.default_timeline_cfg, + node_state, + new_meterial_durations, + texts=inputs.get('texts', []), + tts_res=tts_res, + tts_indices_map=inputs.get('text_indices_map', {}), + music=music, + clip_uuids=[], + ) + + inputs.update({ + "music": music, + "tts_res": tts_res, + }) + + return { + 'timeline_source_data': inputs, + 'music_offset': music_offset, + 'new_meterial_durations': new_meterial_durations, + 'speeds': speeds, + 'time_margins': time_margins, + 'text_start_timestamps': text_start_timestamps, + 'text_durations': text_durations, + 'text_clip_maps': text_clip_maps, + 'tts_start_timestamps': tts_start_timestamps, + } + + def _combine_tool_outputs(self, node_state, outputs): + """ + Change output format. + """ + tracks, video, subtitles, voiceover, bgm = [], [], [], [], [] + timeline_source_data = outputs.get('timeline_source_data', {}) + + # Video track + clip_ids = timeline_source_data.get('clip_ids', []) + clip_group_ids = timeline_source_data.get('clip_group_ids', []) + kinds = timeline_source_data.get('types', []) + fps = timeline_source_data.get('fps', []) + sizes = timeline_source_data.get('sizes', []) + source_paths = timeline_source_data.get('clips', []) + clip_durations = timeline_source_data.get('clip_durations', []) + clip_durations = [x if x > 0 else self.default_timeline_cfg.img_default_duration for x in clip_durations] + new_meterial_durations = outputs.get('new_meterial_durations', {}) + playback_rates = outputs.get('speeds', []) + + timeline_start = 0 + for clip_id, clip_group_id, kind, _fps, source_path, clip_duration, new_meterial_duration, playback_rate, size in \ + zip(clip_ids, clip_group_ids, kinds, fps, source_paths, clip_durations, new_meterial_durations, playback_rates, sizes): + video.append({ + "clip_id": clip_id, + "group_id": clip_group_id, + "kind": kind, + "fps": _fps, + "size": size, + "source_path": source_path, + "source_window": { + "start": 0, + "end": min(clip_duration, new_meterial_duration), + "duration": min(clip_duration, new_meterial_duration) + }, + "timeline_window": { + "start": timeline_start, + "end": timeline_start + new_meterial_duration, + "duration": new_meterial_duration + }, + "playback_rate": playback_rate + }) + timeline_start += new_meterial_duration + + # Subtitles track + if timeline_source_data.get('texts'): + text_group_ids = timeline_source_data.get('text_group_ids', []) + text_unit_ids = timeline_source_data.get('text_unit_ids', []) + text_index_in_group = timeline_source_data.get('text_index_in_group', []) + texts = [x for item in timeline_source_data.get('texts', []) for x in item] + text_start_timestamps = [st for item in outputs.get('text_start_timestamps', []) for st in item] + text_durations = [t for item in outputs.get('text_durations', []) for t in item] + + for text_group_id, text_unit_id, index_in_group, text, text_start_timestamp, text_duration in \ + zip(text_group_ids, text_unit_ids, text_index_in_group, texts, text_start_timestamps, text_durations): + subtitles.append({ + "group_id": text_group_id, + "unit_id": text_unit_id, + "index_in_group": index_in_group, + "text": text, + "timeline_window": { + "start": text_start_timestamp, + "end": text_start_timestamp + text_duration + } + }) + + # Voiceover track + if timeline_source_data.get('tts_res'): + tts_group_ids = timeline_source_data.get('tts_group_ids', []) + voiceover_ids = timeline_source_data.get('voiceover_ids', []) + tts_durations = timeline_source_data.get('tts_durations', []) + tts_paths = timeline_source_data.get('tts_paths', []) + tts_start_timestamps = outputs.get('tts_start_timestamps', []) + + for tts_group_id, voiceover_id, tts_duration, tts_start_timestamp, tts_path in \ + zip(tts_group_ids, voiceover_ids, tts_durations, tts_start_timestamps, tts_paths): + voiceover.append({ + "group_id": tts_group_id, + "voiceover_id": voiceover_id, + "source_window": { + "start": 0, + "end": tts_duration, + "duration": tts_duration + }, + "timeline_window": { + "start": tts_start_timestamp, + "end": tts_start_timestamp + tts_duration, + "duration": tts_duration + }, + "path": tts_path + }) + + # Bgm track + if timeline_source_data.get('music'): + music_info = timeline_source_data.get('music', {}) + music_duration = music_info.get("duration", 0) + video_duration = int(sum(new_meterial_durations)) + loop_num = video_duration // music_duration + for i in range(loop_num + 1): + bgm.append({ + "bgm_id": music_info.get("bgm_id", ""), + "source_window": { + "start": 0, + "end": music_duration if i != loop_num else video_duration - loop_num * music_duration + }, + "path": music_info.get("path", 0) + }) + + # Merge all tracks + tracks = { + "video": video, + "subtitles": subtitles, + "voiceover": voiceover, + "bgm": bgm, + } + return {"tracks": tracks} + + def _parse_input(self, node_state: NodeState, inputs, **kwargs): + + split_shots = inputs.get("split_shots", {}) + group_clips = inputs.get("group_clips", {}) + generate_script = inputs.get("generate_script", {}) + music = inputs.get("music_rec", None).get("bgm", {}) + tts_res = inputs.get("tts", {}).get("voiceover", []) + use_beats = inputs.get("use_beats", False) + texts, types = [], [] + clips, clip_ids, clip_idxes = [], [], [] + clip_group_ids = [] + clip_durations = [] + text_group_ids, text_unit_ids, text_index_in_group = [], [], [] + text_indices_map = {} + tts_group_ids, voiceover_ids, tts_durations, tts_paths = [], [], [], [] + + # Get clips and duration + groups = group_clips.get("groups", []) + for i, group in enumerate(groups): + group_clip_ids = group.get('clip_ids', []) + clip_ids += group_clip_ids + clip_idxes += [int(item.split('_')[-1]) for item in group_clip_ids] + clip_group_ids += [group.get('group_id', []) for _ in group_clip_ids] + text_indices_map[i] = len(group.get('clip_ids', [])) + + clip_durations = [split_shots.get('clips', [])[idx-1].get('source_ref', {}).get('duration', 0) for idx in clip_idxes] + start_times = [split_shots.get('clips', [])[idx-1].get('source_ref', {}).get('start', 0) for idx in clip_idxes] + clips = [split_shots.get('clips', [])[idx-1].get('path', '') for idx in clip_idxes] + types = [split_shots.get('clips', [])[idx-1].get('kind', '') for idx in clip_idxes] + fps = [split_shots.get('clips', [])[idx-1].get('fps', None) for idx in clip_idxes] + + # Get text info + for item in generate_script.get('group_scripts', []): + texts.append([sub_item.get('text', '') for sub_item in item.get('subtitle_units', [])]) + text_unit_ids += [sub_item.get('unit_id', '') for sub_item in item.get('subtitle_units', [])] + text_group_ids += [item.get('group_id', '') for _ in item.get('subtitle_units', [])] + text_index_in_group += [i for i in range(len(item.get('subtitle_units', [])))] + + # Get tts info + for item in tts_res: + tts_group_ids.append(item.get('group_id', '')) + voiceover_ids.append(item.get('voiceover_id', '')) + tts_durations.append(item.get('duration', '')) + tts_paths.append(item.get('path', '')) + + # For save + sizes = [[split_shots.get('clips', [])[idx-1].get('source_ref', {}).get('width', 576), split_shots.get('clips', [])[idx-1].get('source_ref', {}).get('height', 1024)] for idx in clip_idxes] + + return { + 'types': types, + 'texts': texts, + 'text_unit_ids': text_unit_ids, + 'text_group_ids': text_group_ids, + 'text_index_in_group': text_index_in_group, + 'clips': clips, + 'clip_ids': clip_ids, + 'clip_group_ids': clip_group_ids, + 'fps': fps, + 'sizes': sizes, + 'clip_durations': clip_durations, + 'start_times': start_times, + 'text_indices_map': text_indices_map, + 'music': music, + 'tts_res': tts_res, + 'tts_group_ids': tts_group_ids, + 'voiceover_ids': voiceover_ids, + 'tts_durations': tts_durations, + 'tts_paths': tts_paths, + 'is_on_beats': use_beats, + 'title_clip_duration': 0, + } \ No newline at end of file diff --git a/src/open_storyline/nodes/core_nodes/recommend_effects.py b/src/open_storyline/nodes/core_nodes/recommend_effects.py new file mode 100644 index 0000000000000000000000000000000000000000..0c5a8ba6a7c6ea7d262f3fd1fcc2a4e2166f87f7 --- /dev/null +++ b/src/open_storyline/nodes/core_nodes/recommend_effects.py @@ -0,0 +1,116 @@ +from typing import Any, Dict, List +from pathlib import Path + +import numpy as np + +from open_storyline.utils.element_filter import ElementFilter +from open_storyline.utils.prompts import get_prompt +from open_storyline.nodes.core_nodes.base_node import BaseNode, NodeMeta +from open_storyline.nodes.node_state import NodeState +from open_storyline.nodes.node_schema import RecommendTransitionInput, RecommendTextInput +from open_storyline.utils.parse_json import parse_json_dict +from src.open_storyline.config import Settings +from open_storyline.utils.register import NODE_REGISTRY + +@NODE_REGISTRY.register() +class RecommendTransitionNode(BaseNode): + meta = NodeMeta( + name="elementrec_transition", + description="Recommend transition effects according to user needs and segment count, ensuring transition list length equals group count", + node_id="elementrec_transition", + node_kind="transition_rec", + require_prior_kind=['group_clips'], + default_require_prior_kind=[], + next_available_node=["plan_timeline"], + ) + + input_schema = RecommendTransitionInput + + async def default_process( + self, + node_state: NodeState, + inputs: Dict[str, Any], + ) -> Any: + node_state.node_summary.info_for_user(f"[{self.meta.node_id}] Transition effect not used") + return [] + + + async def process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Any: + duration = inputs.get('duration', 1000) # default 1000ms + node_state.node_summary.info_for_user( + f"[{self.meta.node_id}] Adding fade transitions: {duration}ms fade-in at start, {duration}ms fade-out at end" + ) + return [ + { + 'type': 'fade_in', + 'position': 'opening', + 'duration': duration + }, + { + 'type': 'fade_out', + 'position': 'ending', + 'duration': duration + } + ] + + +@NODE_REGISTRY.register() +class RecommendTextNode(BaseNode): + meta = NodeMeta( + name="elementrec_text", + description="Recommend text effects according to user needs", + node_id="elementrec_text", + node_kind="text_rec", + require_prior_kind=["generate_script"], + default_require_prior_kind=[], + next_available_node=["plan_timeline"], + ) + + input_schema = RecommendTextInput + + + def __init__(self, server_cfg: Settings) -> None: + super().__init__(server_cfg) + self.text_filter = ElementFilter(json_path=server_cfg.recommend_text.font_info_path) + + async def default_process( + self, + node_state: NodeState, + inputs: Dict[str, Any], + ) -> Any: + self.text_filter.filter() + node_state.node_summary.info_for_user(f"[{self.meta.node_id}] Using default font") + return [{"font_name": "Noto Sans SC", "font_color": inputs.get("font_color", (255,255,255,255))}] + + async def process( + self, + node_state: NodeState, + inputs: Dict[str, Any], + ) -> Any: + user_request = inputs.get("user_request", "") + filter_include = inputs.get("filter_include", {}) + group_scripts = inputs.get("generate_script", {}).get("group_scripts", {}) + + candidates = self.text_filter.filter(filter_include=filter_include).copy() + + font_paths = [cand.pop("font_path", None) for cand in candidates] + llm = node_state.llm + system_prompt = get_prompt("elementrec_text.system", lang=node_state.lang) + user_prompt = get_prompt("elementrec_text.user", lang=node_state.lang, scripts=group_scripts, candidates=candidates, user_request=user_request) + raw = await llm.complete( + system_prompt=system_prompt, + user_prompt=user_prompt, + temperature=0.1, + top_p=0.9, + max_tokens=2048, + model_preferences=None, + ) + try: + selected_json = parse_json_dict(raw) + except: + selected_json = (raw or "").strip() if raw else "Error: Unable to parse the model output" + node_state.node_summary.add_error(selected_json) + return None + selected_json.update({"font_color": inputs.get("font_color", (255,255,255,255))}) + node_state.node_summary.info_for_user(f"[{self.meta.node_id}] Use font `{selected_json['font_name']}`") + return [selected_json] \ No newline at end of file diff --git a/src/open_storyline/nodes/core_nodes/render_video.py b/src/open_storyline/nodes/core_nodes/render_video.py new file mode 100644 index 0000000000000000000000000000000000000000..6361e0e67a1ef7c50f4ecb7a84c67d61948151d6 --- /dev/null +++ b/src/open_storyline/nodes/core_nodes/render_video.py @@ -0,0 +1,1008 @@ +import os +import tempfile +import time +import uuid +import asyncio +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple +import json + +import numpy as np +from open_storyline.utils.logging import MCPMoviePyLogger +from PIL import Image, ImageDraw, ImageFont, ImageOps +from open_storyline.utils.register import NODE_REGISTRY + +# MoviePy import compatibility (v2 preferred) +try: + from moviepy import ( + VideoFileClip, + AudioFileClip, + ImageClip, + VideoClip, + ColorClip, + CompositeVideoClip, + CompositeAudioClip, + concatenate_videoclips, + concatenate_audioclips, + vfx, + ) +except Exception: # pragma: no cover + from moviepy.editor import ( # type: ignore + VideoFileClip, + AudioFileClip, + ImageClip, + VideoClip, + ColorClip, + CompositeVideoClip, + CompositeAudioClip, + concatenate_videoclips, + concatenate_audioclips, + vfx, + ) + +from src.open_storyline.config import Settings +from open_storyline.nodes.core_nodes.base_node import BaseNode, NodeMeta +from open_storyline.nodes.node_state import NodeState +from open_storyline.nodes.node_schema import RenderVideoInput +from open_storyline.utils.util import get_video_rotation + +# ============================================================================= +# Constants +# ============================================================================= + +MILLISECONDS_PER_SECOND: float = 1000.0 + +MAX_MEDIA_DIMENSION_PX: int = 1080 # requirement: any media <=1080 +DEFAULT_OUTPUT_MAX_DIMENSION_PX: int = 1080 + +DEFAULT_OUTPUT_ASPECT_RATIO: float = 16.0 / 9.0 +BACKGROUND_COLOR_RGB: Tuple[int, int, int] = (0, 0, 0) +CENTER_POSITION = ("center", "center") + +DEFAULT_OUTPUT_FPS: int = 25 + +SUBCLIP_END_SAFETY_MARGIN_S = 1e-3 # + +# Encoding +VIDEO_CODEC: str = "libx264" +AUDIO_CODEC: str = "aac" +FFMPEG_PARAMS: list[str] = ["-preset", "veryfast", "-crf", "23", "-threads", "0"] +TEMP_DIRECTORY_PREFIX: str = "render_video_" +TEMP_AUDIO_FILENAME: str = "temp-audio.m4a" + +# Subtitle baseline +SUBTITLE_BASE_HEIGHT_PX: float = 1080.0 +SUBTITLE_FONT_SIZE_AT_BASE: int = 40 +SUBTITLE_FONT_SIZE_MIN: int = 28 +SUBTITLE_FONT_SIZE_MAX: int = 120 +SUBTITLE_FONT_COLOR: Tuple[int, int, int, int] = (255, 255, 255, 255) +SUBTITLE_MARGIN_BOTTOM_AT_BASE: int = 270 +SUBTITLE_MARGIN_BOTTOM_MIN: int = 40 +SUBTITLE_MARGIN_BOTTOM_MAX: int = 1040 +SUBTITLE_STROKE_WIDTH_AT_BASE: int = 2 +SUBTITLE_STROKE_WIDTH_MIN: int = 0 +SUBTITLE_STROKE_WIDTH_MAX: int = 8 +SUBTITLE_STROKE_COLOR: Tuple[int, int, int, int] = (0, 0, 0, 255) +SUBTITLE_MAX_WIDTH_RATIO: float = 0.90 +SUBTITLE_PADDING_X: int = 20 +SUBTITLE_PADDING_Y: int = 10 + +SOURCE_VIDEO_VOLUME_SCALE = 1.0 +TTS_VOLUME_SCALE: float = 2.0 +BGM_VOLUME_SCALE: float = 0.25 +AUDIO_DURATION_TOLERANCE_SECONDS: float = 0.05 +DEFAULT_CRF = 23 + + +# ============================================================================= +# Small utilities +# ============================================================================= + +def close_quietly(obj: Any) -> None: + try: + if obj is not None: + obj.close() + except Exception: + pass + + +def milliseconds_to_seconds(value: Any) -> float: + try: + return float(value) / MILLISECONDS_PER_SECOND + except Exception: + return 0.0 + + +def clamp_int(value: float, minimum: int, maximum: int) -> int: + return int(max(minimum, min(maximum, round(value)))) + + +def make_even(value: int) -> int: + v = int(value) + if v < 2: + v = 2 + if v % 2 == 1: + v -= 1 + return max(2, v) + + +def parse_aspect_ratio(value: Any) -> Optional[float]: + """ + Accept: + - "16:9" + - float/int like 1.777 + - (w, h) + """ + if value is None: + return None + + if isinstance(value, (int, float)): + r = float(value) + return r if r > 0 else None + + if isinstance(value, str): + text = value.strip() + if ":" in text: + parts = text.split(":") + if len(parts) == 2: + try: + w = float(parts[0].strip()) + h = float(parts[1].strip()) + if w > 0 and h > 0: + return w / h + except Exception: + return None + else: + try: + r = float(text) + return r if r > 0 else None + except Exception: + return None + + if isinstance(value, (tuple, list)) and len(value) == 2: + try: + w = float(value[0]) + h = float(value[1]) + if w > 0 and h > 0: + return w / h + except Exception: + return None + + return None + + +def resolve_output_canvas_size(inputs: Dict[str, Any]) -> Tuple[int, int]: + """ + Requirement: + 1) output aspect ratio decided by inputs + 2) keep output <=1080 (consistent with media<=1080 + performance) + """ + + # Adaptively select the canvas size based on the proportion of the size of the material. + def find_dominant_aspect_ratio(ratios): + if not ratios: + return None + standard_ratios = [9/16, 3/4, 1.0, 4/3, 16/9] + counts = [0] * len(standard_ratios) + + for r in ratios: + idx = min(range(len(standard_ratios)), key=lambda i: abs(standard_ratios[i] - r)) + counts[idx] += 1 + + # apply max count idx + max_count = max(counts) + max_count_idx = counts.index(max_count) + + return standard_ratios[max_count_idx] + + # Specify the aspect ratio and the longest side + video_items = inputs.get('plan_timeline', {}).get("tracks", {}).get("video", []) or [] + ratio = ( + parse_aspect_ratio(inputs.get("aspect_ratio")) + or find_dominant_aspect_ratio([item.get("size")[0] / item.get("size")[1] for item in video_items if item and item.get("size")]) + or DEFAULT_OUTPUT_ASPECT_RATIO + ) + + max_dim = inputs.get("output_max_dimension_px", DEFAULT_OUTPUT_MAX_DIMENSION_PX) + try: + max_dim = int(max_dim) + except Exception: + max_dim = DEFAULT_OUTPUT_MAX_DIMENSION_PX + max_dim = max(2, min(MAX_MEDIA_DIMENSION_PX, max_dim)) + + if ratio >= 1.0: + width = max_dim + height = max(2, int(round(width / ratio))) + else: + height = max_dim + width = max(2, int(round(height * ratio))) + + return (make_even(width), make_even(height)) + + +def build_media_id_to_path_map(load_media: Dict[str, Any]) -> Dict[str, str]: + mapping: Dict[str, str] = {} + for item in (load_media.get("videos") or []) + (load_media.get("images") or []): + media_id = item.get("media_id") + path = item.get("path") + if media_id and path: + mapping[media_id] = path + return mapping + + +def is_image_file(path: str) -> bool: + try: + return Path(path).suffix.lower() in {".png", ".jpg", ".jpeg", ".bmp", ".webp", ".tif", ".tiff"} + except Exception: + return False + + +def make_mask_clip(mask: np.ndarray) -> ImageClip: + # moviepy versions may accept is_mask / ismask + try: + return ImageClip(mask, is_mask=True) + except TypeError: # pragma: no cover + return ImageClip(mask, ismask=True) + + +# ============================================================================= +# Media cache: scale <=1080, drop alpha, and speed up images +# ============================================================================= + +class MediaCache: + def __init__( + self, + *, + include_video_audio: bool, + canvas_size: Tuple[int, int], + clip_compose_mode:str = "padding", + bg_color: Tuple | List | None = None, + ) -> None: + self._include_video_audio = include_video_audio + self._canvas_size = canvas_size + self._clip_compose_mode = clip_compose_mode + self._bg_color = tuple(bg_color) if bg_color else (0, 0, 0) # RGB + + self._video_sources: Dict[str, VideoFileClip] = {} + self._audio_sources: Dict[str, AudioFileClip] = {} + self._audio_to_close: List[AudioFileClip] = [] + + # Key optimization: cache full-canvas RGB frames for images + self._image_padded_frame_cache: Dict[str, np.ndarray] = {} + self._video_size_cache: Dict[str, Tuple[int, int]] = {} + + def close(self) -> None: + for v in self._video_sources.values(): + close_quietly(v) + for a in self._audio_to_close: + close_quietly(a) + + def get_audio(self, path: str) -> AudioFileClip: + cached = self._audio_sources.get(path) + if cached is not None: + return cached + clip = AudioFileClip(path) + self._audio_sources[path] = clip + self._audio_to_close.append(clip) + return clip + + def get_video(self, path: str) -> VideoFileClip: + cached = self._video_sources.get(path) + if cached is not None: + return cached + + src_w, src_h = self._probe_video_size(path) + canvas_w, canvas_h = self._canvas_size + + # fit into canvas and <=1080 + max_w = min(canvas_w, MAX_MEDIA_DIMENSION_PX) + max_h = min(canvas_h, MAX_MEDIA_DIMENSION_PX) + + if src_w > 0 and src_h > 0: + scale = min(max_w / float(src_w), max_h / float(src_h)) + + target_w = make_even(int(src_w * scale)) + target_h = make_even(int(src_h * scale)) + src_ratio = src_w / float(src_h) + canvas_ratio = canvas_w / float(canvas_h) + + clip = VideoFileClip(path, audio=self._include_video_audio, target_resolution=(src_w, src_h)) + + # maybe crop + if self._clip_compose_mode == 'crop': + x0, y0, x1, y1 = self.center_crop_calc((canvas_w, canvas_h), clip.size) + clip = clip.cropped(x0, y0, x1, y1) + + # resize to canvas + if src_ratio >= canvas_ratio: + clip = clip.resized(width=target_w) + else: + clip = clip.resized(height=target_h) + + self._video_sources[path] = clip + return clip + + def get_image(self, path: str) -> np.ndarray: + cached = self._image_padded_frame_cache.get(path) + if cached is not None: + return cached + + canvas_w, canvas_h = self._canvas_size + + with Image.open(path) as image: + try: + image = ImageOps.exif_transpose(image) + except Exception: + pass + + # drop alpha channel: RGBA -> alpha composite on black -> RGB + image = image.convert("RGBA") + black_bg = Image.new("RGBA", image.size, (0, 0, 0, 255)) + try: + image = Image.alpha_composite(black_bg, image).convert("RGB") + except Exception: + image = image.convert("RGB") + + # maybe crop + if self._clip_compose_mode == 'crop': + x0, y0, x1, y1 = self.center_crop_calc(self._canvas_size, image.size) + image = image.crop(box=(x0, y0, x1, y1)) + + # resize to fit (<=1080 and <=canvas) + try: + resample = Image.Resampling.LANCZOS + except Exception: # pragma: no cover + resample = Image.LANCZOS # type: ignore + + scale = min(canvas_w / float(image.width), canvas_h / float(image.height)) + image = image.resize((make_even(scale * image.width), make_even(scale * image.height)), resample=resample) + resized = np.array(image, dtype=np.uint8) + + # build full-canvas frame (black + centered image) + canvas = np.full((canvas_h, canvas_w, 3), fill_value=self._bg_color, dtype=np.uint8) + h, w = resized.shape[0], resized.shape[1] + x0 = max(0, (canvas_w - w) // 2) + y0 = max(0, (canvas_h - h) // 2) + x1 = min(canvas_w, x0 + w) + y1 = min(canvas_h, y0 + h) + canvas[y0:y1, x0:x1] = resized[0 : (y1 - y0), 0 : (x1 - x0)] + + self._image_padded_frame_cache[path] = canvas + return canvas + + def _probe_video_size(self, path: str) -> Tuple[int, int]: + cached = self._video_size_cache.get(path) + if cached is not None: + return cached + + w = h = 0 + # simplest: open once (metadata fetch), then close + try: + tmp = VideoFileClip(path, audio=False) + w = int(getattr(tmp, "w", 0) or 0) + h = int(getattr(tmp, "h", 0) or 0) + close_quietly(tmp) + except Exception: + w, h = 0, 0 + + self._video_size_cache[path] = (w, h) + return w, h + + @staticmethod + def center_crop_calc(canvas_size, media_size): + # unpack sizes + canvas_width, canvas_height = canvas_size + media_width, media_height = media_size + + canvas_ratio = canvas_width / canvas_height + media_ratio = media_width / media_height + + if media_ratio > canvas_ratio: + # crop left and right + crop_width = int(media_height * canvas_ratio) + x1 = (media_width - crop_width) // 2 + return x1, 0, x1 + crop_width, media_height + + elif media_ratio < canvas_ratio: + # crop top and bottom + crop_height = int(media_width / canvas_ratio) + y1 = (media_height - crop_height) // 2 + return 0, y1, media_width, y1 + crop_height + + else: + # same ratio, no crop + return 0, 0, media_width, media_height + + +# ============================================================================= +# Subtitle renderer (RGB + mask; output frames are RGB, no alpha channel) +# ============================================================================= + +class PillowSubtitleRenderer: + def __init__(self, font_path: str) -> None: + self._font_path = font_path + + def render( + self, + subtitle_items: List[Dict[str, Any]], + *, + video_size: Tuple[int, int], + font_color: Tuple[int, int, int, int], + **kwargs, + ) -> List[ImageClip]: + if not self._font_path: + return [] + canvas_w, canvas_h = video_size + scale = (canvas_h / SUBTITLE_BASE_HEIGHT_PX) if canvas_h > 0 else 1.0 + + font_size: int = kwargs.get('font_size') or SUBTITLE_FONT_SIZE_AT_BASE + margin_bottom: int = kwargs.get('margin_bottom') or SUBTITLE_MARGIN_BOTTOM_AT_BASE + stroke_width: int = kwargs.get('stroke_width') or SUBTITLE_STROKE_WIDTH_AT_BASE + stroke_color: Tuple = kwargs.get('stroke_color') or SUBTITLE_STROKE_COLOR + font_size = clamp_int(font_size * scale, SUBTITLE_FONT_SIZE_MIN, SUBTITLE_FONT_SIZE_MAX) + margin_bottom = clamp_int( + margin_bottom * scale, SUBTITLE_MARGIN_BOTTOM_MIN, SUBTITLE_MARGIN_BOTTOM_MAX + ) + stroke_width = clamp_int( + stroke_width * scale, SUBTITLE_STROKE_WIDTH_MIN, SUBTITLE_STROKE_WIDTH_MAX + ) + + clips: List[ImageClip] = [] + for item in subtitle_items: + text = str(item.get("text", "")).strip() + tw = item.get("timeline_window", {}) or {} + start_s = milliseconds_to_seconds(tw.get("start", 0.0)) + end_s = milliseconds_to_seconds(tw.get("end", 0.0)) + dur = end_s - start_s + if not text or dur <= 0: + continue + + clip = self._make_clip( + text=text, + start_s=start_s, + end_s=end_s, + video_size=video_size, + font_size=font_size, + font_color=font_color, + margin_bottom=margin_bottom, + stroke_width=stroke_width, + stroke_color=stroke_color, + ) + if clip is not None: + clips.append(clip) + + return clips + + def _make_clip( + self, + *, + text: str, + start_s: float, + end_s: float, + video_size: Tuple[int, int], + font_size: int, + font_color: Tuple[int, int, int, int], + margin_bottom: int, + stroke_width: int, + stroke_color: Tuple[int, int, int, int], + ) -> Optional[ImageClip]: + canvas_w, canvas_h = video_size + dur = end_s - start_s + if dur <= 0: + return None + + font = self._load_font(font_size) + + max_text_w = max(1, int(canvas_w * SUBTITLE_MAX_WIDTH_RATIO)) + wrapped = self._wrap_text_by_width(text, font, max_text_w) + + measure = Image.new("RGBA", (10, 10), (0, 0, 0, 0)) + draw = ImageDraw.Draw(measure) + bbox = draw.multiline_textbbox( + (0, 0), wrapped, font=font, align="center", stroke_width=stroke_width + ) + text_w = bbox[2] - bbox[0] + text_h = bbox[3] - bbox[1] + + img_w = int(text_w + SUBTITLE_PADDING_X * 2) + img_h = int(text_h + SUBTITLE_PADDING_Y * 2) + + rgba = Image.new("RGBA", (img_w, img_h), (0, 0, 0, 0)) + draw = ImageDraw.Draw(rgba) + draw.multiline_text( + (SUBTITLE_PADDING_X - bbox[0], SUBTITLE_PADDING_Y - bbox[1]), + wrapped, + font=font, + fill=tuple(font_color), + align="center", + stroke_width=stroke_width, + stroke_fill=tuple(stroke_color), + ) + + rgba_arr = np.array(rgba, dtype=np.uint8) + rgb_arr = rgba_arr[:, :, :3] + alpha_mask = (rgba_arr[:, :, 3].astype(np.float32) / 255.0) + + subtitle_clip = ImageClip(rgb_arr).with_mask(make_mask_clip(alpha_mask)) + y = max(0, int(canvas_h - margin_bottom - img_h)) + subtitle_clip = subtitle_clip.with_start(start_s).with_duration(dur).with_position(("center", y)) + return subtitle_clip + + def _load_font(self, font_size: int) -> ImageFont.FreeTypeFont: + try: + if self._font_path and os.path.exists(self._font_path): + return ImageFont.truetype(font=self._font_path, size=font_size) + except Exception: + pass + return ImageFont.load_default() + + @staticmethod + def _wrap_text_by_width(text: str, font: ImageFont.FreeTypeFont, max_width_px: int) -> str: + text = text.strip() + if not text: + return "" + + dummy = Image.new("RGB", (10, 10)) + draw = ImageDraw.Draw(dummy) + + lines: List[str] = [] + for paragraph in text.splitlines(): + paragraph = paragraph.strip() + if not paragraph: + lines.append("") + continue + + current = "" + for ch in paragraph: + candidate = current + ch + if draw.textlength(candidate, font=font) <= max_width_px or not current: + current = candidate + else: + lines.append(current) + current = ch + if current: + lines.append(current) + + return "\n".join(lines) + + +# ============================================================================= +# Audio composer (same behavior as before) +# ============================================================================= + +class AudioTrackComposer: + def __init__(self, *, cache: MediaCache) -> None: + self._cache = cache + + def compose( + self, + *, + voiceover_items: List[Dict[str, Any]], + bgm_items: List[Dict[str, Any]], + final_duration_s: float, + **kwargs, + ): + layers: List[Any] = [] + + # voiceover + for item in voiceover_items: + path = item.get("path") + if not path: + continue + src = self._cache.get_audio(path) + + sw = item.get("source_window", {}) or {} + tw = item.get("timeline_window", {}) or {} + + src_start = milliseconds_to_seconds(sw.get("start", 0.0)) + src_end = milliseconds_to_seconds(sw.get("end", 0.0)) + + src_end = max(src_start, src_end - SUBCLIP_END_SAFETY_MARGIN_S) + src_end = self._clamp_end_to_duration(src, src_end) + if src_end <= src_start: + continue + + tl_start = milliseconds_to_seconds(tw.get("start", 0.0)) + tl_end = milliseconds_to_seconds(tw.get("end", 0.0)) + expected = max(0.0, tl_end - tl_start) + + max_available = max(0.0, src_end - src_start) + expected = min(expected, max_available) + + if expected <= 0: + continue + + sub_end = src_start + expected + + sub_end = self._clamp_end_to_duration(src, sub_end) + if sub_end <= src_start: + continue + + clip = src.subclipped(src_start, sub_end).with_start(tl_start) + clip = clip.with_volume_scaled(kwargs.get('tts_volume_scale') or TTS_VOLUME_SCALE) + layers.append(clip) + + # bgm (concat then loop/trim) + if bgm_items: + segments: List[Any] = [] + for item in bgm_items: + path = item.get("path") + if not path: + continue + src = self._cache.get_audio(path) + sw = item.get("source_window", {}) or {} + src_start = milliseconds_to_seconds(sw.get("start", 0.0)) + src_end = milliseconds_to_seconds(sw.get("end", 0.0)) + if src_end <= src_start: + continue + segments.append(src.subclipped(src_start, src_end)) + + if segments: + bgm = concatenate_audioclips(segments).with_volume_scaled(kwargs.get('bgm_volume_scale') or BGM_VOLUME_SCALE) + if bgm.duration is not None: + if bgm.duration < final_duration_s - AUDIO_DURATION_TOLERANCE_SECONDS: + bgm = self._loop_audio(bgm, final_duration_s) + elif bgm.duration > final_duration_s + AUDIO_DURATION_TOLERANCE_SECONDS: + bgm = bgm.subclipped(0, max(0.0, final_duration_s - SUBCLIP_END_SAFETY_MARGIN_S)) + layers.append(bgm) + + if not layers: + return None + return CompositeAudioClip(layers).with_duration(final_duration_s) + + @staticmethod + def _loop_audio(audio_clip: Any, duration_s: float) -> Any: + try: + n = int(duration_s // audio_clip.duration) + 2 + looped = concatenate_audioclips([audio_clip] * max(1, n)) + return looped.subclipped(0, duration_s) + except Exception: + return audio_clip + + @staticmethod + def _clamp_end_to_duration(clip: Any, end_s: float) -> float: + duration = getattr(clip, "duration", None) + if duration is None: + return end_s + return min(end_s, max(0.0, duration - SUBCLIP_END_SAFETY_MARGIN_S)) + + +# ============================================================================= +# Pipeline (non-overlapping visuals + subtitle overlay) +# ============================================================================= + +class RenderVideoPipeline: + def __init__(self, *, server_cache_dir: Path, font_info_path: Path) -> None: + self._server_cache_dir = server_cache_dir + self.font_info_path = font_info_path + with open(font_info_path, encoding='utf-8') as f: + self.font_info = json.load(f) + self._fontname2path = {font['font_name']: font['font_path'] for font in self.font_info} + + async def render(self, *, node_state: NodeState, inputs: Dict[str, Any]) -> Dict[str, Any]: + load_media: Dict[str, Any] = inputs["load_media"] + tracks: Dict[str, Any] = (inputs.get("plan_timeline") or {}).get("tracks", {}) or {} + transition_rec = inputs.get('transition_rec', []) + text_rec = inputs.get('text_rec', []) + # cut settings + crf = inputs.get('crf', DEFAULT_CRF) + clip_compose_mode = inputs.get('clip_compose_mode', 'crop') # one of `padding` and `crop` + bg_color = inputs.get('bg_color') + font_color = inputs.get('font_color') + font_size = inputs.get('font_size') + margin_bottom = inputs.get('margin_bottom') + bgm_volume_scale = inputs.get('bgm_volume_scale') + tts_volume_scale = inputs.get('tts_volume_scale') + include_video_audio = inputs.get('include_video_audio') + stroke_width = inputs.get('stroke_width') + stroke_color = inputs.get('stroke_color') + + artifact_id: str = node_state.artifact_id + session_id: str = node_state.session_id + outputs_dir: Path = self._server_cache_dir / session_id / artifact_id + outputs_dir.mkdir(parents=True, exist_ok=True) + + video_items = tracks.get("video", []) or [] + subtitle_items = tracks.get("subtitles", []) or [] + voiceover_items = tracks.get("voiceover", []) or [] + bgm_items = tracks.get("bgm", []) or [] + + if not video_items: + raise ValueError("timeline result has no video track") + + output_canvas_size = resolve_output_canvas_size(inputs) + media_map = build_media_id_to_path_map(load_media) + + override_audio = bool(voiceover_items) or bool(bgm_items) # Default: using video audio when music and tts is None. + cache = MediaCache( + include_video_audio=include_video_audio or not override_audio, + canvas_size=output_canvas_size, + clip_compose_mode=clip_compose_mode, + bg_color=bg_color, + ) + + font_path = self._fontname2path.get(text_rec[0]['font_name']) if len(text_rec) > 0 else None + subtitle_renderer = PillowSubtitleRenderer(font_path=font_path) + audio_composer = AudioTrackComposer(cache=cache) + + temp_dir = tempfile.mkdtemp(prefix=TEMP_DIRECTORY_PREFIX) + output_name = f"output_{uuid.uuid4().hex[:8]}_{int(time.time() * 1000)}.mp4" + output_path = str((outputs_dir / output_name).resolve()) + + clips_to_close: List[Any] = [] + subtitle_clips: List[Any] = [] + base_clip = None + final_clip = None + + try: + final_duration_s = self._final_duration_seconds(video_items) + + # Build base video: only concat video track + base_clip, clips_to_close, output_fps = self._build_base_video_concat( + video_items=video_items, + media_map=media_map, + cache=cache, + canvas_size=output_canvas_size, + final_duration_s=final_duration_s, + transition_rec=transition_rec + ) + + # Build subtitle: add subtitle track on base video while `subtitle_clips` is not empty. + subtitle_clips = subtitle_renderer.render( + subtitle_items, + video_size=output_canvas_size, + font_color=font_color, + font_size=font_size, + margin_bottom=margin_bottom, + stroke_width=stroke_width, + stroke_color=stroke_color, + ) + if subtitle_clips: + final_clip = CompositeVideoClip([base_clip, *subtitle_clips]).with_duration(final_duration_s) + else: + final_clip = base_clip + + # Build audios: add music and tts track + if override_audio: + final_audio = audio_composer.compose( + voiceover_items=voiceover_items, + bgm_items=bgm_items, + final_duration_s=final_duration_s, + tts_volume_scale=tts_volume_scale, + bgm_volume_scale=bgm_volume_scale, + ) + if final_audio is not None: + final_clip = final_clip.with_audio(final_audio) + else: + final_clip = final_clip.without_audio() + + loop = asyncio.get_running_loop() + + def report(progress: float, total: float | None, message: str | None): + asyncio.run_coroutine_threadsafe( + node_state.mcp_ctx.report_progress(progress, total, message), + loop, + ) + + logger = MCPMoviePyLogger(report) + + FFMPEG_PARAMS[3] = f"{crf}" # set crf (video quality setting), default is 23 (medium quality) + + await asyncio.to_thread( + final_clip.write_videofile, + output_path, + codec=VIDEO_CODEC, + audio_codec=AUDIO_CODEC, + temp_audiofile=os.path.join(temp_dir, TEMP_AUDIO_FILENAME), + remove_temp=True, + fps=output_fps, + ffmpeg_params=FFMPEG_PARAMS, + logger=logger, + ) + + node_state.node_summary.info_for_user( + f"Video generated successfully, duration: {final_duration_s} seconds, path: {output_path}", + preview_urls=[output_path], + ) + + node_state.node_summary.info_for_llm(f"Video generated successfully, duration: {final_duration_s} seconds, path: {output_path}") + return { + "output_path": output_path, + "output_basename": output_name, + "duration_s": float(final_duration_s), + "output_size": {"width": int(output_canvas_size[0]), "height": int(output_canvas_size[1])}, + } + + finally: + for c in subtitle_clips: + close_quietly(c) + for c in clips_to_close: + close_quietly(c) + close_quietly(base_clip) + close_quietly(final_clip) + cache.close() + + @staticmethod + def _final_duration_seconds(video_items: List[Dict[str, Any]]) -> float: + end_ms = max(float((it.get("timeline_window") or {}).get("end", 0.0)) for it in video_items) + return milliseconds_to_seconds(end_ms) + + def _build_base_video_concat( + self, + *, + video_items: List[Dict[str, Any]], + media_map: Dict[str, str], + cache: MediaCache, + canvas_size: Tuple[int, int], + final_duration_s: float, + transition_rec: List[Dict[str,Any]], + ) -> Tuple[Any, List[Any], float]: + # Force non-overlapping: concat only + sorted_items = sorted(video_items, key=lambda x: float((x.get("timeline_window") or {}).get("start", 0.0))) + + clips: List[Any] = [] + clips_to_close: List[Any] = [] + current_time = 0.0 + + def black_clip(duration: float) -> Any: + c = ColorClip(size=canvas_size, color=BACKGROUND_COLOR_RGB).with_duration(max(0.0, duration)) + clips_to_close.append(c) + return c + + for seg_idx, segment in enumerate(sorted_items): + timeline_window = segment.get("timeline_window", {}) or {} + start_s = milliseconds_to_seconds(timeline_window.get("start", 0.0)) + end_s = milliseconds_to_seconds(timeline_window.get("end", 0.0)) + expected_dur = max(0.0, end_s - start_s) + if expected_dur <= 0: + continue + + # fill gap + if start_s > current_time: + clips.append(black_clip(start_s - current_time)) + current_time = start_s + + seg_clip = RenderVideoPipeline._build_full_canvas_segment( + segment=segment, + media_map=media_map, + cache=cache, + canvas_size=canvas_size, + expected_duration_s=expected_dur, + ) + + if seg_clip is None: + clips.append(black_clip(expected_dur)) + else: + clips.append(seg_clip) + clips_to_close.append(seg_clip) + + current_time = start_s + expected_dur + + # trailing gap + if final_duration_s > current_time: + clips.append(black_clip(final_duration_s - current_time)) + + if not clips: + raise ValueError("no valid video segments") + + base = concatenate_videoclips(clips, method="chain").with_duration(final_duration_s) + + for transition in transition_rec: + transition_type = transition.get('type', "") + duration = transition.get('duration', 1000) / 1000 # ms -> s + if transition.get('position', '') in ('opening', 'ending'): + base = self._get_transition_clip(base, transition_type, duration) + + fps_values = [float(it.get("fps")) for it in video_items if it.get("fps")] + output_fps = max(fps_values) if fps_values else float(getattr(base, "fps", None) or DEFAULT_OUTPUT_FPS) + + return base, clips_to_close, output_fps + + @staticmethod + def _build_full_canvas_segment( + *, + segment: Dict[str, Any], + media_map: Dict[str, str], + cache: MediaCache, + canvas_size: Tuple[int, int], + expected_duration_s: float, + ) -> Optional[Any]: + source_path = segment.get("source_path") or media_map.get(segment.get("media_id")) + if not source_path: + return None + + if is_image_file(source_path): + frame = cache.get_image(source_path) + return ImageClip(frame).with_duration(expected_duration_s) + + # video + source = cache.get_video(source_path) + source_window = segment.get("source_window", {}) or {} + + src_start = milliseconds_to_seconds(source_window.get("start", 0.0)) + + end_ms = source_window.get("end", None) + if end_ms is None: + src_end = float(getattr(source, "duration", 0.0) or 0.0) + else: + src_end = milliseconds_to_seconds(end_ms) + + if src_end <= src_start: + return None + + # Clamp to avoid "end_time > duration" (common near EOF due to rounding/encoding) + source_duration_s = float(getattr(source, "duration", 0.0) or 0.0) + if source_duration_s > 0.0: + if src_start >= source_duration_s: + return None + if src_end > source_duration_s: + src_end = source_duration_s # <= duration + + if src_end <= src_start: + return None + + clip = source.subclipped(src_start, src_end) + + playback_rate = float(segment.get("playback_rate", 1.0) or 1.0) + if playback_rate != 1.0: + clip = clip.with_speed_scaled(playback_rate) + + clip_dur = float(getattr(clip, "duration", 0.0) or 0.0) + if clip_dur > 0.0 and expected_duration_s > clip_dur: + last_t = max(0.0, clip_dur - SUBCLIP_END_SAFETY_MARGIN_S) + # Freeze video/mask at last frame for the remaining duration + clip = clip.time_transform( + lambda t, lt=last_t: min(t, lt), + apply_to=["mask"], + keep_duration=True, + ) + + clip = clip.with_duration(expected_duration_s) + + # pad to full canvas (keep original "center on black" look) + if hasattr(clip, "on_color"): + clip = clip.on_color(size=canvas_size, color=cache._bg_color or BACKGROUND_COLOR_RGB, pos=CENTER_POSITION) + else: # pragma: no cover + bg = ColorClip(size=canvas_size, color=cache._bg_color or BACKGROUND_COLOR_RGB).with_duration(expected_duration_s) + clip = CompositeVideoClip([bg, clip.with_position(CENTER_POSITION)]).with_duration(expected_duration_s) + + return clip.with_duration(expected_duration_s) + + @staticmethod + def _get_transition_clip(clip: VideoClip, transition_type="fade_in", transition_duration=1.0): + + all_transition = { + "": clip, + "fade_in": clip.with_effects([vfx.FadeIn(transition_duration)]), + "fade_out": clip.with_effects([vfx.FadeOut(transition_duration)]), + } + + return all_transition.get(transition_type, clip) + + +# ============================================================================= +# Node entrypoint +# ============================================================================= + +@NODE_REGISTRY.register() +class RenderVideoNode(BaseNode): + meta = NodeMeta( + name="render_video", + description="Render final video from the timeline", + node_id="render_video", + node_kind="render_video", + require_prior_kind=["load_media", "plan_timeline", "transition_rec", "text_rec"], + default_require_prior_kind=["load_media", "plan_timeline", "transition_rec", "text_rec"], + ) + + input_schema = RenderVideoInput + + def __init__(self, server_cfg: Settings) -> None: + super().__init__(server_cfg) + self._pipeline = RenderVideoPipeline(server_cache_dir=self.server_cache_dir,font_info_path=Path(server_cfg.recommend_text.font_info_path)) + + async def default_process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Any: + return await self.process(node_state, inputs) + + async def process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Dict[str, Any]: + return await self._pipeline.render(node_state=node_state, inputs=inputs) \ No newline at end of file diff --git a/src/open_storyline/nodes/core_nodes/script_template_rec.py b/src/open_storyline/nodes/core_nodes/script_template_rec.py new file mode 100644 index 0000000000000000000000000000000000000000..38928b996409603c97f51f928719a5e1a7c379ec --- /dev/null +++ b/src/open_storyline/nodes/core_nodes/script_template_rec.py @@ -0,0 +1,58 @@ +from typing import Any, Dict +from pathlib import Path + +import numpy as np +from open_storyline.nodes.core_nodes.base_node import BaseNode, NodeMeta +from open_storyline.nodes.node_state import NodeState +from open_storyline.utils.recall import StorylineRecall +from open_storyline.utils.element_filter import ElementFilter +from open_storyline.nodes.node_schema import RecommendScriptTemplateInput +from open_storyline.utils.register import NODE_REGISTRY + +@NODE_REGISTRY.register() +class ScriptTemplateRecomendation(BaseNode): + + meta = NodeMeta( + name="script_template_rec", + description="Select an script template (script style) for generation", + node_id="script_template_rec", + node_kind="script_template_rec", + require_prior_kind=[], + default_require_prior_kind=[], + next_available_node=["generate_script"], + ) + + input_schema = RecommendScriptTemplateInput + + def __init__(self, server_cfg): + super().__init__(server_cfg) + self.element_filter = ElementFilter(json_path=self.server_cfg.script_template.script_template_info_path) + self.vectorstore = StorylineRecall.build_vectorstore(self.element_filter.library) + self._top_n = 3 + + async def default_process(self, node_state: NodeState, inputs: Dict[str, Any]): + return {} + + async def process(self, node_state: NodeState, inputs: Dict[str, Any]): + + user_request = inputs.get("user_request", "") + filter_include = inputs.get("filter_include", {}) + filter_exclude = inputs.get("filter_exclude", {}) + + # Step1: Check resources + script_template_dir: Path = self.server_cfg.script_template.script_template_dir.expanduser().resolve() + if not script_template_dir.exists(): + raise FileNotFoundError(f"`script_template_dir` not found: {script_template_dir}") + if not script_template_dir.is_dir(): + raise NotADirectoryError(f"`script_template_dir` is not a directory: {script_template_dir}") + + # Step2: Full Recall + candidates = StorylineRecall.query_top_n(self.vectorstore, query=user_request) + + # Step3: Filter tags + candidates = self.element_filter.filter(candidates, filter_include, filter_exclude) + + if not candidates: + node_state.node_summary.add_error("") + + return {"candidates": candidates[:min(self._top_n, len(candidates))]} \ No newline at end of file diff --git a/src/open_storyline/nodes/core_nodes/search_media.py b/src/open_storyline/nodes/core_nodes/search_media.py new file mode 100644 index 0000000000000000000000000000000000000000..c7b9fc87cdea896556a93ba20d4996022692d0ed --- /dev/null +++ b/src/open_storyline/nodes/core_nodes/search_media.py @@ -0,0 +1,376 @@ +import os +import requests +import time + +from typing import Any, Dict, Optional, ClassVar, Type, Tuple, List +from pydantic import BaseModel + +from pathlib import Path + +from open_storyline.nodes.core_nodes.base_node import NodeMeta, BaseNode +from open_storyline.nodes.node_schema import SearchMediaInput +from open_storyline.nodes.node_state import NodeState +from open_storyline.utils.register import NODE_REGISTRY + +SEARCH_RESULT_PER_PAGE = 40 +MAX_PHOTO_NUMBER = 10 +MAX_VIDEO_NUMBER = 10 +MIN_VIDEO_DURATION = 1 +MAX_VIDEO_DURATION = 30 +DEFAULT_RESULT_NUMBER_PER_PAGE = 50 +DEFAULT_PAGE = 1 + +TARGET_LONG_EDGE_PX = 1080 + +VALID_ORIENTATIONS = {"landscape", "portrait"} +VIDEO_QUALITY_RANK = {"sd": 0, "hd": 1, "uhd": 2} + +@NODE_REGISTRY.register() +class SearchMediaNode(BaseNode): + meta = NodeMeta( + name="search_media", + description="search", + node_id="search_media", + node_kind="search_media", + require_prior_kind=[], + default_require_prior_kind=[], + next_available_node=['load_media'], + ) + input_schema: ClassVar[Type[BaseModel]] = SearchMediaInput + + async def default_process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Dict[str, Any]: + return {} + + async def process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Dict[str, Any]: + pexels_api_key = inputs.get("pexels_api_key", "") + video_saved_paths = [] + image_saved_paths = [] + + if pexels_api_key == "": + pexels_api_key = self.server_cfg.search_media.pexels_api_key + if not pexels_api_key or pexels_api_key == "": + pexels_api_key = os.getenv("PEXELS_API_KEY") + if not pexels_api_key or pexels_api_key == "": + node_state.node_summary.info_for_llm("If the user has not entered their Pexels API key, please remind them to enter it in the sidebar of the webpage.") + raise RuntimeError("Pexels api key not detected. If you use your own pexels key, please fill in the api key in the sidebar or config.toml") + + root_dir = os.path.abspath(os.path.expanduser(self.server_cache_dir)) + media_dir = Path(os.path.join(root_dir, node_state.session_id, "media")) + + search_keyword = inputs.get("search_keyword", "") + photo_number = min(inputs.get("photo_number", MAX_PHOTO_NUMBER), MAX_PHOTO_NUMBER) + video_number = min(inputs.get("video_number", MAX_VIDEO_NUMBER), MAX_VIDEO_NUMBER) + orientation = inputs.get("orientation", "") + min_video_duration = min(max(inputs.get("min_video_duration", MIN_VIDEO_DURATION), MIN_VIDEO_DURATION), MAX_VIDEO_DURATION) + max_video_duration = max(min(inputs.get("max_video_duration", MAX_VIDEO_DURATION), MAX_VIDEO_DURATION), MIN_VIDEO_DURATION) + + if video_number > 0: + video_preview_urls, video_saved_paths = get_video_media_from_pexels( + pexels_api_key=pexels_api_key, + query=search_keyword, + media_dir=media_dir, + video_number=video_number, + orientation=orientation, + min_video_duration=min_video_duration, + max_video_duration=max_video_duration, + ) + node_state.node_summary.info_for_user(f"search media successfully, found {len(video_preview_urls)} videos", preview_urls=video_preview_urls) + + if photo_number > 0: + image_preview_urls, image_saved_paths = get_photo_media_from_pexels( + pexels_api_key=pexels_api_key, + query=search_keyword, + media_dir=media_dir, + photo_number=photo_number, + orientation=orientation, + ) + node_state.node_summary.info_for_user(f"search media successfully, found {len(image_preview_urls)} photos", preview_urls=image_preview_urls) + return {"search_media": video_saved_paths + image_saved_paths} + + +def download_video(url: str, out_path: Path) -> None: + with requests.get(url, stream=True, timeout=120) as r: + r.raise_for_status() + with open(out_path, "wb") as f: + for chunk in r.iter_content(chunk_size=1024 * 1024): + if chunk: + f.write(chunk) + +def search_videos(pexels_api_key: str, query: str, per_page, page) -> dict[str, Any]: + url = "https://api.pexels.com/videos/search" + headers = {"Authorization": pexels_api_key} + params = {"query": query, "per_page": per_page, "page": page} + r = requests.get(url, headers=headers, params=params, timeout=30) + r.raise_for_status() + return r.json() + +def filter_videos( + raw_videos: dict[str, Any], + video_number: int, + orientation: str, + min_video_duration: int, + max_video_duration: int, + ) -> list[str]: + + if video_number <= 0: + return [] + + desired_orientation = _normalize_orientation(orientation) + + results: list[str] = [] + seen: set[str] = set() + + videos = raw_videos.get("videos") or [] + for v in videos: + duration = int(v.get("duration", 0)) + + if duration < int(min_video_duration) or duration > int(max_video_duration): + continue + + w = v.get("width") + h = v.get("height") + if w is None or h is None: + continue + try: + w_i = int(w) + h_i = int(h) + except (TypeError, ValueError): + continue + + if desired_orientation is not None: + actual_orientation = _infer_orientation(w_i, h_i) + if actual_orientation != desired_orientation: + continue + + link = _pick_best_video_link(v.get("video_files") or []) + if not link: + continue + + if link in seen: + continue + + results.append(link) + seen.add(link) + + if len(results) >= video_number: + break + + return results + +def get_video_media_from_pexels( + pexels_api_key: str, + query: str, + media_dir: Path, + video_number: int, + orientation: str, + min_video_duration: int, + max_video_duration: int + ) -> Tuple[list[str], List[Dict[str, Any]]]: + + if video_number <= 0: + return ([], []) + + media_dir.mkdir(parents=True, exist_ok=True) + + collected: list[str] = [] + seen: set[str] = set() + + page = DEFAULT_PAGE + while len(collected) < video_number: + raw_videos = search_videos( + pexels_api_key=pexels_api_key, + query=query, + per_page=DEFAULT_RESULT_NUMBER_PER_PAGE, + page=page, + ) + + batch = filter_videos( + raw_videos=raw_videos, + video_number=video_number - len(collected), + orientation=orientation, + min_video_duration=min_video_duration, + max_video_duration=max_video_duration, + ) + + for url in batch: + if url not in seen: + collected.append(url) + seen.add(url) + + if not raw_videos.get("next_page") or not (raw_videos.get("videos") or []): + break + page += 1 + + video_save_path = [] + ts = int(time.time() * 1000) + for idx, url in enumerate(collected): + out_path = media_dir / f"pexels_video_{ts}_{idx}.mp4" + download_video(url, out_path) + video_save_path.append({'path': str(out_path)}) + + return collected, video_save_path + +def _normalize_orientation(orientation: str) -> Optional[str]: + normalize_orientation = (orientation or "").strip().lower() + return normalize_orientation if normalize_orientation in VALID_ORIENTATIONS else None + +def _infer_orientation(width: int, height: int) -> str: + return "landscape" if width > height else "portrait" + +def _pick_best_video_link(video_files: list[dict[str, Any]]) -> Optional[str]: + """ + Pick a "moderate" MP4 download link. + """ + mp4_candidates: list[dict[str, Any]] = [] + for file_info in video_files or []: + is_mp4 = file_info.get("file_type") == "video/mp4" + has_link = bool(file_info.get("link")) + if is_mp4 and has_link: + mp4_candidates.append(file_info) + + if not mp4_candidates: + return None + + def quality_preference(quality: Any) -> int: + # Higher is better. + quality_str = (str(quality).lower() if quality is not None else "") + if quality_str == "hd": + return 2 + if quality_str == "sd": + return 1 + if quality_str == "uhd": + return 0 + return -1 + + def candidate_score(file_info: dict[str, Any]) -> tuple[int, int, int]: + width_px = int(file_info.get("width", 0)) + height_px = int(file_info.get("height", 0)) + file_size_bytes = int(file_info.get("size", 0)) + + long_edge_px = max(width_px, height_px) + long_edge_distance = abs(long_edge_px - TARGET_LONG_EDGE_PX) + + return ( + -long_edge_distance, + quality_preference(file_info.get("quality")), # hd > sd > uhd > unknown + -file_size_bytes, # smaller file is better + ) + + best_candidate = max(mp4_candidates, key=candidate_score) + return best_candidate.get("link") + +def download_photo(url: str, out_path: Path) -> None: + with requests.get(url, stream=True, timeout=120) as r: + r.raise_for_status() + with open(out_path, "wb") as f: + for chunk in r.iter_content(chunk_size=1024 * 1024): + if chunk: + f.write(chunk) + +def search_photos(pexels_api_key: str, query: str, per_page, page) -> dict[str, Any]: + url = "https://api.pexels.com/v1/search" + headers = {"Authorization": pexels_api_key} + params = {"query": query, "per_page": per_page, "page": page} + r = requests.get(url, headers=headers, params=params, timeout=30) + r.raise_for_status() + return r.json() + +def filter_photos( + raw_photos: dict[str, Any], + photo_number: int, + orientation: str, + ) -> list[str]: + + if photo_number <= 0: + return [] + + desired_orientation = _normalize_orientation(orientation) + + results: list[str] = [] + seen: set[str] = set() + + photos = raw_photos.get("photos") or [] + for p in photos: + w = p.get("width") + h = p.get("height") + if w is None or h is None: + continue + try: + w_i = int(w) + h_i = int(h) + except (TypeError, ValueError): + continue + + if desired_orientation is not None: + actual_orientation = _infer_orientation(w_i, h_i) + if actual_orientation != desired_orientation: + continue + + src = p.get("src") or {} + if desired_orientation is not None: + url = src.get(desired_orientation) or src.get("original") + else: + url = src.get("original") or src.get("large2x") or src.get("large") or src.get("medium") + + if not url: + continue + + if url in seen: + continue + + results.append(url) + seen.add(url) + + if len(results) >= photo_number: + break + + return results + +def get_photo_media_from_pexels( + pexels_api_key: str, + query: str, + media_dir: Path, + photo_number: int, + orientation: str, + ) -> tuple[list[str], list[str]]: + + if photo_number <= 0: + return ([], []) + + media_dir.mkdir(parents=True, exist_ok=True) + + collected: list[str] = [] + seen: set[str] = set() + + page = DEFAULT_PAGE + while len(collected) < photo_number: + raw_photos = search_photos( + pexels_api_key=pexels_api_key, + query=query, + per_page=DEFAULT_RESULT_NUMBER_PER_PAGE, + page=page, + ) + + batch = filter_photos( + raw_photos=raw_photos, + photo_number=photo_number - len(collected), + orientation=orientation, + ) + + for url in batch: + if url not in seen: + collected.append(url) + seen.add(url) + + if not raw_photos.get("next_page") or not (raw_photos.get("photos") or []): + break + page += 1 + + image_save_paths = [] + ts = int(time.time() * 1000) + for idx, url in enumerate(collected): + out_path = media_dir / f"pexels_photo_{ts}_{idx}.jpg" + download_photo(url, out_path) + image_save_paths.append({"path": str(out_path)}) + + return collected, image_save_paths \ No newline at end of file diff --git a/src/open_storyline/nodes/core_nodes/select_bgm.py b/src/open_storyline/nodes/core_nodes/select_bgm.py new file mode 100644 index 0000000000000000000000000000000000000000..aa51b91b911e8210d87c1ddd955d7e9c778757e1 --- /dev/null +++ b/src/open_storyline/nodes/core_nodes/select_bgm.py @@ -0,0 +1,289 @@ +from typing import Any, Dict +from pathlib import Path + +import numpy as np +import random +import librosa + +from open_storyline.nodes.core_nodes.base_node import BaseNode, NodeMeta +from open_storyline.nodes.node_state import NodeState +from open_storyline.nodes.node_schema import SelectBGMInput +from open_storyline.utils.element_filter import ElementFilter +from open_storyline.utils.recall import StorylineRecall +from src.open_storyline.utils.prompts import get_prompt +from open_storyline.utils.parse_json import parse_json_dict +from open_storyline.utils.register import NODE_REGISTRY + +@NODE_REGISTRY.register() +class SelectBGMNode(BaseNode): + meta = NodeMeta( + name="select_bgm", + description="Select appropriate BGM based on user requirements", + node_id="select_bgm", + node_kind="music_rec", + require_prior_kind=[], + default_require_prior_kind=[], + next_available_node=["plan_timeline"], + ) + + input_schema = SelectBGMInput + + def __init__(self, server_cfg): + super().__init__(server_cfg) + self.element_filter = ElementFilter(json_path=f"{self.server_cfg.project.bgm_dir}/meta.json") + self.vectorstore = StorylineRecall.build_vectorstore(self.element_filter.library) + + async def default_process( + self, + node_state: NodeState, + inputs: Dict[str, Any], + ) -> Any: + node_state.node_summary.info_for_user("Failed to choose music") + return {"bgm": {}} + + + async def process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Any: + cfg = self.server_cfg + user_request = inputs.get("user_request", "") + filter_include = inputs.get("filter_include", {}) + filter_exclude = inputs.get("filter_exclude", {}) + bgm_info = await self.recommend(node_state, user_request, filter_include, filter_exclude) + if not bgm_info: + return {"bgm": {}} + + result = self.analyze_music_metrics(bgm_info=bgm_info, sr=cfg.select_bgm.sample_rate, hop_length=cfg.select_bgm.hop_length, frame_length=cfg.select_bgm.frame_length) + if result.get("path"): + node_state.node_summary.info_for_user(f"Successfully choose music", preview_urls = [result.get("path")]) + else: + node_state.node_summary.info_for_user("Failed to choose music") + return {"bgm": result} + + + async def recommend( + self, + node_state: NodeState, + user_request: str, + filter_include: Dict={}, + filter_exclude: Dict={} + ): + + # Step1: Check resources + bgm_dir: Path = self.server_cfg.project.bgm_dir.expanduser().resolve() + if not bgm_dir.exists(): + raise FileNotFoundError(f"bgm_dir not found: {bgm_dir}") + if not bgm_dir.is_dir(): + raise NotADirectoryError(f"bgm_dir is not a directory: {bgm_dir}") + + # Step2: Full Recall + candidates = StorylineRecall.query_top_n(self.vectorstore, query=user_request) + + # Step3: Filter tags + candidates = self.element_filter.filter(candidates, filter_include, filter_exclude) + if not candidates: + raise FileNotFoundError(f"No audio files found in: {bgm_dir}") + + # Step4: LLM Sampling + llm = node_state.llm + system_prompt = get_prompt("select_bgm.system", lang=node_state.lang) + user_prompt = get_prompt("select_bgm.user", lang=node_state.lang, candidates=candidates, user_request=user_request) + raw = await llm.complete( + system_prompt=system_prompt, + user_prompt=user_prompt, + temperature=0.1, + top_p=0.9, + max_tokens=2048, + model_preferences=None, + ) + try: + selected_json = parse_json_dict(raw) + except: + selected_json = (raw or "").strip() if raw else "Error: Unable to parse the model output" + node_state.node_summary.add_error(selected_json) + + if not isinstance(selected_json, Dict) or 'path' not in selected_json: + # Demotion select the first one of candidates + selected_json = candidates[0] + + return selected_json + + + def analyze_music_metrics( + self, + bgm_info: Dict, + sr: int = 22050, + hop_length = 2048, + frame_length = 2048, + + ) -> dict[str, Any]: + path = Path(bgm_info.get("path")) + if not path.exists(): + raise FileNotFoundError(f"File not found: {path}") + + y, sample_rate = self._load_audio_mono(path, sr=sr) + duration = int(librosa.get_duration(y=y, sr=sample_rate) * 1000) + + if y.size < frame_length: + raise RuntimeError("The selected background music is too short.") + + onset_env = librosa.onset.onset_strength(y=y, sr=sample_rate, hop_length=hop_length) + bpm, beat_frames = librosa.beat.beat_track( + onset_envelope=onset_env, + sr=sr, + hop_length=hop_length, + units="frames", + ) + + bpm_val = float(np.atleast_1d(bpm)[0]) + + beat_frames = np.asarray(beat_frames, dtype=int) + + beat_times = self._compute_accent_beats(y=y, sr=sample_rate, beat_frames=beat_frames, hop_length=hop_length) + + rms = librosa.feature.rms( + y=y, + frame_length=frame_length, + hop_length=hop_length + )[0] + + energy_mean = float(np.mean(rms)) + + rms_db = librosa.amplitude_to_db(np.maximum(rms, 1e-10), ref=1.0) + energy_mean_db = float(np.mean(rms_db)) + + lo = float(np.percentile(rms_db, 10.0)) + hi = float(np.percentile(rms_db, 95.0)) + dynamic_range_db = float(hi - lo) + + return { + "bgm_id": bgm_info.get("id"), + "path": str(path), + "duration": duration, + "sample_rate": sample_rate, + "bpm": bpm_val, + "beats": beat_times, + "energy_mean": energy_mean, + "energy_mean_db": energy_mean_db, + "dynamic_range_db": dynamic_range_db, + } + + + @staticmethod + def _load_audio_mono(path: Path, sr: int) -> tuple[np.ndarray, int]: + + try: + y, sr_out = librosa.load(path, sr=sr, mono=True) + return y.astype(np.float32, copy=False), int(sr_out) + except Exception as e1: + + # Librosa failed to read. ffmpeg is used as a fallback + import os + import subprocess + import tempfile + + tmp_wav = None + try: + with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp: + tmp_wav = tmp.name + + cmd = [ + "ffmpeg", + "-y", + "-i", str(path), + "-ac", "1", + "-ar", str(sr), + "-vn", + tmp_wav, + ] + subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + + y, sr_out = librosa.load(tmp_wav, sr=sr, mono=True) + return y.astype(np.float32, copy=False), int(sr_out) + + except FileNotFoundError as e_ffmpeg: + raise RuntimeError( + f"The audio cannot be loaded and ffmpeg is not found." + ) from e_ffmpeg + + except Exception as e2: + raise RuntimeError( + f"The audio cannot be loaded: {type(e1).__name__}: {e1}" + f"Ffmpeg error: {type(e2).__name__}: {e2}" + ) from e2 + finally: + if tmp_wav is not None: + try: + os.remove(tmp_wav) + except Exception: + pass + + + @staticmethod + def _compute_accent_beats( + y: np.ndarray, + sr: int, + beat_frames: np.ndarray, + hop_length: int, + top_pct: float = 70.0, + min_sep_beats: int = 1, # Min beat separation: 1 prevents selecting adjacent beats + use_percussive: bool = True, # Calculate onset strength from percussive component + local_norm_win: int = 8, # Window size for local normalization (measured in beats) + require_local_peak: bool = True # Only retain onsets that are local maxima + ) -> list[float]: + """ + Calculate timestamps of the top `top_pct` percent of drum beats by intensity + """ + + if beat_frames.size == 0: + return [] + + # 1) Use percussive version for onset envelope + y_for_onset = librosa.effects.percussive(y) if use_percussive else y + onset_env = librosa.onset.onset_strength(y=y_for_onset, sr=sr, hop_length=hop_length) + + # 2) Use onset strength at each beat time as beat strength + beat_frames_clip = np.clip(beat_frames.astype(int), 0, len(onset_env) - 1) + strength = onset_env[beat_frames_clip].astype(np.float64) # shape (n_beats,) + + # 3) Local normalization: prevent louder sections from dominating beat selection + if strength.size >= 3 and local_norm_win >= 3: + w = int(local_norm_win) + kernel = np.ones(w, dtype=np.float64) / w + local_mean = np.convolve(strength, kernel, mode="same") + strength_norm = strength / (local_mean + 1e-8) + else: + strength_norm = strength.copy() + + # 4) Select beats in the top top_pct percentile + thr = float(np.percentile(strength_norm, 100.0 - top_pct)) + cand = np.where(strength_norm >= thr)[0] # indices into beats + + # 5) Retain only local peaks to prevent selecting many beats during plateaus + if require_local_peak and cand.size > 0 and strength_norm.size >= 3: + is_peak = np.zeros_like(strength_norm, dtype=bool) + is_peak[1:-1] = (strength_norm[1:-1] >= strength_norm[:-2]) & (strength_norm[1:-1] >= strength_norm[2:]) + is_peak[0] = strength_norm[0] >= strength_norm[1] + is_peak[-1] = strength_norm[-1] >= strength_norm[-2] + cand = cand[is_peak[cand]] + + # 6) Minimum separation suppression + selected = [] + if cand.size > 0: + order = cand[np.argsort(-strength_norm[cand])] + suppressed = np.zeros(strength_norm.size, dtype=bool) + + for idx in order: + if suppressed[idx]: + continue + selected.append(int(idx)) + lo = max(0, idx - min_sep_beats) + hi = min(strength_norm.size, idx + min_sep_beats + 1) + suppressed[lo:hi] = True + + selected = np.array(sorted(selected), dtype=int) + + accent_frames = beat_frames[selected] + accent_times = librosa.frames_to_time(accent_frames, sr=sr, hop_length=hop_length).tolist() + + accent_times_ms = [round(x * 1000) for x in accent_times] + + return accent_times_ms diff --git a/src/open_storyline/nodes/core_nodes/split_shots.py b/src/open_storyline/nodes/core_nodes/split_shots.py new file mode 100644 index 0000000000000000000000000000000000000000..4e47ec9b056fe148a4b52601eaf41233691d796b --- /dev/null +++ b/src/open_storyline/nodes/core_nodes/split_shots.py @@ -0,0 +1,790 @@ +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Dict, Optional, List, Tuple +import csv +import functools +import os +import shutil +import subprocess +import math + + +import numpy as np + +from open_storyline.nodes.core_nodes.base_node import BaseNode, NodeMeta +from open_storyline.nodes.node_schema import SplitShotsInput +from open_storyline.nodes.node_state import NodeState +from open_storyline.nodes.node_summary import NodeSummary +from open_storyline.utils.register import NODE_REGISTRY + +MODEL_CACHE_MAXSIZE = 4 + +# TransNetV2 expects frames with shape [..., 27, 48, 3] in this implementation. +TRANSNETV2_INPUT_HEIGHT = 27 +TRANSNETV2_INPUT_WIDTH = 48 +TRANSNETV2_INPUT_CHANNELS = 3 + +DEFAULT_SCENE_DETECTION_FRAMES_PER_SECOND = 25 +DEFAULT_SCENE_DETECTION_THRESHOLD = 0.5 +DEFAULT_SPLIT_POINT_MINIMUM_GAP_SECONDS = 1e-3 + +DEFAULT_MIN_SHOT_DURATION_MILLISECONDS = 1000 +DEFAULT_MAX_SHOT_DURATION_MILLISECONDS = 30000 + +CLIP_ID_NUMBER_WIDTH = 4 +MILLISECONDS_PER_SECOND = 1000.0 + +FFMPEG_LOGLEVEL = "error" +FFMPEG_PIXEL_FORMAT_RGB24 = "rgb24" +FFMPEG_SCALE_FLAGS = "fast_bilinear" +FFMPEG_STDOUT_PIPE = "pipe:1" + +FFMPEG_ENVIRONMENT_VARIABLE_KEYS = ("IMAGEIO_FFMPEG_EXE", "FFMPEG_BINARY") +SAFE_MAP_ARGS = ["-map", "0:v:0", "-map", "0:a?", "-dn", "-sn"] + +COPY_VIDEO_WHEN_NO_SPLIT = False + +@dataclass(frozen=True) +class VideoSegment: + path: Path + start_seconds: float + end_seconds: float # ffmpeg segment csv might use -1 for "until end" in our wrapper + +# ========================= +# Model / ffmpeg helpers +# ========================= + +@functools.lru_cache(maxsize=MODEL_CACHE_MAXSIZE) +def load_transnetv2_model_cached(weight_path: str, device: str = "auto"): + """ + Load TransNetV2 model with LRU cache. Suitable for service mode. + """ + import torch + from transnetv2_pytorch import TransNetV2 + + model = TransNetV2(device=device) + model.eval() + + state_dict = torch.load(weight_path, map_location=model.device) + model.load_state_dict(state_dict) + return model + + +def resolve_ffmpeg_executable() -> str: + """ + Resolve ffmpeg executable path: + 1) env var IMAGEIO_FFMPEG_EXE / FFMPEG_BINARY + 2) system PATH + 3) imageio-ffmpeg + """ + # 1) Environment variables + for key in FFMPEG_ENVIRONMENT_VARIABLE_KEYS: + configured_value = os.getenv(key) + if not configured_value: + continue + + configured_path = Path(configured_value).expanduser() + if configured_path.exists(): + return str(configured_path) + + # env var may also be just "ffmpeg" or a command name + resolved_from_path = shutil.which(configured_value) + if resolved_from_path: + return resolved_from_path + + # 2) System PATH + ffmpeg_in_path = shutil.which("ffmpeg") + if ffmpeg_in_path: + return ffmpeg_in_path + + # 3) imageio-ffmpeg + try: + import imageio_ffmpeg + ffmpeg_from_imageio = imageio_ffmpeg.get_ffmpeg_exe() + if ffmpeg_from_imageio: + return ffmpeg_from_imageio + except Exception: + pass + + raise RuntimeError("ffmpeg not found (checked env vars, PATH, and imageio-ffmpeg).") + + +def read_video_frames_as_rgb24( + input_video: Path, + ffmpeg_executable: str, + *, + frames_per_second: int = DEFAULT_SCENE_DETECTION_FRAMES_PER_SECOND, + target_width: int = TRANSNETV2_INPUT_WIDTH, + target_height: int = TRANSNETV2_INPUT_HEIGHT, +) -> np.ndarray: + """ + Use ffmpeg to decode frames at fixed FPS and fixed size, output as raw RGB24 bytes. + Returns: np.ndarray with shape [frame_count, target_height, target_width, 3], dtype=uint8 + """ + input_video = Path(input_video) + + video_filter = ( + f"fps={frames_per_second}," + f"scale={target_width}:{target_height}:flags={FFMPEG_SCALE_FLAGS}" + ) + + command = [ + ffmpeg_executable, "-hide_banner", "-loglevel", FFMPEG_LOGLEVEL, "-nostdin", + "-i", str(input_video), + "-an", + "-vf", video_filter, + "-pix_fmt", FFMPEG_PIXEL_FORMAT_RGB24, + "-f", "rawvideo", + FFMPEG_STDOUT_PIPE, + ] + + process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + assert process.stdout is not None and process.stderr is not None + + stdout_bytes, stderr_bytes = process.communicate() + if process.returncode != 0: + raise RuntimeError( + f"ffmpeg frame extraction failed: {input_video}\n" + f"{stderr_bytes.decode('utf-8', errors='replace')}" + ) + + bytes_per_frame = target_width * target_height * TRANSNETV2_INPUT_CHANNELS + frame_count = len(stdout_bytes) // bytes_per_frame + + if frame_count <= 0: + return np.empty((0, target_height, target_width, TRANSNETV2_INPUT_CHANNELS), dtype=np.uint8) + + stdout_bytes = stdout_bytes[: frame_count * bytes_per_frame] + frames = np.frombuffer(stdout_bytes, dtype=np.uint8).reshape( + (frame_count, target_height, target_width, TRANSNETV2_INPUT_CHANNELS) + ) + return frames + + +def detect_scenes_with_transnetv2_without_proxy( + model: Any, + input_video: Path, + ffmpeg_executable: str, + *, + frames_per_second: int = DEFAULT_SCENE_DETECTION_FRAMES_PER_SECOND, + threshold: float = DEFAULT_SCENE_DETECTION_THRESHOLD, +) -> List[Dict[str, Any]]: + """ + No proxy file: + ffmpeg -> frames (uint8) -> model.predict_raw -> predictions_to_scenes_with_data + """ + import torch + + frames_numpy = read_video_frames_as_rgb24( + input_video, + ffmpeg_executable, + frames_per_second=frames_per_second, + target_width=TRANSNETV2_INPUT_WIDTH, + target_height=TRANSNETV2_INPUT_HEIGHT, + ) + + if frames_numpy.size == 0 or frames_numpy.shape[0] == 0: + return [] + + frames_tensor = torch.from_numpy(frames_numpy) # [T, H, W, 3], uint8 + frames_tensor = frames_tensor.unsqueeze(0).contiguous() # [1, T, H, W, 3] + + model_device = getattr(model, "device", None) + if model_device is not None: + frames_tensor = frames_tensor.to(model_device, non_blocking=True) + + with torch.inference_mode(): + single_prediction, _all_prediction = model.predict_raw(frames_tensor) + + prediction = single_prediction.detach().cpu().numpy() + prediction = np.squeeze(prediction) + if prediction.ndim != 1: + prediction = prediction.reshape(-1) + + scenes = model.predictions_to_scenes_with_data( + prediction, + fps=float(frames_per_second), + threshold=float(threshold), + ) + return scenes + + +def convert_scenes_to_split_points_seconds( + scenes: List[Dict[str, Any]], + *, + minimum_gap_seconds: float = DEFAULT_SPLIT_POINT_MINIMUM_GAP_SECONDS, +) -> List[float]: + """ + Convert TransNetV2 scenes to ffmpeg segment split points. + split points: [t1, t2, ...] means segments [0,t1], [t1,t2], ..., [last,end] + """ + end_times: List[float] = [] + last_end_time = 0.0 + + for scene in scenes: + try: + end_time = float(scene.get("end_time", 0.0)) + except Exception: + continue + + if end_time > last_end_time + minimum_gap_seconds: + end_times.append(end_time) + last_end_time = end_time + + # If <=1 scene, don't split + if len(end_times) <= 1: + return [] + + # Remove the last end_time (usually video end) + return end_times[:-1] + +def enforce_shot_duration_constraints_on_split_points_seconds( + split_points_seconds: List[float], + *, + total_duration_milliseconds: int, + min_shot_duration_milliseconds: Optional[int], + max_shot_duration_milliseconds: Optional[int], +) -> List[float]: + """ + 对“切分点(split points)”施加 min/max 时长约束（单位 ms）： + 1) 若某段 < min：通过删除相应切分点，把它与相邻段拼接（优先向后拼；尾段太短则向前拼）。 + 2) 若某段 > max：在该段内部强制均匀切分（允许镜头内仍有镜头切换）。 + + 注意：这里在“调用 ffmpeg 前”调整 split points，从而避免切完后再做文件拼接（性能更好）。 + """ + duration_ms = int(total_duration_milliseconds) + + def _normalize_optional_ms(value: Optional[int], key_name: str) -> Optional[int]: + if value is None: + return None + if isinstance(value, bool): + raise ValueError(f"{key_name} must be int milliseconds, got bool") + try: + value_int = int(value) + except Exception as exc: + raise ValueError(f"{key_name} must be int milliseconds, got {value!r}") from exc + if value_int < 0: + raise ValueError(f"{key_name} must be >= 0, got {value_int}") + if value_int == 0: + return None + return value_int + + min_ms = _normalize_optional_ms(min_shot_duration_milliseconds, "min_shot_duration") + max_ms = _normalize_optional_ms(max_shot_duration_milliseconds, "max_shot_duration") + + if min_ms is not None and max_ms is not None and min_ms > max_ms: + raise ValueError(f"min_shot_duration ({min_ms}) cannot be greater than max_shot_duration ({max_ms}).") + + # seconds -> milliseconds cut points + cut_points_ms = [ + int(round(point_seconds * MILLISECONDS_PER_SECOND)) + for point_seconds in split_points_seconds + ] + cut_points_ms = sorted({c for c in cut_points_ms if 0 < c < duration_ms}) + + # ---------- Step 1: merge short segments (< min) by removing cut points ---------- + if min_ms is not None and cut_points_ms: + merged_cut_points: List[int] = [] + segment_start_ms = 0 + + for cut_ms in cut_points_ms: + segment_length_ms = cut_ms - segment_start_ms + if segment_length_ms < min_ms: + continue + merged_cut_points.append(cut_ms) + segment_start_ms = cut_ms + + if merged_cut_points and (duration_ms - merged_cut_points[-1] < min_ms): + merged_cut_points.pop() + + cut_points_ms = merged_cut_points + + # ---------- Step 2: split long segments (> max) by inserting internal cut points ---------- + if max_ms is not None and max_ms > 0: + cuts_set = set(cut_points_ms) + boundaries = [0] + cut_points_ms + [duration_ms] + + for segment_start_ms, segment_end_ms in zip(boundaries[:-1], boundaries[1:]): + segment_length_ms = segment_end_ms - segment_start_ms + if segment_length_ms <= max_ms: + continue + + # 最少切成 pieces 段，保证每段 <= max_ms + pieces = int(math.ceil(segment_length_ms / max_ms)) + if pieces <= 1: + continue + + # 均匀分配（整数 ms），尽量避免“最后剩一小段” + base = segment_length_ms // pieces + remainder = segment_length_ms % pieces # 前 remainder 段多 1ms + + current = segment_start_ms + for i in range(pieces - 1): + piece_len = base + 1 if i < remainder else base + current += piece_len + if segment_start_ms < current < segment_end_ms: + cuts_set.add(current) + + cut_points_ms = sorted(c for c in cuts_set if 0 < c < duration_ms) + + # milliseconds -> seconds + return [cut_ms / MILLISECONDS_PER_SECOND for cut_ms in cut_points_ms] + +def segment_video_stream_copy_with_ffmpeg( + input_video: Path, + ffmpeg_executable: str, + *, + split_points_seconds: List[float], + output_directory: Path, + filename_prefix: str, + start_index: int = 0, +) -> List[VideoSegment]: + """ + Fast segmentation: stream copy (-c copy) + segment muxer. + Returns segments with start/end in seconds from ffmpeg segment list csv. + """ + output_directory.mkdir(parents=True, exist_ok=True) + + # No split points -> single output copy + if not split_points_seconds: + output_path = output_directory / f"{filename_prefix}_{start_index:0{CLIP_ID_NUMBER_WIDTH}d}.mp4" + command = [ + ffmpeg_executable, "-hide_banner", "-loglevel", FFMPEG_LOGLEVEL, "-nostdin", + "-y", + "-i", str(input_video), + *SAFE_MAP_ARGS, + "-c", "copy", + "-movflags", "+faststart", + str(output_path), + ] + completed = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if completed.returncode != 0: + raise RuntimeError( + f"ffmpeg stream copy failed: {input_video}\n" + f"{completed.stderr.decode('utf-8', errors='replace')}" + ) + return [VideoSegment(path=output_path, start_seconds=0.0, end_seconds=-1.0)] + + split_points_argument = ",".join(f"{t:.3f}" for t in split_points_seconds) + + segment_list_csv_path = output_directory / f"{filename_prefix}_{start_index:0{CLIP_ID_NUMBER_WIDTH}d}.csv" + output_pattern = output_directory / f"{filename_prefix}_%0{CLIP_ID_NUMBER_WIDTH}d.mp4" + + command = [ + ffmpeg_executable, "-hide_banner", "-loglevel", FFMPEG_LOGLEVEL, "-nostdin", + "-y", + "-i", str(input_video), + *SAFE_MAP_ARGS, + "-c", "copy", + "-f", "segment", + "-segment_start_number", str(start_index), + "-segment_list", str(segment_list_csv_path), + "-segment_list_type", "csv", + "-segment_times", split_points_argument, + "-reset_timestamps", "1", + "-segment_format_options", "movflags=+faststart", + str(output_pattern), + ] + + completed = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if completed.returncode != 0: + raise RuntimeError( + f"ffmpeg segment failed: {input_video}\n" + f"{completed.stderr.decode('utf-8', errors='replace')}" + ) + + segments: List[VideoSegment] = [] + with segment_list_csv_path.open("r", encoding="utf-8", newline="") as file_handle: + csv_reader = csv.reader(file_handle) + for row in csv_reader: + if not row or len(row) < 3: + continue + filename, start_time, end_time = row[0], row[1], row[2] + segments.append( + VideoSegment( + path=output_directory / filename, + start_seconds=float(start_time), + end_seconds=float(end_time), + ) + ) + + return segments + + +# ========================= +# Node implementation +# ========================= + +@NODE_REGISTRY.register() +class SplitShotsNode(BaseNode): + meta = NodeMeta( + name="split_shots", + description="Segment input video based on shot boundary detection", + node_id="split_shots", + node_kind="split_shots", + require_prior_kind=["load_media"], + default_require_prior_kind=["load_media"], + next_available_node=["understand_clips", "understand_clips_pro"], + ) + input_schema = SplitShotsInput + + def __init__(self, *args) -> None: + super().__init__(*args) + + self.transnetv2_model = load_transnetv2_model_cached( + str(self.server_cfg.split_shots.transnet_weights), + device=self.server_cfg.split_shots.transnet_device, + ) + self.ffmpeg_executable = resolve_ffmpeg_executable() + + # ------------------------- + # Public entrypoints + # ------------------------- + + async def default_process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Any: + """ + Default behavior: do NOT split shots, just pass-through paths. + (Optimization) No re-save / no re-encode. + """ + output_directory = self._prepare_output_directory(node_state, inputs) + media = self._extract_media(inputs) + + clips: List[Dict[str, Any]] = [] + clip_index = 1 + + for media_item in media: + clip = self._build_clip_without_splitting(media_item=media_item, clip_index=clip_index, node_summary=node_state.node_summary) + clips.append(clip) + clip_index += 1 + + node_state.node_summary.info_for_user("Shot splitting skipped") + return {"clips": clips} + + async def process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Any: + """ + Split shots using TransNetV2 + ffmpeg segment copy. + """ + output_directory = self._prepare_output_directory(node_state, inputs) + media = self._extract_media(inputs) + + clips: List[Dict[str, Any]] = [] + clip_index = 1 + + min_shot_duration_milliseconds = inputs.get("min_shot_duration", DEFAULT_MIN_SHOT_DURATION_MILLISECONDS) + max_shot_duration_milliseconds = inputs.get("max_shot_duration", DEFAULT_MAX_SHOT_DURATION_MILLISECONDS) + + if min_shot_duration_milliseconds > max_shot_duration_milliseconds: + min_shot_duration_milliseconds, max_shot_duration_milliseconds = DEFAULT_MIN_SHOT_DURATION_MILLISECONDS, DEFAULT_MAX_SHOT_DURATION_MILLISECONDS + node_state.node_summary.add_warning( + f"min_shot_duration_milliseconds ({min_shot_duration_milliseconds}) cannot greater than max_shot_duration_milliseconds ({max_shot_duration_milliseconds})" + f"using default config {DEFAULT_MIN_SHOT_DURATION_MILLISECONDS} and ({DEFAULT_MAX_SHOT_DURATION_MILLISECONDS})", + artifact_id = node_state.artifact_id, + ) + + if min_shot_duration_milliseconds < DEFAULT_MIN_SHOT_DURATION_MILLISECONDS: + min_shot_duration_milliseconds = DEFAULT_MIN_SHOT_DURATION_MILLISECONDS + node_state.node_summary.add_warning( + f"min_shot_duration_milliseconds ({min_shot_duration_milliseconds}) too small" + f"using default config {DEFAULT_MIN_SHOT_DURATION_MILLISECONDS}", + artifact_id = node_state.artifact_id, + ) + + if max_shot_duration_milliseconds > DEFAULT_MAX_SHOT_DURATION_MILLISECONDS: + max_shot_duration_milliseconds = DEFAULT_MAX_SHOT_DURATION_MILLISECONDS + node_state.node_summary.add_warning( + f"max_shot_duration_milliseconds ({max_shot_duration_milliseconds}) too great" + f"using default config {DEFAULT_MAX_SHOT_DURATION_MILLISECONDS}", + artifact_id = node_state.artifact_id, + ) + + for media_item in media: + new_clips, clip_index = self._process_single_media_item( + media_item=media_item, + output_directory=output_directory, + starting_clip_index=clip_index, + node_summary=node_state.node_summary, + min_shot_duration_milliseconds=min_shot_duration_milliseconds, + max_shot_duration_milliseconds=max_shot_duration_milliseconds, + ) + clips.extend(new_clips) + + node_state.node_summary.info_for_user( + f"{self.meta.node_id} executed successfully, output clips count: {len(clips)}" + ) + return {"clips": clips} + + # ------------------------- + # Internal helpers + # ------------------------- + + def _prepare_output_directory(self, node_state: NodeState, inputs: Dict[str, Any]) -> Path: + artifact_id = node_state.artifact_id + session_id = node_state.session_id + output_directory = self.server_cache_dir / session_id / artifact_id + output_directory.mkdir(parents=True, exist_ok=True) + return output_directory + + def _extract_media(self, inputs: Dict[str, Any]) -> List[Dict[str, Any]]: + return (inputs.get("load_media") or {}).get("media", []) or [] + + def _format_clip_id(self, clip_index: int) -> str: + return f"clip_{clip_index:0{CLIP_ID_NUMBER_WIDTH}d}" + + def _require_media_id(self, media_item: Dict[str, Any]) -> str: + media_id = media_item.get("media_id") + if not media_id: + raise ValueError(f"media_item missing required field 'media_id': {media_item}") + return str(media_id) + + def _require_media_type(self, media_item: Dict[str, Any]) -> str: + media_type = media_item.get("media_type") + if not media_type: + raise ValueError(f"media_item missing required field 'media_type': {media_item}") + return str(media_type) + + def _require_video_metadata(self, media_id: str, media_item: Dict[str, Any]) -> Dict[str, Any]: + metadata = media_item.get("metadata") or {} + if "duration" not in metadata: + raise ValueError(f"video media_id={media_id} missing metadata.duration") + return metadata + + def _parse_duration_milliseconds(self, media_id: str, metadata: Dict[str, Any]) -> int: + try: + duration_milliseconds = int(metadata["duration"]) + except (TypeError, ValueError): + raise ValueError(f"video media_id={media_id} has invalid metadata.duration: {metadata.get('duration')!r}") + + if duration_milliseconds < 0: + raise ValueError(f"video media_id={media_id} has negative duration: {duration_milliseconds}") + return duration_milliseconds + + def _require_path(self, media_id: str, media_item: Dict[str, Any], *, field_name: str) -> str: + path_value = media_item.get(field_name) + if not path_value: + raise ValueError(f"media_id={media_id} missing required field {field_name!r}") + return str(path_value) + + def _build_clip_without_splitting(self, media_item: Dict[str, Any], clip_index: int, node_summary: NodeSummary) -> Dict[str, Any]: + """ + Build a single clip without cutting: + - image: use orig_path (or fallback to path) + - video: use path (no re-save) + """ + media_id = self._require_media_id(media_item) + media_type = self._require_media_type(media_item) + clip_id = self._format_clip_id(clip_index) + + if media_type == "image": + image_path = media_item.get("orig_path") or media_item.get("path") + if not image_path: + raise ValueError(f"image media_id={media_id} missing 'orig_path'/'path'") + node_summary.info_for_user(f"{clip_id} 分割完成", preview_urls=[image_path]) + return { + "clip_id": clip_id, + "kind": "image", + "path": image_path, + "source_ref": { + "media_id": media_id, + "height": media_item.get("metadata", {}).get("height"), + "width": media_item.get("metadata", {}).get("width"), + }, + } + + if media_type != "video": + raise ValueError(f"unsupported media_type {media_type!r} for media_id={media_id}") + + metadata = self._require_video_metadata(media_id, media_item) + duration_milliseconds = self._parse_duration_milliseconds(media_id, metadata) + video_path = self._require_path(media_id, media_item, field_name="path") + + node_summary.info_for_user(f"{clip_id} split successfully", preview_urls=[video_path]) + return { + "clip_id": clip_id, + "kind": "video", + "path": video_path, + "fps": metadata.get("fps"), + "source_ref": { + "media_id": media_id, + "start": 0, + "end": duration_milliseconds, + "duration": duration_milliseconds, + "height": metadata.get("height"), + "width": metadata.get("width"), + }, + } + + def _process_single_media_item( + self, + *, + media_item: Dict[str, Any], + output_directory: Path, + starting_clip_index: int, + node_summary: NodeSummary, + min_shot_duration_milliseconds: int, + max_shot_duration_milliseconds: int, + ) -> Tuple[List[Dict[str, Any]], int]: + """ + Return: (clips_generated, next_clip_index) + """ + media_id = self._require_media_id(media_item) + media_type = self._require_media_type(media_item) + + if media_type == "image": + clip_id = self._format_clip_id(starting_clip_index) + image_path = media_item.get("orig_path") or media_item.get("path") + if not image_path: + raise ValueError(f"image media_id={media_id} missing 'orig_path'/'path'") + + node_summary.info_for_user(f"{clip_id} split successfully", preview_urls=[image_path]) + clip = { + "clip_id": clip_id, + "kind": "image", + "path": image_path, + "source_ref": { + "media_id": media_id, + "height": media_item.get("metadata", {}).get("height"), + "width": media_item.get("metadata", {}).get("width"), + }, + } + return [clip], starting_clip_index + 1 + + if media_type != "video": + raise ValueError(f"unsupported media_type {media_type!r} for media_id={media_id}") + + video_clips, next_index = self._process_video_media_item( + media_id=media_id, + media_item=media_item, + output_directory=output_directory, + starting_clip_index=starting_clip_index, + node_summary=node_summary, + min_shot_duration_milliseconds=min_shot_duration_milliseconds, + max_shot_duration_milliseconds=max_shot_duration_milliseconds, + ) + return video_clips, next_index + + def _process_video_media_item( + self, + *, + media_id: str, + media_item: Dict[str, Any], + output_directory: Path, + starting_clip_index: int, + node_summary: NodeSummary, + min_shot_duration_milliseconds: int, + max_shot_duration_milliseconds: int, + ) -> Tuple[List[Dict[str, Any]], int]: + metadata = self._require_video_metadata(media_id, media_item) + duration_milliseconds = self._parse_duration_milliseconds(media_id, metadata) + + input_video_path = Path(self._require_path(media_id, media_item, field_name="path")).expanduser() + + # If the media itself is shorter than min_shot_duration: skip segmentation and concatenation entirely + if min_shot_duration_milliseconds is not None and duration_milliseconds < min_shot_duration_milliseconds: + clip_id = self._format_clip_id(starting_clip_index) + node_summary.info_for_user(f"{clip_id} split successfully", preview_urls=[str(input_video_path)]) + clip = { + "clip_id": clip_id, + "kind": "video", + "path": str(input_video_path), + "fps": metadata.get("fps"), + "source_ref": { + "media_id": media_id, + "start": 0, + "end": duration_milliseconds, + "duration": duration_milliseconds, + "height": metadata.get("height"), + "width": metadata.get("width"), + }, + } + return [clip], starting_clip_index + 1 + + # 1) Detect scenes + scenes = detect_scenes_with_transnetv2_without_proxy( + self.transnetv2_model, + input_video_path, + self.ffmpeg_executable, + frames_per_second=DEFAULT_SCENE_DETECTION_FRAMES_PER_SECOND, + threshold=DEFAULT_SCENE_DETECTION_THRESHOLD, + ) + split_points_seconds = convert_scenes_to_split_points_seconds(scenes) + + split_points_seconds = enforce_shot_duration_constraints_on_split_points_seconds( + split_points_seconds, + total_duration_milliseconds=duration_milliseconds, + min_shot_duration_milliseconds=min_shot_duration_milliseconds, + max_shot_duration_milliseconds=max_shot_duration_milliseconds, + ) + + # 2) If no split needed, optionally skip copying + if not split_points_seconds and not COPY_VIDEO_WHEN_NO_SPLIT: + clip_id = self._format_clip_id(starting_clip_index) + node_summary.info_for_user(f"{clip_id} split successfully", preview_urls=[str(input_video_path)]) + clip = { + "clip_id": clip_id, + "kind": "video", + "path": str(input_video_path), + "fps": metadata.get("fps"), + "source_ref": { + "media_id": media_id, + "start": 0, + "end": duration_milliseconds, + "duration": duration_milliseconds, + "height": metadata.get("height"), + "width": metadata.get("width"), + }, + } + return [clip], starting_clip_index + 1 + + # 3) Segment by ffmpeg (-c copy) + filename_prefix = "clip" + segments = segment_video_stream_copy_with_ffmpeg( + input_video=input_video_path, + ffmpeg_executable=self.ffmpeg_executable, + split_points_seconds=split_points_seconds, + output_directory=output_directory, + filename_prefix=filename_prefix, + start_index=starting_clip_index, + ) + + # 4) Build clip list + clips: List[Dict[str, Any]] = [] + clip_index = starting_clip_index + + for segment in segments: + clip_id = self._format_clip_id(clip_index) + + if segment.end_seconds < 0: + start_milliseconds = 0 + end_milliseconds = duration_milliseconds + else: + start_milliseconds = max(0, int(round(segment.start_seconds * MILLISECONDS_PER_SECOND))) + end_milliseconds = max(start_milliseconds, int(round(segment.end_seconds * MILLISECONDS_PER_SECOND))) + + segment_duration_milliseconds = max(0, end_milliseconds - start_milliseconds) + if segment_duration_milliseconds <= 0: + continue + + output_path_string = str(segment.path) + node_summary.info_for_user(f"{clip_id} split successfully", preview_urls=[output_path_string]) + + clips.append( + { + "clip_id": clip_id, + "kind": "video", + "path": output_path_string, + "fps": metadata.get("fps"), + "source_ref": { + "media_id": media_id, + "start": start_milliseconds, + "end": end_milliseconds, + "duration": segment_duration_milliseconds, + "height": metadata.get("height"), + "width": metadata.get("width"), + }, + } + ) + clip_index += 1 + + return clips, clip_index diff --git a/src/open_storyline/nodes/core_nodes/understand_clips.py b/src/open_storyline/nodes/core_nodes/understand_clips.py new file mode 100644 index 0000000000000000000000000000000000000000..fb5442c212810632f61210e21a94ecad30f4e802 --- /dev/null +++ b/src/open_storyline/nodes/core_nodes/understand_clips.py @@ -0,0 +1,214 @@ +from typing import Any, Dict +import asyncio + +from open_storyline.nodes.core_nodes.base_node import BaseNode, NodeMeta +from src.open_storyline.utils.prompts import get_prompt +from open_storyline.utils.parse_json import parse_json_dict +from open_storyline.nodes.node_state import NodeState +from open_storyline.nodes.node_schema import UnderstandClipsInput +from open_storyline.utils.register import NODE_REGISTRY + +@NODE_REGISTRY.register() +class UnderstandClipsNode(BaseNode): + """ + Media Understanding Node + """ + + meta = NodeMeta( + name="understand_clips", + description="Analyze clips and generate descriptions for each. Requires `load_media` and `split_shots` output", + node_id="understand_clips", + node_kind="understand_clips", + require_prior_kind=['load_media', 'split_shots'], + default_require_prior_kind=['load_media', 'split_shots'], + next_available_node=['filter_clips', 'filter_clips_pro'], + ) + + input_schema = UnderstandClipsInput + + async def default_process( + self, + node_state: NodeState, + inputs: Dict[str, Any], + ) -> Any: + clips = inputs["split_shots"]["clips"] + + clip_captions: list[dict[str, Any]] = [] + for clip in clips or []: + clip_captions.append( + { + "clip_id": clip.get("clip_id"), + "caption": "no caption", + "source_ref": { + "media_id": clip.get("source_ref", {}).get("media_id", ""), + } + } + ) + node_state.node_summary.info_for_user(f"Skipped description generation for {len(clips)} clips") + return { + "clip_captions": clip_captions, + "overall": "unknown", + } + + async def process(self, node_state: NodeState, inputs: Dict[str, Any]) -> Any: + """ + inputs: Previous node results read by BaseNode.load_inputs(ctx) + """ + load_media = inputs["media"] + clips = inputs["split_shots"]["clips"] + llm = node_state.llm + system_prompt = get_prompt("understand_clips.system_detail", lang=node_state.lang) + user_prompt = get_prompt("understand_clips.user_detail", lang=node_state.lang) + + + clip_captions: list[dict[str, Any]] = [] + + for clip in clips or []: + clip_id = str(clip.get("clip_id", "") or "").strip() or "(unknown_clip)" + kind = str(clip.get("kind", "") or "").strip().lower() + src = clip.get("source_ref") or {} + + media_id = str(src.get("media_id", "") or "") + media_item = load_media.get(media_id) + + out_item: dict[str, Any] = { + "clip_id": clip_id, + } + + if not media_item: + out_item["caption"] = f"Error: Media not found for media_id={media_id}" + clip_captions.append(out_item) + continue + + path = str(media_item.get("path", "") or "").strip() + if not path: + out_item["caption"] = f"Error: No path specified for media_id={media_id}" + clip_captions.append(out_item) + continue + + # 组装 media + media: list[Any] = [] + + if kind == "image": + media = [{"path": path}] + + elif kind == "video": + in_sec = _safe_float(src.get("start", 0) / 1000.0, 0.0) + + if src.get("end") is not None: + out_sec = _safe_float(src.get("end", 0) / 1000.0, in_sec) + else: + dur = _safe_float(src.get("duration", 0.0), 0.0) + out_sec = in_sec + max(0.0, dur) + + if out_sec <= in_sec: + out_sec = in_sec + 0.1 + + media = [{ + "path": path, + "in_sec": in_sec, + "out_sec": out_sec, + }] + else: + out_item["caption"] = f"Error: Clip kind not supported: {kind}" + clip_captions.append(out_item) + continue + + max_retries = 2 + raw = None + last_exc: Exception | None = None + + for attempt in range(max_retries + 1): + try: + raw = await llm.complete( + system_prompt=system_prompt, + user_prompt=user_prompt, + media=media, + temperature=0.3, + top_p=0.9, + max_tokens=2048, + model_preferences=None, + ) + if raw is not None: + last_exc = None + break + except Exception as e: + last_exc = e + + if attempt < max_retries: + await asyncio.sleep(0.3 * (attempt + 1)) + + if raw is None: + out_item["caption"] = "Error: VLM request failed" + try: + raw_score = obj.get("aes_score") + out_item["aes_score"] = float(str(raw_score).strip()) + except (ValueError, TypeError, AttributeError): + # If the conversion fails (such as "abc", None, "nan", etc.), assign the value -1.0 + out_item["aes_score"] = -1.0 + node_state.node_summary.add_error(repr(last_exc)) + clip_captions.append(out_item) + continue + + try: + obj = parse_json_dict(raw) + except: + text = (raw or "").strip() + out_item["caption"] = text if text else "Error: Unable to parse model output" + clip_captions.append(out_item) + continue + + out_item["caption"] = str(obj.get("caption", "") or "").strip() + out_item["source_ref"] = { + "media_id": clip.get("source_ref", {}).get("media_id", ""), + } + clip_captions.append(out_item) + + desc_lines: list[str] = [] + for desc in clip_captions: + text = str(desc.get("caption")) + desc_lines.append(f"- {desc.get('clip_instance_id')}: {text}") + + overall_summary = "" + if desc_lines: + overall_system_prompt = get_prompt("understand_clips.system_overall", lang=node_state.lang) + overall_user_prompt = get_prompt("understand_clips.user_overall", lang=node_state.lang, clips_captions=desc_lines) + + try: + overall_summary = await llm.complete( + system_prompt=overall_system_prompt, + user_prompt=overall_user_prompt, + media=None, + temperature=0.3, + top_p=0.9, + max_tokens=1024, + model_preferences=None + ) + + except Exception as e: + overall_summary = f"Error: Summary generation failed: {type(e).__name__}: {e}" + node_state.node_summary.info_for_user(f"Clip understanding completed. Analyzed {len(clip_captions)} clips in total. Overall description: {overall_summary}") + return { + "clip_captions": clip_captions, + "overall": overall_summary + } + + + def _parse_input(self, node_state: NodeState, inputs: Dict[str, Any]): + media = inputs["load_media"]["media"] + + load_media: dict[str, dict[str, Any]] = {} + for media_item in media or []: + media_id = media_item.get("media_id") + if media_id: + load_media[str(media_id)] = media_item + inputs.update({"media": load_media}) + return inputs + +def _safe_float(x: Any, default: float = 0.0) -> float: + try: + if x is None: + return default + return float(x) + except Exception: + return default \ No newline at end of file diff --git a/src/open_storyline/nodes/node_manager.py b/src/open_storyline/nodes/node_manager.py new file mode 100644 index 0000000000000000000000000000000000000000..b9e768129c961109f0fdf0949682eeb1d2e97b27 --- /dev/null +++ b/src/open_storyline/nodes/node_manager.py @@ -0,0 +1,168 @@ +from __future__ import annotations +from collections import defaultdict +from typing import Any, Dict, List, Optional, Set + + +from langchain_core.tools.structured import StructuredTool + +from src.open_storyline.storage.agent_memory import ArtifactStore + + +class NodeManager: + def __init__(self, tools: List[StructuredTool] = None): + self.kind_to_node_ids: Dict[str, List[str]] = defaultdict(list) # node_kind -> list of node_ids (sorted) + self.id_to_tool: Dict[str, StructuredTool] = {} # node_id -> StructuredTool + self.id_to_next: Dict[str, List[str]] = {} # node_id -> list of next executable node_ids + self.id_to_priority: Dict[str, int] = {} # node_id -> priority + self.id_to_kind: Dict[str, str] = {} # node_id -> node_kind + + # New: Prerequisite dependency related + self.id_to_require_prior_kind: Dict[str, List[str]] = {} # node_id -> required prerequisite features when executing auto method + self.id_to_default_require_prior_kind: Dict[str, List[str]] = {} # node_id -> prerequisite features needed for default method execution + + # Reverse index: which nodes depend on a specific kind + self.kind_to_dependent_nodes: Dict[str, Set[str]] = defaultdict(set) # kind -> set of node_ids that depend on this feature + self.kind_to_default_dependent_nodes: Dict[str, Set[str]] = defaultdict(set) # kind -> set of node_ids whose default method depends on this feature + + if tools: + self._build(tools) + + def _build(self, tools: List[StructuredTool]): + for tool in tools: + if tool.metadata: + metadata = tool.metadata.get('_meta', {}) + node_id = metadata.get('node_id') + if node_id: + self.add_node(tool) + + def add_node(self, tool: StructuredTool) -> bool: + # metadata is None, failed to add node + if not tool.metadata: + return False + + metadata = tool.metadata.get('_meta', {}) + node_id = metadata.get('node_id') + + if not node_id: + return False + + if node_id in self.id_to_tool: + self.remove_node(node_id) + + node_kind = metadata.get('node_kind', node_id) + priority = metadata.get('priority', 0) + next_nodes = metadata.get('next_available_node', []) + require_prior_kind = metadata.get('require_prior_kind', []) + default_require_prior_kind = metadata.get('default_require_prior_kind', []) + + # Update dependencies + self.id_to_tool[node_id] = tool + self.id_to_priority[node_id] = priority + self.id_to_next[node_id] = next_nodes + self.id_to_kind[node_id] = node_kind + self.id_to_require_prior_kind[node_id] = require_prior_kind + self.id_to_default_require_prior_kind[node_id] = default_require_prior_kind + + # Add to kind_to_node_ids and re-sort + self.kind_to_node_ids[node_kind].append(node_id) + self._sort_kind(node_kind) + + # Update reverse index + for kind in require_prior_kind: + self.kind_to_dependent_nodes[kind].add(node_id) + + for kind in default_require_prior_kind: + self.kind_to_default_dependent_nodes[kind].add(node_id) + + return True + + def remove_node(self, node_id: str, clean_references: bool = True) -> bool: + """ + Delete a node, not used for the time being. + + Args: + node_id: ID of the node to delete + clean_references: Whether to clean up references to this node from other nodes + """ + + if node_id not in self.id_to_tool: + return False + + node_kind = self.id_to_kind[node_id] + + # Clean up reverse index + if node_id in self.id_to_require_prior_kind: + for kind in self.id_to_require_prior_kind[node_id]: + self.kind_to_dependent_nodes[kind].discard(node_id) + if not self.kind_to_dependent_nodes[kind]: + del self.kind_to_dependent_nodes[kind] + + if node_id in self.id_to_default_require_prior_kind: + for kind in self.id_to_default_require_prior_kind[node_id]: + self.kind_to_default_dependent_nodes[kind].discard(node_id) + if not self.kind_to_default_dependent_nodes[kind]: + del self.kind_to_default_dependent_nodes[kind] + + del self.id_to_tool[node_id] + del self.id_to_priority[node_id] + del self.id_to_next[node_id] + del self.id_to_kind[node_id] + + if node_id in self.id_to_require_prior_kind: + del self.id_to_require_prior_kind[node_id] + if node_id in self.id_to_default_require_prior_kind: + del self.id_to_default_require_prior_kind[node_id] + + # Remove from kind group + if node_id in self.kind_to_node_ids[node_kind]: + self.kind_to_node_ids[node_kind].remove(node_id) + + # If no nodes left for this kind, remove the kind + if not self.kind_to_node_ids[node_kind]: + del self.kind_to_node_ids[node_kind] + + # Remove references to this node in other nodes + if clean_references: + for nid in list(self.id_to_next.keys()): + if node_id in self.id_to_next[nid]: + self.id_to_next[nid].remove(node_id) + + return True + + + def _sort_kind(self, kind: str): + """Sort node list for specified kind by priority""" + if kind in self.kind_to_node_ids: + self.kind_to_node_ids[kind].sort( + key=lambda nid: self.id_to_priority[nid], + reverse=True + ) + + def get_tool(self, node_id: str) -> Optional[StructuredTool]: + """Get tool by node_id""" + return self.id_to_tool.get(node_id) + + def check_excutable(self, session_id:str, store: ArtifactStore, all_require_kind: List[str]) -> Dict[str, Any]: + """ + Check if executable and return unexecuted features + """ + collected_output = {} + for req_kind in all_require_kind: + req_ids_queue = self.kind_to_node_ids[req_kind] + # 1. Collect latest outputs from all nodes + valid_outputs = [] + for node_id in req_ids_queue: + output = store.get_latest_meta(node_id=node_id, session_id=session_id) + if output is not None: + valid_outputs.append(output) + + # 2. Identify the most recently created output + if valid_outputs: + latest_output = max(valid_outputs, key=lambda output: output.created_at) + collected_output[req_kind] = latest_output + return { + "excutable": len(collected_output.keys())==len(all_require_kind), + "collected_node": collected_output, + "missing_kind": list(set(all_require_kind) - set(collected_output.keys())) + } + diff --git a/src/open_storyline/nodes/node_schema.py b/src/open_storyline/nodes/node_schema.py new file mode 100644 index 0000000000000000000000000000000000000000..063378a914de8deeb28608f68357b7bbc91147a7 --- /dev/null +++ b/src/open_storyline/nodes/node_schema.py @@ -0,0 +1,448 @@ +from typing import Dict, List, Literal, Any, Annotated, Optional, Union, ClassVar, Type, Tuple +from pydantic import BaseModel, Field, model_validator, constr, conlist + + +class VideoMetadata(BaseModel): + """Video metadata""" + width: int = Field(description="Width") + height: int = Field(description="Height") + duration: float = Field(description="Duration (milliseconds)") + fps: float = Field(description="Video frame rate per second") + has_audio: bool = Field(default=False, description="Whether audio track is present") + + audio_sample_rate_hz: Optional[int] = Field( + None, + gt=0, + description="Audio sample rate (Hz), common values: 44100, 48000" + ) + + @model_validator(mode='after') + def validate_audio_sample_rate(self): + """Audio sample rate is required if audio is present""" + if self.has_audio and self.audio_sample_rate_hz is None: + raise ValueError('audio_sample_rate_hz must be provided when video contains audio') + return self + +class ImageMetadata(BaseModel): + """Image metadata""" + width: int = Field(description="Width") + height: int = Field(description="Height") + + +class Media(BaseModel): + """Single media""" + media_id: str + path: str + media_type: Literal["video", "image", "audio", "unknown"] + metadata: Union[VideoMetadata, ImageMetadata] + extra_info: Optional[Dict[str, Any]] = None + + +class SourceRef(BaseModel): + """ Original media reference information """ + media_id: str + start: float + end: float + duration: float + height: Optional[int] = None + width: Optional[int] = None + + +class Clip(BaseModel): + clip_id: str + language: Optional[str] = None + caption: str = Field(default="", description="Caption describing the media") + media_type: str + path: str + fps: Optional[float] = None + extra_info: Optional[Dict[str, Any]] = Field(default=None, description="Extra metadata") + + +class SubtitleUnit(BaseModel): + """Subtitle segmentation unit""" + unit_id: str = Field( + ..., + description="Unique identifier for subtitle unit", + example="subtitle_0001" + ) + index_in_group: int = Field( + ..., + ge=0, + description="Sequential index within current group (starting from 0)", + example=0 + ) + text: str = Field( + ..., + description="Text content of this subtitle unit", + example="The cat doesn't understand what KPI means" + ) + + +class GroupClips(BaseModel): + """Video group - Visual material organization""" + group_id: str = Field( + ..., + description="Unique identifier for the group", + example="group_0001" + ) + summary: str = Field( + ..., + description="Description of the group's visual style, emotional tone, or editing intent", + example="Start with the calmest, most healing shots to establish the mood." + ) + clip_ids: List[str] = Field( + ..., + description="List of video clip IDs used in this group, arranged in playback order", + example=["clip_0003", "clip_0002"] + ) + + +class GroupScript(BaseModel): + """Group script content""" + group_id: str = Field( + ..., + description="Unique identifier for the group", + example="group_0001" + ) + raw_text: str = Field( + ..., + description="original script content for this group", + example="The cat doesn't understand what KPI means, the cat only knows the sun is shining today" + ) + subtitle_units: List = Field( + ..., + description="List of subtitle segmentation units for precise control of subtitle display rhythm" + ) + + +class Voiceover(BaseModel): + """Single voiceover/narration item""" + group_id: str = Field(..., description="Group ID, e.g., group_0001") + voiceover_id: str = Field(..., description="Voiceover ID, e.g., voiceover_0001") + path: str = Field(..., description="Voiceover file path") + duration: int = Field(..., description="Voiceover duration (milliseconds)", gt=0) + + +class BGM(BaseModel): + """Background music""" + bgm_id: str = Field(..., description="BGM ID, e.g., bgm_0003") + path: str = Field(..., description="BGM file path") + duration: int = Field(..., description="BGM duration (milliseconds)", gt=0) + bpm: float = Field(..., description="Beats per minute", gt=0) + beats: List[int] = Field(default_factory=list, description="List of beat timestamps (milliseconds)") + + +class TimeWindow(BaseModel): + start: int = Field(..., description="Start time (milliseconds)") + end: int = Field(..., description="End time (milliseconds)") + + +class AudioMix(BaseModel): + gain_db: float = Field(default=0.0, description="Gain in decibels") + ducking: Optional[Any] = Field(default=None, description="Ducking effect configuration") + + +class ClipTrack(BaseModel): + clip_id: str + source_window: TimeWindow + timeline_window: TimeWindow + + +class BgmTrack(BaseModel): + bgm_id: str + timeline_window: TimeWindow + mix: AudioMix + + +class SubtitleTrack(BaseModel): + text: str + timeline_window: TimeWindow + + +class VoiceoverTrack(BaseModel): + media_id: str + timeline_window: TimeWindow + + +class TimelineTracks(BaseModel): + video: List[ClipTrack] = Field(default_factory=list) + subtitles: List[SubtitleTrack] = Field(default_factory=list) + voiceover: List[VoiceoverTrack] = Field(default_factory=list) + bgm: List[BgmTrack] = Field(default_factory=list) + + +class BaseInput(BaseModel): + mode: Literal["auto", "skip", "default"] = Field( + default="auto", + description="auto: Automatic mode; skip: Skip mode; default: Default mode" + ) + + +class LoadMediaInput(BaseInput): + ... + +class SearchMediaInput(BaseInput): + mode: Literal["auto", "skip", "default"] = Field( + default="auto", + description="auto: Automatically search media from pexels; skip: skip search; default: skip search" + ) + photo_number: Annotated[int, Field(default=0, description="The number of images the user wants to obtain")] + video_number: Annotated[int, Field(default=5, description="The number of videos the user wants to obtain")] + search_keyword: Annotated[str, Field(default="scenery", description="Keyword of the media the user wants to obtain. Only one keyword is allowed; multiple keywords are not permitted.")] + orientation: Literal["landscape", "portrait"] = Field( + default="landscape", + description="landscape: The screen is wider horizontally and narrower vertically, making it suitable for computer screens, landscape images, etc;portrait: The screen is higher vertically and narrower horizontally, making it suitable for mobile browsing and close-up shots of people." + ) + min_video_duration: Annotated[int, Field(default=1, description="The shortest duration of footage requested by the user in seconds.")] + max_video_duration: Annotated[int, Field(default=30, description="The longest duration of footage requested by the user in seconds.")] + +class LoadMediaOutput(BaseModel): + media: List[Media] = Field( + default_factory=list, + description="List of media" + ) + + +class SplitShotsInput(BaseInput): + mode: Literal["auto", "skip", "default"] = Field( + default="auto", + description="auto: Automatically segment shots based on scene changes, treat images as single shots; skip: Do not segment shots; default: Use default segmentation method" + ) + min_shot_duration: Annotated[int, Field(default=1000, description="Segmented shots must not be shorter than this duration (unit: milliseconds)")] + max_shot_duration: Annotated[int, Field(default=10000, description="If a single shot exceeds this duration, force segmentation (unit: milliseconds)")] + +class SplitShotsOutput(BaseModel): + clip_captions: List[Clip] = Field(default_factory=list, description="List of clips after splitting shots") + overall: Dict[str, str] + + +class UnderstandClipsInput(BaseModel): + mode: Literal["auto", "skip", "default"] = Field( + default="auto", + description="auto: Generate descriptions based on media content; skip: Do not generate descriptions; default: Use default description generation method" + ) + +class UnderstandClipsOutput(BaseModel): + clip_captions: List[Clip] = Field(default_factory=list, description="List of clips after understanding clips") + overall: Dict[str, str] + +class FilterClipsInput(BaseModel): + mode: Literal["auto", "skip", "default"] = Field( + default="auto", + description="auto: Filter clips based on user requirements; skip: Skip filtering; default: Use default filtering method" + ) + user_request: Annotated[str, Field(default="", description="User's requirements for clip filtering; if none provided, formulate one based on media materials and other editing requirements.")] = "" + +class FilterClipsOutput(BaseModel): + clip_captions: List[Clip] = Field(default_factory=list, description="List of clips") + overall: Dict[str, str] + overall: Dict[str, str] + + +class GroupClipsInput(BaseModel): + mode: Literal["auto", "skip", "default"] = Field( + default="auto", + description="auto: Organize clips in a logical order based on narrative flow of media content and user's sequencing requirements; skip: Skip sorting; default: Use default ordering method" + ) + user_request: Annotated[str, Field(default="", description="User's requirements for media organization order; if none provided, arrange in a logical narrative sequence following standard conventions.")] + +class GroupClipsOutput(BaseModel): + groups: List[GroupClips] = Field(default_factory=list, description="List of clips") + + +class GenerateScriptInput(BaseModel): + mode: Literal["auto", "skip", "default"] = Field( + default="auto", + description="auto: Generate appropriate script based on media content and user's script requirements; skip: Skip, do not add subtitles; default: Use default script" + ) + user_request: Annotated[str, Field(default="", description="User's requirements for the script.")] + custom_script: Dict[str, Any] = Field( + default={}, + description="If user has specific character-level editing requirements for script/title, pass the edited custom script and title through this parameter. Format should be based on the original script generation output format but with the subtitle_units field removed. In this case, mode must use `auto`, other modes are prohibited" + ) + +class GenerateScriptOutput(BaseModel): + group_scripts: List[GroupScript] + title: Optional[str] + + +class GenerateVoiceoverInput(BaseInput): + mode: Literal["auto", "skip", "default"] = Field( + default="auto", + description="auto: Generate appropriate voiceover based on media content and user's voice requirements; skip: Skip voiceover; default: Use default voiceover" + ) + user_request: Annotated[str, Field(default="", description="User's requirements for voiceover.")] + +class RecommendScriptTemplateInput(BaseInput): + mode: Literal["auto", "skip", "default"] = Field( + default="auto", + description="auto: Select an appropriate copywriting template based on the material content and user's requirements for voiceover style; skip: Skip;" + ) + user_request: Annotated[str, Field(default="", description="User's specific requirements for the script style.")] + filter_include: Annotated[ + Dict[str, List[str]], + Field( + description=( + "Positive filter conditions. Multiple dimensions are combined with AND, " + "multiple values within the same dimension are combined with OR.\n" + "Supported dimensions:\n" + "- tags: category, one or more of " + "[Life, Food, Beauty, Entertainment, Travel, Tech, Business, Vehicle, Health, Family, Pets, Knowledge]" + ) + ) + ] = {} + filter_exclude: Annotated[ + Dict[str, List[Union[str]]], + Field( + description=( + "Negative filter conditions. Items matching these conditions will be excluded. " + "The semantics are the same as filter_include. " + "Supported dimensions: tags, id." + ) + ) + ] = {} + + +class GenerateVoiceoverOutput(BaseModel): + voiceover: List[Voiceover] = Field(default_factory=list, description="Voiceover list") + + +class SelectBGMInput(BaseInput): + mode: Literal["auto", "skip", "default"] = Field( + default="auto", + description="auto: Select appropriate music based on media content and user's music requirements; skip: Do not use music; default: Use default music" + ) + user_request: Annotated[str, Field(default="", description="User's requirements for background music.")] + filter_include: Annotated[ + Dict[str, List[Union[str, int]]], + Field( + description=( + "Positive filter conditions. Multiple dimensions are combined with AND, " + "multiple values within the same dimension are combined with OR.\n" + "Supported dimensions:\n" + "- mood: music emotion, one or more of " + "[Dynamic, Chill, Happy, Sorrow, Romantic, Calm, Excited, Healing, Inspirational]\n" + "- scene: usage scene, one or more of " + "[Vlog, Travel, Relaxing, Emotion, Transition, Outdoor, Cafe, Evening, Scenery, Food, Date, Club]\n" + "- genre: music genre, one or more of " + "[Pop, BGM, Electronic, R&B/Soul, Hip Hop/Rap, Rock, Jazz, Folk, Classical, Chinese Style]\n" + "- lang: lyric language, one or more of [bgm, en, zh, ko, ja]\n" + "- id: specific music ids (int)" + ) + ) + ] = {} + filter_exclude: Annotated[ + Dict[str, List[Union[str, int]]], + Field( + description=( + "Negative filter conditions. Items matching these conditions will be excluded. " + "The semantics are the same as filter_include. " + "Supported dimensions: mood, scene, genre, lang, id." + ) + ) + ] = {} + +class SelectBGMOutput(BaseModel): + bgm: List[BGM] = Field(default_factory=list, description="BGM list") + + +class RecommendTransitionInput(BaseInput): + mode: Literal["auto", "skip", "default"] = Field( + default="auto", + description="auto: add fade in and fade out transitions at beginning and end; skip: Do not use transitions; default: Use default transitions", + ) + duration: Annotated[int, Field(default=1000, description="Duration of the transition in milliseconds")] + +class RecommendTransitionOutput(BaseInput): + ... + + +class RecommendTextInput(BaseInput): + mode: Literal["auto", "skip", "default"] = Field( + default="auto", + description="auto: Select appropriate font style and color based on user's subtitle font style requirements; default: Use default font", + ) + user_request: Annotated[str, Field(default="", description="User's requirements for font style")] + filter_include: Annotated[ + Dict[str, List[Union[str, int]]], + Field( + description=( + "Positive filter conditions. Multiple dimensions are combined with AND, " + "multiple values within the same dimension are combined with OR.\n" + "Supported dimensions:\n" + "- class: Font type, one or more" + "[Creative, Handwriting, Calligraphy, Basic]\n" + ) + ) + ] = {} + +class RecommendTextOutput(BaseInput): + ... + +class PlanTimelineInput(BaseInput): + use_beats: Annotated[bool, Field(default=True, description="Whether clip transitions should sync with BGM beats")] + +class PlanTimelineOutput(BaseModel): + tracks: List[TimelineTracks] = Field(default_factory=list, description="Timeline track collection") + +class RenderVideoInput(BaseInput): + aspect_ratio: Annotated[str | None, Field( + default=None, + description="When explicitly specified, forces the canvas to one of 16:9, 4:3, 1:1, 3:4, 9:16. If unset, the system automatically infers the most suitable aspect ratio." + )] + output_max_dimension_px: Annotated[int | None, Field( + default=None, + description="Maximum output size in pixels (longest side); defaults to 1080 and works with the aspect ratio." + )] + clip_compose_mode: Annotated[str, Field( + default="padding", + description="" \ + "How to fit media into the canvas: " \ + "'padding' keeps aspect ratio and fills empty areas with a solid color; " \ + "'crop' center-crops media to match the canvas aspect ratio." + )] + bg_color: Annotated[Tuple[int] | List[int] | None, Field( + default=(0, 0, 0), + description="Background color for canvas padding, specified as an (R, G, B) tuple (no alpha channel)." + )] + crf: Annotated[int, Field( + default=23, + description="CRF value (10–30), lower = better quality, larger file" + )] + + # font parameters + font_color: Annotated[Tuple[int, int, int, int], Field( + default=(255, 255, 255, 255), + description="Font color, RGBA format (R, G, B, A), values range 0-255") + ] + font_size: Annotated[int, Field( + default=40, + description="Font size in pixels. Recommended range: 28–120." + )] + margin_bottom: Annotated[int, Field( + default=270, + description="Bottom margin for subtitles in pixels. Defaults to 80; valid range: 40–1040." + )] + stroke_width: Annotated[int, Field( + default=2, + description="Text stroke width (px), typically 0–8" + )] + stroke_color: Annotated[Tuple[int, int, int, int], Field( + default=(0, 0, 0, 255), + description="Text stroke color in RGBA format", + )] + + # audio + bgm_volume_scale: Annotated[float, Field( + default=0.25, + description="Background music volume multiplier, range 0.0–3.0 (1.0 = default volume)" + )] + tts_volume_scale: Annotated[float, Field( + default=2.0, + description="TTS volume multiplier, range 0.0–3.0 (1.0 = default volume)" + )] + include_video_audio: Annotated[bool, Field( + default=False, + description="Whether to include the original video audio track" + )] + diff --git a/src/open_storyline/nodes/node_state.py b/src/open_storyline/nodes/node_state.py new file mode 100644 index 0000000000000000000000000000000000000000..e4b27583d3daeb5668ae4dfb31150df5a07921ec --- /dev/null +++ b/src/open_storyline/nodes/node_state.py @@ -0,0 +1,17 @@ +from dataclasses import dataclass + +from open_storyline.mcp.sampling_requester import SamplingLLMClient +from open_storyline.nodes.node_summary import NodeSummary + +from mcp.server.fastmcp import Context +from mcp.server.session import ServerSession + +@dataclass +class NodeState: + """Node execution state""" + session_id: str + artifact_id: str + lang: str + node_summary: NodeSummary + llm: SamplingLLMClient + mcp_ctx: Context[ServerSession, object] diff --git a/src/open_storyline/nodes/node_summary.py b/src/open_storyline/nodes/node_summary.py new file mode 100644 index 0000000000000000000000000000000000000000..44ff3cd781f40050faf0d181675c72f3993fd318 --- /dev/null +++ b/src/open_storyline/nodes/node_summary.py @@ -0,0 +1,236 @@ +from dataclasses import dataclass, field +from typing import Any, Dict, List, Tuple, Optional +import logging +from datetime import datetime + +from open_storyline.utils.logging import get_logger + + +@dataclass +class LogEntry: + """Single log entry""" + level: str + message: str + timestamp: str + artifact_id: Optional[str] = None + extra_data: Dict[str, Any] = field(default_factory=dict) + +@dataclass +class NodeSummary: + """ + Node Execution Status Summary - Reuses existing logger module + + Features: + 1. ERROR - Error messages for LLM + 2. WARNING - Warning messages for LLM + 3. DEBUG - Debug information for developers + 4. INFO_LLM - Detailed information for LLM + 5. INFO_USER - Brief information for users + + Capabilities: + - Reuses get_logger() configuration + - Hierarchical log storage and extraction + - Colored console output + - Log compression functionality + - Supports artifact tracking + """ + ERROR: str = "ERROR" + DEBUG: str = "DEBUG" + WARNING: str = "WARNING" + INFO_LLM: str = "INFO_LLM" + INFO_USER: str = "INFO_USER" + LOGGER_LEVELS: Tuple[str, ...] = (ERROR, DEBUG, WARNING, INFO_LLM, INFO_USER) + + + # Log storage + log_error: List[LogEntry] = field(default_factory=list) + log_warn: List[LogEntry] = field(default_factory=list) + log_info_llm: List[LogEntry] = field(default_factory=list) + log_info_user: List[LogEntry] = field(default_factory=list) + log_debug: List[LogEntry] = field(default_factory=list) + + # Artifact mapping + artifact_warnings: Dict[str, List[str]] = field(default_factory=dict) + artifact_errors: Dict[str, List[str]] = field(default_factory=dict) + + # Configuration options + logger_name: Optional[str] = field(default=None) + auto_console: bool = field(default=True) # Auto output to console + summary_levels: Optional[List[str]] = field(default=None) + + # Internal state + _logger: Optional[logging.Logger] = field(default=None, init=False, repr=False) + + def __post_init__(self): + """Initialize logger - reuses get_logger""" + if self.logger_name is None: + self.logger_name = "NodeSummary" + self._logger = get_logger(self.logger_name) + if self.summary_levels is None: + self.summary_levels = [self.ERROR, self.WARNING, self.INFO_LLM, self.INFO_USER] + + def _log_to_console(self, level: int, message: str, artifact_id: Optional[str] = None): + """Output to console (using configured logger)""" + if not self.auto_console: + return + + prefix = f"[ARTIFACT:{artifact_id}] " if artifact_id else "" + self._logger.log(level, f"{prefix}{message}") + + def add_error(self, message: str, artifact_id: Optional[str] = None, **kwargs: Any): + """Log error messages - for LLM""" + entry = LogEntry( + level=self.ERROR, + message=message, + timestamp=datetime.now().isoformat(), + artifact_id=artifact_id, + extra_data=kwargs + ) + self.log_error.append(entry) + + if artifact_id: + self.artifact_errors.setdefault(artifact_id, []).append(message) + + self._log_to_console(logging.ERROR, message, artifact_id) + + def add_warning(self, message: str, artifact_id: Optional[str] = None, **kwargs: Any): + """Log warning messages - for LLM""" + entry = LogEntry( + level="WARNING", + message=message, + timestamp=datetime.now().isoformat(), + artifact_id=artifact_id, + extra_data=kwargs + ) + self.log_warn.append(entry) + + if artifact_id: + self.artifact_warnings.setdefault(artifact_id, []).append(message) + + self._log_to_console(logging.WARNING, message, artifact_id) + + def info_for_llm(self, message: str, artifact_id: Optional[str] = None, **kwargs: Any): + """Log detailed information - for LLM""" + entry = LogEntry( + level="INFO_LLM", + message=message, + timestamp=datetime.now().isoformat(), + artifact_id=artifact_id, + extra_data=kwargs + ) + self.log_info_llm.append(entry) + self._log_to_console(logging.INFO, f"[{self.INFO_LLM}] {message}", artifact_id) + + def info_for_user(self, message: str, artifact_id: Optional[str] = None, **kwargs: Any): + """Log general information - for users""" + entry = LogEntry( + level="INFO_USER", + message=message, + timestamp=datetime.now().isoformat(), + artifact_id=artifact_id, + extra_data=kwargs + ) + self.log_info_user.append(entry) + self._log_to_console(logging.INFO, f"[{self.INFO_USER}] {message}", artifact_id) + + def debug_for_dev(self, message: str, artifact_id: Optional[str] = None, **kwargs: Any): + """Log debug information - for developers""" + entry = LogEntry( + level=self.DEBUG, + message=message, + timestamp=datetime.now().isoformat(), + artifact_id=artifact_id, + extra_data=kwargs + ) + self.log_debug.append(entry) + self._log_to_console(logging.DEBUG, f"[{self.DEBUG}] {message}", artifact_id) + + def get_logs_by_level( + self, + level:str , + compress_log: bool=False, # 暂时未实现 TODO + ) -> Dict[str,Any]: + self.all_logs = { + self.ERROR: self.log_error, + self.DEBUG: self.log_debug, + self.WARNING: self.log_warn, + self.INFO_LLM: self.log_info_llm, + self.INFO_USER: self.log_info_user + } + + selected_log = self.all_logs[level] + + return self._extract_log(selected_log) + + def _extract_log( + self, + log_content: List[LogEntry], + ) -> Dict[str,Any]: + """ + Extract log content into string format. + + Args: + log_content: List of log entries + + Returns: + Formatted log string with each log entry on a separate line + """ + if not log_content: + return {} + + log_lines: List[str] = [] + extra_data_list: List[Dict[str,Any]] = [] + for entry in log_content: + log_line = f"[{entry.timestamp}] {entry.message}" + + if entry.artifact_id: + log_line += f" [artifact_id: {entry.artifact_id}]" + + log_lines.append(log_line) + extra_data_list.append(entry.extra_data) + result: Dict[str,Any] = { + "log_lines": "\n".join(log_lines), + "extra_data_list": extra_data_list + } + return result + + def _get_preview_urls( + self, + extra_data_list: List[Dict[str,Any]], + ) -> List[str]: + preview_urls: List[str] = [] + for extra_data in extra_data_list: + preview_urls.extend([str(url) for url in extra_data.get('preview_urls', [])]) + return preview_urls + + def get_summary( + self, + artifact_id: str, + compress_log: bool=True, + **kwargs: Dict[str,Any], + ) -> Dict[str,Any]: + summary: Dict[str,Any] = {} + preview_urls: List[str] = [] + if self.summary_levels is None: + return summary + + for level in self.summary_levels: + summary_log = self.get_logs_by_level(level, compress_log) + log_lines = summary_log.get('log_lines', "") + extra_data_list = summary_log.get('extra_data_list', []) + preview_urls.extend(self._get_preview_urls(extra_data_list)) + summary[level] = log_lines + + summary['preview_urls'] = preview_urls + summary['artifact_id'] = artifact_id + return summary + + def clear(self): + """Clear all logs""" + self.log_error.clear() + self.log_warn.clear() + self.log_info_llm.clear() + self.log_info_user.clear() + self.log_debug.clear() + self.artifact_warnings.clear() + self.artifact_errors.clear() \ No newline at end of file diff --git a/src/open_storyline/skills/skills_io.py b/src/open_storyline/skills/skills_io.py new file mode 100644 index 0000000000000000000000000000000000000000..3be7059ea39139613289543852b70ccb473ff42c --- /dev/null +++ b/src/open_storyline/skills/skills_io.py @@ -0,0 +1,83 @@ + + +import aiofiles +from pathlib import Path +from skillkit import SkillManager +from skillkit.integrations.langchain import create_langchain_tools +from langchain.agents import create_agent +from langchain_openai import ChatOpenAI +from langchain.messages import HumanMessage + +async def load_skills( + skill_dir: str=".storyline/skills" +): + # Discover skills + manager = SkillManager(skill_dir=skill_dir) + await manager.adiscover() + + # Convert to LangChain tools + tools = create_langchain_tools(manager) + return tools + +async def dump_skills( + skill_name: str = '', + skill_dir: str = '', + skill_content: str = '', + **kwargs, +): + + clean_name = skill_name.strip() + if not clean_name: + return { + "status": "error", + "message": "skill_name cannot be empty" + } + + base_path = Path.cwd() + + # Project_Root + skill_dir + skill_name/ + target_path = base_path / skill_dir / f"cutskill_{clean_name}" + + # Fix name: SKILL.md + target_file_path = target_path / "SKILL.md" + + # Path Traversal Protection + try: + final_path = target_file_path.resolve() + if base_path not in final_path.parents: + return { + "status": "error", + "message": f"Security Alert: Writing to paths outside the project directory is forbidden: {final_path}" + } + except Exception as e: + return { + "status": "error", + "message": f"Path resolution error: {str(e)}" + } + + # Start write + try: + if not target_path.exists(): + target_path.mkdir(parents=True, exist_ok=True) + + async with aiofiles.open(final_path, mode='w', encoding='utf-8') as f: + await f.write(skill_content) + + return { + "status": "success", + "message": f"Skill '{clean_name}' successfully created.", + "dir_path": str(target_path), + "file_path": str(final_path), + "size_bytes": len(skill_content.encode('utf-8')) + } + + except PermissionError: + return { + "status": "error", + "message": f"Permission denied: Cannot write to directory {target_path}" + } + except Exception as e: + return { + "status": "error", + "message": f"Write operation failed: {str(e)}" + } diff --git a/src/open_storyline/storage/__init__.py b/src/open_storyline/storage/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/src/open_storyline/storage/agent_memory.py b/src/open_storyline/storage/agent_memory.py new file mode 100644 index 0000000000000000000000000000000000000000..70504ea52f34f2fe563396caf08b60c8d81027fc --- /dev/null +++ b/src/open_storyline/storage/agent_memory.py @@ -0,0 +1,152 @@ +from __future__ import annotations +from dataclasses import dataclass, asdict +from pathlib import Path +from typing import Any, List, Optional, Tuple +import json +import time + +from open_storyline.storage.file import FileCompressor +from open_storyline.utils.logging import get_logger + +logger = get_logger(__name__) + + +@dataclass +class ArtifactMeta: + session_id: str + artifact_id: str + node_id: str + path: str + summary: Optional[str] + created_at: float + +class ArtifactStore: + def __init__(self, artifacts_dir: str | Path, session_id: str) -> None: + self.artifacts_dir = Path(artifacts_dir) + self.session_id = session_id + self.blobs_dir = self.artifacts_dir / session_id + self.meta_path = self.blobs_dir / "meta.json" + self.blobs_dir.mkdir(parents=True, exist_ok=True) + if (not self.meta_path.exists()) or self.meta_path.stat().st_size == 0: + self._save_meta_list([]) + + def _load_meta_list(self) -> List[ArtifactMeta]: + if not self.meta_path.exists(): + return [] + with self.meta_path.open("r", encoding="utf-8") as f: + data = json.load(f) + return [ArtifactMeta(**item) for item in data] + + def _save_meta_list(self, metas: List[ArtifactMeta]): + with self.meta_path.open("w", encoding="utf-8") as f: + json.dump([asdict(m) for m in metas], f, ensure_ascii=False, indent=2) + + def _append_meta(self, meta: ArtifactMeta) -> None: + metas = self._load_meta_list() + metas.append(meta) + self._save_meta_list(metas) + + def _is_media_list(self, items) -> bool: + """Check if it is a valid media list""" + return isinstance(items, list) and all(isinstance(i, dict) for i in items) + + def _save_single_media(self, item: dict, store_dir: Path, artifact_id: str) -> None: + """Save a single media file""" + base64_data = item.pop('base64', None) + if not base64_data: + return + + file_path = store_dir / item.get('path', '') + logger.info(f"Saving media: artifact={artifact_id}, path={file_path}") + + FileCompressor.decompress_from_string(base64_data, file_path) + item['path'] = str(file_path) + + def _save_media(self, tool_execute_result, store_dir, artifact_id): + """Process tool execution results and save media files""" + if not isinstance(tool_execute_result, dict): + return + + for items in tool_execute_result.values(): + if self._is_media_list(items): + for item in items: + self._save_single_media(item, store_dir, artifact_id) + else: + self._save_media(items, store_dir, artifact_id) + + def save_result( + self, + session_id, + node_id, + data: Any, + search_media_dir: Optional[Path] = None + ) -> ArtifactMeta: + # Save intermediate results as JSON and include file information in meta.json for tracking + create_time = time.time() + artifact_id = data['artifact_id'] + summary = data['summary'] + tool_excute_result = data['tool_excute_result'] + store_dir = self.blobs_dir / node_id + file_path = store_dir / f"{artifact_id}.json" + + if not store_dir.exists(): + store_dir.mkdir(parents=True, exist_ok=True) + + if search_media_dir is None: + search_media_dir = store_dir + self._save_media(tool_excute_result, search_media_dir, artifact_id) + + save_data = { + "payload": tool_excute_result, + "session_id": session_id, + "artifact_id": artifact_id, + 'node_id': node_id, + 'create_time': create_time, + } + with file_path.open("w", encoding='utf-8') as f: + json.dump(save_data, f, ensure_ascii=False, indent=2) + logger.info(f"[Node `{node_id}`] save result to {file_path}") + + meta = ArtifactMeta( + session_id=session_id, + artifact_id=artifact_id, + node_id=node_id, + path=str(file_path), + summary=summary, + created_at=create_time, + ) + self._append_meta(meta) + return meta + + def load_result(self, artifact_id: str) -> Tuple[ArtifactMeta, Any]: + metas = self._load_meta_list() + meta = next((m for m in metas if m.artifact_id == artifact_id), None) + + if meta is None: + msg = f"artifact `{artifact_id}` not found" + return None, msg + + with open(meta.path, "r", encoding="utf-8") as f: + data = json.load(f) + return meta, data + + def generate_artifact_id(self, node_id): + unique_id = time.time() + artifact_id = f"{node_id}_{unique_id}" + return artifact_id + + def get_latest_meta( + self, + *, + node_id: str, + session_id: str, + ) -> Optional[ArtifactMeta]: + metas = self._load_meta_list() + candidates = [ + m for m in metas + if m.node_id == node_id + and m.session_id == session_id + ] + if not candidates: + return None + return max(candidates, key=lambda m: m.created_at) \ No newline at end of file diff --git a/src/open_storyline/storage/file.py b/src/open_storyline/storage/file.py new file mode 100644 index 0000000000000000000000000000000000000000..e080fbba1851d8cfcf30044057c75d8f04cb1060 --- /dev/null +++ b/src/open_storyline/storage/file.py @@ -0,0 +1,145 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +import gzip +import base64 +import zlib +import json +from pathlib import Path +from dataclasses import dataclass, asdict +from typing import Union, Optional +import hashlib + +@dataclass +class CompressedFile: + """Data class for compressed file information""" + filename: str + original_size: int + compressed_size: int + compression_ratio: str + method: str + md5: str + base64: str + +class FileCompressor: + """File compression and encoding utility class""" + + @staticmethod + def calculate_md5(data: bytes) -> str: + """Calculate MD5 hash of data""" + return hashlib.md5(data).hexdigest() + + @staticmethod + def compress_and_encode( + file_path: Union[str, Path], + method: str = 'gzip' + ) -> CompressedFile: + """ + Compresses a file and encodes it in Base64. + :param file_path: Path to the file. + :param method: Compression method ('gzip' or 'zlib'). + :return: A CompressedFile object containing the encoded data and metadata. + """ + file_path = Path(file_path) + + if not file_path.exists(): + raise FileNotFoundError(f"File not found: {file_path}") + + with open(file_path, 'rb') as f: + original_data = f.read() + + original_md5 = hashlib.md5(original_data).hexdigest() + original_size = len(original_data) + + if method == 'gzip': + compressed_data = gzip.compress( + original_data, + ) + elif method == 'zlib': + compressed_data = zlib.compress( + original_data, + ) + else: + raise ValueError(f"Unsupported compression method: {method}") + + compressed_size = len(compressed_data) + + encoded_data = base64.b64encode(compressed_data).decode('utf-8') + + return CompressedFile( + filename=file_path.name, + original_size=original_size, + compressed_size=compressed_size, + compression_ratio=f"{(1 - compressed_size/original_size)*100:.2f}%", + method=method, + md5=original_md5, + base64=encoded_data + ) + + @staticmethod + def decode_and_decompress( + encoded_file: CompressedFile, + output_path: Optional[Union[str, Path]] = None + ) -> bytes: + + compressed_data = base64.b64decode(encoded_file.base64) + + method = encoded_file.method + if method == 'gzip': + original_data = gzip.decompress(compressed_data) + elif method == 'zlib': + original_data = zlib.decompress(compressed_data) + else: + raise ValueError(f"Unsupported compression method: {method}") + + decoded_md5 = hashlib.md5(original_data).hexdigest() + if decoded_md5 != encoded_file.md5: + raise ValueError("MD5 checksum verification failed — the file may be corrupted.") + + if output_path: + output_path = Path(output_path) + output_path.parent.mkdir(parents=True, exist_ok=True) + with open(output_path, 'wb') as f: + f.write(original_data) + + return original_data + + @staticmethod + def save_encoded_to_json(encoded_file: CompressedFile, json_path: Union[str, Path]): + json_path = Path(json_path) + json_path.parent.mkdir(parents=True, exist_ok=True) + + with open(json_path, 'w', encoding='utf-8') as f: + json.dump(asdict(encoded_file), f, indent=2, ensure_ascii=False) + + @staticmethod + def load_encoded_from_json(json_path: Union[str, Path]) -> CompressedFile: + json_path = Path(json_path) + + if not json_path.exists(): + raise FileNotFoundError(f"JSON file not found: {json_path}") + + with open(json_path, 'r', encoding='utf-8') as f: + return CompressedFile(**json.load(f)) + + @staticmethod + def decompress_from_string( + encoded_string: str, + output_path: Union[str, Path], + method: str = 'gzip' + ) -> bytes: + + compressed_data = base64.b64decode(encoded_string) + + if method == 'gzip': + original_data = gzip.decompress(compressed_data) + elif method == 'zlib': + original_data = zlib.decompress(compressed_data) + else: + raise ValueError(f"Unsupported compression method: {method}") + + output_path = Path(output_path) + output_path.parent.mkdir(parents=True, exist_ok=True) + with open(output_path, 'wb') as f: + f.write(original_data) + + return original_data \ No newline at end of file diff --git a/src/open_storyline/storage/session_manager.py b/src/open_storyline/storage/session_manager.py new file mode 100644 index 0000000000000000000000000000000000000000..40d3c3deb9abf10911cb05f20dda8ceb8e727151 --- /dev/null +++ b/src/open_storyline/storage/session_manager.py @@ -0,0 +1,167 @@ +import shutil +import uuid +import time +import threading +from pathlib import Path +from typing import Callable, Optional + +from open_storyline.utils.logging import get_logger +from src.open_storyline.storage.agent_memory import ArtifactStore + +logger = get_logger(__name__) + + +class SessionLifecycleManager: + """ + Lifecycle Manager + Responsibilities: + 1. Create and clean up artifacts directory + 2. Create and clean up .server_cache directory + 3. Produce ArtifactStore instances + """ + def __init__( + self, + artifacts_root: str | Path, + cache_root: str | Path, + max_items: int = 256, + retention_days: int = 3, + enable_cleanup: bool = False, + ): + self.artifacts_root = Path(artifacts_root) + self.cache_root = Path(cache_root) + self.max_items = max_items + self.retention_days = retention_days + self.enable_cleanup = enable_cleanup + + # Ensure project root directory exists + self.artifacts_root.mkdir(parents=True, exist_ok=True) + self.cache_root.mkdir(parents=True, exist_ok=True) + + # Concurrency control: prevent multiple cleanup threads from interfering with each other + self._cleanup_lock = threading.Lock() + self._is_cleaning = False + + def _safe_rmtree(self, path: Path): + """More robust directory deletion method""" + def onerror(func, path, exc_info): + import stat + import os + if not os.access(path, os.W_OK): + os.chmod(path, stat.S_IWUSR) + func(path) + else: + logger.warning(f"[Lifecycle] Failed to remove {path}: {exc_info[1]}") + + if path.is_dir(): + shutil.rmtree(path, onerror=onerror) + else: + path.unlink(missing_ok=True) + + def _cleanup_dir(self, target_dir: Path, exclude_name: str = None, filter_func: Callable[[Path], bool] = None): + """ + Cleanup strategy: remove expired items first, then enforce quantity limit + """ + if not target_dir.exists(): + return + + try: + # 1. Calculate expiration timestamp cutoff + now = time.time() + # 86400 second = 1 day + cutoff_time = now - (self.retention_days * 86400) + + valid_items = [] # 没过期且合法的 Session + expired_items = [] # 已经过期的 Session + + # 2. Iterate and check + for p in target_dir.iterdir(): + # (A) Filter check (is it a directory, is it a UUID) + if filter_func and not filter_func(p): + continue + + # (B) Protect currently in-use items (don't delete even if expired, to prevent running tasks from crashing) + if exclude_name and p.name == exclude_name: + continue + + # (C) Check last modification time + mtime = p.stat().st_mtime + if mtime < cutoff_time: + # Exceeded retention_days, add to expired list + expired_items.append(p) + else: + # Not yet expired, add to valid list + valid_items.append(p) + + # 3. Phase 1: Delete all expired items + for item in expired_items: + logger.info(f"[Lifecycle] Deleting expired item (> {self.retention_days} days): {item.name}") + self._safe_rmtree(item) + + # 4. Phase 2: If remaining items still exceed max_items, delete the oldest + if len(valid_items) > self.max_items: + # Sort by time (oldest -> newest) + valid_items.sort(key=lambda x: x.stat().st_mtime) + + num_to_delete = len(valid_items) - self.max_items + logger.info(f"[Lifecycle] Item count {len(valid_items)} > limit {self.max_items}. Deleting {num_to_delete} oldest.") + + for item in valid_items[:num_to_delete]: + logger.info(f"[Lifecycle] Deleting excess item: {item.name}") + self._safe_rmtree(item) + + except Exception as e: + logger.error(f"[Lifecycle] Error cleaning {target_dir}: {e}") + + def cleanup_expired_sessions(self, current_session_id: Optional[str] = None): + """ + Trigger cleanup for all managed directories + Use lock to ensure only one cleanup task runs at a time + """ + if not self.enable_cleanup: + return + + # Try acquiring the lock; if it fails (cleanup in progress), skip this round + # Non-blocking approach suitable for high-frequency calls + if not self._cleanup_lock.acquire(blocking=False): + return + + def artifact_filter(p: Path) -> bool: + return p.is_dir() and self._is_valid_session_id(p.name) + + try: + self._is_cleaning = True + # Clean up artifacts + self._cleanup_dir(self.artifacts_root, exclude_name=current_session_id, filter_func=artifact_filter) + # Clean up server_cache + self._cleanup_dir(self.cache_root, exclude_name=current_session_id, filter_func=artifact_filter) + finally: + self._is_cleaning = False + self._cleanup_lock.release() + + def _is_valid_session_id(self, name: str) -> bool: + # 1. Quick filter: length must be 32 characters + if len(name) != 32: + return False + + # 2. Try to parse as UUID + try: + val = uuid.UUID(name) + return val.hex == name and val.version == 4 + except (ValueError, AttributeError): + return False + + + + def get_artifact_store(self, session_id: str) -> ArtifactStore: + # 1. Trigger cleanup asynchronously + # Even if called concurrently here, the non-blocking lock inside cleanup_expired_sessions handles concurrency issues + if self.enable_cleanup: + threading.Thread( + target=self.cleanup_expired_sessions, + args=(session_id,), + daemon=True, + name=f"CleanupThread-{session_id}" + ).start() + + # 2. Return Store instance + return ArtifactStore(self.artifacts_root, session_id) \ No newline at end of file diff --git a/src/open_storyline/utils/__init__.py b/src/open_storyline/utils/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..0d0c3a0be8532ec607bdaa032ae102f3584d6d0d --- /dev/null +++ b/src/open_storyline/utils/__init__.py @@ -0,0 +1 @@ +# copy from storyline，之后需要替换 \ No newline at end of file diff --git a/src/open_storyline/utils/element_filter.py b/src/open_storyline/utils/element_filter.py new file mode 100644 index 0000000000000000000000000000000000000000..08db11252e0ba914fc8aa6b45f8b22e0c65f4651 --- /dev/null +++ b/src/open_storyline/utils/element_filter.py @@ -0,0 +1,122 @@ +import json +import random +from typing import Any, Dict, List, Optional, Union + + +FilterValue = Union[str, List[str]] +FilterDict = Dict[str, FilterValue] + + +class ElementFilter: + """ + Generic filter for structured element libraries (music, effects, stickers, etc.) + """ + + def __init__( + self, + library: Optional[List[Dict[str, Any]]] = None, + json_path: Optional[str] = None, + ): + self.library: List[Dict[str, Any]] = [] + + if library is not None: + self.library = library + elif json_path is not None: + self.update(json_path) + else: + raise ValueError("Either library or json_path must be provided") + + def update( + self, + json_path: Optional[str] = None, + library: Optional[List[Dict[str, Any]]] = None, + ) -> None: + """Reload or replace the element library.""" + if library is not None: + self.library = library + return + + if json_path is None: + raise ValueError("update() requires json_path or library") + + with open(json_path, "r", encoding="utf-8") as f: + data = json.load(f) + + if not isinstance(data, list): + raise ValueError("Library JSON must be a list of dicts") + + self.library = data + + def filter( + self, + candidates: Optional[List[Dict[str, Any]]] = None, + filter_include: Optional[FilterDict] = None, + filter_exclude: Optional[FilterDict] = None, + fallback_n: int = 10, + ) -> List[Dict[str, Any]]: + """ + Filter elements by include / exclude conditions. + + - candidates: candidates need to filter + - include: fields that must match + - exclude: fields that must NOT match + - fallback_n: random fallback size if result is empty + """ + candidates = candidates or self.library + include = filter_include or {} + exclude = filter_exclude or {} + + results = [] + + for item in candidates: + if not self._match_include(item, include): + continue + + if self._match_exclude(item, exclude): + continue + + results.append(item) + + if not results and fallback_n > 0: + return random.sample( + self.library, min(fallback_n, len(self.library)) + ) + + return results + + @staticmethod + def _normalize(value: Any) -> List[str]: + """Normalize scalar or list values into a list of strings.""" + if value is None: + return [] + if isinstance(value, list): + return [str(v) for v in value] + return [str(value)] + + def _match_include(self, item: Dict[str, Any], include: FilterDict) -> bool: + """All include conditions must be satisfied.""" + for key, expected in include.items(): + if key not in item: + return False + + item_values = set(self._normalize(item[key])) + expected_values = set(self._normalize(expected)) + + if not item_values & expected_values: + return False + + return True + + def _match_exclude(self, item: Dict[str, Any], exclude: FilterDict) -> bool: + """Any exclude condition matched will reject the item.""" + for key, forbidden in exclude.items(): + if key not in item: + continue + + item_values = set(self._normalize(item[key])) + forbidden_values = set(self._normalize(forbidden)) + + if item_values & forbidden_values: + return True + + return False \ No newline at end of file diff --git a/src/open_storyline/utils/emoji.py b/src/open_storyline/utils/emoji.py new file mode 100644 index 0000000000000000000000000000000000000000..55dae93ff7696e2e9e6d5d7544d22abfed1279c2 --- /dev/null +++ b/src/open_storyline/utils/emoji.py @@ -0,0 +1,66 @@ +import re +import json +import emoji +import os + +EMOJI_PATTERN = re.compile( + "[\U0001F600-\U0001F64F" + "\U0001F300-\U0001F5FF" + "\U0001F680-\U0001F6FF" + "\U0001F1E0-\U0001F1FF" + "\U00002700-\U000027BF" + "\U0001F900-\U0001F9FF" + "\U00002600-\U000026FF" + "\U0001F700-\U0001F77F" + "\U0001F780-\U0001F7FF" + "\U0001F800-\U0001F8FF" + "\U0001F0A0-\U0001F0FF" + "\U0001F201-\U0001F2FF" + "\U0001F300-\U0001F3F0" + "\U00002300-\U000023FF" + "\U0001F004" + "\U00002B06" + "\u200D" + "]+", flags=re.UNICODE +) +EMOJI_PATH = './resource/unicode_emojis.json' + +class EmojiManager: + def __init__(self): + if os.path.exists(EMOJI_PATH) is False: + emoji_unicode_pattern_list = [] + else: + with open(EMOJI_PATH, "r", encoding="utf-8") as f: + emoji_unicode_pattern_list = json.load(f) + emoji_unicode_pattern_list = sorted(emoji_unicode_pattern_list, key=len, reverse=True) + self.emoji_unicode_pattern_re = re.compile("|".join(re.escape(e) for e in emoji_unicode_pattern_list)) + + def remove_emoji(self, text): + text = self.emoji_unicode_pattern_re.sub('', text) + # Use wide-range Unicode regex + text = EMOJI_PATTERN.sub('', text) + + return text + + def is_all_emoji(self, text: str) -> bool: + text = text.replace(" ", "") + if not text: + return False + + # If the entire string is in the emoji table, return True directly + if self.emoji_unicode_pattern_re.fullmatch(text): + return True + + # Otherwise, check character by character + for ch in text: + if not (self.emoji_unicode_pattern_re.fullmatch(ch) or EMOJI_PATTERN.fullmatch(ch)): + return False + return True + + @staticmethod + def is_emoji(ch: str) -> bool: + if EMOJI_PATTERN.match(ch): + return True + if emoji.is_emoji(ch): + return True + return False diff --git a/src/open_storyline/utils/logging.py b/src/open_storyline/utils/logging.py new file mode 100644 index 0000000000000000000000000000000000000000..0cc5adc0959647645c5f79639a499715b5dc5bc2 --- /dev/null +++ b/src/open_storyline/utils/logging.py @@ -0,0 +1,213 @@ +import sys +import logging +import colorlog +import time +from pathlib import Path +from datetime import datetime +from typing import Optional, Union, Dict, Any, Callable +from logging.handlers import RotatingFileHandler +from functools import wraps, lru_cache +from contextlib import contextmanager +from proglog import ProgressBarLogger + +@contextmanager +def silence_logging(): + logging.disable() + try: + yield + finally: + logging.disable(logging.NOTSET) + +# Log format +LOG_FORMAT = "%(asctime)s - %(name)s - %(levelname)s - %(message)s (%(filename)s:%(lineno)d)" +# Log colors mapping +LOG_COLOR_MAP = { + "DEBUG": "cyan", + "INFO": "green", + "WARNING": "yellow", + "ERROR": "red", + "CRITICAL": "bold_red", +} +# Log levels mapping +LOG_LEVEL_MAP = { + "debug": logging.DEBUG, + "info": logging.INFO, + "warning": logging.WARNING, + "error": logging.ERROR, + "critical": logging.CRITICAL +} + + +@lru_cache(maxsize=128) +def get_logger( + name: Optional[str] = None, +) -> logging.Logger: + """Get a configured color logger + + Args: + name: Logger name + + Returns: + Logger instance + """ + # Get calling module name + if name is None: + frame = sys._getframe(1) + name = frame.f_globals.get("__name__", "__main__") + + # Logger config + level = "debug" + do_console = True + do_file = False + log_dir = "logs" + date_format = "%Y-%m-%d %H:%M:%S" + + # Create logger + logger = logging.getLogger(name) + + # Set logging level + level = LOG_LEVEL_MAP.get(level.lower(), logging.INFO) + logger.setLevel(level) + logger.propagate = False # Prevent propagation to root logger + logger.handlers.clear() # Clear existing handlers + + # Add console handler + if do_console: + console_handler = colorlog.StreamHandler() + console_handler.setLevel(level) + colored_formatter = colorlog.ColoredFormatter( + f"%(log_color)s{LOG_FORMAT}", + datefmt=date_format, + log_colors=LOG_COLOR_MAP + ) + console_handler.setFormatter(colored_formatter) + logger.addHandler(console_handler) + + # Add file handler + if do_file: + # Create log directory + log_path = Path(log_dir) + log_path.mkdir(parents=True, exist_ok=True) + + # Create log filename + module_name = name.split(".")[-1] + timestamp = datetime.now().strftime("%Y-%m-%d") + filename_template = "{timestamp}.log" + log_file = log_path / filename_template.format( + module=module_name, + timestamp=timestamp + ) + + # Setup file handler + file_handler = RotatingFileHandler( + log_file, + maxBytes=10 * 1024 * 1024, + backupCount=5, + encoding="utf-8" + ) + file_handler.setLevel(level) + file_formatter = logging.Formatter(LOG_FORMAT, datefmt=date_format) + file_handler.setFormatter(file_formatter) + logger.addHandler(file_handler) + + return logger + + +def log_exception(func=None, logger=None, level=logging.ERROR): + def decorator(fn): + nonlocal logger + if logger is None: + logger = get_logger(name=fn.__module__) + + @wraps(fn) + def wrapper(*args, **kwargs): + try: + return fn(*args, **kwargs) + except Exception as e: + logger.log(level, f"Exception in {fn.__name__}: {str(e)}", exc_info=True) + raise # Re-raise exception + return wrapper + + # Support direct @log_exception usage + if func is not None: + return decorator(func) + return decorator + + +def log_time(func=None, logger=None, level=logging.DEBUG): + def decorator(fn): + nonlocal logger + if logger is None: + logger = get_logger(name=fn.__module__) + + @wraps(fn) + def wrapper(*args, **kwargs): + start_time = time.perf_counter() + result = fn(*args, **kwargs) + elapsed_time = time.perf_counter() - start_time + logger.log(level, f"Function {fn.__name__} ellapsed: {elapsed_time:.3f}s") + return result + return wrapper + + if func is not None: + return decorator(func) + return decorator + +from proglog import TqdmProgressBarLogger + +class MCPMoviePyLogger(TqdmProgressBarLogger): + def __init__(self, report: Callable[[float, Optional[float], Optional[str]], None]): + super().__init__(logged_bars="all", leave_bars=False, print_messages=True) + self._report = report + self._last_ts = 0.0 + self._last_p = -1.0 + self._seen = set() + + def bars_callback(self, bar, attr, value, old_value=None): + super().bars_callback(bar, attr, value, old_value) + if bar not in ("frame_index", "t", "chunk"): + return + if attr != "index": + return + st = self.bars.get(bar) or {} + idx, tot = st.get("index"), st.get("total") + if idx is None or not tot: + return + p = float(idx) / float(tot) + p = max(0.0, min(1.0, p)) + + now = time.monotonic() + if p < 1.0 and (now - self._last_ts) < 0.2 and (p - self._last_p) < 0.002: + return + self._last_ts, self._last_p = now, p + + self._report(float(idx), float(tot), f"rendering {p*100:.1f}%") + +if __name__ == "__main__": + + # Create logger with configuration + logger = get_logger() + logger.debug("Debug message") + logger.info("Info message") + + # Test logging decorators + @log_exception + @log_time + def sample_function(x, y): + import time + time.sleep(0.1) + return x + y + + # Test function logging + result = sample_function(10, 20) + logger.info(f"Function result: {result}") + + # Test exception logging + @log_exception + def dumb_func(): + return 1 / 0 + + try: + dumb_func() + except ZeroDivisionError: + logger.info("Exception was logged") diff --git a/src/open_storyline/utils/media_handler.py b/src/open_storyline/utils/media_handler.py new file mode 100644 index 0000000000000000000000000000000000000000..da837c7b0f97ebae328b09d0e69ef2be95ae71cc --- /dev/null +++ b/src/open_storyline/utils/media_handler.py @@ -0,0 +1,30 @@ +import os +from pathlib import Path +from typing import Union + +_MEDIA_EXTS_IMG = {".png", ".jpg", ".jpeg", ".gif", ".bmp", ".webp"} +_MEDIA_EXTS_VID = {".mp4", ".mov", ".avi", ".mkv", ".webm", ".m4v"} + +def scan_media_dir(media_dir: Union[Path, str]) -> dict: + image_num, video_num = 0, 0 + media_dir = Path(media_dir) + media_dir.mkdir(parents=True, exist_ok=True) + + for path in media_dir.iterdir(): + name = path.name + if name.startswith("."): + continue + if not path.is_file(): + continue + + ext = path.suffix.lower() + + if ext in _MEDIA_EXTS_IMG: + image_num += 1 + elif ext in _MEDIA_EXTS_VID: + video_num += 1 + + return { + "image number in user's media library": image_num, + "video number in user's media library": video_num, + } diff --git a/src/open_storyline/utils/parse_json.py b/src/open_storyline/utils/parse_json.py new file mode 100644 index 0000000000000000000000000000000000000000..ef4043a2de6f27d44b1f88dbae2248f72a5a9fe2 --- /dev/null +++ b/src/open_storyline/utils/parse_json.py @@ -0,0 +1,200 @@ +import json +import re +from typing import Any, Dict, Optional, Iterable + +def try_parse_tool_call(text:str) -> Optional[Dict[str, Any]]: + """ + Return dict if text is a valid tool call JSON, otherwise return None + """ + try: + obj = parse_json_dict(text) + except: + return None + + if obj.get("action") != "call_tool": + return None + if "tool" not in obj: + return None + + args = obj.get("arguments", {}) + if args is not None and not isinstance(args, dict): + return None + + return obj + +# Support ```json ... ``` and ```jsonc ... ``` (can remove jsonc if needed) +_CODE_FENCE_RE = re.compile( + r"```(?:json|jsonc)\s*(.*?)\s*```", + flags=re.IGNORECASE | re.DOTALL, +) + + +def _strip_trailing_commas_once(s: str) -> str: + """ + Remove trailing commas before '}' or ']' in JSON text (single pass). + Note: Skips content inside strings, won't remove commas within strings. + """ + out = [] + in_str = False + escape = False + i = 0 + n = len(s) + + while i < n: + c = s[i] + + if in_str: + out.append(c) + if escape: + escape = False + elif c == "\\": + escape = True + elif c == '"': + in_str = False + i += 1 + continue + + # not in string + if c == '"': + in_str = True + out.append(c) + i += 1 + continue + + if c == ",": + # look ahead to next non-whitespace + j = i + 1 + while j < n and s[j] in " \t\r\n": + j += 1 + if j < n and s[j] in "}]": + # drop this comma + i += 1 + continue + + out.append(c) + i += 1 + + return "".join(out) + + +def _strip_trailing_commas(s: str, max_passes: int = 10) -> str: + """ + Remove extra commas before '}' or ']' in JSON text (single pass). + Note: String content is skipped, so commas inside strings won't be removed. + """ + for _ in range(max_passes): + s2 = _strip_trailing_commas_once(s) + if s2 == s: + return s2 + s = s2 + return s # best effort + + +def _extract_balanced_object(text: str, start: int) -> Optional[str]: + """ + Extract a balanced JSON object substring {...} starting from text[start] == '{'. + Correctly skips braces within strings. + """ + depth = 0 + in_str = False + escape = False + + for i in range(start, len(text)): + c = text[i] + + if in_str: + if escape: + escape = False + elif c == "\\": + escape = True + elif c == '"': + in_str = False + continue + + if c == '"': + in_str = True + continue + + if c == "{": + depth += 1 + elif c == "}": + depth -= 1 + if depth == 0: + return text[start : i + 1] + + return None + + +def _iter_fenced_json_blocks(text: str) -> Iterable[str]: + for m in _CODE_FENCE_RE.finditer(text): + block = m.group(1) + if block is not None: + yield block.strip() + + +def _iter_object_candidates(text: str) -> Iterable[str]: + """ + Enumerate all possible {...} substrings in arbitrary text (in order of appearance). + """ + for idx, ch in enumerate(text): + if ch == "{": + cand = _extract_balanced_object(text, idx) + if cand: + yield cand + + +def parse_json_dict(text: str) -> Dict[str, Any]: + """ + Parse a JSON object (dict) from arbitrary text. + + Supports: + 1) Markdown fenced JSON code blocks: ```json ... ``` + 2) JSON surrounded by extra text + 3) Removing trailing commas before '}' or ']' + + Args: + text: Input string to parse + Returns: + Parsed dictionary + Raises: + ValueError: Cannot find a valid JSON dict to parse + TypeError: Input text is not a string + """ + if not isinstance(text, str): + raise TypeError(f"text must be str, got {type(text).__name__}") + + # Try fenced block first, then try the entire text + search_spaces = list(_iter_fenced_json_blocks(text)) + search_spaces.append(text) + + last_err: Optional[Exception] = None + + for space in search_spaces: + # If starts with '{', try to extract a balanced object from the beginning first (to avoid trailing noise) + candidates = [] + stripped = space.lstrip().lstrip("\ufeff") # 顺便去 BOM + if stripped.startswith("{"): + first = _extract_balanced_object(stripped, 0) + if first: + candidates.append(first) + + # Also try objects appearing at any position in the text + candidates.extend(_iter_object_candidates(space)) + + # Deduplicate (avoid retrying same substrings) + seen = set() + for cand in candidates: + if cand in seen: + continue + seen.add(cand) + + cleaned = _strip_trailing_commas(cand).strip() + try: + obj = json.loads(cleaned) + if isinstance(obj, dict): + return obj + except Exception as e: + last_err = e + continue + + raise ValueError("No valid JSON object (dict) found in input") from last_err \ No newline at end of file diff --git a/src/open_storyline/utils/prompts.py b/src/open_storyline/utils/prompts.py new file mode 100644 index 0000000000000000000000000000000000000000..de90948890d75de4e05bde97f602894f37b7662a --- /dev/null +++ b/src/open_storyline/utils/prompts.py @@ -0,0 +1,100 @@ +from pathlib import Path +from typing import Dict, Any +import re + +PROMPTS_DIR = Path("prompts/tasks") + + +class PromptBuilder: + """Builder for fixed templates with dynamic inputs""" + + def __init__(self, prompts_dir: Path = PROMPTS_DIR): + self.prompts_dir = prompts_dir + self._cache: Dict[str, str] = {} + + def _load_template(self, task: str, role: str, lang: str) -> str: + """Load template file""" + cache_key = f"{task}:{role}:{lang}" + + if cache_key in self._cache: + return self._cache[cache_key] + + # prompts/tasks/filter_clips/zh/system.md + template_path = self.prompts_dir / task / lang / f"{role}.md" + + if not template_path.exists(): + raise FileNotFoundError(f"Template not found: {template_path}") + + content = template_path.read_text(encoding='utf-8') + self._cache[cache_key] = content + return content + + def render(self, task: str, role: str, lang: str = "zh", **variables: Any) -> str: + """Render single template""" + template = self._load_template(task, role, lang) + return re.sub(r"{{(.*?)}}", lambda m: str(variables[m.group(1)]), template) + + def build(self, task: str, lang: str = "zh", **user_vars: Any) -> Dict[str, str]: + """ + Build a complete prompt pair + + Args: + task: Task name, e.g., "filter_clips" + lang: Language, defaults to "zh" + **user_vars: User input variables (passed to user.md template) + + Returns: + {"system": "...", "user": "..."} + + Example: + builder.build( + "filter_clips", + clip_data="...", + requirements="Keep exciting clips" + ) + """ + return { + "system": self.render(task, "system", lang), + "user": self.render(task, "user", lang, **user_vars) + } + + +# Global singleton +_builder = PromptBuilder() + + +def get_prompt(name: str, lang: str = "zh", **kwargs:Any) -> str: + """ + 获取单个 prompt + + Args: + name: "task.role" 格式，如 "filter_clips.system" + lang: 语言 + **kwargs: 模板变量 + + Example: + get_prompt("filter_clips.system") + get_prompt("filter_clips.user", clip_data="...") + """ + parts = name.split(".") + if len(parts) != 2: + raise ValueError(f"Invalid format: '{name}', expected 'task/role'") + + task, role = parts + return _builder.render(task, role, lang, **kwargs) + + +def build_prompts(task: str, lang: str = "zh", **user_vars: Any) -> Dict[str, str]: + """ + Get a single prompt. + + Args: + name: Format "task.role", e.g., "filter_clips.system" + lang: Language + **kwargs: Template variables + + Example: + get_prompt("filter_clips.system") + get_prompt("filter_clips.user", clip_data="...") + """ + return _builder.build(task, lang, **user_vars) \ No newline at end of file diff --git a/src/open_storyline/utils/recall.py b/src/open_storyline/utils/recall.py new file mode 100644 index 0000000000000000000000000000000000000000..76add1f74ea4cf1190b50e1a5eceb63f6ac25c35 --- /dev/null +++ b/src/open_storyline/utils/recall.py @@ -0,0 +1,63 @@ +from langchain_huggingface import HuggingFaceEmbeddings +from langchain_core.documents import Document +from langchain_community.vectorstores.faiss import FAISS +import os + +class StorylineRecall: + @staticmethod + def build_vectorstore( + data: list[dict], + field: str = "description", + model_name: str = "./.storyline/models/all-MiniLM-L6-v2", + device: str = "cpu" + ): + """ + Build a FAISS vectorstore using a local HuggingFace embedding model. + + Args: + data: list of dicts + field: which text field to embed + model_name: HuggingFace model identifier + device: "cpu" or "cuda" if available + + Returns: + FAISS vectorstore + """ + if not os.path.exists(model_name): + model_name = "sentence-transformers/all-MiniLM-L6-v2" + + # Create embeddings using HF model + embeddings = HuggingFaceEmbeddings( + model_name=model_name, + model_kwargs={"device": device} + ) + + # Construct LangChain Documents + docs = [] + for item in data: + text = item.get(field, "") + if text: + docs.append(Document(page_content=text, metadata=item)) + + if not docs: + print(f"[RECALL - Build vectorstore] Cannot find field: {field}, return None.") + return None + # Build FAISS + vectorstore = FAISS.from_documents(docs, embeddings) + return vectorstore + + @staticmethod + def query_top_n(vectorstore, query: str, n: int = 32): + """ + Query the vectorstore and return top-N original dicts. + + Args: + vectorstore: FAISS + query: query string + n: number of results + + Returns: + list of original dict entries + """ + results = vectorstore.similarity_search(query, k=n) + return [doc.metadata for doc in results] \ No newline at end of file diff --git a/src/open_storyline/utils/register.py b/src/open_storyline/utils/register.py new file mode 100644 index 0000000000000000000000000000000000000000..86b6515631d646e78a2dbb7cf7e0109f0847c26f --- /dev/null +++ b/src/open_storyline/utils/register.py @@ -0,0 +1,73 @@ +# core/registry.py +import pkgutil +import importlib +from typing import Optional +import os + +class Registry: + def __init__(self): + self._items = {} + + + def register(self, name: Optional[str] = None, override: bool = False): + """ + Class decorator to register a class in the registry. + + Args: + name (str, optional): Custom registration name. Defaults to module_name.ClassName. + override (bool): If True, will replace an existing class with the same name. Defaults to False. + """ + def decorator(cls): + reg_name = name or f"{cls.__name__}" + if reg_name in self._items: + if override: + print(f"[Registry] {reg_name} already registered, override=True -> replacing") + else: + raise KeyError(f"[Registry] {reg_name} already registered, override=False") + + self._items[reg_name] = cls + print(f"[Registry] Registered: {reg_name}") + return cls + return decorator + + + def get(self, name: str, default=None): + """Get a registered class by name. Returns `default` if not found.""" + return self._items.get(name, default) + + + def list(self): + """Return a list of all registered names.""" + return list(self._items.keys()) + + + def __len__(self): + return len(self._items) + + + def clear(self): + """Clear all registered classes.""" + self._items.clear() + + + def scan_package(self, package_name: str): + """ + Scan a Python package and its subpackages, import modules to trigger @REGISTRY.register(). + + Args: + package_name (str): Name of the package, e.g., "nodes" + """ + package = importlib.import_module(package_name) + if not hasattr(package, "__path__"): + # Not a package, skip scanning + print(f"[Registry] {package_name} is not a package, skipping scan") + return + + + for finder, modname, ispkg in pkgutil.walk_packages(package.__path__, package.__name__ + "."): + importlib.import_module(modname) + print(f"[Registry] Scanned module: {modname}") + + +# Global registry instance +NODE_REGISTRY = Registry() \ No newline at end of file diff --git a/src/open_storyline/utils/util.py b/src/open_storyline/utils/util.py new file mode 100644 index 0000000000000000000000000000000000000000..1e151cf4ad40fb5967ea7c007c93ede68efd1c12 --- /dev/null +++ b/src/open_storyline/utils/util.py @@ -0,0 +1,47 @@ + +import subprocess, json +from PIL import Image, ExifTags + +def get_video_rotation(path): + + info = json.loads( + subprocess.check_output([ + "ffprobe", + "-v", "error", + "-select_streams", "v:0", + "-show_entries", "stream_side_data=rotation", + "-of", "json", + str(path), + ]) + ) + + return next( + (sd["rotation"] + for sd in info.get("streams", [{}])[0].get("side_data_list", []) + if "rotation" in sd), + 0, + ) + +def get_image_rotation(path: str) -> int: + """ + Get the rotation angle of an image based on EXIF Orientation. + Returns 0, 90, 180, or 270 degrees. + """ + try: + img = Image.open(path) + exif = img._getexif() + if not exif: + return 0 + + # Search for the key corresponding to the Orientation tag + orientation_key = next( + (k for k, v in ExifTags.TAGS.items() if v == "Orientation"), None + ) + if not orientation_key: + return 0 + orientation = exif.get(orientation_key, 1) + + # Convert to rotation Angle + return {3: 180, 6: 270, 8: 90}.get(orientation, 0) + except Exception: + return 0 \ No newline at end of file diff --git a/web/index.html b/web/index.html new file mode 100644 index 0000000000000000000000000000000000000000..2045c8f125f68c8fe500787017d30787c140600d --- /dev/null +++ b/web/index.html @@ -0,0 +1,466 @@ + + + + + + + + + OpenStoryline + + + + + + + + + + + + +
+ +
+
+ + + OpenStoryline + + v1.0.0 +
+ +
+
+ + + Github + + + + + Docs + + + + + Node Map + +
+
+ 中 + + + + EN +
+
+
+ + + +
+ + + + + + + + + + +
+ + + + +
+ +
+ +
+ + +
+ + + + +
+
+ + + + + + + + +
+ + + + + diff --git a/web/node_map/node_map.html b/web/node_map/node_map.html new file mode 100644 index 0000000000000000000000000000000000000000..aa607e8de1944e172453466d262a8f9cdfc0a391 --- /dev/null +++ b/web/node_map/node_map.html @@ -0,0 +1,1055 @@ + + + + + + Workflow Node Map + + + +
+
+
Workflow Node Map
+
Reads workflow.json and renders a draggable node map (minimal grayscale).
+
+ +
+ + +
+ +
+
Tip: drag nodes; click the header to expand/collapse.
+
+
+ +
+
+ +
+ +
+
+

Loading…

+

Reading JSON and rendering nodes…

+
+
+
+ + + + diff --git a/web/node_map/workflow.json b/web/node_map/workflow.json new file mode 100644 index 0000000000000000000000000000000000000000..407822d58aa41849a6d961a0c4ebe472ea61bb6e --- /dev/null +++ b/web/node_map/workflow.json @@ -0,0 +1,712 @@ +{ + "workflow_meta": { + "id": "video_editing_pipeline", + "name": "Video Editing Workflow", + "description": "Automated video editing pipeline node map, including pro/normal nodes" + }, + "nodes": [ + { + "id": "load_media", + "name": "Load Media", + "kind": "load_media", + "pro": false, + "description": "Loads and indexes input media. Entry point with no dependencies; required by all downstream operations.", + "dependencies": [], + "next_nodes": [ + "split_shots" + ], + "input_schema": [ + { + "name": "mode", + "type": "Enum", + "options": [ + "auto", + "skip", + "default" + ], + "description": "auto: Automatic mode; skip: Skip mode; default: Default mode", + "default": "auto" + } + ], + "output_schema": [ + { + "name": "media", + "type": "List[Media]", + "description": "List of media objects with metadata" + } + ] + }, + { + "id": "search_media", + "name": "Search Media", + "kind": "search_media", + "pro": false, + "description": "Search media from external sources (e.g., Pexels).", + "dependencies": [], + "next_nodes": [ + "load_media" + ], + "input_schema": [ + { + "name": "mode", + "type": "Enum", + "options": [ + "auto", + "skip", + "default" + ], + "description": "auto: Automatically search media from pexels; skip: skip search; default: skip search", + "default": "auto" + }, + { + "name": "search_keyword", + "type": "String", + "description": "Keywords of the media the user wants to obtain", + "default": "scenery" + }, + { + "name": "video_number", + "type": "Integer", + "description": "The number of videos the user wants to obtain", + "default": 5 + }, + { + "name": "photo_number", + "type": "Integer", + "description": "The number of images the user wants to obtain", + "default": 0 + }, + { + "name": "orientation", + "type": "Enum", + "options": [ + "landscape", + "portrait" + ], + "description": "landscape: wider horizontally; portrait: higher vertically", + "default": "landscape" + }, + { + "name": "min_video_duration", + "type": "Integer", + "description": "The shortest duration of footage requested by the user (seconds)", + "default": 1 + }, + { + "name": "max_video_duration", + "type": "Integer", + "description": "The longest duration of footage requested by the user (seconds)", + "default": 30 + } + ], + "output_schema": [ + { + "name": "media", + "type": "List[Media]", + "description": "Retrieved media objects" + } + ] + }, + { + "id": "split_shots", + "name": "Split Shots", + "kind": "split_shots", + "pro": false, + "description": "Automatically segment videos into shots based on scene changes; treat images as single shots.", + "dependencies": [ + "load_media" + ], + "next_nodes": [ + "understand_clips" + ], + "input_schema": [ + { + "name": "mode", + "type": "Enum", + "options": [ + "auto", + "skip", + "default" + ], + "description": "auto: Automatically segment shots based on scene changes, treat images as single shots; skip: Do not segment shots; default: Use default segmentation method", + "default": "auto" + }, + { + "name": "min_shot_duration", + "type": "Integer", + "description": "Segmented shots must not be shorter than this duration (milliseconds)", + "default": 1000 + }, + { + "name": "max_shot_duration", + "type": "Integer", + "description": "If a single shot exceeds this duration, force segmentation (milliseconds)", + "default": 10000 + } + ], + "output_schema": [ + { + "name": "clip_captions", + "type": "List[Clip]", + "description": "List of clips after splitting shots" + }, + { + "name": "overall", + "type": "Dict[String,String]", + "description": "Overall summary/metadata (key-value)" + } + ] + }, + { + "id": "understand_clips", + "name": "Understand Clips", + "kind": "understand_clips", + "pro": false, + "description": "Generate descriptions/captions for each clip based on media content.", + "dependencies": [ + "split_shots" + ], + "next_nodes": [ + "filter_clips", + "recommend_script_template" + ], + "input_schema": [ + { + "name": "mode", + "type": "Enum", + "options": [ + "auto", + "skip", + "default" + ], + "description": "auto: Generate descriptions based on media content; skip: Do not generate descriptions; default: Use default description generation method", + "default": "auto" + } + ], + "output_schema": [ + { + "name": "clip_captions", + "type": "List[Clip]", + "description": "List of clips after understanding clips" + }, + { + "name": "overall", + "type": "Dict[String,String]", + "description": "Overall summary/metadata (key-value)" + } + ] + }, + { + "id": "filter_clips", + "name": "Filter Clips", + "kind": "filter_clips", + "pro": false, + "description": "Filter clips based on user requirements.", + "dependencies": [ + "understand_clips" + ], + "next_nodes": [ + "group_clips" + ], + "input_schema": [ + { + "name": "mode", + "type": "Enum", + "options": [ + "auto", + "skip", + "default" + ], + "description": "auto: Filter clips based on user requirements; skip: Skip filtering; default: Use default filtering method", + "default": "auto" + }, + { + "name": "user_request", + "type": "String", + "description": "User's requirements for clip filtering; if none provided, formulate one based on materials and other editing requirements.", + "default": "" + } + ], + "output_schema": [ + { + "name": "clip_captions", + "type": "List[Clip]", + "description": "List of clips" + }, + { + "name": "overall", + "type": "Dict[String,String]", + "description": "Overall summary/metadata (key-value)" + } + ] + }, + { + "id": "group_clips", + "name": "Group Clips", + "kind": "group_clips", + "pro": false, + "description": "Organize clips into narrative groups and arrange playback order.", + "dependencies": [ + "filter_clips" + ], + "next_nodes": [ + "generate_script" + ], + "input_schema": [ + { + "name": "mode", + "type": "Enum", + "options": [ + "auto", + "skip", + "default" + ], + "description": "auto: Organize clips in a logical order based on narrative flow and user's sequencing requirements; skip: Skip sorting; default: Use default ordering method", + "default": "auto" + }, + { + "name": "user_request", + "type": "String", + "description": "User's requirements for media organization order; if none provided, arrange in a logical narrative sequence following standard conventions.", + "default": "" + } + ], + "output_schema": [ + { + "name": "groups", + "type": "List[GroupClips]", + "description": "List of clip groups with ordering and summaries" + } + ] + }, + { + "id": "generate_script", + "name": "Generate Script", + "kind": "generate_script", + "pro": false, + "description": "Generate script/subtitles for each group; supports passing a custom_script override (subtitle_units removed).", + "dependencies": [ + "group_clips" + ], + "next_nodes": [ + "generate_voiceover", + "recommend_text" + ], + "input_schema": [ + { + "name": "mode", + "type": "Enum", + "options": [ + "auto", + "skip", + "default" + ], + "description": "auto: Generate appropriate script based on media content and user's script requirements; skip: Skip, do not add subtitles; default: Use default script", + "default": "auto" + }, + { + "name": "user_request", + "type": "String", + "description": "User's requirements for the script.", + "default": "" + }, + { + "name": "custom_script", + "type": "Dict[String,Any]", + "description": "If user has specific character-level editing requirements for script/title, pass the edited custom script and title through this parameter. Format should be based on the original script generation output format but with the subtitle_units field removed. In this case, mode must use `auto`, other modes are prohibited.", + "default": {} + } + ], + "output_schema": [ + { + "name": "group_scripts", + "type": "List[GroupScript]", + "description": "Group-level script content including subtitle units" + }, + { + "name": "title", + "type": "Optional[String]", + "description": "Optional generated title" + } + ] + }, + { + "id": "recommend_script_template", + "name": "Recommend Script Template", + "kind": "recommend_script_template", + "pro": false, + "description": "Recommend/select a copywriting template based on material content and user's requirements. (Output schema not defined in the provided schema file.)", + "dependencies": [ + "understand_clips" + ], + "next_nodes": [], + "input_schema": [ + { + "name": "mode", + "type": "Enum", + "options": [ + "auto", + "skip", + "default" + ], + "description": "auto: Select an appropriate copywriting template based on the material content and user's requirements for voiceover style; skip: Skip;", + "default": "auto" + }, + { + "name": "user_request", + "type": "String", + "description": "User's specific requirements for the script style.", + "default": "" + }, + { + "name": "filter_include", + "type": "Dict[String,List[String]]", + "description": "Positive filter conditions. Multiple dimensions are combined with AND, multiple values within the same dimension are combined with OR. Supported dimensions: tags (one or more of [Life, Food, Beauty, Entertainment, Travel, Tech, Business, Vehicle, Health, Family, Pets, Knowledge]).", + "default": {} + }, + { + "name": "filter_exclude", + "type": "Dict[String,List[String]]", + "description": "Negative filter conditions. Items matching these conditions will be excluded. Supported dimensions: tags, id.", + "default": {} + } + ], + "output_schema": [] + }, + { + "id": "generate_voiceover", + "name": "Generate Voiceover", + "kind": "generate_voiceover", + "pro": false, + "description": "Generate voiceover/narration items (e.g., TTS) for each group.", + "dependencies": [ + "generate_script" + ], + "next_nodes": [ + "select_bgm", + "plan_timeline" + ], + "input_schema": [ + { + "name": "mode", + "type": "Enum", + "options": [ + "auto", + "skip", + "default" + ], + "description": "auto: Generate appropriate voiceover based on media content and user's voice requirements; skip: Skip voiceover; default: Use default voiceover", + "default": "auto" + }, + { + "name": "user_request", + "type": "String", + "description": "User's requirements for voiceover.", + "default": "" + } + ], + "output_schema": [ + { + "name": "voiceover", + "type": "List[Voiceover]", + "description": "Voiceover list" + } + ] + }, + { + "id": "select_bgm", + "name": "Select BGM", + "kind": "select_bgm", + "pro": false, + "description": "Select appropriate background music (BGM) based on material content and user requirements; supports include/exclude filters.", + "dependencies": [ + "generate_script" + ], + "next_nodes": [ + "recommend_transition", + "plan_timeline" + ], + "input_schema": [ + { + "name": "mode", + "type": "Enum", + "options": [ + "auto", + "skip", + "default" + ], + "description": "auto: Select appropriate music based on media content and user's music requirements; skip: Do not use music; default: Use default music", + "default": "auto" + }, + { + "name": "user_request", + "type": "String", + "description": "User's requirements for background music.", + "default": "" + }, + { + "name": "filter_include", + "type": "Dict[String,List[Union[String,Integer]]]", + "description": "Positive filter conditions. Supported dimensions: mood [Dynamic, Chill, Happy, Sorrow, Romantic, Calm, Excited, Healing, Inspirational]; scene [Vlog, Travel, Relaxing, Emotion, Transition, Outdoor, Cafe, Evening, Scenery, Food, Date, Club]; genre [Pop, BGM, Electronic, R&B/Soul, Hip Hop/Rap, Rock, Jazz, Folk, Classical, Chinese Style]; lang [bgm, en, zh, ko, ja]; id (int).", + "default": {} + }, + { + "name": "filter_exclude", + "type": "Dict[String,List[Union[String,Integer]]]", + "description": "Negative filter conditions. Items matching these conditions will be excluded. Supported dimensions: mood, scene, genre, lang, id.", + "default": {} + } + ], + "output_schema": [ + { + "name": "bgm", + "type": "List[BGM]", + "description": "BGM list" + } + ] + }, + { + "id": "recommend_transition", + "name": "Recommend Transition", + "kind": "recommend_transition", + "pro": false, + "description": "Recommend transitions (e.g., fade in/out) at beginning and end.", + "dependencies": [ + "select_bgm" + ], + "next_nodes": [ + "render_video" + ], + "input_schema": [ + { + "name": "mode", + "type": "Enum", + "options": [ + "auto", + "skip", + "default" + ], + "description": "auto: add fade in and fade out transitions at beginning and end; skip: Do not use transitions; default: Use default transitions", + "default": "auto" + }, + { + "name": "duration", + "type": "Integer", + "description": "Duration of the transition in milliseconds", + "default": 1000 + } + ], + "output_schema": [ + { + "name": "mode", + "type": "Enum", + "description": "auto / skip / default mode (output model is BaseInput)" + } + ] + }, + { + "id": "recommend_text", + "name": "Recommend Text", + "kind": "recommend_text", + "pro": false, + "description": "Recommend subtitle font style/color based on user's requirements; supports include filters.", + "dependencies": [ + "generate_script" + ], + "next_nodes": [ + "render_video" + ], + "input_schema": [ + { + "name": "mode", + "type": "Enum", + "options": [ + "auto", + "skip", + "default" + ], + "description": "auto: Select appropriate font style and color based on user's subtitle font style requirements; default: Use default font", + "default": "auto" + }, + { + "name": "user_request", + "type": "String", + "description": "User's requirements for font style", + "default": "" + }, + { + "name": "filter_include", + "type": "Dict[String,List[Union[String,Integer]]]", + "description": "Positive filter conditions. Supported dimensions: class [Creative, Handwriting, Calligraphy, Basic].", + "default": {} + } + ], + "output_schema": [ + { + "name": "mode", + "type": "Enum", + "description": "auto / skip / default mode (output model is BaseInput)" + } + ] + }, + { + "id": "plan_timeline", + "name": "Plan Timeline", + "kind": "plan_timeline", + "pro": false, + "description": "Plan timeline tracks for video, subtitles, voiceover, and BGM; optionally sync transitions with BGM beats.", + "dependencies": [ + "group_clips", + "generate_script", + "generate_voiceover", + "select_bgm" + ], + "next_nodes": [ + "render_video" + ], + "input_schema": [ + { + "name": "mode", + "type": "Enum", + "options": [ + "auto", + "skip", + "default" + ], + "description": "auto: Automatic mode; skip: Skip mode; default: Default mode", + "default": "auto" + }, + { + "name": "use_beats", + "type": "Boolean", + "description": "Whether clip transitions should sync with BGM beats", + "default": true + } + ], + "output_schema": [ + { + "name": "tracks", + "type": "List[TimelineTracks]", + "description": "Timeline track collection" + } + ] + }, + { + "id": "render_video", + "name": "Render Video", + "kind": "render_video", + "pro": false, + "description": "Render/export the final video with canvas, subtitle, audio, and encoding parameters. (Output schema not defined in the provided schema file.)", + "dependencies": [ + "plan_timeline" + ], + "next_nodes": [], + "input_schema": [ + { + "name": "mode", + "type": "Enum", + "options": [ + "auto", + "skip", + "default" + ], + "description": "auto: Automatic mode; skip: Skip mode; default: Default mode", + "default": "auto" + }, + { + "name": "aspect_ratio", + "type": "Optional[String]", + "description": "Canvas aspect ratio override: one of 16:9, 4:3, 1:1, 3:4, 9:16; auto if unset.", + "default": null + }, + { + "name": "output_max_dimension_px", + "type": "Optional[Integer]", + "description": "Maximum output size in pixels (longest side); defaults to 1080 and works with the aspect ratio.", + "default": null + }, + { + "name": "clip_compose_mode", + "type": "String", + "description": "How to fit media into the canvas: 'padding' keeps aspect ratio and fills empty areas; 'crop' center-crops media to match the canvas aspect ratio.", + "default": "crop" + }, + { + "name": "bg_color", + "type": "Tuple[Integer,Integer,Integer] | List[Integer] | Null", + "description": "Background color for canvas padding, specified as an (R, G, B) tuple (no alpha channel).", + "default": [ + 0, + 0, + 0 + ] + }, + { + "name": "crf", + "type": "Integer", + "description": "CRF value (10–30), lower = better quality, larger file", + "default": 23 + }, + { + "name": "font_color", + "type": "Tuple[Integer,Integer,Integer,Integer]", + "description": "Font color, RGBA format (R, G, B, A), values range 0-255", + "default": [ + 255, + 255, + 255, + 255 + ] + }, + { + "name": "font_size", + "type": "Integer", + "description": "Font size in pixels. Recommended range: 28–120.", + "default": 40 + }, + { + "name": "margin_bottom", + "type": "Integer", + "description": "Bottom margin for subtitles in pixels. Defaults to 80; valid range: 40–600.", + "default": 80 + }, + { + "name": "stroke_width", + "type": "Integer", + "description": "Text stroke width (px), typically 0–8", + "default": 2 + }, + { + "name": "stroke_color", + "type": "Tuple[Integer,Integer,Integer,Integer]", + "description": "Text stroke color in RGBA format", + "default": [ + 0, + 0, + 0, + 255 + ] + }, + { + "name": "bgm_volume_scale", + "type": "Float", + "description": "Background music volume multiplier, range 0.0–3.0 (1.0 = default volume)", + "default": 1.0 + }, + { + "name": "tts_volume_scale", + "type": "Float", + "description": "TTS volume multiplier, range 0.0–3.0 (1.0 = default volume)", + "default": 1.0 + }, + { + "name": "include_video_audio", + "type": "Boolean", + "description": "Whether to include the original video audio track", + "default": false + } + ], + "output_schema": [] + } + ] +} diff --git a/web/static/app.js b/web/static/app.js new file mode 100644 index 0000000000000000000000000000000000000000..4c160da28c17e2af2ad85452de5281d12151f6fd --- /dev/null +++ b/web/static/app.js @@ -0,0 +1,3661 @@ +// /static/app.js +const $ = (sel) => document.querySelector(sel); +const SIDEBAR_COLLAPSED_KEY = "openstoryline_sidebar_collapsed"; +const DEVBAR_COLLAPSED_KEY = "openstoryline_devbar_collapsed"; +const AUDIO_PREVIEW_MAX = 3; +const CUSTOM_MODEL_KEY = "__custom__"; + +// ========================================================= +// i18n (zh/en) + lang persistence +// ========================================================= +const __OS_LANG_STORAGE_KEY = "openstoryline_lang_v1"; + +const QUICK_PROMPTS = [ + { zh: "详细介绍一下你能做什么", en: "Please describe in detail what you can do." }, + { zh: "帮我找10个夏日海滩素材，剪一个欢快的旅行vlog", en: "Please help me find some summer beach footage and edit it into a 30-second travel vlog." }, + { zh: "我准备长期批量做同类视频，先帮我剪一条示范成片；之后把这套偏好总结成可复用的剪辑风格 Skill。", en: "I plan to produce similar videos in batches over a long period. First, help me edit a sample video; then, help me summarize this set of preferences into a reusable editing style skill."}, + { zh: "根据我的素材内容，仿照鲁迅文风生成文案。", en: "Based on my footage, please generate a Shakespearean-style video script."}, + { zh: "帮我找一些中国春节相关素材，筛选出最有年味的场景，选择喜庆的 BGM", en: "Please help me find some materials related to Chinese New Year, filter out the most festive scenes, and choose celebratory background music."}, +]; + +const __OS_I18N = { + zh: { + // topbar + "main.greeting": "🎬 你好，创作者", + "topbar.lang_title": "切换语言", + "topbar.lang_aria": "语言切换", + "topbar.lang_zh": "中", + "topbar.lang_en": "EN", + "topbar.link1": "github 链接", + "topbar.link2": "使用手册", + "topbar.node_map": "节点地图", + + // aria + "aria.sidebar": "侧边栏", + "aria.sidebar_scroll": "侧边栏滚动区", + "aria.sidebar_model_select": "对话模型选择", + "composer.placeholder": "提出任何剪辑需求（Enter 发送，shift + Enter 换行）", + "assistant.placeholder": "正在调用大模型中…", + "composer.quick_prompt": "插入提示语", + + // sidebar + "sidebar.toggle": "收起/展开侧边栏", + "sidebar.new_chat": "创建新对话", + "sidebar.model_label": "对话模型", + "sidebar.model_select_aria": "选择对话模型", + "sidebar.custom_model_box_aria": "自定义模型配置", + "sidebar.custom_model_title": "自定义模型", + "sidebar.custom_llm_subtitle": "LLM（对话/文案）", + "sidebar.custom_llm_model_ph": "模型名称，例如 deepseek-chat / gpt-4o-mini", + "sidebar.custom_llm_baseurl_ph": "Base URL，例如 https://api.xxx.com/v1", + "sidebar.custom_llm_apikey_ph": "API Key", + "sidebar.custom_vlm_subtitle": "VLM（素材理解）", + "sidebar.custom_vlm_model_ph": "模型名称，例如 qwen-vl-plus / gpt-4o", + "sidebar.custom_vlm_baseurl_ph": "Base URL，例如 https://api.xxx.com/v1", + "sidebar.custom_vlm_apikey_ph": "API Key", + "sidebar.custom_hint": "提示：API Key 仅用于本会话的服务端调用；页面与 Tool trace 会自动脱敏，不会显示明文。", + "sidebar.tts_box_aria": "TTS 服务配置", + "sidebar.tts_title": "TTS 配置", + "sidebar.tts_provider_select_aria": "选择 TTS 服务厂家", + "sidebar.tts_default": "使用默认配置", + "sidebar.tts_hint": "提示：字段留空将使用 config.toml 中的配置。", + "sidebar.tts_field_suffix": "（留空则使用服务器默认）", + "sidebar.use_custom_model": "使用自定义模型", + "sidebar.llm_label": "LLM 模型", + "sidebar.vlm_label": "VLM 模型", + "sidebar.llm_select_aria": "选择 LLM 模型", + "sidebar.vlm_select_aria": "选择 VLM 模型", + "sidebar.custom_llm_title": "LLM 自定义模型", + "sidebar.custom_vlm_title": "VLM 自定义模型", + "sidebar.custom_llm_box_aria": "LLM 自定义模型配置", + "sidebar.custom_vlm_box_aria": "VLM 自定义模型配置", + + "sidebar.pexels_box_aria": "Pexels API Key 配置", + "sidebar.pexels_title": "Pexels 配置", + "sidebar.pexels_mode_select_aria": "选择 Pexels Key 模式", + "sidebar.pexels_default": "使用默认配置", + "sidebar.pexels_custom": "使用自定义 key", + "sidebar.pexels_apikey_ph": "Pexels API Key", + "sidebar.pexels_hint": "提示：默认配置会优先使用 config.toml 的 search_media.pexels_api_key；为空时工具内部会从环境变量读取。", + + "sidebar.help.cta": "点击查看配置教程", + "sidebar.help.llm": "LLM 主要用于对话，在工具内部也被用来生成文案/分组/选择BGM等。", + "sidebar.help.vlm": "VLM 用于素材理解（图像/视频理解）。自定义时请确认模型支持多模态输入。", + "sidebar.help.pexels": "Pexels 用于搜索网络素材。免责声明：OpenStoryline 搜索的网络素材均来自Pexels，通过Pexels下载的素材仅用于体验Open-Storyline剪辑效果，不允许再分发或出售。我们只提供工具，所有通过本工具下载和使用的素材（如 Pexels 图像）都由用户自行通过 API 获取，我们不对用户生成的视频内容、素材的合法性或因使用本工具导致的任何版权/肖像权纠纷承担责任。使用时请遵循 Pexels 的许可协议。", + "sidebar.help.tts": "用于从文案生成配音。", + "sidebar.help.pexels_home_link": "点击进入 Pexels 官方网站", + "sidebar.help.pexels_terms_link": "查看 Pexels 用户协议", + + // common + "common.retry_after_suffix": "（{seconds}s后再试）", + + // toast + "toast.interrupt_failed": "打断失败：{msg}", + "toast.pending_limit": "待发送素材已达上限（{max} 个），请先发送/删除后再上传。", + "toast.pending_limit_partial": "最多还能上传 {remain} 个素材（上限 {max}）。将只上传前 {remain} 个。", + "toast.uploading": "正在上传素材中… {pct}%", + "toast.uploading_file": "正在上传素材（{i}/{n}）：{name}… {pct}%", + "toast.upload_failed": "上传失败：{msg}", + "toast.delete_failed": "删除失败：{msg}", + "toast.uploading_cannot_send": "素材正在上传中，上传完成后才能发送。", + "toast.uploading_interrupt_send": "素材正在上传中，暂时无法发送新消息。已为你打断当前回复；上传完成后再按 Enter 发送。", + + // tools + "tool.card.default_name": "工具调用", + "tool.card.fallback_name": "MCP 工具", + + "tool.preview.render_title": "成片预览", + "tool.preview.other_videos": "其它视频（点击预览）", + "tool.preview.videos": "视频（点击预览）", + "tool.preview.images": "图片（点击预览）", + "tool.preview.audio": "音频", + "tool.preview.listen": "试听", + "tool.preview.split_shots": "镜头切分结果（点击预览）", + + "tool.preview.btn_modal": "弹窗预览", + "tool.preview.btn_open": "打开", + + "tool.preview.more_items": "还有 {n} 个未展示", + "tool.preview.more_audios": "还有 {n} 个音频未展示", + + "tool.preview.label.audio": "音频 {i}", + "tool.preview.label.video": "视频 {i}", + "tool.preview.label.image": "图片 {i}", + "tool.preview.label.shot": "镜头 {i}", + + "preview.unsupported": "该类型暂不支持内嵌预览：", + "preview.open_download": "打开/下载", + }, + en: { + // topbar + "main.greeting": "🎬 Hi, creator", + "topbar.lang_title": "Switch language", + "topbar.lang_aria": "Language switch", + "topbar.lang_zh": "中", + "topbar.lang_en": "EN", + "topbar.link1": "github link", + "topbar.link2": "user guide", + "topbar.node_map": "node map", + + // aria + "aria.sidebar": "Sidebar", + "aria.sidebar_scroll": "Sidebar scroll area", + "aria.sidebar_model_select": "Chat model selector", + "composer.placeholder": "Make any editing requests (Enter to send, Shift + Enter for line break)", + "assistant.placeholder": "Calling the LLM…", + "composer.quick_prompt": "Insert a preset prompt", + + // sidebar + "sidebar.toggle": "Collapse/expand sidebar", + "sidebar.new_chat": "New chat", + "sidebar.model_label": "Chat model", + "sidebar.model_select_aria": "Select chat model", + "sidebar.custom_model_box_aria": "Custom model settings", + "sidebar.custom_model_title": "Custom model", + "sidebar.custom_llm_subtitle": "LLM (chat/copywriting)", + "sidebar.custom_llm_model_ph": "Model name, e.g. deepseek-chat / gpt-4o-mini", + "sidebar.custom_llm_baseurl_ph": "Base URL, e.g. https://api.xxx.com/v1", + "sidebar.custom_llm_apikey_ph": "API key", + "sidebar.custom_vlm_subtitle": "VLM (media understanding)", + "sidebar.custom_vlm_model_ph": "Model name, e.g. qwen-vl-plus / gpt-4o", + "sidebar.custom_vlm_baseurl_ph": "Base URL, e.g. https://api.xxx.com/v1", + "sidebar.custom_vlm_apikey_ph": "API key", + "sidebar.custom_hint": "Note: API keys are used only for server-side calls in this session. They are masked in the UI and tool trace.", + "sidebar.tts_box_aria": "TTS configuration", + "sidebar.tts_title": "TTS", + "sidebar.tts_provider_select_aria": "Select a TTS provider", + "sidebar.tts_default": "Use default configuration", + "sidebar.tts_hint": "Note: leaving fields empty will fall back to config.toml.", + "sidebar.tts_field_suffix": " (leave empty to use server default)", + "sidebar.use_custom_model": "Use custom model", + "sidebar.llm_label": "LLM model", + "sidebar.vlm_label": "VLM model", + "sidebar.llm_select_aria": "Select LLM model", + "sidebar.vlm_select_aria": "Select VLM model", + "sidebar.custom_llm_title": "Custom LLM", + "sidebar.custom_vlm_title": "Custom VLM", + "sidebar.custom_llm_box_aria": "Custom LLM settings", + "sidebar.custom_vlm_box_aria": "Custom VLM settings", + + "sidebar.pexels_box_aria": "Pexels API key settings", + "sidebar.pexels_title": "Pexels", + "sidebar.pexels_mode_select_aria": "Select Pexels key mode", + "sidebar.pexels_default": "Use default configuration", + "sidebar.pexels_custom": "Use custom key", + "sidebar.pexels_apikey_ph": "Pexels API key", + "sidebar.pexels_hint": "Note: default mode prefers config.toml (search_media.pexel_api_key). If empty, the tool will fall back to environment variables.", + + "sidebar.help.cta": "Click to view the configuration guide", + "sidebar.help.llm": "LLM is used for chat/copywriting.", + "sidebar.help.vlm": "VLM is used for media understanding (image/video).", + "sidebar.help.pexels": "Pexels is used for media search. Disclaimer: The online content searched by OpenStoryline is all from Pexels. Footage downloaded via Pexels is for the sole purpose of experiencing Open-Storyline editing effects and may not be redistributed or sold. We only provide the tool. All materials downloaded and used through this tool (such as Pexels images) are obtained by the user through the API. We are not responsible for the legality of user-generated video content or materials, or for any copyright/portrait rights disputes arising from the use of this tool. Please comply with the Pexels license agreement when using it.", + "sidebar.help.tts": "TTS is used to generate voiceover from text.", + "sidebar.help.pexels_home_link": "Visit the official Pexels website", + "sidebar.help.pexels_terms_link": "View Pexels Terms", + + // common + "common.retry_after_suffix": " (retry in {seconds}s)", + + // toast + "toast.interrupt_failed": "Interrupt failed: {msg}", + "toast.pending_limit": "Pending media limit reached ({max}). Please send/delete before uploading more.", + "toast.pending_limit_partial": "You can upload at most {remain} more file(s) (limit {max}). Only the first {remain} will be uploaded.", + "toast.uploading": "Uploading media… {pct}%", + "toast.uploading_file": "Uploading ({i}/{n}): {name}… {pct}%", + "toast.upload_failed": "Upload failed: {msg}", + "toast.delete_failed": "Delete failed: {msg}", + "toast.uploading_cannot_send": "Media is uploading. Please wait until it finishes before sending.", + "toast.uploading_interrupt_send": "Media is uploading, so a new message can't be sent yet. I interrupted the current reply; press Enter after the upload finishes.", + + // tools + "tool.card.default_name": "Tool call", + "tool.card.fallback_name": "MCP Tool", + + "tool.preview.render_title": "Rendered preview", + "tool.preview.other_videos": "Other videos (click to preview)", + "tool.preview.videos": "Videos (click to preview)", + "tool.preview.images": "Images (click to preview)", + "tool.preview.audio": "Audio", + "tool.preview.listen": "Listen", + "tool.preview.split_shots": "Shot splitting results (click to preview)", + + "tool.preview.btn_modal": "Open preview", + "tool.preview.btn_open": "Open", + + "tool.preview.more_items": "{n} more not shown", + "tool.preview.more_audios": "{n} more audio clip(s) not shown", + + "tool.preview.label.audio": "Audio {i}", + "tool.preview.label.video": "Video {i}", + "tool.preview.label.image": "Image {i}", + "tool.preview.label.shot": "Shot {i}", + + "preview.unsupported": "This type can't be previewed inline:", + "preview.open_download": "Open/Download", + } +}; + +function __osNormLang(x) { + const s = String(x || "").trim().toLowerCase(); + if (s === "en" || s.startsWith("en-")) return "en"; + return "zh"; +} + +function __osLoadLang() { + try { + const v = localStorage.getItem(__OS_LANG_STORAGE_KEY); + return v ? __osNormLang(v) : null; + } catch { + return null; + } +} + +function __osSaveLang(lang) { + try { localStorage.setItem(__OS_LANG_STORAGE_KEY, lang); } catch {} +} + +function __osFormat(tpl, vars) { + const s = String(tpl ?? ""); + return s.replace(/\{(\w+)\}/g, (_, k) => { + if (!vars || vars[k] == null) return ""; + return String(vars[k]); + }); +} + +function __t(key, vars) { + const lang = __osNormLang(window.OPENSTORYLINE_LANG || "zh"); + const table = __OS_I18N[lang] || __OS_I18N.zh; + const raw = (table && table[key] != null) ? table[key] : (__OS_I18N.zh[key] ?? key); + return __osFormat(raw, vars); +} + +function __applyI18n(root = document) { + // textContent + root.querySelectorAll("[data-i18n]").forEach((el) => { + const k = el.getAttribute("data-i18n"); + if (!k) return; + el.textContent = __t(k); + }); + + // attributes + root.querySelectorAll("[data-i18n-title]").forEach((el) => { + const k = el.getAttribute("data-i18n-title"); + if (!k) return; + el.setAttribute("title", __t(k)); + }); + + root.querySelectorAll("[data-i18n-aria-label]").forEach((el) => { + const k = el.getAttribute("data-i18n-aria-label"); + if (!k) return; + el.setAttribute("aria-label", __t(k)); + }); + + root.querySelectorAll("[data-i18n-placeholder]").forEach((el) => { + const k = el.getAttribute("data-i18n-placeholder"); + if (!k) return; + el.setAttribute("placeholder", __t(k)); + }); +} + +// TTS 动态字段 placeholder（suffix）重渲染： +// - 创建 input 时会写入 data-os-ph-base / data-os-ph-suffix +function __rerenderTtsFieldPlaceholders(root = document) { + root.querySelectorAll("input[data-os-ph-base]").forEach((el) => { + const base = String(el.getAttribute("data-os-ph-base") || ""); + const needSuffix = el.getAttribute("data-os-ph-suffix") === "1"; + el.setAttribute("placeholder", needSuffix ? `${base}${__t("sidebar.tts_field_suffix")}` : base); + }); +} + +function __osApplyHelpLinks(root = document) { + const lang = __osNormLang(window.OPENSTORYLINE_LANG || "zh"); + const nodes = (root || document).querySelectorAll(".sidebar-help[data-help-zh], .sidebar-help[data-help-en]"); + + nodes.forEach((a) => { + const zh = a.getAttribute("data-help-zh") || ""; + const en = a.getAttribute("data-help-en") || ""; + const href = (lang === "en") ? (en || zh) : (zh || en); + if (href) a.setAttribute("href", href); + }); +} + +function __osApplyTooltipLinks(root = document) { + const lang = __osNormLang(window.OPENSTORYLINE_LANG || "zh"); + + const nodes = (root || document).querySelectorAll( + ".sidebar-help-tooltip-link[data-terms-zh], .sidebar-help-tooltip-link[data-terms-en], " + + ".sidebar-help-tooltip-link[data-pexels-home-zh], .sidebar-help-tooltip-link[data-pexels-home-en]" + ); + + const pickHref = (el) => { + const homeZh = el.getAttribute("data-pexels-home-zh") || ""; + const homeEn = el.getAttribute("data-pexels-home-en") || ""; + const termsZh = el.getAttribute("data-terms-zh") || ""; + const termsEn = el.getAttribute("data-terms-en") || ""; + + const zh = homeZh || termsZh; + const en = homeEn || termsEn; + + return (lang === "en") ? (en || zh) : (zh || en); + }; + + const open = (el, ev) => { + if (ev) { + ev.preventDefault(); + ev.stopPropagation(); + } + const href = pickHref(el); + if (!href) return; + window.open(href, "_blank", "noopener,noreferrer"); + }; + + nodes.forEach((el) => { + if (el.__osTooltipLinkBound) return; + el.__osTooltipLinkBound = true; + + el.addEventListener("click", (e) => open(el, e), true); + + el.addEventListener("keydown", (e) => { + if (e.key === "Enter" || e.key === " ") open(el, e); + }, true); + }); +} + +function __osEnsureLeadingSlash(s) { + s = String(s ?? "").trim(); + if (!s) return ""; + return s.startsWith("/") ? s : ("/" + s); +} + + +function __osAppendToCurrentUrl(suffix) { + const suf = __osEnsureLeadingSlash(suffix); + if (!suf) return ""; + + const u = new URL(window.location.href); + + const h = String(u.hash || ""); + if (h.startsWith("#/") || h.startsWith("#!/")) { + const isBang = h.startsWith("#!/"); + const route = isBang ? h.slice(2) : h.slice(1); // "/xxx..." + const routeNoTrail = route.replace(/\/+$/, ""); + + if (routeNoTrail.endsWith(suf)) return `${u.origin}${u.pathname}${isBang ? "#!" : "#"}${routeNoTrail}`; + + return `${u.origin}${u.pathname}${isBang ? "#!" : "#"}${routeNoTrail}${suf}`; + } + u.search = ""; + u.hash = ""; + + let path = u.pathname || "/"; + + if (!path.endsWith("/")) { + const last = path.split("/").pop() || ""; + if (last.includes(".")) { + path = path.slice(0, path.length - last.length); // 留下末尾的 "/" + } + } + + const base = `${u.origin}${path}`.replace(/\/+$/, ""); + return `${base}${suf}`; +} + +function __osApplyTopbarLinks(root = document) { + const lang = __osNormLang(window.OPENSTORYLINE_LANG || "zh"); + const nodes = (root || document).querySelectorAll( + ".topbar-link[data-link-zh], .topbar-link[data-link-en], .topbar-link[data-link-suffix], .topbar-link[data-link-suffix-zh], .topbar-link[data-link-suffix-en]" + ); + + nodes.forEach((a) => { + // 1) 动态 suffix：优先 + const sufZh = a.getAttribute("data-link-suffix-zh") || ""; + const sufEn = a.getAttribute("data-link-suffix-en") || ""; + const suf = a.getAttribute("data-link-suffix") || ""; + + const pickedSuffix = (lang === "en") ? (sufEn || sufZh || suf) : (sufZh || sufEn || suf); + if (pickedSuffix) { + const href = __osAppendToCurrentUrl(pickedSuffix); + if (href) a.setAttribute("href", href); + return; + } + + // 2) 静态 zh/en URL + const zh = a.getAttribute("data-link-zh") || ""; + const en = a.getAttribute("data-link-en") || ""; + const href = (lang === "en") ? (en || zh) : (zh || en); + if (href) a.setAttribute("href", href); + }); +} + + +function __applyLang(lang, { persist = true } = {}) { + const v = __osNormLang(lang); + window.OPENSTORYLINE_LANG = v; + + if (persist) __osSaveLang(v); + + document.body.classList.toggle("lang-en", v === "en"); + document.body.classList.toggle("lang-zh", v === "zh"); + document.documentElement.lang = (v === "en") ? "en" : "zh-CN"; + + __applyI18n(document); + __rerenderTtsFieldPlaceholders(document); + __osApplyHelpLinks(document); + __osApplyTopbarLinks(document); + __osApplyTooltipLinks(document); +} + +// init once +(() => { + const stored = __osLoadLang(); + const initial = stored || __osNormLang(document.documentElement.lang || "zh"); + __applyLang(initial, { persist: stored != null }); // 有存储就保留；没存储就不写入 +})(); + + +class ApiClient { + async createSession() { + const r = await fetch("/api/sessions", { method: "POST" }); + if (!r.ok) throw new Error(await r.text()); + return await r.json(); + } + + async getSession(sessionId) { + const r = await fetch(`/api/sessions/${encodeURIComponent(sessionId)}`); + if (!r.ok) throw new Error(await r.text()); + return await r.json(); + } + + async getTtsUiSchema() { + const r = await fetch("/api/meta/tts", { method: "GET" }); + if (!r.ok) throw new Error(await this._readFetchError(r)); + return await r.json(); // { default_provider, providers:[...] } + } + + async cancelTurn(sessionId) { + const r = await fetch(`/api/sessions/${encodeURIComponent(sessionId)}/cancel`, { method: "POST" }); + if (!r.ok) throw new Error(await this._readFetchError(r)); + return await r.json(); + } + + async _readFetchError(r) { + const t = await r.text(); + try { + const j = JSON.parse(t); + // 兼容 middleware/接口的 429: {detail:"Too Many Requests", retry_after:n} + if (j && typeof j === "object") { + const ra = (j.retry_after != null) ? Number(j.retry_after) : (j.detail && j.detail.retry_after != null ? Number(j.detail.retry_after) : null); + + if (typeof j.detail === "string") return ra != null ? `${j.detail}${__t("common.retry_after_suffix", { seconds: ra })}` : j.detail; + if (j.detail && typeof j.detail === "object") { + const msg = j.detail.message || j.detail.detail || j.detail.error || JSON.stringify(j.detail); + return ra != null ? `${msg}${__t("common.retry_after_suffix", { seconds: ra })}` : msg; + } + if (typeof j.message === "string") return ra != null ? `${j.message}${__t("common.retry_after_suffix", { seconds: ra })}` : j.message; + } + } catch {} + return t || `HTTP ${r.status}`; + } + + async initResumableMedia(sessionId, file, { chunkSize } = {}) { + const r = await fetch(`/api/sessions/${encodeURIComponent(sessionId)}/media/init`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + filename: file.name, + size: file.size, + mime_type: file.type, + last_modified: file.lastModified, + chunk_size: chunkSize, // 服务端可忽略（以服务端配置为准） + }), + }); + if (!r.ok) throw new Error(await this._readFetchError(r)); + return await r.json(); + } + + uploadResumableChunk(sessionId, uploadId, index, blob, onProgress) { + return new Promise((resolve, reject) => { + const form = new FormData(); + form.append("index", String(index)); + // 这里用 blob（分片），而不是整文件 + form.append("chunk", blob, "chunk"); + + const xhr = new XMLHttpRequest(); + xhr.open( + "POST", + `/api/sessions/${encodeURIComponent(sessionId)}/media/${encodeURIComponent(uploadId)}/chunk`, + true + ); + + xhr.upload.onprogress = (e) => { + if (typeof onProgress === "function") { + const loaded = e && typeof e.loaded === "number" ? e.loaded : 0; + const total = e && typeof e.total === "number" ? e.total : (blob ? blob.size : 0); + onProgress(loaded, total); + } + }; + + xhr.onload = () => { + const ok = xhr.status >= 200 && xhr.status < 300; + if (ok) { + try { resolve(JSON.parse(xhr.responseText || "{}")); } + catch (e) { resolve({}); } + return; + } + + // 错误：尽量把 JSON detail 解析成可读信息 + const text = xhr.responseText || ""; + let msg = text || `HTTP ${xhr.status}`; + try { + const j = JSON.parse(text); + const ra = (j && typeof j === "object" && j.retry_after != null) ? Number(j.retry_after) : null; + if (j && typeof j.detail === "string") msg = ra != null ? `${j.detail}${__t("common.retry_after_suffix", { seconds: ra })}` : j.detail; + else if (j && typeof j.detail === "object") { + const m = j.detail.message || j.detail.detail || j.detail.error || JSON.stringify(j.detail); + msg = ra != null ? `${m}${__t("common.retry_after_suffix", { seconds: ra })}` : m; + } + } catch {} + reject(new Error(msg)); + }; + + xhr.onerror = () => reject(new Error("network error")); + xhr.send(form); + }); + } + + async completeResumableMedia(sessionId, uploadId) { + const r = await fetch(`/api/sessions/${encodeURIComponent(sessionId)}/media/${encodeURIComponent(uploadId)}/complete`, { + method: "POST", + }); + if (!r.ok) throw new Error(await this._readFetchError(r)); + return await r.json(); // { media, pending_media } + } + + async cancelResumableMedia(sessionId, uploadId) { + try { + await fetch(`/api/sessions/${encodeURIComponent(sessionId)}/media/${encodeURIComponent(uploadId)}/cancel`, { method: "POST" }); + } catch {} + } + + // 单文件：init -> chunk... -> complete + async uploadMediaChunked(sessionId, file, { chunkSize, onProgress } = {}) { + const init = await this.initResumableMedia(sessionId, file, { chunkSize }); + const uploadId = init.upload_id; + const cs = Number(init.chunk_size) || Number(chunkSize) || (32 * 1024 * 1024); + + const totalChunks = Number(init.total_chunks) || Math.ceil((file.size || 0) / cs) || 1; + + let confirmed = 0; // 已完成分片字节数（本文件内） + try { + for (let i = 0; i < totalChunks; i++) { + const start = i * cs; + const end = Math.min(file.size, start + cs); + const blob = file.slice(start, end); + + await this.uploadResumableChunk(sessionId, uploadId, i, blob, (loaded) => { + if (typeof onProgress === "function") { + // confirmed + 当前分片已上传字节 + onProgress(Math.min(file.size, confirmed + (loaded || 0)), file.size); + } + }); + + confirmed += blob.size; + if (typeof onProgress === "function") onProgress(Math.min(file.size, confirmed), file.size); + } + + return await this.completeResumableMedia(sessionId, uploadId); + } catch (e) { + // 失败尽量清理服务端临时文件 + await this.cancelResumableMedia(sessionId, uploadId); + throw e; + } + } + + + async deletePendingMedia(sessionId, mediaId) { + const r = await fetch( + `/api/sessions/${encodeURIComponent(sessionId)}/media/pending/${encodeURIComponent(mediaId)}`, + { method: "DELETE" } + ); + if (!r.ok) throw new Error(await r.text()); + return await r.json(); + } +} + +class WsClient { + constructor(url, onEvent) { + this.url = url; + this.onEvent = onEvent; + this.ws = null; + this._timer = null; + this._closedByUser = false; + } + + connect() { + this._closedByUser = false; + this.ws = new WebSocket(this.url); + + this.ws.onopen = () => { + // 心跳（可选） + this._timer = setInterval(() => { + if (this.ws && this.ws.readyState === 1) { + this.send("ping", {}); + } + }, 25000); + }; + + this.ws.onmessage = (e) => { + let msg; + try { msg = JSON.parse(e.data); } catch { return; } + if (this.onEvent) this.onEvent(msg); + }; + + this.ws.onclose = (ev) => { + if (this._timer) clearInterval(this._timer); + this._timer = null; + + console.warn("[ws] closed", { + code: ev?.code, + reason: ev?.reason, + wasClean: ev?.wasClean, + }); + + if (this._closedByUser) return; + + // session 不存在就不要重连 + if (ev && ev.code === 4404) { + localStorage.removeItem("openstoryline_session_id"); + location.reload(); + return; + } + + setTimeout(() => this.connect(), 1000); + }; + } + + close() { + this._closedByUser = true; + if (this._timer) clearInterval(this._timer); + this._timer = null; + if (this.ws) { + try { this.ws.close(1000, "client switch session"); } catch {} + this.ws = null; + } + } + + send(type, data) { + if (!this.ws || this.ws.readyState !== 1) return; + this.ws.send(JSON.stringify({ type, data })); + } +} + +class ChatUI { + constructor() { + this.chatEl = $("#chat"); + this.pendingBarEl = $("#pendingBar"); + this.pendingRowEl = $("#pendingRow"); + this.toastEl = $("#toast"); + // developer + this.devLogEl = $("#devLog") + this.devDomByID = new Map() + + this.modalEl = $("#modal"); + this.modalBackdrop = $("#modalBackdrop"); + this.modalClose = $("#modalClose"); + this.modalContent = $("#modalContent"); + + this.toolDomById = new Map(); + this.toolMediaDomById = new Map(); + this.currentAssistant = null; // { bubbleEl, rawText } + + this.mdStreaming = true; // 是否启用流式 markdown + this._mdRaf = 0; // requestAnimationFrame id + this._mdTimer = null; // setTimeout id + this._mdLastRenderAt = 0; // 上次渲染时间 + this._mdRenderInterval = 80; // 渲染时间间隔 + + this._toolUi = this._loadToolUiConfig(); + + this.scrollBtnEl = $("#scrollToBottomBtn"); + this._bindScrollJumpBtn(); + this._bindScrollWatcher(); + + this._toastI18n = null; + } + + setSessionId(sessionId) { + this._sessionId = sessionId; + const s = `session_id: ${sessionId}`; + const el = $("#sidebarSid"); + if (el) el.textContent = s; + } + + _setToastText(text) { + this.toastEl.textContent = String(text ?? ""); + this.toastEl.classList.remove("hidden"); + } + + showToast(text) { + this._toastI18n = null; + this._setToastText(text); + } + + showToastI18n(key, vars) { + this._toastI18n = { key: String(key || ""), vars: vars || {} }; + this._setToastText(__t(key, vars)); + } + + rerenderToast() { + if (!this.toastEl || this.toastEl.classList.contains("hidden")) return; + if (!this._toastI18n || !this._toastI18n.key) return; + this._setToastText(__t(this._toastI18n.key, this._toastI18n.vars)); + } + + rerenderAssistantPlaceholder() { + const cur = this.currentAssistant; + if (!cur || !cur.bubbleEl) return; + + if ((cur.rawText || "").trim()) return; + + const key = cur._placeholderKey; + if (!key) return; + + this.setBubbleContent(cur.bubbleEl, __t(key)); + } + + + hideToast() { + this.toastEl.classList.add("hidden"); + } + + + _docScrollHeight() { + const de = document.documentElement; + return (de && de.scrollHeight) ? de.scrollHeight : document.body.scrollHeight; + } + + isNearBottom(threshold = 160) { + const top = window.scrollY || window.pageYOffset || 0; + const h = window.innerHeight || 0; + return (top + h) >= (this._docScrollHeight() - threshold); + } + + _updateScrollJumpBtnVisibility(force) { + if (!this.scrollBtnEl) return; + + let show; + if (force === true) show = true; + else if (force === false) show = false; + else show = !this.isNearBottom(); + + this.scrollBtnEl.classList.toggle("hidden", !show); + } + + scrollToBottom({ behavior = "smooth" } = {}) { + requestAnimationFrame(() => { + window.scrollTo({ top: this._docScrollHeight(), behavior }); + }); + } + + maybeAutoScroll(wasNearBottom, { behavior = "auto" } = {}) { + if (wasNearBottom) { + this.scrollToBottom({ behavior }); + this._updateScrollJumpBtnVisibility(false); + } else { + this._updateScrollJumpBtnVisibility(true); + } + } + + _bindScrollJumpBtn() { + if (!this.scrollBtnEl || this._scrollBtnBound) return; + this._scrollBtnBound = true; + + this.scrollBtnEl.addEventListener("click", (e) => { + e.preventDefault(); + this.scrollToBottom({ behavior: "smooth" }); + this._updateScrollJumpBtnVisibility(false); + }); + } + + _bindScrollWatcher() { + if (this._scrollWatchBound) return; + this._scrollWatchBound = true; + + const handler = () => this._updateScrollJumpBtnVisibility(); + window.addEventListener("scroll", handler, { passive: true }); + window.addEventListener("resize", handler, { passive: true }); + + requestAnimationFrame(handler); + } + + + clearAll() { + this.chatEl.innerHTML = ""; + + // 停掉所有假进度条 timer + for (const [, dom] of this.toolDomById) { + if (dom && dom._fakeTimer) { + clearInterval(dom._fakeTimer); + dom._fakeTimer = null; + } + } + + this.toolDomById.clear(); + this.currentAssistant = null; + + if (this.devLogEl) this.devLogEl.innerHTML = ""; + this.devDomByID.clear() + + // 清掉 tool 外部媒体块 + if (this.toolMediaDomById) { + for (const [, dom] of this.toolMediaDomById) { + try { dom?.wrap?.remove(); } catch {} + } + this.toolMediaDomById.clear(); + } + + } + + setBubbleContent(bubbleEl, text, { markdown = true } = {}) { + const s = String(text ?? ""); + + // 纯文本模式：用于 user bubble（避免 marked 生成

导致默认 margin 撑大气泡） + if (!markdown || !window.marked || !window.DOMPurify) { + bubbleEl.textContent = s; + return; + } + + if (!this._mdInited) { + window.marked.setOptions({ + gfm: true, + breaks: true, + headerIds: false, + mangle: false, + }); + + window.DOMPurify.addHook("afterSanitizeAttributes", (node) => { + if (node.tagName === "A") { + node.setAttribute("target", "_blank"); + node.setAttribute("rel", "noopener noreferrer"); + } + }); + + this._mdInited = true; + } + + const rawHtml = window.marked.parse(s); + const safeHtml = window.DOMPurify.sanitize(rawHtml, { USE_PROFILES: { html: true } }); + bubbleEl.innerHTML = safeHtml; + } + + + renderPendingMedia(pendingMedia) { + this.pendingRowEl.innerHTML = ""; + if (!pendingMedia || !pendingMedia.length) { + this.pendingBarEl.classList.add("hidden"); + return; + } + this.pendingBarEl.classList.remove("hidden"); + + for (const a of pendingMedia) { + this.pendingRowEl.appendChild(this.renderMediaThumb(a, { removable: true })); + } + } + + mediaTag(kind) { + if (kind === "image") return "IMG"; + if (kind === "video") return "VID"; + return ""; + } + + renderMediaThumb(media, { removable } = { removable: false }) { + const el = document.createElement("div"); + el.className = "media-item"; + el.title = media.name || ""; + + const img = document.createElement("img"); + img.src = media.thumb_url; + img.alt = media.name || ""; + el.appendChild(img); + + const tag = document.createElement("div"); + tag.className = "media-tag"; + tag.textContent = this.mediaTag(media.kind); + el.appendChild(tag); + + if (media.kind === "video") { + const play = document.createElement("div"); + play.className = "media-play"; + el.appendChild(play); + } + + el.addEventListener("click", (e) => { + if (e.target?.classList?.contains("media-remove")) return; + this.openPreview(media); + }); + + if (removable) { + const rm = document.createElement("div"); + rm.className = "media-remove"; + rm.textContent = "×"; + rm.dataset.mediaId = media.id; + el.appendChild(rm); + } + + return el; + } + + renderAttachmentsRow(attachments, alignRight) { + if (!attachments || !attachments.length) return null; + + const wrap = document.createElement("div"); + wrap.className = "attach-wrap"; + if (alignRight) wrap.classList.add("align-right"); + + const row = document.createElement("div"); + row.className = "attach-row"; + + for (const a of attachments) { + row.appendChild(this.renderMediaThumb(a, { removable: false })); + } + + wrap.appendChild(row); + return wrap; + } + + appendUserMessage(text, attachments) { + const wrap = document.createElement("div"); + wrap.className = "msg user"; + + const container = document.createElement("div"); + container.style.maxWidth = "78%"; + + const attachRow = this.renderAttachmentsRow(attachments, true); + if (attachRow) container.appendChild(attachRow); + + const bubble = document.createElement("div"); + bubble.className = "bubble"; + this.setBubbleContent(bubble, text, { markdown: false }); + container.appendChild(bubble); + + wrap.appendChild(container); + this.chatEl.appendChild(wrap); + this.scrollToBottom({ behavior: "smooth" }); + this._updateScrollJumpBtnVisibility(false); + } + + startAssistantMessage({ placeholder = true } = {}) { + const wasNearBottom = this.isNearBottom(); + const wrap = document.createElement("div"); + wrap.className = "msg assistant"; + + const bubble = document.createElement("div"); + bubble.className = "bubble"; + + const phKey = "assistant.placeholder"; + if (placeholder) { + this.setBubbleContent(bubble, __t(phKey)); + } else { + this.setBubbleContent(bubble, ""); + } + + wrap.appendChild(bubble); + this.chatEl.appendChild(wrap); + this.maybeAutoScroll(wasNearBottom, { behavior: "auto" }); + + this.currentAssistant = { + wrapEl: wrap, + bubbleEl: bubble, + rawText: "", + _placeholderKey: placeholder ? phKey : null, + }; + } + + + + + _normalizeStreamingMarkdown(s) { + s = String(s ?? "").replace(/\r\n?/g, "\n"); + + const ticks = (s.match(/```/g) || []).length; + if (ticks % 2 === 1) s += "\n```"; + + return s; + } + + _renderAssistantStreaming(cur) { + this._mdLastRenderAt = Date.now(); + + const wasNearBottom = this.isNearBottom(160); + + const md = this._normalizeStreamingMarkdown(cur.rawText); + this.setBubbleContent(cur.bubbleEl, md); + + if (wasNearBottom) this.scrollToBottom({ behavior: "auto" }); + else this._updateScrollJumpBtnVisibility(true); + } + + appendAssistantDelta(delta) { + console.log("md deps", !!window.marked, !!window.DOMPurify); + + if (!this.currentAssistant) this.startAssistantMessage({ placeholder: false }); + + const cur = this.currentAssistant; + cur.rawText += (delta || ""); + + // 节流：避免每 token 都 parse + sanitize + const now = Date.now(); + const due = now - this._mdLastRenderAt >= this._mdRenderInterval; + + if (due) { + this._renderAssistantStreaming(cur); + return; + } + + if (this._mdTimer) return; + const wait = Math.max(0, this._mdRenderInterval - (now - this._mdLastRenderAt)); + this._mdTimer = setTimeout(() => { + this._mdTimer = null; + if (this.currentAssistant) this._renderAssistantStreaming(this.currentAssistant); + }, wait); + } + + finalizeAssistant(text) { + const wasNearBottom = this.isNearBottom(); + if (!this.currentAssistant) { + this.startAssistantMessage({ placeholder: false}); + } + const cur = this.currentAssistant; + cur.rawText = (text ?? cur.rawText ?? "").trim(); + this.setBubbleContent(cur.bubbleEl, cur.rawText || "（未生成最终答复）"); + this.currentAssistant = null; + this.maybeAutoScroll(wasNearBottom, { behavior: "auto" }); + } + + // 结束当前 assistant 分段（用于 tool.start 前封口） + flushAssistantSegment() { + const wasNearBottom = this.isNearBottom(); + const cur = this.currentAssistant; + if (!cur) return; + + const text = (cur.rawText || "").trim(); + if (!text) { + // 没有任何 token（只有占位文案）=> 直接移除 + if (cur.wrapEl) cur.wrapEl.remove(); + } else { + this.setBubbleContent(cur.bubbleEl, text); + } + + this.currentAssistant = null; + this.maybeAutoScroll(wasNearBottom, { behavior: "auto" }); + } + + // 结束整个 turn（对应后端 assistant.end） + endAssistantTurn(text) { + const wasNearBottom = this.isNearBottom(); + const s = String(text ?? "").trim(); + + if (this.currentAssistant) { + const cur = this.currentAssistant; + + // 如果服务端给了最终文本，以服务端为准 + if (s) cur.rawText = s; + + const finalText = (cur.rawText || "").trim(); + if (!finalText) { + if (cur.wrapEl) cur.wrapEl.remove(); + } else { + this.setBubbleContent(cur.bubbleEl, finalText); + } + + this.currentAssistant = null; + this.maybeAutoScroll(wasNearBottom, { behavior: "auto" }); + return; + } + + // 没有正在流的 bubble：只有当确实有文本时才新建一条 + if (s) { + this.startAssistantMessage({ placeholder: false }); + const cur = this.currentAssistant; + cur.rawText = s; + this.setBubbleContent(cur.bubbleEl, s); + this.currentAssistant = null; + this.scrollToBottom(); + } + } + + _loadToolUiConfig() { + const cfg = (window.OPENSTORYLINE_TOOL_UI && typeof window.OPENSTORYLINE_TOOL_UI === "object") + ? window.OPENSTORYLINE_TOOL_UI + : {}; + + const labels = + (cfg.labels && typeof cfg.labels === "object") ? cfg.labels : + (window.OPENSTORYLINE_TOOL_LABELS && typeof window.OPENSTORYLINE_TOOL_LABELS === "object") ? window.OPENSTORYLINE_TOOL_LABELS : + {}; + + const estimatesMs = + (cfg.estimates_ms && typeof cfg.estimates_ms === "object") ? cfg.estimates_ms : + (cfg.estimatesMs && typeof cfg.estimatesMs === "object") ? cfg.estimatesMs : + (window.OPENSTORYLINE_TOOL_ESTIMATES && typeof window.OPENSTORYLINE_TOOL_ESTIMATES === "object") ? window.OPENSTORYLINE_TOOL_ESTIMATES : + {}; + + const defaultEstimateMs = Number(cfg.default_estimate_ms ?? cfg.defaultEstimateMs ?? 8000); + const tickMs = Number(cfg.tick_ms ?? cfg.tickMs ?? 120); + const capRunning = Number(cfg.cap_running_progress ?? cfg.capRunningProgress ?? 0.99); + + return { + labels, + estimatesMs, + defaultEstimateMs: (Number.isFinite(defaultEstimateMs) && defaultEstimateMs > 0) ? defaultEstimateMs : 8000, + tickMs: (Number.isFinite(tickMs) && tickMs >= 30) ? tickMs : 120, + capRunningProgress: (Number.isFinite(capRunning) && capRunning > 0 && capRunning < 1) ? capRunning : 0.99, + + // autoOpenWhileRunning: (cfg.auto_open_while_running != null) ? !!cfg.auto_open_while_running : false, + // autoCollapseOnDone: (cfg.auto_collapse_on_done != null) ? !!cfg.auto_collapse_on_done : false, + + hideRawToolName: (cfg.hide_raw_tool_name != null) ? !!cfg.hide_raw_tool_name : true, + showRawToolNameInDev: (cfg.show_raw_tool_name_in_dev != null) ? !!cfg.show_raw_tool_name_in_dev : false, + }; + } + + _toolFullName(server, name) { + return `${server || ""}.${name || ""}`.replace(/^\./, ""); + } + + _toolDisplayName(server, name) { + const full = this._toolFullName(server, name); + const labels = (this._toolUi && this._toolUi.labels) || {}; + + const hit = + labels[full] ?? + labels[name] ?? + labels[String(full).toLowerCase()] ?? + labels[String(name).toLowerCase()]; + + if (hit != null) { + if (typeof hit === "string") return String(hit); + + if (hit && typeof hit === "object") { + const lang = __osNormLang(window.OPENSTORYLINE_LANG || "zh"); + const v = hit[lang] ?? hit.zh ?? hit.en; + if (v != null) return String(v); + } + } + + if (this._toolUi && this._toolUi.hideRawToolName) return __t("tool.card.default_name"); + return full || __t("tool.card.fallback_name"); + } + + _toolEstimateMs(server, name) { + const full = this._toolFullName(server, name); + const map = (this._toolUi && this._toolUi.estimatesMs) || {}; + const v = map[full] ?? map[name]; + const ms = Number(v); + if (Number.isFinite(ms) && ms > 0) return ms; + return (this._toolUi && this._toolUi.defaultEstimateMs) ? this._toolUi.defaultEstimateMs : 8000; + } + + _normToolState(s) { + s = String(s || ""); + if (s === "running") return "running"; + if (s === "error" || s === "failed") return "error"; + if (s === "success" || s === "complete" || s === "done") return "success"; + return "running"; + } + + _calcFakeProgress(dom) { + const est = Math.max(1, Number(dom._fakeEstimateMs || 8000)); + const startAt = Number(dom._fakeStartAt || Date.now()); + const cap = (this._toolUi && this._toolUi.capRunningProgress) ? this._toolUi.capRunningProgress : 0.99; + + const elapsed = Math.max(0, Date.now() - startAt); + const raw = elapsed / est; + + // 慢了就停 99% + const p = Math.min(Math.max(raw, 0), cap); + + dom._fakeProgress = p; + return p; + } + + _updateFakeProgress(dom) { + if (!dom || !dom.data) return; + if (this._normToolState(dom.data.state) !== "running") return; + + const p = this._calcFakeProgress(dom); + + if (dom.fill) dom.fill.style.width = `${Math.round(p * 100)}%`; + + // 百分比：最多显示 99% + const pct = Math.min(99, Math.max(0, Math.floor(p * 100))); + if (dom.pctEl) dom.pctEl.textContent = `${pct}%`; + } + + _ensureFakeProgress(dom, { server, name, progress } = {}) { + if (!dom) return; + + dom._fakeEstimateMs = this._toolEstimateMs(server, name); + + const cap = (this._toolUi && this._toolUi.capRunningProgress) ? this._toolUi.capRunningProgress : 0.99; + const init = Math.min(Math.max(Number(progress) || 0, 0), cap); + + if (!Number.isFinite(dom._fakeInitProgress)) dom._fakeInitProgress = init; + else dom._fakeInitProgress = Math.max(dom._fakeInitProgress, init); + + if (!Number.isFinite(dom._fakeStartAt)) dom._fakeStartAt = NaN; + + this._updateFakeProgress(dom); + if (dom._fakeTimer) return; + + if (dom._fakeDelayTimer) return; + + const tickMs = (this._toolUi && this._toolUi.tickMs) ? this._toolUi.tickMs : 120; + const delayMs = (this._toolUi && Number.isFinite(this._toolUi.fakeDelayMs)) + ? Math.max(0, Number(this._toolUi.fakeDelayMs)) + : 2000; + + dom._fakeDelayTimer = setTimeout(() => { + dom._fakeDelayTimer = null; + + if (!dom || !dom.data) return; + + const st = this._normToolState(dom.data.state); + if (st !== "running") return; + + if (dom._progressMode === "real") return; + + if (dom._fakeTimer) return; + + const init2 = Math.min(Math.max(Number(dom._fakeInitProgress) || 0, 0), cap); + dom._fakeStartAt = Date.now() - init2 * dom._fakeEstimateMs; + this._updateFakeProgress(dom); + + dom._fakeTimer = setInterval(() => { + if (!dom || !dom.data) { + if (dom && dom._fakeTimer) clearInterval(dom._fakeTimer); + if (dom) dom._fakeTimer = null; + return; + } + + const st2 = this._normToolState(dom.data.state); + if (st2 !== "running") { + if (dom._fakeTimer) clearInterval(dom._fakeTimer); + dom._fakeTimer = null; + return; + } + + if (dom._progressMode === "real") { + if (dom._fakeTimer) clearInterval(dom._fakeTimer); + dom._fakeTimer = null; + return; + } + + this._updateFakeProgress(dom); + }, tickMs); + }, delayMs); + } + + _stopFakeProgress(dom) { + if (!dom) return; + + if (dom._fakeDelayTimer) { + clearTimeout(dom._fakeDelayTimer); + dom._fakeDelayTimer = null; + } + + if (dom._fakeTimer) { + clearInterval(dom._fakeTimer); + dom._fakeTimer = null; + } + + dom._fakeStartAt = NaN; + dom._fakeProgress = 0; + dom._fakeInitProgress = NaN; + } + + _summaryToObject(summary) { + if (summary == null) return null; + if (typeof summary === "object") return summary; + + if (typeof summary === "string") { + // 后端可能把 summary 转成 JSON 字符串 + try { + const obj = JSON.parse(summary); + return (obj && typeof obj === "object") ? obj : null; + } catch { + return null; + } + } + return null; + } + + // tool 卡片：按 tool_call_id upsert（可折叠、极简、带状态符号） + upsertToolCard(tool_call_id, patch) { + const wasNearBottom = this.isNearBottom(); + const clamp01 = (n) => Math.max(0, Math.min(1, Number.isFinite(n) ? n : 0)); + const safeStringify = (x) => { + try { return JSON.stringify(x); } catch { return String(x ?? ""); } + }; + const truncate = (s, n = 160) => { + s = String(s ?? ""); + return s.length > n ? (s.slice(0, n) + "…") : s; + }; + const normState = (s) => { + s = String(s || ""); + if (s === "running") return "running"; + if (s === "error" || s === "failed") return "error"; + if (s === "success" || s === "complete" || s === "done") return "success"; + return "running"; + }; + + let dom = this.toolDomById.get(tool_call_id); + + if (!dom) { + const wrap = document.createElement("div"); + wrap.className = "msg assistant"; + + const details = document.createElement("details"); + details.className = "tool-card"; + details.open = false; // 强制默认折叠 + + const head = document.createElement("summary"); + head.className = "tool-head"; + + // 单行：状态符号 + 工具名 + args 预览（ellipsis） + const line = document.createElement("div"); + line.className = "tool-line"; + + const left = document.createElement("div"); + left.className = "tool-left"; + + const statusEl = document.createElement("span"); + statusEl.className = "tool-status"; + + const nameEl = document.createElement("span"); + nameEl.className = "tool-name"; + + left.appendChild(statusEl); + left.appendChild(nameEl); + + const argsPreviewEl = document.createElement("div"); + argsPreviewEl.className = "tool-args-preview"; + + line.appendChild(left); + line.appendChild(argsPreviewEl); + + // 自定义短进度条 + 百分比 + const progRow = document.createElement("div"); + progRow.className = "tool-progress-row"; + + const prog = document.createElement("div"); + prog.className = "tool-progress"; + + const fill = document.createElement("div"); + fill.className = "tool-progress-fill"; + prog.appendChild(fill); + + const pctEl = document.createElement("span"); + pctEl.className = "tool-progress-pct"; + pctEl.textContent = "0%"; + + progRow.appendChild(prog); + progRow.appendChild(pctEl); + + head.appendChild(line); + head.appendChild(progRow); + + // 展开内容：args + summary + const bodyWrap = document.createElement("div"); + bodyWrap.className = "tool-body-wrap"; + + const pre = document.createElement("pre"); + pre.className = "tool-body"; + + const preview = document.createElement("div"); + preview.className = "tool-preview"; + preview.style.display = "none"; // 永久隐藏：不在 tool-card 内展示媒体 + + bodyWrap.appendChild(pre); + bodyWrap.appendChild(preview); + + details.appendChild(head); + details.appendChild(bodyWrap); + + wrap.appendChild(details); + this.chatEl.appendChild(wrap); + this.maybeAutoScroll(wasNearBottom, { behavior: "auto" }); + + dom = { + wrap, details, statusEl, nameEl, argsPreviewEl, progRow, prog, fill, pctEl, pre, preview, + data: { server: "", name: "", args: undefined, message: "", summary: null, state: "running", progress: 0 }, + _progressMode: "fake", + }; + this.toolDomById.set(tool_call_id, dom); + } + + // merge patch -> dom.data（关键：progress/end 不传 args 时要保留 start 的 args） + const d = dom.data || {}; + const merged = { + server: (patch && patch.server != null) ? patch.server : d.server, + name: (patch && patch.name != null) ? patch.name : d.name, + state: (patch && patch.state != null) ? patch.state : d.state, + progress: (patch && typeof patch.progress === "number") ? patch.progress : d.progress, + message: (patch && Object.prototype.hasOwnProperty.call(patch, "message")) ? (patch.message || "") : d.message, + summary: (patch && Object.prototype.hasOwnProperty.call(patch, "summary")) ? patch.summary : d.summary, + args: (patch && Object.prototype.hasOwnProperty.call(patch, "args")) ? patch.args : d.args, + }; + dom.data = merged; + + if (patch && patch.__progress_mode === "real") { + dom._progressMode = "real"; + } + + const st = this._normToolState(merged.state); + + const displayName = this._toolDisplayName(merged.server, merged.name); + dom.nameEl.textContent = displayName; + + // 状态符号 + dom.statusEl.classList.remove("is-running", "is-success", "is-error"); + if (st === "running") { + dom.statusEl.textContent = ""; + dom.statusEl.classList.add("is-running"); + } else if (st === "success") { + dom.statusEl.textContent = "✓"; + dom.statusEl.classList.add("is-success"); + } else { + dom.statusEl.textContent = "!"; + dom.statusEl.classList.add("is-error"); + } + + // args 预览（单行） + dom.argsPreviewEl.style.display = "none"; + dom.argsPreviewEl.textContent = ""; + + if (st === "running") { + dom.progRow.style.display = "flex"; + + if (merged.message) { + dom.argsPreviewEl.style.display = "block"; + dom.argsPreviewEl.textContent = merged.message; + } else { + dom.argsPreviewEl.style.display = "none"; + dom.argsPreviewEl.textContent = ""; + } + + if (dom._progressMode === "real") { + this._stopFakeProgress(dom); + + const p = clamp01(merged.progress); + if (dom.fill) dom.fill.style.width = `${Math.round(p * 100)}%`; + if (dom.pctEl) dom.pctEl.textContent = `${Math.round(p * 100)}%`; + } else { + this._ensureFakeProgress(dom, { + server: merged.server, + name: merged.name, + progress: merged.progress, + }); + this._updateFakeProgress(dom); + } + } else { + this._stopFakeProgress(dom); + + dom.argsPreviewEl.style.display = "none"; + dom.argsPreviewEl.textContent = ""; + + dom.progRow.style.display = "none"; + dom.fill.style.width = "0%"; + if (dom.pctEl) dom.pctEl.textContent = "0%"; + } + + + // 展开体内容（完整展示参数/消息/结果摘要） + const lines = []; + if (merged.args != null) lines.push(`args = ${JSON.stringify(merged.args, null, 2)}`); + if (merged.message) lines.push(`message: ${merged.message}`); + if (merged.summary != null) { + // 把“可见的 \n”解码成真实换行 + const unescapeVisible = (s) => { + if (typeof s !== "string") return s; + return s + .replace(/\\r\\n/g, "\n") + .replace(/\\n/g, "\n") + .replace(/\\r/g, "\r") + .replace(/\\t/g, "\t"); + }; + + let obj = merged.summary; + if (typeof obj === "string") { + try { obj = JSON.parse(obj); } + catch { obj = null; } + } + + let v = (obj && typeof obj === "object") ? obj["INFO_USER"] : undefined; + + if (typeof v === "string") { + v = unescapeVisible(v); + + const t = v.trim(); + if ((t.startsWith("{") && t.endsWith("}")) || (t.startsWith("[") && t.endsWith("]"))) { + try { v = JSON.stringify(JSON.parse(t), null, 2); } catch {} + } + lines.push(`\n${v}`); + } else if (v != null) { + lines.push(`${JSON.stringify(v, null, 2)}`); + } else { + lines.push(``); + } + } + + + dom.pre.textContent = lines.join("\n\n").trim(); + + if (merged && merged.summary != null) { + this._upsertToolMediaMessage(tool_call_id, merged, dom); + } else { + // 没 summary 就清理对应媒体块（通常发生在 running/progress 阶段） + this._removeToolMediaMessage(tool_call_id); + } + } + + // 语言切换时：把已存在的 tool 卡片标题也刷新 + rerenderToolCards() { + if (!this.toolDomById) return; + + for (const [, dom] of this.toolDomById) { + const d = dom?.data || {}; + if (dom?.nameEl) { + dom.nameEl.textContent = this._toolDisplayName(d.server, d.name); + } + } + } + + appendDevSummary(tool_call_id, { server, name, summary, is_error } = {}) { + // 只有 developer mode 才输出 + if (!document.body.classList.contains("dev-mode")) return; + if (!this.devLogEl) return; + if (!tool_call_id) return; + + const fullName = `${server || ""}.${name || ""}`.replace(/^\./, "") || "MCP Tool"; + const headText = `${fullName} (${tool_call_id})${is_error ? " [error]" : ""}`; + + let summaryText = ""; + if (summary == null) { + summaryText = "（无 summary）"; + } else if (typeof summary === "string") { + summaryText = summary; + } else { + try { summaryText = JSON.stringify(summary, null, 2); } + catch { summaryText = String(summary); } + } + + let dom = this.devDomByID.get(tool_call_id); + if (!dom) { + const item = document.createElement("div"); + item.className = "devlog-item"; + + const head = document.createElement("div"); + head.className = "devlog-head"; + head.textContent = headText; + + const pre = document.createElement("pre"); + pre.className = "devlog-pre"; + pre.textContent = summaryText; + + item.appendChild(head); + item.appendChild(pre); + + this.devLogEl.appendChild(item); + this.devDomByID.set(tool_call_id, { item, head, pre }); + } else { + dom.head.textContent = headText; + dom.pre.textContent = summaryText; + } + + // 自动滚到底部，便于实时追踪 + requestAnimationFrame(() => { + const el = this.devLogEl; + if (!el) return; + el.scrollTop = el.scrollHeight; + }); + } + + // 工具调用结果中展示视频、图片、音频 + _stripUrlQueryHash(u) { + return String(u ?? "").split("#")[0].split("?")[0]; + } + + _basenameFromUrl(u) { + const s = this._stripUrlQueryHash(u); + const parts = s.split(/[\\/]/); + return parts[parts.length - 1] || s; + } + + _guessMediaKindFromUrl(u) { + const s = this._stripUrlQueryHash(u).toLowerCase(); + const m = s.match(/\.([a-z0-9]+)$/); + const ext = m ? "." + m[1] : ""; + + if ([".png", ".jpg", ".jpeg", ".gif", ".webp", ".bmp", ".svg"].includes(ext)) return "image"; + if ([".mp4", ".mov", ".webm", ".mkv", ".avi", ".m4v"].includes(ext)) return "video"; + if ([".mp3", ".wav", ".m4a", ".aac", ".flac", ".ogg", ".opus"].includes(ext)) return "audio"; + return "unknown"; + } + + _isSafeMediaUrl(u) { + const s = String(u ?? "").trim(); + if (!s) return false; + try { + const parsed = new URL(s, window.location.href); + const proto = String(parsed.protocol || "").toLowerCase(); + // allow: same-origin relative -> becomes http(s) here; allow absolute http(s) and blob + return proto === "http:" || proto === "https:" || proto === "blob:"; + } catch { + return false; + } + } + + _getPreviewUrlsFromSummary(summary) { + let obj = summary; + if (typeof obj === "string") { + try { obj = JSON.parse(obj); } catch { return []; } + } + const urls = obj && obj.preview_urls; + if (!Array.isArray(urls)) return []; + return urls.filter((u) => typeof u === "string" && u.trim()); + } + + _extractMediaItemsFromSummary(summary) { + const raws = this._getPreviewUrlsFromSummary(summary); + const out = []; + const seen = new Set(); + + for (const raw of raws) { + const url = this._normalizePreviewUrl(raw); + if (!url) continue; + + // 关键：kind 用 raw 判定（因为 /preview?path=... 本身不带后缀） + const kind = this._guessMediaKindFromUrl(String(raw)); + if (kind === "unknown") continue; + + const key = this._stripUrlQueryHash(String(raw)); + if (seen.has(key)) continue; + seen.add(key); + + out.push({ + url, // 可访问 URL：网络/或 /api/.../preview?path=... + kind, + name: this._basenameFromUrl(String(raw)), + }); + } + + return out; + } + + _makeToolPreviewTitle(text) { + const t = document.createElement("div"); + t.className = "tool-preview-title"; + t.textContent = String(text ?? ""); + return t; + } + + _makeInlineVideoBlock(item, title) { + const block = document.createElement("div"); + block.className = "tool-preview-block"; + + if (title) block.appendChild(this._makeToolPreviewTitle(title)); + + const v = document.createElement("video"); + v.style.objectFit = "contain"; + v.style.objectPosition = "center"; + v.className = "tool-inline-video"; + v.controls = true; + v.preload = "metadata"; + v.playsInline = true; + v.src = item.url; + block.appendChild(v); + + const actions = document.createElement("div"); + actions.className = "tool-preview-actions"; + + const btn = document.createElement("button"); + btn.type = "button"; + btn.className = "tool-preview-btn"; + btn.textContent = __t("tool.preview.btn_modal"); + btn.addEventListener("click", (e) => { + e.preventDefault(); + e.stopPropagation(); + this.openPreview({ kind: "video", file_url: item.url, name: item.name }); + }); + actions.appendChild(btn); + + const link = document.createElement("a"); + link.className = "tool-preview-link"; + link.href = item.url; + link.target = "_blank"; + link.rel = "noopener noreferrer"; + link.textContent = __t("tool.preview.btn_open"); + actions.appendChild(link); + + block.appendChild(actions); + + return block; + } + + _makeAudioListBlock(items, title, { maxItems = AUDIO_PREVIEW_MAX } = {}) { + const block = document.createElement("div"); + block.className = "tool-preview-block"; + + if (title) block.appendChild(this._makeToolPreviewTitle(title)); + + const list = document.createElement("div"); + list.className = "tool-audio-list"; + + const show = items.slice(0, maxItems); + show.forEach((it, idx) => { + const row = document.createElement("div"); + row.className = "tool-audio-item"; + + const label = document.createElement("div"); + label.className = "tool-media-label"; + label.textContent = it.name || __t("tool.preview.label.audio", { i: idx + 1 }); + row.appendChild(label); + + const a = document.createElement("audio"); + a.controls = true; + a.preload = "metadata"; + a.src = it.url; + row.appendChild(a); + + list.appendChild(row); + }); + + block.appendChild(list); + + if (items.length > maxItems) { + const more = document.createElement("div"); + more.className = "tool-media-more"; + more.textContent = __t("tool.preview.more_audios", { n: items.length - maxItems }); + block.appendChild(more); + } + + return block; + } + + _makeMediaGridBlock(items, { title, kind, labelKey, maxItems = 12 } = {}) { + const block = document.createElement("div"); + block.className = "tool-preview-block"; + + if (title) block.appendChild(this._makeToolPreviewTitle(title)); + + const grid = document.createElement("div"); + grid.className = "tool-media-grid"; + + // 根据宽高给 thumb 打标签，动态调整 aspect-ratio + const applyThumbAspect = (thumb, w, h) => { + const W = Number(w) || 0; + const H = Number(h) || 0; + if (!(W > 0 && H > 0)) return; + + thumb.classList.remove("is-portrait", "is-square"); + const r = W / H; + + // square: 0.92~1.08 + if (r >= 0.92 && r <= 1.08) { + thumb.classList.add("is-square"); + return; + } + // portrait: r < 1 + if (r < 1) { + thumb.classList.add("is-portrait"); + } + }; + + const show = items.slice(0, maxItems); + show.forEach((it, idx) => { + const btn = document.createElement("button"); + btn.type = "button"; + btn.className = "tool-media-item"; + btn.title = it.name || it.url; + + const thumb = document.createElement("div"); + thumb.className = "tool-media-thumb"; + + if (kind === "image") { + const img = document.createElement("img"); + img.src = it.url; + img.alt = it.name || ""; + + // FIX(1): 强制不裁切（不依赖 CSS 是否命中/是否被覆盖） + img.style.objectFit = "contain"; + img.style.objectPosition = "center"; + + img.addEventListener("load", () => { + applyThumbAspect(thumb, img.naturalWidth, img.naturalHeight); + }); + + thumb.appendChild(img); + } else if (kind === "video") { + const v = document.createElement("video"); + v.preload = "metadata"; + v.muted = true; + v.playsInline = true; + + // FIX(1): 强制不裁切 + v.style.objectFit = "contain"; + v.style.objectPosition = "center"; + + const apply = () => applyThumbAspect(thumb, v.videoWidth, v.videoHeight); + // 先绑定，再设置 src，避免缓存命中导致事件丢失 + v.addEventListener("loadedmetadata", apply, { once: true }); + // 少数浏览器/资源场景 loadedmetadata 不稳定，再用 loadeddata 兜底一次 + v.addEventListener("loadeddata", apply, { once: true }); + + v.src = it.url; + + thumb.appendChild(v); + if (v.readyState >= 1) apply(); + + const play = document.createElement("div"); + play.className = "tool-media-play"; + thumb.appendChild(play); + } + + btn.appendChild(thumb); + + const label = document.createElement("div"); + label.className = "tool-media-label"; + const fallbackKey = + labelKey || + (kind === "video" ? "tool.preview.label.video" : "tool.preview.label.image"); + + label.textContent = it.name || __t(fallbackKey, { i: idx + 1 }); + btn.appendChild(label); + + btn.addEventListener("click", (e) => { + e.preventDefault(); + e.stopPropagation(); + this.openPreview({ kind, file_url: it.url, name: it.name }); + }); + + grid.appendChild(btn); + }); + + block.appendChild(grid); + + if (items.length > maxItems) { + const more = document.createElement("div"); + more.className = "tool-media-more"; + more.textContent = __t("tool.preview.more_items", { n: items.length - maxItems }); + block.appendChild(more); + } + + return block; + } + + _removeToolMediaMessage(tool_call_id) { + const dom = this.toolMediaDomById && this.toolMediaDomById.get(tool_call_id); + if (dom) { + try { dom.wrap?.remove(); } catch {} + this.toolMediaDomById.delete(tool_call_id); + } + } + + // 在 chat 列表中，把“媒体预览块”插在 tool-card 后面（不放进 tool-card） + _upsertToolMediaMessage(tool_call_id, merged, toolCardDom) { + if (!tool_call_id) return; + + const summary = merged?.summary; + if (summary == null) { + // 没 summary 就不展示（也可选择清理旧的） + this._removeToolMediaMessage(tool_call_id); + return; + } + + // 从 summary.preview_urls 提取媒体 + const media = this._extractMediaItemsFromSummary(summary); + if (!media || !media.length) { + this._removeToolMediaMessage(tool_call_id); + return; + } + + // 已存在就复用（并确保位置在 tool-card 之后） + let dom = this.toolMediaDomById.get(tool_call_id); + + const wasNearBottom = this.isNearBottom(); + + if (!dom) { + const wrap = document.createElement("div"); + wrap.className = "msg assistant tool-media-msg"; + + const card = document.createElement("div"); + card.className = "media-card"; + + const preview = document.createElement("div"); + // 复用现有 tool-preview 的样式与内部 block 结构 + preview.className = "tool-preview"; + + card.appendChild(preview); + wrap.appendChild(card); + + // 插入到 tool-card 之后（保证顺序：tool card -> media） + if (toolCardDom && toolCardDom.wrap && toolCardDom.wrap.parentNode) { + toolCardDom.wrap.after(wrap); + } else { + this.chatEl.appendChild(wrap); + } + + dom = { wrap, card, preview }; + this.toolMediaDomById.set(tool_call_id, dom); + + this.maybeAutoScroll(wasNearBottom, { behavior: "auto" }); + } else { + // 如果 DOM 顺序被打乱，强制挪回 tool-card 后面 + try { + if (toolCardDom && toolCardDom.wrap && dom.wrap && toolCardDom.wrap.nextSibling !== dom.wrap) { + toolCardDom.wrap.after(dom.wrap); + } + } catch {} + } + + this._renderToolMediaPreview({ preview: dom.preview, details: null }, merged); + } + + + _renderToolMediaPreview(dom, merged) { + if (!dom || !dom.preview) return; + + const st = this._normToolState(merged?.state); + const summary = merged?.summary; + + // running 且无 summary：清空，避免复用上一轮残留 + if (st === "running" && summary == null) { + dom.preview.innerHTML = ""; + dom.preview._lastMediaKey = ""; + return; + } + + if (summary == null) { + dom.preview.innerHTML = ""; + dom.preview._lastMediaKey = ""; + return; + } + + const lang = __osNormLang(window.OPENSTORYLINE_LANG || "zh"); + + let key = ""; + try { + key = (typeof summary === "string") ? summary : JSON.stringify(summary); + } catch { + key = String(summary); + } + + const combinedKey = `${lang}::${key}`; + if (dom.preview._lastMediaKey === combinedKey) return; + dom.preview._lastMediaKey = combinedKey; + + const media = this._extractMediaItemsFromSummary(summary); + if (!media.length) { + dom.preview.innerHTML = ""; + return; + } + + const toolName = String(merged?.name || "").toLowerCase(); + const toolFull = String(this._toolFullName(merged?.server, merged?.name) || "").toLowerCase(); + + const isSplitShots = toolName.includes("split_shots") || toolFull.includes("split_shots"); + const isRender = toolName.includes("render") || toolFull.includes("render"); + const isTtsOrMusic = + toolName.includes("tts") || toolFull.includes("tts") || + toolName.includes("music") || toolFull.includes("music"); + + const videos = media.filter((x) => x.kind === "video"); + const audios = media.filter((x) => x.kind === "audio"); + const images = media.filter((x) => x.kind === "image"); + + dom.preview.innerHTML = ""; + + // Render：成片直接内嵌展示（第一条 video） + if (isRender && videos.length) { + dom.preview.appendChild(this._makeInlineVideoBlock(videos[0], __t("tool.preview.render_title"))); + + const restVideos = videos.slice(1); + if (restVideos.length) { + dom.preview.appendChild(this._makeMediaGridBlock(restVideos, { + title: __t("tool.preview.other_videos"), + kind: "video", + labelKey: "tool.preview.label.video", + maxItems: 8, + })); + } + + if (audios.length) { + dom.preview.appendChild(this._makeAudioListBlock(audios, __t("tool.preview.audio"))); + } + + if (images.length) { + dom.preview.appendChild(this._makeMediaGridBlock(images, { + title: __t("tool.preview.images"), + kind: "image", + labelKey: "tool.preview.label.image", + maxItems: 12, + })); + } + + // 关键节点：完成后默认展开，做到“直接展示成片” + if (st !== "running" && dom.details) dom.details.open = true; + return; + } + + // 配音/音乐：优先展示试听 + if (isTtsOrMusic && audios.length) { + dom.preview.appendChild(this._makeAudioListBlock(audios, __t("tool.preview.listen"))); + if (st !== "running" && dom.details) dom.details.open = true; + } + + // 镜头切分：展示切分后视频（可点击弹窗预览） + if (videos.length) { + dom.preview.appendChild(this._makeMediaGridBlock(videos, { + title: isSplitShots ? __t("tool.preview.split_shots") : __t("tool.preview.videos"), + kind: "video", + labelKey: isSplitShots ? "tool.preview.label.shot" : "tool.preview.label.video", + maxItems: isSplitShots ? 12 : 8, + })); + if (isSplitShots && st !== "running" && dom.details) dom.details.open = true; + } + + // 图片 + if (images.length) { + dom.preview.appendChild(this._makeMediaGridBlock(images, { + title: __t("tool.preview.images"), + kind: "image", + labelKey: "tool.preview.label.image", + maxItems: 12, + })); + } + + // 其它工具也可能产生音频：给一个通用展示 + if (!isTtsOrMusic && audios.length) { + dom.preview.appendChild(this._makeAudioListBlock(audios, __t("tool.preview.audio"))); + } + } + + _isLikelyLocalPath(s) { + s = String(s ?? "").trim(); + if (!s) return false; + // 相对路径：.xxx 或 xxx/yyy；绝对路径：/xxx/yyy + if (s.startsWith(".") || s.startsWith("/")) return true; + // windows 盘符（可选兜底） + if (/^[a-zA-Z]:[\\/]/.test(s)) return true; + return false; + } + + + + // 只认为“显式 scheme”的才是网络 URL，避免把 .server_cache/... 误判成 http(s) 相对 URL + _isAbsoluteNetworkUrl(s) { + s = String(s ?? "").trim().toLowerCase(); + return s.startsWith("http://") || s.startsWith("https://") || s.startsWith("blob:"); + } + + // 已经是你服务端可直接访问的相对路径（不要再走 preview 代理） + _isServedRelativeUrlPath(s) { + s = String(s ?? "").trim(); + return s.startsWith("/api/") || s.startsWith("/static/"); + } + + // 判断“服务器本地路径” + // - .server_cache/.. + // - ./xxx/.. + // - /abs/path/.. （但排除 /api/, /static/） + // - windows: C:\... + // - 其它不带 scheme 且包含 / 或 \ 的相对路径（例如 outputs/xxx.mp4） + _isLikelyServerLocalPath(s) { + s = String(s ?? "").trim(); + if (!s) return false; + + if (this._isServedRelativeUrlPath(s)) return false; // 已可访问 + + if (/^[a-zA-Z]:[\\/]/.test(s)) return true; // Windows drive + if (s.startsWith(".") || s.startsWith("./") || s.startsWith(".\\")) return true; + + if (s.startsWith("/")) return true; // 绝对路径（同样排除 /api,/static 已在上面处理） + + // 没 scheme，但像路径（含斜杠） + if (!this._isAbsoluteNetworkUrl(s) && (s.includes("/") || s.includes("\\"))) return true; + + return false; + } + + _localPathToPreviewUrl(p) { + const sid = this._sessionId; + if (!sid) return null; + return `/api/sessions/${encodeURIComponent(sid)}/preview?path=${encodeURIComponent(String(p ?? ""))}`; + } + + // 将 preview_urls 里的 raw 字符串转为真正可在浏览器加载的 URL + _normalizePreviewUrl(raw) { + const s = String(raw ?? "").trim(); + if (!s) return null; + + // 1) 已可访问的相对 URL + if (this._isServedRelativeUrlPath(s)) return s; + + // 2) 显式网络 URL + if (this._isAbsoluteNetworkUrl(s)) return s; + + // 3) 本地路径 -> preview 代理 + if (this._isLikelyServerLocalPath(s)) return this._localPathToPreviewUrl(s); + + return null; + } + + + openPreview(media) { + if (!this._modalBound) this.bindModalClose(); + + this.modalContent.innerHTML = ""; + this.modalEl.classList.remove("hidden"); + + const preferSrc = media.local_url || media.file_url; + + if (media.kind === "image") { + const img = document.createElement("img"); + img.src = preferSrc; + img.alt = media.name || ""; + this.modalContent.appendChild(img); + return; + } + + if (media.kind === "video") { + const v = document.createElement("video"); + v.src = preferSrc; + v.controls = true; + v.autoplay = true; + v.preload = "metadata"; + this.modalContent.appendChild(v); + return; + } + + if (media.kind === "audio") { + const a = document.createElement("audio"); + a.src = preferSrc; + a.controls = true; + a.autoplay = true; + a.preload = "metadata"; + this.modalContent.appendChild(a); + return; + } + + const box = document.createElement("div"); + box.className = "file-fallback"; + + const pad = document.createElement("div"); + pad.style.padding = "16px"; + + const tip = document.createElement("div"); + tip.style.color = "rgba(0,0,0,0.75)"; + tip.style.marginBottom = "8px"; + tip.textContent = __t("preview.unsupported"); + pad.appendChild(tip); + + const name = document.createElement("div"); + name.style.fontFamily = "ui-monospace,monospace"; + name.style.fontSize = "12px"; + name.style.marginBottom = "12px"; + name.textContent = media.name || media.id || ""; + pad.appendChild(name); + + const link = document.createElement("a"); + link.href = media.file_url || preferSrc || "#"; + link.target = "_blank"; + link.rel = "noopener"; + link.textContent = __t("preview.open_download"); + pad.appendChild(link); + + box.appendChild(pad); + this.modalContent.appendChild(box); + } + + closePreview() { + this.modalEl.classList.add("hidden"); + this.modalContent.innerHTML = ""; + } + + rerenderToolMediaPreviews() { + if (!this.toolMediaDomById) return; + + for (const [tool_call_id, mediaDom] of this.toolMediaDomById) { + const toolDom = this.toolDomById && this.toolDomById.get(tool_call_id); + const merged = toolDom && toolDom.data; + if (!mediaDom || !mediaDom.preview || !merged) continue; + + this._renderToolMediaPreview({ preview: mediaDom.preview, details: null }, merged); + } + } + + + bindModalClose() { + // 防止重复绑定（openPreview 里也会兜底调用一次） + if (this._modalBound) return; + this._modalBound = true; + + const close = (e) => { + if (e) { + e.preventDefault(); + e.stopPropagation(); + // 同一元素上其它监听也停掉，避免“关闭后又被底层点击重新打开” + if (typeof e.stopImmediatePropagation === "function") e.stopImmediatePropagation(); + } + this.closePreview(); + }; + + // 1) 明确绑定 backdrop/close + if (this.modalBackdrop) { + this.modalBackdrop.addEventListener("click", close, true); // capture + this.modalBackdrop.addEventListener("pointerdown", close, true); // 兼容移动端/某些浏览器 + } + if (this.modalClose) { + this.modalClose.addEventListener("click", close, true); + this.modalClose.addEventListener("pointerdown", close, true); + } + + // 2) 兜底：document capture 里判断“点到内容区外”就关闭 + document.addEventListener("click", (e) => { + if (!this.modalEl || this.modalEl.classList.contains("hidden")) return; + + const t = e.target; + + // 点到 close（或其子元素） => 关闭 + if (this.modalClose && (t === this.modalClose || this.modalClose.contains(t))) { + close(e); + return; + } + + // 点到内容区内部 => 不关闭（允许操作 video controls/滚动等） + if (this.modalContent && (t === this.modalContent || this.modalContent.contains(t))) { + return; + } + + // 其他任何地方（含 click 穿透到页面底层）=> 关闭 + close(e); + }, true); + + // 3) Esc 关闭 + document.addEventListener("keydown", (e) => { + if (!this.modalEl || this.modalEl.classList.contains("hidden")) return; + if (e.key === "Escape") { + e.preventDefault(); + this.closePreview(); + } + }, true); + } + + + escapeHtml(s) { + return String(s).replace(/[&<>"']/g, (c) => ({ + "&":"&","<":"<",">":">",'"':""","'":"'" + }[c])); + } +} + +class App { + constructor() { + this.api = new ApiClient(); + this.ui = new ChatUI(); + this.ws = null; + + this.sessionId = null; + this.pendingMedia = []; + + this.llmSelect = $("#llmModelSelect"); + this.vlmSelect = $("#vlmModelSelect"); + + this.llmModels = []; + this.vlmModels = []; + + this.llmModel = null; + this.vlmModel = null; + + // custom model section + this.customLlmSection = $("#customLlmSection"); + this.customVlmSection = $("#customVlmSection"); + + // Custom model UI + this.customLlmModel = $("#customLlmModel"); + this.customLlmBaseUrl = $("#customLlmBaseUrl"); + this.customLlmApiKey = $("#customLlmApiKey"); + this.customVlmModel = $("#customVlmModel"); + this.customVlmBaseUrl = $("#customVlmBaseUrl"); + this.customVlmApiKey = $("#customVlmApiKey"); + + // TTS UI + this.ttsBox = $("#ttsBox"); + this.ttsProviderSelect = $("#ttsProviderSelect"); + this.ttsProviderFieldsHost = $("#ttsProviderFields"); + this.ttsUiSchema = null; + + // Pexels UI + this.pexelsBox = $("#pexelsBox"); + this.pexelsKeyModeSelect = $("#pexelsKeyModeSelect"); + this.pexelsCustomKeyBox = $("#pexelsCustomKeyBox"); + this.pexelsApiKeyInput = $("#pexelsApiKeyInput"); + + this.limits = { + max_media_per_session: 30, + max_pending_media_per_session: 30, + upload_chunk_bytes: 8 * 1024 * 1024, + }; + + this.localObjectUrlByMediaId = new Map(); + + this.fileInput = $("#fileInput"); + this.uploadBtn = $("#uploadBtn"); + this.promptInput = $("#promptInput"); + this.sendBtn = $("#sendBtn"); + this.quickPromptBtn = $("#quickPromptBtn"); + this._quickPromptIdx = 0; + this.sidebarToggleBtn = $("#sidebarToggle"); + this.createDialogBtn = $("#createDialogBtn"); + this.devbarToggleBtn = $("#devbarToggle"); + this.devbarEl = $("#devbar"); + + this.canceling = false; + + // 保存“发送箭头”的原始 SVG + this._sendIconSend = this.sendBtn ? this.sendBtn.innerHTML : ""; + + // “打断”图标：白色实心正方形 + this._sendIconStop = ` + + `; + + this.streaming = false; + this.uploading = false; + + this.langToggle = $("#langToggle"); + this.lang = __osNormLang(window.OPENSTORYLINE_LANG || "zh"); + + this._langWasStored = (__osLoadLang() != null); + + } + + wsUrl(sessionId) { + const proto = location.protocol === "https:" ? "wss" : "ws"; + return `${proto}://${location.host}/ws/sessions/${encodeURIComponent(sessionId)}/chat`; + } + + async bootstrap() { + // this.restoreSidebarState(); + // this.restoreDevbarState(); + this.ui.bindModalClose(); + this.bindUI(); + this._setLang(this.lang, { persist: false, syncServer: false }); + await this.loadTtsUiSchema(); + + // 复用 localStorage session；如果失效就创建新 session + const saved = localStorage.getItem("openstoryline_session_id"); + if (saved) { + try { + const snap = await this.api.getSession(saved); + await this.useSession(saved, snap); + return; + } catch { + localStorage.removeItem("openstoryline_session_id"); + } + } + + await this.newSession(); + } + + async loadTtsUiSchema() { + let schema = null; + try { + schema = await this.api.getTtsUiSchema(); + } catch (e) { + console.warn("[tts] failed to load /api/meta/tts:", e); + } + + this.ttsUiSchema = schema; + this._renderTtsUiFromSchema(schema); + } + + _renderTtsUiFromSchema(schema) { + if (!this.ttsProviderSelect || !this.ttsProviderFieldsHost) return; + + const providers = (schema && Array.isArray(schema.providers)) ? schema.providers : []; + const before = String(this.ttsProviderSelect.value || "").trim(); + + this.ttsProviderSelect.innerHTML = ""; + const opt0 = document.createElement("option"); + opt0.value = ""; + opt0.textContent = __t("sidebar.tts_default"); + this.ttsProviderSelect.appendChild(opt0); + + for (const v of providers) { + const provider = String(v?.provider || "").trim(); + if (!provider) continue; + const label = String(v?.label || provider); + + const opt = document.createElement("option"); + opt.value = provider; + opt.textContent = label; + this.ttsProviderSelect.appendChild(opt); + } + + this.ttsProviderFieldsHost.innerHTML = ""; + + for (const v of providers) { + const provider = String(v?.provider || "").trim(); + if (!provider) continue; + + const block = document.createElement("div"); + block.className = "sidebar-tts-fields hidden"; + block.dataset.ttsProvider = provider; + + const fields = Array.isArray(v?.fields) ? v.fields : []; + + for (const f of fields) { + const key = String(f?.key || "").trim(); + if (!key) continue; + + const label = String(f?.label || key).trim(); + + const required = !!f?.required; + const secret = !!f?.secret; + + const input = document.createElement("input"); + input.className = "sidebar-input"; + input.type = secret ? "password" : "text"; + input.autocomplete = "off"; + const basePh = String(f?.placeholder || label).trim(); + const needSuffix = !f?.placeholder; // 仅当 schema 没给 placeholder 时，加“留空使用默认”的 suffix + + input.setAttribute("data-os-ph-base", basePh); + input.setAttribute("data-os-ph-suffix", needSuffix ? "1" : "0"); + + const ph = needSuffix ? `${basePh}${__t("sidebar.tts_field_suffix")}` : basePh; + input.placeholder = ph; + + input.setAttribute("data-os-persist", `sidebar.tts.${provider}.${key}`); + + input.dataset.ttsKey = key; + + block.appendChild(input); + } + + this.ttsProviderFieldsHost.appendChild(block); + } + + try { __osHydratePersistedFields(this.ttsBox || document); } catch {} + try { __osBindPersistedFields(this.ttsBox || document); } catch {} + + if (before) { + this.ttsProviderSelect.value = before; + } else { + this.ttsProviderSelect.value = ""; + } + + try { this.ttsProviderSelect.dispatchEvent(new Event("change", { bubbles: true })); } catch {} + } + + // restoreSidebarState() { + // const v = localStorage.getItem(SIDEBAR_COLLAPSED_KEY); + + // if (v === null) { + // // 首次访问：默认收起，并写入本地存储（后续刷新保持一致） + // document.body.classList.add("sidebar-collapsed"); + // localStorage.setItem(SIDEBAR_COLLAPSED_KEY, "1"); + // return; + // } + + // // 已有配置：1 收起，0 展开 + // document.body.classList.toggle("sidebar-collapsed", v === "1"); + // } + + // restoreDevbarState() { + // const v = localStorage.getItem(DEVBAR_COLLAPSED_KEY); + + // if (v === null) { + // // 首次访问：默认收起 + // document.body.classList.add("devbar-collapsed"); + // localStorage.setItem(DEVBAR_COLLAPSED_KEY, "1"); + // return; + // } + + // document.body.classList.toggle("devbar-collapsed", v === "1"); + // } + + _updateSendButtonUI() { + if (!this.sendBtn) return; + + if (this.streaming) { + this.sendBtn.innerHTML = this._sendIconStop; + this.sendBtn.setAttribute("aria-label", "打断"); + this.sendBtn.title = "打断"; + } else { + this.sendBtn.innerHTML = this._sendIconSend; + this.sendBtn.setAttribute("aria-label", "发送"); + this.sendBtn.title = "发送"; + } + } + + async interruptTurn() { + if (!this.sessionId) return; + if (!this.streaming) return; + if (this.canceling) return; + + this.canceling = true; + this._updateComposerDisabledState(); + + try { + await this.api.cancelTurn(this.sessionId); + // 不需要本地立刻 finalize，等后端 assistant.end 来收尾并把上下文写干净 + } catch (e) { + this.canceling = false; + this._updateComposerDisabledState(); + this.ui.showToastI18n("toast.interrupt_failed", { msg: (e && (e.message || e)) || "" }); + setTimeout(() => this.ui.hideToast(), 1600); + } + } + + + toggleDevbar() { + document.body.classList.toggle("devbar-collapsed"); + // const collapsed = document.body.classList.contains("devbar-collapsed"); + // localStorage.setItem(DEVBAR_COLLAPSED_KEY, collapsed ? "1" : "0"); + } + + setDeveloperMode(enabled) { + const on = !!enabled; + const devbar = this.devbarEl || $("#devbar"); + if (!devbar) return; + + if (on) { + document.body.classList.add("dev-mode"); + devbar.classList.remove("hidden"); + } else { + document.body.classList.remove("dev-mode"); + devbar.classList.add("hidden"); + } + } + + toggleSidebar() { + document.body.classList.toggle("sidebar-collapsed"); + // const collapsed = document.body.classList.contains("sidebar-collapsed"); + // localStorage.setItem(SIDEBAR_COLLAPSED_KEY, collapsed ? "1" : "0"); + } + + _setLang(lang, { persist = true, syncServer = true } = {}) { + const v = __osNormLang(lang); + if (!v) return; + + __applyLang(v, { persist }); + + this.lang = v; + if (persist) this._langWasStored = true; + + if (this.langToggle) this.langToggle.checked = (v === "en"); + + this._rerenderLangDynamicBits(); + + if (this.ui && typeof this.ui.rerenderToast === "function") { + this.ui.rerenderToast(); + } + + try { this.ui?.rerenderAssistantPlaceholder?.(); } catch {} + try { this.ui?.rerenderToolCards?.(); } catch {} + try { this.ui?.rerenderToolMediaPreviews?.(); } catch {} + + if (syncServer) this._pushLangToServer(); + } + + _rerenderLangDynamicBits() { + const apply = (sel) => { + if (!sel) return; + const opt = sel.querySelector(`option[value="${CUSTOM_MODEL_KEY}"]`); + if (opt) opt.textContent = __t("sidebar.use_custom_model"); + }; + + apply(this.llmSelect); + apply(this.vlmSelect); + + if (this.ttsProviderSelect) { + const opt0 = this.ttsProviderSelect.querySelector('option[value=""]'); + if (opt0) opt0.textContent = __t("sidebar.tts_default"); + } + + __rerenderTtsFieldPlaceholders(document); + } + + _pushLangToServer() { + if (!this.ws) return; + this.ws.send("session.set_lang", { lang: this.lang }); + } + + applySnapshotLimits(snapshot) { + const lim = (snapshot && snapshot.limits) ? snapshot.limits : {}; + const toInt = (v, d) => { + const n = Number(v); + return Number.isFinite(n) && n > 0 ? n : d; + }; + + this.limits = { + max_media_per_session: toInt(lim.max_media_per_session, this.limits.max_media_per_session || 30), + max_pending_media_per_session: toInt(lim.max_pending_media_per_session, this.limits.max_pending_media_per_session || 30), + upload_chunk_bytes: toInt(lim.upload_chunk_bytes, this.limits.upload_chunk_bytes || (8 * 1024 * 1024)), + }; + } + + applySnapshotModels(snapshot) { + const llmModels = + (snapshot && Array.isArray(snapshot.llm_models)) ? snapshot.llm_models : + (snapshot && Array.isArray(snapshot.chat_models)) ? snapshot.chat_models : []; + + const llmCurrent = + (snapshot && typeof snapshot.llm_model_key === "string") ? snapshot.llm_model_key : + (snapshot && typeof snapshot.chat_model_key === "string") ? snapshot.chat_model_key : ""; + + const vlmModels = (snapshot && Array.isArray(snapshot.vlm_models)) ? snapshot.vlm_models : []; + const vlmCurrent = (snapshot && typeof snapshot.vlm_model_key === "string") ? snapshot.vlm_model_key : ""; + + // 确保至少有一个选项 + const llmList = (llmModels && llmModels.length) ? llmModels.slice() : (llmCurrent ? [llmCurrent] : []); + const vlmList = (vlmModels && vlmModels.length) ? vlmModels.slice() : (vlmCurrent ? [vlmCurrent] : []); + + this.llmModels = llmList; + this.vlmModels = vlmList; + + // render LLM select + if (this.llmSelect) { + this.llmSelect.innerHTML = ""; + for (const m of llmList) { + const opt = document.createElement("option"); + opt.value = m; + opt.textContent = (m === CUSTOM_MODEL_KEY) ? __t("sidebar.use_custom_model") : m; + this.llmSelect.appendChild(opt); + } + let selected = ""; + if (llmCurrent && llmList.includes(llmCurrent)) selected = llmCurrent; + else if (llmList.length) selected = llmList[0]; + this.llmModel = selected || null; + if (this.llmModel) this.llmSelect.value = this.llmModel; + } + + // render VLM select + if (this.vlmSelect) { + this.vlmSelect.innerHTML = ""; + for (const m of vlmList) { + const opt = document.createElement("option"); + opt.value = m; + opt.textContent = (m === CUSTOM_MODEL_KEY) ? __t("sidebar.use_custom_model") : m; + this.vlmSelect.appendChild(opt); + } + let selected = ""; + if (vlmCurrent && vlmList.includes(vlmCurrent)) selected = vlmCurrent; + else if (vlmList.length) selected = vlmList[0]; + this.vlmModel = selected || null; + if (this.vlmModel) this.vlmSelect.value = this.vlmModel; + } + + this._syncConfigPanels(); + } + + + _syncConfigPanels() { + const llmCustom = (this.llmModel === CUSTOM_MODEL_KEY); + const vlmCustom = (this.vlmModel === CUSTOM_MODEL_KEY); + + if (this.customLlmSection) this.customLlmSection.classList.toggle("hidden", !llmCustom); + if (this.customVlmSection) this.customVlmSection.classList.toggle("hidden", !vlmCustom); + + const provider = (this.ttsProviderSelect && this.ttsProviderSelect.value) + ? String(this.ttsProviderSelect.value).trim() + : ""; + + const host = this.ttsProviderFieldsHost || $("#ttsProviderFields"); + if (host) { + host.querySelectorAll("[data-tts-provider]").forEach((el) => { + const v = String(el.dataset.ttsProvider || ""); + el.classList.toggle("hidden", !provider || v !== provider); + }); + } + + // ---- Pexels custom key show/hide ---- + const pMode = (this.pexelsKeyModeSelect && this.pexelsKeyModeSelect.value) + ? String(this.pexelsKeyModeSelect.value).trim() + : "default"; + + const showCustomPexels = (pMode === "custom"); + if (this.pexelsCustomKeyBox) this.pexelsCustomKeyBox.classList.toggle("hidden", !showCustomPexels); + } + + + _readCustomModelsFromUI() { + const s = (x) => String(x ?? "").trim(); + return { + llm: { + model: s(this.customLlmModel?.value), + base_url: s(this.customLlmBaseUrl?.value), + api_key: s(this.customLlmApiKey?.value), + }, + vlm: { + model: s(this.customVlmModel?.value), + base_url: s(this.customVlmBaseUrl?.value), + api_key: s(this.customVlmApiKey?.value), + }, + }; + } + + _validateCustomModels(cfg, { needLlm = false, needVlm = false } = {}) { + const llm = cfg?.llm || {}; + const vlm = cfg?.vlm || {}; + const miss = (x) => !x || !String(x).trim(); + + if (needLlm && (miss(llm.model) || miss(llm.base_url) || miss(llm.api_key))) { + return "custom llm config is incomplete: please fill in model/base_url/api_key"; + } + if (needVlm && (miss(vlm.model) || miss(vlm.base_url) || miss(vlm.api_key))) { + return "custom vlm config is incomplete: please fill in model/base_url/api_key"; + } + return ""; + } + + + _readTtsConfigFromUI() { + const provider = (this.ttsProviderSelect && this.ttsProviderSelect.value) + ? String(this.ttsProviderSelect.value).trim() + : ""; + if (!provider) return null; + + const host = this.ttsProviderFieldsHost || $("#ttsProviderFields"); + const params = {}; + + if (host) { + const block = host.querySelector(`[data-tts-provider="${provider}"]`); + if (block) { + const fields = block.querySelectorAll("input[data-tts-key], select[data-tts-key], textarea[data-tts-key]"); + fields.forEach((el) => { + const k = String(el.dataset.ttsKey || "").trim(); + if (!k) return; + const v = String(el.value ?? "").trim(); + if (v !== "") params[k] = v; + }); + } + } + + // 统一 payload：{ provider, :{...} } + const out = { provider }; + out[provider] = params; // 允许为空 {} + return out; + } + + _readPexelsConfigFromUI() { + if (!this.pexelsKeyModeSelect) return null; + + const modeRaw = String(this.pexelsKeyModeSelect.value || "").trim(); + const mode = (modeRaw === "custom") ? "custom" : "default"; + + let api_key = ""; + if (mode === "custom" && this.pexelsApiKeyInput) { + api_key = String(this.pexelsApiKeyInput.value || "").trim(); + } + + return { mode, api_key }; + } + + + _makeChatSendPayload(text, attachment_ids) { + const payload = { text, attachment_ids, lang: this.lang || "zh" }; + + if (this.llmModel) payload.llm_model = this.llmModel; + if (this.vlmModel) payload.vlm_model = this.vlmModel; + + const rc = {}; + + const needLlmCustom = (this.llmModel === CUSTOM_MODEL_KEY); + const needVlmCustom = (this.vlmModel === CUSTOM_MODEL_KEY); + + if (needLlmCustom || needVlmCustom) { + const cm = this._readCustomModelsFromUI(); + const err = this._validateCustomModels(cm, { needLlm: needLlmCustom, needVlm: needVlmCustom }); + if (err) return { error: err }; + + rc.custom_models = {}; + if (needLlmCustom) rc.custom_models.llm = cm.llm; + if (needVlmCustom) rc.custom_models.vlm = cm.vlm; + } + + const tts = this._readTtsConfigFromUI(); + if (tts) rc.tts = tts; + + const pexels = this._readPexelsConfigFromUI(); + if (pexels) { + rc.search_media = { pexels }; + } + + if (Object.keys(rc).length) payload.service_config = rc; + return { payload }; + } + + + setChatModel(model) { + const m = String(model || "").trim(); + if (!m) return; + this.chatModel = m; + } + + + clearLocalObjectUrls() { + for (const [, url] of this.localObjectUrlByMediaId) { + try { URL.revokeObjectURL(url); } catch {} + } + this.localObjectUrlByMediaId.clear(); + } + + bindLocalUrlsToMedia(list) { + const arr = Array.isArray(list) ? list : []; + return arr.map((a) => { + const url = a && a.id ? this.localObjectUrlByMediaId.get(a.id) : null; + return url ? { ...a, local_url: url } : a; + }); + } + + revokeLocalUrl(mediaId) { + const url = this.localObjectUrlByMediaId.get(mediaId); + if (url) { + try { URL.revokeObjectURL(url); } catch {} + this.localObjectUrlByMediaId.delete(mediaId); + } + } + + _updateComposerDisabledState() { + // - streaming=true：sendBtn 是“打断键”，必须可点（除非正在 canceling） + // - streaming=false：uploading=true 时不能发送 => 禁用 + const disableSend = this.canceling ? true : (!this.streaming && this.uploading); + if (this.sendBtn) this.sendBtn.disabled = disableSend; + + if (this.uploadBtn) this.uploadBtn.disabled = !!this.uploading; + + this._updateSendButtonUI(); + } + + _autosizePrompt() { + const el = this.promptInput; + if (!el) return; + + // 读取 CSS 的 max-height（比如 180px），读不到就 fallback + const cs = window.getComputedStyle(el); + const mh = parseFloat(cs.maxHeight); + const maxPx = Number.isFinite(mh) && mh > 0 ? mh : 180; + + // 先让它回到 auto，才能正确拿到 scrollHeight + el.style.height = "auto"; + + const next = Math.min(el.scrollHeight, maxPx); + el.style.height = next + "px"; + + // 没超过上限：隐藏滚动条；超过上限：出现滚动条 + el.style.overflowY = (el.scrollHeight > maxPx) ? "auto" : "hidden"; + } + + _nextQuickPromptText() { + const list = Array.isArray(QUICK_PROMPTS) ? QUICK_PROMPTS : []; + if (!list.length) return ""; + + const idx = (Number(this._quickPromptIdx) || 0) % list.length; + this._quickPromptIdx = idx + 1; + + const item = list[idx]; + const lang = __osNormLang(this.lang || "zh"); + + if (typeof item === "string") return item.trim(); + if (item && typeof item === "object") { + const v = item[lang] ?? item.zh ?? item.en ?? ""; + return String(v || "").trim(); + } + return String(item ?? "").trim(); + } + + _insertIntoPrompt(text) { + const el = this.promptInput; + const insertText = String(text || "").trim(); + if (!el || !insertText) return; + + const cur = String(el.value || ""); + + if (!cur.trim()) { + el.value = insertText; + try { el.setSelectionRange(el.value.length, el.value.length); } catch {} + el.focus(); + this._autosizePrompt(); + return; + } + + const start = (typeof el.selectionStart === "number") ? el.selectionStart : cur.length; + const end = (typeof el.selectionEnd === "number") ? el.selectionEnd : cur.length; + + const before = cur.slice(0, start); + const after = cur.slice(end); + + const isCollapsed = start === end; + const isAtEnd = isCollapsed && end === cur.length; + + const sep = (isAtEnd && before && !before.endsWith("\n")) ? "\n" : ""; + + el.value = before + sep + insertText + after; + + const caret = (before + sep + insertText).length; + try { el.setSelectionRange(caret, caret); } catch {} + + el.focus(); + this._autosizePrompt(); + } + + bindUI() { + // sidebar + if (this.sidebarToggleBtn) { + this.sidebarToggleBtn.addEventListener("click", () => this.toggleSidebar()); + } + if (this.createDialogBtn) { + this.createDialogBtn.addEventListener("click", () => this.newSession()); + } + + if (this.llmSelect) { + this.llmSelect.addEventListener("change", () => { + const v = (this.llmSelect.value || "").trim(); + if (v) this.llmModel = v; + this._syncConfigPanels(); + }); + } + + if (this.vlmSelect) { + this.vlmSelect.addEventListener("change", () => { + const v = (this.vlmSelect.value || "").trim(); + if (v) this.vlmModel = v; + this._syncConfigPanels(); + }); + } + + if (this.ttsProviderSelect) { + this.ttsProviderSelect.addEventListener("change", () => this._syncConfigPanels()); + } + + if (this.pexelsKeyModeSelect) { + this.pexelsKeyModeSelect.addEventListener("change", () => this._syncConfigPanels()); + } + + // devbar toggle（仅 developer_mode=true 时 devbar 会显示） + if (this.devbarToggleBtn) { + this.devbarToggleBtn.addEventListener("click", () => this.toggleDevbar()); + } + + // uploader + this.uploadBtn.addEventListener("click", () => this.fileInput.click()); + + this.fileInput.addEventListener("change", async () => { + let files = Array.from(this.fileInput.files || []); + this.fileInput.value = ""; + if (!files.length) return; + + // 会话内 pending 上限 + const maxPending = Number(this.limits.max_pending_media_per_session || 30); + const remain = Math.max(0, maxPending - (this.pendingMedia.length || 0)); + if (remain <= 0) { + this.ui.showToastI18n("toast.pending_limit", { max: maxPending }); + setTimeout(() => this.ui.hideToast(), 1600); + return; + } + if (files.length > remain) { + this.ui.showToastI18n("toast.pending_limit_partial", { remain, max: maxPending }); + setTimeout(() => this.ui.hideToast(), 1400); + files = files.slice(0, remain); + } + + const totalBytes = Math.max(1, files.reduce((s, f) => s + (f.size || 0), 0)); + let confirmedBytesAll = 0; + + this.uploading = true; + this._updateComposerDisabledState(); + + try { + this.ui.showToastI18n("toast.uploading", { pct: 0 }); + + // 分片 + for (let i = 0; i < files.length; i++) { + const f = files[i]; + + // 预先创建 ObjectURL（用于 (3) 预览走本地缓存） + const localUrl = URL.createObjectURL(f); + + try { + const resp = await this.api.uploadMediaChunked(this.sessionId, f, { + chunkSize: this.limits.upload_chunk_bytes, + onProgress: (loadedInFile, fileTotal) => { + const overallLoaded = Math.min(totalBytes, confirmedBytesAll + (loadedInFile || 0)); + const pct = Math.round((overallLoaded / totalBytes) * 100); + this.ui.showToastI18n("toast.uploading_file", { i: i + 1, n: files.length, name: f.name, pct }); + }, + }); + + // 上传完成：把 media_id -> localUrl 绑定起来 + if (resp && resp.media && resp.media.id) { + this.localObjectUrlByMediaId.set(resp.media.id, localUrl); + } else { + // 理论不应发生；发生就释放 + try { URL.revokeObjectURL(localUrl); } catch {} + } + + confirmedBytesAll += (f.size || 0); + + // pending 更新（绑定 local_url 后再渲染） + this.setPending((resp && resp.pending_media) ? resp.pending_media : []); + } catch (e) { + // 本文件失败：释放 URL，避免泄漏 + try { URL.revokeObjectURL(localUrl); } catch {} + throw e; + } + } + + this.ui.hideToast(); + } catch (e) { + this.ui.hideToast(); + this.ui.showToastI18n("toast.upload_failed", { msg: (e && (e.message || e)) || "" }); + setTimeout(() => this.ui.hideToast(), 1800); + } finally { + this.uploading = false; + this._updateComposerDisabledState(); + } + }); + + + // pending 删除：用事件委托 + $("#pendingRow").addEventListener("click", async (e) => { + const el = e.target; + if (!el.classList.contains("media-remove")) return; + const mediaId = el.dataset.mediaId; + if (!mediaId) return; + + try { + const resp = await this.api.deletePendingMedia(this.sessionId, mediaId); + this.revokeLocalUrl(mediaId); + this.setPending(resp.pending_media || []); + } catch (err) { + this.ui.showToastI18n("toast.delete_failed", { msg: (err && (err.message || err)) || "" }); + setTimeout(() => this.ui.hideToast(), 1600); + } + }); + + // send + this.sendBtn.addEventListener("click", () => this.sendPrompt({ source: "button" })); + this.promptInput.addEventListener("keydown", (e) => { + // 避免中文输入法“正在组词/选词”时按 Enter 误触发发送 + if (e.isComposing || e.keyCode === 229) return; + + if (e.key === "Enter" && !e.shiftKey) { + e.preventDefault(); + this.sendPrompt({ source: "enter" }); + } + }); + + //quick prompt fill + if (this.quickPromptBtn && !this._quickPromptBound) { + this._quickPromptBound = true; + + this.quickPromptBtn.addEventListener("click", (e) => { + e.preventDefault(); + + const t = this._nextQuickPromptText(); + if (!t) return; + + this.promptInput.value = t; + this._autosizePrompt(); + this.promptInput.focus(); + try { this.promptInput.setSelectionRange(t.length, t.length); } catch {} + + this.quickPromptBtn.classList.add("is-active"); + setTimeout(() => this.quickPromptBtn.classList.remove("is-active"), 160); + }); + } + + // PATCH: prompt 自动长高 + if (this.promptInput && !this._promptAutoResizeBound) { + this._promptAutoResizeBound = true; + + const resize = () => this._autosizePrompt(); + this.promptInput.addEventListener("input", resize); + window.addEventListener("resize", resize, { passive: true }); + + // 首次初始化/切换会话后确保高度正确 + requestAnimationFrame(resize); + } + + // lang toggle + if (this.langToggle) { + this.langToggle.checked = (this.lang === "en"); + + this.langToggle.addEventListener("change", () => { + const next = this.langToggle.checked ? "en" : "zh"; + this._setLang(next, { persist: true, syncServer: true }); + }); + } + } + + setPending(list) { + const arr = this.bindLocalUrlsToMedia(Array.isArray(list) ? list : []); + this.pendingMedia = arr; + this.ui.renderPendingMedia(this.pendingMedia); + } + + async newSession() { + const snap = await this.api.createSession(); + await this.useSession(snap.session_id, snap); + } + + async useSession(sessionId, snapshot) { + this.streaming = false; + this.uploading = false; + this._updateComposerDisabledState(); + + this.sessionId = sessionId; + + const snapLang = snapshot && snapshot.lang; + if (!this._langWasStored && snapLang) { + this._setLang(snapLang, { persist: true, syncServer: false }); + } else { + this._setLang(this.lang, { persist: false, syncServer: false }); + } + + // 切会话：清掉上一会话的本地缓存 URL，避免泄漏 + this.clearLocalObjectUrls(); + + // 从后端 snapshot 读取 limits（按素材个数限制/分片大小等） + this.applySnapshotLimits(snapshot); + this.applySnapshotModels(snapshot); + + localStorage.setItem("openstoryline_session_id", sessionId); + + this.setDeveloperMode(!!snapshot.developer_mode); + + this.ui.setSessionId(sessionId); + this.ui.clearAll(); + + // 回放 history + const history = snapshot.history || []; + for (const item of history) { + if (item.role === "user") { + this.ui.appendUserMessage(item.content || "", item.attachments || []); + } else if (item.role === "assistant") { + this.ui.startAssistantMessage({placeholder: false}); + this.ui.finalizeAssistant(item.content || ""); + } else if (item.role === "tool") { + this.ui.upsertToolCard(item.tool_call_id, { + server: item.server, + name: item.name, + state: item.state, + args: item.args, + progress: item.progress, + message: item.message, + summary: item.summary, + }); + + if (item.summary != null) { + this.ui.appendDevSummary(item.tool_call_id, { + server: item.server, + name: item.name, + summary: item.summary, + is_error: item.state === "error", + }); + } + } + } + + this.setPending(snapshot.pending_media || []); + this.connectWs(); + } + + connectWs() { + if (this.ws) this.ws.close(); + + this.ws = new WsClient(this.wsUrl(this.sessionId), (evt) => this.onWsEvent(evt)); + this.ws.connect(); + } + + onWsEvent(evt) { + const { type, data } = evt || {}; + if (type === "session.snapshot") { + // 一般用不上（useSession 已经回放了），但保留兼容 + this.setDeveloperMode(!!data.developer_mode); + this.ui.setSessionId(data.session_id); + this.applySnapshotModels(data || {}); + + const serverLang = data && data.lang; + const sv = __osNormLang(serverLang); + if (sv && sv !== this.lang) { + if (this._langWasStored) { + this._pushLangToServer(); + } else { + this._setLang(sv, { persist: true, syncServer: false }); + } + } + + this.setPending(data.pending_media || []); + return; + } + + if (type === "chat.user") { + // 以服务端为准更新 pending（避免客户端/服务端状态漂移） + this.setPending((data || {}).pending_media || []); + return; + } + + if (type === "assistant.start") { + this.streaming = true; + this._updateComposerDisabledState(); + this.ui.startAssistantMessage({placeholder: true}); + return; + } + + if (type === "assistant.flush") { + this.ui.flushAssistantSegment(); + return; + } + + if (type === "assistant.delta") { + this.ui.appendAssistantDelta((data || {}).delta || ""); + return; + } + + if (type === "assistant.end") { + this.streaming = false; + this.canceling = false; + this._updateComposerDisabledState(); + this.ui.endAssistantTurn((data || {}).text || ""); + return; + } + + if (type === "tool.start") { + this.ui.upsertToolCard(data.tool_call_id, { + server: data.server, + name: data.name, + state: "running", + args: data.args || {}, + progress: 0, + }); + return; + } + + if (type === "tool.progress") { + this.ui.upsertToolCard(data.tool_call_id, { + server: data.server, + name: data.name, + state: "running", + progress: typeof data.progress === "number" ? data.progress : 0, + message: data.message || "", + __progress_mode: "real", + }); + return; + } + + if (type === "tool.end") { + this.ui.upsertToolCard(data.tool_call_id, { + server: data.server, + name: data.name, + state: data.is_error ? "error" : "success", + summary: (data && Object.prototype.hasOwnProperty.call(data, "summary")) ? data.summary : null, + }); + this.ui.appendDevSummary(data.tool_call_id, { + server: data.server, + name: data.name, + summary: data.summary, + is_error: !!data.is_error, + }); + return; + } + + if (type === "chat.cleared") { + this.streaming = false; + this.canceling = false; + this._updateComposerDisabledState(); + this.ui.clearAll(); + return; + } + + if (type === "error") { + this.streaming = false; + this.canceling = false; + this._updateComposerDisabledState(); + + const msg = String((data || {}).message || "unknown error"); + const partial = String((data || {}).partial_text || "").trim(); + + // 用 endAssistantTurn 结束当前流式气泡： + // - 有 partial：保留已输出内容，并追加错误说明 + // - 无 partial：直接显示错误 + const text = partial + ? `${partial}\n\n（发生错误：${msg}）` + : `发生错误：${msg}`; + + this.ui.endAssistantTurn(text); + return; + } + } + + sendPrompt({ source = "button" } = {}) { + if (!this.ws) return; + + const text = (this.promptInput.value || "").trim(); + + if (this.streaming) { + // Enter 防误触：输入为空 -> 不打断、不发送 + if (source === "enter" && !text) { + return; + } + + // Enter 且有文本：打断 + 发送新 prompt + if (source === "enter" && text) { + if (this.canceling) return; + + // 上传中提示并仅打断（让旧输出停掉），等用户上传完再回车发送 + if (this.uploading) { + this.ui.showToastI18n("toast.uploading_interrupt_send", {}); + setTimeout(() => this.ui.hideToast(), 1600); + this.interruptTurn(); // 有意图（非空）=> 仍然打断 + return; + } + + const attachments = this.pendingMedia.slice(); + const attachment_ids = attachments.map(a => a.id); + + // 1) 立即在 UI 插入 user 气泡（体验更顺滑） + this.ui.appendUserMessage(text, attachments); + this.setPending([]); + + // 2) 清空输入框 + this.promptInput.value = ""; + this._autosizePrompt(); + + // 3) 触发打断（异步，不 await） + this.interruptTurn(); + + // 4) 立即把新消息发到 WS（服务器会在旧 turn 结束后按序处理） + const built = this._makeChatSendPayload(text, attachment_ids); + if (built.error) { + this.ui.showToast(built.error); + setTimeout(() => this.ui.hideToast(), 1800); + return; + } + this.ws.send("chat.send", built.payload); + + return; + } + + // 其它情况（按钮点击/停止图标）：打断 + this.interruptTurn(); + return; + } + + // ----------------------------- + // 非 streaming：正常发送 + // ----------------------------- + if (this.uploading) { + this.ui.showToastI18n("toast.uploading_cannot_send", {}); + setTimeout(() => this.ui.hideToast(), 1400); + return; + } + + if (!text) return; + + const attachments = this.pendingMedia.slice(); + const attachment_ids = attachments.map(a => a.id); + + this.ui.appendUserMessage(text, attachments); + this.setPending([]); + + this.promptInput.value = ""; + this._autosizePrompt(); + + const built = this._makeChatSendPayload(text, attachment_ids); + if (built.error) { + this.ui.showToast(built.error); + setTimeout(() => this.ui.hideToast(), 1800); + return; + } + this.ws.send("chat.send", built.payload); + } + +} + +new App().bootstrap(); +/* ========================================================= + PATCH (mobile viewport / keyboard safe area) + - updates CSS vars: --kb, --composer-h, --vvh + ========================================================= */ +(function () { + const root = document.documentElement; + const composer = document.querySelector(".composer"); + if (!root || !composer) return; + + let raf = 0; + + const compute = () => { + raf = 0; + + const vv = window.visualViewport; + const layoutH = window.innerHeight || document.documentElement.clientHeight || 0; + + const vvH = vv ? vv.height : layoutH; + const vvTop = vv ? vv.offsetTop : 0; + + // Keyboard overlay height (0 on most desktops) + const kb = vv ? Math.max(0, layoutH - (vvH + vvTop)) : 0; + + root.style.setProperty("--vvh", `${Math.round(vvH)}px`); + root.style.setProperty("--kb", `${Math.round(kb)}px`); + + const ch = composer.getBoundingClientRect().height || 0; + if (ch > 0) root.style.setProperty("--composer-h", `${Math.round(ch)}px`); + }; + + const schedule = () => { + if (raf) return; + raf = requestAnimationFrame(compute); + }; + + compute(); + + // Window resize / orientation + window.addEventListener("resize", schedule, { passive: true }); + window.addEventListener("orientationchange", schedule, { passive: true }); + + // iOS: focusing inputs changes visual viewport + document.addEventListener("focusin", schedule, true); + document.addEventListener("focusout", schedule, true); + + // visualViewport gives the best signal on mobile browsers + if (window.visualViewport) { + window.visualViewport.addEventListener("resize", schedule, { passive: true }); + window.visualViewport.addEventListener("scroll", schedule, { passive: true }); + } + + // composer height changes (pending bar / textarea autosize) + if (window.ResizeObserver) { + const ro = new ResizeObserver(schedule); + ro.observe(composer); + } +})(); + +/* ========================================================= + Persist sidebar config across refresh (keys, base_url, etc.) + ========================================================= */ + +const __OS_PERSIST_STORAGE = window.sessionStorage; // <- 改成 localStorage 即可“关浏览器也还在” +const __OS_PERSIST_KEY = "openstoryline_user_config_v1"; + +function __osSafeParseJson(s, fallback) { + try { + const v = JSON.parse(s); + return (v && typeof v === "object") ? v : fallback; + } catch { + return fallback; + } +} + +function __osLoadConfig() { + return __osSafeParseJson(__OS_PERSIST_STORAGE.getItem(__OS_PERSIST_KEY), {}); +} + +function __osSaveConfig(cfg) { + try { + __OS_PERSIST_STORAGE.setItem(__OS_PERSIST_KEY, JSON.stringify(cfg || {})); + } catch (e) { + console.warn("[persist] save failed:", e); + } +} + +function __osGetByPath(obj, path) { + if (!obj || !path) return undefined; + const parts = String(path).split(".").filter(Boolean); + let cur = obj; + for (const p of parts) { + if (!cur || typeof cur !== "object") return undefined; + cur = cur[p]; + } + return cur; +} + +function __osSetByPath(obj, path, value) { + const parts = String(path).split(".").filter(Boolean); + if (!parts.length) return; + let cur = obj; + for (let i = 0; i < parts.length - 1; i++) { + const k = parts[i]; + if (!cur[k] || typeof cur[k] !== "object") cur[k] = {}; + cur = cur[k]; + } + cur[parts[parts.length - 1]] = value; +} + +const __osPendingSelectValues = new Map(); + +function __osApplySelectValue(selectEl, desiredValue) { + const desired = String(desiredValue ?? ""); + const before = selectEl.value; + selectEl.value = desired; + + const ok = selectEl.value === desired; + if (ok && before !== selectEl.value) { + // 触发你现有的 UI 联动逻辑（显示/隐藏 box 等） + selectEl.dispatchEvent(new Event("change", { bubbles: true })); + } + return ok; +} + +function __osObserveSelectOptions(selectEl) { + if (selectEl.__osSelectObserver) return; + + const observer = new MutationObserver(() => { + const desired = __osPendingSelectValues.get(selectEl); + if (desired == null) return; + + if (__osApplySelectValue(selectEl, desired)) { + __osPendingSelectValues.delete(selectEl); + observer.disconnect(); + selectEl.__osSelectObserver = null; + } + }); + + observer.observe(selectEl, { childList: true, subtree: true }); + selectEl.__osSelectObserver = observer; +} + +function __osHydratePersistedFields(root = document) { + const cfg = __osLoadConfig(); + const nodes = root.querySelectorAll("[data-os-persist]"); + + nodes.forEach((el) => { + const key = el.getAttribute("data-os-persist"); + if (!key) return; + + const v = __osGetByPath(cfg, key); + if (v == null) return; + + const tag = (el.tagName || "").toLowerCase(); + const type = String(el.type || "").toLowerCase(); + + try { + if (type === "checkbox") { + el.checked = !!v; + } else if (tag === "select") { + // 如果选项是异步加载的（比如 modelSelect），先尝试设置，不行就等 options 出来再设置 + if (!__osApplySelectValue(el, v)) { + __osPendingSelectValues.set(el, String(v)); + __osObserveSelectOptions(el); + } else { + // 已成功设置，确保联动触发一次（有些情况下 before==after 不触发） + el.dispatchEvent(new Event("change", { bubbles: true })); + } + } else { + el.value = String(v); + } + } catch {} + }); + + root.querySelectorAll('select[data-os-persist]').forEach((sel) => { + try { sel.dispatchEvent(new Event("change", { bubbles: true })); } catch {} + }); + + return cfg; +} + +function __osBindPersistedFields(root = document) { + let cfg = __osLoadConfig(); + + const nodes = root.querySelectorAll("[data-os-persist]"); + nodes.forEach((el) => { + const key = el.getAttribute("data-os-persist"); + if (!key) return; + + if (el.__osPersistBound) return; + el.__osPersistBound = true; + + const handler = () => { + const tag = (el.tagName || "").toLowerCase(); + const type = String(el.type || "").toLowerCase(); + + let v; + if (type === "checkbox") v = !!el.checked; + else if (tag === "select") v = String(el.value ?? ""); + else v = String(el.value ?? ""); + + __osSetByPath(cfg, key, v); + __osSaveConfig(cfg); + }; + + el.addEventListener("input", handler); + el.addEventListener("change", handler); + }); + + return { + getConfig: () => (cfg = __osLoadConfig()), + clear: () => { + __OS_PERSIST_STORAGE.removeItem(__OS_PERSIST_KEY); + cfg = {}; + }, + saveNow: () => __osSaveConfig(cfg), + }; +} + +function __osInitPersistSidebarConfig() { + __osHydratePersistedFields(document); + window.OPENSTORYLINE_PERSIST = __osBindPersistedFields(document); // 可选：调试用 +} + +if (document.readyState === "loading") { + window.addEventListener("DOMContentLoaded", __osInitPersistSidebarConfig); +} else { + __osInitPersistSidebarConfig(); +} diff --git a/web/static/dice.png b/web/static/dice.png new file mode 100644 index 0000000000000000000000000000000000000000..bb284b38b7034ffc1f792eb8f8c82bf163e22f03 Binary files /dev/null and b/web/static/dice.png differ diff --git a/web/static/github.png b/web/static/github.png new file mode 100644 index 0000000000000000000000000000000000000000..d2a496c43115725fbc49b5a6a99d183139cfaafb Binary files /dev/null and b/web/static/github.png differ diff --git a/web/static/node_map.png b/web/static/node_map.png new file mode 100644 index 0000000000000000000000000000000000000000..2fbff57ab380ea8e7840b20260d7ebc970eb5c8b Binary files /dev/null and b/web/static/node_map.png differ diff --git a/web/static/style.css b/web/static/style.css new file mode 100644 index 0000000000000000000000000000000000000000..e1e0d643f6076778210f7a3eb767a3233f4ce43f --- /dev/null +++ b/web/static/style.css @@ -0,0 +1,2074 @@ +:root{ + --os-font: ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Inter", "PingFang SC", "Microsoft YaHei", sans-serif; + --mono: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono","Courier New", monospace; + + --bg: #ffffff; + --text: #0b0b0c; + --muted: rgba(11,11,12,0.62); + + --surface: rgba(255,255,255,0.92); + --surface-2: rgba(0,0,0,0.03); + + --border: rgba(11,11,12,0.14); + --border-weak: rgba(11,11,12,0.08); + + --shadow-soft: 0 12px 32px rgba(0,0,0,0.06); + --shadow: 0 22px 60px rgba(0,0,0,0.10); + + --radius-lg: 28px; + --radius-md: 18px; + --radius-sm: 12px; + + --maxw: 920px; + --ring: 0 0 0 3px rgba(0,0,0,0.12); + + --topbar-h: 56px; + + --sidebar-w: 260px; + --sidebar-collapsed-w: 56px; + + --devbar-w: 360px; + --devbar-collapsed-w: 56px; + + --sidebar-panel-gap: 14px; +} + +@media (prefers-color-scheme: dark){ + :root{ + --bg: #0b0b0c; + --text: #f4f4f5; + --muted: rgba(244,244,245,0.60); + + --surface: rgba(20,20,22,0.92); + --surface-2: rgba(255,255,255,0.07); + + --border: rgba(244,244,245,0.16); + --border-weak: rgba(244,244,245,0.10); + + --shadow-soft: 0 14px 36px rgba(0,0,0,0.38); + --shadow: 0 24px 70px rgba(0,0,0,0.58); + + --ring: 0 0 0 3px rgba(255,255,255,0.14); + } +} + +*{ box-sizing: border-box; } +html, body{ height: 100%; } + +body{ + margin: 0; + font-family: var(--os-font); + color: var(--text); + background: var(--bg); + -webkit-font-smoothing: antialiased; + -moz-osx-font-smoothing: grayscale; + text-rendering: geometricPrecision; + + --sidebar-current: var(--sidebar-w); + --content-offset: var(--sidebar-current); + + --devbar-current: 0px; + --right-offset: 0px; + + /* 让 fixed 元素始终居中在内容区（左右侧边栏之间） */ + --center-shift: calc((var(--content-offset) - var(--right-offset)) * 0.5); +} + +body.sidebar-collapsed{ + --sidebar-current: var(--sidebar-collapsed-w); + --content-offset: var(--sidebar-current); +} + +body.dev-mode{ + --devbar-current: var(--devbar-w); + --right-offset: var(--devbar-current); +} +body.dev-mode.devbar-collapsed{ + --devbar-current: var(--devbar-collapsed-w); + --right-offset: var(--devbar-current); +} + +/* 移动端：sidebar/devbar 采用 overlay，不挤压内容区 */ +@media (max-width: 760px){ + body{ --content-offset: var(--sidebar-collapsed-w); } + body.dev-mode, + body.dev-mode.devbar-collapsed{ --right-offset: 0px; } +} + +.hidden{ display:none !important; } + +::selection{ background: rgba(0,0,0,0.10); } +@media (prefers-color-scheme: dark){ + ::selection{ background: rgba(255,255,255,0.14); } +} + +/* 主内容区给两侧栏预留空间 */ +.main{ + margin-left: var(--content-offset); + margin-right: var(--right-offset); + min-height: 100vh; +} + +/* ========================================================= + 0) Sidebar：左侧可收起 + ========================================================= */ +.sidebar{ + position: fixed; + left: 0; + top: 0; + height: 100vh; + width: var(--sidebar-current); + z-index: 55; + + overflow: hidden; + + background: var(--surface); + border-right: 1px solid var(--border-weak); + box-shadow: var(--shadow-soft); + backdrop-filter: blur(10px); + + transition: width 0.18s ease; +} + +.sidebar-inner{ + height: 100%; + padding: 12px; + display: flex; + flex-direction: column; + gap: 10px; +} + +.sidebar-icon-btn{ + width: 44px; + height: 44px; + border-radius: 999px; + border: 1px solid var(--border-weak); + background: transparent; + color: var(--text); + display: grid; + place-items: center; + cursor: pointer; +} +.sidebar-icon-btn:hover{ + background: var(--surface-2); + border-color: var(--border); +} + +.sidebar-action{ + width: 100%; + height: 44px; + border-radius: 16px; + border: 1px solid var(--border-weak); + background: transparent; + color: var(--text); + display:flex; + align-items:center; + gap: 10px; + padding: 0 12px; + cursor:pointer; +} +.sidebar-action:hover{ + background: var(--surface-2); + border-color: var(--border); +} +.sidebar-action.primary{ border-color: var(--border); } + +.sidebar-action-icon{ + width: 20px; + height: 20px; + display: grid; + place-items:center; + font-size: 18px; + line-height: 1; + flex: 0 0 auto; +} +.sidebar-action-text{ + font-size: 13px; + color: var(--text); + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} + +.devbar-sid{ + font-family: var(--mono); + font-size: 12px; + color: var(--muted); + padding: 6px 2px 0; + white-space: normal; + word-break: break-all; + user-select: text; +} +body.dev-mode.devbar-collapsed .devbar-sid{ display:none; } + +/* 收起态：变成窄轨道，隐藏文字 */ +body.sidebar-collapsed .sidebar-inner{ padding: 12px 6px; } +body.sidebar-collapsed .sidebar-action{ + width: 44px; + padding: 0; + justify-content: center; + border-radius: 999px; +} +body.sidebar-collapsed .sidebar-action-text{ display:none; } + +/* 模型选择下拉框 */ +.sidebar-model{ + width: 100%; + display: flex; + flex-direction: column; + gap: 6px; + padding: 2px; +} +.sidebar-model-label{ + font-size: 12px; + color: var(--muted); + padding: 2px 6px 0; + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} +.sidebar-model-select{ + width: 100%; + height: 36px; + border-radius: 12px; + border: 1px solid var(--border-weak); + background: var(--surface-2); + color: var(--text); + padding: 0 10px; + font-size: 13px; + outline: none; +} +.sidebar-model-select:hover{ border-color: var(--border); } +.sidebar-model-select:focus{ + border-color: var(--border); + box-shadow: var(--ring); +} +body.sidebar-collapsed .sidebar-model{ display: none; } + +/* ========================================================= + Sidebar panels (Custom Model / TTS) + ========================================================= */ +.sidebar-panel{ + width: 100%; + border: 1px solid var(--border-weak); + border-radius: 16px; + background: var(--surface); + padding: 10px 12px; + display: flex; + flex-direction: column; + gap: 8px; +} + +.sidebar-scroll .sidebar-panel + .sidebar-panel{ + margin-top: var(--sidebar-panel-gap); +} + +.sidebar-panel-title{ + font-size: 12px; + color: var(--muted); + padding: 2px 2px 0; + + display: flex; + align-items: center; + gap: 6px; + + overflow: visible; +} + +.sidebar-panel-title-text{ + flex: 0 1 auto; + min-width: 0; + + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} + +/* ========================================================= + Sidebar title help ( ? ) + tooltip + ========================================================= */ + +.sidebar-help{ + position: relative; + flex: 0 0 auto; + display: inline-flex; + align-items: center; + justify-content: center; + text-decoration: none; + cursor: pointer; +} + +.sidebar-help-icon{ + width: 18px; + height: 18px; + border-radius: 999px; + + border: 1px solid var(--border-weak); + background: var(--surface-2); + color: var(--muted); + + display: grid; + place-items: center; + + font-size: 11px; + font-weight: 600; + line-height: 1; +} + +.sidebar-help:hover .sidebar-help-icon{ + border-color: var(--border); + color: var(--text); +} + +.sidebar-help:focus-visible .sidebar-help-icon{ + box-shadow: var(--ring); +} + +.sidebar-help{ + position: relative; +} + +.sidebar-help-tooltip{ + position: absolute; + z-index: 60; + + left: 0px; + right: auto; + max-width: calc(var(--sidebar-current) / 2); + white-space: normal; + max-height: min(35vh, 180px); + top: calc(100% + 6px); + + display: inline-block; + width: max-content; + + overflow: auto; + + padding: 2px 6px; + border-radius: 12px; + border: 1px solid var(--border-weak); + background: var(--surface); + box-shadow: var(--shadow-soft); + + color: var(--text); + font-size: 12px; + line-height: 1.45; + + white-space: pre-line; + overflow-wrap: anywhere; + + opacity: 0; + visibility: hidden; + transform: translateY(-4px); + pointer-events: none; + + transition: opacity .12s ease, transform .12s ease, visibility .12s ease; +} + +.sidebar-help-tooltip::before{ + content: ""; + position: absolute; + top: -6px; + left: auto; + right: 10px; + width: 10px; + height: 10px; + transform: rotate(45deg); + + background: var(--surface); + border-left: 1px solid var(--border-weak); + border-top: 1px solid var(--border-weak); +} + +.sidebar-help:hover .sidebar-help-tooltip, +.sidebar-help:focus-visible .sidebar-help-tooltip, +.sidebar-help-tooltip:hover{ + opacity: 1; + visibility: visible; + transform: translateY(0); + pointer-events: auto; +} + +.sidebar-help-tooltip-body{ + display: block; + color: var(--text); +} + +.sidebar-help-tooltip-cta{ + display: block; + margin-top: 2px; + color: var(--muted); + text-decoration: underline; + text-underline-offset: 2px; +} + +.sidebar-help-tooltip-text{ + text-decoration: underline; + text-underline-offset: 2px; +} + +/* show tooltip on hover/focus */ +.sidebar-help:hover .sidebar-help-tooltip, +.sidebar-help:focus-visible .sidebar-help-tooltip{ + opacity: 1; + visibility: visible; + transform: translateY(0); + pointer-events: auto; +} +/* hover bridge */ +.sidebar-help{ + position: relative; +} + +.sidebar-help::after{ + content: ""; + position: absolute; + left: -10px; + right: -10px; + top: 100%; + height: 14px; + background: transparent; +} + +.sidebar-help:hover .sidebar-help-tooltip, +.sidebar-help:focus-visible .sidebar-help-tooltip, +.sidebar-help-tooltip:hover{ + opacity: 1; + visibility: visible; + transform: translateY(0); + pointer-events: auto; +} + +.sidebar-subtitle{ + font-size: 12px; + color: var(--muted); + margin-top: 4px; +} +.sidebar-input{ + width: 100%; + height: 34px; + border-radius: 12px; + border: 1px solid var(--border-weak); + background: var(--surface-2); + color: var(--text); + padding: 0 10px; + font-size: 13px; + outline: none; +} +.sidebar-input:hover{ border-color: var(--border); } +.sidebar-input:focus{ + border-color: var(--border); + box-shadow: var(--ring); +} +.sidebar-divider{ + height: 1px; + background: var(--border-weak); + margin: 6px 0 2px; +} +.sidebar-hint{ + font-size: 11px; + color: var(--muted); + line-height: 1.45; +} +.sidebar-tts-fields{ + display: flex; + flex-direction: column; + gap: 8px; +} + +/* 收起侧边栏时隐藏配置面板 */ +body.sidebar-collapsed .sidebar-panel{ display: none; } + +/* flex 容器允许子项正确计算滚动高度 */ +.sidebar-inner{ + min-height: 0; /* 不加这个，很多浏览器下滚动区会失效 */ +} + +/* 顶部固定区：不参与滚动，不允许被挤压 */ +.sidebar-top{ + display: flex; + flex-direction: column; + gap: 10px; + flex: 0 0 auto; + min-height: 0; +} + +/* 滚动区：承载“模型配置 / TTS 配置”等长内容 */ +.sidebar-scroll{ + flex: 1 1 auto; + min-height: 0; + overflow-y: auto; + overflow-x: hidden; + padding-bottom: 12px; /* 避免最后一个输入框贴底被遮挡 */ + overscroll-behavior: contain; + -webkit-overflow-scrolling: touch; +} + +.sidebar-icon-btn, +.sidebar-action{ + flex: 0 0 auto; + flex-shrink: 0; +} + +.sidebar-action{ + min-height: 44px; +} + +body.sidebar-collapsed .sidebar-scroll{ + display: none; +} + + +/* ========================================================= + 1) 顶部栏 Topbar + ========================================================= */ +.topbar{ + position: sticky; + top: 0; + z-index: 45; + height: var(--topbar-h); + + display: flex; + align-items: center; + padding: 0 16px; + + background: var(--surface); + border-bottom: 1px solid var(--border-weak); + backdrop-filter: blur(10px); + + box-shadow: 0 10px 24px rgba(0,0,0,0.04); +} +@media (prefers-color-scheme: dark){ + .topbar{ box-shadow: 0 16px 34px rgba(0,0,0,0.32); } +} + +.topbar > .brand{ + width: 100%; + margin: 0; + display: flex; + align-items: baseline; + gap: 10px; + min-width: 0; +} + +.topbar .brand{ + font-size: 24px; + font-weight: 700; + letter-spacing: -0.03em; + color: var(--text); + opacity: 0.92; + + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} + +.topbar .ver{ + display: inline-flex; + align-items: center; + height: 20px; + padding: 0 8px; + border-radius: 999px; + + border: 1px solid var(--border-weak); + background: var(--surface-2); + color: var(--muted); + + font-family: var(--mono); + font-size: 11px; + font-weight: 600; + letter-spacing: 0; +} + +.topbar .actions{ + margin-left: auto; + display: flex; + align-items: center; + gap: 8px; +} + +@media (max-width: 640px){ + .topbar{ height: 52px; padding: 0 12px; } + .topbar > .brand{ width: min(var(--maxw), calc(100% - 1.25rem)); } +} + +/* ========================================================= + 2) 聊天区 + ========================================================= */ +.chat{ + width: min(var(--maxw), calc(100% - 2rem)); + margin: 0 auto; + padding: 24px 0 240px; +} +#chat:empty{ padding: 0; } + +/* 空白页标题 */ +.hero{ + display: none; + position: fixed; + left: calc(50% + var(--center-shift)); + top: 18vh; + transform: translateX(-50%); + width: min(var(--maxw), calc(100vw - var(--content-offset) - var(--right-offset) - 2rem)); + text-align: center; + pointer-events: none; +} +#chat:empty ~ .hero{ display:block; } + +.hero-title{ + margin: 0; + font-size: clamp(34px, 4.2vw, 52px); + line-height: 1.06; + font-weight: 650; + letter-spacing: -0.03em; +} + +/* 消息布局 */ +.msg{ + margin: 0; + padding: 10px 0; + display: flex; +} +.msg.assistant{ justify-content:flex-start; } +.msg.user{ justify-content:flex-end; } + +/* 允许 flex 子项收缩（附件横向滚动的关键） */ +.msg.user > div, +.msg.assistant > div{ min-width: 0; } + +/* 覆盖 JS inline max-width（建议从 JS 源头移除） */ +.msg.user > div{ + max-width: min(70%, 560px) !important; + min-width: 0; +} + +.bubble{ + max-width: min(74ch, 100%); + font-size: 15px; + line-height: 1.68; + letter-spacing: -0.01em; + white-space: normal; + word-break: break-word; +} + +.msg.assistant .bubble{ + padding: 0; + background: transparent; + border: none; + box-shadow: none; +} + +.msg.user .bubble{ + padding: 8px 12px; + background: var(--surface-2); + border: 1px solid var(--border-weak); + border-radius: 16px; + + font-size: 14.5px; + line-height: 1.55; + letter-spacing: -0.005em; + + max-width: min(60ch, 72vw); + white-space: pre-wrap; +} + +.bubble a{ + color: inherit; + text-decoration: underline; + text-underline-offset: 2px; +} +.bubble a:hover{ opacity: 0.88; } + +.bubble pre{ + margin: 12px 0; + padding: 12px 14px; + border-radius: 16px; + background: var(--surface-2); + border: 1px solid var(--border-weak); + overflow: auto; + font-family: var(--mono); + font-size: 13px; + line-height: 1.6; +} +.bubble code{ + font-family: var(--mono); + font-size: 0.92em; + padding: 2px 6px; + border-radius: 10px; + background: var(--surface-2); + border: 1px solid var(--border-weak); +} +.bubble pre code{ + padding: 0; + border: none; + background: transparent; +} + +/* Markdown 内容排版增强（assistant bubble 内） */ +.msg.assistant .bubble h1, +.msg.assistant .bubble h2, +.msg.assistant .bubble h3{ + margin: 18px 0 10px; + line-height: 1.25; + letter-spacing: -0.02em; +} +.msg.assistant .bubble p{ margin: 10px 0; } +.msg.assistant .bubble ul, +.msg.assistant .bubble ol{ margin: 10px 0 10px 1.2em; } +.msg.assistant .bubble blockquote{ + margin: 12px 0; + padding: 8px 12px; + border-left: 3px solid var(--border); + background: var(--surface-2); + border-radius: 12px; + color: var(--muted); +} +.msg.assistant .bubble hr{ + border: none; + border-top: 1px solid var(--border-weak); + margin: 14px 0; +} + +/* ========================================================= + 3) 附件缩略图（消息内 & 待发送素材） + ========================================================= */ +.attach-row{ + display: flex; + gap: 8px; + padding: 6px 0 0; +} + +/* 两类素材条：不换行 + 横向滚动 + 移动端顺滑 */ +.media-row, +.attach-row{ + display: flex; + gap: 8px; + + max-width: 100%; + min-width: 0; + flex-wrap: nowrap; + + overflow-x: auto; + overflow-y: hidden; + + -webkit-overflow-scrolling: touch; + scrollbar-width: thin; +} + +/* WebKit scrollbar */ +.media-row::-webkit-scrollbar, +.attach-row::-webkit-scrollbar{ height: 6px; } +.media-row::-webkit-scrollbar-thumb, +.attach-row::-webkit-scrollbar-thumb{ + background: var(--border); + border-radius: 999px; +} +.media-row::-webkit-scrollbar-track, +.attach-row::-webkit-scrollbar-track{ background: transparent; } + +/* 用户消息附件：必须从左开始排版，保证可滚动 */ +.msg.user .attach-row{ + padding-top: 4px; + justify-content: flex-start !important; + margin-left: 0 !important; + + width: 100%; + min-width: 0; + + overscroll-behavior-x: contain; + touch-action: pan-x; +} + +/* 外层负责靠右，内层负责滚动 */ +.attach-wrap{ + max-width: 100%; + min-width: 0; +} +.attach-wrap.align-right{ margin-left: auto; } +.attach-wrap .attach-row{ + width: 100%; + min-width: 0; + justify-content: flex-start; +} + +/* media item */ +.media-item{ + position: relative; + flex: 0 0 auto; + width: 64px; + height: 64px; + border-radius: 14px; + overflow: hidden; + background: var(--surface-2); + border: 1px solid var(--border-weak); + cursor: pointer; +} +.media-item:hover{ border-color: var(--border); } + +.media-item img{ + width: 100%; + height: 100%; + object-fit: contain; + object-position: center; + display:block; +} + +.media-tag{ + position: absolute; + left: 6px; + top: 6px; + font-size: 10px; + padding: 2px 6px; + border-radius: 999px; + background: rgba(255,255,255,0.86); + border: 1px solid rgba(0,0,0,0.10); + color: rgba(0,0,0,0.72); +} +@media (prefers-color-scheme: dark){ + .media-tag{ + background: rgba(0,0,0,0.55); + border-color: rgba(255,255,255,0.12); + color: rgba(255,255,255,0.78); + } +} + +.media-play{ + position: absolute; + right: 8px; + bottom: 8px; + width: 0; + height: 0; + border-left: 14px solid rgba(255,255,255,0.92); + border-top: 9px solid transparent; + border-bottom: 9px solid transparent; + filter: drop-shadow(0 1px 2px rgba(0,0,0,0.35)); +} + +.media-remove{ + position: absolute; + right: 6px; + top: 6px; + width: 22px; + height: 22px; + border-radius: 999px; + border: 1px solid rgba(255,255,255,0.70); + background: rgba(0,0,0,0.55); + color: rgba(255,255,255,0.96); + font-weight: 700; + line-height: 20px; + text-align: center; + cursor: pointer; +} +@media (prefers-color-scheme: dark){ + .media-remove{ + border-color: rgba(255,255,255,0.18); + background: rgba(255,255,255,0.12); + color: rgba(255,255,255,0.96); + } +} + +/* FIX: media remove "×" optical center */ +.media-remove{ + display: grid; + place-items: center; + padding: 0; + line-height: 0; + font-size: 0; /* hide text glyph × */ + -webkit-appearance: none; + appearance: none; +} + +.media-remove{ --x-size: 10px; --x-thick: 2px; --x-nudge-y: 4.5px; } + +.media-remove::before, +.media-remove::after{ + content: ""; + width: var(--x-size); + height: var(--x-thick); + background: currentColor; + border-radius: 999px; + grid-area: 1 / 1; + pointer-events: none; + display: block; + place-self: center; +} + +.media-remove::before{ transform: translateY(var(--x-nudge-y, 0px)) rotate(45deg); } +.media-remove::after{ transform: translateY(var(--x-nudge-y, 0px)) rotate(-45deg); } + + +/* ========================================================= + 4) 输入框（两层结构） + ========================================================= */ +.file-input{ display:none; } + +.composer{ + position: fixed; + left: calc(50% + var(--center-shift)); + bottom: calc(24px + env(safe-area-inset-bottom)); + transform: translateX(-50%); + z-index: 45; + + width: min(var(--maxw), calc(100vw - var(--content-offset) - var(--right-offset) - 2rem)); + padding: 10px 12px; + border-radius: var(--radius-lg); + + background: var(--surface); + border: 1px solid var(--border-weak); + box-shadow: var(--shadow-soft); + backdrop-filter: blur(10px); + + display: flex; + flex-direction: column; + flex-wrap: nowrap; + align-items: stretch; + gap: 8px; +} + +#chat:empty ~ .composer{ + top: 55vh; + bottom: auto; + transform: translate(-50%, -50%); +} + +.composer:focus-within{ + border-color: var(--border); + box-shadow: var(--shadow-soft), var(--ring); +} + +/* 待发送素材：内嵌在输入框里 */ +.pending{ + display:flex; + gap: 8px; + padding: 6px 2px 2px; + margin: 0 0 2px; + border-bottom: 1px solid var(--border-weak); + overflow: hidden; +} +.media-bar-title{ display:none; } +.pending .media-row{ flex: 1 1 auto; } + +/* prompt 单行更紧凑 */ +.composer-top{ + width: 100%; + padding: 0 2px; +} + +.prompt{ + width: 100%; + min-height: 40px; + max-height: 180px; + + border: none; + outline: none; + background: transparent; + + resize: none; + + overflow-y: hidden; /* 超过 max-height 时建议由 JS 切换为 auto */ + overflow-x: hidden; + + font-family: inherit; + font-size: 15px; + line-height: 20px; + color: var(--text); + padding: 10px 8px; +} +.prompt::placeholder{ color: var(--muted); } + +.composer-actions{ + display: flex; + align-items: center; + gap: 8px; + padding-top: 10px; + border-top: 1px solid var(--border-weak); +} +.composer-actions-spacer{ + flex: 1 1 auto; + min-width: 0; +} + +/* 左侧 + */ +.icon-btn{ + width: 44px; + height: 44px; + border-radius: 999px; + border: 1px solid var(--border-weak); + background: transparent; + color: var(--text); + display: grid; + place-items: center; + cursor: pointer; + flex: 0 0 auto; +} +.icon-btn:hover{ + background: var(--surface-2); + border-color: var(--border); +} +.icon-btn:active{ transform: translateY(1px); } +.icon-btn:disabled{ + opacity: 0.35; + cursor: not-allowed; + transform: none; +} + +/* 发送 */ +.send-btn{ + width: 44px; + height: 44px; + border-radius: 999px; + + border: 1px solid transparent; + background: #000; + color: #fff; + + display: grid; + place-items: center; + cursor: pointer; + flex: 0 0 auto; + + transition: transform 0.08s ease, opacity 0.18s ease; +} +@media (prefers-color-scheme: dark){ + .send-btn{ background: #fff; color: #000; } +} +.send-btn:hover{ transform: translateY(-1px); } +.send-btn:active{ transform: translateY(0); } +.send-btn:disabled{ + opacity: 0.35; + cursor: not-allowed; + transform: none; +} + +/* SVG 统一 */ +.sidebar-icon-btn svg, +.devbar-icon-btn svg, +.icon-btn svg, +.send-btn svg, +.scroll-bottom svg{ + width: 20px; + height: 20px; + fill: none; + stroke: currentColor; + stroke-width: 2.2; + stroke-linecap: round; + stroke-linejoin: round; +} + +/* ========================================================= + 5) toast / tool-card / modal + ========================================================= */ +.toast{ + position: fixed; + left: calc(50% + var(--center-shift)); + transform: translateX(-50%); + bottom: calc(24px + env(safe-area-inset-bottom) + 86px); + z-index: 60; + + padding: 10px 12px; + border-radius: 14px; + + border: 1px solid var(--border-weak); + background: var(--surface); + box-shadow: var(--shadow-soft); + + color: var(--text); + font-size: 13px; + letter-spacing: -0.01em; +} + +details.tool-card{ + width: min(480px, 100%); + border: 1px solid var(--border-weak); + border-radius: 16px; + background: var(--surface); + overflow: hidden; +} +details.tool-card > summary{ list-style: none; } +details.tool-card > summary::-webkit-details-marker{ display:none; } +details.tool-card > summary::marker{ content:""; } + +.tool-head{ cursor: pointer; padding: 12px 14px; } +details.tool-card[open] .tool-head{ border-bottom: 1px solid var(--border-weak); } + +.media-card{ + width: min(480px, 100%); + border: 1px solid var(--border-weak); + border-radius: 16px; + background: var(--surface); + padding: 10px 14px 12px; +} +.media-card .tool-preview{ margin-top: 0; } +.msg.assistant.tool-media-msg{ padding-top: 6px; } + +.tool-line{ + display: flex; + align-items: center; + gap: 10px; + min-width: 0; +} +.tool-left{ + display: flex; + align-items: center; + gap: 8px; + min-width: 0; + flex: 0 1 auto; +} +.tool-status{ + width: 14px; + height: 14px; + flex: 0 0 auto; + display: inline-flex; + align-items: center; + justify-content: center; + box-sizing: border-box; + color: var(--muted); +} +.tool-status.is-running::before{ + content: ""; + width: 12px; + height: 12px; + box-sizing: border-box; + border: 2px solid var(--muted); + border-top-color: transparent; + border-radius: 999px; + animation: os_tool_spin 0.8s linear infinite; +} +.tool-status.is-success, +.tool-status.is-error{ color: var(--text); } + +.tool-name{ + font-size: 13px; + color: var(--muted); + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} +.tool-args-preview{ + font-family: var(--mono); + font-size: 12px; + color: var(--muted); + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; + min-width: 0; + flex: 1 1 auto; + text-align: right; +} + +.tool-progress{ + margin-top: 10px; + width: min(240px, 100%); + height: 6px; + border-radius: 999px; + background: var(--surface-2); + overflow: hidden; +} +.tool-progress-fill{ + height: 100%; + width: 0%; + background: var(--text); + border-radius: 999px; + transition: width 0.12s linear; +} + +.tool-body-wrap{ padding: 10px 14px 12px; } +.tool-body{ + margin: 0; + font-family: var(--mono); + font-size: 12px; + line-height: 1.6; + color: var(--muted); + white-space: pre-wrap; + overflow-wrap: anywhere; + word-break: break-word; +} + +.tool-preview{ + margin-top: 10px; + display: flex; + flex-direction: column; + gap: 10px; +} +.tool-preview-block{ + display: flex; + flex-direction: column; + gap: 8px; +} +.tool-preview-title{ + font-size: 12px; + color: var(--muted); + user-select: none; +} + +.tool-inline-video{ + width: 100%; + max-height: 360px; + border-radius: 12px; + border: 1px solid var(--border-weak); + background: rgba(0,0,0,0.06); + object-fit: contain; +} + +.tool-preview-actions{ + display: flex; + align-items: center; + gap: 10px; +} +.tool-preview-btn{ + height: 30px; + padding: 0 10px; + border-radius: 999px; + border: 1px solid var(--border-weak); + background: transparent; + color: var(--text); + font-size: 12px; + cursor: pointer; +} +.tool-preview-btn:hover{ + background: var(--surface-2); + border-color: var(--border); +} +.tool-preview-link{ + font-size: 12px; + color: var(--muted); + text-decoration: underline; + text-underline-offset: 2px; +} +.tool-preview-link:hover{ opacity: 0.88; } + +/* Grid thumbnails */ +.tool-media-grid{ + display: grid; + grid-template-columns: repeat(auto-fill, minmax(118px, 1fr)); + gap: 10px; +} +.tool-media-item{ + border: none; + background: transparent; + padding: 0; + text-align: left; + cursor: pointer; +} +.tool-media-thumb{ + width: 100%; + aspect-ratio: 16 / 9; + border-radius: 12px; + overflow: hidden; + background: var(--surface-2); + border: 1px solid var(--border-weak); + position: relative; + display: grid; + place-items: center; +} +.tool-media-thumb.is-portrait{ aspect-ratio: 9 / 16; } +.tool-media-thumb.is-square{ aspect-ratio: 1 / 1; } + +.tool-media-thumb img, +.tool-media-thumb video{ + width: 100%; + height: 100%; + object-fit: contain; + display: block; +} +.tool-media-play{ + position: absolute; + right: 10px; + bottom: 10px; + width: 0; + height: 0; + border-left: 16px solid rgba(255,255,255,0.92); + border-top: 10px solid transparent; + border-bottom: 10px solid transparent; + filter: drop-shadow(0 1px 2px rgba(0,0,0,0.35)); + pointer-events: none; +} +.tool-media-label{ + margin-top: 6px; + font-size: 12px; + color: var(--muted); + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} +.tool-media-more{ + font-size: 12px; + color: var(--muted); +} + +/* Audio preview */ +.tool-audio-list{ + display: flex; + flex-direction: column; + gap: 10px; +} +.tool-audio-item{ + border: 1px solid var(--border-weak); + background: var(--surface-2); + border-radius: 12px; + padding: 10px 10px 8px; +} +.tool-audio-item audio{ width: 100%; } + +/* 进度条行 */ +.tool-progress-row{ + display: flex; + align-items: center; + gap: 10px; + margin-top: 8px; +} +.tool-progress-pct{ + font-family: var(--mono); + font-size: 12px; + line-height: 1; + color: var(--muted); + min-width: 38px; + text-align: right; + user-select: none; + flex: 0 0 auto; +} +.tool-progress-row .tool-progress{ + flex: 1 1 auto; + min-width: 0; +} + +@keyframes os_tool_spin{ + from{ transform: rotate(0deg); } + to{ transform: rotate(360deg); } +} + +/* modal */ +.modal{ position: fixed; inset: 0; z-index: 80; } +.modal-backdrop{ + position: absolute; + inset: 0; + background: rgba(0,0,0,0.55); + z-index: 0; +} +.modal-body{ + position: absolute; + left: 50%; + top: 50%; + transform: translate(-50%, -50%); + width: fit-content; + max-width: 92vw; + max-height: 86vh; + background: var(--surface); + border: 1px solid var(--border-weak); + border-radius: 18px; + overflow: hidden; + box-shadow: var(--shadow); + z-index: 1; +} +.modal-content{ position: relative; z-index: 1; padding: 0; } + +.modal-close{ + position: absolute; + right: 12px; + top: 12px; + width: 36px; + height: 36px; + border-radius: 999px; + border: 1px solid var(--border-weak); + background: var(--surface-2); + color: var(--text); + cursor: pointer; + + /* 居中 + 去掉默认按钮内边距 */ + display: grid; + place-items: center; + padding: 0; + line-height: 1; + + font-size: 20px; + z-index: 10; + pointer-events: auto; +} + +.modal-close:hover{ opacity: 0.88; } + +/* FIX: modal close "×" optical center */ +.modal-close{ + display: grid; + place-items: center; + padding: 0; + line-height: 0; + font-size: 0; /* hide text glyph × */ + -webkit-appearance: none; + appearance: none; +} + +.modal-close{ --x-size: 16px; --x-thick: 2px; --x-nudge-y: 8px; } + +.modal-close::before, +.modal-close::after{ + content: ""; + width: var(--x-size); + height: var(--x-thick); + background: currentColor; + border-radius: 999px; + grid-area: 1 / 1; + pointer-events: none; + display: block; + place-self: center; +} + +.modal-close::before{ transform: translateY(var(--x-nudge-y, 0px)) rotate(45deg); } +.modal-close::after{ transform: translateY(var(--x-nudge-y, 0px)) rotate(-45deg); } + +.modal-content img, +.modal-content video{ + max-width: 100%; + max-height: 86vh; + width: auto; + height: auto; + display: block; + margin: 0 auto; +} +.modal-content audio{ + width: min(720px, 92vw); + display: block; + padding: 16px; +} +.modal-content .file-fallback{ + padding: 16px; + color: var(--muted); + font-size: 13px; +} + +/* ========================================================= + 6) 小屏适配 + ========================================================= */ +@media (max-width: 640px){ + .chat{ width: calc(100% - 1.25rem); padding: 24px 0 240px; } + #chat:empty{ padding: 0; } + + .composer{ + width: calc(100vw - var(--content-offset) - var(--right-offset) - 1.25rem); + bottom: calc(14px + env(safe-area-inset-bottom)); + padding: 10px 10px; + border-radius: 24px; + } + #chat:empty ~ .composer{ + top: auto; + bottom: calc(18px + env(safe-area-inset-bottom)); + transform: translateX(-50%); + } + + .msg.user > div{ max-width: 92% !important; } + .msg.user .bubble{ max-width: 100%; } +} + +/* ========================================================= + 7) Developer mode：Right sidebar (devbar) + ========================================================= */ +.devbar{ + position: fixed; + right: 0; + top: 0; + height: 100vh; + width: var(--devbar-current); + z-index: 56; + + overflow: hidden; + + background: var(--surface); + border-left: 1px solid var(--border-weak); + box-shadow: var(--shadow-soft); + backdrop-filter: blur(10px); + + transition: width 0.18s ease; +} +.devbar-inner{ + height: 100%; + padding: 12px; + display: flex; + flex-direction: column; + gap: 10px; +} + +.devbar-icon-btn{ + width: 44px; + height: 44px; + border-radius: 999px; + border: 1px solid var(--border-weak); + background: transparent; + color: var(--text); + display: grid; + place-items: center; + cursor: pointer; +} +.devbar-icon-btn:hover{ + background: var(--surface-2); + border-color: var(--border); +} + +.devbar-title{ + font-size: 13px; + color: var(--muted); + padding: 0 2px 4px; + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} +.devbar-log{ + flex: 1 1 auto; + overflow: auto; + border-top: 1px solid var(--border-weak); + padding-top: 10px; +} + +.devlog-item{ + border: 1px solid var(--border-weak); + border-radius: 14px; + background: var(--surface); + padding: 10px 12px; + margin-bottom: 10px; +} +.devlog-head{ + font-family: var(--mono); + font-size: 12px; + color: var(--muted); + margin-bottom: 8px; + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} +.devlog-pre{ + margin: 0; + font-family: var(--mono); + font-size: 12px; + line-height: 1.55; + color: var(--muted); + white-space: pre-wrap; + word-break: break-word; +} + +body.dev-mode.devbar-collapsed .devbar-inner{ padding: 12px 6px; } +body.dev-mode.devbar-collapsed .devbar-title, +body.dev-mode.devbar-collapsed .devbar-log{ display: none; } + +/* 右下角“到底部”按钮 */ +.scroll-bottom{ + position: fixed; + right: calc(48px + var(--right-offset) + env(safe-area-inset-right)); + bottom: calc(24px + env(safe-area-inset-bottom) + 120px); + z-index: 62; + + width: 44px; + height: 44px; + border-radius: 999px; + + border: 1px solid var(--border-weak); + background: var(--surface); + box-shadow: var(--shadow-soft); + backdrop-filter: blur(10px); + + display: grid; + place-items: center; + cursor: pointer; +} +.scroll-bottom:hover{ + background: var(--surface-2); + border-color: var(--border); +} +.scroll-bottom:active{ transform: translateY(1px); } + +/* 移动端适配 */ + +/* 1) runtime vars + safe-area fallbacks */ +:root{ + --vvh: 100vh; /* visual viewport height (px) */ + --kb: 0px; /* keyboard overlay inset (px) */ + --composer-h: 140px; /* measured composer height (px) */ + --composer-gap: 24px; /* distance from bottom edge */ + + /* safe-area insets fallback */ + --sat: 0px; + --sar: 0px; + --sab: 0px; + --sal: 0px; + + /* solid surface fallback for browsers without backdrop-filter */ + --surface-solid: #ffffff; +} +@media (prefers-color-scheme: dark){ + :root{ --surface-solid: #141416; } +} + +@supports (padding-top: env(safe-area-inset-top)){ + :root{ + --sat: env(safe-area-inset-top); + --sar: env(safe-area-inset-right); + --sab: env(safe-area-inset-bottom); + --sal: env(safe-area-inset-left); + } +} +/* legacy iOS (constant()) */ +@supports (padding-top: constant(safe-area-inset-top)){ + :root{ + --sat: constant(safe-area-inset-top); + --sar: constant(safe-area-inset-right); + --sab: constant(safe-area-inset-bottom); + --sal: constant(safe-area-inset-left); + } +} + +/* 2) typography / tap */ +html{ + -webkit-text-size-adjust: 100%; + text-size-adjust: 100%; +} +button, input, textarea, select, a{ + -webkit-tap-highlight-color: transparent; +} +button, input, textarea, select{ + font: inherit; +} + +/* 3) keep center between sidebars*/ +body{ + --center-shift: calc(((var(--content-offset) - var(--right-offset)) * 0.5) + ((var(--sal) - var(--sar)) * 0.5)); +} + +/* 4) dvh for modern mobile browsers (address bar) */ +.main{ min-height: 100vh; min-height: 100dvh; } +.sidebar, .devbar{ height: 100vh; height: 100dvh; } + +/* 5) add iOS Safari prefix for backdrop-filter */ +.sidebar, .topbar, .composer, .toast, .devbar, .scroll-bottom, .modal-body{ + backdrop-filter: blur(10px); + -webkit-backdrop-filter: blur(10px); +} + +/* 6) no backdrop-filter: use solid surfaces for readability */ +@supports not ((-webkit-backdrop-filter: blur(1px)) or (backdrop-filter: blur(1px))){ + .sidebar, .topbar, .composer, .toast, .devbar, .scroll-bottom, .modal-body{ + background: var(--surface-solid); + } +} + +/* 7) safe-area top for the sticky header */ +.topbar{ + height: calc(var(--topbar-h) + var(--sat)); + padding: var(--sat) 16px 0; +} +@media (max-width: 640px){ + .topbar{ + height: calc(52px + var(--sat)); + padding: var(--sat) 12px 0; + } + :root{ --composer-gap: 14px; } /* compact bottom gap on mobile */ +} + +/* 8) safe-area top for sidebars so the first button isn't under the status bar */ +.sidebar-inner{ padding-top: calc(12px + var(--sat)); } +body.sidebar-collapsed .sidebar-inner{ padding-top: calc(12px + var(--sat)); } + +.devbar-inner{ padding-top: calc(12px + var(--sat)); } +body.dev-mode.devbar-collapsed .devbar-inner{ padding-top: calc(12px + var(--sat)); } + +/* 9) dynamic chat bottom padding = composer height + gaps (avoid last msg covered) */ +.chat{ + padding-bottom: calc(var(--composer-h) + var(--composer-gap) + var(--sab) + var(--kb) + 24px); +} +@media (max-width: 640px){ + .chat{ + padding-bottom: calc(var(--composer-h) + var(--composer-gap) + var(--sab) + var(--kb) + 18px); + } +} + +/* 10) hero width avoids safe-area left/right */ +.hero{ + width: min(var(--maxw), calc(100vw - var(--content-offset) - var(--right-offset) - 2rem - var(--sal) - var(--sar))); +} + +/* 11) composer: bottom uses safe-area + keyboard inset; width avoids safe-area */ +.composer{ + bottom: calc(var(--composer-gap) + var(--sab) + var(--kb)); + width: min(var(--maxw), calc(100vw - var(--content-offset) - var(--right-offset) - 2rem - var(--sal) - var(--sar))); +} +@media (max-width: 640px){ + .composer{ + width: calc(100vw - var(--content-offset) - var(--right-offset) - 1.25rem - var(--sal) - var(--sar)); + bottom: calc(var(--composer-gap) + var(--sab) + var(--kb)); + } + #chat:empty ~ .composer{ + bottom: calc(18px + var(--sab) + var(--kb)); + } +} + +/* 12) toast / scroll-to-bottom always stays above composer (and keyboard) */ +.toast{ + bottom: calc(var(--composer-gap) + var(--sab) + var(--kb) + var(--composer-h) + 12px); +} +.scroll-bottom{ + right: calc(48px + var(--right-offset) + var(--sar)); + bottom: calc(var(--composer-gap) + var(--sab) + var(--kb) + var(--composer-h) + 16px); +} + +/* 13) iOS Safari: prevent focus auto-zoom on textarea/select (font-size >= 16px) */ +@media (max-width: 640px){ + .prompt{ font-size: 16px; line-height: 22px; } + .sidebar-model-select{ font-size: 16px; } +} + +/* 14) touch devices: avoid sticky :hover */ +@media (hover: none) and (pointer: coarse){ + .sidebar-icon-btn:hover, + .sidebar-action:hover, + .icon-btn:hover, + .ghost-icon-btn:hover, + .tool-preview-btn:hover, + .devbar-icon-btn:hover{ + background: transparent; + border-color: var(--border-weak); + opacity: 0.68; + } + + .sidebar-model-select:hover{ + background: var(--surface-2); + border-color: var(--border-weak); + } + + .scroll-bottom:hover{ + background: var(--surface); + border-color: var(--border-weak); + } + + .send-btn:hover{ transform: none; } + .tool-preview-link:hover, + .modal-close:hover{ opacity: 1; } + .media-item:hover{ border-color: var(--border-weak); } +} + +/* ========================================================= + Lang switch (topbar) + ========================================================= */ +.lang-switch{ + display: flex; + align-items: center; + gap: 8px; + padding: 6px 10px; + /* border: 1px solid var(--border-weak); */ + border: 0; + border-radius: 999px; + background: var(--surface-2); +} + +.lang-chip{ + font-size: 12px; + font-weight: 650; + color: var(--muted); + user-select: none; +} + +body.lang-zh .lang-chip.lang-zh, +body.lang-en .lang-chip.lang-en{ + color: var(--text); +} + +.lang-toggle{ + position: relative; + width: 44px; + height: 24px; + display: inline-block; + cursor: pointer; +} + +.lang-toggle input{ + opacity: 0; + width: 0; + height: 0; +} + +.lang-slider{ + position: absolute; + inset: 0; + border-radius: 999px; + background: var(--surface); + border: 1px solid var(--border-weak); + transition: border-color .18s ease, background .18s ease; +} + +.lang-slider::before{ + content: ""; + position: absolute; + width: 18px; + height: 18px; + left: 3px; + top: 50%; + transform: translateY(-50%); + border-radius: 999px; + background: var(--text); + transition: transform .18s ease; +} + +/* checked => English */ +.lang-toggle input:checked + .lang-slider::before{ + transform: translate(20px, -50%); +} + +.lang-toggle input:focus-visible + .lang-slider{ + box-shadow: var(--ring); +} + +/* ========================================================= + Devbar collapsed: 只保留右侧中间一个小箭头（不显示一整列） + ========================================================= */ + +body.dev-mode.devbar-collapsed{ + --devbar-current: 0px; /* 覆盖原来的 56px */ + --right-offset: 0px; +} + +body.dev-mode.devbar-collapsed .devbar{ + width: 0; + background: transparent; + border-left: 0; + box-shadow: none; + backdrop-filter: none; + -webkit-backdrop-filter: none; + overflow: visible; +} + +body.dev-mode.devbar-collapsed .devbar-inner{ padding: 0; } +body.dev-mode.devbar-collapsed .devbar-title, +body.dev-mode.devbar-collapsed .devbar-log, +body.dev-mode.devbar-collapsed .devbar-sid{ display: none !important; } + +body.dev-mode.devbar-collapsed #devbarToggle{ + position: fixed; + top: 50%; + right: calc(10px + var(--sar)); + transform: translateY(-50%); + + width: 36px; + height: 36px; + border-radius: 999px; + + background: var(--surface); + border: 1px solid var(--border-weak); + box-shadow: var(--shadow-soft); + + z-index: 70; +} + +body.dev-mode.devbar-collapsed #devbarToggle svg{ + width: 18px; + height: 18px; +} + +body.dev-mode:not(.devbar-collapsed) #devbarToggle svg{ + transform: rotate(180deg); +} + +.topbar > .brand{ align-items: center; gap: 10px; } +.brand-img{ height: 48px; width: auto; display: block; } +@media (max-width: 640px){ .brand-img{ height: 22px; } } + +#uploadBtn svg{ + stroke-width: 2; +} + +.sidebar-fields{ + display: flex; + flex-direction: column; + gap: 8px; +} + +/* ========================================================= + Ghost icon buttons + ========================================================= */ +.ghost-icon-btn{ + width: 44px; + height: 44px; + border-radius: 999px; + border: 1px solid var(--border-weak); + background: transparent; + color: var(--text); + + display: grid; + place-items: center; + cursor: pointer; + flex: 0 0 auto; + + opacity: 0.68; + + transition: + opacity .12s ease, + background .12s ease, + border-color .12s ease, + transform .08s ease; + + text-decoration: none; + user-select: none; +} + +.ghost-icon-btn:hover{ + opacity: 1; + background: var(--surface-2); + border-color: var(--border); +} + +.ghost-icon-btn:active, +.ghost-icon-btn.is-active{ + opacity: 1; + background: var(--surface-2); + border-color: var(--border); + transform: translateY(1px); +} + +.ghost-icon-btn:focus-visible{ + box-shadow: var(--ring); +} + +.ghost-icon-btn img.os-icon{ + width: 20px; + height: 20px; + display: block; + object-fit: contain; + pointer-events: none; +} + +.ghost-icon-btn.sm{ + width: 44px; + height: 44px; +} +.ghost-icon-btn.sm img.os-icon{ + width: 36px; + height: 36px; +} + +.topbar-links{ + display: flex; + align-items: center; + gap: 6px; +} + +@media (max-width: 640px){ + .topbar-links{ gap: 4px; } + .ghost-icon-btn.sm{ width: 34px; height: 34px; } + .ghost-icon-btn.sm img.os-icon{ width: 18px; height: 18px; } +} + +.topbar-links .ghost-icon-btn{ + border: 0; + background: transparent; + border-radius: 0; + opacity: 0.68; +} + +.topbar-links .ghost-icon-btn:hover, +.topbar-links .ghost-icon-btn:active, +.topbar-links .ghost-icon-btn.is-active{ + background: transparent; + opacity: 1; +} + +#quickPromptBtn{ + width: 48px; + height: 48px; +} + +#quickPromptBtn img.os-icon{ + width: 26px; + height: 26px; +} + +/* 顶栏三按钮与语言切换的间距 */ +.topbar-links{ + margin-right: 30px; +} + +.topbar-pill{ + display: inline-flex; + align-items: center; + gap: 8px; + + height: 36px; + padding: 0 12px 0 10px; + border-radius: 999px; + + border: 0; + background: transparent; + opacity: 0.72; + + color: var(--text); + text-decoration: none; + cursor: pointer; + user-select: none; + + transition: opacity .12s ease, background .12s ease, transform .08s ease; +} + +.topbar-pill:hover{ + opacity: 1; + background: var(--surface-2); +} +.topbar-pill:active{ + opacity: 1; + background: var(--surface-2); + transform: translateY(1px); +} + + +.topbar-pill .os-icon{ + width: 30px; + height: 30px; + display: block; + object-fit: contain; + pointer-events: none; +} + +.topbar-pill-text{ + font-size: 13px; + font-weight: 650; + letter-spacing: -0.01em; + white-space: nowrap; + color: var(--muted); +} +.topbar-pill:hover .topbar-pill-text, +.topbar-pill:active .topbar-pill-text{ + color: var(--text); +} + +@media (max-width: 640px){ + .topbar-pill-text{ display: none; } + .topbar-pill{ + padding: 0 10px; + gap: 0; + } +} + +#quickPromptBtn{ + margin-right: 6px; +} + +/* .sidebar-help-tooltip{ + width: clamp(220px, 28vw, 320px); +} + +.sidebar-help-tooltip-body{ + text-align: justify; + text-justify: inter-ideograph; +} + +.sidebar-help-tooltip-cta{ + text-align: left; +} */ + +.sidebar-help-tooltip-link{ + display: inline-block; + margin-top: 6px; + font-size: 12px; + color: var(--text); + text-decoration: underline; + text-underline-offset: 2px; + cursor: pointer; + opacity: 0.92; +} + +.sidebar-help-tooltip-link:hover{ + opacity: 1; +} \ No newline at end of file diff --git a/web/static/user_guide.png b/web/static/user_guide.png new file mode 100644 index 0000000000000000000000000000000000000000..4273d166e1b7873caaf1f472fda384b49fbfad94 Binary files /dev/null and b/web/static/user_guide.png differ