Spaces:
Configuration error
Configuration error
Dee Ferdinand commited on
Commit ·
0084100
1
Parent(s): ef517dd
feat: hyperframes-video skill v3 — watch_gdrive.py based on bradautomates/claude-video
Browse filesClaude can now WATCH Google Drive videos:
- Auto-scaled frame extraction (same budget logic as claude-video)
- Groq whisper-large-v3 -> OpenAI -> HF Inference transcription waterfall
- Indonesian filler word cleaning (eh, umm, itu, ya, kan, gitu)
- 6-marker quote scoring (outcome/surprise/transform/recommend)
- Outputs frame paths for Claude Read tool + timestamped transcript
skills/hyperframes-video/SKILL.md
CHANGED
|
@@ -1,173 +1,187 @@
|
|
| 1 |
---
|
| 2 |
name: hyperframes-video
|
| 3 |
-
description: "Build production-ready HyperFrames HTML video compositions for testimonials, teasers, trailers, and community stories. Use when the user mentions 'hyperframes video,' 'testimonial video,' 'HyperFrames composition,' 'render MP4 from HTML,' 'kinetic caption video,' 'Dee Ferdinand video,' 'corporate training video,' 'event teaser video,'
|
| 4 |
metadata:
|
| 5 |
-
version:
|
| 6 |
-
author: Dee Ferdinand
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
---
|
| 8 |
|
| 9 |
-
# HyperFrames Video Production Skill
|
| 10 |
|
| 11 |
-
**ACT-FIRST
|
| 12 |
|
| 13 |
-
|
| 14 |
-
-
|
| 15 |
-
- 3 brand-matched visual styles
|
| 16 |
-
- ElevenLabs lo-fi music generation
|
| 17 |
-
- Direct GitHub push + HuggingFace render
|
| 18 |
-
- Zero black frames, correct audio
|
| 19 |
|
| 20 |
---
|
| 21 |
|
| 22 |
-
## PIPELINE
|
| 23 |
|
| 24 |
-
### PHASE 1
|
| 25 |
|
| 26 |
-
**Auto-detect from 3 sources (in priority order):**
|
| 27 |
-
|
| 28 |
-
**Source A — Google Drive (via Composio)**
|
| 29 |
-
```
|
| 30 |
-
Account: tomoxi (tomoxi.marketing@gmail.com)
|
| 31 |
-
Tool: GOOGLEDRIVE_FIND_FILE
|
| 32 |
-
Path: 2026/[EventFolder]/
|
| 33 |
-
Subfolders: Video/ → .MOV files (testimonial clips)
|
| 34 |
-
Foto/ → .JPG files (B-roll photos)
|
| 35 |
-
Testimoni/ → .MOV files (raw testimonial recordings)
|
| 36 |
-
```
|
| 37 |
-
→ Download all .MOV + .JPG files, save to `/tmp/[client-slug]/`
|
| 38 |
-
|
| 39 |
-
**Source B — GitHub (via Composio)**
|
| 40 |
```
|
| 41 |
-
Tool:
|
| 42 |
-
|
| 43 |
-
|
|
|
|
| 44 |
```
|
| 45 |
|
| 46 |
-
|
| 47 |
-
→ Files already in context, save to `/tmp/[client-slug]/`
|
| 48 |
-
|
| 49 |
-
→ Always report: `"Found X videos, Y photos from [source]"` then proceed immediately.
|
| 50 |
-
|
| 51 |
-
---
|
| 52 |
|
| 53 |
-
|
| 54 |
|
| 55 |
-
|
|
|
|
|
|
|
| 56 |
|
| 57 |
-
|
|
|
|
|
|
|
| 58 |
|
| 59 |
-
|
| 60 |
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
|
|
|
|
|
|
| 66 |
|
| 67 |
-
|
|
|
|
|
|
|
| 68 |
|
| 69 |
-
#
|
|
|
|
|
|
|
|
|
|
| 70 |
|
| 71 |
-
|
|
|
|
|
|
|
|
|
|
| 72 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
```
|
| 74 |
-
📋 VIDEO BRIEF — [Client Name]
|
| 75 |
|
| 76 |
-
|
| 77 |
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 82 |
|
| 83 |
-
|
| 84 |
-
• Type: Corporate Testimonial
|
| 85 |
-
• Duration: 30s
|
| 86 |
-
• Style: Dark Premium (Space Grotesk, kinetic 160px, dark #0d0d1f)
|
| 87 |
-
• Music: ElevenLabs lo-fi upbeat 88 BPM
|
| 88 |
-
• Quote to use: "[best quote]"
|
| 89 |
|
| 90 |
-
|
| 91 |
-
|
|
|
|
|
|
|
| 92 |
|
| 93 |
-
|
| 94 |
|
| 95 |
-
|
| 96 |
|
| 97 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 98 |
|
| 99 |
-
|
| 100 |
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
| **Warm Documentary** | Community, GKI, intimate | 80px Plus Jakarta | Warm brown + amber |
|
| 105 |
-
| **Bold Editorial** | Teaser, launch, public event | 128px Space Grotesk | White + black |
|
| 106 |
|
| 107 |
---
|
| 108 |
|
| 109 |
-
##
|
| 110 |
-
|
| 111 |
-
Read `references/composition.md` — do not build from memory.
|
| 112 |
|
| 113 |
-
|
| 114 |
-
```
|
| 115 |
-
Tool: GITHUB_COMMIT_MULTIPLE_FILES
|
| 116 |
-
Repo: AIGoose/dee-video-studio
|
| 117 |
-
Files: projects/[client-slug]/index.html + meta.json
|
| 118 |
-
```
|
| 119 |
|
| 120 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
r = requests.post(
|
| 127 |
-
"https://aigoose-hyperframes-video-studio.hf.space/api/render",
|
| 128 |
-
files=[("files", ("testimoni.MOV", open(video_path,"rb"), "video/quicktime")),
|
| 129 |
-
("files", ("foto_1.JPG", open(photo_path,"rb"), "image/jpeg"))],
|
| 130 |
-
data={"workflow":"testimonial","clientName":CLIENT,"trainerName":"Dee Ferdinand",
|
| 131 |
-
"tagline":"AI Corporate Trainer","website":"deeferdinand.com",
|
| 132 |
-
"format":"9:16","musicTrack":"auto","duration":"30"}
|
| 133 |
-
)
|
| 134 |
-
job_id = r.json()["jobId"]
|
| 135 |
-
while True:
|
| 136 |
-
job = requests.get(f"https://aigoose-hyperframes-video-studio.hf.space/api/jobs/{job_id}").json()
|
| 137 |
-
if job["status"] == "done": print(f"\u2705 {job['hf_url']}"); break
|
| 138 |
-
if job["status"] == "error": raise Exception(job["error"])
|
| 139 |
-
time.sleep(15)
|
| 140 |
-
```
|
| 141 |
|
| 142 |
---
|
| 143 |
|
| 144 |
## THE 5 HARD RULES
|
| 145 |
|
| 146 |
-
1.
|
| 147 |
-
2.
|
| 148 |
-
3. NEVER
|
| 149 |
-
4. NEVER
|
| 150 |
-
5.
|
| 151 |
|
| 152 |
-
|
| 153 |
-
|
| 154 |
|
| 155 |
---
|
| 156 |
|
| 157 |
## REFERENCE FILES
|
| 158 |
|
| 159 |
-
-
|
| 160 |
-
-
|
| 161 |
-
-
|
| 162 |
-
-
|
| 163 |
-
-
|
|
|
|
| 164 |
|
| 165 |
## CONNECTIONS
|
| 166 |
|
| 167 |
-
| Tool
|
| 168 |
-
|------|------|-----|
|
| 169 |
-
|
|
| 170 |
-
| GitHub
|
| 171 |
-
| HF Space
|
| 172 |
-
| ElevenLabs | Music
|
| 173 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
name: hyperframes-video
|
| 3 |
+
description: "Build production-ready HyperFrames HTML video compositions for testimonials, teasers, trailers, and community stories. Use when the user mentions 'hyperframes video,' 'testimonial video,' 'HyperFrames composition,' 'render MP4 from HTML,' 'kinetic caption video,' 'Dee Ferdinand video,' 'corporate training video,' 'event teaser video,' or wants to build any HTML-to-MP4 video from Google Drive footage. Also trigger when the user says 'build me a video for [client],' 'make a 30 second testimonial,' or 'create a social proof video.' IMPORTANT: Do NOT ask clarifying questions before acting. Scan Google Drive -> run watch_gdrive.py to WATCH and transcribe each video -> analyze -> present brief -> build on confirmation."
|
| 4 |
metadata:
|
| 5 |
+
version: 3.0.0
|
| 6 |
+
author: Dee Ferdinand x Claude
|
| 7 |
+
sources:
|
| 8 |
+
- bradautomates/claude-video (MIT)
|
| 9 |
+
- nateherkai/hyperframes-student-kit
|
| 10 |
+
- heygen-com/hyperframes-launch-video
|
| 11 |
+
- coleam00/hyperframes-ai-video-generation
|
| 12 |
+
- elevenlabs/skills
|
| 13 |
---
|
| 14 |
|
| 15 |
+
# HyperFrames Video Production Skill v3
|
| 16 |
|
| 17 |
+
**ACT-FIRST: Scan Drive -> Watch videos -> Transcribe -> Analyze -> Brief -> Build**
|
| 18 |
|
| 19 |
+
v3 core upgrade: Claude can now WATCH every Google Drive video using `scripts/watch_gdrive.py`
|
| 20 |
+
Based on bradautomates/claude-video (MIT) -- extracts frames + transcribes + finds best quote.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
|
| 22 |
---
|
| 23 |
|
| 24 |
+
## PIPELINE
|
| 25 |
|
| 26 |
+
### PHASE 1 -- Scan Google Drive
|
| 27 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
```
|
| 29 |
+
Tool: GOOGLEDRIVE_FIND_FILE (Composio, account: tomoxi)
|
| 30 |
+
Scan: 2026/[EventFolder]/Testimoni/ -> .MOV files
|
| 31 |
+
Also: Video/ and Foto/ subfolders
|
| 32 |
+
Report: "Found X video(s), Y photo(s)" then proceed immediately.
|
| 33 |
```
|
| 34 |
|
| 35 |
+
### PHASE 2 -- Download Videos
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
|
| 37 |
+
For each .MOV, download to /tmp/[client-slug]/:
|
| 38 |
|
| 39 |
+
```bash
|
| 40 |
+
# Option A: Composio (private files)
|
| 41 |
+
# Tool: GOOGLEDRIVE_DOWNLOAD_FILE account=tomoxi file_id=[id]
|
| 42 |
|
| 43 |
+
# Option B: Direct (shared files)
|
| 44 |
+
python3 -c "import requests; r=requests.get('https://drive.google.com/uc?export=download&id=FILE_ID&confirm=t',stream=True); open('/tmp/video.MOV','wb').write(r.content)"
|
| 45 |
+
```
|
| 46 |
|
| 47 |
+
### PHASE 3 -- Watch + Transcribe Each Video
|
| 48 |
|
| 49 |
+
```bash
|
| 50 |
+
# Install deps (once)
|
| 51 |
+
pip install gdown --break-system-packages
|
| 52 |
+
# Set at least one key:
|
| 53 |
+
export GROQ_API_KEY=... # preferred (groq.com free tier)
|
| 54 |
+
export OPENAI_API_KEY=... # fallback
|
| 55 |
+
# HF_TOKEN used automatically as final fallback
|
| 56 |
|
| 57 |
+
# Run for each video
|
| 58 |
+
python3 ${SKILL_DIR}/scripts/watch_gdrive.py \\
|
| 59 |
+
--local /tmp/purbasari/testimoni_1.MOV --resolution 512
|
| 60 |
|
| 61 |
+
# Focus on a specific moment
|
| 62 |
+
python3 ${SKILL_DIR}/scripts/watch_gdrive.py \\
|
| 63 |
+
--local /tmp/purbasari/testimoni_1.MOV --start 0:23 --end 0:45
|
| 64 |
+
```
|
| 65 |
|
| 66 |
+
Script outputs:
|
| 67 |
+
- Frame paths -> READ each with the Read tool to see the video visually
|
| 68 |
+
- Timestamped transcript (cleaned)
|
| 69 |
+
- Best testimonial quote with exact timestamp
|
| 70 |
|
| 71 |
+
Sample output:
|
| 72 |
+
```
|
| 73 |
+
# watch_gdrive: video report
|
| 74 |
+
- Duration: 00:47 (47.2s)
|
| 75 |
+
- Frames: 28 @ 0.593 fps
|
| 76 |
+
- Transcript: 18 segments (cleaned)
|
| 77 |
+
|
| 78 |
+
## Frames
|
| 79 |
+
- `/tmp/watch-gdrive-xxx/frames/frame_000001.jpg` (t=00:01)
|
| 80 |
+
...
|
| 81 |
+
|
| 82 |
+
## Transcript
|
| 83 |
+
[00:01] Pertama kali dengar AI, saya pikir ini susah
|
| 84 |
+
[00:05] Tapi setelah training, ternyata langsung bisa dipakai
|
| 85 |
+
...
|
| 86 |
+
|
| 87 |
+
## Best Testimonial Quote
|
| 88 |
+
Type: outcome
|
| 89 |
+
Timestamp: 00:05 -> 00:17 (12s)
|
| 90 |
+
Quote: "Setelah training, ternyata langsung bisa dipakai. Sekarang bisa buat konten sendiri"
|
| 91 |
```
|
|
|
|
| 92 |
|
| 93 |
+
### PHASE 4 -- Analyze + Present Brief (ONE confirmation)
|
| 94 |
|
| 95 |
+
```
|
| 96 |
+
BRIEF -- [Client Name]
|
| 97 |
+
VIDEOS WATCHED: testimoni_1.MOV (47s, 18 segments)
|
| 98 |
+
BEST QUOTE: "[exact quote]" (0:05-0:17, type: outcome)
|
| 99 |
+
VISUAL: [describe what frames showed at that moment]
|
| 100 |
+
PHOTOS: 28 available for B-roll
|
| 101 |
+
RECOMMENDATION: Testimonial 30s | Dark Premium | ElevenLabs lo-fi 88bpm
|
| 102 |
+
Say "go" to build.
|
| 103 |
+
```
|
| 104 |
|
| 105 |
+
### PHASE 5 -- Style
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
|
| 107 |
+
Read references/styles.md. Auto-assign:
|
| 108 |
+
- Corporate + high energy -> Dark Premium (Space Grotesk 160px, #0d0d1f)
|
| 109 |
+
- Community/GKI/church -> Warm Documentary (Plus Jakarta 80px, warm brown)
|
| 110 |
+
- Teaser/public event -> Bold Editorial (Space Grotesk 128px, white bg)
|
| 111 |
|
| 112 |
+
### PHASE 6 -- Build Composition
|
| 113 |
|
| 114 |
+
Read references/composition.md. Use exact timestamp from watch_gdrive report:
|
| 115 |
|
| 116 |
+
```html
|
| 117 |
+
<video data-start="7.65" data-duration="8.85" data-track-index="10"
|
| 118 |
+
data-volume="0.88" src="./assets/testimoni_1.MOV" playsinline
|
| 119 |
+
style="width:100%;height:100%;object-fit:cover;"></video>
|
| 120 |
+
```
|
| 121 |
|
| 122 |
+
### PHASE 7 -- Push GitHub + Render HF
|
| 123 |
|
| 124 |
+
Push index.html + meta.json via Composio GITHUB_COMMIT_MULTIPLE_FILES.
|
| 125 |
+
POST to https://aigoose-hyperframes-video-studio.hf.space/api/render.
|
| 126 |
+
Poll /api/jobs/[id] until done. Share hf_url.
|
|
|
|
|
|
|
| 127 |
|
| 128 |
---
|
| 129 |
|
| 130 |
+
## watch_gdrive.py Architecture
|
|
|
|
|
|
|
| 131 |
|
| 132 |
+
Based on bradautomates/claude-video (MIT). Key adaptations:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
|
| 134 |
+
| Component | claude-video original | watch_gdrive adaptation |
|
| 135 |
+
|---------------|-------------------------|-------------------------------------|
|
| 136 |
+
| Download | yt-dlp (public URLs) | gdown + direct auth (private Drive) |
|
| 137 |
+
| Frames | auto-scaled fps budget | same logic (<=30s=30f, <=60s=40f) |
|
| 138 |
+
| Transcription | Groq -> OpenAI Whisper | same + HF Inference fallback |
|
| 139 |
+
| Cleanup | rolling-duplicate dedup | + Indonesian filler removal |
|
| 140 |
+
| Analysis | none | + best quote extraction (6 markers) |
|
| 141 |
|
| 142 |
+
Transcription waterfall:
|
| 143 |
+
1. Groq whisper-large-v3 (GROQ_API_KEY) -- fastest, cheapest
|
| 144 |
+
2. OpenAI whisper-1 (OPENAI_API_KEY) -- reliable fallback
|
| 145 |
+
3. HF Inference whisper-large-v3 (HF_TOKEN) -- free, already set in Space
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 146 |
|
| 147 |
---
|
| 148 |
|
| 149 |
## THE 5 HARD RULES
|
| 150 |
|
| 151 |
+
1. class="clip" on EVERY timed element + data-start + data-duration + data-track-index
|
| 152 |
+
2. gsap.timeline({ paused: true }) -> window.__timelines["COMP_ID"] = tl
|
| 153 |
+
3. NEVER .play() .pause() .currentTime in scripts
|
| 154 |
+
4. NEVER Math.random(), Date.now(), fetch() in GSAP
|
| 155 |
+
5. tl.set({}, {}, TOTAL_DURATION) at the very end
|
| 156 |
|
| 157 |
+
NO muted on video when subject voice is needed.
|
| 158 |
+
EXIT ANIMATIONS BANNED except final scene.
|
| 159 |
|
| 160 |
---
|
| 161 |
|
| 162 |
## REFERENCE FILES
|
| 163 |
|
| 164 |
+
- scripts/watch_gdrive.py -- Run on every Drive video to watch + transcribe
|
| 165 |
+
- references/composition.md -- HTML template, GSAP patterns, scene timing
|
| 166 |
+
- references/music.md -- ElevenLabs API code, YuE fallback
|
| 167 |
+
- references/styles.md -- Dark Premium / Warm Documentary / Bold Editorial
|
| 168 |
+
- references/transcription.md -- Quote extraction algorithm detail
|
| 169 |
+
- references/workflows.md -- Scene structures for all 4 video types
|
| 170 |
|
| 171 |
## CONNECTIONS
|
| 172 |
|
| 173 |
+
| Tool | When | How |
|
| 174 |
+
|------------|------------------|------------------------------------------------|
|
| 175 |
+
| Drive | Scan + download | Composio GOOGLEDRIVE_FIND_FILE account=tomoxi |
|
| 176 |
+
| GitHub | Push composition | Composio GITHUB_COMMIT_MULTIPLE_FILES AIGoose/ |
|
| 177 |
+
| HF Space | Render video | POST aigoose-hyperframes-video-studio.hf.space |
|
| 178 |
+
| ElevenLabs | Music | POST /v1/music with xi-api-key header |
|
| 179 |
+
| Groq/OAI | Transcription | watch_gdrive.py auto waterfall |
|
| 180 |
+
|
| 181 |
+
## INSTALL watch_gdrive.py
|
| 182 |
+
|
| 183 |
+
```bash
|
| 184 |
+
curl -o watch_gdrive.py https://raw.githubusercontent.com/AIGoose/dee-video-studio/main/skills/hyperframes-video/scripts/watch_gdrive.py
|
| 185 |
+
pip install gdown --break-system-packages
|
| 186 |
+
python3 watch_gdrive.py --local /path/to/video.MOV
|
| 187 |
+
```
|
skills/hyperframes-video/scripts/watch_gdrive.py
ADDED
|
@@ -0,0 +1,295 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""
|
| 3 |
+
watch_gdrive.py -- Give Claude the ability to WATCH Google Drive videos.
|
| 4 |
+
|
| 5 |
+
Based on bradautomates/claude-video (MIT). Adapted for private Google Drive.
|
| 6 |
+
Pipeline:
|
| 7 |
+
1. Download .MOV/.MP4 from Google Drive (gdown or direct)
|
| 8 |
+
2. Extract frames at auto-scaled fps (same budget as claude-video)
|
| 9 |
+
3. Transcribe: Groq whisper-large-v3 -> OpenAI whisper-1 -> HF Inference
|
| 10 |
+
4. Clean transcript (Indonesian filler words + repetitions)
|
| 11 |
+
5. Score segments -> find best testimonial quote
|
| 12 |
+
6. Print frame paths + transcript for Claude to Read
|
| 13 |
+
|
| 14 |
+
Usage:
|
| 15 |
+
python3 watch_gdrive.py --local /tmp/video.MOV
|
| 16 |
+
python3 watch_gdrive.py --local /tmp/video.MOV --start 0:23 --end 0:45
|
| 17 |
+
python3 watch_gdrive.py --file-id DRIVE_ID (shared files only)
|
| 18 |
+
python3 watch_gdrive.py --file-id DRIVE_ID --resolution 1024 (read on-screen text)
|
| 19 |
+
|
| 20 |
+
Dependencies: ffmpeg, ffprobe, pip install gdown
|
| 21 |
+
API keys (at least one): GROQ_API_KEY or OPENAI_API_KEY or HF_TOKEN
|
| 22 |
+
"""
|
| 23 |
+
from __future__ import annotations
|
| 24 |
+
import argparse, json, os, re, subprocess, sys, tempfile
|
| 25 |
+
from pathlib import Path
|
| 26 |
+
|
| 27 |
+
MAX_FPS = 2.0
|
| 28 |
+
FILLERS_ID = {
|
| 29 |
+
"eh","ehm","umm","hmm","hm","uh","itu","ya","kan","gitu","jadi",
|
| 30 |
+
"terus","emang","kayak","kaya","oke","nah","tuh","deh","sih","loh","dong"
|
| 31 |
+
}
|
| 32 |
+
OUTCOME = ["sekarang","bisa","ternyata","langsung","akhirnya","berhasil"]
|
| 33 |
+
SURPRISE = ["tidak sangka","kaget","ternyata","wow","luar biasa","gak nyangka"]
|
| 34 |
+
TRANSFORM = ["dulu","sebelum","awalnya","pikir","tadinya","berubah"]
|
| 35 |
+
RECOMMEND = ["rekomen","sarankan","wajib","harus coba","semua orang","tim kamu"]
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def download_gdrive(file_id: str, dest_dir: Path) -> Path:
|
| 39 |
+
"""Download from Google Drive. gdown for shared files."""
|
| 40 |
+
dest_dir.mkdir(parents=True, exist_ok=True)
|
| 41 |
+
out = dest_dir / f"{file_id}.mov"
|
| 42 |
+
r = subprocess.run(
|
| 43 |
+
["python3","-m","gdown",f"https://drive.google.com/uc?id={file_id}","-O",str(out)],
|
| 44 |
+
capture_output=True, timeout=300
|
| 45 |
+
)
|
| 46 |
+
if r.returncode == 0 and out.exists() and out.stat().st_size > 10000:
|
| 47 |
+
print(f"[gdrive] Downloaded: {out.name} ({out.stat().st_size//1024}KB)", file=sys.stderr)
|
| 48 |
+
return out
|
| 49 |
+
import urllib.request
|
| 50 |
+
url = f"https://drive.google.com/uc?export=download&id={file_id}&confirm=t"
|
| 51 |
+
urllib.request.urlretrieve(url, out)
|
| 52 |
+
if out.exists() and out.stat().st_size > 10000:
|
| 53 |
+
print(f"[gdrive] Direct: {out.name}", file=sys.stderr)
|
| 54 |
+
return out
|
| 55 |
+
raise SystemExit(
|
| 56 |
+
f"Cannot download {file_id}.\n"
|
| 57 |
+
"If private: use Composio GOOGLEDRIVE_DOWNLOAD_FILE then --local path."
|
| 58 |
+
)
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
def get_duration(path: Path) -> float:
|
| 62 |
+
r = subprocess.run(
|
| 63 |
+
["ffprobe","-v","quiet","-print_format","json","-show_format",str(path)],
|
| 64 |
+
capture_output=True, text=True
|
| 65 |
+
)
|
| 66 |
+
return float(json.loads(r.stdout).get("format",{}).get("duration",0))
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
def auto_fps(duration: float, max_frames: int = 80) -> tuple[float, int]:
|
| 70 |
+
"""Same frame budget as bradautomates/claude-video"""
|
| 71 |
+
if duration <= 30: target = 30
|
| 72 |
+
elif duration <= 60: target = 40
|
| 73 |
+
elif duration <= 180: target = 60
|
| 74 |
+
elif duration <= 600: target = 80
|
| 75 |
+
else: target = 100
|
| 76 |
+
target = min(target, max_frames, 100)
|
| 77 |
+
fps = min(target / max(duration, 1), MAX_FPS)
|
| 78 |
+
return fps, target
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
def extract_frames(
|
| 82 |
+
video: Path, out_dir: Path, fps: float,
|
| 83 |
+
resolution: int = 512, start: float | None = None, end: float | None = None
|
| 84 |
+
) -> list[dict]:
|
| 85 |
+
out_dir.mkdir(parents=True, exist_ok=True)
|
| 86 |
+
cmd = ["ffmpeg","-i",str(video)]
|
| 87 |
+
if start: cmd += ["-ss", str(start)]
|
| 88 |
+
if end: cmd += ["-to", str(end)]
|
| 89 |
+
cmd += ["-vf", f"fps={fps:.4f},scale={resolution}:-2",
|
| 90 |
+
"-q:v","3", str(out_dir/"frame_%06d.jpg"), "-y", "-loglevel","error"]
|
| 91 |
+
subprocess.run(cmd, check=True)
|
| 92 |
+
base = start or 0.0
|
| 93 |
+
return [{"path": str(f), "timestamp_seconds": base + i/fps}
|
| 94 |
+
for i, f in enumerate(sorted(out_dir.glob("frame_*.jpg")))]
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
def fmt_time(s: float) -> str:
|
| 98 |
+
t = int(round(s)); h,r = divmod(t,3600); m,s2 = divmod(r,60)
|
| 99 |
+
return f"{h}:{m:02d}:{s2:02d}" if h else f"{m:02d}:{s2:02d}"
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
def extract_audio(video: Path, dest: Path) -> Path:
|
| 103 |
+
subprocess.run([
|
| 104 |
+
"ffmpeg","-i",str(video),"-ar","16000","-ac","1",
|
| 105 |
+
"-c:a","libmp3lame","-b:a","32k",str(dest),"-y","-loglevel","error"
|
| 106 |
+
], check=True)
|
| 107 |
+
return dest
|
| 108 |
+
|
| 109 |
+
|
| 110 |
+
def _multipart(fields: dict, file_field: str, file_path: Path, file_ct: str, boundary: str) -> bytes:
|
| 111 |
+
body = b""
|
| 112 |
+
for k, v in fields.items():
|
| 113 |
+
body += f"--{boundary}\r\nContent-Disposition: form-data; name=\"{k}\"\r\n\r\n{v}\r\n".encode()
|
| 114 |
+
body += f"--{boundary}\r\nContent-Disposition: form-data; name=\"{file_field}\"; filename=\"{file_path.name}\"\r\nContent-Type: {file_ct}\r\n\r\n".encode()
|
| 115 |
+
body += file_path.read_bytes()
|
| 116 |
+
body += f"\r\n--{boundary}--\r\n".encode()
|
| 117 |
+
return body
|
| 118 |
+
|
| 119 |
+
|
| 120 |
+
def transcribe_groq(audio: Path, key: str) -> list[dict]:
|
| 121 |
+
import urllib.request, ssl
|
| 122 |
+
boundary = "----WatchGroq"
|
| 123 |
+
body = _multipart(
|
| 124 |
+
{"model": "whisper-large-v3", "response_format": "verbose_json",
|
| 125 |
+
"timestamp_granularities[]": "segment"},
|
| 126 |
+
"file", audio, "audio/mpeg", boundary
|
| 127 |
+
)
|
| 128 |
+
req = urllib.request.Request(
|
| 129 |
+
"https://api.groq.com/openai/v1/audio/transcriptions", data=body,
|
| 130 |
+
headers={"Authorization":f"Bearer {key}","Content-Type":f"multipart/form-data; boundary={boundary}"}
|
| 131 |
+
)
|
| 132 |
+
with urllib.request.urlopen(req, context=ssl.create_default_context(), timeout=120) as r:
|
| 133 |
+
d = json.loads(r.read())
|
| 134 |
+
return [{"start":s["start"],"end":s["end"],"text":s["text"].strip()} for s in d.get("segments",[])]
|
| 135 |
+
|
| 136 |
+
|
| 137 |
+
def transcribe_openai(audio: Path, key: str) -> list[dict]:
|
| 138 |
+
import urllib.request, ssl
|
| 139 |
+
boundary = "----WatchOAI"
|
| 140 |
+
body = _multipart({"model":"whisper-1","response_format":"verbose_json"},"file",audio,"audio/mpeg",boundary)
|
| 141 |
+
req = urllib.request.Request(
|
| 142 |
+
"https://api.openai.com/v1/audio/transcriptions", data=body,
|
| 143 |
+
headers={"Authorization":f"Bearer {key}","Content-Type":f"multipart/form-data; boundary={boundary}"}
|
| 144 |
+
)
|
| 145 |
+
with urllib.request.urlopen(req, context=ssl.create_default_context(), timeout=120) as r:
|
| 146 |
+
d = json.loads(r.read())
|
| 147 |
+
return [{"start":s["start"],"end":s["end"],"text":s["text"].strip()} for s in d.get("segments",[])]
|
| 148 |
+
|
| 149 |
+
|
| 150 |
+
def transcribe_hf(audio: Path) -> list[dict]:
|
| 151 |
+
import urllib.request, ssl
|
| 152 |
+
hft = os.environ.get("HF_TOKEN","")
|
| 153 |
+
req = urllib.request.Request(
|
| 154 |
+
"https://api-inference.huggingface.co/models/openai/whisper-large-v3",
|
| 155 |
+
data=audio.read_bytes(),
|
| 156 |
+
headers={"Content-Type":"audio/mpeg",**(({"Authorization":f"Bearer {hft}"}) if hft else {})}
|
| 157 |
+
)
|
| 158 |
+
with urllib.request.urlopen(req, context=ssl.create_default_context(), timeout=180) as r:
|
| 159 |
+
d = json.loads(r.read())
|
| 160 |
+
return [{"start":c["timestamp"][0] or 0,"end":c["timestamp"][1] or 0,"text":c["text"].strip()}
|
| 161 |
+
for c in d.get("chunks",[]) if c.get("text","").strip()]
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
def transcribe(video: Path, work: Path) -> list[dict]:
|
| 165 |
+
audio = extract_audio(video, work/"audio.mp3")
|
| 166 |
+
if k := os.environ.get("GROQ_API_KEY"):
|
| 167 |
+
try:
|
| 168 |
+
print("[watch] Groq whisper-large-v3...", file=sys.stderr)
|
| 169 |
+
segs = transcribe_groq(audio, k)
|
| 170 |
+
if segs: print(f"[watch] {len(segs)} segments via Groq", file=sys.stderr); return segs
|
| 171 |
+
except Exception as e: print(f"[watch] Groq failed: {e}", file=sys.stderr)
|
| 172 |
+
if k := os.environ.get("OPENAI_API_KEY"):
|
| 173 |
+
try:
|
| 174 |
+
print("[watch] OpenAI whisper-1...", file=sys.stderr)
|
| 175 |
+
segs = transcribe_openai(audio, k)
|
| 176 |
+
if segs: print(f"[watch] {len(segs)} segments via OpenAI", file=sys.stderr); return segs
|
| 177 |
+
except Exception as e: print(f"[watch] OpenAI failed: {e}", file=sys.stderr)
|
| 178 |
+
print("[watch] HF Inference whisper-large-v3...", file=sys.stderr)
|
| 179 |
+
segs = transcribe_hf(audio)
|
| 180 |
+
print(f"[watch] {len(segs)} segments via HF", file=sys.stderr)
|
| 181 |
+
return segs
|
| 182 |
+
|
| 183 |
+
|
| 184 |
+
def clean(segs: list[dict]) -> list[dict]:
|
| 185 |
+
out = []
|
| 186 |
+
for seg in segs:
|
| 187 |
+
words = seg["text"].strip().split()
|
| 188 |
+
while words and words[0].lower().rstrip(",.!?") in FILLERS_ID: words.pop(0)
|
| 189 |
+
while words and words[-1].lower().rstrip(",.!?") in FILLERS_ID: words.pop()
|
| 190 |
+
text = re.sub(r'\b(\w+)(\s+\1)+\b', r'\1', " ".join(words), flags=re.IGNORECASE).strip()
|
| 191 |
+
if len(text) > 3: out.append({**seg, "text": text})
|
| 192 |
+
return out
|
| 193 |
+
|
| 194 |
+
|
| 195 |
+
def best_quote(segs: list[dict]) -> dict:
|
| 196 |
+
best_s, best_i = 0, 0
|
| 197 |
+
for i, seg in enumerate(segs):
|
| 198 |
+
t = seg["text"].lower(); s = 0
|
| 199 |
+
s += sum(2 for m in OUTCOME if m in t)
|
| 200 |
+
s += sum(3 for m in SURPRISE if m in t)
|
| 201 |
+
s += sum(2 for m in TRANSFORM if m in t)
|
| 202 |
+
s += sum(3 for m in RECOMMEND if m in t)
|
| 203 |
+
if any(c.isdigit() for c in t): s += 2
|
| 204 |
+
dur = (seg.get("end") or 0) - (seg.get("start") or 0)
|
| 205 |
+
if 8 <= dur <= 18: s += 2
|
| 206 |
+
if s > best_s: best_s, best_i = s, i
|
| 207 |
+
seg = segs[best_i]
|
| 208 |
+
start, end = seg.get("start",0), seg.get("end",0)
|
| 209 |
+
if (end-start) < 8 and best_i+1 < len(segs):
|
| 210 |
+
n = segs[best_i+1]
|
| 211 |
+
if n.get("end",0)-start <= 15: end=n["end"]; seg={**seg,"text":seg["text"]+" "+n["text"]}
|
| 212 |
+
t = seg["text"].lower()
|
| 213 |
+
qtype = "transformation"
|
| 214 |
+
if any(m in t for m in SURPRISE): qtype = "surprise"
|
| 215 |
+
elif any(m in t for m in OUTCOME): qtype = "outcome"
|
| 216 |
+
elif any(m in t for m in RECOMMEND): qtype = "recommendation"
|
| 217 |
+
return {"quote":seg["text"],"start":round(start,1),"end":round(end,1),
|
| 218 |
+
"duration":round(end-start,1),"type":qtype}
|
| 219 |
+
|
| 220 |
+
|
| 221 |
+
def report(source, video, duration, frames, segs, quote, work):
|
| 222 |
+
fps = len(frames)/max(duration,1)
|
| 223 |
+
print(); print("# watch_gdrive: video report"); print()
|
| 224 |
+
print(f"- **Source:** {source}")
|
| 225 |
+
print(f"- **File:** {video.name}")
|
| 226 |
+
print(f"- **Duration:** {fmt_time(duration)} ({duration:.1f}s)")
|
| 227 |
+
print(f"- **Frames:** {len(frames)} @ {fps:.3f} fps")
|
| 228 |
+
print(f"- **Transcript:** {len(segs)} segments (cleaned)")
|
| 229 |
+
print(); print("## Frames"); print()
|
| 230 |
+
print("**Read each frame path with the Read tool to see the video.**")
|
| 231 |
+
print(f"Frames at: `{work/'frames'}`"); print()
|
| 232 |
+
for f in frames: print(f"- `{f['path']}` (t={fmt_time(f['timestamp_seconds'])})")
|
| 233 |
+
print(); print("## Transcript"); print()
|
| 234 |
+
print("```")
|
| 235 |
+
for seg in segs: print(f"[{fmt_time(seg.get('start',0))}] {seg['text']}")
|
| 236 |
+
print("```")
|
| 237 |
+
print(); print("## Best Testimonial Quote"); print()
|
| 238 |
+
print(f"**Type:** {quote['type']}")
|
| 239 |
+
print(f"**Timestamp:** {fmt_time(quote['start'])} -> {fmt_time(quote['end'])} ({quote['duration']}s)")
|
| 240 |
+
print(f'**Quote:** "{quote["quote"]}"')
|
| 241 |
+
print(); print("---"); print(f"_Work dir: `{work}`_")
|
| 242 |
+
|
| 243 |
+
|
| 244 |
+
def parse_time_str(s: str | None) -> float | None:
|
| 245 |
+
if not s: return None
|
| 246 |
+
p = s.split(":")
|
| 247 |
+
if len(p)==1: return float(p[0])
|
| 248 |
+
if len(p)==2: return int(p[0])*60+float(p[1])
|
| 249 |
+
return int(p[0])*3600+int(p[1])*60+float(p[2])
|
| 250 |
+
|
| 251 |
+
|
| 252 |
+
def main():
|
| 253 |
+
ap = argparse.ArgumentParser(description="Watch Google Drive video for Claude")
|
| 254 |
+
src = ap.add_mutually_exclusive_group(required=True)
|
| 255 |
+
src.add_argument("--file-id", help="Google Drive file ID")
|
| 256 |
+
src.add_argument("--local", help="Local video path")
|
| 257 |
+
ap.add_argument("--start", help="Focus start (SS or MM:SS)")
|
| 258 |
+
ap.add_argument("--end", help="Focus end (SS or MM:SS)")
|
| 259 |
+
ap.add_argument("--resolution", type=int, default=512)
|
| 260 |
+
ap.add_argument("--max-frames", type=int, default=80)
|
| 261 |
+
ap.add_argument("--out-dir")
|
| 262 |
+
args = ap.parse_args()
|
| 263 |
+
|
| 264 |
+
work = Path(args.out_dir).expanduser() if args.out_dir else Path(tempfile.mkdtemp(prefix="watch-gdrive-"))
|
| 265 |
+
work.mkdir(parents=True, exist_ok=True)
|
| 266 |
+
print(f"[watch] working dir: {work}", file=sys.stderr)
|
| 267 |
+
|
| 268 |
+
video = (Path(args.local).expanduser().resolve() if args.local
|
| 269 |
+
else download_gdrive(args.file_id, work/"download"))
|
| 270 |
+
if not video.exists(): raise SystemExit(f"File not found: {video}")
|
| 271 |
+
source = str(video) if args.local else f"drive://{args.file_id}"
|
| 272 |
+
|
| 273 |
+
start_sec, end_sec = parse_time_str(args.start), parse_time_str(args.end)
|
| 274 |
+
duration = get_duration(video)
|
| 275 |
+
eff_dur = (end_sec or duration) - (start_sec or 0)
|
| 276 |
+
fps, _ = auto_fps(eff_dur, args.max_frames)
|
| 277 |
+
|
| 278 |
+
print(f"[watch] {int(fps*eff_dur)} frames at {fps:.3f}fps over {eff_dur:.1f}s...", file=sys.stderr)
|
| 279 |
+
frames = extract_frames(video, work/"frames", fps, args.resolution, start_sec, end_sec)
|
| 280 |
+
print(f"[watch] {len(frames)} frames extracted", file=sys.stderr)
|
| 281 |
+
|
| 282 |
+
segs_raw = transcribe(video, work)
|
| 283 |
+
if start_sec or end_sec:
|
| 284 |
+
lo, hi = start_sec or 0, end_sec or float("inf")
|
| 285 |
+
segs_raw = [s for s in segs_raw if s.get("end",0)>=lo and s.get("start",0)<=hi]
|
| 286 |
+
segs = clean(segs_raw)
|
| 287 |
+
quote = (best_quote(segs) if segs else
|
| 288 |
+
{"quote":"N/A","start":0,"end":0,"duration":0,"type":"none"})
|
| 289 |
+
|
| 290 |
+
report(source, video, duration, frames, segs, quote, work)
|
| 291 |
+
return 0
|
| 292 |
+
|
| 293 |
+
|
| 294 |
+
if __name__ == "__main__":
|
| 295 |
+
raise SystemExit(main())
|