programmersd commited on
Commit
a9334c7
·
1 Parent(s): aa1c07c
Files changed (6) hide show
  1. README.md +33 -78
  2. app.py +83 -68
  3. packages.txt +7 -11
  4. pipeline.py +50 -78
  5. queue_manager.py +19 -54
  6. utils.py +9 -108
README.md CHANGED
@@ -12,48 +12,27 @@ pinned: false
12
 
13
  # 🎬 PDF → Manim Animation Pipeline
14
 
15
- A Hugging Face Spaces Gradio app that converts any PDF into an animated Manim video and delivers it to your inbox.
16
 
17
  ---
18
 
19
- ## 🚀 Quick Start
20
 
21
- ### 1. Upload to Hugging Face Spaces
22
-
23
- Create a new Space on [huggingface.co/spaces](https://huggingface.co/spaces) with:
24
- - **SDK:** Gradio
25
- - **Hardware:** CPU Basic (or better — more RAM = faster renders)
26
-
27
- Upload all files in this folder.
28
-
29
- ---
30
-
31
- ### 2. Set Environment Secrets
32
-
33
- In your Space settings → **Secrets**, add:
34
-
35
- | Secret Name | Value |
36
- |-----------------|--------------------------------------------|
37
- | `SMTP_EMAIL` | Your Gmail address (e.g. `you@gmail.com`) |
38
- | `SMTP_PASSWORD` | Gmail App Password (16-char, not login pw) |
39
-
40
- **How to get a Gmail App Password:**
41
- 1. Enable 2-Step Verification on your Google account.
42
- 2. Go to [myaccount.google.com/apppasswords](https://myaccount.google.com/apppasswords).
43
- 3. Create a new app password for "Mail".
44
-
45
- > ⚠️ **Never commit secrets to your repo.** Only use the Spaces secret store.
46
 
47
  ---
48
 
49
- ### 3. Usage
50
 
51
- 1. Open your Space URL.
52
- 2. Upload a PDF file.
53
- 3. Enter your email address *(saved in browser — no need to re-enter)*.
54
- 4. Enter your **Gemini API key** *(saved in browser)* get one free at [aistudio.google.com](https://aistudio.google.com).
55
- 5. Click **Generate Video**.
56
- 6. Watch live status updates — the generated Manim code appears in-app, and the video arrives in your inbox!
 
57
 
58
  ---
59
 
@@ -66,61 +45,39 @@ PDF Upload
66
  Extract Text (pypdf)
67
 
68
 
69
- Gemini 2.5 Flash (google-genai)
70
- │ → generated code shown live in UI
71
-
72
- Manim Render (-qm, 1280×720, 30fps)
73
 
74
 
75
- Video < 24MB? ──YES──▶ Email Attachment
76
- │ │
77
- NO │
78
- ▼ │
79
- Catbox.moe Upload │
80
- │ │
81
- ▼ │
82
- Email Link ◀─────────────────┘
83
 
84
 
85
- Cleanup temp files
86
  ```
87
 
88
  ---
89
 
90
- ## ⚙️ Configuration
91
-
92
- | Setting | Default | Location |
93
- |--------------------|--------------|---------------------------------|
94
- | Max concurrent jobs| 100 | `app.py` → `JobQueue(max_jobs=)`|
95
- | Thread workers | 8 | `app.py` → `JobQueue(max_workers=)`|
96
- | Manim quality | `-qm` 720p30 | `pipeline.py` → `_render_manim` |
97
- | Render timeout | 600s | `pipeline.py` → `_render_manim` |
98
- | Catbox retries | 5 | `utils.py` |
99
-
100
- ---
101
-
102
  ## 📁 File Structure
103
 
104
  ```
105
  .
106
- ├── app.py # Gradio UI + streaming status + BrowserState persistence
107
- ├── pipeline.py # Job pipeline orchestrator
108
- ├── queue_manager.py # Thread-pool queue with state tracking
109
- ├── utils.py # PDF, Gemini, email, Catbox helpers
110
- ├── requirements.txt # Python dependencies
111
- ├── packages.txt # System packages (LaTeX, ffmpeg, cairo…)
112
- └── README.md # This file
113
  ```
114
 
115
  ---
116
 
117
- ## 🔒 Security Notes
 
 
118
 
119
- - Gemini API keys are passed per-request and **never stored server-side**.
120
- - Email/API key persisted in **browser localStorage** (client only).
121
- - SMTP credentials live only in HF Secrets (env vars).
122
- - All temporary files deleted after pipeline completion.
123
- - Each job uses a UUID to prevent path collisions.
124
 
125
  ---
126
 
@@ -128,9 +85,7 @@ Cleanup temp files
128
 
129
  | Issue | Fix |
130
  |-------|-----|
131
- | `SMTP_EMAIL not set` | Add the secret in HF Space settings |
132
- | `latex not found` | Ensure `packages.txt` is in your Space root |
133
- | Render timeout | Use a higher-tier Space hardware |
134
- | Gemini API error | Check your API key and quota |
135
- | Empty PDF | PDF must have selectable text (not scanned image) |
136
- | Large video | Catbox fallback used automatically |
 
12
 
13
  # 🎬 PDF → Manim Animation Pipeline
14
 
15
+ Upload a PDF Gemini generates a Manim animation watch it in-app and download the artifacts.
16
 
17
  ---
18
 
19
+ ## 🚀 Deploy on Hugging Face Spaces
20
 
21
+ 1. Create a new Space (SDK: Gradio, any CPU hardware).
22
+ 2. Upload all files in this folder.
23
+ 3. No secrets required users supply their own Gemini API key in the UI.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
24
 
25
  ---
26
 
27
+ ## 🖥️ Usage
28
 
29
+ 1. Upload a PDF.
30
+ 2. Enter your **Gemini API key** (saved in your browser — enter once).
31
+ 3. Click **Generate Video**.
32
+ 4. Watch live status and the generated Manim code appear in real time.
33
+ 5. When complete:
34
+ - ▶️ **Video player**watch the animation directly in the browser.
35
+ - ⬇️ **Artifacts ZIP** — download `<job_id>.py` + `OutputVideo.mp4`.
36
 
37
  ---
38
 
 
45
  Extract Text (pypdf)
46
 
47
 
48
+ Gemini 3 Flash Preview → code shown live in UI
 
 
 
49
 
50
 
51
+ Manim Render (-qm, 1280×720, 30fps)
 
 
 
 
 
 
 
52
 
53
 
54
+ Video Player + artifacts_<job_id>.zip download
55
  ```
56
 
57
  ---
58
 
 
 
 
 
 
 
 
 
 
 
 
 
59
  ## 📁 File Structure
60
 
61
  ```
62
  .
63
+ ├── app.py # Gradio UI streaming status, video player, zip download
64
+ ├── pipeline.py # Job orchestrator
65
+ ├── queue_manager.py # Thread-pool queue (max 100 jobs)
66
+ ├── utils.py # PDF extraction, Gemini streaming, code sanitisation
67
+ ├── requirements.txt # Python deps
68
+ ├── packages.txt # System deps (LaTeX, ffmpeg, cairo…)
69
+ └── README.md
70
  ```
71
 
72
  ---
73
 
74
+ ## ⚙️ System Packages (`packages.txt`)
75
+
76
+ Manim requires LaTeX and ffmpeg. These are installed automatically by HF Spaces:
77
 
78
+ - `ffmpeg`, `libcairo2-dev`, `libpango1.0-dev`
79
+ - `texlive-latex-base`, `texlive-latex-extra`, `texlive-fonts-recommended`
80
+ - `dvipng`, `dvisvgm`, `ghostscript`
 
 
81
 
82
  ---
83
 
 
85
 
86
  | Issue | Fix |
87
  |-------|-----|
88
+ | `latex not found` | Ensure `packages.txt` is at the repo root |
89
+ | Render timeout | Use higher-tier hardware or shorten animation in prompt |
90
+ | Gemini API error | Check API key and quota at aistudio.google.com |
91
+ | Empty PDF | PDF must have selectable text (not a scanned image) |
 
 
app.py CHANGED
@@ -8,14 +8,13 @@ import atexit
8
  import queue
9
  import threading
10
  import uuid
11
- from pathlib import Path
12
 
13
  import gradio as gr
14
 
15
  from queue_manager import JobQueue, State
16
  from pipeline import run_pipeline
17
 
18
- # ── Asyncio cleanup fix (suppresses Invalid file descriptor errors) ───────────
19
  def _cleanup_event_loop():
20
  try:
21
  loop = asyncio.get_event_loop()
@@ -26,60 +25,79 @@ def _cleanup_event_loop():
26
 
27
  atexit.register(_cleanup_event_loop)
28
 
29
- # ── Global queue ──────────────────────────────────────────────────────────────
30
  job_queue = JobQueue(max_workers=8, max_jobs=100)
31
 
32
 
33
- # ── Pipeline streaming generator ──────────────────────────────────────────────
34
- def submit_and_stream(pdf_file, email: str, api_key: str):
35
  """
36
- Generator — yields (status_md, code_text, code_visible) tuples live.
37
- Drives the entire UI update without any polling button.
 
 
38
  """
39
 
 
 
 
 
 
 
 
 
 
 
 
40
  # ── Validate ──────────────────────────────────────────────────────────────
41
  if pdf_file is None:
42
- yield "❌ Please upload a PDF file.", "", gr.update(visible=False)
43
- return
44
- if not email or "@" not in email:
45
- yield "❌ Please enter a valid email address.", "", gr.update(visible=False)
46
  return
47
  if not api_key or len(api_key) < 10:
48
- yield "❌ Please enter a valid Gemini API key.", "", gr.update(visible=False)
49
  return
50
  if job_queue.is_full():
51
- yield "⚠️ Queue is full (max 100 jobs). Please try again shortly.", "", gr.update(visible=False)
52
  return
53
 
54
- job_id = uuid.uuid4().hex
55
  pdf_path = pdf_file.name
 
56
 
57
- # Thread-safe channel for status + code updates
58
  update_q: queue.Queue = queue.Queue()
59
 
60
  def status_cb(state: State, message: str = "", code: str | None = None):
61
  update_q.put((state, message, code))
62
 
 
 
63
  def _run():
64
  try:
65
- run_pipeline(
66
  job_id=job_id,
67
  pdf_path=pdf_path,
68
- email=email,
69
  gemini_api_key=api_key,
70
  status_cb=status_cb,
71
  )
72
- update_q.put((State.DONE, "Video sent to your inbox! 🎉", None))
 
73
  except Exception as exc:
74
- update_q.put((State.FAILED, str(exc), None))
75
  finally:
76
  update_q.put(None) # sentinel
77
 
78
- thread = threading.Thread(target=_run, daemon=True)
79
- thread.start()
 
 
 
 
 
 
80
 
81
  code_so_far = ""
82
- yield f"⏳ **Queued** — Starting…\n\n*Job ID: `{job_id}`*", "", gr.update(visible=False)
83
 
84
  while True:
85
  item = update_q.get()
@@ -87,58 +105,50 @@ def submit_and_stream(pdf_file, email: str, api_key: str):
87
  break
88
 
89
  state, message, code = item
90
-
91
- icons = {
92
- State.QUEUED: "⏳",
93
- State.RUNNING: "⚙️",
94
- State.UPLOADING: "☁️",
95
- State.SENDING: "📧",
96
- State.DONE: "✅",
97
- State.FAILED: "❌",
98
- }
99
- icon = icons.get(state, "❓")
100
- status_text = f"{icon} **{state.value.title()}** — {message}\n\n*Job ID: `{job_id}`*"
101
-
102
- if code is not None:
103
  code_so_far = code
104
 
105
- yield (
 
 
 
 
 
 
 
106
  status_text,
107
- code_so_far,
108
- gr.update(visible=bool(code_so_far)),
 
 
 
 
109
  )
110
 
111
 
112
  # ── UI ────────────────────────────────────────────────────────────────────────
113
  with gr.Blocks(title="PDF → Manim Video") as demo:
114
 
115
- gr.Markdown(
116
- """
117
- # 🎬 PDF → Manim Animation Pipeline
118
- Upload a PDF, enter your details, and receive an animated video in your inbox.
119
- """
120
- )
121
 
122
- # Persistent browser-side storage (survives page refresh)
123
- saved_email = gr.BrowserState("")
124
- saved_api_key = gr.BrowserState("")
125
 
126
  with gr.Row():
 
127
  with gr.Column(scale=1):
128
  pdf_input = gr.File(label="📄 Upload PDF", file_types=[".pdf"])
129
- email_input = gr.Textbox(
130
- label="📧 Your Email",
131
- placeholder="you@example.com",
132
- )
133
  api_key_input = gr.Textbox(
134
  label="🔑 Gemini API Key",
135
  placeholder="AIza…",
136
  type="password",
 
137
  )
138
  submit_btn = gr.Button("🚀 Generate Video", variant="primary")
139
 
 
140
  with gr.Column(scale=1):
141
  status_md = gr.Markdown("*Submit a job to see live status here.*")
 
142
  code_box = gr.Code(
143
  label="📝 Generated Manim Code",
144
  language="python",
@@ -146,33 +156,38 @@ with gr.Blocks(title="PDF → Manim Video") as demo:
146
  interactive=False,
147
  )
148
 
 
 
 
 
 
 
 
 
 
 
 
 
149
  gr.Markdown(
150
  """
151
  ---
152
- **Notes:**
153
- - Processing typically takes 2–5 minutes.
154
- - The video will be emailed once rendered; large files use a Catbox.moe link.
155
- - Your Gemini API key is used only for this request and never stored server-side.
156
- - Requires `SMTP_EMAIL` and `SMTP_PASSWORD` environment secrets on the Space.
157
- """
158
- )
159
 
160
- # ── Restore persisted values on page load ─────────────────────────────────
161
- demo.load(
162
- fn=lambda e, k: (e, k),
163
- inputs=[saved_email, saved_api_key],
164
- outputs=[email_input, api_key_input],
165
  )
166
 
167
- # ── Save values to browser storage whenever they change ───────────────────
168
- email_input.change(fn=lambda v: v, inputs=[email_input], outputs=[saved_email])
169
  api_key_input.change(fn=lambda v: v, inputs=[api_key_input], outputs=[saved_api_key])
170
 
171
- # ── Streaming job submission ───────────────────────────────────────────────
172
  submit_btn.click(
173
  fn=submit_and_stream,
174
- inputs=[pdf_input, email_input, api_key_input],
175
- outputs=[status_md, code_box, code_box],
176
  )
177
 
178
 
 
8
  import queue
9
  import threading
10
  import uuid
 
11
 
12
  import gradio as gr
13
 
14
  from queue_manager import JobQueue, State
15
  from pipeline import run_pipeline
16
 
17
+ # ── Asyncio cleanup (suppresses "Invalid file descriptor" noise on shutdown) ──
18
  def _cleanup_event_loop():
19
  try:
20
  loop = asyncio.get_event_loop()
 
25
 
26
  atexit.register(_cleanup_event_loop)
27
 
28
+ # ── Global job queue ──────────────────────────────────────────────────────────
29
  job_queue = JobQueue(max_workers=8, max_jobs=100)
30
 
31
 
32
+ # ── Streaming pipeline ────────────────────────────────────────────────────────
33
+ def submit_and_stream(pdf_file, api_key: str):
34
  """
35
+ Generator — yields tuples:
36
+ (status_md, code_str, code_visible, video_path, video_visible,
37
+ zip_path, zip_visible)
38
+ live-streamed to the Gradio UI.
39
  """
40
 
41
+ def _emit(status, code="", code_vis=False, video=None, vid_vis=False, zip_p=None, zip_vis=False):
42
+ return (
43
+ status,
44
+ code,
45
+ gr.update(visible=code_vis),
46
+ video,
47
+ gr.update(visible=vid_vis),
48
+ zip_p,
49
+ gr.update(visible=zip_vis),
50
+ )
51
+
52
  # ── Validate ──────────────────────────────────────────────────────────────
53
  if pdf_file is None:
54
+ yield _emit("❌ Please upload a PDF file.")
 
 
 
55
  return
56
  if not api_key or len(api_key) < 10:
57
+ yield _emit("❌ Please enter a valid Gemini API key.")
58
  return
59
  if job_queue.is_full():
60
+ yield _emit("⚠️ Queue is full (max 100 jobs). Please try again shortly.")
61
  return
62
 
63
+ job_id = uuid.uuid4().hex
64
  pdf_path = pdf_file.name
65
+ job_queue.register(job_id)
66
 
67
+ # Thread-safe update channel
68
  update_q: queue.Queue = queue.Queue()
69
 
70
  def status_cb(state: State, message: str = "", code: str | None = None):
71
  update_q.put((state, message, code))
72
 
73
+ result_holder: dict = {}
74
+
75
  def _run():
76
  try:
77
+ result = run_pipeline(
78
  job_id=job_id,
79
  pdf_path=pdf_path,
 
80
  gemini_api_key=api_key,
81
  status_cb=status_cb,
82
  )
83
+ result_holder.update(result)
84
+ update_q.put((State.DONE, "✅ Render complete!", result.get("code")))
85
  except Exception as exc:
86
+ update_q.put((State.FAILED, f"❌ {exc}", None))
87
  finally:
88
  update_q.put(None) # sentinel
89
 
90
+ threading.Thread(target=_run, daemon=True).start()
91
+
92
+ icons = {
93
+ State.QUEUED: "⏳",
94
+ State.RUNNING: "⚙️",
95
+ State.DONE: "✅",
96
+ State.FAILED: "❌",
97
+ }
98
 
99
  code_so_far = ""
100
+ yield _emit(f"⏳ **Queued** — Starting…\n\n*Job `{job_id}`*")
101
 
102
  while True:
103
  item = update_q.get()
 
105
  break
106
 
107
  state, message, code = item
108
+ if code:
 
 
 
 
 
 
 
 
 
 
 
 
109
  code_so_far = code
110
 
111
+ status_text = (
112
+ f"{icons.get(state,'❓')} **{state.value.title()}** — {message}"
113
+ f"\n\n*Job `{job_id}`*"
114
+ )
115
+ is_done = state == State.DONE
116
+ is_failed = state == State.FAILED
117
+
118
+ yield _emit(
119
  status_text,
120
+ code = code_so_far,
121
+ code_vis = bool(code_so_far),
122
+ video = result_holder.get("video_path") if is_done else None,
123
+ vid_vis = is_done,
124
+ zip_p = result_holder.get("zip_path") if is_done else None,
125
+ zip_vis = is_done,
126
  )
127
 
128
 
129
  # ── UI ────────────────────────────────────────────────────────────────────────
130
  with gr.Blocks(title="PDF → Manim Video") as demo:
131
 
132
+ gr.Markdown("# 🎬 PDF → Manim Animation Pipeline\nUpload a PDF and get a downloadable Manim animation.")
 
 
 
 
 
133
 
134
+ saved_api_key = gr.BrowserState("") # persisted in browser localStorage
 
 
135
 
136
  with gr.Row():
137
+ # ── Left column: inputs ───────────────────────────────────────────────
138
  with gr.Column(scale=1):
139
  pdf_input = gr.File(label="📄 Upload PDF", file_types=[".pdf"])
 
 
 
 
140
  api_key_input = gr.Textbox(
141
  label="🔑 Gemini API Key",
142
  placeholder="AIza…",
143
  type="password",
144
+ info="Saved in your browser — you only need to enter this once.",
145
  )
146
  submit_btn = gr.Button("🚀 Generate Video", variant="primary")
147
 
148
+ # ── Right column: outputs ─────────────────────────────────────────────
149
  with gr.Column(scale=1):
150
  status_md = gr.Markdown("*Submit a job to see live status here.*")
151
+
152
  code_box = gr.Code(
153
  label="📝 Generated Manim Code",
154
  language="python",
 
156
  interactive=False,
157
  )
158
 
159
+ video_player = gr.Video(
160
+ label="🎬 Rendered Animation",
161
+ visible=False,
162
+ interactive=False,
163
+ )
164
+
165
+ zip_download = gr.File(
166
+ label="⬇️ Download Artifacts (.py + .mp4)",
167
+ visible=False,
168
+ interactive=False,
169
+ )
170
+
171
  gr.Markdown(
172
  """
173
  ---
174
+ **Notes:** Processing typically takes 2–5 minutes depending on animation complexity.
175
+ The artifacts ZIP contains the generated `.py` source and the rendered `.mp4`.
176
+ Your API key is never stored server-side.
 
 
 
 
177
 
178
+ Have fun! 🎬 If you liked it, feel free to share it with your friends and family.
179
+ """
 
 
 
180
  )
181
 
182
+ # ── Restore API key from browser on load ──────────────────────────────────
183
+ demo.load(fn=lambda k: k, inputs=[saved_api_key], outputs=[api_key_input])
184
  api_key_input.change(fn=lambda v: v, inputs=[api_key_input], outputs=[saved_api_key])
185
 
186
+ # ── Streaming submit ───────────────────────────────────────────────────────
187
  submit_btn.click(
188
  fn=submit_and_stream,
189
+ inputs=[pdf_input, api_key_input],
190
+ outputs=[status_md, code_box, code_box, video_player, video_player, zip_download, zip_download],
191
  )
192
 
193
 
packages.txt CHANGED
@@ -1,15 +1,11 @@
1
- build-essential
2
- python3-dev
3
- libcairo2-dev
4
- libpango1.0-dev
5
  ffmpeg
6
- sox
7
- ghostscript
8
- dvipng
9
- dvisvgm
10
- texlive
11
  texlive-latex-extra
 
12
  texlive-fonts-extra
13
- texlive-latex-recommended
14
  texlive-science
15
- tipa
 
 
 
 
 
 
 
 
 
1
  ffmpeg
2
+ texlive-latex-base
 
 
 
 
3
  texlive-latex-extra
4
+ texlive-fonts-recommended
5
  texlive-fonts-extra
 
6
  texlive-science
7
+ dvipng
8
+ dvisvgm
9
+ ghostscript
10
+ libcairo2-dev
11
+ libpango1.0-dev
pipeline.py CHANGED
@@ -1,74 +1,79 @@
1
  """
2
- pipeline.py — Main pipeline: PDF → Gemini → Manim Upload/Email Cleanup
3
  """
4
 
5
  from __future__ import annotations
6
 
7
- import os
8
- import shutil
9
  import subprocess
10
  import textwrap
11
  import time
 
12
  from pathlib import Path
13
  from typing import Callable
14
 
15
  from queue_manager import State
16
- from utils import (
17
- extract_pdf_text,
18
- generate_manim_code,
19
- send_video_email,
20
- upload_to_catbox,
21
- sanitize_manim_code,
22
- )
23
 
24
- MEDIA_ROOT = Path("media/videos")
25
- JOBS_ROOT = Path("jobs")
 
26
 
27
 
28
  def run_pipeline(
29
  job_id: str,
30
  pdf_path: str,
31
- email: str,
32
  gemini_api_key: str,
33
  status_cb: Callable[[State, str, str | None], None],
34
- ) -> None:
35
- job_dir = JOBS_ROOT / job_id
 
 
 
 
 
 
 
 
36
  script_path = job_dir / f"{job_id}.py"
37
  video_path = MEDIA_ROOT / job_id / "720p30" / "OutputVideo.mp4"
 
38
 
39
  job_dir.mkdir(parents=True, exist_ok=True)
 
40
 
41
- try:
42
- # 1. Extract PDF
43
- status_cb(State.RUNNING, "📖 Extracting text from PDF…", None)
44
- pdf_text = extract_pdf_text(pdf_path)
45
- if not pdf_text.strip():
46
- raise ValueError("PDF appears empty or has no selectable text.")
47
 
48
- # 2. Generate Manim code via Gemini
49
- status_cb(State.RUNNING, "🤖 Generating Manim code with Gemini…", None)
50
- prompt = _build_prompt(pdf_text)
51
- raw_code = generate_manim_code(prompt, gemini_api_key)
52
- manim_code = sanitize_manim_code(raw_code, job_id)
53
 
54
- # Surface generated code to UI
55
- status_cb(State.RUNNING, "✏️ Manim code generated — starting render…", manim_code)
56
 
57
- script_path.write_text(manim_code, encoding="utf-8")
 
 
58
 
59
- # 3. Render
60
- status_cb(State.RUNNING, "🎬 Rendering animation (this may take a few minutes)…", None)
61
- _render_manim(script_path, job_id)
62
 
63
- if not video_path.exists():
64
- raise FileNotFoundError(f"Rendered video not found at {video_path}")
 
 
 
65
 
66
- # 4. Deliver
67
- status_cb(State.SENDING, "📧 Sending video to your inbox…", None)
68
- _deliver_video(str(video_path), email, status_cb)
69
-
70
- finally:
71
- _cleanup(job_dir, video_path)
72
 
73
 
74
  # ── Helpers ───────────────────────────────────────────────────────────────────
@@ -85,7 +90,7 @@ def _build_prompt(pdf_text: str) -> str:
85
  - Use only standard Manim Community v0.18+ API.
86
  - Output ONLY valid Python code. No explanations, no markdown fences.
87
  - Keep runtime under 90 seconds.
88
- - Avoid custom LaTeX; prefer Text() over MathTex() where possible.
89
  - Animations should be clear, readable, and professional.
90
 
91
  Document content:
@@ -95,7 +100,7 @@ def _build_prompt(pdf_text: str) -> str:
95
  """).strip()
96
 
97
 
98
- def _render_manim(script_path: Path, job_id: str, max_retries: int = 2) -> None:
99
  cmd = [
100
  "manim",
101
  str(script_path),
@@ -104,48 +109,15 @@ def _render_manim(script_path: Path, job_id: str, max_retries: int = 2) -> None:
104
  "--media_dir", "media",
105
  "--disable_caching",
106
  ]
 
107
  for attempt in range(max_retries + 1):
108
  result = subprocess.run(cmd, capture_output=True, text=True, timeout=600)
109
  if result.returncode == 0:
110
  return
 
111
  if attempt < max_retries:
112
  time.sleep(5 * (attempt + 1))
113
 
114
  raise RuntimeError(
115
- f"Manim render failed after {max_retries + 1} attempts.\n"
116
- f"STDERR: {result.stderr[-3000:]}"
117
  )
118
-
119
-
120
- def _deliver_video(
121
- video_path: str,
122
- email: str,
123
- status_cb: Callable,
124
- ) -> None:
125
- file_size_mb = Path(video_path).stat().st_size / (1024 * 1024)
126
-
127
- if file_size_mb <= 24:
128
- try:
129
- send_video_email(email, video_path)
130
- return
131
- except Exception as exc:
132
- status_cb(State.UPLOADING, f"⚠️ Attachment failed ({exc}) — uploading to Catbox…", None)
133
- else:
134
- status_cb(State.UPLOADING, f"📦 {file_size_mb:.1f} MB video — uploading to Catbox…", None)
135
-
136
- url = upload_to_catbox(video_path)
137
- send_video_email(email, video_path=None, catbox_url=url)
138
-
139
-
140
- def _cleanup(*paths) -> None:
141
- for p in paths:
142
- if p is None:
143
- continue
144
- path = Path(p)
145
- try:
146
- if path.is_dir():
147
- shutil.rmtree(path, ignore_errors=True)
148
- elif path.is_file():
149
- path.unlink(missing_ok=True)
150
- except Exception:
151
- pass
 
1
  """
2
+ pipeline.py — PDF → Gemini → Manim renderreturn artifacts
3
  """
4
 
5
  from __future__ import annotations
6
 
 
 
7
  import subprocess
8
  import textwrap
9
  import time
10
+ import zipfile
11
  from pathlib import Path
12
  from typing import Callable
13
 
14
  from queue_manager import State
15
+ from utils import extract_pdf_text, generate_manim_code, sanitize_manim_code
 
 
 
 
 
 
16
 
17
+ MEDIA_ROOT = Path("media/videos")
18
+ JOBS_ROOT = Path("jobs")
19
+ ARTIFACTS = Path("artifacts")
20
 
21
 
22
  def run_pipeline(
23
  job_id: str,
24
  pdf_path: str,
 
25
  gemini_api_key: str,
26
  status_cb: Callable[[State, str, str | None], None],
27
+ ) -> dict:
28
+ """
29
+ Run the full pipeline and return:
30
+ {
31
+ "video_path": str, # absolute path to the rendered .mp4
32
+ "zip_path": str, # absolute path to artifacts_<job_id>.zip
33
+ "code": str, # generated Manim source
34
+ }
35
+ """
36
+ job_dir = JOBS_ROOT / job_id
37
  script_path = job_dir / f"{job_id}.py"
38
  video_path = MEDIA_ROOT / job_id / "720p30" / "OutputVideo.mp4"
39
+ zip_path = ARTIFACTS / f"artifacts_{job_id}.zip"
40
 
41
  job_dir.mkdir(parents=True, exist_ok=True)
42
+ ARTIFACTS.mkdir(parents=True, exist_ok=True)
43
 
44
+ # 1. Extract PDF text
45
+ status_cb(State.RUNNING, "📖 Extracting text from PDF…", None)
46
+ pdf_text = extract_pdf_text(pdf_path)
47
+ if not pdf_text.strip():
48
+ raise ValueError("PDF appears empty or has no selectable text.")
 
49
 
50
+ # 2. Generate Manim code via Gemini
51
+ status_cb(State.RUNNING, "🤖 Generating Manim code with Gemini…", None)
52
+ prompt = _build_prompt(pdf_text)
53
+ raw_code = generate_manim_code(prompt, gemini_api_key)
54
+ manim_code = sanitize_manim_code(raw_code)
55
 
56
+ status_cb(State.RUNNING, "✏️ Code generated starting render…", manim_code)
57
+ script_path.write_text(manim_code, encoding="utf-8")
58
 
59
+ # 3. Render
60
+ status_cb(State.RUNNING, "🎬 Rendering animation (this may take a few minutes)…", None)
61
+ _render_manim(script_path)
62
 
63
+ if not video_path.exists():
64
+ raise FileNotFoundError(f"Rendered video not found at {video_path}")
 
65
 
66
+ # 4. Package artifacts zip
67
+ status_cb(State.RUNNING, "📦 Packaging artifacts zip…", None)
68
+ with zipfile.ZipFile(zip_path, "w", zipfile.ZIP_DEFLATED) as zf:
69
+ zf.write(script_path, arcname=f"{job_id}.py")
70
+ zf.write(video_path, arcname="OutputVideo.mp4")
71
 
72
+ return {
73
+ "video_path": str(video_path.resolve()),
74
+ "zip_path": str(zip_path.resolve()),
75
+ "code": manim_code,
76
+ }
 
77
 
78
 
79
  # ── Helpers ───────────────────────────────────────────────────────────────────
 
90
  - Use only standard Manim Community v0.18+ API.
91
  - Output ONLY valid Python code. No explanations, no markdown fences.
92
  - Keep runtime under 90 seconds.
93
+ - Avoid custom LaTeX preambles; prefer Text() over MathTex() where possible.
94
  - Animations should be clear, readable, and professional.
95
 
96
  Document content:
 
100
  """).strip()
101
 
102
 
103
+ def _render_manim(script_path: Path, max_retries: int = 2) -> None:
104
  cmd = [
105
  "manim",
106
  str(script_path),
 
109
  "--media_dir", "media",
110
  "--disable_caching",
111
  ]
112
+ stderr_tail = ""
113
  for attempt in range(max_retries + 1):
114
  result = subprocess.run(cmd, capture_output=True, text=True, timeout=600)
115
  if result.returncode == 0:
116
  return
117
+ stderr_tail = result.stderr[-3000:]
118
  if attempt < max_retries:
119
  time.sleep(5 * (attempt + 1))
120
 
121
  raise RuntimeError(
122
+ f"Manim render failed after {max_retries + 1} attempts.\nSTDERR: {stderr_tail}"
 
123
  )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
queue_manager.py CHANGED
@@ -1,5 +1,5 @@
1
  """
2
- queue_manager.py — Thread-pool-based job queue with status tracking.
3
  """
4
 
5
  from __future__ import annotations
@@ -13,12 +13,10 @@ from typing import Any, Callable
13
 
14
 
15
  class State(str, Enum):
16
- QUEUED = "queued"
17
- RUNNING = "running"
18
- UPLOADING = "uploading"
19
- SENDING = "sending"
20
- DONE = "done"
21
- FAILED = "failed"
22
 
23
 
24
  @dataclass
@@ -34,66 +32,33 @@ class JobStatus:
34
  self.message = message
35
  self.updated_at = time.time()
36
 
37
- def display(self) -> str:
38
- icons = {
39
- State.QUEUED: "⏳",
40
- State.RUNNING: "⚙️",
41
- State.UPLOADING: "☁️",
42
- State.SENDING: "📧",
43
- State.DONE: "✅",
44
- State.FAILED: "❌",
45
- }
46
- icon = icons.get(self.state, "❓")
47
- lines = [f"{icon} **{self.state.value.title()}**"]
48
- if self.message:
49
- lines.append(self.message)
50
- lines.append(f"*Job ID: `{self.job_id}`*")
51
- return "\n\n".join(lines)
52
-
53
 
54
  class JobQueue:
55
  def __init__(self, max_workers: int = 8, max_jobs: int = 100) -> None:
56
- self._max_jobs = max_jobs
57
- self._executor = ThreadPoolExecutor(max_workers=max_workers)
58
  self._jobs: dict[str, JobStatus] = {}
59
  self._lock = threading.Lock()
60
 
61
  def is_full(self) -> bool:
62
  with self._lock:
63
  active = sum(
64
- 1
65
- for s in self._jobs.values()
66
- if s.state in (State.QUEUED, State.RUNNING, State.UPLOADING, State.SENDING)
67
  )
68
  return active >= self._max_jobs
69
 
70
- def enqueue(self, job_id: str, fn: Callable, kwargs: dict[str, Any]) -> JobStatus:
71
- status = JobStatus(job_id=job_id)
72
  with self._lock:
73
- self._jobs[job_id] = status
74
-
75
- def _run():
76
- with self._lock:
77
- self._jobs[job_id].update(State.RUNNING, "Pipeline started…")
78
- try:
79
- fn(status_cb=self._make_cb(job_id), **kwargs)
80
- with self._lock:
81
- self._jobs[job_id].update(State.DONE, "Video sent to your inbox! 🎉")
82
- except Exception as exc:
83
- with self._lock:
84
- self._jobs[job_id].update(State.FAILED, f"Error: {exc}")
85
-
86
- self._executor.submit(_run)
87
- return status
88
-
89
- def _make_cb(self, job_id: str) -> Callable[[State, str], None]:
90
- def cb(state: State, message: str = "") -> None:
91
- with self._lock:
92
- if job_id in self._jobs:
93
- self._jobs[job_id].update(state, message)
94
 
95
- return cb
 
 
 
96
 
97
- def get_status(self, job_id: str) -> JobStatus | None:
 
98
  with self._lock:
99
- return self._jobs.get(job_id)
 
 
1
  """
2
+ queue_manager.py — Thread-pool job queue with status tracking.
3
  """
4
 
5
  from __future__ import annotations
 
13
 
14
 
15
  class State(str, Enum):
16
+ QUEUED = "queued"
17
+ RUNNING = "running"
18
+ DONE = "done"
19
+ FAILED = "failed"
 
 
20
 
21
 
22
  @dataclass
 
32
  self.message = message
33
  self.updated_at = time.time()
34
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
35
 
36
  class JobQueue:
37
  def __init__(self, max_workers: int = 8, max_jobs: int = 100) -> None:
38
+ self._max_jobs = max_jobs
39
+ self._executor = ThreadPoolExecutor(max_workers=max_workers)
40
  self._jobs: dict[str, JobStatus] = {}
41
  self._lock = threading.Lock()
42
 
43
  def is_full(self) -> bool:
44
  with self._lock:
45
  active = sum(
46
+ 1 for s in self._jobs.values()
47
+ if s.state in (State.QUEUED, State.RUNNING)
 
48
  )
49
  return active >= self._max_jobs
50
 
51
+ def get_status(self, job_id: str) -> JobStatus | None:
 
52
  with self._lock:
53
+ return self._jobs.get(job_id)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
54
 
55
+ def _set_state(self, job_id: str, state: State, message: str = "") -> None:
56
+ with self._lock:
57
+ if job_id in self._jobs:
58
+ self._jobs[job_id].update(state, message)
59
 
60
+ def register(self, job_id: str) -> JobStatus:
61
+ status = JobStatus(job_id=job_id)
62
  with self._lock:
63
+ self._jobs[job_id] = status
64
+ return status
utils.py CHANGED
@@ -1,21 +1,15 @@
1
  """
2
- utils.py — PDF extraction, Gemini LLM, email, Catbox upload helpers.
3
  """
4
 
5
  from __future__ import annotations
6
 
7
- import os
8
  import re
9
- import smtplib
10
- import ssl
11
- import time
12
- from email.message import EmailMessage
13
- from pathlib import Path
14
 
15
- import requests
16
  from google import genai
17
  from google.genai import types
18
 
 
19
  # ── PDF Text Extraction ───────────────────────────────────────────────────────
20
 
21
  def extract_pdf_text(pdf_path: str) -> str:
@@ -33,7 +27,7 @@ def extract_pdf_text(pdf_path: str) -> str:
33
 
34
  # ── Gemini LLM ────────────────────────────────────────────────────────────────
35
 
36
- def generate_manim_code(prompt_text: str, api_key: str):
37
  """Stream Manim code from Gemini 3 Flash Preview and return it as a string."""
38
  client = genai.Client(api_key=api_key)
39
  model = "gemini-3-flash-preview"
@@ -59,20 +53,20 @@ def generate_manim_code(prompt_text: str, api_key: str):
59
  ):
60
  if chunk.text:
61
  code += chunk.text
62
- print(chunk.text, end="") # optional live streaming output
63
 
64
  return code
65
 
66
- def sanitize_manim_code(raw: str, job_id: str) -> str:
 
 
 
67
  """
68
- Strip markdown fences, ensure correct imports and class name,
69
- and add job-specific media dir hint.
70
  """
71
- # Remove ```python ... ``` fences
72
  code = re.sub(r"^```(?:python)?\s*", "", raw.strip(), flags=re.MULTILINE)
73
  code = re.sub(r"\s*```$", "", code.strip(), flags=re.MULTILINE)
74
 
75
- # Ensure manim import is present
76
  if "from manim import" not in code and "import manim" not in code:
77
  code = "from manim import *\n\n" + code
78
 
@@ -80,96 +74,3 @@ def sanitize_manim_code(raw: str, job_id: str) -> str:
80
  code = re.sub(r"class\s+\w+\s*\(\s*Scene\s*\)", "class OutputVideo(Scene)", code)
81
 
82
  return code
83
-
84
-
85
- # ── Email ─────────────────────────────────────────────────────────────────────
86
-
87
- def send_video_email(
88
- to_email: str,
89
- video_path: str | None = None,
90
- catbox_url: str | None = None,
91
- ) -> None:
92
- """Send email with video attachment or Catbox link."""
93
- smtp_email = os.environ["SMTP_EMAIL"]
94
- smtp_password = os.environ["SMTP_PASSWORD"]
95
-
96
- msg = EmailMessage()
97
- msg["Subject"] = "🎬 Your Manim Animation is Ready!"
98
- msg["From"] = smtp_email
99
- msg["To"] = to_email
100
-
101
- if catbox_url:
102
- body = (
103
- "Your animated video has been generated and uploaded.\n\n"
104
- f"Download it here (link valid for ~3 days):\n{catbox_url}\n\n"
105
- "Enjoy your animation!"
106
- )
107
- msg.set_content(body)
108
- else:
109
- msg.set_content(
110
- "Your animated video is attached to this email.\n\nEnjoy your animation!"
111
- )
112
- with open(video_path, "rb") as f:
113
- msg.add_attachment(
114
- f.read(),
115
- maintype="video",
116
- subtype="mp4",
117
- filename="animation.mp4",
118
- )
119
-
120
- context = ssl.create_default_context()
121
- _smtp_send_with_retry(msg, smtp_email, smtp_password, context)
122
-
123
- # Delete video after successful send
124
- if video_path and Path(video_path).exists():
125
- Path(video_path).unlink(missing_ok=True)
126
-
127
-
128
- def _smtp_send_with_retry(
129
- msg: EmailMessage,
130
- smtp_email: str,
131
- smtp_password: str,
132
- context: ssl.SSLContext,
133
- max_retries: int = 3,
134
- ) -> None:
135
- backoff = 2
136
- for attempt in range(max_retries):
137
- try:
138
- with smtplib.SMTP_SSL("smtp.gmail.com", 465, context=context) as server:
139
- server.login(smtp_email, smtp_password)
140
- server.send_message(msg)
141
- return
142
- except smtplib.SMTPException as exc:
143
- if attempt == max_retries - 1:
144
- raise
145
- time.sleep(backoff)
146
- backoff *= 2
147
-
148
-
149
- # ── Catbox Upload ─────────────────────────────────────────────────────────────
150
-
151
- API_URL = "https://catbox.moe/user/api.php"
152
-
153
-
154
- def upload_to_catbox(path: str, max_retries: int = 5) -> str:
155
- """Upload a file to Catbox.moe with exponential backoff."""
156
- file_path = Path(path)
157
- backoff = 1
158
- for attempt in range(max_retries):
159
- try:
160
- with file_path.open("rb") as f:
161
- r = requests.post(
162
- API_URL,
163
- data={"reqtype": "fileupload"},
164
- files={"fileToUpload": f},
165
- timeout=120,
166
- )
167
- if r.status_code == 200 and r.text.startswith("https://files.catbox.moe/"):
168
- return r.text.strip()
169
- raise RuntimeError(f"Catbox returned unexpected response: {r.text[:200]}")
170
- except Exception:
171
- if attempt == max_retries - 1:
172
- raise
173
- time.sleep(backoff)
174
- backoff *= 2
175
- raise RuntimeError("Catbox upload failed after all retries.")
 
1
  """
2
+ utils.py — PDF extraction, Gemini LLM, and Manim code helpers.
3
  """
4
 
5
  from __future__ import annotations
6
 
 
7
  import re
 
 
 
 
 
8
 
 
9
  from google import genai
10
  from google.genai import types
11
 
12
+
13
  # ── PDF Text Extraction ───────────────────────────────────────────────────────
14
 
15
  def extract_pdf_text(pdf_path: str) -> str:
 
27
 
28
  # ── Gemini LLM ────────────────────────────────────────────────────────────────
29
 
30
+ def generate_manim_code(prompt_text: str, api_key: str) -> str:
31
  """Stream Manim code from Gemini 3 Flash Preview and return it as a string."""
32
  client = genai.Client(api_key=api_key)
33
  model = "gemini-3-flash-preview"
 
53
  ):
54
  if chunk.text:
55
  code += chunk.text
56
+ print(chunk.text, end="", flush=True)
57
 
58
  return code
59
 
60
+
61
+ # ── Manim Code Sanitisation ───────────────────────────────────────────────────
62
+
63
+ def sanitize_manim_code(raw: str) -> str:
64
  """
65
+ Strip markdown fences, ensure correct imports and class name.
 
66
  """
 
67
  code = re.sub(r"^```(?:python)?\s*", "", raw.strip(), flags=re.MULTILINE)
68
  code = re.sub(r"\s*```$", "", code.strip(), flags=re.MULTILINE)
69
 
 
70
  if "from manim import" not in code and "import manim" not in code:
71
  code = "from manim import *\n\n" + code
72
 
 
74
  code = re.sub(r"class\s+\w+\s*\(\s*Scene\s*\)", "class OutputVideo(Scene)", code)
75
 
76
  return code