Spaces:
Runtime error
Runtime error
akashyadav758 Claude Opus 4.8 (1M context) commited on
Commit Β·
7869843
1
Parent(s): 645c976
Document Postgres persistence, selfapi space name, and UA/Cloudflare fix in README
Browse filesAdd a Persistence section with the Neon DATABASE_URL setup guide, a
self-hosting section for docker-compose, a UA-must-match-OS note, fix the
stale chrome -> selfapi space name/URLs and title, and update the boot
sequence + troubleshooting to match the current start_hf.sh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
README.md
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
---
|
| 2 |
-
title:
|
| 3 |
emoji: π
|
| 4 |
colorFrom: gray
|
| 5 |
colorTo: blue
|
|
@@ -8,14 +8,16 @@ app_port: 3001
|
|
| 8 |
pinned: false
|
| 9 |
---
|
| 10 |
|
| 11 |
-
#
|
| 12 |
|
| 13 |
-
A [Hugging Face Space](https://huggingface.co/spaces/akash1313/
|
| 14 |
persistent, headless **Chrome for Testing** browser inside Docker with three custom
|
| 15 |
unpacked extensions auto-loaded, plus a live web monitor to watch the browser in real time.
|
| 16 |
|
| 17 |
-
The browser stays logged in across restarts
|
| 18 |
-
|
|
|
|
|
|
|
| 19 |
|
| 20 |
---
|
| 21 |
|
|
@@ -44,9 +46,10 @@ run yourself β see each subproject's README for setup):
|
|
| 44 |
| [`free-gemini-api`](free-gemini-api/README.md) | Go reverse-proxy | local server | Gemini text / Imagen 3 images / Gemini video |
|
| 45 |
| [`flow-agent`](flow-agent/README.md) | Python FastAPI + CLI | `127.0.0.1:8100` | Google Flow T2V / V2V / I2V video + T2I / I2I images |
|
| 46 |
|
| 47 |
-
> β οΈ
|
| 48 |
-
>
|
| 49 |
-
>
|
|
|
|
| 50 |
|
| 51 |
---
|
| 52 |
|
|
@@ -59,7 +62,10 @@ HF Space (port 3001)
|
|
| 59 |
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 60 |
β Docker container (Debian 12) β
|
| 61 |
β β
|
| 62 |
-
β
|
|
|
|
|
|
|
|
|
|
| 63 |
β (Go, :3001) β β
|
| 64 |
β βΌ β
|
| 65 |
β Chrome for Testing 145 (--headless=new) β
|
|
@@ -73,18 +79,59 @@ HF Space (port 3001)
|
|
| 73 |
|
| 74 |
`start_hf.sh` is the boot sequence:
|
| 75 |
|
|
|
|
|
|
|
| 76 |
1. **Profile repair & cleanup** β fixes a nested `data/data` layout and removes cache/junk
|
| 77 |
to keep the persistent profile small.
|
| 78 |
2. **Lock removal** β clears `Singleton*` / `LOCK` files left by an unclean shutdown.
|
| 79 |
3. **Preferences injection** β enables `extensions.ui.developer_mode` and sets a clean
|
| 80 |
`exit_type` so the "restore pages?" popup never appears.
|
| 81 |
-
4. **Monitor server** β starts the Go monitor on `:3001` and symlinks `chrome.log`
|
| 82 |
5. **CDP proxy** β `socat` exposes the DevTools port (9223 β 127.0.0.1:9222).
|
| 83 |
-
6. **Auto-
|
|
|
|
| 84 |
7. **Launch Chrome** β Chrome for Testing in `--headless=new` with all three extensions.
|
| 85 |
|
| 86 |
---
|
| 87 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 88 |
## Why Chrome for Testing (not regular Chrome)
|
| 89 |
|
| 90 |
Branded **Google Chrome β₯ 128 silently ignores `--load-extension`** (removed by the
|
|
@@ -103,12 +150,29 @@ that binary. Everything else (monitor, CDP proxy, flags) is unchanged.
|
|
| 103 |
|
| 104 |
---
|
| 105 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
## Endpoints
|
| 107 |
|
| 108 |
| URL | Description |
|
| 109 |
|-----|-------------|
|
| 110 |
-
| `https://akash1313-
|
| 111 |
-
| `https://akash1313-
|
| 112 |
| `127.0.0.1:9222` (in-container) | Chrome DevTools Protocol |
|
| 113 |
| `:9223` | CDP proxied for external tools (via socat) |
|
| 114 |
|
|
@@ -119,7 +183,10 @@ that binary. Everything else (monitor, CDP proxy, flags) is unchanged.
|
|
| 119 |
```
|
| 120 |
chrome-space/
|
| 121 |
βββ Dockerfile # builds on akashyadav758/chrome, installs CfT, copies extensions
|
|
|
|
| 122 |
βββ start_hf.sh # container entrypoint / Chrome boot sequence
|
|
|
|
|
|
|
| 123 |
βββ chatgpt-free-api/ # ChatGPT Free API server + gpt-extension
|
| 124 |
βββ free-gemini-api/ # Free Gemini API server + gemini-extension
|
| 125 |
βββ flow-agent/ # Flow automation CLI + Flow-extension
|
|
@@ -127,9 +194,38 @@ chrome-space/
|
|
| 127 |
|
| 128 |
---
|
| 129 |
|
| 130 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 131 |
|
| 132 |
-
|
| 133 |
|
| 134 |
```bash
|
| 135 |
# Build
|
|
@@ -147,22 +243,12 @@ Open <http://localhost:3001> to watch the browser.
|
|
| 147 |
|
| 148 |
---
|
| 149 |
|
| 150 |
-
## Deploy
|
| 151 |
-
|
| 152 |
-
This repo **is** the HF Space β pushing to `main` triggers an automatic rebuild:
|
| 153 |
-
|
| 154 |
-
```bash
|
| 155 |
-
git push origin main
|
| 156 |
-
```
|
| 157 |
-
|
| 158 |
-
> `hf-token` is gitignored β keep your Hugging Face token out of commits.
|
| 159 |
-
|
| 160 |
-
---
|
| 161 |
-
|
| 162 |
## Troubleshooting
|
| 163 |
|
| 164 |
| Symptom | Cause / fix |
|
| 165 |
|---------|-------------|
|
|
|
|
|
|
|
| 166 |
| `chrome://extensions/` empty, no error in log | Branded Chrome ignoring `--load-extension` β use Chrome for Testing (see above) |
|
| 167 |
| `COPY failed: ... not found` at build | Extension source dir name in the Dockerfile doesn't match disk |
|
| 168 |
| Extension listed but service worker idle | MV3 headless quirk β open the extension's toggle or hit **Update** on the extensions page |
|
|
|
|
| 1 |
---
|
| 2 |
+
title: Selfapi
|
| 3 |
emoji: π
|
| 4 |
colorFrom: gray
|
| 5 |
colorTo: blue
|
|
|
|
| 8 |
pinned: false
|
| 9 |
---
|
| 10 |
|
| 11 |
+
# Selfapi β Headless Chrome + Unpacked Extensions
|
| 12 |
|
| 13 |
+
A [Hugging Face Space](https://huggingface.co/spaces/akash1313/selfapi) that runs a
|
| 14 |
persistent, headless **Chrome for Testing** browser inside Docker with three custom
|
| 15 |
unpacked extensions auto-loaded, plus a live web monitor to watch the browser in real time.
|
| 16 |
|
| 17 |
+
The browser stays logged in across restarts β on Hugging Face the profile is snapshotted to
|
| 18 |
+
a **Postgres database** (see [Persistence](#persistence-staying-logged-in) below); on a
|
| 19 |
+
self-hosted Docker setup it lives in a named volume. The bundled extensions sync cookies /
|
| 20 |
+
automate workflows for ChatGPT, Google Gemini, and Google Flow.
|
| 21 |
|
| 22 |
---
|
| 23 |
|
|
|
|
| 46 |
| [`free-gemini-api`](free-gemini-api/README.md) | Go reverse-proxy | local server | Gemini text / Imagen 3 images / Gemini video |
|
| 47 |
| [`flow-agent`](flow-agent/README.md) | Python FastAPI + CLI | `127.0.0.1:8100` | Google Flow T2V / V2V / I2V video + T2I / I2I images |
|
| 48 |
|
| 49 |
+
> β οΈ The HF Space builds **only the root `Dockerfile`** β it runs **the browser + the 3
|
| 50 |
+
> extensions + the monitor**. It does **not** start the backend servers. To run the full
|
| 51 |
+
> stack (browser **and** all three API servers sharing one loopback), use
|
| 52 |
+
> `docker-compose.yml` for self-hosting β see [Self-hosting](#self-hosting-full-stack).
|
| 53 |
|
| 54 |
---
|
| 55 |
|
|
|
|
| 62 |
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 63 |
β Docker container (Debian 12) β
|
| 64 |
β β
|
| 65 |
+
β profilesync βββΊ restores profile from Postgres on boot β
|
| 66 |
+
β (Go) and backs it up every 5 min / on exit β
|
| 67 |
+
β β β
|
| 68 |
+
β monitor (3-tab) ββ€ serves the live UI + /chrome.log β
|
| 69 |
β (Go, :3001) β β
|
| 70 |
β βΌ β
|
| 71 |
β Chrome for Testing 145 (--headless=new) β
|
|
|
|
| 79 |
|
| 80 |
`start_hf.sh` is the boot sequence:
|
| 81 |
|
| 82 |
+
0. **Restore profile** β if `DATABASE_URL` is set, `profilesync restore` pulls the last
|
| 83 |
+
Chrome profile snapshot from Postgres so logins survive an HF restart/rebuild.
|
| 84 |
1. **Profile repair & cleanup** β fixes a nested `data/data` layout and removes cache/junk
|
| 85 |
to keep the persistent profile small.
|
| 86 |
2. **Lock removal** β clears `Singleton*` / `LOCK` files left by an unclean shutdown.
|
| 87 |
3. **Preferences injection** β enables `extensions.ui.developer_mode` and sets a clean
|
| 88 |
`exit_type` so the "restore pages?" popup never appears.
|
| 89 |
+
4. **Monitor server** β starts the Go 3-tab monitor on `:3001` and symlinks `chrome.log`.
|
| 90 |
5. **CDP proxy** β `socat` exposes the DevTools port (9223 β 127.0.0.1:9222).
|
| 91 |
+
6. **Auto-backup** β if `DATABASE_URL` is set, `profilesync backup` runs every 5 min (plus
|
| 92 |
+
a final backup on `SIGTERM`/`SIGINT` shutdown).
|
| 93 |
7. **Launch Chrome** β Chrome for Testing in `--headless=new` with all three extensions.
|
| 94 |
|
| 95 |
---
|
| 96 |
|
| 97 |
+
## Persistence (staying logged in)
|
| 98 |
+
|
| 99 |
+
Hugging Face Spaces on the **free `cpu-basic` tier have no persistent disk** β the container
|
| 100 |
+
filesystem (including `/home/chrome/data`) is **wiped on every restart, rebuild, or factory
|
| 101 |
+
reboot**. To keep logins, the profile is snapshotted to an **external Postgres database**.
|
| 102 |
+
|
| 103 |
+
How it works (the `profilesync` Go tool, built into the image):
|
| 104 |
+
|
| 105 |
+
- **On boot:** restores the profile from Postgres (`profilesync restore`).
|
| 106 |
+
- **Every 5 min + on shutdown:** backs the profile up (`profilesync backup`).
|
| 107 |
+
- Stores it in one row: `chrome_profile(id, updated_at, data bytea)` β table auto-created.
|
| 108 |
+
- **All of this is gated on the `DATABASE_URL` env var.** If it is not set, restore/backup
|
| 109 |
+
are silently skipped and logins are lost on restart.
|
| 110 |
+
|
| 111 |
+
### Setup β one-time
|
| 112 |
+
|
| 113 |
+
1. **Create a free Postgres** at [neon.tech](https://neon.tech) β new project β copy the
|
| 114 |
+
connection string. It looks like:
|
| 115 |
+
```
|
| 116 |
+
postgresql://USER:PASSWORD@HOST.neon.tech/DBNAME?sslmode=require
|
| 117 |
+
```
|
| 118 |
+
2. **Add it as a Space secret:** HF Space β **Settings** β **Variables and secrets** β
|
| 119 |
+
**New secret**
|
| 120 |
+
- **Name:** `DATABASE_URL`
|
| 121 |
+
- **Value:** the connection string from step 1
|
| 122 |
+
3. Saving the secret **auto-restarts** the Space. Because nothing was backed up yet, the
|
| 123 |
+
first boot has nothing to restore β so **log in once** after the secret is set. Within
|
| 124 |
+
5 minutes that login is snapshotted to Postgres and will survive all future restarts.
|
| 125 |
+
|
| 126 |
+
> π‘ Verify a backup landed: connect to the DB and run
|
| 127 |
+
> `SELECT id, updated_at, octet_length(data) FROM chrome_profile;` β a row with a non-zero
|
| 128 |
+
> size means the profile is persisted.
|
| 129 |
+
|
| 130 |
+
> π The connection string is a credential β keep it only in the HF secret. A local copy in
|
| 131 |
+
> `neon-db` is **gitignored** so it never reaches this public repo.
|
| 132 |
+
|
| 133 |
+
---
|
| 134 |
+
|
| 135 |
## Why Chrome for Testing (not regular Chrome)
|
| 136 |
|
| 137 |
Branded **Google Chrome β₯ 128 silently ignores `--load-extension`** (removed by the
|
|
|
|
| 150 |
|
| 151 |
---
|
| 152 |
|
| 153 |
+
## User-Agent must match the OS
|
| 154 |
+
|
| 155 |
+
The Space runs on **Linux**, so Chrome (and the backend HTTP clients) advertise a **Linux**
|
| 156 |
+
User-Agent:
|
| 157 |
+
|
| 158 |
+
```
|
| 159 |
+
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36
|
| 160 |
+
```
|
| 161 |
+
|
| 162 |
+
A **macOS UA on a Linux box is an inconsistency** that Cloudflare Turnstile flags, causing
|
| 163 |
+
the ChatGPT "verify you are human" check to loop forever (the tick never clears). The browser
|
| 164 |
+
UA lives in `start_hf.sh`; the ChatGPT backend (`chatgpt-free-api/cookies.go`) uses the **same
|
| 165 |
+
UA and Chrome version (145)** so the `cf_clearance` cookie β which Cloudflare binds to the
|
| 166 |
+
exact UA β stays valid across the browser and the backend.
|
| 167 |
+
|
| 168 |
+
---
|
| 169 |
+
|
| 170 |
## Endpoints
|
| 171 |
|
| 172 |
| URL | Description |
|
| 173 |
|-----|-------------|
|
| 174 |
+
| `https://akash1313-selfapi.hf.space/` | Live Chrome monitor UI |
|
| 175 |
+
| `https://akash1313-selfapi.hf.space/chrome.log` | Streaming Chrome log (debugging) |
|
| 176 |
| `127.0.0.1:9222` (in-container) | Chrome DevTools Protocol |
|
| 177 |
| `:9223` | CDP proxied for external tools (via socat) |
|
| 178 |
|
|
|
|
| 183 |
```
|
| 184 |
chrome-space/
|
| 185 |
βββ Dockerfile # builds on akashyadav758/chrome, installs CfT, copies extensions
|
| 186 |
+
βββ docker-compose.yml # full self-host stack: browser + 3 API servers on one loopback
|
| 187 |
βββ start_hf.sh # container entrypoint / Chrome boot sequence
|
| 188 |
+
βββ profilesync/ # Go tool: Chrome profile <-> Postgres snapshot
|
| 189 |
+
βββ monitor/ # Go 3-tab live monitor (GPT / Gemini / Flow)
|
| 190 |
βββ chatgpt-free-api/ # ChatGPT Free API server + gpt-extension
|
| 191 |
βββ free-gemini-api/ # Free Gemini API server + gemini-extension
|
| 192 |
βββ flow-agent/ # Flow automation CLI + Flow-extension
|
|
|
|
| 194 |
|
| 195 |
---
|
| 196 |
|
| 197 |
+
## Deploy
|
| 198 |
+
|
| 199 |
+
This repo **is** the HF Space β pushing to `main` triggers an automatic rebuild:
|
| 200 |
+
|
| 201 |
+
```bash
|
| 202 |
+
git push origin main
|
| 203 |
+
```
|
| 204 |
+
|
| 205 |
+
First-time / persistence checklist:
|
| 206 |
+
|
| 207 |
+
1. Push the code (above).
|
| 208 |
+
2. Set the `DATABASE_URL` secret β see [Persistence](#persistence-staying-logged-in).
|
| 209 |
+
3. Log in once in the monitor UI; the profile auto-backs up within 5 min.
|
| 210 |
+
|
| 211 |
+
> `hf-token` and `neon-db` are gitignored β keep your HF token and DB credentials out of
|
| 212 |
+
> commits.
|
| 213 |
+
|
| 214 |
+
---
|
| 215 |
+
|
| 216 |
+
## Self-hosting (full stack)
|
| 217 |
+
|
| 218 |
+
To run the browser **and** all three backend API servers locally, use `docker-compose.yml`.
|
| 219 |
+
The server containers join the chrome container's network namespace
|
| 220 |
+
(`network_mode: "service:chrome"`), so they all share one `127.0.0.1` β exactly what the
|
| 221 |
+
extensions expect. The Chrome profile persists in a named Docker volume (no Postgres needed).
|
| 222 |
+
|
| 223 |
+
```bash
|
| 224 |
+
docker compose up --build -d # browser + chatgpt + gemini + flow
|
| 225 |
+
docker compose down # profile survives in the chrome-profile volume
|
| 226 |
+
```
|
| 227 |
|
| 228 |
+
Or just the browser image:
|
| 229 |
|
| 230 |
```bash
|
| 231 |
# Build
|
|
|
|
| 243 |
|
| 244 |
---
|
| 245 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 246 |
## Troubleshooting
|
| 247 |
|
| 248 |
| Symptom | Cause / fix |
|
| 249 |
|---------|-------------|
|
| 250 |
+
| Login lost after restart/rebuild | `DATABASE_URL` not set (or DB unreachable) β set the Postgres secret, see [Persistence](#persistence-staying-logged-in) |
|
| 251 |
+
| ChatGPT Cloudflare "verify you are human" loops, tick never passes | UA/OS mismatch (macOS UA on Linux) β ensure the **Linux** UA in `start_hf.sh`. If it still loops, it's the **datacenter IP** (HF = AWS); route through a residential proxy via `--proxy-server=http://user:pass@host:port` |
|
| 252 |
| `chrome://extensions/` empty, no error in log | Branded Chrome ignoring `--load-extension` β use Chrome for Testing (see above) |
|
| 253 |
| `COPY failed: ... not found` at build | Extension source dir name in the Dockerfile doesn't match disk |
|
| 254 |
| Extension listed but service worker idle | MV3 headless quirk β open the extension's toggle or hit **Update** on the extensions page |
|