akashyadav758 Claude Opus 4.8 (1M context) commited on
Commit
7869843
Β·
1 Parent(s): 645c976

Document Postgres persistence, selfapi space name, and UA/Cloudflare fix in README

Browse files

Add a Persistence section with the Neon DATABASE_URL setup guide, a
self-hosting section for docker-compose, a UA-must-match-OS note, fix the
stale chrome -> selfapi space name/URLs and title, and update the boot
sequence + troubleshooting to match the current start_hf.sh.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files changed (1) hide show
  1. README.md +113 -27
README.md CHANGED
@@ -1,5 +1,5 @@
1
  ---
2
- title: Chrome
3
  emoji: πŸ“Š
4
  colorFrom: gray
5
  colorTo: blue
@@ -8,14 +8,16 @@ app_port: 3001
8
  pinned: false
9
  ---
10
 
11
- # Chrome Space β€” Headless Chrome + Unpacked Extensions
12
 
13
- A [Hugging Face Space](https://huggingface.co/spaces/akash1313/chrome) that runs a
14
  persistent, headless **Chrome for Testing** browser inside Docker with three custom
15
  unpacked extensions auto-loaded, plus a live web monitor to watch the browser in real time.
16
 
17
- The browser stays logged in across restarts (persistent profile), and the bundled
18
- extensions sync cookies / automate workflows for ChatGPT, Google Gemini, and Google Flow.
 
 
19
 
20
  ---
21
 
@@ -44,9 +46,10 @@ run yourself β€” see each subproject's README for setup):
44
  | [`free-gemini-api`](free-gemini-api/README.md) | Go reverse-proxy | local server | Gemini text / Imagen 3 images / Gemini video |
45
  | [`flow-agent`](flow-agent/README.md) | Python FastAPI + CLI | `127.0.0.1:8100` | Google Flow T2V / V2V / I2V video + T2I / I2I images |
46
 
47
- > ⚠️ This Space runs **only the browser + the 3 extensions**. It does **not** start the
48
- > backend servers β€” run those locally (or in their own containers) and the extensions will
49
- > connect once the matching port is reachable.
 
50
 
51
  ---
52
 
@@ -59,7 +62,10 @@ HF Space (port 3001)
59
  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
60
  β”‚ Docker container (Debian 12) β”‚
61
  β”‚ β”‚
62
- β”‚ monitor-server ──┐ serves the live UI + /chrome.log β”‚
 
 
 
63
  β”‚ (Go, :3001) β”‚ β”‚
64
  β”‚ β–Ό β”‚
65
  β”‚ Chrome for Testing 145 (--headless=new) β”‚
@@ -73,18 +79,59 @@ HF Space (port 3001)
73
 
74
  `start_hf.sh` is the boot sequence:
75
 
 
 
76
  1. **Profile repair & cleanup** β€” fixes a nested `data/data` layout and removes cache/junk
77
  to keep the persistent profile small.
78
  2. **Lock removal** β€” clears `Singleton*` / `LOCK` files left by an unclean shutdown.
79
  3. **Preferences injection** β€” enables `extensions.ui.developer_mode` and sets a clean
80
  `exit_type` so the "restore pages?" popup never appears.
81
- 4. **Monitor server** β€” starts the Go monitor on `:3001` and symlinks `chrome.log` into it.
82
  5. **CDP proxy** β€” `socat` exposes the DevTools port (9223 β†’ 127.0.0.1:9222).
83
- 6. **Auto-navigate** β€” a tiny WebSocket client drives the first tab to `chrome://extensions/`.
 
84
  7. **Launch Chrome** β€” Chrome for Testing in `--headless=new` with all three extensions.
85
 
86
  ---
87
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
88
  ## Why Chrome for Testing (not regular Chrome)
89
 
90
  Branded **Google Chrome β‰₯ 128 silently ignores `--load-extension`** (removed by the
@@ -103,12 +150,29 @@ that binary. Everything else (monitor, CDP proxy, flags) is unchanged.
103
 
104
  ---
105
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
106
  ## Endpoints
107
 
108
  | URL | Description |
109
  |-----|-------------|
110
- | `https://akash1313-chrome.hf.space/` | Live Chrome monitor UI |
111
- | `https://akash1313-chrome.hf.space/chrome.log` | Streaming Chrome log (debugging) |
112
  | `127.0.0.1:9222` (in-container) | Chrome DevTools Protocol |
113
  | `:9223` | CDP proxied for external tools (via socat) |
114
 
@@ -119,7 +183,10 @@ that binary. Everything else (monitor, CDP proxy, flags) is unchanged.
119
  ```
120
  chrome-space/
121
  β”œβ”€β”€ Dockerfile # builds on akashyadav758/chrome, installs CfT, copies extensions
 
122
  β”œβ”€β”€ start_hf.sh # container entrypoint / Chrome boot sequence
 
 
123
  β”œβ”€β”€ chatgpt-free-api/ # ChatGPT Free API server + gpt-extension
124
  β”œβ”€β”€ free-gemini-api/ # Free Gemini API server + gemini-extension
125
  └── flow-agent/ # Flow automation CLI + Flow-extension
@@ -127,9 +194,38 @@ chrome-space/
127
 
128
  ---
129
 
130
- ## Local development
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
131
 
132
- Requires Docker. The base image is `linux/amd64` only.
133
 
134
  ```bash
135
  # Build
@@ -147,22 +243,12 @@ Open <http://localhost:3001> to watch the browser.
147
 
148
  ---
149
 
150
- ## Deploy
151
-
152
- This repo **is** the HF Space β€” pushing to `main` triggers an automatic rebuild:
153
-
154
- ```bash
155
- git push origin main
156
- ```
157
-
158
- > `hf-token` is gitignored β€” keep your Hugging Face token out of commits.
159
-
160
- ---
161
-
162
  ## Troubleshooting
163
 
164
  | Symptom | Cause / fix |
165
  |---------|-------------|
 
 
166
  | `chrome://extensions/` empty, no error in log | Branded Chrome ignoring `--load-extension` β†’ use Chrome for Testing (see above) |
167
  | `COPY failed: ... not found` at build | Extension source dir name in the Dockerfile doesn't match disk |
168
  | Extension listed but service worker idle | MV3 headless quirk β€” open the extension's toggle or hit **Update** on the extensions page |
 
1
  ---
2
+ title: Selfapi
3
  emoji: πŸ“Š
4
  colorFrom: gray
5
  colorTo: blue
 
8
  pinned: false
9
  ---
10
 
11
+ # Selfapi β€” Headless Chrome + Unpacked Extensions
12
 
13
+ A [Hugging Face Space](https://huggingface.co/spaces/akash1313/selfapi) that runs a
14
  persistent, headless **Chrome for Testing** browser inside Docker with three custom
15
  unpacked extensions auto-loaded, plus a live web monitor to watch the browser in real time.
16
 
17
+ The browser stays logged in across restarts β€” on Hugging Face the profile is snapshotted to
18
+ a **Postgres database** (see [Persistence](#persistence-staying-logged-in) below); on a
19
+ self-hosted Docker setup it lives in a named volume. The bundled extensions sync cookies /
20
+ automate workflows for ChatGPT, Google Gemini, and Google Flow.
21
 
22
  ---
23
 
 
46
  | [`free-gemini-api`](free-gemini-api/README.md) | Go reverse-proxy | local server | Gemini text / Imagen 3 images / Gemini video |
47
  | [`flow-agent`](flow-agent/README.md) | Python FastAPI + CLI | `127.0.0.1:8100` | Google Flow T2V / V2V / I2V video + T2I / I2I images |
48
 
49
+ > ⚠️ The HF Space builds **only the root `Dockerfile`** β†’ it runs **the browser + the 3
50
+ > extensions + the monitor**. It does **not** start the backend servers. To run the full
51
+ > stack (browser **and** all three API servers sharing one loopback), use
52
+ > `docker-compose.yml` for self-hosting β€” see [Self-hosting](#self-hosting-full-stack).
53
 
54
  ---
55
 
 
62
  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
63
  β”‚ Docker container (Debian 12) β”‚
64
  β”‚ β”‚
65
+ β”‚ profilesync ──► restores profile from Postgres on boot β”‚
66
+ β”‚ (Go) and backs it up every 5 min / on exit β”‚
67
+ β”‚ β”‚ β”‚
68
+ β”‚ monitor (3-tab) ── serves the live UI + /chrome.log β”‚
69
  β”‚ (Go, :3001) β”‚ β”‚
70
  β”‚ β–Ό β”‚
71
  β”‚ Chrome for Testing 145 (--headless=new) β”‚
 
79
 
80
  `start_hf.sh` is the boot sequence:
81
 
82
+ 0. **Restore profile** β€” if `DATABASE_URL` is set, `profilesync restore` pulls the last
83
+ Chrome profile snapshot from Postgres so logins survive an HF restart/rebuild.
84
  1. **Profile repair & cleanup** β€” fixes a nested `data/data` layout and removes cache/junk
85
  to keep the persistent profile small.
86
  2. **Lock removal** β€” clears `Singleton*` / `LOCK` files left by an unclean shutdown.
87
  3. **Preferences injection** β€” enables `extensions.ui.developer_mode` and sets a clean
88
  `exit_type` so the "restore pages?" popup never appears.
89
+ 4. **Monitor server** β€” starts the Go 3-tab monitor on `:3001` and symlinks `chrome.log`.
90
  5. **CDP proxy** β€” `socat` exposes the DevTools port (9223 β†’ 127.0.0.1:9222).
91
+ 6. **Auto-backup** β€” if `DATABASE_URL` is set, `profilesync backup` runs every 5 min (plus
92
+ a final backup on `SIGTERM`/`SIGINT` shutdown).
93
  7. **Launch Chrome** β€” Chrome for Testing in `--headless=new` with all three extensions.
94
 
95
  ---
96
 
97
+ ## Persistence (staying logged in)
98
+
99
+ Hugging Face Spaces on the **free `cpu-basic` tier have no persistent disk** β€” the container
100
+ filesystem (including `/home/chrome/data`) is **wiped on every restart, rebuild, or factory
101
+ reboot**. To keep logins, the profile is snapshotted to an **external Postgres database**.
102
+
103
+ How it works (the `profilesync` Go tool, built into the image):
104
+
105
+ - **On boot:** restores the profile from Postgres (`profilesync restore`).
106
+ - **Every 5 min + on shutdown:** backs the profile up (`profilesync backup`).
107
+ - Stores it in one row: `chrome_profile(id, updated_at, data bytea)` β€” table auto-created.
108
+ - **All of this is gated on the `DATABASE_URL` env var.** If it is not set, restore/backup
109
+ are silently skipped and logins are lost on restart.
110
+
111
+ ### Setup β€” one-time
112
+
113
+ 1. **Create a free Postgres** at [neon.tech](https://neon.tech) β†’ new project β†’ copy the
114
+ connection string. It looks like:
115
+ ```
116
+ postgresql://USER:PASSWORD@HOST.neon.tech/DBNAME?sslmode=require
117
+ ```
118
+ 2. **Add it as a Space secret:** HF Space β†’ **Settings** β†’ **Variables and secrets** β†’
119
+ **New secret**
120
+ - **Name:** `DATABASE_URL`
121
+ - **Value:** the connection string from step 1
122
+ 3. Saving the secret **auto-restarts** the Space. Because nothing was backed up yet, the
123
+ first boot has nothing to restore β€” so **log in once** after the secret is set. Within
124
+ 5 minutes that login is snapshotted to Postgres and will survive all future restarts.
125
+
126
+ > πŸ’‘ Verify a backup landed: connect to the DB and run
127
+ > `SELECT id, updated_at, octet_length(data) FROM chrome_profile;` β€” a row with a non-zero
128
+ > size means the profile is persisted.
129
+
130
+ > πŸ”’ The connection string is a credential β€” keep it only in the HF secret. A local copy in
131
+ > `neon-db` is **gitignored** so it never reaches this public repo.
132
+
133
+ ---
134
+
135
  ## Why Chrome for Testing (not regular Chrome)
136
 
137
  Branded **Google Chrome β‰₯ 128 silently ignores `--load-extension`** (removed by the
 
150
 
151
  ---
152
 
153
+ ## User-Agent must match the OS
154
+
155
+ The Space runs on **Linux**, so Chrome (and the backend HTTP clients) advertise a **Linux**
156
+ User-Agent:
157
+
158
+ ```
159
+ Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36
160
+ ```
161
+
162
+ A **macOS UA on a Linux box is an inconsistency** that Cloudflare Turnstile flags, causing
163
+ the ChatGPT "verify you are human" check to loop forever (the tick never clears). The browser
164
+ UA lives in `start_hf.sh`; the ChatGPT backend (`chatgpt-free-api/cookies.go`) uses the **same
165
+ UA and Chrome version (145)** so the `cf_clearance` cookie β€” which Cloudflare binds to the
166
+ exact UA β€” stays valid across the browser and the backend.
167
+
168
+ ---
169
+
170
  ## Endpoints
171
 
172
  | URL | Description |
173
  |-----|-------------|
174
+ | `https://akash1313-selfapi.hf.space/` | Live Chrome monitor UI |
175
+ | `https://akash1313-selfapi.hf.space/chrome.log` | Streaming Chrome log (debugging) |
176
  | `127.0.0.1:9222` (in-container) | Chrome DevTools Protocol |
177
  | `:9223` | CDP proxied for external tools (via socat) |
178
 
 
183
  ```
184
  chrome-space/
185
  β”œβ”€β”€ Dockerfile # builds on akashyadav758/chrome, installs CfT, copies extensions
186
+ β”œβ”€β”€ docker-compose.yml # full self-host stack: browser + 3 API servers on one loopback
187
  β”œβ”€β”€ start_hf.sh # container entrypoint / Chrome boot sequence
188
+ β”œβ”€β”€ profilesync/ # Go tool: Chrome profile <-> Postgres snapshot
189
+ β”œβ”€β”€ monitor/ # Go 3-tab live monitor (GPT / Gemini / Flow)
190
  β”œβ”€β”€ chatgpt-free-api/ # ChatGPT Free API server + gpt-extension
191
  β”œβ”€β”€ free-gemini-api/ # Free Gemini API server + gemini-extension
192
  └── flow-agent/ # Flow automation CLI + Flow-extension
 
194
 
195
  ---
196
 
197
+ ## Deploy
198
+
199
+ This repo **is** the HF Space β€” pushing to `main` triggers an automatic rebuild:
200
+
201
+ ```bash
202
+ git push origin main
203
+ ```
204
+
205
+ First-time / persistence checklist:
206
+
207
+ 1. Push the code (above).
208
+ 2. Set the `DATABASE_URL` secret β€” see [Persistence](#persistence-staying-logged-in).
209
+ 3. Log in once in the monitor UI; the profile auto-backs up within 5 min.
210
+
211
+ > `hf-token` and `neon-db` are gitignored β€” keep your HF token and DB credentials out of
212
+ > commits.
213
+
214
+ ---
215
+
216
+ ## Self-hosting (full stack)
217
+
218
+ To run the browser **and** all three backend API servers locally, use `docker-compose.yml`.
219
+ The server containers join the chrome container's network namespace
220
+ (`network_mode: "service:chrome"`), so they all share one `127.0.0.1` β€” exactly what the
221
+ extensions expect. The Chrome profile persists in a named Docker volume (no Postgres needed).
222
+
223
+ ```bash
224
+ docker compose up --build -d # browser + chatgpt + gemini + flow
225
+ docker compose down # profile survives in the chrome-profile volume
226
+ ```
227
 
228
+ Or just the browser image:
229
 
230
  ```bash
231
  # Build
 
243
 
244
  ---
245
 
 
 
 
 
 
 
 
 
 
 
 
 
246
  ## Troubleshooting
247
 
248
  | Symptom | Cause / fix |
249
  |---------|-------------|
250
+ | Login lost after restart/rebuild | `DATABASE_URL` not set (or DB unreachable) β†’ set the Postgres secret, see [Persistence](#persistence-staying-logged-in) |
251
+ | ChatGPT Cloudflare "verify you are human" loops, tick never passes | UA/OS mismatch (macOS UA on Linux) β€” ensure the **Linux** UA in `start_hf.sh`. If it still loops, it's the **datacenter IP** (HF = AWS); route through a residential proxy via `--proxy-server=http://user:pass@host:port` |
252
  | `chrome://extensions/` empty, no error in log | Branded Chrome ignoring `--load-extension` β†’ use Chrome for Testing (see above) |
253
  | `COPY failed: ... not found` at build | Extension source dir name in the Dockerfile doesn't match disk |
254
  | Extension listed but service worker idle | MV3 headless quirk β€” open the extension's toggle or hit **Update** on the extensions page |