Spaces:
Sleeping
Sleeping
| title: Selfapi-v2 | |
| emoji: 📊 | |
| colorFrom: gray | |
| colorTo: blue | |
| sdk: docker | |
| app_port: 3001 | |
| pinned: false | |
| # Selfapi-v2 — Headless Chrome + Unpacked Extensions | |
| A [Hugging Face Space](https://huggingface.co/spaces/akash1313/selfapi-v2) that runs a | |
| persistent, headless **Chrome for Testing** browser inside Docker with three custom | |
| unpacked extensions auto-loaded, plus a live web monitor to watch the browser in real time. | |
| The browser stays logged in across restarts — on Hugging Face the profile is snapshotted to | |
| a **Postgres database** (see [Persistence](#persistence-staying-logged-in) below); on a | |
| self-hosted Docker setup it lives in a named volume. The bundled extensions sync cookies / | |
| automate workflows for ChatGPT, Google Gemini, and Google Flow. | |
| --- | |
| ## Bundled extensions | |
| | Extension | Source dir | Loaded at | What it does | | |
| |-----------|-----------|-----------|--------------| | |
| | **Gpt Agent Api** | `free-Chatgpt-api/gpt-extension` | `/opt/gpt-extension` | Auto-syncs ChatGPT cookies to a local ChatGPT Free API server | | |
| | **Gemini Api Agent** | `free-gemini-api/gemini-extension` | `/opt/gemini-extension` | Auto-syncs Google Gemini cookies to a local Free Gemini API server | | |
| Both are Manifest V3 extensions with background service workers. Each subfolder | |
| also contains the matching backend server. | |
| --- | |
| ## Backend servers | |
| Each extension is the browser half of a standalone project. The extension uses your | |
| authenticated browser session and forwards requests to a **local** API server (which you | |
| run yourself — see each subproject's README for setup): | |
| | Project | Server | Talks to | Purpose | | |
| |---------|--------|----------|---------| | |
| | `free-Chatgpt-api` | Go | `127.0.0.1:9225` | ChatGPT backend API from your logged-in session | | |
| | `free-gemini-api` | Go reverse-proxy | local server | Gemini text / Imagen 3 images / Gemini video | | |
| | `flow-agent` | Python FastAPI + CLI | `127.0.0.1:8100` | Google Flow T2V / V2V / I2V video + T2I / I2I images | | |
| > ✅ The HF Space now runs **all three servers in the same container** as Chrome, and the | |
| > monitor doubles as a **single API gateway** that fronts them on one public port — see | |
| > [API gateway](#api-gateway). (The extensions dial their servers on hardcoded | |
| > `127.0.0.1:9225/9226/9227`, so the servers *must* share Chrome's localhost — a separate | |
| > Space can't reach them.) `docker-compose.yml` does the same wiring for self-hosting. | |
| --- | |
| ## How it works | |
| ``` | |
| HF Space (port 3001) | |
| │ | |
| ▼ | |
| ┌─────────────────────────────────────────────────────────┐ | |
| │ Docker container (Debian 12) │ | |
| │ │ | |
| │ profilesync ──► restores profile from Postgres on boot │ | |
| │ (Go) and backs it up every 5 min / on exit │ | |
| │ │ │ | |
| │ monitor (3-tab) ─┤ serves the live UI + /chrome.log │ | |
| │ (Go, :3001) │ │ | |
| │ ▼ │ | |
| │ Chrome for Testing 145 (--headless=new) │ | |
| │ --load-extension=/opt/gpt,/opt/gemini,/opt/flow │ | |
| │ --remote-debugging-port=9222 (CDP) │ | |
| │ --user-data-dir=/home/chrome/data (persistent) │ | |
| │ ▲ │ | |
| │ socat :9223 ─────┘ forwards external CDP → 127.0.0.1 │ | |
| └─────────────────────────────────────────────────────────┘ | |
| ``` | |
| `start_hf.sh` is the boot sequence: | |
| 0. **Restore profile** — if `DATABASE_URL` is set, `profilesync restore` pulls the last | |
| Chrome profile snapshot from Postgres so logins survive an HF restart/rebuild. | |
| 1. **Profile repair & cleanup** — fixes a nested `data/data` layout and removes cache/junk | |
| to keep the persistent profile small. | |
| 2. **Lock removal** — clears `Singleton*` / `LOCK` files left by an unclean shutdown. | |
| 3. **Preferences injection** — enables `extensions.ui.developer_mode` and sets a clean | |
| `exit_type` so the "restore pages?" popup never appears. | |
| 4. **Monitor server** — starts the Go 3-tab monitor on `:3001` and symlinks `chrome.log`. | |
| 5. **CDP proxy** — `socat` exposes the DevTools port (9223 → 127.0.0.1:9222). | |
| 6. **Auto-backup** — if `DATABASE_URL` is set, `profilesync backup` runs every 5 min (plus | |
| a final backup on `SIGTERM`/`SIGINT` shutdown). | |
| 7. **Launch Chrome** — Chrome for Testing in `--headless=new` with all three extensions. | |
| --- | |
| ## Persistence (staying logged in) | |
| Hugging Face Spaces on the **free `cpu-basic` tier have no persistent disk** — the container | |
| filesystem (including `/home/chrome/data`) is **wiped on every restart, rebuild, or factory | |
| reboot**. To keep logins, the profile is snapshotted to an **external Postgres database**. | |
| How it works (the `profilesync` Go tool, built into the image): | |
| - **On boot:** restores the profile from Postgres (`profilesync restore`). | |
| - **Every 5 min + on shutdown:** backs the profile up (`profilesync backup`). | |
| - Stores it in one row: `chrome_profile(id, updated_at, data bytea)` — table auto-created. | |
| - **All of this is gated on the `DATABASE_URL` env var.** If it is not set, restore/backup | |
| are silently skipped and logins are lost on restart. | |
| ### Setup — one-time | |
| 1. **Create a free Postgres** at [neon.tech](https://neon.tech) → new project → copy the | |
| connection string. It looks like: | |
| ``` | |
| postgresql://USER:PASSWORD@HOST.neon.tech/DBNAME?sslmode=require | |
| ``` | |
| 2. **Add it as a Space secret:** HF Space → **Settings** → **Variables and secrets** → | |
| **New secret** | |
| - **Name:** `DATABASE_URL` | |
| - **Value:** the connection string from step 1 | |
| 3. Saving the secret **auto-restarts** the Space. Because nothing was backed up yet, the | |
| first boot has nothing to restore — so **log in once** after the secret is set. Within | |
| 5 minutes that login is snapshotted to Postgres and will survive all future restarts. | |
| > 💡 Verify a backup landed: connect to the DB and run | |
| > `SELECT id, updated_at, octet_length(data) FROM chrome_profile;` — a row with a non-zero | |
| > size means the profile is persisted. | |
| > 🔒 The connection string is a credential — keep it only in the HF secret. A local copy in | |
| > `neon-db` is **gitignored** so it never reaches this public repo. | |
| --- | |
| ## Why Chrome for Testing (not regular Chrome) | |
| Branded **Google Chrome ≥ 128 silently ignores `--load-extension`** (removed by the | |
| [March 2025 Chrome RFC](https://developer.chrome.com/blog/remote-debugging-port) to curb | |
| abuse). On a branded build the extensions never load and `chrome://extensions/` shows an | |
| empty list with **no error in the log**. | |
| The fix is to run **[Chrome for Testing](https://googlechromelabs.github.io/chrome-for-testing/)** | |
| — the unbranded build of the same Chrome version where `--load-extension` still works. The | |
| Dockerfile downloads CfT `145.0.7632.117` to `/opt/chrome-linux64` and `start_hf.sh` launches | |
| that binary. Everything else (monitor, CDP proxy, flags) is unchanged. | |
| > Alternatives if you ever upgrade: use the CDP `Extensions.loadUnpacked` method | |
| > (requires `--remote-debugging-pipe` + `--enable-unsafe-extension-debugging`), or plain | |
| > Chromium. Both also keep `--load-extension`-style loading working. | |
| --- | |
| ## User-Agent must match the OS | |
| The Space runs on **Linux**, so Chrome (and the backend HTTP clients) advertise a **Linux** | |
| User-Agent: | |
| ``` | |
| Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 | |
| ``` | |
| A **macOS UA on a Linux box is an inconsistency** that Cloudflare Turnstile flags, causing | |
| the ChatGPT "verify you are human" check to loop forever (the tick never clears). The browser | |
| UA lives in `start_hf.sh`; the ChatGPT backend (`free-Chatgpt-api/cookies.go`) uses the **same | |
| UA and Chrome version (145)** so the `cf_clearance` cookie — which Cloudflare binds to the | |
| exact UA — stays valid across the browser and the backend. | |
| --- | |
| ## Endpoints | |
| | URL | Description | | |
| |-----|-------------| | |
| | `https://akash1313-selfapi-v2.hf.space/` | Live Chrome monitor UI — **key required** (open `/?key=YOUR_API_KEY` once; it sets a cookie) | | |
| | `https://akash1313-selfapi-v2.hf.space/chrome.log` | Streaming Chrome log — **key required** | | |
| | `https://akash1313-selfapi-v2.hf.space/gpt/…` | ChatGPT API (gateway → `127.0.0.1:9225`) | | |
| | `https://akash1313-selfapi-v2.hf.space/gemini/…` | Gemini API (gateway → `127.0.0.1:8000`) | | |
| | `https://akash1313-selfapi-v2.hf.space/flow/…` | Flow API (gateway → `127.0.0.1:8101`) | | |
| | `127.0.0.1:9222` (in-container) | Chrome DevTools Protocol | | |
| | `:9223` | CDP proxied for external tools (via socat) | | |
| --- | |
| ## API gateway | |
| The monitor (`monitor/main.go`) is also a reverse-proxy gateway: HF exposes only one public | |
| port (`3001`), so all three backend APIs are reached under one base URL by path prefix. The | |
| gateway strips the prefix and forwards to the server on localhost: | |
| | Prefix | Backend | Example endpoints | | |
| |--------|---------|-------------------| | |
| | `/gpt/` | free-Chatgpt-api `:9225` | `/gpt/api/chat`, `/gpt/v1/chat/completions`, `/gpt/health` | | |
| | `/gemini/` | free-gemini-api `:8000` | `/gemini/chat`, `/gemini/status`, `/gemini/output/*` | | |
| | `/flow/` | flow-agent `:8101` | `/flow/generate/video`, `/flow/generate/image`, `/flow/health` | | |
| ### Auth — required | |
| **Every** route — the API prefixes, the monitor UI (`/`), `/api/*`, `/chrome.log`, `/logs/*` — is | |
| gated by an **`API_KEY`** Space secret. Send it as a Bearer token, `?key=`, or (for the browser UI) | |
| open `/?key=YOUR_API_KEY` once to set an `apikey` cookie. **Fail-closed:** if `API_KEY` is unset | |
| every route returns `503`, so the logged-in accounts are never exposed. This Space is also | |
| **private** on Hugging Face, so the URL itself requires HF auth — defense in depth. | |
| 1. Set the Space secret **`API_KEY`** (Settings → Variables and secrets) to a strong random string. | |
| 2. Call it: | |
| ```bash | |
| curl -X POST https://akash1313-selfapi-v2.hf.space/gpt/api/chat \ | |
| -H "Authorization: Bearer $API_KEY" \ | |
| -H "Content-Type: application/json" \ | |
| -d '{"message":"hello"}' | |
| ``` | |
| No/!wrong key → `401`. Wrong path → forwarded to the backend (may `404` there). | |
| --- | |
| ## Project layout | |
| ``` | |
| chrome-space/ | |
| ├── Dockerfile # builds on akashyadav758/chrome, installs CfT, copies extensions | |
| ├── docker-compose.yml # full self-host stack: browser + 3 API servers on one loopback | |
| ├── start_hf.sh # container entrypoint / Chrome boot sequence | |
| ├── profilesync/ # Go tool: Chrome profile <-> Postgres snapshot | |
| ├── monitor/ # Go 3-tab live monitor (GPT / Gemini / Flow) | |
| ├── free-Chatgpt-api/ # ChatGPT Free API server + gpt-extension | |
| ├── free-gemini-api/ # Free Gemini API server + gemini-extension | |
| └── flow-agent/ # Flow automation CLI + Flow-extension | |
| ``` | |
| --- | |
| ## Deploy | |
| This repo **is** the HF Space — pushing to `main` triggers an automatic rebuild: | |
| ```bash | |
| git push origin main | |
| ``` | |
| First-time / persistence checklist: | |
| 1. Push the code (above). | |
| 2. Set the `DATABASE_URL` secret — see [Persistence](#persistence-staying-logged-in). | |
| 3. Log in once in the monitor UI; the profile auto-backs up within 5 min. | |
| > `hf-token` and `neon-db` are gitignored — keep your HF token and DB credentials out of | |
| > commits. | |
| --- | |
| ## Self-hosting (full stack) | |
| To run the browser **and** all three backend API servers locally, use `docker-compose.yml`. | |
| The server containers join the chrome container's network namespace | |
| (`network_mode: "service:chrome"`), so they all share one `127.0.0.1` — exactly what the | |
| extensions expect. The Chrome profile persists in a named Docker volume (no Postgres needed). | |
| ```bash | |
| docker compose up --build -d # browser + chatgpt + gemini + flow | |
| docker compose down # profile survives in the chrome-profile volume | |
| ``` | |
| Or just the browser image: | |
| ```bash | |
| # Build | |
| docker build --platform linux/amd64 -t chrome-space . | |
| # Run | |
| docker run -d --name chrome-space -p 3001:3001 --platform linux/amd64 chrome-space | |
| # Verify the 3 extensions loaded (look for service_worker targets) | |
| docker exec chrome-space curl -s http://127.0.0.1:9222/json \ | |
| | python3 -c 'import json,sys;[print(t["type"],t["url"]) for t in json.load(sys.stdin)]' | |
| ``` | |
| Open <http://localhost:3001> to watch the browser. | |
| --- | |
| ## Troubleshooting | |
| | Symptom | Cause / fix | | |
| |---------|-------------| | |
| | Login lost after restart/rebuild | `DATABASE_URL` not set (or DB unreachable) → set the Postgres secret, see [Persistence](#persistence-staying-logged-in) | | |
| | ChatGPT Cloudflare "verify you are human" loops, tick never passes | UA/OS mismatch (macOS UA on Linux) — ensure the **Linux** UA in `start_hf.sh`. If it still loops, it's the **datacenter IP** (HF = AWS); route through a residential proxy via `--proxy-server=http://user:pass@host:port` | | |
| | `chrome://extensions/` empty, no error in log | Branded Chrome ignoring `--load-extension` → use Chrome for Testing (see above) | | |
| | `COPY failed: ... not found` at build | Extension source dir name in the Dockerfile doesn't match disk | | |
| | Extension listed but service worker idle | MV3 headless quirk — open the extension's toggle or hit **Update** on the extensions page | | |
| | `Failed to connect to the bus` spam in log | Harmless D-Bus warnings in a container with no system bus | | |
| | "Restore pages?" popup | Handled by the `exit_type=Normal` Preferences injection in `start_hf.sh` | | |