Blame for dev/HANDOFF.md

# HANDOFF — DOS MQTT LAB

Read this first. Everything below was checked against the real running page, not assumed.
Project owner language: Traditional Chinese. Replies: short, factual, no emojis, say limitations honestly.

## 1. What it is (and what it is not)

* Space: `HelloSun/dosmqtt` (static, public). Live: https://hellosun-dosmqtt.static.hf.space/
* Two pages: `index.html` (MQTT, two machines) and `gradio.html` (DOS asks a Gradio Space a question, one machine, section 9).
* Two **FreeDOS 1.4** machines ("A" listener, "B" sender) run side by side in the page on the **v86** x86 emulator.
* Inside each DOS: Crynwr **NE2000 packet driver** -> mTCP **DHCP.EXE** (gets 192.168.86.100) ->
  **A.EXE / B.EXE**, real DOS programs (Open Watcom + mTCP library) doing DNS, TCP and **MQTT 3.1.1** (QoS 0) themselves.
* `A.EXE`: connect, subscribe, print `[hh:mm:ss] A rx: <msg>`, 1 s later print `[hh:mm:ss] A echo: <msg>`.
* `B.EXE`: run once: connect, publish `Hello from DOS B at hh:mm:ss`, disconnect, exit. The page types `B` every 5 s
  (DOS has no scheduler). The page types `A` once.
* **Honest limits**: only the last hop is not TCP: the browser cannot open raw TCP, and a static Space has no server,
  so `site/bridge.js` relays the guest's TCP streams on **port 1883** to a public broker's **MQTT-over-WSS** endpoint.
  MQTT is identical on both transports, so the bytes DOS writes are exactly what the broker receives (verified with an
  independent raw-TCP subscriber, see section 5). "Background" A is a second VM because DOS is single-tasking.
  js-dos 6.22 (an earlier version of this Space) has no TCP/IP at all, which is why v86 is used now.

## 2. Data path

```
A.EXE/B.EXE (mTCP TCP/IP stack, DOS)
   -> packet driver NE2000.COM (INT 0x60, I/O 0xC000, IRQ 10)
   -> v86 emulated NE2000 (PCI)                              DHCP + DNS answered by v86 itself
   -> v86 "wisp" network adapter (terminates guest TCP)      DNS = DoH to cloudflare-dns.com (needs internet)
   -> WebSocket('wisp://mqttbridge.invalid/?m=A|B')          <- monkeypatched in bridge.js (Proxy on window.WebSocket)
   -> FakeWispServer (bridge.js): per TCP stream opens
      new WebSocket('wss://<broker>', 'mqtt')  (port must be 1883, else stream refused)
   -> public broker
```
* WISP frames used: 1 CONNECT, 2 DATA, 3 CONTINUE (flow-control credit, we hand out 128 on stream 0 and per DATA), 4 CLOSE.
* Data the guest sends before the WSS is open is queued; broker->guest data is cut into 1 KB frames.
* Both VMs use the same IP; they are isolated (wisp backend isolates guests).

## 3. Layout and how to build

See `README.md` for the tree. Build chain (`tools/build_all.sh`, a few seconds once deps are cached):

1. `tools/fetch_deps.sh` downloads v86 0.5.472 (npm), SeaBIOS/VGABIOS, FreeDOS 1.4 floppy, Crynwr NE2000.COM,
   mTCP 2026-10-10 (bin + src), Open Watcom **1.9** into `deps/`, and checks pinned SHA-256s (edit the hashes on purpose when upgrading).
2. `src/build.sh` -> `build/A.EXE`, `build/B.EXE` (16-bit DOS compact model, `-0 -mc -oh -os -s -zp2 -zpw -we`).
   mTCP sources are mirrored in lowercase first (Linux is case sensitive, mTCP is not).
3. `src/make_image.sh` -> `build/dosmqtt.img` (FreeDOS boot floppy, strips installer files, adds boot files + NE2000.COM + DHCP.EXE + A/B.EXE; ~1 MB stays free).
4. `src/build_assets.py` -> `dist/assets.b64.js` (v86.wasm, BIOSes, floppy as **gzip+base64 text**) and `dist/libv86.js`.
   Reason: HF static Spaces answer binary files with a CORS-less redirect; text files are served directly.
5. `build_all.sh` also copies `site/*` and `src/` into `dist/`. **`dist/` is exactly what gets uploaded to the Space.**

Rebuilt EXEs equal the deployed ones except 1 byte (mTCP embeds the build date `Oct 24 2026` vs `Oct 25 2026`).

`site/app.js` flow: unpack assets (`DecompressionStream`) -> start two `V86` instances -> poll the DOS text screens until
`TCP/IP is up` appears (printed by `FDAUTO.BAT`) -> enable the start button -> on start: type `SET MQTT_HOST/PORT/TOPIC`
in both machines, type `A` in machine A, wait for `subscribed to` on A's screen, then every 5 s type `B` in machine B
(skipped and logged if B's prompt `X:\>` is not the last screen line). Results are read **from the DOS screens**
(`emu.screen_adapter.get_text_screen()`), regex on `A rx:` / `A echo:` / `B sent:`; the log also shows bridge events.
A and B read host/port/topic from the DOS environment (`MQTT_HOST`, `MQTT_PORT` default 1883, `MQTT_TOPIC`) or from the command line `A <broker> <port> <topic>`.

## 4. Hard-won facts (do not rediscover these)

* **NE2000 is NOT at 0x300.** SeaBIOS re-maps the PCI card: I/O **0xC000, IRQ 10**. Load with `NE2000 0x60 10 0xC000`.
  (Read from `emu.v86.cpu.devices.pci.device_spaces`: BAR0 `c001`, dword 0x3c `10a`.) Packet interrupt 0x60 must match `PACKETINT 0x60` in `MTCP.CFG`.
* DHCP.EXE rewrites `A:\MTCP.CFG` on the floppy (writable). `MTCPCFG=A:\MTCP.CFG` is set in `FDAUTO.BAT`.
* **Use Open Watcom 1.9, not 2.0 beta.** EXEs from 2.0 beta end with `End: heap is corrupted!` (mTCP's `_heapchk`) even for an empty program
  (`_heapchk` returned 3). With 1.9 it returns 0. `-we` (warnings are errors) bites on struct padding (W880): order fields so no padding is needed.
* mTCP needs `#include "global.cfg"` lowercase in `a.cfg/b.cfg` on Linux; the configs are copies of mTCP's `SAMPLE.CFG` with `TCP_MAX_SOCKETS 2`.
* **Typing into DOS:** `emu.keyboard_send_text(text, 25)` (25 ms per char). Without the delay FreeCOM drops characters
  (`SET PATH=...` came out as `FRESET MTCPCFG=...`). Give each command >= 250 ms before the next.
* **Screen scaling:** in text mode v86 draws a DOM text layer (80x25, 14 px monospace), not the canvas. `screen_set_scale()` and
  `use_graphical_text` turned the screen black/cropped. `fitScreen()` in `app.js` scales whichever layer is visible with a CSS `transform`.
* **v86's fake TCP never retransmits and ignores the guest's receive window.** It sends 1460-byte segments one after another
  (waiting only for the ACK of the previous one). If the guest has no room for a segment it drops it, never ACKs, and the connection
  hangs for good (seen in `emu.v86.network_adapter.tcp_conn[...]`: `pending: true`, `sendBuf` > 0). With a 2 KB receive buffer
  G.EXE hung about 1 time in 9 answers (and once for 167 s); with `setRecvBuffer(8192)` and 1 KB reads: 30/30 answers, 0 hangs.
  mTCP allows 512..16384 bytes. Any DOS program that receives more than a few hundred bytes must use a big buffer and drain it fast.
  A.EXE/B.EXE still use 1024 (`RECV_BUFFER_SIZE` in `mqtt.h`): fine for their tiny MQTT packets, a risk if payloads ever grow.
* B uses a pseudo-random local TCP port per run; DNS answers rotate between several A records (that is normal).
* DOS clock is **UTC**; mTCP timer granularity is the 18.2 Hz BIOS tick (55 ms), so "1 s" echo reads 0.8-1.2 s from the host.
* **Broker quality differs a lot** (CONNECT -> CONNACK over WSS from headless Chromium in this sandbox; 12 attempts per broker, HiveMQ also 30):
  `broker.emqx.io:8084` 12/12 fast (~560 ms); `test.mosquitto.org:8081` 12/12 fast (~390 ms);
  `broker.hivemq.com:8884` only 7/12 (and 19/30 in another run) fast, the rest took ~10 s or never answered (an idle WS without CONNECT is closed at ~10 s);
  `mqtt.eclipseprojects.io/mqtt` 0/12. Default is EMQX. B's 10 s CONNACK timeout then shows `MQTT: CONNECT refused or timed out`.
* A sends PINGREQ every 30 s (keepalive 60 s) and keeps <= 8 pending echoes. A has no reconnect: restart the VMs ("停止並重新開機").
* The "stop" button can't send Ctrl-Break; it reboots both VMs (`vm.restart()`), then waits for `TCP/IP is up` again.

## 5. Testing (Playwright + independent MQTT peer)

`pip install playwright && playwright install chromium`; Chromium flags `--no-sandbox --disable-gpu --disable-dev-shm-usage`.
* `python3 tests/ask.py <gradio.html url> "question" shot.png`: asks one question on the Gradio page and prints DOS/bridge/check results.
* `tests/serve_harness.sh`: bare v86 page on :8801 (verbose bridge logging) for manual DOS poking via `drive2.py`
  (`session([(cmd, wait_s), ...])` prints the DOS screen + bridge log).
* `python3 -m http.server 8802 --directory dist`, then
  `python3 tests/full.py http://127.0.0.1:8802/ <topic> <seconds> <shot.png>` runs the whole page (also works on the live URL).
* **Always cross-check with something that is not the page**: `python3 tests/sub.py <topic> <secs> broker.emqx.io` (raw TCP 1883 subscriber)
  must print every `Hello from DOS B at ...`; `python3 tests/pub.py <topic> <delay> <n>` publishes to test A on its own.
  Use the same topic in the page (`full.py` fills it) and the same broker host (default EMQX).
* Results when this was written: local 8/8 matched, 8 messages seen by the independent subscriber; **live URL** 9 B runs, 9 sent, A received 9 and echoed 8
  (the 9th echo was still within its 1 s delay when sampled); the independent subscriber saw all 9.
* Use `kill <pid>`; `pkill -f <name>` kills your own shell.

## 6. Deploy rules (the owner was angry about breakage before)

1. **Only write to `HelloSun/dosmqtt`.** Never touch `HelloSun/c_dos`, `flare-gamet2` or any other Space (reading `c_dos` for reference is fine).
2. **Never delete remote files.** Upload with `HfApi.create_commit` + `CommitOperationAdd` only; compare `list_repo_files` before/after and report `deleted: []`.
3. Account name is `HelloSun` (case matters: `hellosun/...` gave 403 "no rights to create a space"). Auth: read the token from
   `/root/.cache/huggingface/token`. Env `HF_TOKEN` is invalid. A token was once pasted into chat; it is exposed and
   lacks create rights: advise revoking it. **Never write a token into any file.** `hf` CLI is broken here (typer/click mismatch): use the Python API.
4. After upload HF needs ~1 min to publish; verify with `curl` for a new symbol and then run `tests/full.py` on the live URL.

```python
from huggingface_hub import HfApi, CommitOperationAdd
from pathlib import Path
api = HfApi(token=Path("/root/.cache/huggingface/token").read_text().strip()); repo = "HelloSun/dosmqtt"
before = set(api.list_repo_files(repo, repo_type="space"))
ops = [CommitOperationAdd(path_in_repo=str(p.relative_to(root)), path_or_fileobj=str(p)) for p in files]
api.create_commit(repo_id=repo, repo_type="space", operations=ops, commit_message="...")
assert not (before - set(api.list_repo_files(repo, repo_type="space")))
```
Space file set: web files (both pages) + `assets.b64.js` + `gradio-img.b64.js` + `libv86.js` + `src/` + licenses, and `dev/` (this package, `dev/dosmqtt-dev.zip`, `dev/HANDOFF.md`).
Unused leftovers from the old js-dos version, **deliberately not deleted**: `js-dos.js`, `emu.b64.js`, `dos.b64.js`, `LICENSE-js-dos.txt`.

## 9. DOS asks a Gradio Space (`gradio.html`)

Verified with `tencent/Hy3` (public, Gradio 6.12, endpoint `/chat`). Live page: https://hellosun-dosmqtt.static.hf.space/gradio.html

```
G.EXE (DOS, mTCP, plain HTTP/1.0 on port 80)
   -> NE2000 -> v86 wisp adapter -> gradio-bridge.js (FakeWispServer)
   -> fetch('https://<host>.hf.space/gradio_api/...')      (CORS: Spaces echo the caller's Origin, incl. localhost)
```
* `G.EXE` (`src/g.cpp`): `G <question>`; host from env `GRADIO_HOST` (default `tencent-hy3.hf.space`). It does the documented Gradio calls itself:
  `POST /gradio_api/call/chat` with `{"data":[question, system_prompt, [], "no_think", null, 0, 0, null, ""]}` -> `{"event_id":...}`, then
  `GET /gradio_api/call/chat/<event_id>` (server-sent events: N x `generating`, then `complete`). It prints one `.` per `generating` frame,
  decodes the first string of the `complete` frame (`[[content, reasoning, tool_calls, history]]`, `\uXXXX` -> ASCII, other chars -> `?`) and word-wraps it as `A> ...`.
* **All 9 parameters are required** on the raw `/call` API. With only `message` the stream answers `event: error / data: null` (the official `gradio_client` fills the defaults).
* The system prompt `Answer in plain ASCII English in at most 40 words.` keeps replies small (whole stream ~3 KB, complete frame ~0.5 KB) and ASCII, because FreeDOS has no CJK font.
* `site/gradio-bridge.js`: parses one HTTP request per TCP stream (port 80 only), relays it with `fetch` and answers `HTTP/1.0 <status>` with `Content-Length`,
  then closes the stream. **Allow-list**: host must match `*.hf.space`, path must start with `/gradio_api/`, GET/POST only, body <= 8 KB, no credentials added, 150 s timeout.
  The bridge also parses the SSE text to log `Gradio answer: ...` (captured for the self-check).
* `site/gradio.html` + `gradio.js`: types `SET GRADIO_HOST=...`, `CLS`, `G <question>` into DOS, waits for the prompt `A:\>`, then **compares the answer
  printed on the DOS screen with the answer the bridge captured from Gradio** (tag `DOS = GRADIO`). `window.dosVm` exposes the v86 instance for tests.
* Assets: the page reuses `assets.b64.js` (wasm/BIOS) and adds `gradio-img.b64.js` (its own floppy, `build/gradio.img`, boot file `src/dos/FDAUTO_G.BAT`, prints `TCP/IP is up`).
  `bridge.js`/`app.js`/`index.html` (MQTT page) were deliberately left untouched; `gradio-bridge.js` duplicates ~60 lines of WISP framing. A shared module would be a clean refactor.
* Limits: public Spaces only (no token is ever sent), non-ASCII becomes `?`, one reply is returned when the stream ends (DOS shows dots while it streams),
  the question is sent to the chosen Space, so never type secrets. A Space whose `/chat` has other parameters needs `src/g.cpp` adapted (look at `/gradio_api/info`).
* Results: 30/30 consecutive questions answered with the 8 KB buffer, each `DOS = GRADIO`; `17 times 23` -> DOS printed `391`.

## 7. Licensing

A.EXE/B.EXE link mTCP (GPL-3.0-or-later), so our DOS sources, scripts and web code are GPL-3.0-or-later and the mTCP source archive
is shipped in `third_party/`. FreeDOS (GPL-2.0), v86 (BSD-2-Clause), SeaBIOS (LGPL-3.0) and the Crynwr packet driver
(source must stay available: Crynwr 11.x sources are at https://archive.org/details/PKTD11A_ZIP) are listed in `site/NOTICE.md`.
Open Watcom is only a build tool and is not redistributed.

## 8. Known limits / ideas

* QoS 0 only, no TLS and no auth in the DOS clients (TLS is done by the browser's WSS). Topic/payload are small (payload <= 95 chars in A).
* B is started by the page typing `B`; if the previous run is still going the tick is skipped (log shows it). A real in-DOS loop would need a `B.EXE -loop` mode.
* Only port 1883 is bridged. Other ports are refused on purpose.
* The page needs `DecompressionStream`, WebAssembly and internet (DoH + broker). No offline mode.
* Ideas: reconnect in A, in-DOS 5 s loop for B, MQTT QoS 1, mobile on-screen keyboard, a selectable second bridge (own WISP server) for brokers without WSS.