Fix loop diagram alignment
Browse files
README.md
CHANGED
|
@@ -1,3 +1,4 @@
|
|
|
|
|
| 1 |
---
|
| 2 |
title: Tiny Computer Agent
|
| 3 |
emoji: π±οΈ
|
|
@@ -27,13 +28,14 @@ same machine it is controlling.
|
|
| 27 |
## The idea, in one loop
|
| 28 |
|
| 29 |
```
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
|
|
|
| 37 |
```
|
| 38 |
|
| 39 |
1. **Observe** β grab the screen, downscale it, optionally attach OCR text or the
|
|
@@ -51,43 +53,43 @@ The harness does the heavy lifting, so the model does not have to be clever:
|
|
| 51 |
| Job | Who does it |
|
| 52 |
|---|---|
|
| 53 |
| Finding *where* a button is | OCR / accessibility tree / element index β not the model |
|
| 54 |
-
| Choosing *what* to do | tiny VLM over a **closed action menu** (click, type, key, scroll, wait, done) |
|
| 55 |
-
| Knowing
|
| 56 |
-
| Recovering from a bad step | retry policy in `loop.py`, not
|
| 57 |
|
| 58 |
-
That keeps the agent offline,
|
| 59 |
-
|
| 60 |
|
| 61 |
## Candidate backends
|
| 62 |
|
| 63 |
-
Pluggable behind one interface; nothing here
|
| 64 |
|
| 65 |
-
- **Moondream 2** / **SmolVLM-256M / 500M** β
|
| 66 |
- **Qwen2-VL-2B-Instruct**, **Florence-2-base** β stronger grounding, still tiny
|
| 67 |
- **OCR + text-only planner** β no VLM at all; the cheapest credible baseline
|
| 68 |
-
- **mock brain** β deterministic scripted actions, used
|
| 69 |
|
| 70 |
## Intended layout
|
| 71 |
|
| 72 |
```
|
| 73 |
tiny-computer-agent/
|
| 74 |
-
ββ README.md
|
| 75 |
-
ββ tiny_computer_agent.py # marimo notebook: drive/watch the loop
|
| 76 |
-
ββ src/tiny_computer_agent/
|
| 77 |
-
β
|
| 78 |
-
β
|
| 79 |
-
β
|
| 80 |
-
β
|
| 81 |
-
β
|
| 82 |
-
ββ LICENSE
|
| 83 |
```
|
| 84 |
|
| 85 |
## Safety, because this thing has your mouse
|
| 86 |
|
| 87 |
-
- Run it in a **VM, container, or throwaway desktop account**
|
| 88 |
- `--dry-run` logs every action without executing it, so you can read the plan first.
|
| 89 |
- Deny-list for shells, terminals, payment flows and password managers.
|
| 90 |
-
-
|
| 91 |
- **Local-first, for real:** screen pixels and keystrokes never leave the machine.
|
| 92 |
|
| 93 |
## Roadmap
|
|
@@ -96,7 +98,7 @@ tiny-computer-agent/
|
|
| 96 |
- [ ] OCR / AT-SPI element index, and click-by-element instead of click-by-pixel
|
| 97 |
- [ ] tiny-VLM grounding backend (transformers + llama.cpp GGUF)
|
| 98 |
- [ ] marimo UI: live screen preview, action log, pause / resume / step
|
| 99 |
-
- [ ] append-only run traces so failed episodes can be replayed and scored
|
| 100 |
|
| 101 |
## License
|
| 102 |
|
|
|
|
| 1 |
+
|
| 2 |
---
|
| 3 |
title: Tiny Computer Agent
|
| 4 |
emoji: π±οΈ
|
|
|
|
| 28 |
## The idea, in one loop
|
| 29 |
|
| 30 |
```
|
| 31 |
+
observe think act verify
|
| 32 |
+
+---------+ +---------+ +---------+ +---------+
|
| 33 |
+
| screen |--->| tiny |--->| mouse / |--->| next |
|
| 34 |
+
| frame | | model | | keyboard| | frame |
|
| 35 |
+
+---------+ +---------+ +---------+ +----+----+
|
| 36 |
+
|
|
| 37 |
+
<-------------------------------------------------+
|
| 38 |
+
repeat until done, or the step budget runs out
|
| 39 |
```
|
| 40 |
|
| 41 |
1. **Observe** β grab the screen, downscale it, optionally attach OCR text or the
|
|
|
|
| 53 |
| Job | Who does it |
|
| 54 |
|---|---|
|
| 55 |
| Finding *where* a button is | OCR / accessibility tree / element index β not the model |
|
| 56 |
+
| Choosing *what* to do | a tiny VLM over a **closed action menu** (click, type, key, scroll, wait, done) |
|
| 57 |
+
| Knowing whether it worked | the next frame, plus harness-side checks |
|
| 58 |
+
| Recovering from a bad step | the retry policy in `loop.py`, not the weights |
|
| 59 |
|
| 60 |
+
That keeps the agent offline, fast to start, and small enough that a laptop GPU
|
| 61 |
+
(or plain CPU) can run it on the same box it is controlling.
|
| 62 |
|
| 63 |
## Candidate backends
|
| 64 |
|
| 65 |
+
Pluggable behind one interface; nothing here needs a network call at run time.
|
| 66 |
|
| 67 |
+
- **Moondream 2** / **SmolVLM-256M / 500M** β a few hundred MB, safe default
|
| 68 |
- **Qwen2-VL-2B-Instruct**, **Florence-2-base** β stronger grounding, still tiny
|
| 69 |
- **OCR + text-only planner** β no VLM at all; the cheapest credible baseline
|
| 70 |
+
- **mock brain** β deterministic scripted actions, used to test the harness itself
|
| 71 |
|
| 72 |
## Intended layout
|
| 73 |
|
| 74 |
```
|
| 75 |
tiny-computer-agent/
|
| 76 |
+
βββ README.md
|
| 77 |
+
βββ tiny_computer_agent.py # marimo notebook: drive / watch the loop
|
| 78 |
+
βββ src/tiny_computer_agent/
|
| 79 |
+
β βββ observe.py # screen capture + OCR / accessibility tree
|
| 80 |
+
β βββ brain.py # model backends behind one interface
|
| 81 |
+
β βββ act.py # action vocabulary + pyautogui / xdotool executor
|
| 82 |
+
β βββ loop.py # observe β think β act β verify
|
| 83 |
+
β βββ safety.py # dry-run, allow/deny lists, kill switch, step budget
|
| 84 |
+
βββ LICENSE
|
| 85 |
```
|
| 86 |
|
| 87 |
## Safety, because this thing has your mouse
|
| 88 |
|
| 89 |
+
- Run it in a **VM, container, or throwaway desktop account** β not your real session.
|
| 90 |
- `--dry-run` logs every action without executing it, so you can read the plan first.
|
| 91 |
- Deny-list for shells, terminals, payment flows and password managers.
|
| 92 |
+
- A step budget plus a global hotkey kill switch before anything touches the pointer.
|
| 93 |
- **Local-first, for real:** screen pixels and keystrokes never leave the machine.
|
| 94 |
|
| 95 |
## Roadmap
|
|
|
|
| 98 |
- [ ] OCR / AT-SPI element index, and click-by-element instead of click-by-pixel
|
| 99 |
- [ ] tiny-VLM grounding backend (transformers + llama.cpp GGUF)
|
| 100 |
- [ ] marimo UI: live screen preview, action log, pause / resume / step
|
| 101 |
+
- [ ] append-only run traces, so failed episodes can be replayed and scored
|
| 102 |
|
| 103 |
## License
|
| 104 |
|