hkblockchaintechnology commited on
Commit
3fcc46c
Β·
verified Β·
1 Parent(s): 1d1dbda

Fix loop diagram alignment

Browse files
Files changed (1) hide show
  1. README.md +29 -27
README.md CHANGED
@@ -1,3 +1,4 @@
 
1
  ---
2
  title: Tiny Computer Agent
3
  emoji: πŸ–±οΈ
@@ -27,13 +28,14 @@ same machine it is controlling.
27
  ## The idea, in one loop
28
 
29
  ```
30
- observe think act verify
31
- β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”
32
- β”‚ screen │──────▢│ tiny │─────▢│ mouse / │─────▢│ next β”‚
33
- β”‚ frame β”‚ β”‚ model β”‚ β”‚ keyboardβ”‚ β”‚ frame β”‚
34
- β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”˜
35
- β”‚
36
- β—€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
 
37
  ```
38
 
39
  1. **Observe** β€” grab the screen, downscale it, optionally attach OCR text or the
@@ -51,43 +53,43 @@ The harness does the heavy lifting, so the model does not have to be clever:
51
  | Job | Who does it |
52
  |---|---|
53
  | Finding *where* a button is | OCR / accessibility tree / element index β€” not the model |
54
- | Choosing *what* to do | tiny VLM over a **closed action menu** (click, type, key, scroll, wait, done) |
55
- | Knowing if it worked | the next screenshot, plus harness-side checks |
56
- | Recovering from a bad step | retry policy in `loop.py`, not in the weights |
57
 
58
- That keeps the agent offline, instant to start, and small enough that a laptop
59
- GPU (or plain CPU) is enough to run it β€” on the same box it is controlling.
60
 
61
  ## Candidate backends
62
 
63
- Pluggable behind one interface; nothing here requires a network call at run time.
64
 
65
- - **Moondream 2** / **SmolVLM-256M / 500M** β€” ~hundreds of MB, safe default
66
  - **Qwen2-VL-2B-Instruct**, **Florence-2-base** β€” stronger grounding, still tiny
67
  - **OCR + text-only planner** β€” no VLM at all; the cheapest credible baseline
68
- - **mock brain** β€” deterministic scripted actions, used for harness tests
69
 
70
  ## Intended layout
71
 
72
  ```
73
  tiny-computer-agent/
74
- β”œβ”€ README.md
75
- β”œβ”€ tiny_computer_agent.py # marimo notebook: drive/watch the loop
76
- β”œβ”€ src/tiny_computer_agent/
77
- β”‚ β”œβ”€ observe.py # screen capture + OCR / accessibility tree
78
- β”‚ β”œβ”€ brain.py # model backends behind one interface
79
- β”‚ β”œβ”€ act.py # action vocabulary + pyautogui / xdotool executor
80
- β”‚ β”œβ”€ loop.py # observe β†’ think β†’ act β†’ verify
81
- β”‚ └─ safety.py # dry-run, allow/deny lists, kill switch, step budget
82
- └─ LICENSE
83
  ```
84
 
85
  ## Safety, because this thing has your mouse
86
 
87
- - Run it in a **VM, container, or throwaway desktop account**, never your real session.
88
  - `--dry-run` logs every action without executing it, so you can read the plan first.
89
  - Deny-list for shells, terminals, payment flows and password managers.
90
- - Step budget + a global hotkey kill switch before anything touches the pointer.
91
  - **Local-first, for real:** screen pixels and keystrokes never leave the machine.
92
 
93
  ## Roadmap
@@ -96,7 +98,7 @@ tiny-computer-agent/
96
  - [ ] OCR / AT-SPI element index, and click-by-element instead of click-by-pixel
97
  - [ ] tiny-VLM grounding backend (transformers + llama.cpp GGUF)
98
  - [ ] marimo UI: live screen preview, action log, pause / resume / step
99
- - [ ] append-only run traces so failed episodes can be replayed and scored
100
 
101
  ## License
102
 
 
1
+
2
  ---
3
  title: Tiny Computer Agent
4
  emoji: πŸ–±οΈ
 
28
  ## The idea, in one loop
29
 
30
  ```
31
+ observe think act verify
32
+ +---------+ +---------+ +---------+ +---------+
33
+ | screen |--->| tiny |--->| mouse / |--->| next |
34
+ | frame | | model | | keyboard| | frame |
35
+ +---------+ +---------+ +---------+ +----+----+
36
+ |
37
+ <-------------------------------------------------+
38
+ repeat until done, or the step budget runs out
39
  ```
40
 
41
  1. **Observe** β€” grab the screen, downscale it, optionally attach OCR text or the
 
53
  | Job | Who does it |
54
  |---|---|
55
  | Finding *where* a button is | OCR / accessibility tree / element index β€” not the model |
56
+ | Choosing *what* to do | a tiny VLM over a **closed action menu** (click, type, key, scroll, wait, done) |
57
+ | Knowing whether it worked | the next frame, plus harness-side checks |
58
+ | Recovering from a bad step | the retry policy in `loop.py`, not the weights |
59
 
60
+ That keeps the agent offline, fast to start, and small enough that a laptop GPU
61
+ (or plain CPU) can run it on the same box it is controlling.
62
 
63
  ## Candidate backends
64
 
65
+ Pluggable behind one interface; nothing here needs a network call at run time.
66
 
67
+ - **Moondream 2** / **SmolVLM-256M / 500M** β€” a few hundred MB, safe default
68
  - **Qwen2-VL-2B-Instruct**, **Florence-2-base** β€” stronger grounding, still tiny
69
  - **OCR + text-only planner** β€” no VLM at all; the cheapest credible baseline
70
+ - **mock brain** β€” deterministic scripted actions, used to test the harness itself
71
 
72
  ## Intended layout
73
 
74
  ```
75
  tiny-computer-agent/
76
+ β”œβ”€β”€ README.md
77
+ β”œβ”€β”€ tiny_computer_agent.py # marimo notebook: drive / watch the loop
78
+ β”œβ”€β”€ src/tiny_computer_agent/
79
+ β”‚ β”œβ”€β”€ observe.py # screen capture + OCR / accessibility tree
80
+ β”‚ β”œβ”€β”€ brain.py # model backends behind one interface
81
+ β”‚ β”œβ”€β”€ act.py # action vocabulary + pyautogui / xdotool executor
82
+ β”‚ β”œβ”€β”€ loop.py # observe β†’ think β†’ act β†’ verify
83
+ β”‚ └── safety.py # dry-run, allow/deny lists, kill switch, step budget
84
+ └── LICENSE
85
  ```
86
 
87
  ## Safety, because this thing has your mouse
88
 
89
+ - Run it in a **VM, container, or throwaway desktop account** β€” not your real session.
90
  - `--dry-run` logs every action without executing it, so you can read the plan first.
91
  - Deny-list for shells, terminals, payment flows and password managers.
92
+ - A step budget plus a global hotkey kill switch before anything touches the pointer.
93
  - **Local-first, for real:** screen pixels and keystrokes never leave the machine.
94
 
95
  ## Roadmap
 
98
  - [ ] OCR / AT-SPI element index, and click-by-element instead of click-by-pixel
99
  - [ ] tiny-VLM grounding backend (transformers + llama.cpp GGUF)
100
  - [ ] marimo UI: live screen preview, action log, pause / resume / step
101
+ - [ ] append-only run traces, so failed episodes can be replayed and scored
102
 
103
  ## License
104