Initial commit

README.md

@@ -0,0 +1,104 @@
+# Desktop harness — first slice
+
+Manual SSH and X11 controls. There is **no model loop, persistent conversation, scheduler, or Jev integration yet**. The package name is a placeholder.
+
+## On berlin (or your development computer)
+
+Requires Node.js 22+ and an SSH client. Configure an SSH alias `home` for the VM's desktop user, with key authentication. Connect manually first to verify and save its host key:
+
+```sh
+ssh home
+```
+
+The CLI requires known host keys and noninteractive authentication. It does not bypass SSH verification.
+
+```sh
+npm install
+npm run build
+npm run home -- install home
+```
+
+## Inside home
+
+Install dependencies:
+
+```sh
+sudo pacman -S --needed python python-pillow xdotool xclip
+```
+
+From a terminal **inside the logged-in XFCE desktop**, run:
+
+```sh
+python3 ~/.local/lib/desktop-harness/desktop.py --register-session
+```
+
+This records the current graphical session environment for SSH requests. Installation also creates `~/.config/autostart/desktop-harness-session.desktop`, so subsequent XFCE logins refresh registration automatically. Re-running `install` updates both the helper and its autostart entry. No periodic refresh is needed within the same session. Keep the desktop logged in and unlocked for these initial checks.
+
+To test automatic registration, log out and back into XFCE, then capture from berlin without running the registration command manually. The helper assumes one graphical session for this account; it does not log in, unlock the screen, or choose between concurrent sessions. To disable automatic registration, remove the autostart file.
+
+## Try capturing
+
+From berlin:
+
+```sh
+npm run home -- capture home artifacts/first.png
+```
+
+Open the PNG. Check dimensions and that it shows the actual desktop rather than a blank or different display.
+
+## Try input
+
+Open Mousepad or another ordinary GUI text editor manually, focus a blank document, and create `actions.json` on berlin:
+
+```json
+{
+    "actions": [
+        { "type": "text", "text": "Hello from berlin!\nUnicode: café — こんにちは" },
+        { "type": "wait", "milliseconds": 300 }
+    ]
+}
+```
+
+```sh
+npm run home -- act home actions.json artifacts/typed.png
+```
+
+Verify the text and screenshot. Text insertion **replaces the clipboard** and pastes with Ctrl+V; this is for ordinary GUI text fields, not terminals (which often need Ctrl+Shift+V). For shell work use the shell command below.
+
+Other action examples:
+
+```json
+{
+    "expectedSize": [1280, 800],
+    "actions": [
+        { "type": "click", "x": 300, "y": 200, "button": "left" },
+        { "type": "keys", "keys": ["ctrl", "a"] },
+        { "type": "scroll", "direction": "down", "steps": 2 }
+    ]
+}
+```
+
+Use coordinates from your own screenshot, not these example coordinates. `expectedSize` is optional and rejects input if the screen size changed. Supported actions are click, scroll, keys, text, and wait. Drag is deferred. Actions are serialized by a VM-side lock; avoid manual interaction while a request runs. The full request is validated before any actions execute, but runtime failures can still leave partial effects.
+
+## Try shell execution
+
+```sh
+npm run home -- shell home 'printf "hello\n"; uname -a'
+npm run home -- shell home 'sleep 10' 2
+```
+
+The second command should time out. GNU `timeout` runs inside the VM and sends TERM, then KILL after two seconds. This is not a sandbox: commands have the desktop user's permissions, and processes that deliberately detach can escape the timeout. Output is limited to 32 MiB across stdout/stderr; exceeding it disconnects SSH and reports an unknown outcome. Commands must be one quoted local argument. Default timeout is 30 seconds, maximum 300.
+
+## Local checks
+
+```sh
+npm run check
+npm test
+python3 -m unittest discover -s test -p 'test_*.py'
+```
+
+Desktop dependencies are imported only when operating the display, so validation tests can run without X11 or Pillow.
+
+## Next milestone
+
+Once capture, Unicode insertion, key combinations, and shell timeout work on the real VM, add the llama.cpp adapter, durable tool-call records, and a single model/tool loop. Wake/sleep and restart recovery follow. No SSH connection or real graphical session is available in the development sandbox, so those checks must be run on your setup.