Desktop harness — first slice

Manual SSH and X11 controls. There is no model loop, persistent conversation, scheduler, or Jev integration yet. The package name is a placeholder.

On berlin (or your development computer)

Requires Node.js 22+ and an SSH client. Configure an SSH alias home for the VM's desktop user, with key authentication. Connect manually first to verify and save its host key:

ssh home

The CLI requires known host keys and noninteractive authentication. It does not bypass SSH verification.

npm install
npm run build
npm run home -- install home

Inside home

Install dependencies:

sudo pacman -S --needed python python-pillow xdotool xclip

From a terminal inside the logged-in XFCE desktop, run:

python3 ~/.local/lib/desktop-harness/desktop.py --register-session

This records the current graphical session environment for SSH requests. Installation also creates ~/.config/autostart/desktop-harness-session.desktop, so subsequent XFCE logins refresh registration automatically. Re-running install updates both the helper and its autostart entry. No periodic refresh is needed within the same session. Keep the desktop logged in and unlocked for these initial checks.

To test automatic registration, log out and back into XFCE, then capture from berlin without running the registration command manually. The helper assumes one graphical session for this account; it does not log in, unlock the screen, or choose between concurrent sessions. To disable automatic registration, remove the autostart file.

Try capturing

From berlin:

npm run home -- capture home artifacts/first.png

Open the PNG. Check dimensions and that it shows the actual desktop rather than a blank or different display.

Try input

Open Mousepad or another ordinary GUI text editor manually, focus a blank document, and create actions.json on berlin:

{
    "actions": [
        { "type": "text", "text": "Hello from berlin!\nUnicode: café — こんにちは" },
        { "type": "wait", "milliseconds": 300 }
    ]
}

npm run home -- act home actions.json artifacts/typed.png

Verify the text and screenshot. Text insertion replaces the clipboard and pastes with Ctrl+V; this is for ordinary GUI text fields, not terminals (which often need Ctrl+Shift+V). For shell work use the shell command below.

Other action examples:

{
    "expectedSize": [1280, 800],
    "actions": [
        { "type": "click", "x": 300, "y": 200, "button": "left" },
        { "type": "keys", "keys": ["ctrl", "a"] },
        { "type": "scroll", "direction": "down", "steps": 2 }
    ]
}

Use coordinates from your own screenshot, not these example coordinates. expectedSize is optional and rejects input if the screen size changed. Supported actions are click, scroll, keys, text, and wait. Drag is deferred. Actions are serialized by a VM-side lock; avoid manual interaction while a request runs. The full request is validated before any actions execute, but runtime failures can still leave partial effects.

Try shell execution

npm run home -- shell home 'printf "hello\n"; uname -a'
npm run home -- shell home 'sleep 10' 2

The second command should time out. GNU timeout runs inside the VM and sends TERM, then KILL after two seconds. This is not a sandbox: commands have the desktop user's permissions, and processes that deliberately detach can escape the timeout. Output is limited to 32 MiB across stdout/stderr; exceeding it disconnects SSH and reports an unknown outcome. Commands must be one quoted local argument. Default timeout is 30 seconds, maximum 300.

Local checks

npm run check
npm test
python3 -m unittest discover -s test -p 'test_*.py'

Desktop dependencies are imported only when operating the display, so validation tests can run without X11 or Pillow.

Next milestone

Once capture, Unicode insertion, key combinations, and shell timeout work on the real VM, add the llama.cpp adapter, durable tool-call records, and a single model/tool loop. Wake/sleep and restart recovery follow. No SSH connection or real graphical session is available in the development sandbox, so those checks must be run on your setup.

4.2 KiB Raw Blame History