Desktop harness — first slice

Manual SSH and X11 controls. There is no model loop, persistent conversation, scheduler, or Jev integration yet. The package name is a placeholder.

On berlin (or your development computer)

Requires Node.js 22.13+ (for built-in SQLite) and an SSH client. Configure an SSH alias home for the VM's desktop user, with key authentication. Connect manually first to verify and save its host key:

ssh home

The CLI requires known host keys and noninteractive authentication. It does not bypass SSH verification.

npm install
npm run build
npm run home -- install home

Inside home

Install dependencies:

sudo pacman -S --needed python python-pillow xdotool xclip

From a terminal inside the logged-in XFCE desktop, run:

python3 ~/.local/lib/desktop-harness/desktop.py --register-session

This records the current graphical session environment for SSH requests. Installation also creates ~/.config/autostart/desktop-harness-session.desktop, so subsequent XFCE logins refresh registration automatically. Re-running install updates both the helper and its autostart entry. No periodic refresh is needed within the same session. Keep the desktop logged in and unlocked for these initial checks.

To test automatic registration, log out and back into XFCE, then capture from berlin without running the registration command manually. The helper assumes one graphical session for this account; it does not log in, unlock the screen, or choose between concurrent sessions. To disable automatic registration, remove the autostart file.

Try capturing

From berlin:

npm run home -- capture home artifacts/first.png

Open the PNG. Check dimensions and that it shows the actual desktop rather than a blank or different display.

Try input

Open Mousepad or another ordinary GUI text editor manually, focus a blank document, and create actions.json on berlin:

{
    "actions": [
        { "type": "text", "text": "Hello from berlin!\nUnicode: café — こんにちは" },
        { "type": "wait", "milliseconds": 300 }
    ]
}

npm run home -- act home actions.json artifacts/typed.png

Verify the text and screenshot. Text insertion replaces the clipboard and pastes with Ctrl+V; this is for ordinary GUI text fields, not terminals (which often need Ctrl+Shift+V). For shell work use the shell command below.

Other action examples:

{
    "expectedSize": [1280, 800],
    "actions": [
        { "type": "click", "x": 300, "y": 200, "button": "left" },
        { "type": "keys", "keys": ["ctrl", "a"] },
        { "type": "scroll", "direction": "down", "steps": 2 }
    ]
}

Use coordinates from your own screenshot, not these example coordinates. expectedSize is optional and rejects input if the screen size changed. Supported actions are click, scroll, keys, text, and wait. Drag is deferred. Actions are serialized by a VM-side lock; avoid manual interaction while a request runs. The full request is validated before any actions execute, but runtime failures can still leave partial effects.

Try shell execution

npm run home -- shell home 'printf "hello\n"; uname -a'
npm run home -- shell home 'sleep 10' 2

The second command should time out. GNU timeout runs inside the VM and sends TERM, then KILL after two seconds. This is not a sandbox: commands have the desktop user's permissions, and processes that deliberately detach can escape the timeout. Output is limited to 32 MiB across stdout/stderr; exceeding it disconnects SSH and reports an unknown outcome. Commands must be one quoted local argument. Default timeout is 30 seconds, maximum 300.

Local checks

npm run check
npm test
python3 -m unittest discover -s test -p 'test_*.py'

Desktop dependencies are imported only when operating the display, so validation tests can run without X11 or Pillow.

Configuration and SQLite

cp .env.example .env
npm run build
npm run setup:check

.env.example contains berlin's llama.cpp origin and model ID. Edit .env for your installation. The origin excludes /v1; future model requests will append the API path. DATABASE_PATH defaults to ./data/token.sqlite, relative to the process working directory. .env and database files are ignored by Git. The check validates configuration and opens/checks SQLite; it does not contact the model or operate the desktop.

src/database.ts exports a general openDatabase() connection using Node's built-in SQLite, with foreign keys, WAL, and a five-second busy timeout. The caller owns the connection and must close it. Use one connection in the eventual supervisor and pass it to domain modules; it is not tied to conversations or logs. Append numbered SQL migrations to migrations as actual schemas are introduced (projects, schedules, messages, etc.). Applied migration SQL is recorded and checked against subsequent builds. Pending migrations run transactionally; incompatible history fails rather than silently changing existing data. There is intentionally no speculative domain schema or ORM yet. Back up live databases using a SQLite-aware backup mechanism, not by copying only the main file while WAL is active.

Next milestone

Once capture, Unicode insertion, key combinations, and shell timeout work on the real VM, add the llama.cpp adapter, durable tool-call records, and a single model/tool loop. Wake/sleep and restart recovery follow. No SSH connection or real graphical session is available in the development sandbox, so those checks must be run on your setup.

5.5 KiB Raw Blame History