token/README.md

Desktop harness — first slice

SSH and X11 controls plus a bounded llama.cpp tool loop. Conversations and tool operations are recorded in SQLite. Sleep/wake, automatic resumption, and Jev integration are not implemented. The package name is a placeholder.

On berlin (or your development computer)

Requires Node.js 22.13+ (for built-in SQLite) and an SSH client. Configure an SSH alias home for the VM's desktop user, with key authentication. Connect manually first to verify and save its host key:

ssh home

The CLI requires known host keys and noninteractive authentication. It does not bypass SSH verification.

npm install
npm run build
npm run home -- install home

Inside home

Install dependencies:

sudo pacman -S --needed python python-pillow xdotool xclip

From a terminal inside the logged-in XFCE desktop, run:

python3 ~/.local/lib/desktop-harness/desktop.py --register-session

This records the current graphical session environment for SSH requests. Installation also creates ~/.config/autostart/desktop-harness-session.desktop, so subsequent XFCE logins refresh registration automatically. Re-running install updates both the helper and its autostart entry. No periodic refresh is needed within the same session. Keep the desktop logged in and unlocked for these initial checks.

To test automatic registration, log out and back into XFCE, then capture from berlin without running the registration command manually. The helper assumes one graphical session for this account; it does not log in, unlock the screen, or choose between concurrent sessions. To disable automatic registration, remove the autostart file.

Try capturing

From berlin:

npm run home -- capture home artifacts/first.png

Open the PNG. Check dimensions and that it shows the actual desktop rather than a blank or different display.

Coordinate convention

Set DESKTOP_COORDINATE_SPACE=normalized_1000 in .env for models that emit coordinates on a 0–1000 scale. Set pixels for literal screenshot coordinates. Omission defaults to pixels for backward compatibility; .env.example selects normalized coordinates. Both the manual home act CLI and the model loop use this setting. Existing pixel-based action files must be converted or run with DESKTOP_COORDINATE_SPACE=pixels npm run home -- act ....

The model's tool description and observations state the convention. The VM helper converts normalized clicks exactly once using round(value × (dimension - 1) / 1000) with nonnegative half-up rounding. (0,0) and (1000,1000) map to the first and last screen pixels. Screenshots are not resized. Conversion uses current display dimensions and is returned in tool results; the model loop logs supplied and pixel coordinates. Reinstall the VM helper after updating this code. Environment changes take effect on the next CLI invocation.

Try input

Open Mousepad or another ordinary GUI text editor manually, focus a blank document, and create actions.json on berlin:

{
    "actions": [
        { "type": "text", "text": "Hello from berlin!\nUnicode: café — こんにちは" },
        { "type": "wait", "milliseconds": 300 }
    ]
}

npm run home -- act home actions.json artifacts/typed.png

Verify the text and screenshot. Text insertion replaces the clipboard and pastes with Ctrl+V; this is for ordinary GUI text fields, not terminals (which often need Ctrl+Shift+V). For shell work use the shell command below.

Other action examples:

{
    "expectedSize": [1280, 800],
    "actions": [
        { "type": "click", "ymin": 190, "xmin": 280, "ymax": 210, "xmax": 320, "button": "left" },
        { "type": "keys", "keys": ["ctrl", "a"] },
        { "type": "scroll", "direction": "down", "steps": 2 }
    ]
}

Use coordinates from your own screenshot, not these example coordinates. Coordinate clicks draw a bounding box with ymin, xmin, ymax, and xmax placing the target in the middle of the box (the click is delivered to the center of the box). expectedSize is optional and rejects input if the screen size changed. Supported actions are click, click_target, scroll, keys, text, and wait. Drag is deferred. Actions are serialized by a VM-side lock; avoid manual interaction while a request runs. The full request is validated before any actions execute, but runtime failures can still leave partial effects.

After a non-empty action batch, the helper waits 500 ms before taking the resulting screenshot and scanning accessibility targets. Capture-only requests have no added delay. This allows ordinary UI updates to paint; slow loads still need an explicit wait or another observation.

AT-SPI numbered targets

Install the accessibility bindings inside home:

sudo pacman -S --needed at-spi2-core python-atspi

Enable Settings → Accessibility → Enable assistive technologies in XFCE if disabled, then log out/in and reopen applications. The helper must use the graphical session's D-Bus address; session registration captures it automatically.

Rebuild and reinstall from berlin:

npm run build
npm run home -- install home
npm run home -- capture home artifacts/targets.png

Focus Mousepad or another GTK application before capturing. The PNG has magenta element bounds and yellow numbered labels; an unmarked original is saved as artifacts/targets.png.raw.png. CLI output includes targets, observationId, and any accessibility warning. Only actionable, enabled, showing elements in active accessible windows are included. There are traversal, target-count, and three-second collection limits. Missing support or timeouts leave pixel interaction available. This does not add OCR or infer targets from image content.

To click a label, copy the actual ID and observation ID into an action file:

{
    "actions": [
        {"type": "click_target", "target": "3", "observationId": "COPY-FROM-CAPTURE"},
        {"type": "text", "text": "Hello from an accessibility target"}
    ]
}

A target click must be first, and only one is allowed per request. The helper checks it against the latest capture and a fresh accessibility scan (application/path, name, role, and bounds) before clicking its center. Any intervening capture invalidates earlier IDs. This detects many stale targets, but is not an atomic guarantee against moving UI or occlusion. For small, overlapping, or poorly exposed elements, use a fresh screenshot and coordinate clicks in the configured coordinate space instead. AT-SPI target IDs always resolve directly to pixels, regardless of DESKTOP_COORDINATE_SPACE; target bounds are reported in pixels. Avoid changing windows manually while the model operates.

The model receives annotated screenshots and a text list of targets. Original images are retained beside the annotated run artifacts. Targets depend on the application's accessibility support: games, launchers, menus, and custom interfaces may expose none. Start testing with a focused editor, then try the launcher/browser. If a normal editor shows no targets, check accessibility settings, restart the app, and re-register the graphical session.

Try shell execution

npm run home -- shell home 'printf "hello\n"; uname -a'
npm run home -- shell home 'sleep 10' 2

The second command should time out. GNU timeout runs inside the VM and sends TERM, then KILL after two seconds. This is not a sandbox: commands have the desktop user's permissions, and processes that deliberately detach can escape the timeout. Output is limited to 32 MiB across stdout/stderr; exceeding it disconnects SSH and reports an unknown outcome. Commands must be one quoted local argument. Default timeout is 30 seconds, maximum 300.

Local checks

npm run check
npm test
python3 -m unittest discover -s test -p 'test_*.py'

Desktop dependencies are imported only when operating the display, so validation tests can run without X11 or Pillow.

Configuration and SQLite

cp .env.example .env
npm run build
npm run setup:check

.env.example contains berlin's llama.cpp origin and model ID. Edit .env for your installation. The origin excludes /v1; future model requests will append the API path. DATABASE_PATH defaults to ./data/token.sqlite, relative to the process working directory. .env and database files are ignored by Git. The check validates configuration and opens/checks SQLite; it does not contact the model or operate the desktop.

src/database.ts exports a general openDatabase() connection using Node's built-in SQLite, with foreign keys, WAL, and a five-second busy timeout. The caller owns the connection and must close it. Use one connection in the eventual supervisor and pass it to domain modules; it is not tied to conversations or logs. Append numbered SQL migrations to migrations as actual schemas are introduced (projects, schedules, messages, etc.). Applied migration SQL is recorded and checked against subsequent builds. Pending migrations run transactionally; incompatible history fails rather than silently changing existing data. The first migration stores commissioning runs, conversation messages, and tool operations. Other domains can add independent tables through subsequent migrations; there is no ORM. Back up live databases using a SQLite-aware backup mechanism, not by copying only the main file while WAL is active.

Model connection and editor test

npm install
npm run build
npm run agent -- probe home

The probe captures the desktop, sends the screenshot to llama.cpp, and requests a description and a proposed capture tool call. It prints but does not execute the proposed calls. Check that the description matches the screen and the call is desktop with empty actions. This tests the actual vision and tool-call message format.

Then, with the desktop unlocked and no unsaved work in the editor:

npm run agent -- run home 'Use the desktop to open Mousepad, type a short greeting, and save it as /home/user/harness-test.txt. Use shell to read that file and verify its contents, then stop. Do not modify any other existing files.' 8

run really executes model-generated desktop actions and arbitrary shell commands with the VM user's permissions. Watch the first runs. Do not manually interact with the desktop concurrently. The optional turn limit defaults to 8, maximum 30; each turn permits at most eight sequential tool calls. Ctrl+C stops inference or waits for the current tool to finish before stopping. It does not undo effects. There is no interactive pause/resume yet.

Runs, messages, and operation results are stored in SQLite. Screenshots live alongside the database under artifacts/<run-id>/; the latest screenshot is sent to the model, while older ones remain on disk. Reasoning returned by the endpoint is retained as part of its assistant message. Only the commissioning instructions are used—no identity prompt or resident history is introduced.

A final response without calls ends the run. Truncated model responses are rejected without executing their calls. Connection or desktop errors can leave partial effects; tool errors explicitly tell the model not to assume otherwise. If the supervisor is killed, operations left as started have unknown outcomes. Runs are not automatically resumed or retried: inspect them before starting another test. A new invocation creates a new conversation. A turn bound limits these initial sessions; token-budget enforcement and long-term context management remain future work.

The development environment cannot currently resolve berlin, so end-to-end model and desktop testing must be done from your machine. Next: persistent sleep/wake and deliberate restart recovery.