117 lines
5.5 KiB
Markdown
117 lines
5.5 KiB
Markdown
# Desktop harness — first slice
|
|
|
|
Manual SSH and X11 controls. There is **no model loop, persistent conversation, scheduler, or Jev integration yet**. The package name is a placeholder.
|
|
|
|
## On berlin (or your development computer)
|
|
|
|
Requires Node.js 22.13+ (for built-in SQLite) and an SSH client. Configure an SSH alias `home` for the VM's desktop user, with key authentication. Connect manually first to verify and save its host key:
|
|
|
|
```sh
|
|
ssh home
|
|
```
|
|
|
|
The CLI requires known host keys and noninteractive authentication. It does not bypass SSH verification.
|
|
|
|
```sh
|
|
npm install
|
|
npm run build
|
|
npm run home -- install home
|
|
```
|
|
|
|
## Inside home
|
|
|
|
Install dependencies:
|
|
|
|
```sh
|
|
sudo pacman -S --needed python python-pillow xdotool xclip
|
|
```
|
|
|
|
From a terminal **inside the logged-in XFCE desktop**, run:
|
|
|
|
```sh
|
|
python3 ~/.local/lib/desktop-harness/desktop.py --register-session
|
|
```
|
|
|
|
This records the current graphical session environment for SSH requests. Installation also creates `~/.config/autostart/desktop-harness-session.desktop`, so subsequent XFCE logins refresh registration automatically. Re-running `install` updates both the helper and its autostart entry. No periodic refresh is needed within the same session. Keep the desktop logged in and unlocked for these initial checks.
|
|
|
|
To test automatic registration, log out and back into XFCE, then capture from berlin without running the registration command manually. The helper assumes one graphical session for this account; it does not log in, unlock the screen, or choose between concurrent sessions. To disable automatic registration, remove the autostart file.
|
|
|
|
## Try capturing
|
|
|
|
From berlin:
|
|
|
|
```sh
|
|
npm run home -- capture home artifacts/first.png
|
|
```
|
|
|
|
Open the PNG. Check dimensions and that it shows the actual desktop rather than a blank or different display.
|
|
|
|
## Try input
|
|
|
|
Open Mousepad or another ordinary GUI text editor manually, focus a blank document, and create `actions.json` on berlin:
|
|
|
|
```json
|
|
{
|
|
"actions": [
|
|
{ "type": "text", "text": "Hello from berlin!\nUnicode: café — こんにちは" },
|
|
{ "type": "wait", "milliseconds": 300 }
|
|
]
|
|
}
|
|
```
|
|
|
|
```sh
|
|
npm run home -- act home actions.json artifacts/typed.png
|
|
```
|
|
|
|
Verify the text and screenshot. Text insertion **replaces the clipboard** and pastes with Ctrl+V; this is for ordinary GUI text fields, not terminals (which often need Ctrl+Shift+V). For shell work use the shell command below.
|
|
|
|
Other action examples:
|
|
|
|
```json
|
|
{
|
|
"expectedSize": [1280, 800],
|
|
"actions": [
|
|
{ "type": "click", "x": 300, "y": 200, "button": "left" },
|
|
{ "type": "keys", "keys": ["ctrl", "a"] },
|
|
{ "type": "scroll", "direction": "down", "steps": 2 }
|
|
]
|
|
}
|
|
```
|
|
|
|
Use coordinates from your own screenshot, not these example coordinates. `expectedSize` is optional and rejects input if the screen size changed. Supported actions are click, scroll, keys, text, and wait. Drag is deferred. Actions are serialized by a VM-side lock; avoid manual interaction while a request runs. The full request is validated before any actions execute, but runtime failures can still leave partial effects.
|
|
|
|
## Try shell execution
|
|
|
|
```sh
|
|
npm run home -- shell home 'printf "hello\n"; uname -a'
|
|
npm run home -- shell home 'sleep 10' 2
|
|
```
|
|
|
|
The second command should time out. GNU `timeout` runs inside the VM and sends TERM, then KILL after two seconds. This is not a sandbox: commands have the desktop user's permissions, and processes that deliberately detach can escape the timeout. Output is limited to 32 MiB across stdout/stderr; exceeding it disconnects SSH and reports an unknown outcome. Commands must be one quoted local argument. Default timeout is 30 seconds, maximum 300.
|
|
|
|
## Local checks
|
|
|
|
```sh
|
|
npm run check
|
|
npm test
|
|
python3 -m unittest discover -s test -p 'test_*.py'
|
|
```
|
|
|
|
Desktop dependencies are imported only when operating the display, so validation tests can run without X11 or Pillow.
|
|
|
|
## Configuration and SQLite
|
|
|
|
```sh
|
|
cp .env.example .env
|
|
npm run build
|
|
npm run setup:check
|
|
```
|
|
|
|
`.env.example` contains berlin's llama.cpp origin and model ID. Edit `.env` for your installation. The origin excludes `/v1`; future model requests will append the API path. `DATABASE_PATH` defaults to `./data/token.sqlite`, relative to the process working directory. `.env` and database files are ignored by Git. The check validates configuration and opens/checks SQLite; it does not contact the model or operate the desktop.
|
|
|
|
`src/database.ts` exports a general `openDatabase()` connection using Node's built-in SQLite, with foreign keys, WAL, and a five-second busy timeout. The caller owns the connection and must close it. Use one connection in the eventual supervisor and pass it to domain modules; it is not tied to conversations or logs. Append numbered SQL migrations to `migrations` as actual schemas are introduced (projects, schedules, messages, etc.). Applied migration SQL is recorded and checked against subsequent builds. Pending migrations run transactionally; incompatible history fails rather than silently changing existing data. There is intentionally no speculative domain schema or ORM yet. Back up live databases using a SQLite-aware backup mechanism, not by copying only the main file while WAL is active.
|
|
|
|
## Next milestone
|
|
|
|
Once capture, Unicode insertion, key combinations, and shell timeout work on the real VM, add the llama.cpp adapter, durable tool-call records, and a single model/tool loop. Wake/sleep and restart recovery follow. No SSH connection or real graphical session is available in the development sandbox, so those checks must be run on your setup.
|