feat: support normalizing coordinate spaces
This commit is contained in:
@@ -46,6 +46,12 @@ npm run home -- capture home artifacts/first.png
|
||||
|
||||
Open the PNG. Check dimensions and that it shows the actual desktop rather than a blank or different display.
|
||||
|
||||
## Coordinate convention
|
||||
|
||||
Set `DESKTOP_COORDINATE_SPACE=normalized_1000` in `.env` for models that emit coordinates on a 0–1000 scale. Set `pixels` for literal screenshot coordinates. Omission defaults to `pixels` for backward compatibility; `.env.example` selects normalized coordinates. Both the manual `home act` CLI and the model loop use this setting. Existing pixel-based action files must be converted or run with `DESKTOP_COORDINATE_SPACE=pixels npm run home -- act ...`.
|
||||
|
||||
The model's tool description and observations state the convention. The VM helper converts normalized clicks exactly once using `round(value × (dimension - 1) / 1000)` with nonnegative half-up rounding. `(0,0)` and `(1000,1000)` map to the first and last screen pixels. Screenshots are not resized. Conversion uses current display dimensions and is returned in tool results; the model loop logs supplied and pixel coordinates. Reinstall the VM helper after updating this code. Environment changes take effect on the next CLI invocation.
|
||||
|
||||
## Try input
|
||||
|
||||
Open Mousepad or another ordinary GUI text editor manually, focus a blank document, and create `actions.json` on berlin:
|
||||
|
||||
Reference in New Issue
Block a user