Overview
The unified computer action API allows for granular control over all aspects of the Bytebot virtual desktop environment through a single endpoint. It replaces multiple specific endpoints with a unified interface that handles various computer actions like mouse movements, clicks, key presses, and more.Endpoint
Method | URL | Description |
---|---|---|
POST | /computer-use | Execute computer actions in the virtual desktop |
Request Format
All requests to the unified endpoint follow this format:action
parameter determines which operation to perform, and the remaining parameters depend on the specific action.
Available Actions
move_mouse
Move the mouse cursor to a specific position. Parameters:Parameter | Type | Required | Description |
---|---|---|---|
coordinates | Object | Yes | The target coordinates to move to |
coordinates.x | Number | Yes | X coordinate |
coordinates.y | Number | Yes | Y coordinate |
trace_mouse
Move the mouse along a path of coordinates. Parameters:Parameter | Type | Required | Description |
---|---|---|---|
path | Array | Yes | Array of coordinate objects for the mouse path |
path[].x | Number | Yes | X coordinate for each point in the path |
path[].y | Number | Yes | Y coordinate for each point in the path |
holdKeys | Array | No | Keys to hold while moving along the path |
click_mouse
Perform a mouse click at the current or specified position. Parameters:Parameter | Type | Required | Description |
---|---|---|---|
coordinates | Object | No | The coordinates to click (uses current if omitted) |
coordinates.x | Number | Yes* | X coordinate |
coordinates.y | Number | Yes* | Y coordinate |
button | String | Yes | Mouse button: ‘left’, ‘right’, or ‘middle’ |
clickCount | Number | Yes | Number of clicks to perform |
holdKeys | Array | No | Keys to hold while clicking (e.g., [‘ctrl’, ‘shift’]) |
press_mouse
Press or release a mouse button at the current or specified position. Parameters:Parameter | Type | Required | Description |
---|---|---|---|
coordinates | Object | No | The coordinates to press/release (uses current if omitted) |
coordinates.x | Number | Yes* | X coordinate |
coordinates.y | Number | Yes* | Y coordinate |
button | String | Yes | Mouse button: ‘left’, ‘right’, or ‘middle’ |
press | String | Yes | Action: ‘up’ or ‘down’ |
drag_mouse
Click and drag the mouse from one point to another. Parameters:Parameter | Type | Required | Description |
---|---|---|---|
path | Array | Yes | Array of coordinate objects for the drag path |
path[].x | Number | Yes | X coordinate for each point in the path |
path[].y | Number | Yes | Y coordinate for each point in the path |
button | String | Yes | Mouse button: ‘left’, ‘right’, or ‘middle’ |
holdKeys | Array | No | Keys to hold while dragging |
scroll
Scroll up, down, left, or right. Parameters:Parameter | Type | Required | Description |
---|---|---|---|
coordinates | Object | No | The coordinates to scroll at (uses current if omitted) |
coordinates.x | Number | Yes* | X coordinate |
coordinates.y | Number | Yes* | Y coordinate |
direction | String | Yes | Scroll direction: ‘up’, ‘down’, ‘left’, ‘right’ |
scrollCount | Number | Yes | Number of scroll steps |
holdKeys | Array | No | Keys to hold while scrolling |
type_keys
Type a sequence of keyboard keys. Parameters:Parameter | Type | Required | Description |
---|---|---|---|
keys | Array | Yes | Array of keys to type in sequence |
delay | Number | No | Delay between key presses (ms) |
press_keys
Press or release keyboard keys. Parameters:Parameter | Type | Required | Description |
---|---|---|---|
keys | Array | Yes | Array of keys to press or release |
press | String | Yes | Action: ‘up’ or ‘down’ |
type_text
Type a text string with optional delay. Parameters:Parameter | Type | Required | Description |
---|---|---|---|
text | String | Yes | The text to type |
delay | Number | No | Delay between characters in milliseconds (default: 0) |
paste_text
Paste text to the current cursor position. This is especially useful for special characters that aren’t on the standard keyboard. Parameters:Parameter | Type | Required | Description |
---|---|---|---|
text | String | Yes | The text to paste, including special characters and emojis |
wait
Wait for a specified duration. Parameters:Parameter | Type | Required | Description |
---|---|---|---|
duration | Number | Yes | Wait duration in milliseconds |
screenshot
Capture a screenshot of the desktop. Parameters: None required Example:cursor_position
Get the current position of the mouse cursor. Parameters: None required Example:application
Switch between different applications or navigate to the desktop/directory. Parameters:Parameter | Type | Required | Description |
---|---|---|---|
application | String | Yes | The application to switch to. See available options below. |
firefox
- Mozilla Firefox web browser1password
- Password managerthunderbird
- Email clientvscode
- Visual Studio Code editorterminal
- Terminal/console applicationdesktop
- Switch to desktopdirectory
- File manager/directory browser
write_file
Write a file to the desktop environment filesystem. Parameters:Parameter | Type | Required | Description |
---|---|---|---|
path | String | Yes | File path (absolute or relative to /home/user/Desktop) |
data | String | Yes | Base64 encoded file content |
read_file
Read a file from the desktop environment filesystem. Parameters:Parameter | Type | Required | Description |
---|---|---|---|
path | String | Yes | File path (absolute or relative to /home/user/Desktop) |