Computer Action

Execute actions like mouse movements, clicks, keyboard input, and screenshots in the Bytebot desktop environment.

Request

action

string

required

The type of computer action to perform. Must be one of: move_mouse, trace_mouse, click_mouse, press_mouse, drag_mouse, scroll, type_keys, press_keys, type_text, wait, screenshot, cursor_position.

Mouse Actions

move_mouse

coordinates

object

required

The target coordinates to move to.

Show coordinates properties

number

required

X coordinate (horizontal position)

number

required

Y coordinate (vertical position)

Example Request

{
  "action": "move_mouse",
  "coordinates": {
    "x": 100,
    "y": 200
  }
}

trace_mouse

path

array

required

Array of coordinate objects for the mouse path.

Show path

object

Show properties

number

required

X coordinate

number

required

Y coordinate

holdKeys

array

Keys to hold while moving the mouse along the path.

Example Request

{
  "action": "trace_mouse",
  "path": [
    { "x": 100, "y": 100 },
    { "x": 150, "y": 150 },
    { "x": 200, "y": 200 }
  ],
  "holdKeys": ["shift"]
}

click_mouse

coordinates

object

The coordinates to click (uses current cursor position if omitted).

Show coordinates properties

number

required

X coordinate (horizontal position)

number

required

Y coordinate (vertical position)

button

string

required

Mouse button to click. Must be one of: left, right, middle.

clickCount

number

required

Number of clicks to perform.

holdKeys

array

Keys to hold while clicking (e.g., [‘ctrl’, ‘shift’])

Show holdKeys

string

Key name

Example Request

{
  "action": "click_mouse",
  "coordinates": {
    "x": 150,
    "y": 250
  },
  "button": "left",
  "clickCount": 2
}

press_mouse

coordinates

object

The coordinates to press/release (uses current cursor position if omitted).

Show coordinates properties

number

required

X coordinate (horizontal position)

number

required

Y coordinate (vertical position)

button

string

required

Mouse button to press/release. Must be one of: left, right, middle.

press

string

required

Whether to press or release the button. Must be one of: up, down.

Example Request

{
  "action": "press_mouse",
  "coordinates": {
    "x": 150,
    "y": 250
  },
  "button": "left",
  "press": "down"
}

drag_mouse

path

array

required

Array of coordinate objects for the drag path.

Show path

object

Show properties

number

required

X coordinate

number

required

Y coordinate

button

string

required

Mouse button to use for dragging. Must be one of: left, right, middle.

holdKeys

array

Keys to hold while dragging.

Example Request

{
  "action": "drag_mouse",
  "path": [
    { "x": 100, "y": 100 },
    { "x": 200, "y": 200 }
  ],
  "button": "left"
}

scroll

coordinates

object

The coordinates to scroll at (uses current cursor position if omitted).

Show coordinates properties

number

required

X coordinate

number

required

Y coordinate

direction

string

required

Scroll direction. Must be one of: up, down, left, right.

scrollCount

number

required

Number of scroll steps to perform.

holdKeys

array

Keys to hold while scrolling.

Example Request

{
  "action": "scroll",
  "direction": "down",
  "scrollCount": 5
}

Keyboard Actions

type_keys

keys

array

required

Array of keys to type in sequence.

Show keys

string

Key name

delay

number

Delay between key presses in milliseconds.

Example Request

{
  "action": "type_keys",
  "keys": ["a", "b", "c", "enter"],
  "delay": 50
}

press_keys

keys

array

required

Array of keys to press or release.

Show keys

string

Key name

press

string

required

Whether to press or release the keys. Must be one of: up, down.

Example Request

{
  "action": "press_keys",
  "keys": ["ctrl", "shift", "esc"],
  "press": "down"
}

type_text

paste_text

text

string

required

The text to paste. Useful for special characters that aren’t on the standard keyboard.

Example Request

{
  "action": "paste_text",
  "text": "Special characters: ©®™€¥£ émojis 🎉"
}

System Actions

wait

screenshot

cursor_position

application

string

required

The application to switch to. Available options: firefox, 1password, thunderbird, vscode, terminal, desktop, directory.

Example Request

{
  "action": "application",
  "application": "firefox"
}

Available Applications:

firefox - Mozilla Firefox web browser
1password - Password manager
thunderbird - Email client
vscode - Visual Studio Code editor
terminal - Terminal/console application
desktop - Switch to desktop
directory - File manager/directory browser

Response

Responses vary based on the action performed:

Default Response

Most actions return a simple success response:

{
  "success": true
}

Screenshot Response

Returns the screenshot as a base64 encoded string:

{
  "success": true,
  "data": {
    "image": "base64_encoded_image_data"
  }
}

Cursor Position Response

Returns the current cursor position:

{
  "success": true,
  "data": {
    "x": 123,
    "y": 456
  }
}

Error Response

{
  "success": false,
  "error": "Error message"
}

Code Examples

curl -X POST http://localhost:9990/computer-use \
  -H "Content-Type: application/json" \
  -d '{"action": "move_mouse", "coordinates": {"x": 100, "y": 200}}'

Body

application/json

Response

200

application/json

Successful response

The response is of type object.

Overview

Agent API

Computer Control API

Request

Mouse Actions

Keyboard Actions

System Actions

Response

Default Response

Screenshot Response

Cursor Position Response

Error Response

Code Examples

Body

Response

Overview

Agent API

Computer Control API

​Request

​Mouse Actions

​Keyboard Actions

​System Actions

​Response

​Default Response

​Screenshot Response

​Cursor Position Response

​Error Response

​Code Examples

Body

Response

Request

Mouse Actions

Keyboard Actions

System Actions

Response

Default Response

Screenshot Response

Cursor Position Response

Error Response

Code Examples