Computer Action

Execute actions like mouse movements, clicks, keyboard input, and screenshots in the Bytebot desktop environment.

Request

action

string

required

The type of computer action to perform. Must be one of: move_mouse, click_mouse, drag_mouse, scroll, press_key, type_text, wait, screenshot, cursor_position.

Mouse Actions

coordinates

object

required

The target coordinates to move to.

Example Request

{
  "action": "move_mouse",
  "coordinates": {
    "x": 100,
    "y": 200
  }
}

coordinates

object

The coordinates to click (uses current cursor position if omitted).

button

string

required

Mouse button to click. Must be one of: left, right, middle.

numClicks

number

Number of clicks to perform. Default: 1

holdKeys

array

Keys to hold while clicking (e.g., [‘ctrl’, ‘shift’])

Example Request

{
  "action": "click_mouse",
  "coordinates": {
    "x": 150,
    "y": 250
  },
  "button": "left",
  "numClicks": 2
}

path

array

required

Array of coordinate objects for the drag path.

button

string

required

Mouse button to use for dragging. Must be one of: left, right, middle.

holdKeys

array

Keys to hold while dragging.

Example Request

{
  "action": "drag_mouse",
  "path": [
    { "x": 100, "y": 100 },
    { "x": 200, "y": 200 }
  ],
  "button": "left"
}

coordinates

object

The coordinates to scroll at (uses current cursor position if omitted).

direction

string

required

Scroll direction. Must be one of: up, down, left, right.

amount

number

required

Scroll amount in pixels.

holdKeys

array

Keys to hold while scrolling.

Example Request

{
  "action": "scroll",
  "direction": "down",
  "amount": 100
}

Keyboard Actions

key

string

required

The key to press (e.g., ‘a’, ‘enter’, ‘tab’).

modifiers

array

Modifier keys to hold (e.g., [‘alt’, ‘shift’]).

Example Request

{
  "action": "press_key",
  "key": "tab",
  "modifiers": ["alt"]
}

System Actions

Response

success

boolean

Indicates if the operation was successful.

data

object

Action-specific response data.

error

string

Error message if success is false.

Response Examples

Code Examples

curl -X POST http://localhost:9990/computer-use \
  -H "Content-Type: application/json" \
  -d '{"action": "move_mouse", "coordinates": {"x": 100, "y": 200}}'

Overview

Endpoints

Request

Mouse Actions

Keyboard Actions

System Actions

Response

Response Examples

Code Examples

Overview

Endpoints

​Request

​Mouse Actions

​Keyboard Actions

​System Actions

​Response

​Response Examples

​Code Examples

Request

Mouse Actions

Keyboard Actions

System Actions

Response

Response Examples

Code Examples