Agent System

​

Overview

The Bytebot Agent System transforms a simple desktop container into an intelligent, autonomous computer user. By combining Claude AI with structured task management, it can understand natural language requests and execute complex workflows just like a human would.

​

How the AI Agent Works

​

The Brain: Claude AI Integration

At the heart of Bytebot is Claude, Anthropic’s advanced AI assistant. The agent:

1. Understands Context: Processes your natural language requests with full conversation history
2. Plans Actions: Breaks down complex tasks into executable computer actions
3. Adapts in Real-time: Adjusts its approach based on what it sees on screen
4. Learns from Feedback: Improves task execution through conversation

​

Conversation Flow

​

Task Management System

​

Task Lifecycle

Tasks move through a structured lifecycle:

​

Task Properties

Each task contains:

• Description: What needs to be done
• Priority: Urgent, High, Medium, or Low
• Status: Current state in the lifecycle
• Type: Immediate or Scheduled
• History: All messages and actions taken

​

Smart Task Processing

The agent processes tasks intelligently:

1. Priority Queue: Urgent tasks run first
2. Error Recovery: Automatically retries failed actions
3. Human in the Loop: Asks for help when stuck
4. Context Preservation: Maintains conversation history across sessions

​

Real-world Capabilities

​

What the Agent Can Do

​

Example Use Cases

​

Research Assistant

User: "Find the top 5 project management tools and compare their pricing"

Agent Actions:
1. Opens browser
2. Searches for project management tools
3. Visits each tool's website
4. Extracts pricing information
5. Creates comparison spreadsheet
6. Takes screenshots of each pricing page

​

Form Automation

User: "Fill out the vendor application form with data from our company profile"

Agent Actions:
1. Opens the form URL
2. Reads company profile document
3. Maps data to form fields
4. Fills out each section
5. Uploads required documents
6. Submits and saves confirmation

​

Email Management

User: "Check my emails and create a summary of action items"

Agent Actions:
1. Opens email client
2. Reads unread messages
3. Identifies action items
4. Creates organized task list
5. Drafts response templates
6. Flags important messages

​

Technical Architecture

​

Core Components

1. NestJS Agent Service

   • Manages task queue with BullMQ
   • Integrates with Anthropic API
   • Handles WebSocket connections
   • Coordinates with desktop API

2. Message System

   • Structured conversation format
   • Supports text and images
   • Maintains full context
   • Enables rich interactions

3. Database Schema

   Copy

   Tasks: id, description, status, priority, timestamps
   Messages: id, task_id, role, content, timestamps
   Summaries: id, task_id, content, parent_id
   

4. Computer Action Bridge

   • Translates AI decisions to desktop actions
   • Handles screenshots and feedback
   • Manages action timing
   • Provides error handling

​

API Endpoints

Key endpoints for programmatic control:

// Create a new task
POST /tasks
{
  "description": "Your task description",
  "priority": "HIGH",
  "type": "IMMEDIATE"
}

// Get task status
GET /tasks/:id

// Send a message
POST /tasks/:id/messages
{
  "content": "Additional instructions"
}

// Get task history
GET /tasks/:id/messages

​

Chat UI Features

The web interface provides:

​

Real-time Interaction

• Live chat with the AI agent
• Instant status updates
• Progress indicators
• Error notifications

​

Visual Feedback

• Embedded desktop viewer
• Screenshot history
• Action replay
• Task timeline

​

Task Management

• Create and prioritize tasks
• View active and completed tasks
• Export conversation logs
• Manage task queues

​

Security & Privacy

​

Data Isolation

• All processing happens in your infrastructure
• No data sent to external services (except Claude API)
• Conversations stored locally
• Complete audit trail

​

Access Control

• Configurable authentication
• API key management
• Network isolation options
• Role-based permissions (coming soon)

​

Extending the Agent

​

Custom Tools

Add specialized capabilities:

// Register a custom tool
agent.registerTool({
  name: 'database_query',
  description: 'Query internal database',
  execute: async (params) => {
    // Your implementation
  }
});

​

Workflow Templates

Create reusable task templates:

name: "Daily Report"
steps:
  - action: "screenshot"
    target: "dashboard"
  - action: "extract_data"
    format: "table"
  - action: "create_document"
    template: "daily_report"

​

Integration Points

• Webhook notifications
• External API calls
• Custom AI prompts
• Plugin system (coming soon)

​

Performance Tuning

​

Optimization Settings

# Concurrent task limit
MAX_CONCURRENT_TASKS=1

# Task timeout (ms)
TASK_TIMEOUT=300000

# Screenshot quality (0-100)
SCREENSHOT_QUALITY=80

# Message history limit
MAX_CONTEXT_MESSAGES=50

​

Best Practices

1. Clear Instructions: Be specific about desired outcomes
2. Break Down Complex Tasks: Use multiple smaller tasks for better results
3. Provide Context: Include relevant files or URLs
4. Monitor Progress: Watch the desktop view for real-time feedback
5. Review Results: Verify outputs meet requirements

​

Troubleshooting

​

Next Steps