Copilot Coding Agent for Azure DevOps

English | 中文

A fully autonomous code agent powered by GitHub Copilot CLI and Azure Pipelines. This system implements autonomous code implementation and code review through Work Item assignments and Pull Request comments with designated reviewers.

Demo

• Work Item: a python app for addition
• Pull Request: Copilot Generated PR for work item #13

🏗️ System Architecture

graph TB
    subgraph "Application Repository (Target Repo)"
        A[Azure DevOps Work Item/PR] -->|Assign Work Item to Copilot| C[Service Hook Trigger]
        A[Azure DevOps Work Item/PR] -->|PR Comment @copilot-agent| C[Service Hook Trigger]
    end

    subgraph "Webhook Service"
        C -->|HTTP POST| D[Flask Service]
        D -->|Validate & Extract| E[Pipeline Parameters]
        E -->|Trigger| F[Azure DevOps API]
    end

    subgraph "Copilot Coding Agent Repository"
        F -->|Start Pipeline| G[Azure Pipeline]
        G -->|Work Item Job| H[Work Item Workflow Job]
        G -->|PR Comment Job| I[PR Comment Workflow Job]
        G -->|PR Code Review Job| J[PR Code Review Workflow Job]
        H -->|7 Tasks: ack → plan → create_pr → implement → finalize| K[Copilot CLI]
        I -->|1 Task: Process Comment| K
        J -->|1 Task: Code Review| K
        K -->|Generate Code| L[Git Commit & Push]
    end

    L -->|Update| A
    K -->|Post Comment| A

    style D fill:#e1f5ff
    style G fill:#fff4e1
    style K fill:#e8f5e9

Loading

System Components

1. Application Repository: Your application code repository where actual development happens
2. Webhook Service: Flask-based relay service that captures Azure DevOps Service Hook events
3. Copilot Coding Agent Repository: Azure Pipeline orchestrator that runs coding automation

Workflow Overview

Work Item Assignment Workflow (Full Automation):

Assign Work Item to Copilot → Service Hook → Trigger Pipeline →
Acknowledge Work Item → Generate TODO Plan → Create PR →
Implement Code → Push Changes → Update PR & Work Item

PR Comment Workflow (Quick Updates):

Comment @copilot-agent in PR → Service Hook → Trigger Pipeline →
Acknowledge → Implement Changes → Push to Source Branch →
Post Summary Comment

PR Code Review Workflow (Code Review):

Add Copilot as PR Reviewer → Service Hook → Trigger Pipeline →
Acknowledge → Clone Repository → Analyze Code Changes →
Generate Review Report → Post Review Comment

📋 Prerequisites

Required Tools

• Azure DevOps Organization and Project (with API access)
• Azure Pipelines (for CI/CD execution)
• GitHub Copilot CLI access and subscription
• Docker (optional, for running webhook service in containers)

Required Permissions

• Application Repository: Contributor role (to receive Service Hooks and create PRs)
• Copilot Coding Agent Repository: Project Administrator role (to configure Pipelines)
• Azure DevOps Personal Access Token (PAT) with the following scopes:
  • Code (Read & Write)
  • Work Items (Read & Write)
  • Build (Read & Execute)

🚀 Administrator Setup Guide

Step 1: Create Copilot Service Account (Optional but Recommended)

It's recommended to create a dedicated Azure DevOps user account for the Copilot coding agent for better permission management and activity auditing. You can use an existing account, but it's not recommended.

1. Create a new Azure DevOps account named "Copilot" or similar

2. Generate a Personal Access Token (PAT) for this account:

   • Go to User Security → Personal Access Tokens
   • Click New Token
   • Token name: copilot-automation
   • Scopes:
     • Code (Read & Write)
     • Work Items (Read & Write)
     • Build (Read & Execute)
   • Create and securely save the token

3. Grant appropriate permissions to this user (choose one approach):

   • Option A (Recommended for Org-wide Use): Set as Azure DevOps Project Collection Administrator or project Project Administrator
     • This allows the Copilot user to access all repositories under the organization or project
     • More convenient when managing multiple projects
   • Option B (Recommended for Scoped Access): Add this user as Contributor to specific application repositories
     • More granular permission control, suitable for scenarios requiring limited access scope
   • This user will be assigned Work Items and create Pull Requests

Step 2: Setup Copilot Coding Agent Repository

Use the Copilot user created in Step 1

1. Import this repository to your Azure DevOps

   • Using the Copilot user created in Step 1, import the repository in Azure DevOps
   • Repository URL:

     https://github.com/satomic/ado-copilot-coding-agent.git

     

2. Configure Pipeline Variable Group

   Go to Pipelines → Library → + Variable group, create variable group copilot-agent-vars:

   Variable Name	Description	Secret
   ADO_PAT	Personal Access Token (from Step 1)	✅
   GITHUB_TOKEN	GitHub Copilot CLI access token with valid GitHub Copilot subscription	✅

   

   About GITHUB_TOKEN, this is a fine-grained personal access token with "Copilot Requests" permission enabled:

   • Visit https://github.com/settings/personal-access-tokens/new
   • Under "Permissions", click "add permissions" and select "Copilot Requests"
   • Generate your token

3. Create Azure Pipeline

   • Go to Pipelines → New Pipeline
   • Select Azure Repos Git
   • Select your Copilot coding agent repository
   • Select Existing Azure Pipelines YAML file
   • Select /azure-pipelines.yml
   • Save the Pipeline and note the Pipeline Definition ID, which can be found in the Pipeline URL. In the example below, the ID is 3.

4. Create Agent Pool

   • Go to Project Settings → Agent Pools
   • Create a new Agent Pool named copilot-coding-agent-runner
   • Add a new self-hosted Agent, follow the prompts to download and configure the Agent
   • For Linux, note that you cannot use the root account, use PAT for authentication
   • Ensure the Agent is running and successfully connected

5. Configure Copilot CLI Access

   I've already built a Docker image satomic/copilot-cli:latest that includes:

   • GitHub Copilot CLI installed
   • Pre-configured authentication that reads the GITHUB_TOKEN environment variable

   Or build your own image with Copilot CLI access.

Step 3: Deploy Webhook Service

1. Create .env file

   cat > .env << EOF
   ADO_ORG_URL=https://dev.azure.com/your-organization
   ADO_PROJECT=your-project-name
   ADO_PAT=your-personal-access-token
   PIPELINE_DEFINITION_ID=your-pipeline-definition-id
   WEBHOOK_SECRET_TOKEN=
   COPILOT_AGENT_USERNAME=copilot
   COPILOT_AGENT_COMMIT_EMAIL=copilot@github.com
   COPILOT_AGENT_USER_ID=  # Copilot user's Azure DevOps GUID
   LISTEN_HOST=0.0.0.0
   LISTEN_PORT=8080
   EOF

   • ADO_ORG_URL: Your Azure DevOps organization URL
   • ADO_PROJECT: Target project name
   • ADO_PAT: Personal Access Token generated in Step 1
   • PIPELINE_DEFINITION_ID: Pipeline Definition ID created in Step 2
   • COPILOT_AGENT_USERNAME: Copilot username created in Step 1, used to match Work Item assignee (System.AssignedTo newValue must start with this prefix), used to identify mentions in PR comments (e.g., @copilot or @<copilot>)
   • COPILOT_AGENT_USER_ID: Copilot user's GUID, used to identify mentions in @<USER-ID> format. For details, see: COPILOT_USER_ID_GUIDE

   This configuration ensures:

   1. Work Item Assignment: Triggers when a Work Item is assigned to the Copilot user
   2. PR Comment Processing: Only processes comments containing @copilot, @<copilot>, or @<USER-ID>
   3. Avoid Self-Response: Automatically filters comments posted by Copilot itself to prevent infinite loops

2. Run with Docker

   docker run -itd \
     --name ado-copilot-coding-agent-hook \
     -p 8080:8080 \
     --env-file .env \
     --restart unless-stopped \
     satomic/ado-copilot-coding-agent-hook:latest

3. Run from Source (Optional)

   git clone https://github.com/satomic/ado-copilot-coding-agent.git
   cd ado-copilot-coding-agent/
   python3 main.py

4. Hook Address Get the public address of the webhook service, for example:

   • http://your-server-ip:8080/webhook

Step 4: Configure Service Hooks in Application Project

Regular developers who want to use the Copilot coding agent only need to configure Service Hooks in their application project, without needing access to the Copilot coding agent repository.

1. Go to your Azure DevOps Project → Project Settings → Service hooks

2. Create Work Item Updated Hook

   • Click + to create subscription
   • Select Web Hooks
   • Trigger: Work item updated
   • URL: http://your-server-ip:8080/webhook
   • Click Finish

3. Create Pull Request Commented Hook

   • Click + to create subscription
   • Select Web Hooks
   • Trigger: Pull request commented on
   • URL: http://your-server-ip:8080/webhook
   • Click Finish

4. Create Pull Request Updated Hook (for Reviewer Review)

   • Click + to create subscription
   • Select Web Hooks
   • Trigger: Pull request updated
   • URL: http://your-server-ip:8080/webhook
   • Click Finish
   • Note: Triggers code review when Copilot user is added as a PR Reviewer

5. Test Service Hook

   • Click Test to send test event
   • Check webhook service logs for successful receipt
   • Verify HTTP 200/202 response

Step 5: Verification

1. Test Work Item Assignment

   • Create a test Work Item in the application project and link it to a branch in the target code repository
   • Assign it to the Copilot user
   • Observe the Azure Pipeline trigger in the Copilot coding agent repository
   • Verify PR creation and code implementation

2. Test PR Comment

   • Create a test PR in the application repository
   • Comment: @copilot add a hello world function
   • Verify pipeline execution and code changes

3. Test PR Code Review

   • Create a test PR in the application repository
   • Add the Copilot user to the PR's Reviewers list
   • Observe the Azure Pipeline trigger in the Copilot coding agent repository
   • Verify code review comment posted to PR

4. Check Logs

   # Webhook service logs
   docker logs -f ado-copilot-coding-agent-hook
   
   # Check saved webhook payloads
   ls -la hooks/
   
   # Check pipeline logs
   # Go to Copilot coding agent repository → Pipelines → Runs

🔧 Configuration Reference

Intermediate Files (Auto-excluded from Git)

Files generated during execution but excluded from commits:

• patch_raw.txt - Copilot raw output
• todo.md / todo_completed.md - Task checklist
• plan.json - Execution plan
• commit_msg.txt - Generated commit message
• pr_summary.txt - Change summary

🐛 Troubleshooting

Service Hook Not Triggering

1. Check Service Hook Delivery

   • Azure DevOps Project → Project Settings → Service hooks
   • View subscription history
   • Look for 2xx status codes

2. Verify Webhook Service

   curl -X POST https://webhook.yourdomain.com/webhook \
     -H "Content-Type: application/json" \
     -d '{"test": true}'

Pipeline Not Triggering

1. Verify Pipeline Definition ID

   • Ensure PIPELINE_DEFINITION_ID in .env is correct
   • Get the correct ID from the URL in Azure DevOps Pipelines

2. Check PAT Permissions

   • Ensure PAT has Build (Read & Execute) permission
   • Ensure PAT is not expired

Copilot CLI Errors

1. Verify GITHUB_TOKEN

   • Ensure token has Copilot Requests permission
   • Ensure GitHub account has a valid Copilot subscription

2. Check Docker Image

   • Ensure using the latest satomic/copilot-cli:latest image
   • Check container logs for authentication errors

Git Push Failures

1. Verify ADO_PAT Permissions

   • Ensure PAT has Code (Read & Write) permission
   • Ensure Copilot user has Contributor permission to the repository

2. Check Branch Policies

   • Some branches may have protection rules
   • Ensure Copilot user can push to target branch