Skip to content

Add AI Triage Workflow and LLM Guidelines for Automated Issue Management #1789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chagong merged 2 commits into from
Sep 4, 2025
Merged

Add AI Triage Workflow and LLM Guidelines for Automated Issue Management #1789

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d5669a1
Initial plan
Copilot Sep 3, 2025
472de6f
Add AI triage workflow and LLM guidelines for vscode-java-test
Copilot Sep 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions .github/llms.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Test Runner for Java

Test Runner for Java is a lightweight extension to run and debug Java test cases in Visual Studio Code. It works with Language Support for Java by Red Hat to provide comprehensive Java testing capabilities.

## Features
- Run & Debug JUnit/TestNG Test Cases
- Test discovery and execution
- Test result visualization
- Integration with Java project structure
- Support for Maven and Gradle projects

## Label
When labeling an issue, follow the rules below per label category:
### General Rules
- Analyze if the issue is related with the scope of Java testing functionality in VS Code. If not, STOP labelling IMMEDIATELY.
- Assign label per category.
- If a category is not applicable or you're unsure, you may skip it.
- Do not assign multiple labels within the same category, unless explicitly allowed as an exception.

### Issue Type Labels
- [bug]: Primary label for real bug issues
- [enhancement]: Primary label for enhancement issues
- [documentation]: Primary label for documentation issues
- [question]: Primary label for question issues

### Test Framework Labels
- [junit]: Issues specific to JUnit test framework
- [testng]: Issues specific to TestNG test framework

### Project Type Labels
- [maven]: Issues related to Maven projects
- [gradle]: Issues related to Gradle projects

### Component Labels
- [test-discovery]: Issues related to finding and discovering tests
- [test-execution]: Issues related to running tests
- [test-debugging]: Issues related to debugging tests
- [test-reporting]: Issues related to test results and reporting
- [ui]: Issues related to user interface and experience

### Priority Labels
- [high-priority]: Critical issues that significantly impact functionality
- [low-priority]: Nice-to-have improvements or minor issues

### Status Labels
- [waiting-for-user-info]: Waiting for additional information from the user
- [investigating]: Issue is being investigated
- [needs-reproduction]: Issue needs to be reproduced
- [fixed-next-release]: Issue has been fixed and will be in the next release
122 changes: 122 additions & 0 deletions .github/workflows/triage-agent.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
name: AI Triage - Label and Comment on New Issues
on:
issues:
types: [opened]
workflow_dispatch:
inputs:
issue_number:
description: 'Issue number to triage (manual run). e.g. 123'
required: true

permissions:
issues: write
contents: read

jobs:
label_and_comment:
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@v2

- name: Get issue data
id: get_issue
uses: actions/github-script@v6
with:
script: |
const eventName = context.eventName;
let issue;
if (eventName === 'workflow_dispatch') {
const inputs = context.payload.inputs || {};
const issueNumber = inputs.issue_number || inputs.issueNumber;
if (!issueNumber) core.setFailed('Input issue_number is required for manual run.');
const { data } = await github.rest.issues.get({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: parseInt(issueNumber, 10),
});
issue = data;
} else if (context.payload.issue) {
issue = context.payload.issue;
} else {
core.setFailed('No issue information found in the event payload.');
}
core.setOutput('id', String(issue.number));
core.setOutput('user', String((issue.user && issue.user.login) || ''));
core.setOutput('title', String(issue.title || ''));
core.setOutput('body', String(issue.body || ''));
const labelNames = (issue.labels || []).map(label => label.name);
core.setOutput('labels', JSON.stringify(labelNames));

- name: Call Azure Function
id: call_azure_function
env:
PAYLOAD: >-
{
"authToken": "${{ secrets.GITHUB_TOKEN }}",
"repoId": "microsoft/vscode-java-test",
"issueData": {
"id": ${{ steps.get_issue.outputs.id }},
"user": ${{ toJson(steps.get_issue.outputs.user) }},
"title": ${{ toJson(steps.get_issue.outputs.title) }},
"body": ${{ toJson(steps.get_issue.outputs.body) }},
"labels": ${{ steps.get_issue.outputs.labels }}
},
"mode": "DirectUpdate"
}

run: |
# Make the HTTP request with improved error handling and timeouts
echo "Making request to triage agent..."

# Add timeout handling and better error detection
set +e # Don't exit on curl failure
response=$(timeout ${{ vars.TRIAGE_AGENT_TIMEOUT }} curl \
--max-time 0 \
--connect-timeout 30 \
--fail-with-body \
--silent \
--show-error \
--write-out "HTTPSTATUS:%{http_code}" \
--header "Content-Type: application/json" \
--request POST \
--data "$PAYLOAD" \
${{ secrets.TRIAGE_FUNCTION_LINK }} 2>&1)

curl_exit_code=$?
set -e # Re-enable exit on error

echo "Curl exit code: $curl_exit_code"

# Check if curl command timed out or failed
if [ $curl_exit_code -eq 124 ]; then
echo "❌ Request timed out after 650 seconds"
exit 1
elif [ $curl_exit_code -ne 0 ]; then
echo "❌ Curl command failed with exit code: $curl_exit_code"
echo "Response: $response"
exit 1
fi

# Extract HTTP status code and response body
http_code=$(echo "$response" | grep -o "HTTPSTATUS:[0-9]*" | cut -d: -f2)
response_body=$(echo "$response" | sed 's/HTTPSTATUS:[0-9]*$//')

echo "HTTP Status Code: $http_code"

# Validate HTTP status code
if [ -z "$http_code" ]; then
echo "❌ Failed to extract HTTP status code from response"
echo "Raw response: $response"
exit 1
fi

# Check if the request was successful
if [ "$http_code" -ge 200 ] && [ "$http_code" -lt 300 ]; then
echo "✅ Azure Function call succeeded"
else
echo "❌ Azure Function call failed with status code: $http_code"
echo "Response: $response_body"
exit 1
fi
Loading