MCP Issue Labeler GitHub Triage Bot

[anannya03]

MCP Issue Labeler GitHub Triage Bot

This PR adds an issue labeling system for the microsoft/mcp repository using Azure AI Search, Azure OpenAI, and RAG (Retrieval-Augmented Generation).

Changes

MCP Labeler (McpOpenAiLabeler)

• Automatically predicts Server and Tool labels for new MCP issues
• Uses RAG to retrieve 15 similar labeled issues for context
• Applies rule-based reasoning with confidence thresholding (0.9)
• Prevents data leakage via excludeIssueId filter
• Achieves ~95% accuracy on evaluation dataset

MCP-Specific Infrastructure (McpTriageRag)

• Query preprocessing to remove tool name examples (e.g., "like Storage")
• Filters for labeled issues only (Server ne null)

Evaluation Tool (Mcp.Evaluator)

• Console app for testing labeler accuracy against ground truth
• Single issue prediction, batch evaluation, and test dataset extraction modes
• Exports results to CSV with detailed metrics