Skip to content

Conversation

@MaiLinhP
Copy link
Member

Added an AzureSdkIssueGeneratorService Function App that will reference azure-sdk-for-net issues to generate synthetic issues for other repos and add TSV file to blob storage for model training if needed.

Replaced all old ML labeler processes with streamlined IssueLabelerMLPipeline. Most of this pipeline is meant to be run locally for more control over model training processes. It includes:

  1. Downloader: Download applicable issues to local .tsv file.
  2. Trainer: Use downloaded issues (along with synthetic issues generated from AzureSdkIssueGeneratorService if available) to train Category and Service models. This will also provide per label statistics as well as a list of labels that need more data for training. This list can be passed back into AzureSdkIssueGeneratorService to generate issues with specific labels.
  3. Predictor: Use the category and service model zip files produced from Trainer to predict issues.
  4. Tester: Test models on all labeled issues for # of matches, mismatches, and no predictions. Also provides statistics on how models perform.
  5. DevOpsModelRetrainer: Process to be handed to DevOps team to periodically retrain models automatically. This download issues, train models along with synthetic data if any, then replace current models with new ones.

Unified config setup.

Previously, most function app configs use {org}/{repo}:{key} or defaults:{key}. However, as ModelHolderFactory is a singleton and didn't have per repo config at runtime, we were doing roundabout logics to configure model files (check old LegacyLabeler and IModelHolderFactoryLite.cs for details). Per repo config is now available to ModelHolderFactory, which simplifies many configs and cut out unnecessary name translation/extraction logic.

Simplified the flow from AzureSdkIssueLabelerService to Labeler.

Previously, there were many layers - ModelHolder, ModelHolderFactory, ModelHolderFactoryLite, Labeler, LabelerLite, LegacyLabeler. Replaced LegacyLabeler with MLLabeler, simplified the rest down to ModelHolder and ModelHolderFactory.

Previously, ModelHolder was also unaware of which label type it is, and the ordering of service vs category models in ModelHolderFactory was dictated by order listed in defaults:IssueModel.{repo}.BlobConfigNames. This made code unreadable without constantly referencing app config. ModelHolder now has modelType which makes ordering irrelevant, simplifies call to config and making code much more readable.

Cleaned up codebase

Deleted all unused references and refactored functions.

Copilot AI review requested due to automatic review settings December 10, 2025 22:40
@microsoft-github-policy-service
Copy link
Contributor

@MaiLinhP please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

  • (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@microsoft-github-policy-service agree
  • (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"
Contributor License Agreement

Contribution License Agreement

This Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
and conveys certain license rights to Microsoft Corporation and its affiliates (“Microsoft”) for Your
contributions to Microsoft open source projects. This Agreement is effective as of the latest signature
date below.

  1. Definitions.
    “Code” means the computer software code, whether in human-readable or machine-executable form,
    that is delivered by You to Microsoft under this Agreement.
    “Project” means any of the projects owned or managed by Microsoft and offered under a license
    approved by the Open Source Initiative (www.opensource.org).
    “Submit” is the act of uploading, submitting, transmitting, or distributing code or other content to any
    Project, including but not limited to communication on electronic mailing lists, source code control
    systems, and issue tracking systems that are managed by, or on behalf of, the Project for the purpose of
    discussing and improving that Project, but excluding communication that is conspicuously marked or
    otherwise designated in writing by You as “Not a Submission.”
    “Submission” means the Code and any other copyrightable material Submitted by You, including any
    associated comments and documentation.
  2. Your Submission. You must agree to the terms of this Agreement before making a Submission to any
    Project. This Agreement covers any and all Submissions that You, now or in the future (except as
    described in Section 4 below), Submit to any Project.
  3. Originality of Work. You represent that each of Your Submissions is entirely Your original work.
    Should You wish to Submit materials that are not Your original work, You may Submit them separately
    to the Project if You (a) retain all copyright and license information that was in the materials as You
    received them, (b) in the description accompanying Your Submission, include the phrase “Submission
    containing materials of a third party:” followed by the names of the third party and any licenses or other
    restrictions of which You are aware, and (c) follow any other instructions in the Project’s written
    guidelines concerning Submissions.
  4. Your Employer. References to “employer” in this Agreement include Your employer or anyone else
    for whom You are acting in making Your Submission, e.g. as a contractor, vendor, or agent. If Your
    Submission is made in the course of Your work for an employer or Your employer has intellectual
    property rights in Your Submission by contract or applicable law, You must secure permission from Your
    employer to make the Submission before signing this Agreement. In that case, the term “You” in this
    Agreement will refer to You and the employer collectively. If You change employers in the future and
    desire to Submit additional Submissions for the new employer, then You agree to sign a new Agreement
    and secure permission from the new employer before Submitting those Submissions.
  5. Licenses.
  • Copyright License. You grant Microsoft, and those who receive the Submission directly or
    indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license in the
    Submission to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute
    the Submission and such derivative works, and to sublicense any or all of the foregoing rights to third
    parties.
  • Patent License. You grant Microsoft, and those who receive the Submission directly or
    indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license under
    Your patent claims that are necessarily infringed by the Submission or the combination of the
    Submission with the Project to which it was Submitted to make, have made, use, offer to sell, sell and
    import or otherwise dispose of the Submission alone or with the Project.
  • Other Rights Reserved. Each party reserves all rights not expressly granted in this Agreement.
    No additional licenses or rights whatsoever (including, without limitation, any implied licenses) are
    granted by implication, exhaustion, estoppel or otherwise.
  1. Representations and Warranties. You represent that You are legally entitled to grant the above
    licenses. You represent that each of Your Submissions is entirely Your original work (except as You may
    have disclosed under Section 3). You represent that You have secured permission from Your employer to
    make the Submission in cases where Your Submission is made in the course of Your work for Your
    employer or Your employer has intellectual property rights in Your Submission by contract or applicable
    law. If You are signing this Agreement on behalf of Your employer, You represent and warrant that You
    have the necessary authority to bind the listed employer to the obligations contained in this Agreement.
    You are not expected to provide support for Your Submission, unless You choose to do so. UNLESS
    REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, AND EXCEPT FOR THE WARRANTIES
    EXPRESSLY STATED IN SECTIONS 3, 4, AND 6, THE SUBMISSION PROVIDED UNDER THIS AGREEMENT IS
    PROVIDED WITHOUT WARRANTY OF ANY KIND, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY OF
    NONINFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
  2. Notice to Microsoft. You agree to notify Microsoft in writing of any facts or circumstances of which
    You later become aware that would make Your representations in this Agreement inaccurate in any
    respect.
  3. Information about Submissions. You agree that contributions to Projects and information about
    contributions may be maintained indefinitely and disclosed publicly, including Your name and other
    information that You submit with Your Submission.
  4. Governing Law/Jurisdiction. This Agreement is governed by the laws of the State of Washington, and
    the parties consent to exclusive jurisdiction and venue in the federal courts sitting in King County,
    Washington, unless no federal subject matter jurisdiction exists, in which case the parties consent to
    exclusive jurisdiction and venue in the Superior Court of King County, Washington. The parties waive all
    defenses of lack of personal jurisdiction and forum non-conveniens.
  5. Entire Agreement/Assignment. This Agreement is the entire agreement between the parties, and
    supersedes any and all prior agreements, understandings or communications, written or oral, between
    the parties relating to the subject matter hereof. This Agreement may be assigned by Microsoft.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR represents a significant architectural refactoring of the issue-labeler project, replacing legacy ML labeling infrastructure with a modern streamlined pipeline approach.

Key Changes

  • New Synthetic Issue Generation: Added AzureSdkIssueGeneratorService Function App to generate synthetic issues from azure-sdk-for-net issues for improved model training
  • Streamlined ML Pipeline: Replaced old ML labeler processes with IssueLabelerMLPipeline containing local tools for downloading, training, testing, and predicting issue labels
  • Simplified Architecture: Removed Hubbup.MikLabelModel project and replaced LegacyLabeler with MLLabeler, eliminating multiple abstraction layers and simplifying the codebase
  • Unified Configuration: Consolidated config setup with per-repository support in ModelHolderFactory, removing complex name translation logic

Reviewed changes

Copilot reviewed 123 out of 123 changed files in this pull request and generated 28 comments.

Show a summary per file
File Description
IssueLabelerService/Labelers/MLLabeler.cs New ML labeler replacing LegacyLabeler with simplified prediction logic
IssueLabelerService/IssueGenerator/OpenAiIssueGenerator.cs New issue generator service for creating synthetic training data
IssueLabelerMLPipeline/src/ Complete new ML pipeline with Downloader, Trainer, Predictor, Tester, and DevOpsModelRetrainer tools
IssueLabeler.Shared/ModelHolder.cs Refactored to include modelType awareness and simplified blob loading
IssueLabeler.Shared/IModelHolderFactory.cs New interface with per-repository configuration support
Hubbup.MikLabelModel/ Entire project and dependencies removed
tests/Hubbup.MikLabelModel.Tests/ Test project removed as part of cleanup
Comments suppressed due to low confidence (2)

tools/issue-labeler/src/IssueLabelerService/AzureSdkIssueGeneratorService.cs:46

  • This assignment to result is useless, since its value is never read.
    tools/issue-labeler/src/IssueLabelerService/AzureSdkIssueGeneratorService.cs:51
  • This assignment to answer is useless, since its value is never read.

}

var config = _configurationService.GetDefault();
TriageOutput result = new TriageOutput {};
Copy link

Copilot AI Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused variable 'result' is created but never used. The return statement from the issue generator service is assigned to 'answer' (line 51), but then the function returns without using either variable. This appears to be incomplete implementation or should return a proper response.

Copilot uses AI. Check for mistakes.
Comment on lines +1 to +5
public enum LabelType
{
Category = 1,
Service = 2
} No newline at end of file
Copy link

Copilot AI Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing copyright and license header. For consistency with other files in the project, add the standard copyright header at the top of this file.

Copilot uses AI. Check for mistakes.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 . We do need that.

Comment on lines +236 to +254
public static string FormatTemplate(string template, Dictionary<string, string> replacements, ILogger logger)
{
if (string.IsNullOrEmpty(template))
return string.Empty;

string result = template;

foreach (var replacement in replacements)
{
if (!result.Contains($"{{{replacement.Key}}}"))
{
logger.LogWarning($"Replacement value for {replacement.Key} does not exist in {template}.");
}
result = result.Replace($"{{{replacement.Key}}}", replacement.Value);
}

// Replace escaped newlines with actual newlines
return result.Replace("\\n", "\n");
}
Copy link

Copilot AI Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method 'FormatTemplate' appears to be a duplicate of similar functionality in 'AzureSdkIssueLabelerService' class (line 67 references it). Consider extracting this method to a shared utility class to avoid code duplication and improve maintainability.

Copilot uses AI. Check for mistakes.
Comment on lines +91 to +97
foreach (var issue in issues)
{
if (issue.Category is null || issue.Service is null || issue.Title is null || issue.Body is null
|| !applicableCategoryLabels.Contains(issue.Category) || !applicableServiceLabels.Contains(issue.Service)) continue;

await writer.WriteLineAsync(FormatIssueRecord(issue.Category, issue.Service, issue.Title, issue.Body));
}
Copy link

Copilot AI Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This foreach loop implicitly filters its target sequence - consider filtering the sequence explicitly using '.Where(...)'.

Copilot uses AI. Check for mistakes.
Comment on lines +154 to +158
foreach (var obj in arr)
{
if (obj is JObject jobj)
allSources.Add(jobj);
}
Copy link

Copilot AI Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This foreach loop implicitly filters its target sequence - consider filtering the sequence explicitly using '.Where(...)'.

Copilot uses AI. Check for mistakes.
private IModelHolderFactoryLite _modelHolderFactory;
private ILogger<LabelerFactory> _logger;
private ILabelerLite _labeler;
private IModelHolderFactory _modelHolderFactory;
Copy link

Copilot AI Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Field '_modelHolderFactory' can be 'readonly'.

Copilot uses AI. Check for mistakes.
{
public class OpenAiIssueGenerator : IIssueGeneratorService
{
private RepositoryConfiguration _config;
Copy link

Copilot AI Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Field '_config' can be 'readonly'.

Copilot uses AI. Check for mistakes.
public class OpenAiIssueGenerator : IIssueGeneratorService
{
private RepositoryConfiguration _config;
private TriageRag _ragService;
Copy link

Copilot AI Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Field '_ragService' can be 'readonly'.

Copilot uses AI. Check for mistakes.
{
private RepositoryConfiguration _config;
private TriageRag _ragService;
private ILogger<IssueGeneratorFactory> _logger;
Copy link

Copilot AI Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Field '_logger' can be 'readonly'.

Copilot uses AI. Check for mistakes.
private TriageRag _ragService;
private ILogger<IssueGeneratorFactory> _logger;
private readonly KnowledgeAgentRetrievalClient _retrievalClient;
private BlobServiceClient _blobClient;
Copy link

Copilot AI Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Field '_blobClient' can be 'readonly'.

Copilot uses AI. Check for mistakes.
Copy link
Member

@jeo02 jeo02 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

Wish I could have figured out GraphQL myself, but it was quite difficult to use. Do you encounter less rate limiting through GraphQL vs using the API through octokit? I imagine it should be quite an improvement.

Mostly just nits on the styling. I have nothing against top-level statements (even though the constant comments made it seem so) but since everything else is in classes I think we should keep it consistent.

Also saw a couple of classes that are declared all in one file. To find it easier it would be better to just declare them separately under Models folder.

The synthetic issues are really cool !!

Comment on lines +4 to +6
using Azure.Identity;
using Azure.Storage.Blobs;
using Microsoft.Extensions.Configuration;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason this class is all top level rather than being a class?

Comment on lines +1 to +2
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also all top level would it not be better to have it be a class?

/// <returns>The downloaded issues as an async enumerable collection of tuples containing the issue and its predicate-matched label (when only one matcing label is found).</returns>
public static async IAsyncEnumerable<(Issue Issue, string CategoryLabel, string ServiceLabel)> DownloadIssues(
string githubToken,
string org, string repo,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit)

Suggested change
string org, string repo,
string org,
string repo,

using GraphQL;
using GraphQL.Client.Http;
using GraphQL.Client.Serializer.SystemTextJson;
using Octokit;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
using Octokit;

Using graphql no octokit needed :)

<PackageReference Include="GitHubJwt" />
<PackageReference Include="GraphQL.Client" />
<PackageReference Include="GraphQL.Client.Serializer.SystemTextJson" />
<PackageReference Include="Octokit" />
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

won't need octokit here either

@@ -0,0 +1,55 @@
// Licensed to the .NET Foundation under one or more agreements.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Class per file :) like the other comment

@@ -0,0 +1,193 @@
// Licensed to the .NET Foundation under one or more agreements.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

top level but seeing as so much of the code is structured by classes, I think we should keep consistent and make this a class as well. (Unless its top level for some reason I don't see)

@@ -0,0 +1,236 @@
// Licensed to the .NET Foundation under one or more agreements.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

top level

return bestScore is not null ? (bestScore.Label, bestScore.Score) : ((string?)null, (float?)null);
}

class TestStats
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put this in its own file

@@ -0,0 +1,43 @@
// Licensed to the .NET Foundation under one or more agreements.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

top level

<MicrosoftNETTestSdkVersion>16.11.0</MicrosoftNETTestSdkVersion>
</PropertyGroup>

<PropertyGroup>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this for local support of the Azure Function? I'm wondering if we should move this to a local props file that we ignore so that CI doesn't get confused.

Task<IModelHolder> CreateModelHolder(string owner, string repo, string modelType);
Task<IPredictor> GetPredictor(string owner, string repo, string modelType);
}
public class ModelHolderFactory : IModelHolderFactory
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please split into separate files. In general, .NET has a strong one-class-per-file norm that we'll want to stick with unless there's a type that can't stand on its own. In that case, we'd want to nest it in its parent.

private readonly ConcurrentDictionary<(string, string, string), IModelHolder> models = new ConcurrentDictionary<(string, string, string), IModelHolder>();
private readonly ILogger<ModelHolderFactory> logger;
private readonly IRepositoryConfigurationProvider configurationProvider;
private SemaphoreSlim sem = new SemaphoreSlim(1,1);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. This is a good suggestion.

@@ -0,0 +1,139 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.
using Microsoft.Extensions.Logging;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging;

nit: We generally throw a blank between the header and statements as a convention.

}
public class ModelHolderFactory : IModelHolderFactory
{
private readonly ConcurrentDictionary<(string, string, string), IModelHolder> models = new ConcurrentDictionary<(string, string, string), IModelHolder>();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a quick mention that if you wanted to avoid the "Look at mah Java!" feel, you can shorten up the declarations on either side. Nothing wrong with the verbose form if that's your preference.

 private readonly ConcurrentDictionary<(string, string, string), IModelHolder> models = new();
 private readonly var models = new ConcurrentDictionary<(string, string, string), IModelHolder>();

@@ -0,0 +1,7 @@
<?xml version="1.0" encoding="utf-8"?>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to remove this. Microsoft has a new policy that requires us to pass through a proxy for all NuGet pulls and this resets that.

/// </summary>
/// <param name="text">The text to sanitize.</param>
/// <returns>The sanitized text.</returns>
public static string SanitizeText(string text)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can add .AsSpan() on text and save a bunch of allocations.

@@ -0,0 +1,5 @@
public enum LabelType
{
Category = 1,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we explicitly numbering? Generally, you don't want to do that unless you're actually using the numeric values. We also have an issue here as default(LabelType) ends up being an invalid because it gets initialized as (LabelType)0. We'll want to add an Unknown = 0 at least here.

Comment on lines +1 to +5
public enum LabelType
{
Category = 1,
Service = 2
} No newline at end of file
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 . We do need that.


public struct Args
{
public string? GitHubToken { get; set; }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a file that we borrowed. We'll want to update to our header. I was going to suggest using System.CommandLine but if we inherited this and we know it works, better to not.

@jsquire
Copy link
Member

jsquire commented Jan 9, 2026

GitHub is being really, really laggy today. I left some thoughts but didn't dig into the back half of the files yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants