-
Notifications
You must be signed in to change notification settings - Fork 227
Revamp issue-labeler project #13310
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Revamp issue-labeler project #13310
Conversation
…works with repo specific Configuration
…es to blob storage
|
@MaiLinhP please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
Contributor License AgreementContribution License AgreementThis Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR represents a significant architectural refactoring of the issue-labeler project, replacing legacy ML labeling infrastructure with a modern streamlined pipeline approach.
Key Changes
- New Synthetic Issue Generation: Added
AzureSdkIssueGeneratorServiceFunction App to generate synthetic issues from azure-sdk-for-net issues for improved model training - Streamlined ML Pipeline: Replaced old ML labeler processes with
IssueLabelerMLPipelinecontaining local tools for downloading, training, testing, and predicting issue labels - Simplified Architecture: Removed
Hubbup.MikLabelModelproject and replacedLegacyLabelerwithMLLabeler, eliminating multiple abstraction layers and simplifying the codebase - Unified Configuration: Consolidated config setup with per-repository support in
ModelHolderFactory, removing complex name translation logic
Reviewed changes
Copilot reviewed 123 out of 123 changed files in this pull request and generated 28 comments.
Show a summary per file
| File | Description |
|---|---|
IssueLabelerService/Labelers/MLLabeler.cs |
New ML labeler replacing LegacyLabeler with simplified prediction logic |
IssueLabelerService/IssueGenerator/OpenAiIssueGenerator.cs |
New issue generator service for creating synthetic training data |
IssueLabelerMLPipeline/src/ |
Complete new ML pipeline with Downloader, Trainer, Predictor, Tester, and DevOpsModelRetrainer tools |
IssueLabeler.Shared/ModelHolder.cs |
Refactored to include modelType awareness and simplified blob loading |
IssueLabeler.Shared/IModelHolderFactory.cs |
New interface with per-repository configuration support |
Hubbup.MikLabelModel/ |
Entire project and dependencies removed |
tests/Hubbup.MikLabelModel.Tests/ |
Test project removed as part of cleanup |
Comments suppressed due to low confidence (2)
tools/issue-labeler/src/IssueLabelerService/AzureSdkIssueGeneratorService.cs:46
| } | ||
|
|
||
| var config = _configurationService.GetDefault(); | ||
| TriageOutput result = new TriageOutput {}; |
Copilot
AI
Dec 10, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unused variable 'result' is created but never used. The return statement from the issue generator service is assigned to 'answer' (line 51), but then the function returns without using either variable. This appears to be incomplete implementation or should return a proper response.
| public enum LabelType | ||
| { | ||
| Category = 1, | ||
| Service = 2 | ||
| } No newline at end of file |
Copilot
AI
Dec 10, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing copyright and license header. For consistency with other files in the project, add the standard copyright header at the top of this file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 . We do need that.
| public static string FormatTemplate(string template, Dictionary<string, string> replacements, ILogger logger) | ||
| { | ||
| if (string.IsNullOrEmpty(template)) | ||
| return string.Empty; | ||
|
|
||
| string result = template; | ||
|
|
||
| foreach (var replacement in replacements) | ||
| { | ||
| if (!result.Contains($"{{{replacement.Key}}}")) | ||
| { | ||
| logger.LogWarning($"Replacement value for {replacement.Key} does not exist in {template}."); | ||
| } | ||
| result = result.Replace($"{{{replacement.Key}}}", replacement.Value); | ||
| } | ||
|
|
||
| // Replace escaped newlines with actual newlines | ||
| return result.Replace("\\n", "\n"); | ||
| } |
Copilot
AI
Dec 10, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The method 'FormatTemplate' appears to be a duplicate of similar functionality in 'AzureSdkIssueLabelerService' class (line 67 references it). Consider extracting this method to a shared utility class to avoid code duplication and improve maintainability.
| foreach (var issue in issues) | ||
| { | ||
| if (issue.Category is null || issue.Service is null || issue.Title is null || issue.Body is null | ||
| || !applicableCategoryLabels.Contains(issue.Category) || !applicableServiceLabels.Contains(issue.Service)) continue; | ||
|
|
||
| await writer.WriteLineAsync(FormatIssueRecord(issue.Category, issue.Service, issue.Title, issue.Body)); | ||
| } |
Copilot
AI
Dec 10, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This foreach loop implicitly filters its target sequence - consider filtering the sequence explicitly using '.Where(...)'.
| foreach (var obj in arr) | ||
| { | ||
| if (obj is JObject jobj) | ||
| allSources.Add(jobj); | ||
| } |
Copilot
AI
Dec 10, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This foreach loop implicitly filters its target sequence - consider filtering the sequence explicitly using '.Where(...)'.
| private IModelHolderFactoryLite _modelHolderFactory; | ||
| private ILogger<LabelerFactory> _logger; | ||
| private ILabelerLite _labeler; | ||
| private IModelHolderFactory _modelHolderFactory; |
Copilot
AI
Dec 10, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Field '_modelHolderFactory' can be 'readonly'.
| { | ||
| public class OpenAiIssueGenerator : IIssueGeneratorService | ||
| { | ||
| private RepositoryConfiguration _config; |
Copilot
AI
Dec 10, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Field '_config' can be 'readonly'.
| public class OpenAiIssueGenerator : IIssueGeneratorService | ||
| { | ||
| private RepositoryConfiguration _config; | ||
| private TriageRag _ragService; |
Copilot
AI
Dec 10, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Field '_ragService' can be 'readonly'.
| { | ||
| private RepositoryConfiguration _config; | ||
| private TriageRag _ragService; | ||
| private ILogger<IssueGeneratorFactory> _logger; |
Copilot
AI
Dec 10, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Field '_logger' can be 'readonly'.
| private TriageRag _ragService; | ||
| private ILogger<IssueGeneratorFactory> _logger; | ||
| private readonly KnowledgeAgentRetrievalClient _retrievalClient; | ||
| private BlobServiceClient _blobClient; |
Copilot
AI
Dec 10, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Field '_blobClient' can be 'readonly'.
jeo02
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good
Wish I could have figured out GraphQL myself, but it was quite difficult to use. Do you encounter less rate limiting through GraphQL vs using the API through octokit? I imagine it should be quite an improvement.
Mostly just nits on the styling. I have nothing against top-level statements (even though the constant comments made it seem so) but since everything else is in classes I think we should keep it consistent.
Also saw a couple of classes that are declared all in one file. To find it easier it would be better to just declare them separately under Models folder.
The synthetic issues are really cool !!
| using Azure.Identity; | ||
| using Azure.Storage.Blobs; | ||
| using Microsoft.Extensions.Configuration; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any reason this class is all top level rather than being a class?
| // Licensed to the .NET Foundation under one or more agreements. | ||
| // The .NET Foundation licenses this file to you under the MIT license. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also all top level would it not be better to have it be a class?
| /// <returns>The downloaded issues as an async enumerable collection of tuples containing the issue and its predicate-matched label (when only one matcing label is found).</returns> | ||
| public static async IAsyncEnumerable<(Issue Issue, string CategoryLabel, string ServiceLabel)> DownloadIssues( | ||
| string githubToken, | ||
| string org, string repo, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(nit)
| string org, string repo, | |
| string org, | |
| string repo, |
| using GraphQL; | ||
| using GraphQL.Client.Http; | ||
| using GraphQL.Client.Serializer.SystemTextJson; | ||
| using Octokit; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| using Octokit; |
Using graphql no octokit needed :)
| <PackageReference Include="GitHubJwt" /> | ||
| <PackageReference Include="GraphQL.Client" /> | ||
| <PackageReference Include="GraphQL.Client.Serializer.SystemTextJson" /> | ||
| <PackageReference Include="Octokit" /> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
won't need octokit here either
| @@ -0,0 +1,55 @@ | |||
| // Licensed to the .NET Foundation under one or more agreements. | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Class per file :) like the other comment
| @@ -0,0 +1,193 @@ | |||
| // Licensed to the .NET Foundation under one or more agreements. | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
top level but seeing as so much of the code is structured by classes, I think we should keep consistent and make this a class as well. (Unless its top level for some reason I don't see)
| @@ -0,0 +1,236 @@ | |||
| // Licensed to the .NET Foundation under one or more agreements. | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
top level
| return bestScore is not null ? (bestScore.Label, bestScore.Score) : ((string?)null, (float?)null); | ||
| } | ||
|
|
||
| class TestStats |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Put this in its own file
| @@ -0,0 +1,43 @@ | |||
| // Licensed to the .NET Foundation under one or more agreements. | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
top level
| <MicrosoftNETTestSdkVersion>16.11.0</MicrosoftNETTestSdkVersion> | ||
| </PropertyGroup> | ||
|
|
||
| <PropertyGroup> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this for local support of the Azure Function? I'm wondering if we should move this to a local props file that we ignore so that CI doesn't get confused.
| Task<IModelHolder> CreateModelHolder(string owner, string repo, string modelType); | ||
| Task<IPredictor> GetPredictor(string owner, string repo, string modelType); | ||
| } | ||
| public class ModelHolderFactory : IModelHolderFactory |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please split into separate files. In general, .NET has a strong one-class-per-file norm that we'll want to stick with unless there's a type that can't stand on its own. In that case, we'd want to nest it in its parent.
| private readonly ConcurrentDictionary<(string, string, string), IModelHolder> models = new ConcurrentDictionary<(string, string, string), IModelHolder>(); | ||
| private readonly ILogger<ModelHolderFactory> logger; | ||
| private readonly IRepositoryConfigurationProvider configurationProvider; | ||
| private SemaphoreSlim sem = new SemaphoreSlim(1,1); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1. This is a good suggestion.
| @@ -0,0 +1,139 @@ | |||
| // Copyright (c) Microsoft Corporation. All rights reserved. | |||
| // Licensed under the MIT License. | |||
| using Microsoft.Extensions.Logging; | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| using Microsoft.Extensions.Logging; | |
| using Microsoft.Extensions.Logging; |
nit: We generally throw a blank between the header and statements as a convention.
| } | ||
| public class ModelHolderFactory : IModelHolderFactory | ||
| { | ||
| private readonly ConcurrentDictionary<(string, string, string), IModelHolder> models = new ConcurrentDictionary<(string, string, string), IModelHolder>(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a quick mention that if you wanted to avoid the "Look at mah Java!" feel, you can shorten up the declarations on either side. Nothing wrong with the verbose form if that's your preference.
private readonly ConcurrentDictionary<(string, string, string), IModelHolder> models = new(); private readonly var models = new ConcurrentDictionary<(string, string, string), IModelHolder>();| @@ -0,0 +1,7 @@ | |||
| <?xml version="1.0" encoding="utf-8"?> | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to remove this. Microsoft has a new policy that requires us to pass through a proxy for all NuGet pulls and this resets that.
| /// </summary> | ||
| /// <param name="text">The text to sanitize.</param> | ||
| /// <returns>The sanitized text.</returns> | ||
| public static string SanitizeText(string text) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can add .AsSpan() on text and save a bunch of allocations.
| @@ -0,0 +1,5 @@ | |||
| public enum LabelType | |||
| { | |||
| Category = 1, | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are we explicitly numbering? Generally, you don't want to do that unless you're actually using the numeric values. We also have an issue here as default(LabelType) ends up being an invalid because it gets initialized as (LabelType)0. We'll want to add an Unknown = 0 at least here.
| public enum LabelType | ||
| { | ||
| Category = 1, | ||
| Service = 2 | ||
| } No newline at end of file |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 . We do need that.
|
|
||
| public struct Args | ||
| { | ||
| public string? GitHubToken { get; set; } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks like a file that we borrowed. We'll want to update to our header. I was going to suggest using System.CommandLine but if we inherited this and we know it works, better to not.
|
GitHub is being really, really laggy today. I left some thoughts but didn't dig into the back half of the files yet. |
Added an
AzureSdkIssueGeneratorServiceFunction App that will reference azure-sdk-for-net issues to generate synthetic issues for other repos and add TSV file to blob storage for model training if needed.Replaced all old ML labeler processes with streamlined
IssueLabelerMLPipeline. Most of this pipeline is meant to be run locally for more control over model training processes. It includes:Downloader: Download applicable issues to local .tsv file.Trainer: Use downloaded issues (along with synthetic issues generated fromAzureSdkIssueGeneratorServiceif available) to train Category and Service models. This will also provide per label statistics as well as a list of labels that need more data for training. This list can be passed back intoAzureSdkIssueGeneratorServiceto generate issues with specific labels.Predictor: Use the category and service model zip files produced from Trainer to predict issues.Tester: Test models on all labeled issues for # of matches, mismatches, and no predictions. Also provides statistics on how models perform.DevOpsModelRetrainer: Process to be handed to DevOps team to periodically retrain models automatically. This download issues, train models along with synthetic data if any, then replace current models with new ones.Unified config setup.
Previously, most function app configs use
{org}/{repo}:{key}ordefaults:{key}. However, asModelHolderFactoryis a singleton and didn't have per repo config at runtime, we were doing roundabout logics to configure model files (check oldLegacyLabelerandIModelHolderFactoryLite.csfor details). Per repo config is now available toModelHolderFactory, which simplifies many configs and cut out unnecessary name translation/extraction logic.Simplified the flow from
AzureSdkIssueLabelerServicetoLabeler.Previously, there were many layers -
ModelHolder, ModelHolderFactory, ModelHolderFactoryLite, Labeler, LabelerLite, LegacyLabeler. ReplacedLegacyLabelerwithMLLabeler, simplified the rest down toModelHolderandModelHolderFactory.Previously,
ModelHolderwas also unaware of which label type it is, and the ordering of service vs category models inModelHolderFactorywas dictated by order listed indefaults:IssueModel.{repo}.BlobConfigNames. This made code unreadable without constantly referencing app config.ModelHoldernow hasmodelTypewhich makes ordering irrelevant, simplifies call to config and making code much more readable.Cleaned up codebase
Deleted all unused references and refactored functions.