[Rollout] Production rollout, 2026-01-28

.vault-config/dnceng-amg-int-kv.yaml

@@ -0,0 +1,24 @@
+storageLocation:
+  type: azure-key-vault
+  parameters:
+    subscription: a4fc5514-21a9-4296-bfaf-5c7ee7fa35d1
+    name: dnceng-amg-int-kv
+
+secrets:  
+  # API token for DotNet Status website
+  dotnet-build-bot-dotnet-eng-status-token:
+    type: text
+    parameters:
+      description: API token from https://dotneteng-status-staging.azurewebsites.net/ - Generated using dotnet-build-bot account
+
+  # Authorization header for Deployment Annotations datasource
+  dotneteng-status-auth-header:
+    type: text
+    parameters:
+      description: "Bearer token for status API - Format: Bearer <dotnet-build-bot-dotnet-eng-status-token>"
+
+  # Teams webhook URL for alert notifications
+  fr-bot-notifications-teams-notification-url:
+    type: text
+    parameters:
+      description: Teams Incoming Webhook URL - Do not rotate

.vault-config/dnceng-amg-prod-kv.yaml

@@ -0,0 +1,24 @@
+storageLocation:
+  type: azure-key-vault
+  parameters:
+    subscription: a4fc5514-21a9-4296-bfaf-5c7ee7fa35d1
+    name: dnceng-amg-prod-kv
+
+secrets:
+  # API token for DotNet Status website
+  dotnet-build-bot-dotnet-eng-status-token:
+    type: text
+    parameters:
+      description: API token from https://dotneteng-status.azurewebsites.net/ - Generated using dotnet-build-bot account
+
+  # Authorization header for Deployment Annotations datasource
+  dotneteng-status-auth-header:
+    type: text
+    parameters:
+      description: "Bearer token for status API - Format: Bearer <dotnet-build-bot-dotnet-eng-status-token>"
+
+  # Teams webhook URL for alert notifications
+  fr-bot-notifications-teams-notification-url:
+    type: text
+    parameters:
+      description: Teams Incoming Webhook URL - Do not rotate

Documentation/OnePagers/ops-triage-agent.md

@@ -0,0 +1,174 @@
+# DNCEng Operational Triage Agent — One Pager
+
+## Description
+Automated triage system for DNCEng Tasks under  
+`internal\.NET Engineering Services\Operations`, ensuring transparent and consistent priority assignment.
+
+The architecture is intentionally split into two layers with a **strict separation of responsibilities**:
+
+1. **MCP-Based Triage Tooling (Logic Layer — Pure, Side‑Effect‑Free)**  
+   - Implemented as one or more **local MCP servers**, distributed via **NuGet packages**  
+   - Encapsulates triage logic, rule evaluation, and reasoning generation  
+   - **Performs no Azure DevOps I/O**  
+   - **Requires no authentication, tokens, or permissions**  
+   - Deterministic, testable, and reusable across any environment (Azure job, CLI, Foundry)
+
+2. **Lightweight Agent Layer (Execution / Orchestration Layer — Side‑Effecting)**  
+   - Responsible for **all authentication and authorization**  
+   - Owns all Azure DevOps interactions (queries, writes, comments, updates)  
+   - Invokes MCP logic and applies its results  
+   - Initially an **Azure-scheduled job**, but architected to support **CLI** or **Foundry-based agents** using the same MCP tooling
+
+By design, **all security scopes remain in the agent**, eliminating the need to manage authentication or ADO permissions inside MCP tooling.  With this model, the agent is treated with the same security posture as a user would.
+
+## Deliverable at End of Sprint
+A functioning system where:
+- MCP servers generate pure triage decisions and structured reasoning
+- The agent:
+  - Authenticates to Azure DevOps
+  - Queries for candidate items
+  - Invokes MCP tooling
+  - Writes updates back to Azure DevOps
+- Unprioritized DNCEng Tasks are automatically triaged
+- Items assigned to the agent’s service principal are re-triaged and unassigned
+- A read-only “Triage agent” section is created and appended to
+- Triage decisions follow a versioned wiki-based triage bar
+
+### Metrics
+
+As a metric for evaluating the effectiveness of the triage agent, we should review all work items which are triaged by the agent.  If a user, or users, or triage team have examined an issue and adjusted the assigned priority, then we can quickly generate a measure of how many issues the agent triaged correctly / incorrectly on an on-going basis.  There will be scenarios where the agent actually triaged appropriately and the work item lacked sufficient context for a different priority assignment but a user has additional information and decides to re-triage based on external knowledge but hopefully those occurences are rare.  If it turns out to be a frequent case where external knowledge is not captured in the work item, then we should re-examine our processes to try to encourage more value be added to the work item itself.
+
+## Motivation
+Azure DevOps Tags/Labels must remain strictly **filtering/query metadata**, not programmatic switches.
+
+The team adheres to the principle that **Azure DevOps Tags/Labels must not be used as operational signals**.  Use of Tags / Labels for programmatic decision making / operations leads to non-intuitive experiences and area expertise requirements.
+
+Tags should remain strictly for **querying and filtering**, not automation triggers.
+
+Separating concerns between:
+- **Pure logic (MCP tooling)**  
+- **Execution agent (Azure job, CLI, Foundry)**  
+
+ensures a predictable, inspectable system that avoids “hidden behaviors” and reduces dependency on tribal knowledge.
+
+The system’s architecture reinforces this by:
+- Making MCP tooling **pure business logic with zero dependencies on authentication or external APIs**
+- Keeping **all operational and security responsibilities in the agent layer**
+- Preventing accidental coupling between triage rules and ADO access patterns
+
+This separation:
+- Removes the need for security handling in MCP tooling  
+- Promotes deterministic, portable logic  
+- Preserves a predictable and auditable automation model
+
+## Architectural Overview
+
+### **MCP Tooling (Logic Layer)**
+- Distributed as NuGet-based local MCP servers
+- Purely logical: no network calls, no credentials  
+- Takes structured inputs:
+  - Work item fields supplied by the agent
+  - Triage bar rules
+  - Contextual metadata
+- Outputs structured triage decisions:
+  - Priority recommendation  
+  - Reasoning and decision explanation  
+  - Alternate outcomes if additional data were provided  
+  - Triage bar version used  
+
+**Because MCP tooling is completely side‑effect‑free, it does not require ADO credentials, eliminating the need to manage authentication/authorization at the logic level.**
+
+### **Agent Layer (Execution / Orchestration Layer)**
+The agent is the **only** component that:
+
+- Authenticates to Azure DevOps  
+- Holds authorization scopes  
+- Performs all ADO reads (queries)  
+- Performs all ADO writes (priority updates, comments, section updates)  
+- Handles throttling, retries, and permission boundaries  
+
+The agent simply orchestrates:
+1. Collecting ADO data  
+2. Passing structured inputs to MCP servers  
+3. Applying MCP outputs back to ADO  
+
+### **Why ADO I/O Lives Exclusively in the Agent Layer**
+Keeping all authentication and authorization in the agent:
+
+- Ensures MCP servers remain universal and environment-agnostic  
+- Prevents credential sprawl (no tokens in libraries/packages)  
+- Eliminates the need for MCP security review or privileged access  
+- Makes MCP tooling safe to run in local tools, pipelines, and Foundry agents  
+- Reduces operational blast radius  
+
+**This security simplification is a core architectural advantage.**
+
+## Approach
+
+### 1. Automatic Triage of Unprioritized Tasks
+Agent queries ADO → sends data to MCP logic → applies MCP decision.
+
+### 2. Re‑Triage of Tasks Assigned to the Agent
+Agent re-evaluates tasks assigned to the service principal, unassigns them, and updates ADO.
+
+### 3. “Triage Agent” Read‑Only Section
+Agent writes an append-only, structured section including:
+- MCP reasoning  
+- Priority determination  
+- Data gaps  
+- Bar version used  
+- Timestamp  
+- Wiki link  
+
+### 3a. **Brief Agent Comment on the Work Item**
+Whenever the agent triages or re‑triages a work item, it posts a **short comment** indicating that triage occurred,  
+and includes a **direct link to the “Triage agent” section** where the full historical record and detailed reasoning are stored.
+
+This ensures:
+- Comments stay concise  
+- Users immediately know triage occurred  
+- Full fidelity details remain centralized and well‑structured  
+
+### 4. Wiki‑Published, Versioned Triage Bar
+- Triage bar lives in DNCEng wiki  
+- MCP tooling interprets it  
+- Updating the bar allows all items to be re‑triaged cleanly  
+
+## Security / Telemetry / Test Coverage / Safe Deployment
+
+### **Security**
+- **All authentication and authorization handled by the agent**  
+- MCP tooling requires **no credentials and has no permissions**  
+- No external calls from MCP servers  
+
+This reduces risk and simplifies compliance.
+
+### **Telemetry**
+- MCP evaluations (inputs/outputs)  
+- Agent‑applied operations  
+- Re-triage deltas  
+- User overrides  
+
+### **Test Coverage**
+- MCP: deterministic unit + integration tests  
+- Agent: orchestration tests, mocked MCP calls  
+
+### **Safe Deployment**
+- Wiki-based rule versioning  
+- Dry-run modes  
+- Controlled rollout via agent configuration  
+
+
+## Metrics
+
+## Task Breakdown
+
+| Testable Chunk                   | Tasks                                                       | Cost |
+|----------------------------------|-------------------------------------------------------------|------|
+| MCP triage engine                | Pure logic + triage bar interpreter                        | 2–3 days |
+| MCP server infrastructure        | Local MCP server + NuGet packaging                          | 2–3 days |
+| Agent orchestration logic        | Azure job + ADO I/O + MCP invocation                       | 2–3 days |
+| Triage agent section writer      | Append-only structured section                              | 2 days |
+| Commenting + assignment handling | Minimal comments + unassign logic                           | 1–2 days |
+| Telemetry + monitoring           | Metrics, dashboards                                         | 1–2 days |
+| Testing & rollout                | Unit, integration, and dry-run validation                   | 3 days |

azure-pipelines-pr.yml

@@ -105,4 +105,4 @@ stages:
             dotnet run --project src/SecretManager/Microsoft.DncEng.SecretManager -- validate-all -b src @manifestArgs
           displayName: Verify Secret Usages
 
-        - template: /eng/test.yaml
+        - template: /eng/test.yaml

azure-pipelines.yml

@@ -7,6 +7,8 @@ variables:
   - name: _DotNetArtifactsCategory
     value: .NETCore
   - group: SDL_Settings
+  - name: ServiceConnectionName
+    value: 'Dotnet Engineering services'
 
 trigger:
   batch: true
@@ -195,6 +197,24 @@ extends:
                 contents: '*'
                 targetFolder: $(Build.ArtifactStagingDirectory)\eng
 
+            - task: AzureCLI@2
+              displayName: 'Validate Grafana Bicep Template'
+              inputs:
+                azureSubscription: '$(ServiceConnectionName)'
+                scriptType: 'ps'
+                scriptLocation: 'inlineScript'
+                inlineScript: |
+                  Write-Host "Validating Grafana Bicep template..."
+                  if (!(Test-Path "eng/deployment/azure-managed-grafana.bicep")) {
+                    throw "Bicep template not found: azure-managed-grafana.bicep"
+                  }
+
+                  az bicep build --file eng/deployment/azure-managed-grafana.bicep
+                  if ($LASTEXITCODE -ne 0) {
+                    throw "Bicep template validation failed"
+                  }
+                  Write-Host "SUCCESS: Bicep template validation successful"
+
     - template: /eng/common/templates-official/post-build/post-build.yml@self
       parameters:
         enableSymbolValidation: false
@@ -225,20 +245,26 @@ extends:
             PublishProfile: Int
             ServiceConnectionName: NetHelixStaging
             StatusVariableGroup: DotNetStatus Staging
-            GrafanaHost: https://dotnet-eng-grafana-staging.westus2.cloudapp.azure.com
-            GrafanaKeyVault: dotnet-grafana-staging
-            GrafanaVariableGroup: Dotnet-Grafana-Staging
             ServiceConnectionClientId: 57f299da-15de-4117-b8f6-7c10451926f0
             ServiceConnectionId: 7829de7e-fb4e-4118-8370-475d6bc61905
+            AMGServiceConnectionName: 'Dotnet Engineering services'
+            AMGServiceConnectionId: dd8c2cfc-b9c9-452c-a168-ccd4240ada55
+            AMGServiceConnectionClientId: fc1eb341-aea4-4a11-8f80-d14b8775b2ba
+            AMGDeploymentEnvironment: Staging
+            AMGGrafanaWorkspaceName: dnceng-grafana-staging
+            AMGGrafanaKeyVault: dnceng-amg-int-kv
           ${{ else }}:
             DeploymentEnvironment: Production
             DotNetStatusAppName: dotneteng-status
             DotNetStatusEndpoint: .NET Engineering Deployment Notification - Production
             PublishProfile: Prod
             ServiceConnectionName: NetHelix
             StatusVariableGroup: DotNetStatus Production
-            GrafanaHost: https://dotnet-eng-grafana.westus2.cloudapp.azure.com
-            GrafanaKeyVault: dotnet-grafana
-            GrafanaVariableGroup: Dotnet-Grafana-Production
             ServiceConnectionClientId: fc1eb341-aea4-4a11-8f80-d14b8775b2ba
-            ServiceConnectionId: 4a511f6f-b538-48e6-a389-207e430634d1
+            ServiceConnectionId: 4a511f6f-b538-48e6-a389-207e430634d1
+            AMGServiceConnectionName: 'Dotnet Engineering services'
+            AMGServiceConnectionId: dd8c2cfc-b9c9-452c-a168-ccd4240ada55
+            AMGServiceConnectionClientId: fc1eb341-aea4-4a11-8f80-d14b8775b2ba
+            AMGDeploymentEnvironment: Production
+            AMGGrafanaWorkspaceName: dnceng-grafana
+            AMGGrafanaKeyVault: dnceng-amg-prod-kv