feat(sre-agent): Packaged deployment, Teams delivery, Autonomous mode, UAT-validated scheduled tasks#2111
feat(sre-agent): Packaged deployment, Teams delivery, Autonomous mode, UAT-validated scheduled tasks#2111
Conversation
…, UAT-validated scheduled tasks - Add deploy.sh and deploy.ps1 as single packaged deployment entrypoints - --clone-env, --replace, --destroy modes - B2B tenant fix: az account set before azd up - Clone-env identity leak fix (unset AZURE_ENV_NAME/RG after sourcing) - Switch agent action mode from Review to Autonomous - Review mode blocked all autonomous tool execution including Teams posts - Autonomous mode required for scheduled task delivery without human approval - All 9 scheduled tasks updated with: - Phase 0: read knowledge base before starting work - Teams/knowledge split: financial results to Teams only, operational learnings to knowledge only - PostTeamsMessage delivery instruction in every task - Fix duplicate task bug: post-provision uses srectl scheduledtask apply (idempotent) - Add 3 knowledge docs for agent self-improvement: - onboarding-recommendations.md - teams-notification-guide.md (PostTeamsMessage vs 403 on Graph API) - known-issues-and-workarounds.md (7 error categories from UAT) - Enable experimentalSettings (EnableV2AgentLoop, EnableWorkspaceTools) - Enable execute_python on analytical subagents - README updated as living product contract - 17 tests passing (TC-4.1a, TC-4.2a enforce packaging and delivery contracts) UAT validated: all 9 scheduled tasks executed and delivered to Teams channel. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…wledge/Teams split, memory model fix - Add docs-mslearn/toolkit/hubs/configure-sre.md (duo-reviewed against configure-ai.md and AGENTS.md) - Add to TOC.yml after Configure AI in Copilot Studio - Switch Bicep from Review to Autonomous mode - Fix README to reflect Autonomous mode - Add 3 knowledge docs: onboarding, Teams notification guide, known issues and workarounds - Fix scheduled task persistence: use #remember for operational notes, not git commits - Replace 'save to knowledge base' with #remember in all 9 task prompts - Add Phase 0 (read knowledge before starting) to all 9 tasks - Enforce knowledge/Teams split: financial results to Teams only, operational learnings to memory only - Fix post-provision: srectl scheduledtask apply (idempotent) instead of create - Add known-issues-and-workarounds.md section 0: correct persistence model Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a new redistributable FinOps SRE Agent template under src/templates/sre-agent/, including deploy automation, Bicep infrastructure, Kusto tool catalog, scheduled tasks, knowledge docs, and template validation tests, plus an MSLearn doc entry for configuring the agent with FinOps hubs.
Changes:
- Introduces 21 Kusto-backed
ExtendedAgentTooldefinitions and wires them into multiple autonomous scheduled tasks. - Adds packaged deployment and post-provision automation (
deploy.*,post-provision.*) plus full Bicep infra modules (agent, identity, monitoring, RBAC, optional ADX role). - Adds template tests and documentation updates (template README, catalog, UAT plan, and MSLearn article + TOC entry).
Reviewed changes
Copilot reviewed 66 out of 66 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| src/templates/sre-agent/tools/top-services-by-cost.yaml | New Kusto tool for top services by effective cost. |
| src/templates/sre-agent/tools/top-resource-types-by-cost.yaml | New Kusto tool for top resource types by cost and count. |
| src/templates/sre-agent/tools/top-resource-groups-by-cost.yaml | New Kusto tool for top resource groups by effective cost. |
| src/templates/sre-agent/tools/top-other-transactions.yaml | New Kusto tool to surface large non-usage/non-commitment purchases. |
| src/templates/sre-agent/tools/top-commitment-transactions.yaml | New Kusto tool for commitment purchase transactions with helper columns. |
| src/templates/sre-agent/tools/service-price-benchmarking.yaml | New Kusto tool to benchmark list/contracted/effective costs and savings. |
| src/templates/sre-agent/tools/savings-summary-report.yaml | New Kusto tool to summarize savings and effective savings rate. |
| src/templates/sre-agent/tools/reservation-recommendation-breakdown.yaml | New Kusto tool to analyze reservation recommendations and break-even. |
| src/templates/sre-agent/tools/quarterly-cost-by-resource-group.yaml | New Kusto tool for quarterly-style RG cost rows. |
| src/templates/sre-agent/tools/monthly-cost-trend.yaml | New Kusto tool for monthly billed/effective cost trends. |
| src/templates/sre-agent/tools/monthly-cost-change-percentage.yaml | New Kusto tool for month-over-month cost change percentages. |
| src/templates/sre-agent/tools/costs-enriched-base.yaml | New “base” enriched Costs() query for broader analytics. |
| src/templates/sre-agent/tools/cost-forecasting-model.yaml | New Kusto time-series forecasting tool. |
| src/templates/sre-agent/tools/cost-by-region-trend.yaml | New Kusto tool for regional effective-cost distribution. |
| src/templates/sre-agent/tools/cost-by-financial-hierarchy.yaml | New Kusto tool for hierarchy showback with percent-of-total. |
| src/templates/sre-agent/tools/cost-anomaly-detection.yaml | New Kusto anomaly detection time-series tool. |
| src/templates/sre-agent/tools/commitment-discount-utilization.yaml | New Kusto tool for commitment utilization mix and percent-of-total. |
| src/templates/sre-agent/tools/ai-token-usage-breakdown.yaml | New Kusto tool for Azure OpenAI token/cost breakdown and unit costs. |
| src/templates/sre-agent/tools/ai-model-cost-comparison.yaml | New Kusto tool comparing cost per 1K tokens by model. |
| src/templates/sre-agent/tools/ai-daily-trend.yaml | New Kusto tool for daily AI token/cost trends. |
| src/templates/sre-agent/tools/ai-cost-by-application.yaml | New Kusto tool allocating AI spend by tags (app/team/env/cost center). |
| src/templates/sre-agent/tests/sprint1-artifacts.test.mjs | Adds template validation tests for agents/skills/connectors/infra/scripts/tasks. |
| src/templates/sre-agent/sre-config/scheduled-tasks/ytd-report.yaml | New autonomous YTD finance reporting scheduled task prompt. |
| src/templates/sre-agent/sre-config/scheduled-tasks/mom-report.yaml | New autonomous MoM report scheduled task prompt. |
| src/templates/sre-agent/sre-config/scheduled-tasks/hubs-health-check.yaml | New daily hubs health-check scheduled task prompt. |
| src/templates/sre-agent/sre-config/scheduled-tasks/cost-optimization.yaml | New cost optimization scheduled task prompt (advisor/orphaned/commitments). |
| src/templates/sre-agent/sre-config/scheduled-tasks/capacity-weekly-supply-review.yaml | New weekly capacity supply review scheduled task prompt. |
| src/templates/sre-agent/sre-config/scheduled-tasks/capacity-quarterly-strategy.yaml | New quarterly capacity strategy scheduled task prompt. |
| src/templates/sre-agent/sre-config/scheduled-tasks/capacity-monthly-planning.yaml | New monthly capacity planning scheduled task prompt. |
| src/templates/sre-agent/sre-config/scheduled-tasks/capacity-daily-monitor.yaml | New daily capacity monitor scheduled task prompt. |
| src/templates/sre-agent/sre-config/scheduled-tasks/ai-workload-cost-analysis.yaml | New monthly AI workload cost analysis scheduled task prompt. |
| src/templates/sre-agent/sre-config/knowledge/teams-notification-guide.md | Adds guidance for Teams delivery via built-in PostTeamsMessage tool. |
| src/templates/sre-agent/sre-config/knowledge/onboarding-recommendations.md | Adds onboarding guidance and connector setup constraints. |
| src/templates/sre-agent/sre-config/knowledge/known-issues-and-workarounds.md | Adds UAT-derived known issues/workarounds and data persistence rules. |
| src/templates/sre-agent/sre-config/connectors/finops-hub-kusto.yaml | Adds DataConnector YAML for FinOps Hub Kusto connector contract reference. |
| src/templates/sre-agent/sre-config/agents/ftk-hubs-agent.yaml | Adds/updates hubs specialist agent definition. |
| src/templates/sre-agent/sre-config/agents/ftk-database-query.yaml | Adds/updates KQL specialist agent with tool access list. |
| src/templates/sre-agent/sre-config/agents/finops-practitioner.yaml | Adds/updates FinOps practitioner agent with tools and handoffs. |
| src/templates/sre-agent/sre-config/agents/chief-financial-officer.yaml | Adds/updates CFO agent with tools for finance narratives. |
| src/templates/sre-agent/sre-config/agents/azure-capacity-manager.yaml | Adds/updates capacity manager agent with quota/capacity guardrails. |
| src/templates/sre-agent/scripts/post-provision.sh | Adds bash post-provision automation to apply repo artifacts via srectl. |
| src/templates/sre-agent/scripts/post-provision.ps1 | Adds PowerShell post-provision automation to apply repo artifacts via srectl. |
| src/templates/sre-agent/scripts/deploy.sh | Adds packaged bash deploy wrapper around azd environment workflow. |
| src/templates/sre-agent/scripts/deploy.ps1 | Adds packaged PowerShell deploy wrapper around azd environment workflow. |
| src/templates/sre-agent/plan.md | Adds UAT plan and status tracking for template completeness. |
| src/templates/sre-agent/infra/bicep/resources.bicep | Adds RG-scoped orchestrator wiring identity/monitoring/agent modules. |
| src/templates/sre-agent/infra/bicep/modules/subscription-rbac.bicep | Adds subscription-scope role assignments (Reader + Monitoring Contributor). |
| src/templates/sre-agent/infra/bicep/modules/sre-agent.bicep | Adds SRE Agent resource deployment module + optional data connector resource. |
| src/templates/sre-agent/infra/bicep/modules/monitoring.bicep | Adds Log Analytics + App Insights resources for agent telemetry. |
| src/templates/sre-agent/infra/bicep/modules/identity.bicep | Adds UAMI creation module and outputs. |
| src/templates/sre-agent/infra/bicep/modules/adx-role.json | Adds ARM JSON for ADX AllDatabasesViewer principal assignment (portal flow). |
| src/templates/sre-agent/infra/bicep/modules/adx-role.bicep | Adds Bicep module to assign ADX AllDatabasesViewer. |
| src/templates/sre-agent/infra/bicep/main.parameters.json | Adds azd parameter mapping for main.bicep. |
| src/templates/sre-agent/infra/bicep/main.json | Checks in compiled ARM template output for subscription deployment. |
| src/templates/sre-agent/infra/bicep/main.bicep | Adds subscription entrypoint: RG creation, modules, RBAC, optional ADX role. |
| src/templates/sre-agent/azure.yaml | Adds azd template definition and postprovision hooks. |
| src/templates/sre-agent/README.md | Adds template README with deploy instructions, verification, and connector guidance. |
| src/templates/sre-agent/CATALOG.md | Adds scheduled task catalog and tool gap analysis. |
| src/templates/sre-agent/AGENTS.md | Adds agent instruction metadata and example connection details. |
| docs-mslearn/toolkit/hubs/configure-sre.md | Adds MSLearn article documenting deployment/config of SRE agent with hubs + Teams. |
| docs-mslearn/TOC.yml | Adds TOC entry for the new “Configure an SRE agent” doc. |
| .gitmodules | Adds azcapman submodule pointer for capacity-management lab content. |
- Remove real subscription/tenant IDs from AGENTS.md, replace with placeholders - Fix sre-agent.bicep comment: Review → Autonomous - Fix README architecture diagram: Review → Autonomous - Regenerate main.json from Bicep (Autonomous mode) - Remove PyYAML dependency from post-provision scripts (use grep/Select-String) - Fix KQL isempty()/isnotempty() on integer fields → isnull()/isnotnull() - Tighten #remember in MOM task to exclude financial figures - Update ms.date on changelog to 04/28/2026 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
@MSBrett I'm unable to start working on this because of repository rules that prevent me from pushing to the branch:
See the documentation for more details. |
…custom zone-peers role Sprint 3: 14 Python tools built via TDD gate pipeline (14/14 done). Capacity tools (from azcapman scripts): - vm-quota-usage: ARM REST compute usages with warning/critical thresholds - zone-mapping: ARM REST checkZonePeers for cross-sub zone alignment - capacity-reservation-groups: ARM REST CRG list with instanceView utilization - sku-availability: ARM REST compute SKUs with restriction parsing - data-freshness-check: Kusto REST querying Hub functions (Costs/Prices/Recommendations/Transactions) - resource-graph-query: ARM REST Resource Graph for orphaned resource detection - non-compute-quotas: ARM REST Storage/Network usages + Resource Graph counts - spot-placement-scores: ARM REST spot placement scores API - benefit-recommendations: Cost Management REST benefitRecommendations with pagination Governance tools (from azcapman scripts): - deploy-budget: ARM REST subscription budget creation with validation - deploy-bulk-budgets: Resource Graph discovery + per-sub budget deployment - deploy-anomaly-alert: Cost Management scheduled actions for anomaly detection - deploy-bulk-anomaly-alerts: Resource Graph discovery + per-sub alert deployment - suppress-advisor-recommendations: Advisor REST suppression with GUID-validated IDs Architecture decisions: - All tools use requests + ARM REST (not azure-mgmt-* SDKs — not pre-installed in sandbox) - All tools use ManagedIdentityCredential(client_id) for UAMI auth with DefaultAzureCredential fallback - Entry point is def main(**kwargs) per platform harness requirement - Write tools include input validation (GUID, email, enum, positive amount) - Custom role 'FinOps SRE Zone Peers Reader' added for checkZonePeers/action - data-freshness-check queries Hub functions (Costs/Prices/Recommendations/Transactions) not .show tables UAT: 6/7 tools pass on live agent. zone-mapping needs the custom role deployed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… version - Removed zone-mapping: requires AvailabilityZonePeering feature + custom role that cannot be verified end-to-end in this session. Documented the feature registration prerequisite in README for future reintroduction. - Removed spot-placement-scores: requires Microsoft.Compute/locations/ placementScores/generate/action not included in Reader role. - Fixed deploy-bulk-budgets, deploy-bulk-anomaly-alerts, suppress-advisor- recommendations: replaced azure.mgmt.resourcegraph SDK (not installed in sandbox) with ARM REST API for Resource Graph queries. - Fixed spot-placement-scores API version from 2024-11-01 (nonexistent) to 2024-06-01-preview before removal. - Moved custom role creation from Bicep to post-provision.sh for management group scope flexibility. - Added AvailabilityZonePeering feature registration to README prerequisites. - Verified vm-quota-usage returns 215 real quotas on live agent with UAMI auth. - All 12 remaining Python tools pass local testing (14/14 local, 12 deployed). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ew capacity audits T-4.1: Wired 12 Python tools into all 9 existing scheduled tasks. Each task prompt now references the relevant Python tools by name and instructs the agent to pass client_id for UAMI authentication. Replaced az CLI fallback references with Python tool equivalents. T-4.2: Created 9 new scheduled tasks from azcapman lab mapping: - ComputeUtilizationTrend (weekly): VM quota utilization trends - StoragePaasGrowthForecast (monthly): storage/PaaS growth projection - SkuAvailabilityAudit (weekly): SKU restrictions across regions - NonComputeQuotaAudit (weekly): storage/network quota health - BudgetCoverageAudit (monthly): subscriptions missing budgets - AlertCoverageAudit (monthly): subscriptions missing anomaly alerts - MonitoringScopeValidation (weekly): Hub coverage vs active subscriptions - BenefitRecommendationReview (weekly): reservation/savings plan recommendations - AdvisorSuppressionReview (monthly): stale/expired Advisor suppressions T-4.3: All 18 tasks deployed to live agent via srectl scheduledtask apply. Old duplicates cleaned up. Converted all tasks to v1 YAML format (snake_case) required by the srectl API. T-4.4: README updated with final inventory — 33 tools (21 Kusto + 12 Python), 18 scheduled tasks. T-4.5: post-provision.sh already uses directory glob iteration — no changes needed for new tasks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…se and attempt git commits Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…emove set -e Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…-provision tests - Added EnableCodeInterpreter and EnableMemory to experimentalSettings - Updated tests: removed repo connector assertions, relaxed apply-yaml to accept tool/agent apply Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ation as portal step Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| experimentalSettings: { | ||
| EnableWorkspaceTools: true | ||
| } |
There was a problem hiding this comment.
experimentalSettings is missing EnableV2AgentLoop: true. The repo test suite expects both EnableV2AgentLoop and EnableWorkspaceTools enabled for this template, and leaving it out will break TC-5.2 (and may prevent the newer agent loop behavior the template relies on). Add EnableV2AgentLoop: true alongside EnableWorkspaceTools: true.
| api_version: azuresre.ai/v1 | ||
| kind: ScheduledTask | ||
| metadata: | ||
| name: YTD | ||
| owner: | ||
| version: | ||
| tags: | ||
| spec: | ||
| name: YTD | ||
| cron_expression: '' | ||
| agent_prompt: >- | ||
| # Fiscal Year-to-Date Analysis |
There was a problem hiding this comment.
These sre-config/scheduled-tasks/scheduledtasks/** YAMLs appear to be exported runtime/task-instance artifacts (BOM at start of file, blank owner/version/tags, cron_expression: '', plus fields like start_time, duration_hours, and a separate cron field). They don’t match the canonical scheduled task definitions in sre-config/scheduled-tasks/*.yaml and (since post-provision doesn’t recurse) won’t be applied anyway. Consider removing this exported scheduledtasks/ tree from the redistributable template, or converting it into a clearly separated example/export location so it can’t be mistaken for deployable config.
| spec: | ||
| name: YTD | ||
| description: Fiscal year-to-date analysis with forecast through end of fiscal year | ||
| cron_expression: 0 9 1 * * |
There was a problem hiding this comment.
This task is scheduled for 0 9 1 * *, which is the same cron as other heavyweight monthly tasks (e.g., CapacityMonthlyPlanning and AdvisorSuppressionReview). If these run concurrently they can compete for tool quotas, increase execution time, and spam the Teams channel at the same time. Consider staggering monthly tasks (different minutes/hours) to reduce concurrency and improve reliability.
| spec: | ||
| name: CapacityMonthlyPlanning | ||
| description: Monthly capacity planning cycle — forecast demand, procurement pipeline, governance review | ||
| cron_expression: 0 9 1 * * |
There was a problem hiding this comment.
This task is scheduled for 0 9 1 * *, which is the same cron as other monthly tasks (e.g., YTD and AdvisorSuppressionReview). Running multiple long tasks at the same time can cause tool throttling and longer runtimes. Consider staggering the cron schedules to reduce concurrency.
| spec: | ||
| name: AdvisorSuppressionReview | ||
| description: Monthly review of active Advisor recommendation suppressions for stale or expired decisions | ||
| cron_expression: 0 9 1 * * |
There was a problem hiding this comment.
This task is scheduled for 0 9 1 * *, which overlaps with other monthly tasks (e.g., CapacityMonthlyPlanning and YTD). Consider staggering the cron schedule to avoid multiple monthly reports running concurrently and posting to Teams at the same time.
Add --dry-run flag to post-provision.sh and -DryRun switch to post-provision.ps1. When enabled, scripts log every srectl/az command that would execute without actually calling them. Output uses [DRY-RUN] prefix per category (skill, agent, tool, knowledge, scheduled task). Additional fixes from red-team audit: - Remove Add-RepoConnector from PS1 (parity with .sh safety removal; agent was searching full codebase and attempting git commits) - Unknown arguments now fail closed in .sh instead of being ignored Includes TDD test suite: - tests/test-dry-run.sh (shell contract test) - tests/Test-DryRun.Tests.ps1 (Pester 5 contract test, 5 cases) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Create new top-level docs section at docs-mslearn/toolkit/sre-agent/ with 9 pages covering the full FinOps SRE Agent feature surface: - overview.md: what it is, architecture, cost estimate - deploy.md: prerequisites, azd up, notifications, dry-run, regions - agents.md: 5 subagents, handoff model, 3 skills - tools.md: 33 tools (21 Kusto + 12 Python) grouped by domain - scheduled-tasks.md: 18 deployed tasks by cadence - knowledge.md: knowledge docs, memory system, persistence - security.md: RBAC, managed identity, least privilege, run modes - troubleshooting.md: B2B tenants, known issues, workarounds - template.md: Bicep params, azd outputs, script flags Also: - Add FinOps SRE Agent section to TOC after FinOps hubs - Remove old Configure SRE agent entry from hubs TOC - Add redirect from hubs/configure-sre to sre-agent/overview - Update cross-links in changelog and other docs - Set ms.reviewer to brettwil (micflan no longer at MS) Audited against MS Learn metadata requirements and partner-center-pr/finops/docfx.json globalMetadata contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…udio Add Azure SRE Agent and Azure Copilot Studio as AI integrations alongside the existing GitHub Copilot in the FinOps hubs architecture diagram. All three query the ADX / Fabric normalized data layer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# Conflicts: # docs-mslearn/toolkit/changelog.md
Add safer SRE Agent deployment wiring, remove unsafe teardown validation, and add Azure Data Explorer SKU preflight guidance for FinOps Hub deployments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
T-5.1: Convert all shell scripts to LF line endings and add .gitattributes to enforce LF for *.sh files. Fixes macOS bash 3.2 parse failures that completely blocked deployment. T-5.2: Add hub cluster URI validation to deploy.sh and deploy.ps1 — auto-appends /hub database suffix when missing. Update help text and README examples to document the required format. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Agents have repeatedly corrupted production deployments by running ad-hoc srectl/az commands against live resources outside the deploy.sh release process. Add explicit prohibition at the top of AGENTS.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Require existing FinOps Hub Kusto connections to resolve, provision, post-provision, and verify without false success. Add lock preflight, post-provision marker validation, and connector/ADX assignment checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Stop tracking .mcp.json and ignore it as local AI tooling configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
🛠️ Description
Add the FinOps SRE Agent as a new redistributable template under
src/templates/sre-agent/. This PR delivers the packaged deployment story, Teams notification delivery, and UAT-validated scheduled tasks.Key changes:
scripts/deploy.shandscripts/deploy.ps1as single entrypoints wrappingazd upwith--clone-env,--replace,--destroymodes and B2B tenant fixPostTeamsMessage. Knowledge doc documents the correct pattern (built-in tool, not Graph API)#remember/ synthesized knowledge only. Never financial data in knowledge.srectl scheduledtask applyinstead ofcreateprevents duplicates on re-runEnableV2AgentLoop+EnableWorkspaceToolsenabled by defaultexecute_pythonon analytical subagents for code interpreter accessdocs-mslearn/toolkit/hubs/configure-sre.mdduo-reviewed againstconfigure-ai.mdandAGENTS.mdUAT results: All 9 scheduled tasks executed against live
ftk-sre-test3deployment and delivered to Teams channel. 11 total Teams messages confirmed.Fixes #
📋 Checklist
🔬 How did you test this change?
📦 Deploy to test?
🙋♀️ Do any of the following that apply?
📑 Did you update
docs/changelog.md?📖 Did you update documentation?