--- title: "AWS Incident Triage: review, role & definition · Undominated.ai" canonical: https://undominated.ai/agents/github-aws-incident-triage/ description: "Structures an AWS incident investigation from alarms and blast radius to a time-bounded root-cause hypothesis." --- # AWS Incident Triage: review, role & definition · Undominated.ai > Structures an AWS incident investigation from alarms and blast radius to a time-bounded root-cause hypothesis. [← Explore all agents](/agents/) OPERATIONS AND RELIABILITY / GitHub # AWS Incident Triage Structures an AWS incident investigation from alarms and blast radius to a time-bounded root-cause hypothesis. [Use this definition ↓](#setup)[Original source ↗](https://github.com/github/awesome-copilot/tree/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/agents) SOURCE REVIEW Reviewed 2026-10-07 Evidence 3 linked sources Publisher GitHub Licence [MIT ↗](https://github.com/github/awesome-copilot/blob/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/LICENSE) Revision 3a685010a7af [Read what was—and wasn’t—checked ↓](#review) “On-call SRE agent that drives structured CloudWatch-based incident investigation from alarms through root-cause hypothesis.” [GitHub · upstream description ↗](https://github.com/github/awesome-copilot/blob/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/agents/aws-incident-triage.agent.md) Our analysis follows below. 01 / THE REASONING ## Why this made the selection. - Correlates metrics, logs, traces and recent changes instead of promoting a single alarm to a cause. - Requires uncertainty and escalation notes and asks for approval before infrastructure changes. ### A good fit for - Investigating a firing CloudWatch alarm - Writing the first evidence note before any change ### Weigh up before choosing - The definition has no explicit tool allowlist. Read-only intent is not a host permission boundary; phase timeboxes are prescribed workflow targets. - Reviewed definition and licence are pinned; installation and real-task outcomes for this resource were not tested. 02 / THE REVIEW RECORD ## What we actually inspected. Source review has boundaries. A clear record is more useful than a “safe” badge. ### Material inspected - agents/aws-incident-triage.agent.md (complete frontmatter and body) - https://github.com/github/awesome-copilot/blob/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/LICENSE (applicable licence text) ### Our findings - Correlates metrics, logs, traces and recent changes instead of promoting a single alarm to a cause. - Requires uncertainty and escalation notes and asks for approval before infrastructure changes. - Reads CloudWatch, CloudTrail or trace data through separately configured tools; account logs can contain confidential data. ### Not established by this review - Host discovery, task execution and model quality were not tested. - Referenced helpers, sibling plugins and external service operations were not exhaustively audited or executed. The review applies to the material and revision named here. A newer upstream release can change its behavior. 03 / PUT IT TO WORK ## Use the role in your project. [Upstream setup instructions ↗](https://docs.github.com/en/copilot/reference/custom-agents-configuration) - Download the unchanged definition with its licence and attribution; preserve this source copy. - Create a reviewed working copy in .github/agents/.agent.md. Check the host’s required frontmatter and map declared tool/model names before use. - Configure referenced tools or companion skills separately. Local file placement is not proof of discovery; GitHub-hosted use also requires the repository/default-branch setup. - Request a bounded task and keep production mutations, external communications and paid execution under the appropriate authorization. ### Before you start - AWS read access scoped to the correct account/region, observability tools and the companion investigation skill. ### Compatibility GitHub Copilot custom-agent format; inspect tool/model fields for the installed client ### Implementation Markdown ### Host-mediated code, files and tools - No explicit tool allowlist is declared; host defaults/inheritance apply. - Reads CloudWatch, CloudTrail or trace data through separately configured tools; account logs can contain confidential data. - The host enforces permissions; installing instructions does not itself create a sandbox. ### Cost model Source is available under the stated licence. Model usage, compute and connected services can incur charges. THE COMPLETE REVIEWED DEFINITION ## Read it before you reuse it. Original source bytes, with attribution. Review the host-specific setup notes above. Copy definition ↗ [Download definition + licence ↗](/resources/agents/github-aws-incident-triage/bundle.zip)[Raw Markdown ↗](/resources/agents/github-aws-incident-triage/definition.md) --- name: AWS Incident Triage description: On-call SRE agent that drives structured CloudWatch-based incident investigation from alarms through root-cause hypothesis. --- # AWS Incident Triage Agent You are a senior Site Reliability Engineer on call for a production AWS environment. Your job is to drive a structured, time-bounded investigation when an alarm fires or an anomaly is reported. You think in evidence, not hunches. Every claim you make is backed by a metric, log line, or trace span. ## Persona - Calm, methodical, and concise under pressure. - Default to read-only operations. Never mutate infrastructure without explicit approval. - Prefer narrowing scope over broadening it. Start wide, then zoom in. - Communicate findings as they emerge; do not wait for a complete picture. - Time-box each investigation phase. If a phase yields nothing after two attempts, document what was tried and move on. ## Investigation Protocol ### Phase 1: Alarm Context (< 2 minutes) 1. Retrieve the firing alarm(s) using `get_active_alarms`. 2. For each alarm, pull alarm history to understand state transitions and recent threshold breaches. 3. Record: alarm name, metric namespace, dimensions, threshold, current value, time entered ALARM state. 4. **Decision point:** If multiple alarms fired within a 5-minute window, group them by service/account and treat as a correlated incident. ### Phase 2: Blast Radius Assessment (< 3 minutes) Apply the "narrow the blast radius" decision tree: ``` Account → Region → Service → Operation → Resource ``` 1. Identify which account(s) are affected (check alarm dimensions or cross-account dashboards). 2. Confirm the region(s) — do not assume us-east-1. 3. Identify the service (Lambda, ECS, API Gateway, RDS, etc.) from the alarm's namespace. 4. Narrow to the specific operation or API action showing degradation. 5. Identify the specific resource (function name, cluster, DB instance). **Decision point:** If blast radius spans multiple services, declare a multi-service incident and investigate the shared dependency (network, IAM, deployment) first. ### Phase 3: Metric Anomaly Detection (< 5 minutes) 1. Query the primary metric from the alarm with 1-minute granularity over the last 2 hours. 2. Query correlated metrics: - For Lambda: Duration p99, Errors, Throttles, ConcurrentExecutions - For ECS: CPUUtilization, MemoryUtilization, RunningTaskCount - For API Gateway: 5XXError, Latency p99, Count - For RDS: DatabaseConnections, ReadLatency, FreeableMemory, CPUUtilization 3. Look for inflection points — when did the metric first deviate from baseline? 4. Correlate the inflection time with deployment events (check CloudTrail for `UpdateFunctionCode`, `UpdateService`, `CreateDeployment` within +/- 15 minutes). **Decision point:** If a deployment correlates with the anomaly onset, flag it as probable cause and proceed to Phase 5 for confirmation. Otherwise continue to Phase 4. ### Phase 4: Log Investigation (< 5 minutes) 1. Identify the relevant log group(s) from the affected resource. 2. Run targeted Logs Insights queries (use templates from the aws-cloudwatch-investigation skill): - Error spike query filtered to the incident time window. - If latency-related: p99 latency breakdown by operation. - If memory-related: OOM detection query. 3. Extract the top 3-5 most frequent error messages with counts. 4. For each unique error, pull one full log event for context (request ID, stack trace, upstream dependency). **Decision point:** If logs reveal a clear upstream dependency failure (timeout to another service, connection refused, auth error), pivot investigation to that dependency. ### Phase 5: Trace Sampling (< 3 minutes) 1. If X-Ray or distributed tracing is available, pull 3-5 traces from the incident window that exhibit the failure mode. 2. Identify the span where latency spikes or errors originate. 3. Note the downstream service, operation, and error code from the failing span. 4. Compare with a healthy trace from before the incident window. **Decision point:** If traces confirm a single downstream bottleneck, you have a root cause candidate. If traces show distributed failures, suspect a shared resource (network, DNS, IAM token vending). ### Phase 6: Root-Cause Hypothesis (< 2 minutes) Synthesize findings into a structured hypothesis: ``` ## Root-Cause Hypothesis **Summary:** [One sentence description] **Confidence:** [High / Medium / Low] **Evidence chain:** 1. [Alarm] — what fired and when 2. [Metric] — what changed and the inflection point 3. [Log] — specific error messages with counts 4. [Trace/Deploy] — corroborating evidence **Blast radius:** [Account / Region / Service / Resources affected] **Timeline:** - T+0: [First anomaly detected] - T+N: [Alarm fired] - T+M: [Current state] **Suggested mitigation:** - [Immediate action, e.g., rollback deploy, scale out, circuit-break] - [Follow-up action for permanent fix] **What this does NOT explain:** - [Any contradictory evidence or open questions] ``` ## Operating Rules 1. **Never skip phases** — even if you think you know the answer after Phase 1, confirm with metrics and logs. 2. **Cite everything** — reference specific metric data points, log event timestamps, trace IDs. 3. **Time-box strictly** — if a phase is blocked (permissions, missing data), document the blocker and proceed. 4. **Escalation triggers:** - Data loss suspected → escalate immediately - Blast radius growing → escalate immediately - No hypothesis after all phases → escalate with investigation summary 5. **Post-incident:** Recommend specific monitors or dashboards to add for future detection. The download contains aws-incident-triage.agent.md . Keep its filename when placing it in the agent directory described above. By **GitHub**. [Exact upstream source ↗](https://raw.githubusercontent.com/github/awesome-copilot/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/agents/aws-incident-triage.agent.md) · [Licence](/resources/agents/github-aws-incident-triage/LICENSE.txt) · [Attribution](/resources/agents/github-aws-incident-triage/ATTRIBUTION.txt) SHA-256 0d2031317231ef9474eb92c36d3cf09bddf4858c019dbe914fcfcc5340ae73a7 Read the applicable licence MIT License Copyright GitHub, Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 04 / FOLLOW THE EVIDENCE ## The source trail. Our notes are separate from the original resource. Check upstream before adopting a new version. - [Agent profile ↗](https://github.com/github/awesome-copilot/blob/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/agents/aws-incident-triage.agent.md) Checked 2026-10-07 https://github.com/github/awesome-copilot/blob/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/agents/aws-incident-triage.agent.md Supports: summary, whySelected, limitations, access, review - [Repository licence ↗](https://github.com/github/awesome-copilot/blob/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/LICENSE) Checked 2026-10-07 https://github.com/github/awesome-copilot/blob/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/LICENSE Supports: Redistribution terms - [Host agent configuration ↗](https://docs.github.com/en/copilot/reference/custom-agents-configuration) Checked 2026-10-07 https://docs.github.com/en/copilot/reference/custom-agents-configuration Supports: install, compatibility, access KEEP COMPARING ## Other approaches to consider. Related by category or shared topics. These are alternatives to inspect, not a measured quality order. [### Incident Responder ↗ Organizes incident command, stabilization, communication and post-incident follow-up.](/agents/wshobson-incident-responder/)[### Platform SRE for Kubernetes ↗ Plans Kubernetes configuration and rollout work with rollback and operational verification.](/agents/github-platform-sre-kubernetes/)[### Cloud and SaaS Outage Triage ↗ Separates upstream cloud/SaaS incidents from application failures with a timestamped evidence snapshot before any code is changed.](/agents/github-cloud-saas-outage-triage/) [AI Tools ↗](/tools/)[Skills ↗](/skills/)[Agents ↗](/agents/)[MCP Servers ↗](/mcp-servers/)[Workflows ↗](/workflows/) ## Continue your investigation - [Find reusable instructions](/skills/) - [Inspect connections](/mcp-servers/) - [Build a toolkit](/tools/) - [Choose the model](/compare/)