Skip to content

P1 incident

Severity 1 — restore service first, investigate later

Owner: on-call-teams · Auto-drafted from ansible-p1-runbooks.

First 5 minutes

  1. Confirm blast radius (dashboard link in the alert)
  2. Declare incident / open bridge if customer-impacting
  3. Page on-call: incident-p1 channel

Diagnosis

  • Check the owning service dashboard
  • Identify the failing component / hot resource

Mitigation

ansible-playbook remediate.yml --limit <host>

China residency

In China, mitigation stays in-region — use the local secondary only, never cross-border failover.

Verification

  • Service restored; error rate back to baseline

Escalation

  • L2: platform-devops · L3 / vendor: see contacts in the alert

Post-incident

  • Timeline captured
  • Root-cause ticket raised
  • Runbook updated if steps changed