A unified observability & resilience program for 2,000+ customers across three regions: replace poll-only SolarWinds with a self-hosted Prometheus + Grafana stack, make zone failure a shrug instead of an outage, and fix the ServiceNow change automation that keeps missing “Implementing.” Built as code, documented by itself.
*Modeled midpoint of documented ranges — see The Money. Every figure carries a [validate with finance] flag, mirroring the source plan.
The AZ1 outage wasn’t bad luck — it was a design we never chose on purpose, invisible to a monitoring tool that only polls every 15 minutes, wired to a change process that branches on free text. Each is fixable; together they’re the program.
In West Europe, tier-1 workloads lived entirely in Availability Zone 1 — one fault domain. AZ1 had a platform event and everything went hard-down until manual rebuild. A zone-capable region gives you three zones; we were using one.
100% SolarWinds, 15-minute polling, no webhooks, per-element licensing that rises every year, and thin depth on Kubernetes, Azure and network flow. It never surfaced the zone-affinity risk behind the outage.
Change automation keyed off typed status (“Implementing”, “impl”, typos) and record names. Free text can’t be matched deterministically — changes get mis-routed or missed, and renames create duplicates.
A self-hosted kube-prometheus-stack on AKS: exporters feed an HA Prometheus pair that fans out to alerting, dashboards and long-term storage. No managed/self-hosted split — because Azure Managed Prometheus & Grafana don’t exist in China (21Vianet).
China isn’t “another region,” it’s another cloud. The architecture makes the residency boundary physical: metrics can only aggregate where the law allows.
Full stack on AKS. May hold a read-only Thanos view of Brazil aggregates. Primary dashboards and alert routing.
Full stack on AKS. Metrics stay in-region under LGPD — only non-sensitive aggregates may leave to EMEA.
Fully isolated full stack — own Grafana, own identity. No cross-border scraping or querying, ever. Never point a Global-Azure Prometheus at China.
On-prem & small offices ship exporters to the nearest regional Prometheus, egress limited by network policy. This is also why managed SaaS (Datadog / Grafana Cloud) was rejected — cost at 2,000+ elements plus residency/egress conflicts with China and LGPD.
ADR-0001 makes zone-redundancy mandatory for tier-1 — and the Terraform refuses to deploy a single-zone tier-1 resource. The fix lives in code, not in a wiki nobody reads.
AKS user nodepools span zones [1,2,3] · topologySpreadConstraints + PDB tolerating one-zone loss.
Managed disks on ZRS · PostgreSQL Flexible Server zone-redundant HA (primary + standby in different zones).
Standard, zone-redundant Load Balancer & public IPs · no zonal single point of ingress.
modules/aks fails the Terraform plan if a tier-1 resource has < 2 zones. Plus a Prometheus alert for any single-zone tier-1 workload.
Automation now branches on ServiceNow’s structured choice fields — state and approval — keyed on the immutable sys_id. Renames become updates, not duplicates. ServiceNow stays the system of record; we only consume the read side.
if approval == "approved" and state == −1 (Implement):
→ draft/update runbook /30-runbooks/change/<number>.md
elif state in (3 Closed, 4 Canceled): → close draft
else: → hold pending, no publish
The old check parsed the typed status text and keyed on record name. “Implementing” arrived as “impl”, “in progress”, or a typo — so the real signal (state == −1) was missed, and renames spawned duplicates.
Detection is unchanged and belt-and-braces: a real-time Business Rule → n8n webhook, plus a nightly reconciliation querying sys_updated_on to catch anything missed.
The same events that drive monitoring and change flow also draft the wiki. AI drafts; humans merge. Nothing publishes without review, and every generated page cites its source.
Like-for-like on monitoring: we stop paying SolarWinds and stand up OSS on our own AKS. Resilience is a separate, deliberate investment — and one avoided AZ-class outage dwarfs it.
Range $45k–$215k. Net-positive from year one once the parallel-run window closes.
Zone-redundancy adds $20k–$50k/yr (≈2–3× data-plane footprint on tier-1 + cross-zone transfer) and ~0.5–1.0 FTE of run effort, offset by IaC + auto-docs.
What it buys: a single AZ1-class tier-1 outage across a 2,000-customer B2B estate is six-to-seven figures in SLA credits, churn and trust. The uplift is a rounding error against one outage — it applies to tier-1 only; dev & ephemeral are exempt.
Method: every figure is the documented order-of-magnitude range from the source plan (§4); the “midpoint” columns are ours for the deck. Zone-redundancy and FTE are shown as a distinct resilience line, not folded into the monitoring saving. All figures carry a [validate with finance] flag — confirm the SolarWinds contract exit terms and the parallel-run window before quoting leadership.
21Vianet stack is fully isolated — no cross-border scrape or query. Brazil metrics stay in-region under LGPD; only aggregates leave.
Grafana behind Entra-ID with per-team folders. Key Vault is RBAC, purge-protected, private-endpoint only. Secrets never in code.
AI drafts are pull requests with owner-routed reviewers; low-confidence work is labelled needs-sme. Rollback is git revert.
Every page has a classification field; ACL trimming, secret-scanning and PII redaction run in the adapters before anything reaches the wiki or RAG index.
Postgres, Redis, Service Bus, AI Search and Key Vault on private endpoints; Log Analytics diagnostic settings on every resource.
Prometheus HA pair + n8n HA on AKS, all zone-redundant per ADR-0001 — so the observability stack survives the failures it watches for.
Terraform stands up a zone-redundant AKS + monitoring dev platform in West Europe first — proving the AZ1 fix and the SolarWinds replacement before a single production cutover.