OBSRV · RESILIENCE — PACK
Executive pack · Start here

Everything for the board, in one pack — the story, the money, and a flow you can run live.

One AZ took the platform down. This pack is the fix: self-hosted monitoring in place of poll-only SolarWinds, zone-redundancy enforced in code, the ServiceNow automation repaired — plus the auto-docs pipeline exposed as a working HTTP API you can demo in the room.

~$130knet saved / yr*

The bottom line: zero monitoring licences, a full-region outage taken off the table for a 2,000-customer estate, and documentation that writes itself. The resilience uplift ($20–50k + ~0.5–1.0 FTE) is a rounding error against a single AZ1-class outage. *Every figure carries a [validate with finance] flag.

2,000+
Customers
3
Regions EU/BR/CN
15 min
Old poll blind-spot
$0
Monitoring licences
20 wk
Dev-first rollout
Pros & cons — the honest read

✓ Why we do it

  • Kills the licence bill. SolarWinds $120–250k/yr → $0; OSS on infra we already run.
  • Survives a zone. ADR-0001 makes single-zone tier-1 fail the Terraform plan — the fix can’t be forgotten.
  • Deeper signal. First-class Kubernetes, Azure and network vs. 15-minute polling with no webhooks.
  • Change automation works again. Branches on state + approval, keyed on immutable sys_id — no more missed “Implementing”.
  • Docs that maintain themselves. Every alert, node and change drafts its own page behind a pull request.
  • Productisable. The same stack can be sold to customers.

→ What it costs us

  • We now own monitoring uptime. Mitigated: HA + zone-redundant, so the stack isn’t its own SPOF.
  • Run effort: ~0.5–1.0 FTE. Offset by IaC + auto-docs removing manual toil.
  • Resilience uplift $20–50k/yr. Tier-1 only; dev & ephemeral exempt. One outage dwarfs it.
  • China is a separate cloud. Fully isolated 21Vianet stack; residency sign-off before rollout.
  • Migration window. Parallel-run; cut over per service; SolarWinds retired last.
  • Costs need finance sign-off. All figures are documented ranges, flagged for validation.
What’s in this pack
01 · present on screen
📡

Executive brief

The full scrolling deck: incident, architecture, 3-region residency, the AZ1 before/after, ServiceNow state machine, the money model and the asks.

interactive · light + dark
Open the deck →
02 · print / hand out
🖨️

Print handout

A tight 2-page A4/Letter leave-behind: the case & the money on page one, architecture & delivery on page two. Ink-friendly, print-to-PDF ready.

2 pages · print CSS
Open & print →
03 · run it live
⚡

Live API console

The auto-docs flow as an HTTP API. POST an event, watch the router classify it and the drafter return a review-ready page. Demo it in the room.

stdlib · mock / anthropic
Open console (run make serve) →
Run the live flow
# from the automation/ directory — no new dependencies, offline by default make serve # console → http://127.0.0.1:8099/ # drive it yourself: curl -s localhost:8099/example \ | curl -s -X POST localhost:8099/draft -d @- # → router decision + drafted page # go live against Claude instead of the mock provider: LLM_PROVIDER=anthropic make serve

Endpoints: GET /health · GET /example · POST /draft · POST /pipeline. Nothing publishes without a pull request; restricted content is link-only (never mirrored); API calls write to a sandbox, never the repo. Covered by 7 tests in the suite (27 total, all green).