Give every signal one meaning
| Signal | What it proves | What it does not prove |
|---|---|---|
GET /ping | The HTTP process can answer a small liveness request. | Database, storage, providers, protocols, permissions, or a complete user journey. |
GET /statistics | Runtime/process statistics exposed by the deployed release. | That every value is healthy without a baseline and capacity model. |
| Application probe | A representative read or write and its authorization checks work. | All tables, roles, actions, or external dependencies. |
| Protocol/provider probe | The exact SMTP, IMAP, FTP, storage, or provider path tested is working. | Other protocols, accounts, regions, or operations. |
Collect without exposing internals publicly
Poll liveness from the same network path as the load balancer and collect statistics and logs through an operator-only path. Restrict diagnostic endpoints at the edge if they reveal deployment data. Keep clocks synchronized and attach instance, release, request, route, and tenant-safe correlation fields where available. Never log passwords, bearer tokens, provider credentials, or file bodies.
curl --fail --silent --show-error http://127.0.0.1:6336/ping
curl --fail --silent --show-error http://127.0.0.1:6336/statisticsAlert on user impact and exhaustion
- Sustained liveness or readiness failure, not a single transient sample.
- Elevated server errors, latency, or authorization failures relative to the normal baseline.
- Database connection waits, exhaustion, storage capacity, and backup/restore failure.
- Certificate expiry and renewal failure before clients are affected.
- Mail backlog/retries, scheduled-work delay, provider failure, and cluster degradation when those services are used.
Set initial thresholds from representative load and revise them from incidents. This guide cannot supply one safe global number for every workload.
Make every alert actionable
Each alert needs an owner, severity, user impact, dashboard/log query, immediate containment, escalation path, and recovery check. Test alert delivery and the runbook during a controlled failure. After recovery, verify a real application read/write and the affected dependency rather than clearing the alert on process restart alone.