Lesson 12.1: Final mission: the 2 a.m. call
A complete incident on one machine, solved with tools from every section: read the alerts, find the runaway process and stop it, read a failed service's status and journal, find what filled the disk, remove it, fix the configuration with sed, restart and verify the service, measure server errors in the access log with a regular expression, run a health-check script and schedule it with cron, and write the incident note.
25 min
You will learn to
- Work through an incident in a calm order: look, find the cause, fix, verify, prevent
- Combine ps, kill, systemctl, journalctl, du, find, sed, grep, scripts and cron
- Verify every fix instead of assuming it worked
- Leave an automatic check and a written note behind
Key terms
- Incident
- An unplanned interruption or degradation of a service: the shop is down, the disk is full, logins fail.
- On-call
- The person responsible for responding to alerts during a period, including the night.
- Symptom and root cause
- A symptom is what you notice (connection refused); the root cause is why it happened (debug logging left on). Fixing only symptoms means it comes back.
- Health check
- A small automatic test, run regularly, that says whether a service works: running, reachable, enough disk.
- Post-incident note
- A short written account: timeline, cause, fix and what prevents it next time. Blame-free: about the system, not the person.
← Functions and a real backup script · All lessons · Wrap-up: what you can do now, and what's next →