Troubleshooting methodically
Every administrator faces systems that will not boot, services that will not start, and networks that will not connect. A calm, systematic approach solves problems faster than guessing.
A troubleshooting process
Define the problem precisely: what is failing, since when, for whom, and what changed. Gather evidence from logs, status commands, and metrics. Form a hypothesis, test it with the smallest safe experiment, and repeat. Make one change at a time, record what you did, and confirm the fix. Afterwards, address the root cause and document it so it does not recur.
Services that will not start
systemctl status unit shows the state, exit code, and recent log lines; journalctl -u unit -b shows the full story. Common causes include configuration syntax errors (many daemons offer test modes, such as nginx -t or sshd -t), missing files or permissions, a port already in use (check with ss -ltnp), SELinux or AppArmor denials (check the audit log or journalctl for AVC or apparmor messages), and missing dependencies.
Disk space and inodes
A full file system breaks logging, package updates, databases, and logins. df -h shows which file system is full; du -xh --max-depth=1 / | sort -h finds the largest directories; journalctl --disk-usage shows journal size. Deleted files still held open by a process keep using space until the process closes them or restarts: lsof +L1 lists such files. df -i reveals inode exhaustion from millions of tiny files.
Network connectivity
Work up the layers. ip link checks interfaces are up; ip addr shows addresses; ip route shows the default gateway; ping the gateway, then an external IP address, then a host name to separate local, routing, and DNS problems. resolvectl status or /etc/resolv.conf shows DNS servers; dig name tests resolution. ss -tulpn confirms services are listening; curl -v tests HTTP end to end; and firewall rules may block traffic.
Boot problems
If the boot loader appears, edit the kernel command line to boot into rescue.target or emergency.target, or select an older kernel. If the system drops to an emergency shell, journalctl -xb shows why, often an fstab entry for a missing disk (adding nofail to non-critical mounts prevents this) or a file-system error requiring fsck. Live USB systems can mount and repair an installed system's disks.
Permissions and access problems
'Permission denied' may come from file modes, ownership, missing execute bits on a parent directory, ACLs, mount options such as noexec, SELinux or AppArmor policy, or missing capabilities. namei -l /path/to/file shows the permissions of every directory along a path, quickly revealing where access stops. id shows the user's groups, remembering that new group memberships need a fresh login.
Learning from incidents
After significant problems, teams write blameless post-incident reviews describing the timeline, impact, root causes, and follow-up actions, focusing on improving systems rather than blaming people. Runbooks capture tested procedures for common tasks and failures. Monitoring and alerts are adjusted so similar issues are caught earlier next time.
Worked example
nginx fails to start after a configuration change. systemctl status nginx shows it exited with an error; journalctl -u nginx -b shows 'bind() to 0.0.0.0:80 failed (98: Address already in use)'. ss -ltnp 'sport = :80' reveals an Apache process listening on port 80, installed as a dependency of another package. Stopping and disabling Apache, then running nginx -t and systemctl start nginx, resolves it, and the team records the change.
Key takeaways
Troubleshoot by defining the problem, gathering evidence, testing hypotheses one change at a time, confirming the fix, and documenting root causes. For failing services, read systemctl status and journalctl -u, test configurations, check ports with ss, and look for SELinux or AppArmor denials. For full disks, use df -h, du, journalctl --disk-usage, lsof +L1 for deleted-but-open files, and df -i for inodes. Debug networks layer by layer with ip, ping, dig, ss, and curl. Use rescue or emergency targets, older kernels, nofail, fsck, and live USBs for boot problems. namei -l traces permission problems along paths. Blameless reviews, runbooks, and better alerts turn incidents into improvements.
Flashcard
Define, gather evidence, hypothesise, test, fix
Check your understanding
df shows a disk 100% full, but du finds only half that much data. What is a likely explanation?