Performance tuning and monitoring
Performance work starts with measurement. A systematic approach finds the real bottleneck, whether CPU, memory, disk, or network, before anything is tuned.
A method
Brendan Gregg's USE method checks, for every resource, utilisation (how busy it is), saturation (how much work is queued), and errors. Applying it to CPUs, memory, disks, and network interfaces quickly locates bottlenecks. Change one thing at a time, measure before and after, and prefer fixing inefficient work over adding hardware.
CPU
top, htop, and mpstat -P ALL show overall and per-core use, split into user, system, I/O wait, and steal time (on virtual machines, time taken by the hypervisor for other guests). Load averages compared with core counts indicate saturation. perf top and perf record profile where CPU time goes, down to functions, and flame graphs visualise these profiles.
Memory
free -h shows available memory, which includes reclaimable cache; a low 'available' figure, rising swap use, and high values in vmstat's si and so columns (pages swapped in and out per second) indicate memory pressure. The OOM killer's messages in the kernel log show when memory was exhausted. vm.swappiness, from 0 to 200 on recent kernels and default 60, influences how readily the kernel swaps.
Disk I/O
iostat -xz 1 reports per-device utilisation, throughput, I/O operations per second, average wait times, and queue sizes; high %util with long waits means the device is saturated. iotop shows which processes are generating I/O. Remedies include faster storage such as NVMe SSDs, caching, fixing chatty applications, and appropriate I/O schedulers, such as none or mq-deadline for fast SSDs.
Network
ip -s link shows interface counters including errors and drops; ss -s summarises sockets; ss -ti shows TCP details such as round-trip time and retransmissions; sar -n DEV reports throughput over time; and iperf3 measures achievable bandwidth between hosts. Tuning may involve TCP buffer sizes, congestion control (sysctl net.ipv4.tcp_congestion_control, such as bbr), and interrupt affinity on busy servers.
Observability tools
sar, from the sysstat package, records historical statistics so you can see what happened yesterday at 3 a.m. eBPF, a kernel technology for running safe, sandboxed programs on kernel events, powers tools such as bcc and bpftrace, letting engineers trace latency of disk requests, slow system calls, or TCP retransmits in production with low overhead. Prometheus node_exporter and Grafana provide fleet-wide dashboards.
Tuning safely
Kernel parameters can be changed with sysctl and persisted in /etc/sysctl.d/; tuned on Red Hat-family systems applies profiles such as throughput-performance or virtual-guest. Keep changes documented and reversible, and beware copying 'magic' tuning lists from the internet without understanding them: many are obsolete or harmful on modern kernels.
Worked example
An API's response times spike every afternoon. mpstat shows CPUs 40% busy, so CPU is not saturated; free shows plenty of available memory. iostat -xz 1 shows the data disk at 98% utilisation with average waits of 45 milliseconds during spikes, and iotop points to a reporting job's large queries. Moving the reports to a read replica reduces disk utilisation to 30% and restores normal response times, without any kernel tuning.
Key takeaways
Measure before tuning; the USE method checks utilisation, saturation, and errors for each resource, changing one thing at a time. For CPU, use top, htop, mpstat, load averages, and perf with flame graphs; watch steal time on VMs. For memory, check available memory, swap activity in vmstat, OOM messages, and vm.swappiness. For disks, iostat -xz shows utilisation and waits, and iotop shows culprits. For networks, use ip -s link, ss, sar -n DEV, and iperf3, tuning buffers or congestion control when needed. sar records history; eBPF tools such as bpftrace trace production safely; node_exporter and Grafana give dashboards. Tune with documented, reversible changes via sysctl or tuned.
Flashcard
Utilisation, saturation, errors
Check your understanding
A server is slow. CPUs are 30% busy, memory is plentiful, but iostat shows a disk at 99% utilisation with long wait times. Where is the bottleneck?