Scheduling and monitoring processes
With hundreds of processes and few CPU cores, the kernel's scheduler decides who runs. Monitoring tools show what is running and why a system is slow.
The scheduler
The kernel scheduler shares CPU time among runnable threads. From 2007, Linux used the Completely Fair Scheduler (CFS), which tracks each task's virtual runtime and runs the task that has had the least, weighted by priority. Kernel 6.6 (2023) replaced it with the EEVDF scheduler, which also targets fairness while better handling latency-sensitive tasks.
Nice values and priorities
Users influence scheduling with nice values from -20 (highest priority) to 19 (lowest), default 0. nice -n 10 command starts a command at lower priority; renice changes a running process's value. Only root can raise priority by lowering niceness below 0. Real-time scheduling policies, SCHED_FIFO and SCHED_RR, give strict priority for time-critical tasks such as audio.
Load average
uptime and top show three load averages, over 1, 5, and 15 minutes, counting processes that are runnable or in uninterruptible sleep. On a 4-core machine, a load of 4.0 means roughly full use; 8.0 means tasks are waiting. Because D-state processes count, high load with low CPU use often points to storage or network file system delays.
ps and top
ps shows a snapshot of processes: ps aux lists all with user, CPU and memory percentages, and command; ps -ef shows parent PIDs; ps -o lets you choose columns. top and the friendlier htop show live, sortable views of CPU, memory, and processes, with keys to sort, filter, and send signals. pstree shows the process hierarchy.
Memory and I/O monitoring
free -h shows total, used, and available memory, including buffers and cache, which the kernel reclaims when needed. vmstat reports processes, memory, swap, I/O, and CPU over time. iostat and iotop show disk activity by device and process. If memory runs out, the out-of-memory (OOM) killer terminates a process, chosen by a badness score, logging the event in the kernel log.
Inspecting a process
lsof lists open files, including network sockets: lsof -p PID shows a process's files, and lsof -i :443 shows which process listens on port 443 (ss -ltnp also lists listening sockets). /proc/PID/status, cmdline, and environ reveal details. strace traces a process's system calls, showing exactly what it is asking the kernel to do, which is invaluable for debugging hangs and errors.
Limiting resources
ulimit sets per-process limits such as maximum open files or core-dump size. Control groups (cgroups), particularly cgroup v2, let administrators limit and account for CPU, memory, and I/O for groups of processes; systemd places each service in its own cgroup, and containers rely on cgroups to stop one workload starving others.
Worked example
A server feels slow. uptime shows load averages of 9.1, 8.7, 8.2 on a 4-core machine. top shows CPU mostly idle but many processes in state D. iostat shows the disk 100% busy, and iotop reveals a backup job writing heavily. Running renice 19 on the backup and lowering its I/O priority with ionice -c3 lets interactive work respond again, without stopping the backup.
Key takeaways
The scheduler shares CPUs fairly: CFS from 2007 and EEVDF from kernel 6.6 in 2023. Nice values range from -20 (high priority) to 19 (low); only root can go below 0; SCHED_FIFO and SCHED_RR provide real-time priority. Load averages over 1, 5, and 15 minutes count runnable and D-state tasks, compared with core count. ps, top, htop, and pstree show processes; free, vmstat, iostat, and iotop show memory and I/O; the OOM killer acts when memory runs out. lsof, ss, /proc, and strace inspect processes. ulimit and cgroups limit resources for processes, services, and containers.
Flashcard
Shares CPU time among tasks
Check your understanding
A 2-core server shows a load average of 6 but its CPUs are mostly idle. What is the most likely cause?