Logging with the journal and syslog
When something goes wrong, logs explain why. systemd's journal and traditional syslog files record messages from the kernel, services, and applications.
The systemd journal
systemd-journald collects log messages from the kernel, early boot, services' standard output and error, and the syslog interface, storing them in a structured, indexed binary format with metadata such as the unit, PID, user, and priority. By default it may keep logs only in memory under /run/log/journal unless persistent storage under /var/log/journal is enabled, as most distributions now do.
journalctl basics
journalctl shows the journal. Useful options: -u nginx (one unit), -b (current boot) and -b -1 (previous boot), -k (kernel messages), -f (follow new messages), -p err (priority error and above), --since '1 hour ago' or --since 2026-10-24 --until 2026-10-25, -n 50 (last 50 lines), and -o json for machine-readable output. Combining filters narrows problems quickly.
Priorities
Messages carry syslog priorities from 0 (emerg) through 1 (alert), 2 (crit), 3 (err), 4 (warning), 5 (notice), and 6 (info) to 7 (debug). Filtering by priority, such as journalctl -p warning -b, surfaces significant problems in a boot without the noise of informational messages.
Traditional syslog
Many systems also run a syslog daemon, usually rsyslog, writing text files under /var/log: /var/log/syslog or /var/log/messages for general messages, /var/log/auth.log or /var/log/secure for authentication, /var/log/kern.log for the kernel, and application-specific files such as /var/log/nginx/access.log. Text logs are easy to grep and to forward to central servers.
Log rotation and retention
Logs grow continuously, so logrotate rotates text logs daily or weekly, compressing old files (such as syslog.2.gz) and deleting the oldest after a set number. The journal limits its own size with settings such as SystemMaxUse= in /etc/systemd/journald.conf, and journalctl --vacuum-size=500M or --vacuum-time=2weeks trims it manually.
Centralised logging
With many servers, logs are shipped to a central system for search, alerting, and retention: rsyslog or syslog-ng forwarding, or agents such as Fluent Bit, Vector, or the OpenTelemetry Collector sending to systems such as Elasticsearch or OpenSearch, Loki, or commercial services. Central logs survive when a compromised or crashed machine's local logs do not.
Good logging practice
Applications should log meaningful events with timestamps, severity, and context, such as request IDs, preferably in structured formats like JSON. They must never log secrets such as passwords, tokens, or full card numbers, and should treat personal data carefully. Times are best recorded in UTC to avoid confusion across time zones and daylight-saving changes.
Worked example
A web service failed overnight. journalctl -u web.service --since 'yesterday 22:00' --until 'today 06:00' -p warning shows, at 02:13, 'connection to database refused' repeated, followed by the service exiting. journalctl -u postgresql --since 'today 02:00' reveals the database restarted at 02:12 for automatic updates. The fix: make the web service retry connections and order it After=postgresql.service.
Key takeaways
systemd-journald stores structured, indexed logs from the kernel, services, and syslog, persistently under /var/log/journal when enabled. journalctl filters by unit (-u), boot (-b), kernel (-k), priority (-p), and time (--since, --until), and follows with -f. Priorities run from 0 emerg to 7 debug. rsyslog writes text logs such as /var/log/syslog, auth.log, and application logs. logrotate and journald size limits manage retention. Central logging with forwarding agents enables search and survives host failures. Log useful structured events in UTC, and never log secrets.
Flashcard
Collects structured logs
Check your understanding
A server rebooted unexpectedly overnight. Which command best shows warnings and errors from before the reboot?