Mastering journalctl: Querying the System

In the old days, searching for an error meant using grep across 50 different text files in /var/log. It was slow, and you often missed context.

Because systemd-journal is a binary database, it provides a command called journalctl that acts like a search engine for your server. You can ask it complex questions like: "Show me every error message from the Nginx service that happened between 2 PM and 3 PM yesterday."

In this lesson, we will learn to master the "Query Power" of journalctl.

---

1. Basic Filters: Units and Priorities

The most common way to search is by the Unit (the name of the service).

# Show logs ONLY for the SSH service
sudo journalctl -u ssh

# Show logs for the Kernel (useful for hardware issues)
sudo journalctl -k

Filtering by Priority:

The journal uses the standard Unix priority levels (0-7).

• -p err: Show Errors, Criticals, and Emergencies.
• -p warning: Show warnings and everything more serious.

# Show only Errors and worse for the whole system
sudo journalctl -p err

---

2. The Power of Time: Since and Until

This is where journalctl truly shines. You don't need to know exact timestamps; it understands human-readable time.

# Show everything that happened since yesterday
sudo journalctl --since "yesterday"

# Show everything from the last 15 minutes
sudo journalctl --since "15 min ago"

# Show everything between two specific times
sudo journalctl --since "2026-03-22 14:00:00" --until "2026-03-22 15:00:00"

---

3. Investigating Boots

By default, journalctl shows you everything it has in memory. But what if the server crashed and you want to see the logs from before the reboot?

# List all recorded boots
sudo journalctl --list-boots

# Show logs from the PREVIOUS boot (the one before the current one)
sudo journalctl -b -1

# Show logs from 3 boots ago
sudo journalctl -b -3

---

4. Cleaning the Database: Disk Usage

The journal can grow to several gigabytes if you don't limit it. This can consume your entire hard drive!

# See how much space the journal is currently using
sudo journalctl --disk-usage

# Delete all logs older than 7 days
sudo journalctl --vacuum-time=7d

# Delete old logs until only 1GB remains
sudo journalctl --vacuum-size=1G

---

5. Practical: Connecting the Service to the Error

If a service fails to start, running systemctl status usually only shows the last 10 lines of the error—not enough to fix it. Use journalctl with the -xe flags.

• -x: Add "Expert" explanations of the error (if available).
• -e: Jump directly to the END of the log.

sudo journalctl -u nginx -xe

---

6. Example: A Journal Error Summarizer (Python)

If a server is having issues, you want a summary of the count of errors per service. Here is a Python script that parses the journal for the last hour and counts the failures.

import subprocess
import collections
import re

def summarize_journal_errors():
    """
    Groups errors by service name.
    """
    print("--- 1-Hour Error Summary ---")
    
    # Get errors from the last hour in a machine-readable format
    cmd = ["sudo", "journalctl", "-p", "err", "--since", "1 hour ago", "--no-pager"]
    res = subprocess.run(cmd, capture_output=True, text=True)
    
    lines = res.stdout.split('\n')
    
    # Simple regex to find service names in brackets like [nginx]
    # Format: Mar 23 10:00:00 hostname service_name[PID]: message
    service_counts = collections.Counter()
    
    for line in lines:
        match = re.search(r" \w+\[\d+\]:", line)
        if match:
            service = match.group(0).strip(" []:")
            service_counts[service] += 1
            
    if not service_counts:
        print("[OK] No errors found in the last hour.")
    else:
        for service, count in service_counts.most_common():
            print(f"{service:20} | {count} errors")

if __name__ == "__main__":
    summarize_journal_errors()

---

7. Professional Tip: Use '-o json' for Automation

If you are a developer building a monitoring dashboard, don't try to parse the text output of journalctl. Use the -o json or -o json-pretty flag. This gives you every field of the log as a clean JSON object that your code can process effortlessly.

---

8. Summary

journalctl is the scalpel of the Linux system administrator.

• -u filters by service (Unit).
• --since filters by time.
• -b looks back into previous boots.
• --vacuum keeps your disk from filling up.
• -xe is the standard debugging workflow for service failures.

In the next lesson, we will look at the other side of the coin: rsyslog and Centralized Logging.

Quiz Questions

1. How do you view only the kernel messages (like dmesg) using journalctl?
2. What is the difference between journalctl -b 0 and journalctl -b -1?
3. How would you limit the journal size so it never uses more than 500MB?

---

Continue to Lesson 3: The Classic Syslog—rsyslog and Centralized Logging.