Logging Architecture: The Two Worlds

If a server crashes in the middle of the night and no one is there to see it, does it make a sound? In Linux, the answer is Yes. Every action, every error, and every success is recorded in the system logs.

Professional troubleshooting starts and ends with logs. But modern Linux systems actually use two different logging systems at the same time:

1. The Systemd Journal (journald): A modern, binary-based system that captures everything from boot up to shut down.
2. Traditional Syslog (rsyslog): The classic, plain-text system that organizes logs into files in /var/log.

In this lesson, we will understand how these two systems work together and where exactly you should look when things go wrong.

---

1. The Journal: The High-Speed Recorder

When a service like Nginx starts, its output is captured by journald.

• Format: Binary (you cannot read it with cat).
• Speed: Very fast.
• Scope: Includes kernel messages, early boot messages, and standard output from every service.

Why a binary format?

Because it allows for Metadata. Every log entry in the journal knows the PID of the process, the User ID, the exact nanosecond it occurred, and even which CPU core it was running on.

---

2. The Syslog: The Human-Readable Archive

Most Linux distros still run a "Syslog" daemon (usually rsyslog). It "listens" to the journal and copies important messages into categorized text files.

File Path	What's inside?
/var/log/syslog	The general "Everything" log (Debian/Ubuntu).
/var/log/messages	The general "Everything" log (Red Hat/CentOS).
/var/log/auth.log	Logins, sudo usage, and SSH attempts.
/var/log/kern.log	Kernel errors and hardware issues.
/var/log/dmesg	The most recent hardware boot messages.

---

3. The Flow of Information

It is helpful to visualize the "Logging Pipeline." A single error message goes on a journey:

graph TD
    App[Application Error] --> J[Systemd Journald]
    J -- Binary DB --> Viewer[journalctl command]
    J -- Text Stream --> R[rsyslog daemon]
    R -- Filter & Rule --> F1[/var/log/auth.log]
    R -- Filter & Rule --> F2[/var/log/syslog]

---

4. Practical: Real-Time Tailoring

When you are trying to reproduce a bug, you want to see the logs as they happen.

# Watch the binary journal for the whole system
sudo journalctl -f

# Watch a specific text file for new logins
sudo tail -f /var/log/auth.log

---

5. Identifying the "Noisy" Service

Sometimes a single broken app will flood your logs with 10,000 messages a minute, filling up your hard drive.

# See which log files are the largest
sudo du -sh /var/log/* | sort -h

---

6. Example: A Log Importance Filter (Python)

If you are looking at /var/log/syslog, 99% of it is "Information" that you don't care about. Here is a Python script that parses a log file and only shows you the "CRITICAL" or "ERROR" lines.

import os

def filter_critical_logs(log_path="/var/log/syslog"):
    """
    Parses a text log and pulls out only the entries with high severity.
    """
    if not os.path.exists(log_path):
        print("Log file not found.")
        return

    keywords = ["ERROR", "CRITICAL", "FAILED", "FATAL", "DENIED"]
    
    print(f"--- Critical Events in {log_path} ---")
    
    with open(log_path, 'r', errors='ignore') as f:
        for line in f:
            if any(key in line.upper() for key in keywords):
                print(line.strip())

if __name__ == "__main__":
    filter_critical_logs()

---

7. Professional Tip: Use 'logger' to Test your Setup

If you are writing a script and want to record its progress in the official system logs, don't just use echo. Use the logger command. This ensures your script's messages are timestamped and categorized correctly by the system.

logger "BACKUP SCRIPT: Starting database export now."
# Now go check /var/log/syslog!

---

8. Summary

Logging is the insurance policy of a stable system.

• Journald is the fast, binary foundation of modern Linux.
• Syslog is the organized, text-based archive.
• /var/log/auth.log is your best friend for security investigation.
• journalctl is the tool to read the binary, and tail is the tool to watch the text.
• Use logger to make your own scripts part of the system's history.

In the next lesson, we will master the specialized tool for the binary world: Mastering journalctl.

Quiz Questions

1. Why can't you use cat or grep directly on the /var/log/journal/ files?
2. Which log file would you check if you suspected someone was trying to "Brute-Force" their way into your server via SSH?
3. What is the main benefit of the digital metadata stored in the systemd journal?

---

Continue to Lesson 2: Mastering journalctl—Searching the Modern Records.