System Logging: Listening to the Diary of a Machine

A Linux system is never "silent." Even when you aren't typing, the kernel is chatting with the hardware, services are recording their successful tasks, and security monitors are noting failed login attempts.

In a crisis, the difference between an amateur and a professional is knowing exactly which "diary" to read. Is it a hardware error? (Check dmesg). Is it an application crash? (Check journalctl). Or is it a security breach? (Check /var/log/auth.log).

In this lesson, we will master the two major logging worlds: the older Text Logs and the modern systemd Journal.

---

1. dmesg: The Kernel's Ring Buffer

dmesg (Diagnostic Message) is the "first responder." It displays messages from the Kernel Ring Buffer.

Use Case: Hardware and Boot Issues

Because the kernel starts before the hard drive is even mounted, it needs a place to store messages in memory. dmesg is where you see info about:

• Plug-and-play devices (USB sticks, mice).
• Drive errors.
• Out-of-memory (OOM) kills.

# See the latest hardware events
dmesg | tail -n 20

# Search for Out-of-memory events (the OOM Killer)
dmesg | grep -i "oom"

---

2. journalctl: The Modern System Log

On systems using systemd, almost all logs are now handled by the Journal. Unlike old-school logs, these are stored in a binary format, allowing for very fast searching by time, service, or priority.

The Essential Power-User Flags:

• journalctl -u nginx: See logs for a specific service (e.g., Nginx).
• journalctl -f: Follow the logs in real-time (like tail -f).
• journalctl --since "1 hour ago": Time-based filtering.
• journalctl -p err: Show ONLY errors and critical messages (Priority).

# Find why your SSH login failed today
journalctl -u ssh --since "today" | grep "Failed"

---

3. The Classic Log Hierarchy: /var/log

Even with journalctl, most Linux systems still write plain text logs for compatibility.

Log Path	Purpose
/var/log/syslog	The general-purpose system log (Debian/Ubuntu).
/var/log/messages	The general-purpose log (RHEL/CentOS).
/var/log/auth.log	Security and authentication attempts.
/var/log/kern.log	Kernel-specific text log.
/var/log/nginx/	Application-specific logs (Web server).

---

4. Log Rotation: Keeping the Disks Clean

If a server runs for three years, a log file could grow to 100GB. To prevent this, Linux uses logrotate. It periodically "rotates" the logs:

1. syslog becomes syslog.1.
2. syslog.1 is compressed into syslog.2.gz.
3. The oldest logs are deleted.

Configuration: Found in /etc/logrotate.conf and /etc/logrotate.d/.

---

5. Practical: Finding the "Cause of Death"

If a program suddenly stops running, follow this sequence:

1. Check if the Kernel killed it for using too much RAM: dmesg | grep -i kill
2. Check the service's own logs: journalctl -u myapp --since "10 min ago"
3. Check the general system log for errors around that time: grep "ERROR" /var/log/syslog

---

6. Example: An Automated Log Parser (Python)

If you are building a monitoring agent, you need to parse logs programmatically. Here is a Python script that analyzes /var/log/auth.log to identify potential "Brute Force" hackers (multiple failed logins from same IP).

import re
from collections import Counter

def analyze_auth_failures(log_path="/var/log/auth.log"):
    """
    Scans the security log for failed login attempts.
    """
    failures = []
    
    # Regex for a common failed SSH attempt: 
    # "Failed password for root from 1.2.3.4 port ..."
    pattern = re.compile(r"Failed password for .* from (\d+\.\d+\.\d+\.\d+)")
    
    try:
        with open(log_path, 'r') as f:
            for line in f:
                match = pattern.search(line)
                if match:
                    failures.append(match.group(1))
                    
        # Count occurrences
        stats = Counter(failures).most_common(5)
        
        print(f"Top 5 Suspected Brute-Force IPs:")
        print("-" * 35)
        for ip, count in stats:
            print(f"{ip:15} | {count} failed attempts")
            
    except FileNotFoundError:
        print(f"Error: {log_path} not found. Are you on an Ubuntu/Debian system?")

if __name__ == "__main__":
    analyze_auth_failures()

---

7. Professional Tip: Use 'logger' to Create Your Own Logs

When you write a bash script, you shouldn't just echo errors to the screen. You should send them to the system logs so they can be rotated and monitored properly.

# Send a message to /var/log/syslog with a custom tag
logger -t MY_SCRIPT "Backup process started successfully."

---

8. Summary

Logging is the "Eyes and Ears" of the administrator.

• dmesg is for the Kernel and Hardware.
• journalctl is the modern, high-speed interface for all services.
• /var/log is the traditional home of text logs.
• logrotate ensures that logs don't eat your entire hard drive.

This concludes our module on System Information and Performance Monitoring. You now have the skills to identify your system, monitor its health, and diagnose its failures.

In the next module, we will move beyond a single machine and explore Linux Networking Mastery.

Quiz Questions

1. Where would you look to see messages about a newly plugged-in USB drive?
2. How do you see only the most recent 50 lines of the system journal?
3. What is the benefit of the binary format used by journalctl compared to plain text logs?

---

End of Module 6. Proceed to Module 7: Linux Networking Mastery.