rsyslog: Organizing the System Record

While journalctl is fantastic for searching on a single machine, rsyslog is built for the enterprise.

Why use rsyslog?

1. Readable Files: Highly readable text files in /var/log that any tool can process.
2. Centralization: A hacker can delete logs on a server they just broke into. But if those logs were instantly sent to a Remote Logging Server, the evidence is already safe.
3. Complex Filtering: You can say: "If a message contains the word 'HACKER', send an email to the admin and write it to a special file."

In this lesson, we will master the configuration of the rsyslog daemon.

---

1. Facilities and Severities

rsyslog categorizes every message using two criteria:

I. The Facility (The Category)

• auth / authpriv: Security and login messages.
• mail: Email system messages.
• cron: Scheduled task messages.
• kern: Kernel messages.
• local0 through local7: Reserved for your own custom apps.

II. The Severity (The Emergency Level)

• debug: Lowest level.
• info: General stats.
• warn: Something is odd.
• err: Something is broken.
• crit / emerg: The system is dying.

---

2. The Configuration Logic

The configuration is found in /etc/rsyslog.conf and /etc/rsyslog.d/.

A rule looks like this: facility.priority /path/to/log/file

Practical Examples:

# Record all auth errors to a special file
auth.err                /var/log/security_failures.log

# Record everything of 'info' level EXCEPT mail
*.info;mail.none        /var/log/everything_but_mail.log

# Discard (ignore) all debug messages
*.debug                 ~

---

3. Centralized Logging: The "Log Server"

In a professional setup, you have one "Secure Server" that collects logs from every other machine.

On the Client (The sender):

Tell the client to send its logs via UDP (one @) or TCP (two @@).

# Send all logs to the central server at 192.168.1.100
*.*  @@192.168.1.100:514

On the Server (The receiver):

Enable the "Reception" module in /etc/rsyslog.conf:

module(load="imtcp")
input(type="imtcp" port="514")

---

4. Practical: Custom App Logging

If you want your Python or Bash script to have its own log file in /var/log/myapp.log, you can tell rsyslog to look for sentences coming from your app.

# In /etc/rsyslog.d/myapp.conf
if $programname == 'myapp' then /var/log/myapp.log
& stop

---

5. Troubleshooting: The Log Gap

If you see a log in journalctl but it is missing from /var/log/syslog, it means:

1. The rsyslog daemon is not running.
2. The systemd-journal-upload service is broken.
3. A filtering rule in rsyslog is accidentally dropping the message.

---

6. Example: A Remote Log Connectivity Tester (Python)

If you are using centralized logging, you need to be sure the port is open. Here is a Python script that tests if your central log server is ready to receive data.

import socket

def test_rsyslog_connection(host, port=514):
    """
    Attempts a TCP connection to the rsyslog port.
    """
    print(f"Testing rsyslog connectivity to {host}:{port}...")
    
    try:
        # Create a TCP socket
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.settimeout(2)
        s.connect((host, port))
        print("[OK] Connection Successful. Central server is listening.")
        s.close()
    except Exception as e:
        print(f"[!!!] FAILED: Cannot reach central server. {e}")

if __name__ == "__main__":
    # Change to your real log server IP
    test_rsyslog_connection("127.0.0.1")

---

7. Professional Tip: Check 'template' for Custom Formats

Standard logs can be hard to parse by AI or monitoring tools. rsyslog allows you to create a Template to change how the message looks. You can add the Server's UUID, the ISO timestamp, or even a JSON wrapper to every line.

---

8. Summary

rsyslog is the architect's tool for log management.

• Facilities and Severities allow you to filter the signal from the noise.
• Remote Logging provides security and a "Single Source of Truth."
• /var/log/ is the human-readable result.
• logger is how your apps join the stream.

In the next lesson, we will learn how to keep these files from growing until they crash the server: Mastering logrotate.

Quiz Questions

1. Why is sending logs to a remote server considered a critical security practice?
2. What is the difference between @ and @@ in a remote logging rule?
3. How do you tell rsyslog to stop processing a message once it has been saved to a specific file?

---

Continue to Lesson 4: Log Management—Mastering logrotate.