Network Troubleshooting: Deep Packet Inspection

You've learned to route traffic, translate addresses, and build secure tunnels. But what do you do when the connection still fails? How do you know if a packet is being dropped by a firewall, corrupted by a router, or ignored by an application?

You need to "Listen" to the wire.

In this final networking lesson, we will master Deep Packet Inspection (DPI). We will use tcpdump to capture raw traffic on servers and Wireshark to visualize it on our desktops. We will also use nmap to scan for open ports and identify potential security holes.

1. tcpdump: The Command Line Sniffer

tcpdump is the most powerful tool in a sysadmin's belt. It allows you to watch every packet arriving at or leaving your server in real-time.

Essential Flags:

-i eth0: Which interface to watch.
-n: Don't resolve IP addresses to names (makes it much faster).
-w file.pcap: Save the capture to a file to open in Wireshark later.
-vv: Very verbose (show more headers).

Finding the Needle in the Haystack (Filters):

# Watch only traffic on Port 80 (HTTP)
sudo tcpdump -i eth0 port 80

# Watch only traffic coming from a specific IP
sudo tcpdump -i eth0 src 192.168.1.50

# Watch only SSH (22) traffic that ISN'T from your management PC
sudo tcpdump -i eth0 port 22 and not src 10.0.0.5

2. Wireshark: The Visual X-Ray

While tcpdump is great for quick checks, it's hard to read a 10,000-line text output. Wireshark is a graphical tool that translates raw hex data into human-readable protocols.

The Flow Graph: Shows the "Three-way Handshake" of TCP (SYN -> SYN-ACK -> ACK).
Follow Stream: Allows you to read an entire conversation (like a chat or a web request) as if it were a single text file.

3. nmap: The Network Mapper

nmap is the "Security Guard's Flashlight." It sends specialized packets to multiple ports to see who is listening.

# Quick scan of common ports
nmap 192.168.1.100

# Aggressive scan (OS detection, Version detection)
sudo nmap -A 192.168.1.100

# Service Version detection (Find out EXACTLY which version of Nginx is running)
nmap -sV 192.168.1.100

4. Troubleshooting: The TCP Handshake Failure

If a client says "Connection Timed Out," you should run tcpdump and look for the handshake.

You see SYN but NO SYN-ACK: The server is likely behind a firewall that is dropping the packet.
You see SYN and then a RST (Reset): The firewall is allowing the packet, but no application is actually listening on that port.

5. Practical: The "Stealth" Audit

Use nmap to verify your own firewall rules. If you think you've blocked Port 3306, run a scan from a different machine to be 100% sure.

# Practical: Scan your own public IP for open ports
sudo nmap -sS -p 1-1024 [your_public_ip]

6. Example: A Port Scanning Alert System (Python)

In a production environment, you should be alerted if a "Stranger" starts scanning your ports. Here is a Python script that parses the syslog for signs of an nmap scan.

import time
import os

def monitor_scans(log_path="/var/log/syslog"):
    """
    Watches for firewall log entries that indicate a port scan.
    """
    if not os.path.exists(log_path):
        print(f"Log not found at {log_path}")
        return

    print(f"Monitoring {log_path} for scan activity...")
    
    # We open the file and jump to the end
    with open(log_path, 'r') as f:
        f.seek(0, os.SEEK_END)
        
        while True:
            line = f.readline()
            if not line:
                time.sleep(0.1)
                continue
            
            # Firewalls like UFW record blocks with "[UFW BLOCK]"
            if "[UFW BLOCK]" in line:
                print(f"[!] SCAN ALERT: {line.strip()}")
                # You could add code here to auto-block the IP in nftables!

if __name__ == "__main__":
    monitor_scans()

7. Professional Tip: Use 'pcap' for Evidence

If you are a security consultant, always save your captures as .pcap files. They are the "Fingerprints" of the internet. You can use them to prove that a hack happened, to analyze malware behavior, or to debug a protocol failure that only happens once a day.

8. Summary

Troubleshooting is about moving from "I hope" to "I know."

tcpdump is the tool for capturing raw evidence on the fly.
Wireshark is the tool for deep analysis and visualization.
nmap is the tool for discovery and security verification.
Handshake Analysis is the key to solving "Connection Refused" errors.

This concludes Module 11: Advanced Linux Networking. You now have the skills to architect, secure, and debug the digital nervous system of any enterprise.

In the next module, we move from the network to the machine and explore Disk Management and Filesystems.

Quiz Questions

What is the difference between nmap and tcpdump?
How do you save a tcpdump capture so that you can open it in Wireshark?
What does a "TCP Reset" (RST) packet usually mean in a troubleshooting context?

End of Module 11. Proceed to Module 12: Disk Management and Filesystems.

The Digital Stethoscope: tcpdump and Wireshark