The Capstone Project: The Citadel

You have traveled a long road. From the first ls command in Module 3 to the kernel-level profiling in Module 17 and the global orchestration in Module 19. You are no longer a "User" of Linux; you are an Architect.

But knowledge without application is just data. To prove your mastery, you must build The Citadel.

The Citadel is not just a server; it is a Highly Available Environment designed to stay alive during a disaster and block any attacker. In this final project, we will combine everything you've learned.

1. The Blueprint: Architecture

Your goal is to build the following structure:

Two HAProxy Load Balancers (using Keepalived for a Floating IP).
Two Nginx Web Servers (synced via lsyncd).
One MySQL Primary and One MySQL Replica (configured for Read-Write splitting).
Centralized Logging (All logs sent to a dedicated logging instance).

graph TD
    User((User)) -- VIP --> K[Keepalived / HAProxy]
    K -- Load Balance --> W1[Nginx Web 1]
    K -- Load Balance --> W2[Nginx Web 2]
    W1 -- Write --> DB_M[(MySQL Primary)]
    W2 -- Write --> DB_M
    W1 -- Read --> DB_R[(MySQL Replica)]
    W2 -- Read --> DB_R
    M[Monitor / Logging] -- Watch --> K
    all[Wait... All Servers] -- rsyslog --> M

2. Phase 1: Infrastructure as Code

Objective: Provision the servers.

Write a Terraform script to launch 6 small Linux instances.
Configure a virtual network with a private subnet and a public gateway.
Set up security groups (Firewalls) that only allow Port 80/443 to the Load Balancers and SSH only from your IP.

3. Phase 2: Configuration Management

Objective: Install and configure the software.

Write an Ansible Playbook that:
- Hardens SSH (Key-only, no root login).
- Installs Nginx and MySQL.
- Configures Keepalived and HAProxy.
- Sets up the replication user for MySQL.
- Sets up a "Cron" job to backup the database every night.

4. Phase 3: High Availability and Security

Objective: Make it bulletproof.

Implement SELinux or AppArmor profiles for Nginx to ensure it can't read files outside /var/www.
Configure Fail2Ban to block IPs that try to brute-force your SSH port.
Test the Failover: Shut down the primary Load Balancer and ensure the website stays online.

5. Phase 4: Observability

Objective: See everything.

Set up an AIDE watchlist for /etc and /usr/bin.
Configure a Python script (from Module 16/17) to monitor for database replication lag and RAM pressure.
Verify that logs from the Load Balancers are appearing in your central logging server.

6. The Final Challenge: The Disaster Simulation

Once the Citadel is built, perform these three tests:

The DB Crash: Kill the primary database. Does the replica take over or does your app at least stay in "Read-Only" mode?
The Traffic Spike: Run a load test (e.g., ab -n 1000 -c 100). Watch top and htop to identify the bottleneck.
The Intrusion: Try to edit /etc/shadow from a non-root account. Does PAM and your Audit system notice?

7. Submission and Certification

Building the Citadel is the digital equivalent of a Master's Thesis. If you have successfully implemented these components, you have attained the Level 4: Architect status in the Linux & Networking Mastery program.

8. Summary

Mastery is not a destination; it is a way of working.

Complexity is managed through Automation.
Stability is managed through Redundancy.
Security is managed through Layers.
Performance is managed through Measurement.

This project marks the end of our guided journey. You now possess the keys to the most powerful operating system in history.

Final Lesson 2: Course Wrap-up and the Path to Professional Certification.

The Architect's Proof: Building the Citadel