Module 4 Lesson 5: Optimizing for Large Clusters

When you have 500 developers pushing code every 10 minutes, your CI/CD foundation will start to crack. This lesson is about Scaling the Automation.

1. Runner Concurrency

By default, a runner might only do one thing at a time.

• The Fix: In the config.toml of your GitLab Runner, increase the concurrent setting.
• Caution: More concurrency requires more CPU and RAM. If you set concurrent = 10 but your server only has 2GB of RAM, your runner will crash.

---

2. Distributed Caching

(Review Module 3, Lesson 4). In a large cluster, "Runner A" on Server 1 needs to see the cache created by "Runner B" on Server 2.

• The Fix: Use S3 or GCS as a "Distributed Cache."
• All runners upload their node_modules to the cloud, so any runner in your cluster can pull them down instantly.

---

3. Minimizing "Git Fetches"

If your repository is 5GB (common in Game Dev or AI), downloading the code (fetching) for every small check is slow and kills the network.

variables:
  GIT_STRATEGY: none # Use this for jobs that don't need code (like checking a URL)
  GIT_DEPTH: "1"     # Only download the last commit, not the whole history

---

4. The "Monorepo" Filter

If 1,000 people are in one repo, use Parent/Child pipelines (Module 4, Lesson 2) to ensure that a change in "Service A" doesn't start the build for "Service Z." This saves thousands of dollars in cloud compute costs.

---

5. Summary of Enterprise Scaling

Problem	Solution
Slow Builds	Parallelism + needs keyword
Disk Space	Automatic pruning of runners
Network Congestion	Distributed Caching + GIT_DEPTH: 1
Configuration Bloat	Templates + Extension Fields

---

Exercise: The Architect's Audit

1. Imagine your company repo grows to 10GB. Which YAML variable should you set immediately?
2. If you have 10 runners but your pipeline still takes 1 hour, is the problem Hardware or YAML Design? (How would you tell?)
3. Why is "Distributed Cache" essential for high-availability runners?
4. Research: What is the GitLab "Runner Autoscale" feature for AWS?

---

Summary

You have completed Module 4: Advanced Pipeline Orchestration. You have moved beyond simple scripts and are now designing complex, cross-service automation engines that can handle the scale of a global enterprise.

Next Module: Truth in code: Module 5: Testing and Quality Assurance.