Module 1 Lesson 5: CPU vs GPU vs Apple Silicon
Choosing the right engine for your AI. A technical comparison of how different processors handle LLM workloads.
CPU vs GPU vs Apple Silicon: The AI Engine
Not all silicon is created equal. When you run Ollama, it detects your hardware and chooses the best "engine" to run the model. Understanding how these engines differ will help you optimize your setup.
1. The CPU (The Generalist)
Every computer has a CPU (Central Processing Unit). It is designed to handle a wide variety of tasks—opening tabs, running spreadsheets, managing the OS.
How it handles LLMs:
- Sequential Processing: CPUs have a few very powerful cores (4 to 16 usually). They process tasks one after another.
- The Bottleneck: LLMs require billions of simultaneous mathematical operations (matrix multiplications). A CPU is like a genius math professor who can do any math problem but only one at a time.
- When to use: If you have no dedicated GPU, Ollama will use "CPU Inference." It works, but it’s the slowest method.
2. The GPU (The Specialist)
A GPU (Graphics Processing Unit), specifically from NVIDIA, is the industry standard for AI.
How it handles LLMs:
- Parallel Processing: GPUs have thousands of small, specialized cores (CUDA cores). They can do thousands of simple math problems at the exact same time.
- CUDA: This is NVIDIA's secret sauce—a software layer that allows AI tools to talk directly to the GPU hardware.
- The Advantage: A GPU is like a stadium filled with 5,000 high school students all solving one simple addition problem at the same time. Together, they finish the work of the "genius professor" CPU in a fraction of a second.
- Best for: Windows and Linux users with NVIDIA cards.
Visualizing the Process
graph TD
Start[Input] --> Process[Processing]
Process --> Decision{Check}
Decision -->|Success| End[Complete]
Decision -->|Retry| Process
3. Apple Silicon (The New Contender)
Apple’s M1, M2, M3, and M4 chips changed the game for local AI through an architecture called Unified Memory.
How it handles LLMs:
- Shared Pool: In a traditional PC, the CPU has RAM and the GPU has VRAM. Moving data between them is slow. In a Mac, the CPU and GPU sit on the same chip and share the same memory.
- Metal Express: Apple uses the "Metal" API to accelerate AI, similar to how NVIDIA uses CUDA.
- The Advantage: You can buy a Mac with 128GB of RAM, and the GPU can use almost all of it. On a PC, buying 128GB of GPU VRAM would cost you $10,000+. On a Mac, it's a fraction of that.
- Best for: Laptops and compact workstations (Mac Studio) running very large models.
The Performance Gap
| Engine | Speed (8B Model) | Max Model Size (on 32GB RAM) |
|---|---|---|
| CPU Only | ~1-3 tokens/sec | Limited only by patience |
| NVIDIA GPU (8GB) | ~40-60 tokens/sec | ~8B (Fits in VRAM) |
| Apple M3 Max (36GB) | ~30-40 tokens/sec | ~20B (Fits in Unified Memory) |
Which Should You Get?
- If you want a Laptop: Get a MacBook with at least 16GB of RAM (32GB+ preferred). Apple Silicon is the undisputed king of "AI on the go."
- If you want a Desktop: Build a PC with an NVIDIA RTX 4060 Ti (16GB version) or an RTX 4090. CUDA is still the most compatible and fastest ecosystem for developers.
- If you have an old PC: Don't throw it away! Ollama's CPU mode is perfectly fine for basic prototyping and learning the CLI.
Conclusion
Ollama is "hardware agnostic," meaning it will try to give you the best experience regardless of what's under the hood. However, if you are serious about building AI applications, a GPU (NVIDIA) or Unified Memory (Apple) is a non-negotiable upgrade.
Key Takeaways
- CPUs are slow for AI because they process tasks sequentially.
- GPUs use parallel processing to achieve high-speed generation.
- Apple Silicon is unique because of its high-capacity Unified Memory, allowing it to run massive models that would normally require enterprise-grade GPUs.