Heavy Lifting: Celery, Redis, and Task Queues

FastAPI's built-in background tasks are great for small things, but if you need to process thousands of images, train AI models, or run 24-hour data scraping jobs, you need a Distributed Task Queue.

In this lesson, we explore the industry-standard architecture for heavy background work.

---

1. Why Move Beyond Built-in Tasks?

1. Isolation: If a background task crashes a built-in worker, it could crash your whole API. With a separate queue, only the worker crashes.
2. Concurrency: You can have 1 FastAPI server and 100 Background Workers on different machines.
3. Persistence: If your API server restarts, built-in tasks are lost. With a queue like Redis, tasks stay in the queue until a worker picks them up.

---

2. The Architecture: FastAPI + Redis + Celery

• The Producer: Your FastAPI app. It "Produces" a task and puts it in the queue.
• The Broker: Redis or RabbitMQ. It holds the "Waitlist" of tasks.
• The Consumer: Celery. A separate Python process that reads from the queue and does the work.

# tasks.py (Celery Worker)
from celery import Celery

celery_app = Celery("tasks", broker="redis://localhost:6379/0")

@celery_app.task
def process_video_heavy(video_id: str):
    # This might take 10 minutes!
    return "Done"

# main.py (FastAPI App)
@app.post("/upload")
async def upload_video(video_id: str):
    # Put task in Redis and return immediately
    process_video_heavy.delay(video_id)
    return {"status": "Processing initiated"}

---

3. ARQ: The Async Alternative

If you want a modern, fully async task queue that feels more like FastAPI, check out ARQ. It uses Redis and provides a very similar experience to Celery but with native async/await support throughout.

---

4. Monitoring Tasks

In production, you need to know if your tasks are failing.

• Flower: A real-time web UI for monitoring Celery tasks. It shows you which tasks succeeded, which failed, and how long they took.

---

Visualizing Distributed Tasks

graph LR
    A["FastAPI App (Producer)"] -- "delay()" --> B["Redis (Broker)"]
    B -- "Fetch" --> C["Celery Worker 1"]
    B -- "Fetch" --> D["Celery Worker 2"]
    B -- "Fetch" --> E["Celery Worker 3"]
    
    subgraph "Background Infrastructure"
    B
    C
    D
    E
    end

---

Summary

• Built-in: For light, non-critical I/O (Emails).
• Celery/Redis: For heavy, long-running, or critical tasks (Video processing, AI training).
• Broker: The storage layer that ensures tasks aren't lost during crashes.
• Scalability: Distributed queues allow you to scale your compute independently from your API.

In the next lesson, we wrap up Module 11 with Exercises on async architecture.

---

Exercise: The Architect's Decision

You are building an AI Image Generation API (like Midjourney).

1. When a user hits @app.post("/generate"), should you use a built-in Background Task or an external Task Queue?
2. Why? (Consider time, CPU usage, and reliability).