Module 6 Lesson 4: Pagination & Rate Limits

When an API has 10,000 items, it won't give them all to you at once. It uses Pagination. You get Page 1, then Page 2, and so on.

1. The "Next Page" Pattern

Most APIs (like GitHub or Stripe) provide a next_page URL or a has_more boolean in the JSON.

1. Request: Get Page 1.
2. Action: Process items.
3. Check: {{ $json.has_more === true }}.
4. Loop: If true, increment the page variable and go back to Step 1.

---

2. Page vs. Offset vs. Cursor

• Page: ?page=2. (Simple).
• Offset: ?limit=100&offset=100. (Math required: offset = offset + limit).
• Cursor: ?after=xyz789. (The "ID" of the last item in the previous request).

---

3. Respecting the Speed Limit (Rate Limits)

If an API allows 60 requests per minute, and you try to do 60 in 1 second, you will get a 429 Too Many Requests error.

• The Solution: Add a Wait node (Module 3) at the end of your loop. Set it to 1 second.
• Bonus: Use a Code node to dynamically calculate the wait time based on the X-RateLimit-Remaining header returned by the API!

---

4. Why Use "Split In Batches" here?

Don't. Pagination loops should usually be handled with a Manual Loop (Module 5, Lesson 4) because Split In Batches assumes you already HAVE the full list. In pagination, you are "Building" the list as you go.

---

Exercise: The Collector's Loop

1. Research: How does the Shopify or Airtable API handle pagination? (Is it a cursor or a page number?)
2. Create a manual loop that starts with page = 1. Each loop, increment the page by 1. Stop when page = 5. (Use an IF node).
3. If an API says "Retry after 30 seconds," how would you use n8n's Retry settings (Module 5, Lesson 3) to handle this automatically?
4. Search: What is the "Paginiator" node in the n8n community? (Is it better than a manual loop?)

---

Summary

Pagination and Rate Limits are the "Rules of the Road." By mastering these patterns, you ensure that your automation can ingest any amount of data, no matter how restrictive or large the external API is.

Next Lesson: Troubleshooting: Debugging External API Failures.