Handling Pagination in Make.com APIs

Introduction

You built the perfect automation. The logic is sound, the mappings are clean, and the test run returned a beautiful JSON payload. But a week later, you notice a discrepancy. Your CRM has 5,000 new leads, but your automation only processed 50.

You just hit the Pagination Wall.

Pagination is the method APIs use to break large datasets into manageable chunks, or “pages.” This preserves server performance. For a human browsing a website, clicking “Next Page” is easy. For an automated workflow in Make.com automation, it determines if your system works or fails.

If you are building self-driving ecosystems rather than simple tasks—the core mission of Thinkpeak.ai—incomplete data is unacceptable. An AI agent that only sees the first 50 rows of your sales data isn’t intelligent. It is hallucinating based on a partial reality.

In this guide, we move beyond basic documentation. We will explore how to architect robust pagination workflows in Make.com. We cover everything from simple logic to complex cursor-based navigation used by tools like Shopify and Slack.

The Three Types of API Pagination

Before dragging modules onto your canvas, you must identify how your specific API handles data. Open your API documentation and look for “Pagination.” You will typically find one of three models.

1. Offset or Page-Based (The Classical Model)

This is the most intuitive method. You tell the API to give you “page 3” or to skip the first 200 records and give the next 100.

• Parametreler: page=1, limit=100 OR offset=200, limit=100.
• Make.com Difficulty: Düşük.

2. Cursor-Based (The Modern Standard)

High-volume platforms like Meta, Stripe, and Slack use cursor-based pagination to prevent data drift. The API provides a unique “token” pointing to the last item you received. You must pass this token back to get the next batch.

• Parametreler: starting_after=xh52..., next_page_token=abc....
• Make.com Difficulty: High (requires dynamic looping).

3. Link Header (The “Hidden” Model)

The API doesn’t put pagination details in the JSON body. Instead, it hides the “Next Page URL” inside the HTTP Response Headers.

• Make.com Difficulty: Medium (requires header parsing).

Method 1: The Native HTTP Module Solution

Many users ignore the advanced settings in Make’s standard HTTP > Make a request module. They are unaware that native pagination support exists.

When to use it: Use this for simple, standard APIs. This works best when you just need to increment a query parameter until the data stops.

How to configure:

1. Open your HTTP module.
2. Scroll to the bottom and check “Show advanced settings”.
3. Check the “Pagination” option.

This reveals three new fields:

• URL/Query String: Define the parameter to increment (e.g., Sayfa).
• Merge with parent: Keep this Evet. It merges your pagination params with your original request.
• Condition: This is the “Stop” button. You must write a formula that tells Make when to quit.

The “Condition” Formula:

You usually check if the response body is empty.

{{length(body.data) > 0}}

As long as this formula is TRUE, Make will keep requesting the next page.

Why this often fails: The native feature is opaque. It is hard to debug why a loop stopped. For enterprise-grade reliability, we recommend building the logic explicitly using the methods below.

Method 2: The Repeater Loop

This is the gold standard for controlling your data flow. It separates the counting logic from the fetching logic. This works best for offset pagination.

Scenario: Fetching 5,000 leads, 100 at a time.

Step 1: The “Scout” Request

First, make a single call to the API to get the total count of records.

• Module: HTTP (Make a request)
• URL: .../leads?limit=1
• Çıktı: You receive JSON saying "total": 5000.

Step 2: Calculate the Loops

Kullanım Set Variable module to calculate how many pages you need.

• Formula: {{ceil(1.data.total / 100)}}
• Sonuç: 5000 / 100 = 50 pages.

Step 3: The Repeater

Add the Flow Control Repeater Modül.

• Initial value: 1
• Repeats: Map the “Total Pages” variable you calculated in Step 2.

Step 4: The Paginated Request

Add your main HTTP module after the Repeater.

• URL: .../leads
• Query String: Set limit için 100 ve Sayfa to the i value from the Repeater module.

Step 5: Aggregation

Do not process the data inside the loop immediately. Add an Array Aggregator after the HTTP module.

• Source Module: Repeater.
• Target: Bu data[] array from your HTTP module.

Sonuç: The Repeater runs 50 times. The Aggregator waits for all 50 runs to finish. Then, it releases a single bundle containing all 5,000 leads. You can now map this massive array to your database in one go.

Thinkpeak.ai Pro Tip: If you are dealing with massive datasets (10k+ rows), the Array Aggregator might hit Make’s memory limit. In that case, our Bespoke Engineering team uses streaming architectures to process data in batches.

Method 3: Handling Cursor Pagination

Cursor pagination is tricky because you do not know how many pages exist. You must “keep going” until the API stops giving you a next_token. Make.com lacks a native Do-While loop, so we must engineer one.

Mimari: We use a Repeater set to a high safety limit. We pass the “Cursor” token from one iteration to the next using Make’s variable storage.

Step 1: Initialize the Token

• Module: Tools > Set Variable
• Variable Name: next_cursor
• Value: Leave empty.

Step 2: The Safety Loop

• Module: Repeater
• Repeats: 100 (or a number higher than you ever expect to reach).

Step 3: Retrieve the Current Token

Inside the loop, grab the token from the previous run.

• Module: Tools > Get Variable
• Variable Name: next_cursor

Step 4: The API Call

• Module: HTTP
• Query String: Map cursor to the value from the Get Variable module.

Step 5: Update the Token

• Module: Tools > Set Variable
• Variable Name: next_cursor
• Value: Map the next_page_token from the HTTP response body.

Step 6: The “Break” Logic

If the API returns no token, you must stop the Repeater.

• Module: Flow Control > Router (or a Filter)
• Filter Condition: next_cursor (from the response) Does Not Exist.
• Eylem: Error Handler > Commit. This forces the scenario to succeed and stop processing.

This method works, but it is complex to debug. If you prefer a plug-and-play solution, check the templates at Thinkpeak.ai.

Handling Rate Limits

When you paginate efficiently, you create a new problem: speed. You might hit the API 50 times in 2 seconds. This triggers a 429 Too Many Requests error.

The Solution: The Sleep Directive

1. Right-click your HTTP module.
2. Select “Add Error Handler”.
3. Choose the Break directive.
4. Configure Number of attempts to 3 and Interval to 60 seconds.

This tells Make to wait 60 seconds and try again automatically if the API blocks you.

Blueprints for Success: Thinkpeak.ai’s Approach

Pagination is just one layer of the “Data Integrity” stack. At Thinkpeak.ai, we transform these workflows into autonomous systems.

Enterprise logic often requires more than standard methods. We implement cross-reference pagination to avoid duplicates. We use recursive webhooks for infinite pagination on millions of rows. We also build self-healing pipelines that automatically re-fetch missing intervals.

How We Help:

• Otomasyon Pazaryeri: We provide “Universal API Paginator” templates. These pre-architected modules handle Offset, Cursor, and Header pagination out of the box.
• Ismarlama Dahili Araçlar: We build backend infrastructure for unique data logic. We ensure your internal portals show 100% of the data.
• Özel Yapay Zeka Aracı Geliştirme: Pagination feeds your AI. Our Dijital Çalışanlar use rigorous data ingestion protocols to make accurate decisions.

Common Pitfalls to Avoid

1. The Infinite Loop

Your operations count drains to zero in minutes. This happens when your stop condition is wrong. Always set a hard limit on your Repeater while testing.

2. The Memory Crash

The scenario fails with a “RuntimeError: Memory limit exceeded.” This occurs when using an Array Aggregator on huge datasets. Fix this by processing data in batches. Aggregate 10 pages, upload them, clear memory, and continue.

3. The “Off-By-One” Error

You miss the first page or duplicate the last one. APIs index differently. Some start at page 0, others at page 1. Always manually test the first two requests in Postman or Make’s DevTool.

Sonuç

Handling pagination in Make.com separates amateur tinkerers from professional automation architects. You must master the Repeater module and understand cursor logic. This ensures your business decisions rely on complete, accurate data sets.

Whether syncing leads or feeding an AI model, data integrity is non-negotiable.

Ready to stop fixing broken loops and start building self-driving ecosystems? Visit Thinkpeak.ai for ready-to-use pagination templates or to contact our engineering team.

Sıkça Sorulan Sorular (SSS)

Can I handle pagination without using the Repeater module?

Yes, if the API supports it. You can use the native “Pagination” feature inside the HTTP module settings. However, this offers less visibility and error handling control than the Repeater method. Use the native feature for simple requests and the Repeater for complex logic.

How do I handle APIs that put the “Next URL” in the Link Header?

You need to parse the header. In your HTTP module, map the Header: Link output. Use a Text Parser module with a regex pattern to extract the URL. Then, pass this URL into your next request loop variable.

What happens if my scenario times out while paginating?

Make.com scenarios have a maximum execution time. If your pagination takes longer, use a recursive architecture. Split the job into smaller batches. Have the scenario call a Webhook at the end of its run to trigger the next batch in a fresh execution.

Kaynaklar

• Pagination, sorting and filtering | Make API | Make Developer Hub: https://developers.make.com/api-documentation/pagination-sorting-filtering
• Pagination | Custom Apps Documentation | Make Developer Hub: https://developers.make.com/custom-apps-documentation/component-blocks/api/pagination
• Pagination in list/search modules | Custom Apps Documentation | Make Developer Hub: https://developers.make.com/custom-apps-documentation/debugging-your-app/debugging-of-pagination-in-list-search-modules
• Implementation of Dynamic Pagination Using nextPageToken – Questions – Make Community: https://community.make.com/t/implementation-of-dynamic-pagination-using-nextpagetoken/78641
• Make.com (quick tutorial) How to Handle Next Page Token for any API – YouTube API used in example: https://www.youtube.com/watch?v=6_P9emwOenk