API Pagination Patterns in Clockspring

Most APIs paginate results, but they don’t all do it the same way. Some return a full URL for the next page. Others use page numbers, offsets, or cursor tokens.

In Clockspring, the loop pattern stays the same. What changes is how you extract or build the next request using attributes and Expression Language (EL), and how you validate that request against what the API actually returned.

The One Pattern That Never Changes

Every pagination flow follows this structure:

1. Store request state in FlowFile attributes

2. Call the API using InvokeHTTP

3. Extract pagination information from the response

4. Overwrite the attribute used for the next request

5. Route back to InvokeHTTP if another page exists

6. Exit cleanly when it does not

Pagination logic lives in attributes. Processors stay generic.

Pagination Type 1: Next URL Pagination

How it works

The API response includes the full URL for the next page.

Example response:

{  "next": "https://api.example.com/items?page=2"}

How to handle it

Use EvaluateJsonPath to overwrite the url attribute:

• Property name: url

• JSONPath: $.next

InvokeHTTP uses:

${url}

If url is empty, pagination is complete. No URL construction is required.

Pagination Type 2: Page Number Pagination

How it works

The API expects a page number in the request.

Example request:

https://api.example.com/items?page=1

Attributes to track

base_url = https://api.example.com/items
page = 1

Incrementing the page with EL

After each successful response:

${page:toNumber():plus(1)}

Rebuild the URL:

${base_url}?page=${page}

Pagination Type 3: Offset and Limit Pagination

How it works

The API uses record offsets instead of page numbers.

Example request:

https://api.example.com/items?offset=0&limit=100

Attributes to track

base_url = https://api.example.com/items
offset = 0
limit = 100

Incrementing the offset

${offset:toNumber():plus(${limit})}

Rebuild the URL:

${base_url}?offset=${offset}&limit=${limit}

Pagination Type 4: Cursor or Token Pagination

How it works

The API returns an opaque token indicating where to continue.

Example response:

{  "next_cursor": "eyJpZCI6MTIzfQ=="
}

How to handle it

Extract the token:

• Property name: cursor

• JSONPath: $.next_cursor

Build the next request:

${base_url}?cursor=${cursor}

Stop when the cursor attribute is empty. Never attempt to interpret or modify cursor values.

Don’t Trust Your Counter: Validate Against the API Response

When you build pagination requests yourself using EL, your local counters can drift. If that happens, you can end up requesting pages that do not exist.

Common causes:

• Incorrect starting values

• Retries replaying stale attributes

• Incrementing after a failed request

• API-side pagination changing mid-run

Rule: Only request the next page if the API says it exists.

If the API returns total_pages

Extract values like:

• page = $.page

• total_pages = $.total_pages

Route using conditions such as:

• Continue if: ${page:toNumber():lt(${total_pages:toNumber()})}

• Stop otherwise

This prevents requests like page 82 when only 3 pages exist.

If the API returns has_more

Extract:

• has_more = $.has_more

Route:

• Continue if: ${has_more:equals('true')}

• Stop otherwise

This is often safer than comparing counters.

If the API does not return page counts

For offset-based APIs:

• Stop when the response returns zero records

• Or when the number of records returned is less than the requested limit

Let the API response determine when pagination ends.

Best practice for page-based APIs

If the API returns the current page number in the response, always use that value instead of assuming your counter is correct. The API is the source of truth.

What Changes vs What Stays the Same

Always the same

• Loop structure

• InvokeHTTP configuration

• Attribute-driven routing

Varies by API

• JSONPath expressions

• Attributes you update

• Stop conditions

Once you understand one pagination style, the others are small variations.

Common Mistakes

• Hardcoding URLs instead of using attributes

• Incrementing counters without validating API responses

• Forgetting numeric conversion in EL

• Infinite loops due to missing stop conditions

• Mixing pagination logic with record processing

Pagination should control the loop. Data handling should remain separate.

Summary

Clockspring handles all API pagination styles using the same attribute-driven loop. Expression Language builds the next request, but the API response determines whether another page actually exists.

Trust the API, not your counter, and pagination loops stay predictable, safe, and easy to reuse.