Cache pagination options

Understanding how pagination options impact what data is cached.

Introduction

When you drop an add to cache shape into a process flow, there are two options that you should consider if your selected endpoint paginates the data that is received OR you generate multiple payloads in some other way (for example, via the flow control shape). These options are: save all pages and append.

Together, these two options determine how multiple payloads are cached, so it's important to understand the implications of each.

On this page we focus on paginated data however, the same principles apply whenever multiple payloads are cached, irrespective of whether those payloads are generated via pagination or some other means (for example, via the flow control shape).

Save all pages

When paginated data is pulled from a connection shape, a payload is created for each page - you can see these in the run log payload tab:

If you are caching paginated data and choose to toggle the save all pages option to on, the payload for each page is saved with its page number and a unique key. For example:

The unique key is generated dynamically, by adding the page number to your specified cache key. If the cache is a flow run type, the unique key will also incorporate the flow run id.

It's important to note that every time a connection shape pulls paginated data, page numbers reset to 1.

Append

When the append option is toggled ON, incoming payloads are appended to cache keys. How this works depends on the save all pages option:

Save all pages	Append to cache	Cache behaviour
OFF	OFF	The given cache key is overwritten each time a payload is cached. As such, the cache key will only ever include data from the LAST payload received.
ON	OFF	The first time that multiple payloads are received, each one is saved to its own unique key, against your specified cache key. Next time the cache receives data, any existing unique keys associated with your specified cache key are overwritten. Additional unique keys are created for new payloads/pages as needed). As such, each unique key will only ever contain the latest data for the correlating payload/page number.
OFF	ON	Each payload is appended to your specified cache key. As such, data in the cache key continues to grow with each data pull - nothing is overwritten.
ON	ON	The first time that multiple payloads are received, each one is saved to its own unique key, against your specified cache key. Next time the cache receives data, any existing unique keys associated with your specified cache key are appended with the latest data from correlating payload/page numbers. Additional unique keys are created for new payloads/pages as needed). As such, data associated with existing unique keys continues to grow with each data pull.

The diagram below illustrates this:

For information about setting the append option, please see our Appending data to a cache page.