Cache pagination options
Understanding how pagination options impact what data is cached.
Last updated
Understanding how pagination options impact what data is cached.
Last updated
When you drop an add to cache shape into a process flow, there are two options that you should consider if your selected endpoint paginates the data that is received OR you generate multiple payloads in some other way (for example, via the flow control shape). These options are: save all pages
and append
.
Together, these two options determine how multiple payloads are cached, so it's important to understand the implications of each.
On this page we focus on paginated data however, the same principles apply whenever multiple payloads are cached, irrespective of whether those payloads are generated via pagination or some other means (for example, via the flow control shape).
When paginated data is pulled from a connection shape, a payload is created for each page - you can see these in the run log payload tab:
If you are caching paginated data and choose to toggle the save all pages
option to on
, the payload for each page is saved with its page number and a unique key
. For example:
The unique key
is generated dynamically, by adding the page number to your specified cache key. If the cache is a flow run
type, the unique key
will also incorporate the flow run id
.
It's important to note that every time a connection shape pulls paginated data, page numbers reset to 1.
When the append option is toggled ON, incoming payloads are appended to cache keys. How this works depends on the save all pages option:
The given cache key
is overwritten each time a payload is cached. As such, the cache key
will only ever include data from the LAST payload received.
The first time that multiple payloads are received, each one is saved to its own unique key
, against your specified cache key
.
Next time the cache receives data, any existing unique keys
associated with your specified cache key
are overwritten. Additional unique keys
are created for new payloads/pages as needed).
As such, each unique key
will only ever contain the latest data for the correlating payload/page number.
Each payload is appended to your specified cache key
. As such, data in the cache key continues to grow with each data pull - nothing is overwritten.
The first time that multiple payloads are received, each one is saved to its own unique key
, against your specified cache key
.
Next time the cache receives data, any existing unique keys
associated with your specified cache key
are appended with the latest data from correlating payload/page numbers. Additional unique keys
are created for new payloads/pages as needed).
As such, data associated with existing unique keys
continues to grow with each data pull.
The diagram below illustrates this:
For information about setting the append option, please see our Appending data to a cache page.