Cache pagination options

Understanding how pagination options impact what data is cached.

Introduction

When you drop an add to cache shape into a process flow, there are two options that you should consider if your selected endpoint paginates the data that is pulled - these are: save all pages and append.

Together, these two options determine how paginated data is cached, so it's important to understand the implications of each.

Save all pages

When paginated data is pulled from a connection shape, a payload is created for each page - you can see these in the run log payload tab:

If you are caching paginated data and toggle the save all pages option to ON, the payload for each page is saved to its own cache key.

The name of this key is generated dynamically, by adding the page number as a suffix to your given cache key for the add to cache shape. Consider the example below:

In this example, our given cache key is called demokey and the save all pages option is toggled ON. So, if received data is paginated into 5 pages, there will be 5 payloads to be cached. These would be named:

• demokey.1

• demokey.2

• demokey.3

• demokey.4

• demokey.5

Here, .n is the page number suffix. If/how these cache keys can be accessed depends on how the append option is set.

It's important to note that every time a connection shape pulls paginated data, page numbers reset to 1.

Append

When the append option is toggled ON, payloads are appended to the given cache key. How this works depends on the save all pages option:

• If the save all pages option is toggled OFF and the append option is toggled OFF, the given (static) cache key is overwritten every time the payload for a page is added to the cache. In this scenario, the cache key will always include data for the latest page from the latest pull.

• If the save all pages option is toggled ON and the append option is toggled ON, dynamic cache keys will be created on the first pull, and all subsequent pulls will append paginated payloads to the correlating key (additional cache keys are created for new page numbers, as needed). In this scenario, dynamic cache keys will continue to grow with each data pull as each one will include all payloads received for the correlating page number.

• If the save all pages option is toggled OFF and the append option is toggled ON, the payload for each page is appended to the given (static) cache key. In this scenario, your single (static) cache key continues to grow with each data pull - nothing is overwritten.

• If the save all pages option is toggled ON and the append option is toggled OFF, dynamic cache keys will be created on the first pull, and all subsequent pulls will overwrite data in the correlating key (additional cache keys are created for new page numbers, as needed). In this scenario, dynamic cache keys will only ever contain the latest data for the correlating page number.

The diagram below illustrates the above:

For information about setting the append option, please see our Appending data to a cache page.