Appending data to a cache

Introduction

We've already noted how the add to cache shape can be added to a process flow to cache the entire payload at a given point in the flow. The default behaviour is that when a process flow runs and hits an add to cache shape, any existing data associated with that cache is overwritten with a new payload from the new run.

However, it is possible to append data to a cache, so each time the process flow runs and the add to cache shape is reached, the current cache is appended to the existing cache. This works for any cache type (flow, flow run, and company).

Need to know

• Paginated data. If your connection shape receives paginated data, it's important to understand how the save all pages option works in conjunction with append. For more information please see our cache pagination options page.

• Cache size. Theoretically, if a cache is set to append data and then runs on a regular basis indefinitely, the cache size may grow to an unmanageable size. With this in mind, a limit is in place to ensure that a single cache cannot exceed 50MB.

• Append data format. Appending cached data is supported for JSON only.

• Shared caches. The append to cache operation is not atomic - as such we advise against multiple process flows attempting to update the same cache at the same time.

Using the append option

To use the append option, follow the steps below.

Step 1 Drop an add to cache shape into your process flow in the normal way - create your cache, then select it and add your cache key.

Step 2 Ensure that the save all pages option is set as needed. For more information about how this option affects appended data please see our cache pagination options page.

Step 3 Enable the append option:

Step 4 A path to append to field is displayed:

Here, you need to consider the structure of the payload that you're passing in and specify a path that ensures that each new payload is appended in the right place.

Example

Consider the payload below:

{
  "orders": [
    {
      "id": 1,
      "customer": {
        "email": "[email protected]"
      }
    },
    {
      "id": 2,
      "customer": {
        "email": "[email protected]"
      }
    }
  ]
}

In this case, we want to append new data to the orders object, so our path to append to would be defined as orders. The first time the cache is updated, the payload would be:

[{"orders":[{"id":1,"customer":{"email":"[email protected]"}},{"id":2,"customer":{"email":"[email protected]"}}]}]

The next time the payload is appended, it would be in the form below:

{"0":{"orders":[{"id":1,"customer":{"email":"[email protected]"}},{"id":2,"customer":{"email":"[email protected]"}}]},"orders":[{"orders":[{"id":3,"customer":{"email":"[email protected]"}},{"id":4,"customer":{"email":"[email protected]"}}]}]}

If required, flow variables can be specified here.

Step 5 Save the shape. Next time the process flow runs the data will be cached and appended.

Viewing the appended cache

If you choose to view the payload for an add to cache shape, the payload will always show data from the latest run - for example:

However, when you add a load from cache shape, the payload will show ALL appended data so far - for example: