Introduction

When an add to cache shape is dropped into a process flow, the entire incoming payload is cached and associated with the given cache key. Depending on the cache type, you can load this cache later in the same flow or in a different flow.

In the simplest scenario, your given cache key would be a static value (e.g. customers) and you would use this to load the entire cache (containing perhaps tens, hundreds, even thousands of items) where required. But what if you want to load a specific item from a cache, rather than the whole thing?

This is where dynamic cache keys are so useful.

How it works

To load data from a cache, you configure a load from cache shape with the required cache and a single cache key. All data associated with your given cache key is loaded.

Consider the example incoming payload below, where four records are cached with a static cache key with a value of customers:

[
    {
        "id": 1000000001,
        "first_name": "Jane",
        "last_name": "Smith",
        "items": {
           "itemref": "0000001",
            "item1": "apples",
            "item2": "oranges",
            "item3": "pears"
        }
    },
    {
        "id": 1000000002,
        "first_name": "George",
        "last_name": "Jones",
        "items": {
           "itemref": "0000002",
            "item1": "tangerines",
            "item2": "peaches",
            "item3": "grapes"
        }
    },
    {
        "id": 1000000003,
        "first_name": "Bob",
        "last_name": "Brown",
        "items": {
           "itemref": "0000003",
            "item1": "nectarines",
            "item2": "raspberries",
            "item3": "strawberries"
        }
    },
    {
        "id": 1000000004,
        "first_name": "Marjorie",
        "last_name": "Simpson",
        "items": {
           "itemref": "0000004",
            "item1": "blueberries",
            "item2": "cranberries",
            "item3": "apricots"
        }
    }
]

If we were to configure a load from cache shape to access the customers cache key, all four records would be loaded.

So, in order to load specific items from a cache, the incoming data must be added to a cache in such a way that we can easily target individual items. We need an efficient way to take incoming data, batch it into single-record payloads and add each of these to the cache with its own unique, identifying cache key - i.e.:

We can achieve this as follows:

Need to know

• When you specify a dynamic variable as the cache key, the value for that variable is injected into the key. To prevent the case where large amounts of data are passed into the key, there is a character limit is 128 characters.

• Any combination of payload, flow and metadata variables can be used to form cache key names.

Using a variable to generate dynamic cache keys

Follow the steps below to configure an add to cache shape with a payload variable for generating dynamic cache keys.

Step 1 Drop an add to cache shape into your process flow, where required.

Step 2 In the add to cache shape settings, choose to create cache:

Step 3 Set the cache level and name as required and save changes.

Step 4 Select the cache that you just created - for example:

Step 5 Move down to the cache key field and enter the required key. Here, you use standard payload variable syntax to define your target data element:

[[payload.<schema notation>]]

...where schema notation should be replaced with the notation path to the first occurrence of the required element in the payload which should be used to form the cache key. If required, you can also include a static prefix or suffix. For example:

customer-[[payload.0.id>]]

The output of the payload variable will be used as the cache key.

Step 6 Save the add to cache shape settings.

Loading data from a cache

Cached data can be loaded via our load from cache shape. Please refer to the Load from cache shape section for more information.

Introduction

When you drop an add to cache shape into a process flow, there are two options that you should consider if your selected endpoint paginates the data that is received OR you generate multiple payloads in some other way (for example, via the flow control shape). These options are: save all pages and append.

Together, these two options determine how multiple payloads are cached, so it's important to understand the implications of each.

Save all pages

When paginated data is pulled from a connection shape, a payload is created for each page - you can see these in the run log payload tab:

If you are caching paginated data and choose to toggle the save all pages option to on, the payload for each page is saved with its page number and a unique key. For example:

The unique key is generated dynamically, by adding the page number to your specified cache key. If the cache is a flow run type, the unique key will also incorporate the flow run id.

Append

When the append option is toggled ON, incoming payloads are appended to cache keys. How this works depends on the save all pages option:

The diagram below illustrates this:

Introduction

The add to cache shape is used to cache (i.e. store a copy of) the payload as it stands at that point in the process flow.

You can add as many add to cache shapes as you like in a process flow. For example, you might place want to cache a payload as soon as it gets pulled from a source connection, and again later after it's been transformed. For example:

How long a cached payload remains available depends on the cache level selected when you configured the add to cache shape in your process flow.

Need to know

• When a process flow hits an add to cache shape, all data from the incoming payload is cached. With this in mind, ensure that your incoming data is filtered, split and/or batched as required.

• The default behaviour is for the existing cache to be overwritten each time it is updated. Please see the Appending data to a cache page for information about appending data.

• The maximum cache size is 50MB.

• Cache names must not include full stop (.) or colon (:) characters.

• Cached data is stored in Amazon S3.

Adding & configuring an add to cache shape to a process flow

To add an add to cache shape to a process flow, follow the steps below.

Step 1 Find the point in your process flow where you want to cache the payload - typically this would be after a 'GET' connection shape, or perhaps after data has been mapped or manipulated via a script.

Step 2 Select the add to cache shape from the shapes palette:

Step 3 Click the create cache option:

...cache options are displayed:

Step 4 Click in the cache level > select cache field to choose when/where this cache will be available:

Choose from the following options:

Step 5 Enter a name for this cache:

Step 6 If you have chosen a flow-level or company-level cache, you can set a data retention period to determine when this data will expire - for example:

Step 7 Save changes to exit back to add to cache settings where you can continue with your newly created cache.

Step 8 Click in the select a cache field and select your new cache from the list:

Step 9 Enter a cache key to identify this cache object - for example:

Your cache key can be:

Step 10 If you have multiple incoming payloads (typically where source data is paginated or has been through flow control), you should consider how these payloads are cached. The save all pages option determines cache behaviour for multiple incoming payloads:

• Save all pages toggled ON. All incoming payloads are saved for your cache key. If you access the cache, you'll see each page listed with a page number - for example:
• Save all pages toggled OFF. Data associated with the given cache key is overwritten each time one of the multiple payloads is saved - so only the final payload is saved - for example:

Step 11 Set the append option as required. If this option is toggled ON, incoming data is appended to the existing cache key each time an update is made. If this option is toggled OFF, the cache key is overwritten with new data each time.

Step 12 Save changes. The add to cache shape is added to your process flow, displaying the given name and key - for example:

Can I see the payload for an add to cache shape?

Yes. As with any other process flow shape, you can view the associated payload for an add from cache shape after the process flow has run. To do this, click the shape's tick icon and then select the payload tab in the run log panel - for example:

Multiple payloads

If you place an add to cache shape before a shape which generates multiple payloads (typically, a flow control shape), you can see each payload that is created via the payload dropdown - for example:

Loading data from a cache

Cached data can be loaded via our load from cache shape. Please refer to the Load from cache shape section for more information.

Introduction

We've already noted how the add to cache shape can be added to a process flow to cache the entire payload at a given point in the flow. The default behaviour is that when a process flow runs and hits an add to cache shape, any existing data associated with that cache is overwritten with a new payload from the new run.

However, it is possible to append data to a cache, so each time the process flow runs and the add to cache shape is reached, the current cache is appended to the existing cache. This works for any cache type (flow, flow run, and company).

Need to know

• Paginated data. If your connection shape receives paginated data, it's important to understand how the save all pages option works in conjunction with append. For more information please see our cache pagination options page.

• Cache size. Theoretically, if a cache is set to append data and then runs on a regular basis indefinitely, the cache size may grow to an unmanageable size. With this in mind, a limit is in place to ensure that a single cache cannot exceed 50MB.

• Append data format. Appending cached data is supported for JSON only.

• Shared caches. The append to cache operation is not atomic - as such we advise against multiple process flows attempting to update the same cache at the same time.

Using the append option

To use the append option, follow the steps below.

Step 1 Drop an add to cache shape into your process flow in the normal way - create your cache, then select it and add your cache key.

Step 2 Ensure that the save all pages option is set as needed. For more information about how this option affects appended data please see our cache pagination options page.

Step 3 Enable the append option:

Step 4 A path to append to field is displayed:

Here, you need to consider the structure of the payload that you're passing in and specify a path that ensures that each new payload is appended in the right place.

Step 5 Save the shape. Next time the process flow runs the data will be cached and appended.

Viewing the appended cache

If you choose to view the payload for an add to cache shape, the payload will always show data from the latest run - for example:

However, when you add a load from cache shape, the payload will show ALL appended data so far - for example: