Add to cache shape

Introduction

The add to cache shape is used to cache (i.e. store a copy of) the payload as it stands at that point in the process flow.

You can add as many add to cache shapes as you like in a process flow. For example, you might place want to cache a payload as soon as it gets pulled from a source connection, and again later after it's been transformed. For example:

How long a cached payload remains available depends on the cache level selected when you configured the add to cache shape in your process flow.

Need to know

During routine platform maintenance, cached data may be cleared. While we make a best effort to retain data for up to 7 days, it could be cleared sooner. Please design your process flows accordingly.

• When a process flow hits an add to cache shape, all data from the incoming payload is cached. With this in mind, ensure that your incoming data is filtered, split and/or batched as required.

• The default behaviour is for the existing cache to be overwritten each time it is updated. Please see the Appending data to a cache page for information about appending data.

• The maximum cache size is 50MB.

• Cache names must not include full stop (.) or colon (:) characters.

• Cached data is stored in Amazon S3.

Adding & configuring an add to cache shape to a process flow

To add an add to cache shape to a process flow, follow the steps below.

Step 1 Find the point in your process flow where you want to cache the payload - typically this would be after a 'GET' connection shape, or perhaps after data has been mapped or manipulated via a script.

Step 2 Select the add to cache shape from the shapes palette:

Step 3 Click the create cache option:

...cache options are displayed:

Step 4 Click in the cache level > select cache field to choose when/where this cache will be available:

Choose from the following options:

Cache level	Summary
Flow run	The data associated with this cache is only available while the process flow is running. When a process flow run completes, existing cached data is deleted. How this happens depends on whether the process flow is enabled & deployed. Enabled & deployed process flows In this case, the flow run cache is cleared as soon as the process flow completes. Draft/inactive process flows In this case, we use a TTL (Time to Live) with a default of 2 hours to determine when the cached data is deleted. There's no chance that flow run cached data could be re-used in the TTL deletion window - each time a flow run cache is used, a unique flow run id is added to the cache key used to get and set data. Because every process flow run has a unique run id, there's no possibility for another flow run to access the data from a previous run.
Flow	Data in the cache is retained after the process flow is run, so it can be loaded again within this process flow if required. Cache retention When you choose to add a flow cache, retention options are available so you can decide how long cached data should be retained (you can set a time limit in seconds, minutes, hours, or days). The default setting is 2 hours. This can be updated to a maximum of 7 days.
Company	The data associated with the latest update to this cache is available for use in this process flow and in any other process flows created within your company profile. It's not currently possible to access different versions of a cache. So, each time a process flow runs with the same add to cache shape, the payload for that cache is overwritten/appended with the latest data and it's this that will be available to load from a company cache. Cache retention When you choose to add a company cache, retention options are available so you can decide how long cached data should be retained (you can set a time limit in seconds, minutes, hours, or days). The default setting is 2 hours. This can be updated to a maximum of 7 days.

Show me how the different cache levels work

Step 5 Enter a name for this cache:

The cache name must not include full stop (.) or colon (:) characters.

Step 6 If you have chosen a flow-level or company-level cache, you can set a data retention period to determine when this data will expire - for example:

The data retention period for a flow run-level cache is always 2 hours - this cannot be changed. The maximum retention period for a flow-level or company-level cache is 7 days.

Step 7 Save changes to exit back to add to cache settings where you can continue with your newly created cache.

Step 8 Click in the select a cache field and select your new cache from the list:

Step 9 Enter a cache key to identify this cache object - for example:

Your cache key can be:

Cache key	Summary	Example
Static	Data is cached to the key exactly as it is specified. Typically used when your aim it to load the entire cache later in the flow (or in other flows).	orders
Dynamic	The cache key resolves dynamically using variables Typically used when your aim it to load single or multiple items from the cache later in the flow (or in other flows). For more information please see our Generating dynamic cache keys with variables.	order-[[payload.0.id]]

A cache key cannot exceed 128 characters.

If you are adding a company-level cache, you may want to make a note of the key that you specify here, so it can be shared with other users in your organisation who may want to reference this cache in their process flows.

Step 10 If you have multiple incoming payloads (typically where source data is paginated or has been through flow control), you should consider how these payloads are cached. The save all pages option determines cache behaviour for multiple incoming payloads:

• Save all pages toggled ON. All incoming payloads are saved for your cache key. If you access the cache, you'll see each page listed with a page number - for example:

  
• Save all pages toggled OFF. Data associated with the given cache key is overwritten each time one of the multiple payloads is saved - so only the final payload is saved - for example:

  

It's important to understand how the save all pages option works in conjunction with the append option. If you aren't sure, please see our Cache pagination options page before proceeding.

Step 11 Set the append option as required. If this option is toggled ON, incoming data is appended to the existing cache key each time an update is made. If this option is toggled OFF, the cache key is overwritten with new data each time.

For more information see our Appending data to a cache page.

Step 12 Save changes. The add to cache shape is added to your process flow, displaying the given name and key - for example:

Can I see the payload for an add to cache shape?

Yes. As with any other process flow shape, you can view the associated payload for an add from cache shape after the process flow has run. To do this, click the shape's tick icon and then select the payload tab in the run log panel - for example:

Multiple payloads

If you place an add to cache shape before a shape which generates multiple payloads (typically, a flow control shape), you can see each payload that is created via the payload dropdown - for example:

Loading data from a cache

Cached data can be loaded via our load from cache shape. Please refer to the Load from cache shape section for more information.