Loading multiple items from dynamic cache keys

Introduction

This approach assumes that the cache to be loaded was added with a payload variable for the cache key, and is comprised of multiple, single-record payloads (having been through a flow control shape).

Each of these payloads has its own, unique cache key (when data was added to the cache, this key was generated dynamically by resolving a cache key payload variable).

For more information about this stage, please see Generating dynamic cache keys with payload variables.

When we come to load this data, we must target the required cache keys. In the same way that we use a payload variable to add data to a cache with dynamic cache keys, we can use a payload variable to load data from these keys.

To do this, you configure a load from cache shape with a 'multi-pick' payload variable in the cache key, and ensure that data passed into this shape contains the values required to resolve this variable.

How it works

In summary, you can drop a single load from cache shape into a process flow and specify a payload variable as the required cache key. This must be in the form:

[[payload.*.<element>]]

...where <element> should be replaced with whichever data element you will be passing in to to resolve the cache key. For example:

[[payload.*.id]]

The <element> defined here will be the same data element that was specified in the payload variable for the corresponding add to cache shape.

You then need to pass in any <element> values that should be used to resolve required cache key names. This might be achieved via a connection shape (if values are being generated from another system), or perhaps a manual payload shape. Whichever shape you use must be placed immediately before the load from cache shape.

Example

To help understand how this approach works, we will step through an example.

Suppose we have the scenario where a process flow has been built to receives incoming orders, and another process flow needs to target specific orders received from this flow.

Process flow 1: Add to cache To allow the second process flow access to orders processed by the first, we must add all incoming orders to a company type cache in the first process flow (remember that company type caches can be accessed by any other process flow created for your company profile). To ensure that we can go on to target specific orders from this cache later, we will cache every order in its own cache key, using a payload variable.

Process flow 2: Load from cache To retrieve specific orders from the cache created in the first process flow, we will pass the required order ids into a load from cache shape. These ids will be used to resolve dynamic cache keys, using a payload variable.

Process flow 1: Add to cache

Here, we will batch an 'orders' payload into single order payloads - then we'll add each payload to its own cache key, which is created dynamically from a payload variable. Let's break these steps down:

Step 1: Manual payload

The manual payload shape contains an 'orders' payload with 17 orders in total.

Step 2: Filter

The filter shape ensures that orders are only processed if the id field is not empty.

Step 3: Flow control

The flow control shape is set to create batches of 1 from the payload root level - so every order will be added to its own payload.

Step 4: Add to cache

The add to cache shape is defined to add to a company type cache, named CPT-722. The cache key is created dynamically, where the first part is always order-followed by the value of the first id element found in the incoming payload. E.g. order-5697116045650. All data from the incoming payload will be added to this cache key. Taking our example using flow control, the incoming payload will only ever be a single order.

Step 5: Run flow

When this process flow runs, checking payload information for the add to cache shape shows that 17 payloads have been cached - one payload for each order.

Process flow 2: Load from cache

Here, we will pass the required order ids into a load from cache shape. These ids are then used to resolve dynamic cache keys (via a payload variable) to determine which orders should be loaded. Let's break these steps down:

Step 1: Manual payload

The manual payload shape contains two order ids that we want to load from our cache.

Step 2: Load from cache

The load from cache shape is configured to load data from our CPT-722 cache, targeting dynamic cache keys from order-[[payload.*.id]]. Here, the required cache key(s) will be resolved from all (*) ids found in the incoming payload - in this case order-5693105439058 and order-5697116045650.

Step 3: Run flow

When this process flow runs, checking payload information for the load from cache shape shows that two payloads have been loaded - one for each of our given ids.