Generating dynamic cache keys with variables

Introduction

When an add to cache shape is dropped into a process flow, the entire incoming payload is cached and associated with the given cache key. Depending on the cache type, you can load this cache later in the same flow or in a different flow.

In the simplest scenario, your given cache key would be a static value (e.g. customers) and you would use this to load the entire cache (containing perhaps tens, hundreds, even thousands of items) where required. But what if you want to load a specific item from a cache, rather than the whole thing?

This is where dynamic cache keys are so useful.

How it works

To load data from a cache, you configure a load from cache shape with the required cache and a single cache key. All data associated with your given cache key is loaded.

Consider the example incoming payload below, where four records are cached with a static cache key with a value of customers:

cache key: customers

[
    {
        "id": 1000000001,
        "first_name": "Jane",
        "last_name": "Smith",
        "items": {
           "itemref": "0000001",
            "item1": "apples",
            "item2": "oranges",
            "item3": "pears"
        }
    },
    {
        "id": 1000000002,
        "first_name": "George",
        "last_name": "Jones",
        "items": {
           "itemref": "0000002",
            "item1": "tangerines",
            "item2": "peaches",
            "item3": "grapes"
        }
    },
    {
        "id": 1000000003,
        "first_name": "Bob",
        "last_name": "Brown",
        "items": {
           "itemref": "0000003",
            "item1": "nectarines",
            "item2": "raspberries",
            "item3": "strawberries"
        }
    },
    {
        "id": 1000000004,
        "first_name": "Marjorie",
        "last_name": "Simpson",
        "items": {
           "itemref": "0000004",
            "item1": "blueberries",
            "item2": "cranberries",
            "item3": "apricots"
        }
    }
]

If we were to configure a load from cache shape to access the customers cache key, all four records would be loaded.

So, in order to load specific items from a cache, the incoming data must be added to a cache in such a way that we can easily target individual items. We need an efficient way to take incoming data, batch it into single-record payloads and add each of these to the cache with its own unique, identifying cache key - i.e.:

cache: customerData cache key: customer-1000000001

[
    {
        "id": 1000000001,
        "first_name": "Jane",
        "last_name": "Smith",
        "items": {
           "itemref": "0000001",
            "item1": "apples",
            "item2": "oranges",
            "item3": "pears"
        }
    }
]

cache: customerData cache key: customer-1000000002

[
    {
        "id": 1000000002,
        "first_name": "George",
        "last_name": "Jones",
        "items": {
           "itemref": "0000002",
            "item1": "tangerines",
            "item2": "peaches",
            "item3": "grapes"
        }
    }
]

cache: customerData cache key: customer-1000000003

[
    {
        "id": 1000000003,
        "first_name": "Bob",
        "last_name": "Brown",
        "items": {
           "itemref": "0000003",
            "item1": "nectarines",
            "item2": "raspberries",
            "item3": "strawberries"
        }
    }
]

cache: customerData cache key: customer-1000000004

[
    {
        "id": 1000000004,
        "first_name": "Marjorie",
        "last_name": "Simpson",
        "items": {
           "itemref": "0000004",
            "item1": "blueberries",
            "item2": "cranberries",
            "item3": "apricots"
        }
    }
]

We can achieve this as follows:

Action	Outcome
Place a flow control shape immediately before the add to cache shape and configure it to create batches of 1 at the appropriate level for your data.	The incoming payload is batched into multiple payloads - one payload per data element (e.g. one order per payload, one customer per payload, one product per payload, etc.).
Configure the add to cache shape and specify a payload variable as the cache key, where the variable looks for the first occurrence of a uniquely identifying element in the payload (typically an id or reference number).	The add to cache shape receives and caches single-record payloads from the flow control shape. The cache key for each payload is generated dynamically by resolving the payload variable from each incoming payload.

Flow control is an easy way to batch incoming data into single-record payloads, however you may prefer an alternative approach. The important point is that the add to cache shape must receive single-record payloads - how you achieve this is up to you.

Need to know

• When you specify a dynamic variable as the cache key, the value for that variable is injected into the key. To prevent the case where large amounts of data are passed into the key, there is a character limit is 128 characters.

• Any combination of payload, flow and metadata variables can be used to form cache key names.

• When using dynamic variables for auto-generated cache keys, resolved variable values must not contain full stop (.) or colon (:) characters.

Using a variable to generate dynamic cache keys

Follow the steps below to configure an add to cache shape with a payload variable for generating dynamic cache keys.

These steps assume that you have already defined a flow control shape (or some other means) to ensure that the add to cache shape receives single-record payloads.

Step 1 Drop an add to cache shape into your process flow, where required.

Example

In the example below, we have an incoming payload which includes orders in an orders array:

If we want to add these orders to a cache, we need to split them into batches of 1, so our flow control shape is configured as below:

This will generate two payloads. So, if we go on to drop an add to cache shape immediately after, each of these payloads will be cached (and therefore retrieved) separately.

Step 2 In the add to cache shape settings, choose to create cache:

Step 3 Set the cache level and name as required and save changes.

For more information on these fields please see the add to cache shape page.

Step 4 Select the cache that you just created - for example:

Step 5 Move down to the cache key field and enter the required key. Here, you use standard payload variable syntax to define your target data element:

[[payload.<schema notation>]]

...where schema notation should be replaced with the notation path to the first occurrence of the required element in the payload which should be used to form the cache key. If required, you can also include a static prefix or suffix. For example:

customer-[[payload.0.id>]]

The output of the payload variable will be used as the cache key.

Example

Consider the payload below:

{
  "orders": [
    {
      "id": 1,
      "customer": {
        "email": "[email protected]"
      }
    }
  ]
}

...and our cache key is specified as:

customer-[[payload.0.id>]]

The generated cache key would be:

customer-1

...where 1 is the id associated with the first object in the orders array.The

Our example uses dynamic payload variables however, you can also use metadata variables and/or flow variables. For more information please see dynamics variables section.

When using dynamic variables for auto-generated cache keys, ensure that resolved variable values do not contain full stop (.) or colon (:) characters. For example:

[[payload.id]] resolving to 1234567 is good [[payload.id]] resolving to 123.4567 is bad

Step 6 Save the add to cache shape settings.

Loading data from a cache

Cached data can be loaded via our load from cache shape. Please refer to the Load from cache shape section for more information.