Track data shape

Introduction

The track data shape is used to track processed data, based on field paths that you define. When data passes through a track data shape, the values associated with your defined field paths are tracked - which means they can be reviewed from the tracked data page.

For example, you might want to track all customer_id values that pass through a flow, so at any time you can quickly check if/when/how a given customer record has been processed.

A single incoming payload for any process flow shape should not exceed 500MB.

We recommend processing multiple, smaller payloads rather than one single payload (1000 x 0.5MB payloads are more efficient than 1 x 500MB payload!).

For payloads up to 500MB, consider adding a flow control shape to batch data into multiple, smaller payloads. Payloads exceeding 500MB should be batched at source.

Need to know

Adding & configuring a track data shape

You can add as many track data shapes to a process flow as required. For example, you might place one immediately after a receiving connector to track everything received before anything else happens to the data, and another after the final sending connector to track everything sent into your destination system.

To add and configure a new track data shape, follow the steps below.

Step 1 In your process flow, add the track data shape in the usual way:

Step 2 Configure settings as required - the table below summarises available fields:

Field
Summary

Source instance Source endpoint

If data is coming into the process flow via a connector shape, use these dropdown fields to select appropriate source connector details (i.e. the same instance and endpoint as configured for the previous connector shape). If data is coming into the flow via a non-connector source (such as a manual payload, API request, or webhook) then leave these fields blank.

Entity

If data is coming into the flow via a connector shape, this field will be set as required by default. Otherwise, select the entity type associated with the data field(s) that you want to track.

Note that the selection made here has no impact on how the shape performs - it simply determines how the tracked field is categorised in tracked data summaries.

Direction

If data is received via a connector shape, this field will be set as required by default. Otherwise, select the flow direction (send or receive) associated with the data field(s) that you want to track:

  • If the tracked data is being pulled from a source, set this option to receive

  • If the tracked data is being pushed to a destination, set this option to send

Note that the selection made here has no impact on how the shape performs - it simply determines how the tracked field is categorised in tracked data summaries.

Field paths

Define one or more data fields to be tracked - i.e. fields that you may want to look up in the event of a query.

If multiple fields are specified, these values are tracked as one, concatenated value. To track multiple fields separately, use one shape per field.

If data is received via a connector shape, you can navigate the associated data structure to select a field for tracking - for example: If data is received via a non-connector source (such as a manual payload, API request, or webhook), enter a path to the required field manually. If you need to track meta or flow variables, toggle the use custom value option to on and enter the correct variable syntax. For more information on this option, please see the custom values section below.

Step 3 Save your settings, then access them again:

...you'll now see success criteria options at the bottom of the shape settings drawer:

Step 4 The success criteria options are optional. If you don't need to apply these, then your setup is complete - close the settings drawer and continue with your process flow as required. If you do want to use these options, see below for guidance.

Success criteria options

When data passes through a track data shape, specified data fields are tracked, and by default, tracked data is marked as a success.

However, there may be times when you want to control the conditions under which the status of tracked data is deemed a success or a failure, and to record this outcome for future reference. The success criteria section allows you to:

  • Define filter conditions that must be met for an entity's progress to be reported as a success or failure in summary information for this tracked item.

  • Add a message to be displayed in summary information for this tracked item.

When tracked data is marked as failed, it is still tracked and the shape still processes successfully - for example:

In this run log, notice that tracked data is marked as a failure (1) but tracked data is stored (2), and the track data step succeeds (3).

In this context, tracked data marked as a failure simply means that one or more filters defined for success were not matched, and therefore this item is reported as a failure in associated tracked data summaries.

Success criteria filters

Any conditions that you want to apply can be added via filters. To define a new filter, click the add filter button:

These filters work in the same way as other filters in the dashboard - select/define a field, then set conditions and values.

You can add as many filters as you need - multiple filters work together with an 'AND' operator. Remember that you're defining conditions that must be met for a success outcome - if multiple filters are present, they must ALL be matched. If one or more filters are not matched, the associated tracked data is marked as a failure.

Filters can be based on any field(s) found in your data, irrespective of whether you chose to track them.

The success or failure outcome from these filters is reported in the logs, and also in tracked data summaries - for example:

Success criteria message

You can define a message to be displayed in the tracked data summary for associated tracked data:

This message can be text-only, or any combination of text, payload variables, flow variables, and metadata variables. For example:

In tracked data summaries, this example is shown as:

Messages are added to tracked data summaries when:

  • No success criteria filters are defined

  • Success criteria filters are defined, and the outcome is success

  • Success criteria filters are defined, and the outcome is failure

Custom values

When defining a field path to be tracked, a use custom value toggle option is available:

This option can be used for cases where you want to specify a meta or flow variable value to be tracked, but you don't want that value to be taken from incoming payloads.

For example, suppose your process flow includes a set variables shape, setting a meta-variable as follows for incoming payloads:

name : filename value : AW25.

Now, suppose you want to track this meta-variable value. You'd be forgiven for entering meta-variable syntax ( [[meta.filename]]) in the field path, as below:

However, this would be processed as follows:

  1. Resolve [[meta.filename]]

  2. Set the resolved value (AW25) as the field path key to be tracked

  3. Look for a key named AW25 in the incoming payload

  4. Fail to find a key named AW25 in the incoming payload, so nothing is tracked

The correct approach is to toggle the use custom value toggle option to on and enter meta-variable syntax ( [[meta.filename]]) in the custom value, as below:

This will be processed as follows:

  1. Resolve [[meta.filename]]

  2. Track the resolved value (AW25)

Related pages

PreviousSplit shapeNextThe tracked data page

Last updated