Track data shape
Last updated
Last updated
The track data shape is used to track processed data, based on field paths that you define. When data passes through a track data shape, the values associated with your defined field paths are tracked - which means they can be reviewed from the tracked data page.
For example, you might want to track all customer_id values that pass through a flow, so at any time you can quickly check if/when/how a given customer record has been processed.
Tracked data shape is available for viewing for 15 days after it was last tracked.
Depending on data volumes, allow 10 minutes for tracked data to be visible in data pools.
The track data shape works with incoming payloads from a connection shape, a manual payload, an API request, or a webhook.
JSON payloads are supported.
You can add as many track data shapes to a process flow as required. For example, you might place one immediately after a receiving connector to track everything received before anything else happens to the data, and another after the final sending connector to track everything sent into your destination system.
To add and configure a new track shape, follow the steps below.
Step 1 In your process flow, add the track data shape in the usual way:
Step 2 Configure the settings as required - the table below summarises available fields:
Source instance Source endpoint
Entity
Direction
If the tracked data is being pulled from a source, set this option to receive
If the tracked data is being pushed to a destination, set this option to send
Field paths
Define one or more data fields to be tracked - i.e. fields that you may want to look up in the event of a query.
Step 3 Save your settings., then access them again:
...you'll now see success criteria options at the bottom of the shape settings drawer:
Step 4 The success criteria options are optional. If you don't need to apply these then your setup is complete - close the settings drawer and continue with your process flow as required. If you do want to use these options, see below for guidance.
When data passes through a track data shape, specified data fields are tracked and by default, tracked data is marked as a success.
However, there may be times where you want to control the conditions under which the status of tracked data is deemed a success or a failure, and to record this outcome for future reference. The success criteria section allows you to:
Define filter conditions that must be met for an entity's progress to be reported as a success or failure in summary information for this tracked item.
Add a message to be displayed in summary information for this tracked item.
When tracked data is marked as failed, it is still tracked and the shape still processes successfully - for example:
In this run log, notice that tracked data is marked as a failure (1) but tracked data is stored (2) and the track data step succeeds (3).
In this context, tracked data marked as a 'failure' simply means that one or more filters defined for success were not matched and therefore this item is reported as a failure in associated tracked data summaries.
Any conditions that you want to apply can be added via filters. To define a new filter, click the add filter button:
These filters work in the same way as other filters in the dashboard - select/define a field, then set conditions and values.
You can add as many filters as you need - multiple filters work together with an 'AND' operator. Remember that you're defining conditions that must be met for a success outcome - if multiple filters are present they must ALL be matched. If one or more filters are not matched, the associated tracked data is marked as a failure.
Filters can be based on any field(s) found in your data, irrespective of whether you chosen to track them.
The success or failure outcome from these filters is reported in the logs, and also in tracked data summaries - for example:
You can define a message to be displayed in the tracked data summary for associated tracked data:
This message can be text-only, or any combination of text, payload variables, flow variables, and metadata variables. For example:
In tracked data summaries, this example is shown as:
Messages are added to tracked data summaries when:
No success criteria filters are defined
Success criteria filters are defined and the outcome is success
Success criteria filters are defined and the outcome is failure
If data is coming into the process flow via a , use these dropdown fields to select appropriate source connector details (i.e. the same instance and endpoint as configured for the previous connector shape). If data is coming into the flow via a non-connector source (such as a , , or ) then leave these fields blank.
If data is coming into the flow via a , this field will be set as required by default. Otherwise, select the entity type associated with the data field(s) that you want to track.
Note that the selection made here has no impact on how the shape performs - it simply determines how the tracked field is categorised in .
If data is received via a , this field will be set as required by default. Otherwise, select the flow direction (send or receive) associated with the data field(s) that you want to track:
Note that the selection made here has no impact on how the shape performs - it simply determines how the tracked field is categorised in .
If multiple fields are specified, these values are tracked as one, concatenated value. To track multiple fields separately, use one shape per field.
If data is received via a , you can navigate the associated data structure to select a field for tracking - for example: If data is received via a non-connector source (such as a , , or ), enter a path to the required field manually - for example:
.svg?alt=media&token=ddeb7ae5-d7c7-46f5-baf3-55a3d5b1478f)
