# Track data shape
## Introduction
The `track data` shape is used to track processed data, based on field paths that you define. When data passes through a `track data` shape, the values associated with your defined field paths are tracked - which means they can be [reviewed from the tracked data page](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/track-data-shape/the-tracked-data-page).
For example, you might want to track all `customer_id` values that pass through a flow, so at any time you can quickly check if/when/how a given customer record has been processed.
{% hint style="info" %}
A single incoming payload for [any process flow shape](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes) should not exceed 500MB.
We recommend processing multiple, smaller payloads rather than one single payload (1000 x 0.5MB payloads are more efficient than 1 x 500MB payload!).
For payloads up to 500MB, consider adding a[ flow control shape](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/flow-control-shape) to batch data into multiple, smaller payloads. Payloads exceeding 500MB should be batched at source.
{% endhint %}
## Need to know
* By default, tracked data is [available for viewing](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/track-data-shape/the-tracked-data-page) for 15 days after it was last tracked. This can be increased up to 90 days; please [contact our Sales team](https://www.wearepatchworks.com/pages/contact-us) if you'd like to extend your retention period.
* Data to be tracked can be a maximum of 255 characters (tracked data is typically a specific payload field - for example, a customer ID).
* The track data shape works with incoming payloads from a [connection shape](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape), a [manual payload](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/manual-payload-shape), an [API request](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/broken-reference), or a [webhook](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/trigger-shape/trigger-shape-webhook).
* [Payload](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/manual-payload-shape), [flow](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-versioning), and [metadata](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/dynamic-variables/metadata-variables) variables are supported when defining success messages.
* JSON payloads are supported.
## Adding & configuring a track data shape
You can add as many `track data` shapes to a process flow as required. For example, you might place one immediately after a *receiving* connector to track everything received before anything else happens to the data, and another after the final *sending* connector to track everything sent into your destination system.
To add and configure a new `track data` shape, follow the steps below.
**Step 1**\
In your process flow, add the `track data` shape in the usual way:
**Step 2**\
Configure settings as required - the table below summarises available fields:
| Field | Summary |
| ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Source instance
Source endpoint
| If data is coming into the process flow via a [connector shape](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape), use these dropdown fields to select appropriate source connector details (i.e. the same instance and endpoint as configured for the previous connector shape). If data is coming into the flow via a non-connector source (such as a [manual payload](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/manual-payload-shape), [API request](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/broken-reference), or [webhook](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/trigger-shape/trigger-shape-webhook)) then leave these fields blank. |
| Entity | If data is coming into the flow via a connector shape, this field will be set as required by default. Otherwise, select the entity type associated with the data field(s) that you want to track.
Note that the selection made here has no impact on how the shape performs - it simply determines how the tracked field is categorised on the tracked data page.
|
| Direction | If data is received via a connector shape, this field will be set as required by default. Otherwise, select the flow direction (send or receive) associated with the data field(s) that you want to track:
- If the tracked data is being pulled from a source, set this option to
receive - If the tracked data is being pushed to a destination, set this option to
send
Note that the selection made here has no impact on how the shape performs - it simply determines how the tracked field is categorised on the tracked data page.
|
| Field paths | Define one or more data fields to be tracked - i.e. fields that you may want to look up in the event of a query.
If multiple fields are specified, these values are tracked as one, concatenated value. To track multiple fields separately, use one shape per field.
If data is received via a connector shape, you can navigate the associated data structure to select a field for tracking - for example:
If data is received via a non-connector source (such as a manual payload, API request, or webhook), enter a path to the required field manually.
If you need to track meta or flow variables, toggle the use custom value option to on and enter the correct variable syntax. For more information on this option, please see the custom values section below.
|
**Step 3**\
Save and then re-access settings for this shape. For example:
Now you'll w see `success criteria` options at the bottom of the shape settings drawer:
**Step 4**\
The `success criteria` options are optional. If you don't need to apply these, then your setup is complete - close the settings drawer and continue with your process flow as required. If you do want to use these options, see below for guidance.
### Success criteria options
When data passes through a `track data` shape, specified data fields are tracked, and by default, tracked data is marked as a `success`.
However, there may be times when you want to control the conditions under which the status of tracked data is deemed a `success` or a `failure`*,* and to record this outcome for future reference. The `success criteria` section allows you to:
* Define filter conditions that must be met for an entity's progress to be reported as a `success` or `failure` on the [tracked data page](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/track-data-shape/the-tracked-data-page).
* Add a message to be displayed on the [tracked data page](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/track-data-shape/the-tracked-data-page) for this tracked item.
{% hint style="info" %}
When tracked data is marked as `failed`, it is still tracked and the shape still processes successfully - for example:
In this run log, notice that the tracked data is marked as a `failure` (1) but tracked data is stored (2), and the track data step succeeds (3).
In this context, tracked data marked as a `failure` simply means that one or more filters defined for success were not matched, and therefore, this item is reported as a `failure`.
{% endhint %}
#### Success criteria filters
Any conditions that you want to apply can be added via filters. To define a new filter, click the `add filter` button:
These filters work the same as [other filters](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/filter-shape) in the dashboard - select/define a field, then set conditions and values.
You can add as many filters as you need - multiple filters work together with an 'AND' operator. Remember that you're defining conditions that must be met for a `success` outcome - if multiple filters are present, they must ALL be matched. If one or more filters are not matched, the associated tracked data is marked as a `failure`.
{% hint style="info" %}
Filters can be based on any field(s) found in your data, irrespective of whether you chose to track them.
{% endhint %}
The success or failure outcome from these filters is reported in the logs, and also on the [tracked data page](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/track-data-shape/the-tracked-data-page).
#### Success criteria message
You can define a message to be displayed on the [tracked data page](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/track-data-shape/the-tracked-data-page) for the associated tracked data item:
This message can be text-only, or any combination of text, [payload variables](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/dynamic-variables/payload-variables), [flow variables](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/dynamic-variables/flow-variables), and [metadata variables](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/dynamic-variables/metadata-variables). For example:
You will find these messages on the [tracked data page](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/track-data-shape/the-tracked-data-page). For example:
Messages are added to the [tracked data page](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/track-data-shape/the-tracked-data-page) in any of the following scenarios:
* No success criteria filters are defined
* Success criteria filters are defined, and the outcome is `success`
* Success criteria filters are defined, and the outcome is `failure`
### Custom values
When defining a `field path` to be tracked, a `use custom value` toggle option is available:
This option can be used for cases where you want to specify a meta or flow variable value to be tracked, but you don't want that value to be taken from incoming payloads.
For example, suppose your process flow includes a [set variables shape](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/set-variables-shape), setting a meta-variable as follows for incoming payloads:
`name` : `filename` \
`value` : `AW25`.
Now, suppose you want to track this meta-variable value. You'd be forgiven for entering meta-variable syntax ( `[[meta.filename]]`) in the `field path`, as below:
However, this would be processed as follows:
1. Resolve `[[meta.filename]]`
2. Set the resolved value (`AW25`) as the `field path` key to be tracked
3. Look for a key named `AW25` in the incoming payload
4. Fail to find a key named `AW25` in the incoming payload, so nothing is tracked
The correct approach is to toggle the `use custom value` toggle option to `on` and enter meta-variable syntax ( `[[meta.filename]]`) in the `custom value`, as below:
This will be processed as follows:
1. Resolve `[[meta.filename]]`
2. Track the resolved value (`AW25`)
## Related pages
* [The tracked data page](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/track-data-shape/the-tracked-data-page)
* [Field tagging & tracked fields](https://doc.wearepatchworks.com/product-documentation/developer-hub/connector-builder/building-your-own-connector/4-endpoints/endpoint-options/schema-taxonomy/field-tagging#what-are-tracked-fields)
