Adding & configuring a de-dupe shape
Introduction
The de-dupe shape is used to identify and then remove duplicate entries from an incoming payload. For more background information please see our De-dupe shape page.
Need to know
Tracked de-dupe data is retained for 90 days after it's added to a data pool.
Adding a de-dupe shape
To add and configure a new de-dupe shape, follow the steps below.
Step 1 In your process flow, add the de-dupe shape in the usual way:
Step 2 Select a source integration and endpoint to determine where the incoming payload to be de-duped originates - for example:
If your incoming data is via manual payload, API request, or webhook then you can remove any default source instance and endpoint selections:
Step 3 Move down to the behaviour field and select the required option.
Step 4 Move down to the data pool field and select the required data pool.
Step 5 In the key field, select/enter the data field to be used for matching duplicate records. How you do this depends on how the incoming data is being received - please see the options below:
I want to choose a field from the schema associated with a connector endpoint
If the incoming payload for the de-dupe shape is received from a connection shape, you'll find that the de-dupe shape settings default to the same connection instance and endpoint. In this case, the key field allows you to navigate the schema that's associated with the endpoint, and select the required data item:
I want to specify a field manually
If the incoming payload for the de-dupe shape is received via manual payload, API request, or webhook, there is no associated instance/endpoint and therefore no known data schema. In this case, you should enter the required key field value manually - enter the dot notation path to the required field in your data - for example: *.customerID:
I want to use variables to define a dynamic key field
If the incoming payload for the de-dupe shape is received via manual payload, API request, or webhook, you can generate the key field value dynamically using payload, flow and metadata variables.
Any combination of payload, flow and metadata variables can be used to form cache key names. For more information please see our Dynamic variables section.
Step 5 Select the payload format:
Step 6 Save the shape.
Last updated