Adding & configuring a de-dupe shape

Introduction

The de-dupe shape is used to identify and then remove duplicate entries from an incoming payload. For more background information please see our De-dupe shape page.

Need to know

Tracked de-dupe data is retained for 90 days after it's added to a data pool.

Currently, the de-dupe shape supports JSON payloads only.

Adding a de-dupe shape

To add and configure a new de-dupe shape, follow the steps below.

Step 1 In your process flow, add the de-dupe shape in the usual way:

Step 2 Select a source integration and endpoint to determine where the incoming payload to be de-duped originates - for example:

If your incoming data is via manual payload, inbound API or webhook then you can remove any default source instance and endpoint selections:

Step 3 Move down to the behaviour field and select the required option.

For more information about these options please see our De-dupe shape behaviour section.

Step 4 Move down to the data pool field and select the required data pool.

If necessary, you can create a data pool 'on the fly' using the create data pool option. For more information please see Adding a new data pool via the de-dupe shape.

Step 5 In the key field, select/enter the data field to be used for matching duplicate records. How you do this depends on how the incoming data is being received - please see the options below:

I want to choose a field from the schema associated with a connector endpoint

If the incoming payload for the de-dupe shape is received from a connection shape, you'll find that the de-dupe shape settings default to the same connection instance and endpoint. In this case, the key field allows you you to navigate the schema that's associated with the endpoint, and select the required data item:

I want to specify a field manually

If the incoming payload for the de-dupe shape is received from a manual payload, inbound API or webhook, there is no associated instance/endpoint and therefore no known data schema. In this case, you should enter the required key field value manually - enter the dot notation path to the required field in your data - for example: *.customerID:

I want to use variables to define a dynamic key field

If the incoming payload for the de-dupe shape is received from a manual payload, inbound API or webhook, you can generate the key field value dynamically based on given payload, flow, or metadata variables.

Any combination of payload, flow and metadata variables can be used to form cache key names. For more information please see our Dynamic variables section.

The selection that you make here determines how the payload is adjusted when duplicate data is removed. For more information please see How duplicate data is handled.

Step 5 Select the payload format:

Step 6 Save the shape.