> For the complete documentation index, see [llms.txt](https://doc.wearepatchworks.com/product-documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/advanced-shapes/de-dupe-shape/adding-and-configuring-a-de-dupe-shape.md).

# Adding & configuring a de-dupe shape

## Introduction

The **de-dupe** shape is used to identify and then remove duplicate entries from an incoming payload. For more background information please see our [De-dupe shape](/product-documentation/process-flows/building-process-flows/process-flow-shapes/advanced-shapes/de-dupe-shape.md) page.

## Need to know

{% hint style="danger" %}
Tracked de-dupe data is retained for 90 days after it's added to a data pool.
{% endhint %}

{% hint style="info" %}
Currently, the de-dupe shape supports JSON payloads.
{% endhint %}

## Adding a de-dupe shape

To add and configure a new **de-dupe** shape, follow the steps below.

**Step 1**\
In your process flow, add the **de-dupe** shape in the usual way:

<div align="left"><figure><img src="/files/h8FINyX8ND7u8oRNOctJ" alt="" width="375"><figcaption></figcaption></figure></div>

**Step 2**\
Select a **source integration** and **endpoint** to determine where the incoming payload to be de-duped originates - for example:

<div align="left"><figure><img src="/files/sE3aUoqVDOgJf8RSD05H" alt="" width="352"><figcaption></figcaption></figure></div>

{% hint style="info" %}
If your incoming data is via [manual payload](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/manual-payload-shape.md), [API request](broken://pages/EVn1AesQJtjyoIw7p0HZ), or [webhook](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/trigger-shape/trigger-shape-webhook.md) then you can remove any default source instance and endpoint selections:

<img src="/files/vR8caSjQJ2vjeQOlrHep" alt="" data-size="original">
{% endhint %}

**Step 3**\
Move down to the behaviour field and select the required option.

{% hint style="info" %}
For more information about these options please see our [De-dupe shape behaviour](/product-documentation/process-flows/building-process-flows/process-flow-shapes/advanced-shapes/de-dupe-shape.md#behaviour) section.
{% endhint %}

**Step 4**\
Move down to the **data pool** field and select the required [data pool](/product-documentation/process-flows/building-process-flows/process-flow-shapes/advanced-shapes/de-dupe-shape.md#data-pools).

<div align="left"><figure><img src="/files/X9pt9b3haIS0b7seVmvc" alt="" width="357"><figcaption></figcaption></figure></div>

{% hint style="info" %}
If necessary, you can create a data pool 'on the fly' using the **create data pool** option. For more information please see [Adding a new data pool via the de-dupe shape](/product-documentation/process-flows/building-process-flows/process-flow-shapes/advanced-shapes/de-dupe-shape/working-with-data-pools.md#adding-a-new-data-pool-via-the-de-dupe-shape).
{% endhint %}

**Step 5**\
In the **key field**, select/enter the data field to be used for matching duplicate records. How you do this depends on how the incoming data is being received - please see the options below:

<details>

<summary><img src="/files/mjTTZzyEQN7pfYDU3Ixl" alt="" data-size="line"> I want to choose a field from the schema associated with a connector endpoint</summary>

If the incoming payload for the de-dupe shape is received from a connection shape, you'll find that the de-dupe shape settings default to the same connection instance and endpoint. In this case, the `key field` allows you to navigate the schema that's associated with the endpoint, and select the required data item:

<img src="/files/Q3xGWbAwv8qtXGPwq6nY" alt="" data-size="original">

</details>

<details>

<summary><img src="/files/mjTTZzyEQN7pfYDU3Ixl" alt="" data-size="line"> I want to specify a field manually</summary>

If the incoming payload for the de-dupe shape is received via [manual payload](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/manual-payload-shape.md), [API request](broken://pages/EVn1AesQJtjyoIw7p0HZ), or [webhook](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/trigger-shape/trigger-shape-webhook.md), there is no associated instance/endpoint and therefore no known data schema. In this case, you should enter the required `key field` value manually - enter the dot notation path to the required field in your data - for example: `*.customerID`:

<img src="/files/Ltq4QJfI1n6Wf7WsuRvw" alt="" data-size="original">

</details>

<details>

<summary><img src="/files/mjTTZzyEQN7pfYDU3Ixl" alt="" data-size="line"> I want to use variables to define a dynamic key field</summary>

If the incoming payload for the de-dupe shape is received via [manual payload](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/manual-payload-shape.md), [API request](broken://pages/EVn1AesQJtjyoIw7p0HZ), or [webhook](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/trigger-shape/trigger-shape-webhook.md), you can generate the key field value dynamically using [payload](/product-documentation/process-flows/building-process-flows/dynamic-variables/payload-variables.md), [flow](/product-documentation/process-flows/building-process-flows/dynamic-variables/flow-variables.md) and [metadata](/product-documentation/process-flows/building-process-flows/dynamic-variables/metadata-variables.md) variables.

<img src="/files/syNjVGNiHPyQI5H59IMO" alt="" data-size="original">

Any combination of [payload](/product-documentation/process-flows/building-process-flows/dynamic-variables/payload-variables.md), [flow](/product-documentation/process-flows/building-process-flows/dynamic-variables/flow-variables.md) and [metadata](/product-documentation/process-flows/building-process-flows/dynamic-variables/metadata-variables.md) variables can be used to form cache key names. For more information please see our [Dynamic variables](/product-documentation/process-flows/building-process-flows/dynamic-variables.md) section.

</details>

{% hint style="info" %}
The selection that you make here determines how the payload is adjusted when duplicate data is removed. For more information please see [How duplicate data is handled](/product-documentation/process-flows/building-process-flows/process-flow-shapes/advanced-shapes/de-dupe-shape.md#how-duplicate-data-is-handled).
{% endhint %}

**Step 5**\
Select the payload format:

<div align="left"><figure><img src="/files/gzHXAtQVgIPkxk8UoolM" alt="" width="359"><figcaption></figcaption></figure></div>

**Step 6**\
Save the shape.