Importing & exporting de-dupe data

Introduction

If required, you can import existing data into a de-dupe pool. For example, you may have records that you know have been processed elsewhere and want to ensure that they aren't processed via Patchworks.

Conversely, you can export de-dupe pool data to a CSV file, for use outside of Patchworks.

Need to know

Export file format

De-dupe data exports are completed in CSV format, delimited ONLY with a single comma between fields. The exported file includes four columns with the following headers:

Header	Data Type	Summary
value	varies - review data	The tracked data value
entity_type_id	number	The Patchworks id associated with the entity type being tracked.
created_by	string	The name of the process flow that updated the data pool with this value.
flow_run_id	int	The unique id associated with the flow run that added this value to the data pool.
created_at	string	The date and time that the record was added to the pool (UTC time).

For example:

value,entity_type_id,created_by,flow_run_id,created_at
6980276158744,1,"Demo Data Pools",01jkgd4e7mwxprh7x09fk29421,"2025-02-07 14:31:12"
6980276191512,1,"Demo Data Pools",01jkgd4e7mwxprh7x09fk29421,"2025-02-07 14:31:11"
6980276224280,1,"Demo Data Pools",01jkgd4e7mwxprh7x09fk29421,"2025-02-07 14:31:11"
7063019880728,1,"Demo Data Pools",01jkgd4e7mwxprh7x09fk29421,"2025-02-07 14:31:11"
7105804927256,1,"Demo Data Pools",01jkgd4e7mwxprh7x09fk29421,"2025-02-07 14:31:11"
7105808859416,1,"Demo Data Pools",01jkgd4e7mwxprh7x09fk29421,"2025-02-07 14:31:11"
7105822490904,1,"Demo Data Pools",01jkgd4e7mwxprh7x09fk29421,"2025-02-07 14:31:11"
7208446951704,1,"Demo Data Pools",01jkgd4e7mwxprh7x09fk29421,"2025-02-07 14:31:11"

Imports

Import approach

When de-dupe data values are imported:

• All records in the import file are added to the data pool as new items

• Any existing items in the data pool are unchecked and unchanged

Import file format

To import de-dupe values, the import file must be in csv format and include value and entity_type_id headers. For example:

value,entity_type_id
0000276158744,1

Where:

• The value is the key field value that you are matching on

• The entity_type_id is the internal Patchworks id for the entity type associated with the key field that you are using to match duplicates. This id must be present for every entry in your CSV file. You can download a list of ids by following steps detailed later in this page.

Import files cannot exceed 5MB.

Exporting a de-dupe data pool

To export/download a de-dupe data pool, follow the steps below.

Step 1 Log into the Patchworks dashboard, then select the settings option:

...followed by the file data pools option:

Step 2 Click the name of the data pool that you want to export:

Step 3 With the data pool in edit mode, move to the lower tracked de-dupe data panel and click the download button:

Step 4 The download job is added to a queue and a confirmation message is displayed:

Step 5 When your download is ready, you'll receive an email which includes a link to retrieve the file from the file downloads page. If you can't/don't want to use this link, you can access this page manually - click data pools in the breadcrumb trail at the top of the page:

...followed by the settings element option:

Step 6 Select the file downloads option from the settings page:

Step 7 On the file downloads page, you'll find any exports that have been completed for your company profile in the last hour. Click the download button for your job - the associated CSV file is saved to the default downloads folder for your browser.

This list may include exports from different parts of the dashboard, not just data pools (for example, run log and cross-reference lookup data exports are added here).

Step 8 Click the download button for your job - the associated CSV file is saved to the default downloads folder for your browser.

Download files are cleared after one hour. If you don't manage to download your file within this time, don't worry - just run the export again to create a new one.

Downloading the Patchworks entity id type list

If you want to import data into a de-dupe data pool, you need to ensure that each record in your CSV file includes an entity_type_id. To find which id you should use, follow the steps below to download a current list.

Step 1 Log into the Patchworks dashboard, then select the settings option:

...followed by the file data pools option:

Step 2 Click the download entity types button at the top of the page:

Step 3 A CSV file is saved to the default downloads folder for your browser.

Importing de-dupe data

To import data into a de-dupe data pool, follow the steps below.

Step 1 Log into the Patchworks dashboard, then select the settings option:

...followed by the file data pools option:

Step 2 If you want to import data into an existing data pool, click the name of the required data pool from the list:

Alternatively, you can create a new data pool.

Step 3 Move to the lower tracked de-dupe data panel and click the import button:

Step 4 Navigate to the CSV file that you want to import and select it:

Step 5 The file is uploaded and displayed as a button - click this button to complete the import:

Step 6 The import is completed - existing values are updated and new values are added:

You may need to refresh the page to view the updated data pool.