Re-syncing a previous payload

Introduction

If a process flow fails to complete successfully, you'll see the run displayed in your logs with a failure status - if appropriate, you can use the retry option to run it again.

However, there may be occasions when some time has passed since a flow failed, therefore a retry would not necessarily pick up the same data as the original run. Or, you want to re-sync a specific item in a payload.

This guide walks through the steps required to retrieve the original payload from a failed run and pass this into the process flow to run again.

Most endpoints have a variable or parameter used to target a specific entity (for example, order_id, customer_id, product_id, etc.). When building a process flow, we recommend building an alternative 'flavour' that allows users to specify a unique identifier to target at runtime - this is the most efficient way to sync/re-sync specific items. For more information please see our Targeted syncs - best practice page.

However, if you don't have a process flow in place for targeted syncs, the steps detailed here can be followed.

The steps

This task is documented in four stages:

1. Find the payload to be synced We need to find the payload from the failed process flow that needs to be passed in again, manually.

2. Skip shapes Since we're passing in the required data manually, we must prevent (temporarily) all shapes up to and including the point the payload was taken from running again.

3. Initialise the flow manually with the payload Run the flow manually, passing in the payload found in stage 1.

4. Reinstate skipped shapes Remove the 'skip' option from shapes so the process flow runs normally next time.

This task requires us to (temporarily) prevent some shapes in your process flow from running. As such, please ensure that you complete this task at a time when the process flow is not scheduled to run automatically.

Stage 1: Find the payload

First, we locate and copy the initial payload so it can be passed in again, manually.

The 'initial' payload is the payload you want to pass into the flow to be synced. In the simplest case, this is a payload generated from a first connector step that receives data, but there may be times when you need to take a payload from further down the flow - for example, you might want to process a single-item payload generated from a flow control shape.

Any payloads associated with a process flow run can be accessed retrospectively, via your run logs. To do this, follow the steps below.

Step 1 Select run logs from the left-hand navigation menu:

Step 2 At the top of the run logs table, click the filter icon associated with the status column:

...and select the failure option (so only failed runs are displayed):

Here we assume that you are re-syncing data because a run has failed. If this isn't the case, you can filter by any status to find the required run - however, beware of duplicating data if you re-use data from a run set to anything other than a failure status.

Step 3 Click the ellipses associated with the run that you want to access, then select the view logs option:

Step 4 Logs are displayed with each step in the flow listed in the lower panel - for example:

Our example shows that our flow failed at the assert step, with only a connector step before. So in this case, we need to obtain the entire payload from the previous connector step.

You can retrieve the payload from whichever step in your process flow is most appropriate. In our example, we want to re-sync ALL original data from the connector shape but if - for example - you are looking for a specific record and your process flow includes flow control, this is most likely to be the place to find it.

Step 5 Click the 'eye' icon associated with the shape that you want to view.

...and look for payloads in the lower panel:

These are payloads that were available at the end of the step - you might have a single payload or (for example, if your data is paginated or has been through flow control) you might have multiple payloads as per this example.

Step 6 Click the 'eye' icon associated with the first payload:

...and the payload is displayed:

Step 7 Check the payload to ensure that it contains the right data, then scroll to the end and look to see if the displayed data is truncated - if it ends with a [trimmed] marker, the full payload can't be displayed:

Step 8 What you do next depends on whether the displayed payload ends with a [trimmed] marker:

Payload is NOT trimmed

Click anywhere in the payload panel, then press CTRL+A followed by CTRL+C to copy the data to your clipboard:

Payload is trimmed

Click the download button to download the full payload in a text file - then open this file and press CTRL+A followed by CTRL+C to copy the data to your clipboard:

Stage 2: Skip shapes

Having obtained the required data, we must ensure that all shapes up to including the step associated with your payload are NOT processed the next time the process flow runs.

To do this, access settings for each shape and toggle the skip option on:

With the exception of the trigger shape (which can't and doesn't need to be skipped), you must skip ALL shapes up to and including the shape where you copied the required payload.

For more information about the skip shape option please see our Skipping shapes page.

Stage 3: Initialise the flow manually with payload

Now we're ready to run the process flow manually, passing in the payload copied in stage 1.

Step 1 Select initialise flow with data from the actions bar:

...a payload window is displayed:

Step 2 Paste in the payload that you copied in stage 1, then click the initialise flow button:

Step 3 Ensure that the process flow runs without errors, and check that data has synced as expected.

Stage 4: Reinstate skipped shapes

Having ensured that data has synced successfully, we can reset skipped shapes so the process flow runs normally next time.

To do this, access settings for each shape that you skipped in stage 2 and toggle the skip option back to off: