Introduction

The run logs page is used to access detailed logs and payloads for active and previous process flow runs and your run queue.

Accessing your run logs & queue page

To access the run logs & queue page, follow the steps below.

Step 1 Log into the Patchworks dashboard and select run logs from the left-hand navigation menu:

Step 2 Select the active tab for active and previous runs or the queue tab for pending runs:

Introduction

Once you start syncing data with process flows, we provide the tools to make sure that you're in the driver's seat when it comes to knowing that everything is running as expected and - in the event of any issues - understanding when/where a problem occurs so you can make informed decisions to resolve it.

Your toolkit includes:

• for immediate feedback as a process flow runs. Follow the progress of every step in a process flow with success/failure details, payloads, and request/response information for every shape.

• are retained for one month so you can access success/failure details, payloads, request and response information retrospectively.

• can be viewed so you can see any process flow runs that are pending.

• for failed runs so you're not constantly checking logs for assurance that your process flows are running as expected.

Introduction

Having accessed run logs and selected the view logs option for a given flow run, you can view:

• An insights summary with performance-related information for the process flow

• Detailed logs for each step in the flow

Insights summary

The insights summary is divided into three functional areas:

In the top, right-hand corner (1) you'll find additional options via an ellipses menu - these options will vary, depending on whether the run succeeded or failed. See Additional options for more information.

In the top panel (2), you can see basic details for the process flow (name, id, version and status, and how triggered) together with duration, CPU usage time, data used, and score.

The lower panel (3) shows every step (i.e. every shape) in the flow in processing sequence, with the following information:

Understanding scores

The score is a measure of how efficient/expensive (in terms of processing) your process flows are, based on the volume of data processed per second - 999 is the highest score.

Your score is based on all runs for all process flows associated with your company profile. This includes flows that are:

• Triggered by a schedule, webhook, or event

• Initialised by an API request

• Run manually

• Enabled or disabled when run

• Draft or deployed status when run

If your score is on the low side, it may be that your process flow is necessarily complex - including items such as scripts, transformations, flow control, caches, etc., has an impact on your score.

However, it's always worth checking a low-scored process flow further as there may be places where your process flows can be optimised (for example, does a flow include lots of mapping transformations that could be achieved in a single script?). A quick scan down the shapes list in the lower panel will show, at a glance, which shapes in your flow are taking longer to process or perhaps processing unusually high volumes of data.

If your score is low and you're satisfied that the process flow is built optimally, don't worry too much- scores are just there as a flag to indicate that it's worth checking the efficiency of your flow.

Detailed logs

In the lower panel of the insights summary, you can click any entry to access detailed logs:

In the top section, you can see a full history of what happened during this step - this will often display over multiple pages:

In the lower section, you can view and download any payloads associated with this step. Click the 'view' icon associated with any payload to see the contents:

...the payload is displayed:

Longer payloads are trimmed but you can use the download option to retrieve the full content:

Additional options

The ellipses icon in the top, right-hand corner provides access to additional options. These options vary, depending on whether the run failed or succeeded:

When a run is successful or partially successful, you can download run logs. If the run failed, you can download logs and also retry the run.

Introduction

Having and selected the view logs (classic) option for a given flow run, you can view detailed logs and payloads for that run:

Here, you can scroll down the page to view information for each step of the process flow.

Viewing & downloading payloads

Click the 'eye' icon associated with the run that you want to access:

The log entry expands and further options are available. Click the 'eye' icon to show the payload:

Longer payloads are trimmed but you can use the download option to retrieve the full content:

Introduction

Having accessed the run logs & queue page, the active tab displays active and previous process flow runs - these are listed chronologically based on the run start time:

At a glance, you can see the outcome of a run, the start date/time, the duration, how the run was triggered and whether there were any warnings.

Need to know

• Run logs are retained for 7 days.

• Payloads are retained for 72 hours.

• Payloads are stored in AWS S3 (eu-west-2 London).

• Payloads are encrypted during pull/push operations (HTTPS, TLS 1.2/1.3).

Searching run logs

If you're looking for a specific log, you can use the search field to search by flow run id or by process flow name:

Searching by flow run ID

If you want to find logs for a specific flow run id, type the full id into the search field - for example:

You must enter the id in full - partial elements are not matched.

A flow run id search is not case sensitive - entering 01j6errxr432r4k4dh41y5ttpc is the same as 01J6ERRXR432R4K4DH41Y5TTPC or 01J6eRRxR432r4K4DH41Y5ttPC.

Searching by flow name

If you want to find logs for a process flow that you only know by name, type any part of the name into the search field - for example:

You can enter as much or as little of the name as you like.

A flow name search is not case sensitive - entering CHECK is the same as check or Check.

The default view

Filters

Use filters to refine the list based on the value associated with a particular column. Currently, you can filter logs by:

• Started at date

• Trigger (schedule, manual, API, etc.)

• Has warnings

• Status

To define a filter, click the filter icon associated with a column - for example:

...then make selections from the options provided - for example:

Show stats

The show stats option can be toggled ON to view additional columns in the log table:

When this option is toggled ON, three more columns are shown - these show insights for the performance of the process flow.

Run status

The status of a run will be one of the following:

Warnings

If warnings are given during a run (be it failed or successful), the entry is shown with an icon in the Has warnings column - for example:

Checking the reason for a failed run

Run logs can contain a lot of information and there may be times when you want to quickly check why a run has failed.

When a process flow run is initialised but fails somewhere in the run, its status is set to failed and an expandable arrow is shown to the left of the entry - click this arrow to view a summary of why the failure occurred. For example:

Additional options

Each log entry has an ellipses icon, with access to additional options:

From here you can:

Related information

Introduction

If required, you can download run logs for a completed process flow run in CSV format, for use outside of Patchworks.

Need to know

Run log exports are completed in CSV format. The exported file includes two columns with level and message headers. For example:

level,message
info,"Flow initialised by Manual trigger."
info,"Flow step connector started"
info,"Connecting to Shopify"
error,"Connection to Shopify failed"
warning,"Errno: 60"
error,"Flow step connector failed: attempt (1/2)"
warning,"Error: 'SSL: no alternative certificate subject name matches target host name 'patchworks-docs-demo.myshopify.com.myshopify.com''"
info,"Flow step connector started"
info,"Connecting to Shopify"
error,"Connection to Shopify failed"
warning,"Errno: 60"
warning,"Error: 'SSL: no alternative certificate subject name matches target host name 'patchworks-docs-demo.myshopify.com.myshopify.com''"
error,"Flow step connector failed: attempt (2/2)"
error,"Flow run #01hsdkz31xzfe6jpvcpz7fgtf5 failed."
error,"Error: Internal error, please contact support if this issue persists."

The steps

To download run logs for a specific process flow run, follow the steps below.

Step 1 Log into the Patchworks dashboard and access your run logs & queue.

Step 2 Click the ellipses icon to access additional options for the log entry, then select the download logs option - for example:

Step 3 The download job is added to a queue and a confirmation message is displayed:

Step 4 When your download is ready, you'll receive an email which includes a link to retrieve the file from the file downloads page. If you can't/don't want to use this link, you can access this page manually by selecting the settings option:

...followed by the file downloads option:

Step 5 On the file downloads page, you'll find any exports that have been completed for your company profile in the last hour. Click the download button for your job:

Step 6 The associated CSV file is saved to the default downloads folder for your browser.

Introduction

When a process flow run is initialised automatically (via a trigger shape or a Patchworks API request), the run job is added to your queue.

Manual runs (initialised from the canvas actions bar) and manual retries are NOT added to your queue:

How it works

The time that run jobs remain in the queue depends on various factors which are summarised below:

Points to note:

• Every 05 seconds, the system checks if any process flows are running:

  • If NO, it checks the queue and picks more flows to run - the number of flows picked will be the maximum allowed to meet your concurrent process flows limit.

  • If YES, it checks if our concurrent process flows limit has been reached. If it has, no more flows are picked from the queue. If it has not, the queue is checked and more flows are picked to run. The number of flows picked will be the maximum allowed to meet our concurrent process flows limit.

• If a process flow is triggered manually, it bypasses the queue and is run as soon as possible. A manual run still consumes one flow from your concurrent process flows allowance - so while it's running, one less job can be picked from your queue.

• By default, run jobs are picked from the queue in the sequence they were added. However, if a queue priority has been set for a process flow, it takes precedence. For example, if your queue contains 6 run jobs with no queue priority and then a new job is added for a process flow with a queue priority set to high, this will be the next job to be picked (despite being the last to be added to the queue). Conversely, if a job is queued for a process flow with a queue priority set to lowest, this will be the last job to leave the queue.

Viewing your run queue

You can view all of your pending runs from the queue tab - entries are listed chronologically based on the time that the job was added to the queue:

At a glance, you can see:

• Process flow name (hyperlinked so you can quickly access the flow)

• Trigger

• Queued at time

Deleting a run job from your queue

To delete a run job from your queue (so it doesn't run), click the associated using

When prompted, confirm this action:

Related information

• Viewing logs

• Viewing logs (classic)

• Downloading run logs

• Retrying a failed process flow run

• Email notifications for failed process flow runs

Introduction

If a process flow run fails for any reason, it's displayed in run logs with a status of failed - for example:

You can retry the run from this page. Before doing so, please consider:

• It's advisable to check your process flow setup (filters and de-dupe shapes) to ensure that running the flow again will not duplicate any runs that may have succeeded in the interim period.

• If a process flow includes references to variables, be aware that any values for those variables may not be the same as they would have been if the failed flow had been successful.

• When the retry option is used, the flow version that was set originally is run. If you want to run an updated version of the process flow, you can choose to edit the flow and initialise a run manually instead.

• A process flow must be enabled for the retry option to work.

The steps

Step 1 Log into the Patchworks dashboard and access your run logs & queue.

Step 2 Click the ellipses icon to access additional options for the log entry, then select the retry flow run option - for example:

About retry timings

When the retry option is used for a failed process flow run, the flow run is dispatched immediately*, using the current time to calculate any relative date filters and settings defined in the flow - unlike scheduled process flow runs which are added to a queue.

To ensure that any time elapsed between the time a failed run was originally queued and the time it is retried, the current time for a retry always assumes the queued at time for the original run that failed.

Example

Suppose that a process flow is scheduled (via a trigger shape) to start at 13:00.

• At 13:00, the run is initialised and added to a queue. The queued at time is set to 13:00.

• At 13:03 the queued run is processed. Because the use queued time process flow setting is toggled ON, the queue delay of 3 minutes is accounted for in all relative date/time settings defined in the flow.

• The run fails.

• At 17:00, the run is retried (via the run logs page). The retry happens immediately however, the current time assumes the original queued at time of 13:00.

Introduction

When a process flow is running, the logs panel provides real-time information about the progress of each step:

This is a great way to monitor what's happening during each step of the flow.

Working with real-time logs

As a flow progresses, click the 'tick' icon for a shape to view information for that step - for example:

You can drag this panel wherever you like on the canvas. If you prefer to work with a larger window, click the 'pop-out' icon:

Log details

Details are provided over a series of tabs:

• Logs

• Payloads

• Failed payloads

• Requests

• Responses

Logs

The logs tab provides a commentary about the status of the process flow as it progresses through each step - for example:

Payloads

The payloads tab shows the payload for the selected step in the process flow - this means you can see exactly what's happened to your data at every stage. For example:

If a shape generates multiple payloads (for example, where a flow control shape batches data into smaller payloads, or if incoming data is paginated), you can choose which one to view - for example:

Failed payloads

If the remove failed payloads option is toggled on in process flow settings OR a try/catch shape has found exceptions, this tab displays any payloads that were removed in this run:

If multiple payloads are removed, you can choose which one to view - for example:

Requests

The requests tab shows details of the API request made for the selected shape - for example:

Here you can see the API request URL, method, body content and headers. If multiple requests are made (for example, if a connection fails the first time) you can use the request dropdown list to view each one.

Responses

The responses tab shows details of the API response received for the selected shape - for example:

Here you can see the API response status, body content and headers. If multiple requests were made (for example, if a connection failed the first time) you can use the response dropdown list to view each one.

Introduction

You can configure a process flow to trigger email notifications in the event that a process flow runs and fails. This is a two-stage process:

Need to know

Email notifications for a failed process flow run are only sent if:

• the process flow is

• the process flow is initiated via the (i.e. not for manual runs)

Updating process flow settings to trigger failed run notifications

Having , you should access settings for the required process flow(s) and apply this group to the email failure notifications section. To do this, follow the steps below.

Step 1 that you want to update.

Step 2 Click the 'cog' icon in the toolbar to access general settings for this process flow:

Step 3 From the settings panel, click in the email failure notifications field and select the appropriate group to receive an email notification in the event that this process flow runs and fails:

Step 4 Save changes.