Having accessed run logs and selected the view logs option for a given flow run, you can view:
The insights summary is divided into three functional areas:
In the top, right-hand corner (1) you'll find additional options via an ellipses menu - these options will vary, depending on whether the run succeeded or failed. See Additional options for more information.
In the top panel (2), you can see basic details for the process flow (name, id, version and status, and how triggered) together with duration, CPU usage time, data used, and score.
The lower panel (3) shows every step (i.e. every shape) in the flow in processing sequence, with the following information:
The score is a measure of how efficient/expensive (in terms of processing) your process flows are, based on the volume of data processed per second - 999 is the highest score.
Your score is based on all runs for all process flows associated with your company profile. This includes flows that are:
Triggered by a schedule, webhook, or event
Initialised by an API request
Run manually
Enabled or disabled when run
Draft or deployed status when run
If your score is on the low side, it may be that your process flow is necessarily complex - including items such as scripts, transformations, flow control, caches, etc., has an impact on your score.
However, it's always worth checking a low-scored process flow further as there may be places where your process flows can be optimised (for example, does a flow include lots of mapping transformations that could be achieved in a single script?). A quick scan down the shapes list in the lower panel will show, at a glance, which shapes in your flow are taking longer to process or perhaps processing unusually high volumes of data.
If your score is low and you're satisfied that the process flow is built optimally, don't worry too much- scores are just there as a flag to indicate that it's worth checking the efficiency of your flow.
See our Best practice for building process flows section for advice on building efficient process flows.
In the lower panel of the insights summary, you can click any entry to access detailed logs:
In the top section, you can see a full history of what happened during this step - this will often display over multiple pages:
In the lower section, you can view and download any payloads associated with this step. Click the 'view' icon associated with any payload to see the contents:
...the payload is displayed:
Longer payloads are trimmed but you can use the download option to retrieve the full content:
Payloads are available for 72 hours after the process flow has run.
The ellipses icon in the top, right-hand corner provides access to additional options. These options vary, depending on whether the run failed or succeeded:
When a run is successful or partially successful, you can download run logs. If the run failed, you can download logs and also retry the run.
| Column | Summary |
|---|---|
If a failed payload was removed so that the flow run could continue (either because connector settings are set to do this OR a try/catch shape has found exceptions), it's displayed with a failed payload status - for example:
Name
The name given for the shape in the process flow. If your process flow includes custom names for shapes, these are shown here - otherwise the default name is displayed.
Shape
The underlying microservice for the shape.
Flow id
CPU time
The total amount of time spent processing.
Payload count
The total number of payloads processed.
Payload size
The total size of processed payloads.
Operations received
The number of operations completed to receive data. For more information please see About operations.
Operations sent
The number of operations completed to send data. For more information please see About operations.
Waterfall
A visual representation of where each step starts and finishes.
This page has been updated re. some new features which will be available in our next scheduled release. If you don't see an option that's shown here, please check back after this release.
If a process flow run fails for any reason, it's displayed in run logs with a status of failed - for example:
You can retry the run from this page. Before doing so, please consider:
It's advisable to check your process flow setup (filters and de-dupe shapes) to ensure that running the flow again will not duplicate any runs that may have succeeded in the interim period.
If a process flow includes references to variables, be aware that any values for those variables may not be the same as they would have been if the failed flow had been successful.
When the retry option is used, the flow version that was set originally is run. If you want to run an updated version of the process flow, you can choose to edit the flow and initialise a run manually instead.
A process flow must be enabled for the retry option to work.
Step 1 Log into the Patchworks dashboard and access your run logs & queue.
Step 2 Click the ellipses icon to access additional options for the log entry, then select the retry flow run option - for example:
When the retry option is used for a failed process flow run, the flow run is dispatched immediately*, using the current time to calculate any relative date filters and settings defined in the flow - unlike scheduled process flow runs which are added to a queue.
*Retried runs are not added to a queue, which means they will are processed as soon as possible. Often, this will be immediately, but in busier times a retry job may be placed behind others.
To ensure that any time elapsed between the time a failed run was originally queued and the time it is retried, the current time for a retry always assumes the queued at time for the original run that failed.
Suppose that a process flow is scheduled (via a trigger shape) to start at 13:00.
At 13:00, the run is initialised and added to a queue. The queued at time is set to 13:00.
At 13:03 the queued run is processed. Because the use queued time process flow setting is toggled ON, the queue delay of 3 minutes is accounted for in all relative date/time settings defined in the flow.
The run fails.
At 17:00, the run is retried (via the run logs page). The retry happens immediately however, the current time assumes the original queued at time of 13:00.
Having accessed the run logs & queue page, the active tab displays active and previous process flow runs - these are listed chronologically based on the run start time:
At a glance, you can see the outcome of a run, the start date/time, the duration, how the run was triggered and whether there were any warnings.
Failed runs are shown with an expandable summary of why the run failed, and you can also access additional options to access detailed logs, and more.
Click the name of a process flow to access its setup.
If a process flow is running it is displayed with a 'stop' button that you can use to stop the run.
Run logs are retained for 7 days.
Payloads are retained for 72 hours.
Payloads are stored in AWS S3 (eu-west-2 London).
Payloads are encrypted during pull/push operations (HTTPS, TLS 1.2/1.3).
There is a limit on the number of process flow runs that can be started per minute. This limit varies according to your subscription tier - please see Core subscription tiers for information.
If you're looking for a specific log, you can use the search field to search by flow run id or by process flow name:
If you want to find logs for a specific flow run id, type the full id into the search field - for example:
You must enter the id in full - partial elements are not matched.
A flow run id search is not case sensitive - entering 01j6errxr432r4k4dh41y5ttpc is the same as 01J6ERRXR432R4K4DH41Y5TTPC or 01J6eRRxR432r4K4DH41Y5ttPC.
If you want to find logs for a process flow that you only know by name, type any part of the name into the search field - for example:
You can enter as much or as little of the name as you like.
A flow name search is not case sensitive - entering CHECK is the same as check or Check.
By default, logs are displayed for the last seven days. You can change this using the associated date filter - here, you can clear all filters to show all available logs, or add rules to apply a new date/time range:
Use filters to refine the list based on the value associated with a particular column. Currently, you can filter logs by:
Started at date
Trigger (schedule, manual, API, etc.)
Has warnings
Status
To define a filter, click the filter icon associated with a column - for example:
...then make selections from the options provided - for example:
The show stats option can be toggled ON to view additional columns in the log table:
When this option is toggled ON, three more columns are shown - these show insights for the performance of the process flow.
Further insights are available from the logs associated with each run, and also from your company insights page.
The status of a run will be one of the following:
If warnings are given during a run (be it failed or successful), the entry is shown with an icon in the Has warnings column - for example:
If the run failed, you should view associated logs to check these warnings, then make any changes necessary to your process flow. If the run succeeded, you should still view the logs for warnings in case the issue causes a failure in the future.
Run logs can contain a lot of information and there may be times when you want to quickly check why a run has failed.
When a process flow run is initialised but fails somewhere in the run, its status is set to failed and an expandable arrow is shown to the left of the entry - click this arrow to view a summary of why the failure occurred. For example:
Each log entry has an ellipses icon, with access to additional options:
From here you can:
View logs. Access more insights and detailed logs for this run. For more information please see our viewing logs page.
View logs (classic). If you prefer to work with 'original style' logs, you can access them from here. For more information please see our viewing logs (classic) page.
Download logs. Use this option to download detailed logs in CSV format. For more information please see our downloading run logs page.
Retry the flow run. Use this option to retry a process flow run that previously failed. For more information please see our retrying a failed process flow run page.
Having accessed run logs and selected the view logs (classic) option for a given flow run, you can view detailed logs and payloads for that run:
Here, you can scroll down the page to view information for each step of the process flow.
Click the 'sort' option on the date column to toggle the log sequence in ascending/descending order.
Click the 'eye' icon associated with the run that you want to access:
The log entry expands and further options are available. Click the 'eye' icon to show the payload:
Longer payloads are trimmed but you can use the download option to retrieve the full content:
Payloads are available for 72 hours after the process flow has run.
Patchworks id for the process flow. You'll see this number in the canvas title block (immediately above the title) when editing the flow:
Run succeeded / partially succeeded
Run failed
| Status | Summary |
|---|---|
Success
The process run was completed success without errors.
Partial success
The process flow run was completed successfully however at least one failed payload was removed.
Failed
The process run did not complete due to errors.
Retried
The failed run has been retried.
Stopped
The process flow was stopped manually.
Running
The process flow is currently running.
The date filter displays date/time rule pickers:
If required, you can download run logs for a completed process flow run in CSV format, for use outside of Patchworks.
Run log exports are completed in CSV format. The exported file includes two columns with level and message headers. For example:
To download run logs for a specific process flow run, follow the steps below.
Step 1 Log into the Patchworks dashboard and access your run logs & queue.
Step 2 Click the ellipses icon to access additional options for the log entry, then select the download logs option - for example:
Step 3 The download job is added to a queue and a confirmation message is displayed:
Step 4 When your download is ready, you'll receive an email which includes a link to retrieve the file from the file downloads page. If you can't/don't want to use this link, you can access this page manually by selecting the settings option:
...followed by the file downloads option:
Step 5 On the file downloads page, you'll find any exports that have been completed for your company profile in the last hour. Click the download button for your job:
This list may include exports from different parts of the dashboard, not just run logs (for example, de-dupe and cross-reference lookup data exports are added here).
Step 6 The associated CSV file is saved to the default downloads folder for your browser.
Download files are cleared after one hour. If you don't manage to download your file within this time, don't worry - just download the logs again to create a new one.
The date filter displays date/time rule pickers:
