Introduction

Process flows are built by dragging and dropping shapes onto a canvas, and then configuring those shapes to work in the way you need to exchange data between connector instances.

Process flows are extremely flexible. You can build something very simple to sync data between two instances with standard field mappings - or build more complex flows, perhaps using custom scripts and/or routing data to different paths based on given conditions.

Prerequisites for building a process flow

Before you start building a process flow, make sure that you've installed a connector and added your required instances for any third-party applications that you want to use.

In this section

• Approaching your first process flow

• Techniques for building process flows

• Best practice for building process flows

• Process flow versioning

• Adding a new process flow

• The process flow canvas

• Process flow shapes

• Shape notes

• Dynamic variables

Introduction

In this section, we share some insights on best practice for different aspects building process flows.

Pages in this section

• Scripts - best practice

• Multi environment management - best practice

Introduction

When building and testing your process flows before going live, a common requirement is to build process flows that connect to development/staging instances of your third-party systems - this ensures that you're not working with live data during the testing phase.

This page details our suggested procedure for managing this approach in terms of:

• Adding process flows with different environments

• Updating process flows with different environments

Adding process flows with different environments

Our suggested practice for adding process flows when you want to work with different environments is detailed in the following sections:

1. Add connectors

2. Build & test process flows

3. Duplicate process flows for production

4. Configure process flows for production

5. Housekeeping

Phase 1: add connectors

Step 1 Install or build required connectors for third-party systems to be integrated in your process flows.

Step 2 Add any required instances of these connectors and specify your DEV/STAGE credentials (i.e. credentials that allow you to access data in your DEV/STAGE environment for the third-party system). It's a good idea to indicate DEV/STAGE as part of the instance name. For example:

Step 3 Add any required instances of these connectors again, this time using your PRODUCTION credentials (i.e. credentials that allow you to access data in your PRODUCTION environment for the third-party system). It's a good idea to indicate PRODUCTION as part of the instance name. For example:

Phase 2: build & test process flows

Step 1 Create a new process flow with the required name and description. You may wish to indicate DEV/STAGE as part of the process flow name but this isn't essential if you follow subsequent steps to apply labels.

Step 2 From the process flow canvas, access process flow settings.

Step 3 Move down to the labels section and apply a label that indicates that this process flow is configured for your DEV/STAGE environment - for example:

If required labels don't exist, you can create them 'on the fly' from here, or go to settings > labels for label management. More information is available in our Process flow labels section.

Step 4 Build the process flow. When configuring connection shapes, ensure that you select the DEV/STAGING instance:

Step 5 Build the rest of your process flow as required. If you are required to select instances for other shapes in your flow, always ensure that you select the DEV/STAGING instance.

Step 6 Test carefully to ensure that the right data (i.e. DEV/STAGING data) is processed as expected.

Phase 3: duplicate flows for production

Step 1 When you're satisfied that your process flow is working correctly, duplicate the version that you want to use in your live environment.

Phase 4: configure flows for production

Step 1 Edit the duplicated process flow and access process flow settings.

Step 2 Update the process flow name as required. You may wish to indicate PRODUCTION as part of the process flow name but this isn't essential if you follow subsequent steps to apply labels.

Step 3 Move down to the labels section. Remove the existing DEV/STAGE label and replace it with a label that clearly indicates that this process flow is configured for your PRODUCTION environment - for example:

If required labels don't exist, you can create them 'on the fly' from here, or go to settings > labels for label management. More information is available in our Process flow labels section.

Step 4 Save changes.

Step 5 For every connection shape in the process flow, access settings and change the DEV/STAGING instance to the equivalent PRODUCTION instance:

Step 6 Check all other shapes in your process flow - if instance details are present, ensure that the PRODUCTION instance is selected.

Step 7 Check other shapes in your process flow. Typically, any instance information defined for a connection shape is inherited by subsequent 'child' shapes (which require instance details) in the flow. However, it's always worth double-checking before going live.

Step 8 When you're ready, deploy and enable the process flow.

Phase 5: housekeeping

At this point, you now have different 'environment flavours' of the same process flow - for example:

Keep in mind that your Core subscription tier determines the number of active (i.e. deployed and enabled) process flows that are allowed for each company profile.

Once your PRODUCTION version is in place and running, we advise that you disable the DEV/STAGE version but leave it in place to test any future updates.

Updating process flows with different environments

If you need to update a PRODUCTION process flow, there are two options, summarised below.

Option 1

Edit the draft version of the existing PRODUCTION process flow and deploy changes once you're satisfied that the flow is running correctly.

Option 2

Edit and test the existing DEV/STAGING version of the process flow. Once testing is complete, follow steps in Phase 3, Phase 4 and Phase 5 above to duplicate the process flow and configure PRODUCTION connections. Having done this, you can either:

• Remove the 'old' PRODUCTION process flow

• Retain the 'old' PRODUCTION process flow but apply an ARCHIVE label and ensure it's disabled.

Adding shapes

To add a shape to a process flow, click the + icon at the point you want to place it - for example:

...then choose the type of shape to add:

Depending on the shape, the settings panel will either open immediately so you can provide details before the shape is added to the canvas, or the shape is added to the canvas and you can update settings when you're ready.

Accessing shape settings

To access settings for an existing shape, click the associated 'cog' icon - for example:

The settings panel is displayed, so you can configure the shape as required - don't forget to save changes.

Changing a shape name

When a shape is dropped into the canvas, it's labelled with a generic name - 'map', 'flow', 'split', etc. Sometimes it can be useful to modify these names to something more specific - for example, to give a hint of what the shape's purpose in your flow (particularly if you have multiple shapes of the same type!).

To change the name, simply access shape settings, then click the 'edit' icon associated with the name at the very top of the settings drawer - for example:

The name field can now be edited - update the current name as needed, then click save at the bottom of the settings drawer:

Removing shapes

To remove a shape, click the associated 'cog' icon - for example:

...then click the delete option in the settings panel - for example:

Introduction

The flexibility of process flows means that there's no 'one size fits all' approach - everyone's requirements are different, and the scope is huge. This level of flexibility is a great advantage but on the flip side - where do you start?

Here, we outline the bare bones of a process flow so you know what to consider as a minimum when getting started for the first time.

Testing

A scratchpad area will be available soon. In the meantime, we suggest registering for a sandbox account and experimenting there.

Make sure you create instances with credentials for your third-party application sandbox accounts, rather than live ones!

Key elements of a process flow

In their simplest form, process flows are defined to receive data from one third-party application and send it to another third-party application, perhaps with some data manipulation in between. Key elements are summarised below.

Process flow versions

Process flow can be associated with three version types: draft, deployed and inactive. Before you get started building process flows, we advise reading our Process flow versioning page to make sure you understand how this works.

Introduction

The process flow canvas is where you build and test your process flows in a smart, visual way. This is where you define if, when, what, and how data is synced.

Finding your way around the process flow canvas

The process flow canvas has four main elements - a title, an actions bar, a shapes area, and a (hidden unless activated) options panel:

The process flow title bar shows the name of the process flow, as specified when it was created. The number above the title is the process flow id and the number below is the version number. To change this title, use the settings option from the actions bar.

Options in the actions bar are summarised below:

The main 'shapes area' is where you build your process flow. Start by clicking the + sign associated with the trigger shape, then choose the required shape for your next step, and start building!

When you access process flow settings or to configure a shape in your flow, available settings are displayed in a panel on the right-hand side. For example, when we choose to access settings for a trigger shape, available trigger options are displayed:

Introduction

Process flow settings are used to manage settings and behaviour for the process flow as a whole:

Understanding process flow settings

Process flow settings are summarised below.

Introduction

Patchworks process flows are incredibly flexible. With a range of shapes for receiving, paginating, manipulating, batching, splitting, caching and sending data, you can build highly complex flows in a matter of a minutes.

With this in mind, it's important to understand how data flows through shapes.

How data flows

In the simplest of scenarios, a process flow receives a single payload of unpaginated data and this flows all the way through to completion with no manipulation or batching - one payload is received, processed, and completed.

However, if your incoming data is paginated and/or you introduce shapes capable of generating multiple payloads, it's important to understand how these pass through the flow. Essentially, any payloads that a shape outputs are added to a 'bucket' and it's that bucket that is then passed to the next shape.

So, all payloads from one shape are passed to the next shape in the same context - they don't pass down the entire flow individually.

Demo

The animation below shows how this works.

Shape timeouts and retries

Timeouts

Retries

With some exceptions (detailed below), a further three attempts will be made if a process flow shape fails. Exceptions are summarised below:

Introduction

This page summarises best practice insights for for working with scripts in process flows. Keep in mind that scripts come in two 'flavours':

• Payload scripts. A is configured to run a given script on the incoming payload.

• Transform scripts. A is added to a field mapping (in the ) - this runs a given script on the associated source field before the mapping is completed.

One script is more efficient than multiple scripts

Generally, doing everything you can in a single script is more efficient than processing multiple scripts - it means less downtime between steps, and less chance of being caught in a queue between scripts.

There are times where deploying multiple scripts is preferable from a management perspective - for example, if you're designing generic scripts to use across multiple process flows. In this case, you trade modularity for speed, in most cases.

Payload scripts are more efficient than transform scripts

Payload scripts (i.e. scripts run via the ) are more efficient than .

Effectively, a transform script pauses the map shape, calls the script, then merges that response and continues with the map. Multiple script transforms results in multiple pauses - similar to running multiple scripts back-to-back in the process flow itself.

Debugging scripts

The allows you to edit scripts and test with a given payload, but for detailed debugging, we advise using an IDE with a debugger to step through the code.

Introduction

When you add a process flow, you're given a new, blank to start .

A process flow can be as simple or as complex as you need, and you can add as many as you like - maybe one flow will fulfil all of your requirements, or maybe you'll need several to achieve different things.

The steps

To add a new process flow, follow the steps below.

Step 1 Log in to the , then select process flows > process flows to access the manage process flows page.

Step 2 Click the create new flow button:

Step 3 Enter a name for this flow, then click the next step button.

Introduction

You can use the assert shape is typically used for testing purposes -you can define a static payload which is used to validate that the current payload (i.e. the payload generated up to the point that the assert shape is encountered) is as expected.

Accessing assert payload shape settings

To view/update the settings for an existing assert shape, click the associated 'cog' icon:

Configuring settings for a new assert shape

To configure an assert shape, all you need to do is paste the required payload and save the shape for example:

Introduction

Shapes detailed in this section are available as standard, for all core subscription tiers:

• Assert

• Connection

• Filter

• Flow control

• Manual payload

• Map

• Notify

• Route

• Run process flow

• Set variables

• Split

• Track data

• Trigger

• Try/Catch

Introduction

Typically, a process flow run is triggered and a request for data is made via a connector shape - if the request is successful, data is retrieved and the flow continues.

However, there may be scenarios where you need to control whether the connector shape or process flow run should fail/continue based on information returned from the connection request. To achieve this, you can apply a response script to your connector shape.

How it works

When a response script is applied to a connector shape, the script runs every time a connection is attempted. The script receives the response code, headers, and body from the request and - utilising response_code actions - returns a value determining whether the connector shape/flow run continues or stops.

Response scripts are just like any other custom script, except they receive additional information from the request - see lines 11 to 14 in the example below:

<?php

/**
 * Handler function.
 *
 * @param array $data [
 *      'payload'   => (string|null) the payload as a string|null
 *      'variables' => (array[string]string) any variables as key/value
 *      'meta'      => (array[string]string) any meta as key/value
 *      'flow'      => (array[mixed]) current flow data, including variables
 *      'response'  => [
 *           'headers' => ['Content-Type' => 'application/json', .......],
 *           'body' => '....',
 *           'status' => 200
 *      ]
 *    ]
 *
 * @return array $data Structure as above, plus 'logs' => (array[string]) Logs to be written to flow run log after script execution
 */

function handle($data)
{
    return $data;
}

Implementing a response script

To implement a response script, you should:

1. Write and deploy the required script

2. Apply the script to your connector shape

Step 1: Write & deploy your response script

Response scripts are written and deployed in the usual way, via the custom scripts option. However, two additional options can be used for scripts that you intend to apply via connector shapes: response_code and message.

Response code

The response_code determines how the process flow behaves if a connection request fails. Supported response_code values are:

Message

The message is optional. If supplied, it is output in the run logs.

Example

function handle($data)
{
    // Stops the flow with a message
    $data['response_code'] = 3
    $data['message'] = 'Flow stopped by response script';

    return $data;
}

Step 2: Apply the response script

To apply your response script, access settings for the required connector shape and select your script from the response script dropdown field.

Examples

Scenario 1

Here we handle the scenario where a connection response appears OK because the status code received is 200, but in fact the response body includes a string (Invalid session) which contradicts this. So, when this string is found in the response body, we want to retry the process flow.

In this case we return a response_code of 2 with an message of Invalid session:

<?php
/**
 * Handler function.
 *
 * @param array $data [
 *      'payload'   => (string|null) the payload as a string|null
 *      'variables' => (array[string]string) any variables as key/value
 *      'meta'      => (array[string]string) any meta as key/value
 *.     'response'  => [
 *           'headers' => ['Content-Type' => 'application/json', .......],
 *.          'body' => '....',
 *           'status' => 200
 *      ]
 *    ]
 */
function handle($data)
{
    handle invalid session error in PVX
    $data['response']['status'] = 200;
    $data['response']['body'] = 'Invalid session';
      if (str_contains($data['response']['body'], 'Invalid session')) {
        return [
         'response_code' => 2 // retry flow
         'message' => 'Invalid session'
       ];
     }

Scenario 2

Here we show how the payload received from a connection request is checked for an order number and an order status - retrying the the process flow if a particular order status is found:

<?php
/**
 * Handler function.
 *
 * @param array $data [
 *      'payload'   => (string|null) the payload as a string|null
 *      'variables' => (array[string]string) any variables as key/value
 *      'meta'      => (array[string]string) any meta as key/value
 *.     'response'  => [
 *           'headers' => ['Content-Type' => 'application/json', .......],
 *.          'body' => '....',
 *           'status' => 200
 *      ]
 *    ]
 */
    // check if order is ready to be processed yet
      $data['payload'] = [
        'order_id' => 1,
        'status' => 'Pending',
      ]
      if ($data['payload']['status'] === 'Pending') {
        return [
          'response_code' => 2, // retry flow
          'message' => 'Order not ready for processing, adding flow back to queue'
        ]
      }

Related pages

• Custom scripts

Introduction

There may be times when you need to define a filter based on incoming data matching one of many given values. Conversely, you might want to define a filter based on incoming data NOT matching one of many given values. This can be achieved using the following operators in string-type filters:

• Contains one of many

• Does not contain one of many

Using these operators, you can specify a comma-separated list of values that a record must have/not have in a string-type field, to be a match.

Contains one of many

The contains one of many operator is used to match incoming records if the value of a given field DOES match any item from a provided (comma delimited) list of values. For example, consider the following payload of customer records:

[
    {"name": "John Smith",  "country": "FR"},
    {"name": "Bob Jones",  "country": "BE"},
    {"name": "Hank Smith",  "country": "AU"},
    {"name": "Jane Smith",  "country": "DE"},
    {"name": "Jack Jones",  "country": "SE"},
    {"name": "Hank Smith",  "country": "US"},
    {"name": "Paul Smith",  "country": "IE"}
]

Suppose you only want to process customer records with a European country code in the country field (which is a string type field).

You can add a filter for the country field and select the contains one of many operator - then provide a comma-separated list of acceptable country codes as the value:

The resulting payload would only include records where the country field includes one of the specified values - i.e.:

[
    {"name": "John Smith",  "country": "FR"},
    {"name": "Bob Jones",  "country": "BE"},
    {"name": "Jane Smith",  "country": "DE"},
    {"name": "Jack Jones",  "country": "SE"},
    {"name": "Paul Smith",  "country": "IE"}
]

Does not contain one of many

The does not contain one of many operator is used to match incoming records if the value of a given field DOES NOT match any items from a provided (comma delimited) list of values. For example, consider the following payload of customer records:

[
    {"name": "John Smith",  "country": "FR"},
    {"name": "Bob Jones",  "country": "BE"},
    {"name": "Hank Smith",  "country": "AU"},
    {"name": "Jane Smith",  "country": "DE"},
    {"name": "Jack Jones",  "country": "SE"},
    {"name": "Hank Smith",  "country": "US"},
    {"name": "Paul Smith",  "country": "IE"}
]

Suppose you only want to process customer records that do NOT have US or AU in the country field (which is a string type field).

You can add a filter for the country field and select the does not contain one of many operator - then provide a comma-separated list of unacceptable country codes as the value:

The resulting payload would only include records where the country field does NOT include one of the specified values - i.e.:

[
    {"name": "John Smith",  "country": "FR"},
    {"name": "Bob Jones",  "country": "BE"},
    {"name": "Jane Smith",  "country": "DE"},
    {"name": "Jack Jones",  "country": "SE"},
    {"name": "Paul Smith",  "country": "IE"}
]

List syntax notes

When defining your 'many values' list as the value for a contains one of many or a does not contain one of many filter, there are a couple of things to keep in mind:

• Format

• Spaces

Format

It's important that your 'many values' are specified as a comma-separated list - so in our example:

FR,BE,DE,SE,IE 

...will match as required, but:

FR BE DE SE IE 

...will NOT match as required.

Spaces

Any spaces included in your 'many values' list ARE considered when matching. For example, consider our original payload:

[
    {"name": "John Smith",  "country": "FR"},
    {"name": "Bob Jones",  "country": "BE"},
    {"name": "Hank Smith",  "country": "AU"},
    {"name": "Jane Smith",  "country": "DE"},
    {"name": "Jack Jones",  "country": "SE"},
    {"name": "Hank Smith",  "country": "US"},
    {"name": "Paul Smith",  "country": "IE"}
]

Suppose we are using the contains one of many operator to match all European countries and the value field is defined as below:

FR,BE,DE,SE, IE 

Notice that the final IE list item is preceded by a space. This means that our filter will only match the IE country code if it's preceded by a space in the payload, so the output would be:

[
    {"name": "John Smith",  "country": "FR"},
    {"name": "Bob Jones",  "country": "BE"},
    {"name": "Jane Smith",  "country": "DE"},
    {"name": "Jack Jones",  "country": "SE"}
]

Our IE record (line 8 in the payload) isn't matched because there's no space before the country code in the country field.

Introduction

The Patchworks FTP connector is used to work with data via files on FTP servers in process flows. You might work purely in the FTP environment (for example, copying/moving files between locations), or you might sync data from FTP files into other connectors, or you might use a combination of both! For example, a process flow might be designed to:

1. Pull files from an FTP server

2. Use the data in those files as the payload for subsequent processing (e.g. sync to Shopify)

3. Move files to a different FTP server location

This guide explains the basics of configuring a connection shape with an FTP connector.

About the Patchworks FTP connector

When you add a connection shape and select an FTP connector, you will see that two endpoints are available:

Here:

• FTP GET is used to retrieve files from the given server (i.e. to receive data)

• FTP PUT is used to add/update files on the given server (i.e. to send data)

Configuring FTP endpoints

Having selected either of the two FTP endpoints, configuration options are displayed. The same options are used for both endpoints but in a different sequence, reflecting usage:

For information about these fields please see our Configuring SFTP connections page - details are the same.

Introduction

With our process flow versioning system, you can be sure that a process flow that's currently deployed will never be edited (possibly with breaking changes) while it's in use.

To edit a deployed processed process flow, you take a copy as a draft and work on that - when you're ready, you can then deploy your draft.

Each time that you deploy a new version of a process flow, the previously deployed version is saved as an inactive version for future reference and, if required, future use.

Version types summary

At any given time, a process flow can be associated with one of the following version types:

More about version types

Draft

There is always one draft version of a process flow. The draft version can be edited freely without any possibility of changing or breaking the version that's currently deployed. With a draft version, you can add/update shapes, and change the process flow name.

When you're working with a draft version of a process flow, you can take the following actions:

• Enable/disable the process flow. If you enable a process flow when viewing a draft version, there's no impact on the draft version. However, the deployed version will start to run automatically as per its trigger shape settings.

• Run manually. Use this option to run the draft process flow immediately.

• Deploy. When you deploy a draft version, it becomes the currently deployed version and stays as the current draft - the previously deployed version becomes a new inactive version.

Deployed

The deployed version of a process flow is the one that's currently in use (if it's enabled) or ready for use (if it's disabled).

The deployed version of a process flow cannot be edited - shapes can't be added/updated, and you can't change the name. The only actions that you can take with a deployed version of a process flow are:

• Enable/disable the process flow. Just because a process flow version is deployed, it doesn't necessarily mean that it will be triggered to run automatically as per trigger shape settings. For this to happen, a process flow must be both deployed AND enabled.

• Run manually. Use this option to run the process flow immediately.

• Copy to draft. When you do this, the process flow remains deployed and an exact copy is taken as the current draft version, ready for you to edit - the existing draft version is discarded. This is a good solution if you've been editing a draft but reached the point where you need to restart from a known sound point.

Inactive

Each time a draft version of a process flow is deployed, the previously deployed version becomes an inactive version - so you have a full version history for all deployed versions of a process flow.

An inactive version of a process cannot be edited - shapes can't be added/updated, and you can't change the name. The only actions that you can take with an inactive version of a are:

• Enable/disable the process flow. If you enable a process flow when viewing an inactive version, there's no impact on the inactive version. However, the deployed version will start to run automatically as per its trigger shape settings.

• Run manually. Whilst you can use this option to run the process flow immediately, it's not recommended.

• Copy to draft. When you copy an inactive version to draft, an exact copy is taken as the current draft version, ready for you to edit (the existing draft version is discarded). There's no impact on the deployed version.

• Deploy. When you deploy an inactive version, it becomes the currently deployed version. The previously deployed version becomes a new inactive version, and the existing draft is not affected.

Working with versions

• Accessing process flow versions

• Understanding which version you're viewing

• Switching between versions

• Deploying a draft or inactive version

• Copying to draft

Accessing process flow versions

You can view all versions of a process flow via the settings panel.

Understanding which version you're viewing

When you access a process flow, the version being viewed is noted in the title bar. If you are viewing a deployed or inactive version, you'll see a message advising that edits cannot be made, and the version number is displayed beneath the title.

Switching between versions

To switch between different versions of a process flow, access the versions list and select the required entry.

Deploying a draft or inactive version

Deploying the draft version of a process flow - or deploying an inactive version without editing it as a draft first - is a simple one-click operation from the versions list.

Copying to draft

If you want to edit the currently deployed version of a process flow - or an inactive version - you must first copy it to draft. The existing draft version is replaced by the version you copy.

Standard shapes

Advanced shapes

Introduction

Mappings are at the heart of Patchworks.

When we pull data from one system and push it into another, it’s unlikely that the two will have a like-for-like data structure. By creating mappings between source and target data fields, the Patchworks engine knows how to transform incoming data as needed to update the target system.

The illustration below helps to visualise this:

Accessing map shape settings

The map shape includes everything that you need to map data fields between two connections in a process flow. When you start to create mappings, there are two approaches to consider:

Having added a map shape to a process flow, click the associated 'cog' icon to access settings:

Auto-generating mappings between connections

The generate automatic mapping option is used to auto-generate mappings between your selected source and target connections.

Once auto-generation is complete, mapping rows are added for all fields found in the source data - for example:

Auto-generating mappings for custom connections

• If yes, your custom connector will behave like any of our prebuilt connectors when it comes to auto-generated mappings, adding fully mapped rows for all matched tags.

Adding mappings manually

It's very easy to add individual mapping rows manually, using the add mapping rule option:

Introduction

The flow control shape can be used for cases where you're pulling lots of records from a source connection, but your target connection needs to receive data in small batches. Two common use cases for this shape are:

• A target system can only accept items one at a time

• A target system has a maximum number of records that can be added/updated at one time

The flow control shape takes all received items, splits them into batches of your given number, and sends these batches into the target connection.

Demo

Adding & configuring a flow control shape

Step 1 In your process flow, add the flow control shape in the usual way:

Step 2 Select a source integration and endpoint to determine where the incoming payload to be split originates - for example:

Step 3 Move down to the batch level field and select that data element that you are putting into batches. For example :

Step 4 In the batch size field, enter the number of items to be included in each batch. For example:

Step 5 By default, the payload format is auto-detected but you can set a specific format here if you prefer:

Step 6

If you're creating batches of one record, you can toggle ON the Do not wrap single records in an array option if you want the output to be this:

...rather than this:

Step 7 Save the shape. Now when you run this process flow, data will be split into batches of your given size.

Introduction

When a connector is built, default filters can be applied at the API level, so when a process flow pulls data, the payload received is has already been refined.

However, there may be times where you want to apply additional filters to a payload that's been pulled via a - for example, if the API for a connector does not support particular filters that you need.

Need to know

When specifying a filter value, the maximum number of characters is 1024.

Accessing filter shape settings

To view/update the settings for an existing filter shape, click the associated 'cog' icon:

Configuring settings for a new filter shape

Follow the steps below to configure a filter shape.

Step 1 Select a source integration and endpoint to determine where the incoming payload to be filtered originates.

Step 2 Click the add new filter button:

Step 3 Filter settings are displayed:

From here, you can select a field (from the data schema associated with the source endpoint selected in step 1) - for example:

Alternatively, you can toggle the manual input option to ON and add a manual path.

Step 4 Use remaining operator, type and value options to define the required filter.

Step 5 Use the keep matching? toggle option to choose how matched records should be treated:

Here:

• If the keep matching? option is toggled OFF, matched records are removed from the payload before it moves on down the flow for further processing.

• If the keep matching? option is toggled ON, matched records remain in the onward payload, and all non-matching records will be removed.

Step 6 Click the create button to confirm your settings.

Step 7 The filter is added to the filter shape - you can now add more filters if needed:

Available filter types

When defining a filter, you can choose from the following types:

Related information

Introduction

When adding filters for a string-type field, there may be times when standard operators can't achieve what you need. For example, when defining multiple filter conditions, the default condition is AND - so ALL specified filters must be met for a match. But what if you need an OR condition?

For more complex filtering requirements, regex can be used.

Regex filter example

Let's take the following payload:

Suppose we want to retrieve any items where the value of the fruit field contains peaches OR apples. We might be tempted to add two string-type filters in the filter shape:

However, this wouldn't return any matches because we'd be looking for any records where peaches AND apples are present. Instead, we can define one string-type filter with regex:

Introduction

The connector shape is used to define which connector should be used for sending or receiving data, and then which endpoint.

Any connector instances that have been for your are available to associate with a connector shape. Any endpoints configured for the underlying connector will be available for selection once you've confirmed which instance you're using.

Accessing connector shape settings

When you add a connector shape to a process flow, the is displayed immediately, so you can choose which of your connector instances to use, and which endpoint.

To view/update the settings for an existing connector shape, click the associated 'cog' icon to access the - for example:

Configuring settings for a connector shape

Follow the steps below to configure a connector shape.

Step 1 Click the select a source integration field and choose the instance that you want to use - for example:

Step 2 Select the endpoint that you want to use - for example

Step 3 Depending on how your selected endpoint is configured, you may be required to provide values for one or more variables.

Step 4 Save your changes.

Step 5 Once your selected instance and endpoint settings are saved, go back to edit settings:

Now you can access any optional filter options that are available - for example:

Step 6 The request timeout setting allows you to override the default number of seconds allowed before a request to this endpoint is deemed to have failed - for example:

Step 7 Set error-handling options as required. Available options are summarised below:

Step 8 Set the payload wrapping option as appropriate for the data received from the previous step:

This setting determines how the payload that gets pushed should be handled. Available options are summarised below:

Step 9 If required you can set response handling options:

These options are summarised below:

Step 10 Save your changes.

Introduction

You can use the manual payload shape to define a static payload to be used for onward processing. For example, you might define an email template that gets pushed into an email connection, or you might want to test a process flow for a connector that's currently being by your development team.

Need to know

• The maximum number of characters for a single payload is 100k. Anything larger than this may cause the process flow to fail.

• Any text-based data format is supported (JSON, XML, CSV, plain)) however, keep in mind that subsequent shapes in the flow may only support JSON and XML.

Accessing manual payload shape settings

To view/update the settings for an existing manual payload shape, click the associated 'cog' icon:

Configuring settings for a new manual payload shape

To configure a manual payload shape, all you need to do is paste the required payload and save the shape for example:

Introduction

The Patchworks SFTP connector is used to work with data via files on SFTP servers in process flows. You might work purely in the SFTP environment (for example, copying/moving files between locations), or you might sync data from SFTP files into other connectors, or you might use a combination of both! For example, a process flow might be designed to:

1. Pull files from an SFTP server

2. Use the data in those files as the payload for subsequent processing (e.g. sync to Shopify)

3. Move files to a different SFTP server location

This guide explains the basics of configuring a connection shape with an SFTP connector.

About the Patchworks SFTP connector

Authentication

When you the Patchworks SFTP connector from the and then , you'll find that two authentication methods are available:

Endpoints

When you add a connection shape and select an SFTP connector, you will see that two endpoints are available:

Here:

• SFTP GET UserPass is used to retrieve files from the given server (i.e. to receive data)

• SFTP PUT UserPass is used to add/update files on the given server (i.e. to send data)

Configuring SFTP endpoints

Having selected either of the two SFTP endpoints, configuration options are displayed. The same options are used for both endpoints but in a different sequence, reflecting usage:

These fields are summarised below:

Using an {{original_filename}} variable

In this scenario, we can't know the literal name of the file(s) that the SFTP PUT UserPass endpoint will receive. So, by setting the path field to {{original_filename}}, we can refer back to the filename(s) from the previous SFTP connection step.

Using an {{original_path}} variable

The {{original_path}} variable is used to replicate the path from a previous SFTP connection step. It's most typically used with the SFTP PUT UserPass endpoint.

Using a {{current_filename}} variable

The {{current_path}} variable is used to reference the filename within the current SFTP connection step.

For example, you might want to move existing files to a different SFTP folder. The rename FTP command is an efficient way to do this - for example:

Here, we're using the FTP rename command to effectively move files - we're renaming with a different folder location, with current filenames:

rename:store1/completed_orders/{{current_filename}}

Creating SFTP folders dynamically based on timestamps

Script code

The following four lines of code should be added to your script:

SFTP connection shape path

The path in your SFTP connection shape should be set to:

How it works

Syncing SFTP data to another target connector

Much of the information above focuses on scenarios where you are working with files between different SFTP locations. However, another approach is to take the data in files from an SFTP server and sync that data into another Patchworks connector.

When a process flow includes a source connection for an SFTP server (using the SFTP GET UserPass endpoint) and a non-SFTP target connector (for example, Shopify), data in the retrieved file(s) is used as the incoming payload for the target connector.

More information

For information about working with regular expressions, please see the link below:

Introduction

If you're building multiple process flows with similar requirements for field mappings, you can the configuration for a map shape, and then that configuration into another map shape.

When a map shape configuration is exported, a JSON file is generated and saved (automatically) to the default download folder for your browser. All and associated are exported. You can then import this file to any other map shape within:

• The same process flow

• Other process flows for your company profile

• Other process flows for any of your linked company profiles

Exporting a mapping configuration

To export the configuration for a map shape, follow the steps below:

Step 1 Access the required process flow, then click the settings icon for the map shape that you want to export:

Step 2 Click the export map button:

Step 3 The configuration is exported and saved to your default downloads folder. The filename is always map.json.

Importing a mapping configuration

To import a mapping configuration into a map shape, follow the steps below:

Step 1 Access the required process flow, then click the settings icon for the map shape that you want to update:

Step 2 Click the import map button:

Step 3 Navigate to the downloaded map configuration file on your local drive, then select it.

Step 4 Having selected a valid mapping configuration file to import, the import completes immediately.

Introduction

This page provides guidance on using the to configure field mappings between two .

Configuring a new map shape

Step 1 Click the source endpoint option:

...source and target selection fields are displayed:

Step 2 Use source and target selection fields to choose the required connector instance and associated endpoints to be mapped - for example:

Step 3 Click the generate automatic mapping button:

...when prompted to confirm this operation, click generate mapping:

Step 4 Patchworks attempts to apply mappings between your given source and target automatically. A mapping rule is added for each source data field and, where possible, a matched target field - for example:

From here you can refine mappings as needed. You can:

Step 5

Toggle wrap input payload and wrap output payload options ON/OFF as required.

Where:

• wrap input payload ON. Wraps the incoming payload in an array [ ] ONLY for processing within the map shape.

• wrap output payload ON. Wraps the outgoing payload in an array [ ] ONLY for onward processing.

Click options below for payload examples showing how these options work in practice:

Step 6 Save changes.

Adding a new mapping rule

You can add as many new mapping rules as required to map data between source and target connections.

Specifying a mapping field manually

There may be times where you don't want to (or can't) use the payload fields dropdown select a field from your source/target data schema. In this case, you simply select the manual input field and enter the full schema path for the required field.

Changing display names or fields in an existing rule

You can change the display name and/or the field associated with the source or target for any mapping rule.

Adding a target mapping for a partial mapping rule

Mapping a source field to multiple targets

If required, you can map a source field to multiple target fields - for example, you might need to send a customer order number into two (or more) target fields.

Mapping multiple source fields to a single target

Sometimes it can be useful to map multiple source fields to a single target field. For example, you might have a target connection which expects a single field for 'full name', but a source connection with one field for 'first name' and another field for 'surname'.

Deleting a mapping rule

When you choose to delete a mapping rule, it's removed from the list immediately. However, the deletion is not permanent until you choose to save the mapping shape.

Introduction

Field transformations can be defined to change the value of a data field pulled from a source system before it is sent to its target. A transformation is comprised of one or more .

This page explains how to for a field mapping, and .

Adding a new transformation

To add a new transformation for a field mapping, you start by adding a new transformation and then build the required functions. To do this, follow the steps below:

Step 1 Access the required process flow and then edit the map shape to be updated with a transform:

Step 2 Find the mapping that you want to update, then click the transform icon (between source and target elements). For our example, we're going to add a prefix to the 'id' field:

Step 3 Click the add transform button:

Step 4 Use the select a function field to choose the type of function that you need to use (functions are organised by type):

Step 5 Depending on the type of function you select, additional fields are displayed for you to complete. Update these as required - for our example we're adding some text to be added as a prefix:

Step 6 Now we need to confirm which source field this transform field should be applied for - click the add field button:

Step 7 Select the required field:

Step 8 Accept your changes.

Step 9 Add more fields if necessary.

Step 10 When you're satisfied that all required fields have been added, accept changes and then save the shape settings.

Introduction

The array join transform function is used to join elements of an array as a string, with a user-defined delimiter. For example, you might have product data in an array:

...and need to convert these items to a string before pushing to a single destination field (with each item in the string delimited by a character of your choice):

Adding an array join transform

Step 1 In your process flow, access settings for your map shape:

Step 2 Select the add transform icon for the required mapping rule - for example:

Step 3 Click the add transform button:

Step 4 Click in the name field to access a list of all available transform functions, then select join from the array category:

Step 5 In the delimiter field, enter the character that you want to use to delimit each array field in the string:

Step 6 Click the add field button:

Step 7 Click in source fields and select the source field to be used for this transform:

Step 8 Accept your changes (twice).

Step 9 Save the transform. You'll notice that the transform icon is updated to indicate that a transform function is present for the mapping row - for example:

Transforms in this section

Introduction

Available transform functions are summarised in the following categories:

Array

Date & time

Number

Other

String

Introduction

The custom dynamic date transform function is used to set a target field to the current date and time, based on the date and time that the process flow runs. You can also define rounding and adjustments.

Adding a custom string transform

Step 1 In your process flow, access settings for your map shape:

Step 2 Select the add transform button for the required mapping rule - for example:

Step 3 Click the add transform button:

Step 4 Click in the name field to access a list of all available transform functions, then select custom dynamic date:

Step 5 Optionally, you can add adjustment settings - for example:

These options are summarised below:

Step 6 Accept your changes:

...then save the transformation:

Step 7 Now you can select a target field in the usual way - for example:

...then:

...then:

Step 8 Once your mapping is complete, the row should be displayed without a source field - for example:

From here you can save changes or add more mapping rules as needed. Next time the process flow runs, the custom dynamic date will be mapped to the given target field.