Configuring SFTP connections

Introduction

The Patchworks SFTP connector is used to work with data via files on SFTP servers in process flows. You might work purely in the SFTP environment (for example, copying/moving files between locations), or you might sync data from SFTP files into other connectors, or you might use a combination of both! For example, a process flow might be designed to:

  1. Pull files from an SFTP server

  2. Use the data in those files as the payload for subsequent processing (e.g. sync to Shopify)

  3. Move files to a different SFTP server location

This guide explains the basics of configuring a connection shape with an SFTP connector.

Guidance on this page is for SFTP connections however, they also apply for FTP.

About the Patchworks SFTP connector

Authentication

When you install the Patchworks SFTP connector from the Patchworks marketplace and then add an instance, you'll find that two authentication methods are available:

Further information on these authentication methods can be found on our SFTP (prebuilt connector) page.

Endpoints

When you add a connection shape and select an SFTP connector, you will see that two endpoints are available:

Here:

  • SFTP GET UserPass is used to retrieve files from the given server (i.e. to receive data)

  • SFTP PUT UserPass is used to add/update files on the given server (i.e. to send data)

You may notice that the PUT UserPass endpoint has a GET HTTP method - that's because it's not actually used for SFTP. All we're actually doing here is retrieving host information from the connector instance - you'll set the FTP action later in the endpoint configuration, via an ftp command settings.

Configuring SFTP endpoints

Having selected either of the two SFTP endpoints, configuration options are displayed. The same options are used for both endpoints but in a different sequence, reflecting usage:

These fields are summarised below:

Using an {{original_filename}} variable

If you're processing files between SFTP server locations, the {{original_filename}} variable is used to reference filenames from a previous SFTP connection step. It's most typically used with the SFTP PUT UserPass endpoint. This handles cases where you're taking action with files/data processed by a previous connection shape which is configured to use the SFTP GET UserPass endpoint and retrieve files matching a regular expression path.

In this scenario, we can't know the literal name of the file(s) that the SFTP PUT UserPass endpoint will receive. So, by setting the path field to {{original_filename}}, we can refer back to the filename(s) from the previous SFTP connection step.

Using an {{original_path}} variable

The {{original_path}} variable is used to replicate the path from a previous SFTP connection step. It's most typically used with the SFTP PUT UserPass endpoint.

This handles cases where you're taking action with files/data processed by a previous connection shape which is configured to use the SFTP GET UserPass endpoint to retrieve files matching a regular expression path and you want to replicate the source path in the target location.

Using a {{current_filename}} variable

The {{current_path}} variable is used to reference the filename within the current SFTP connection step.

For example, you might want to move existing files to a different SFTP folder. The rename FTP command is an efficient way to do this - for example:

Here, we're using the FTP rename command to effectively move files - we're renaming with a different folder location, with current filenames:

rename:store1/completed_orders/{{current_filename}}

Creating SFTP folders dynamically based on timestamps

A fairly common requirement is to create folders on an SFTP server which are named according to the current date. This can be achieved using a custom script, as summarised below.

Script code

The following four lines of code should be added to your script:

$timestamp = round(microtime(true) * 100);
$meta = $data['meta'];
$meta['original_filename'] = 'FIXED_TEXT' . '_' . $timestamp . '.xml';
$data['meta'] = $meta;

Our example is PHP - you should change as needed for your preferred language.

SFTP connection shape path

The path in your SFTP connection shape should be set to:

{{original_filename}}

How it works

The data object in the script shape contains three items: payload, meta, and variables.

Our script code creates a timestamp, puts it in to the meta, and then puts the meta into the data. The SFTP shape always checks if there is an original_filename key in the meta and if one exists, this is used.

Syncing SFTP data to another target connector

Much of the information above focuses on scenarios where you are working with files between different SFTP locations. However, another approach is to take the data in files from an SFTP server and sync that data into another Patchworks connector.

When a process flow includes a source connection for an SFTP server (using the SFTP GET UserPass endpoint) and a non-SFTP target connector (for example, Shopify), data in the retrieved file(s) is used as the incoming payload for the target connector.

If multiple files are retrieved from the SFTP server (because the required path in settings for the SFTP connector is defined as a regular expression which matches more than one file), then each matched file is put through subsequent steps in the process flow one at a time, in turn. So, if you retrieve five files from the source SFTP connection, the process flow will run five times.

More information

For information about working with regular expressions, please see the link below:

Last updated