# Configuring SFTP connections

## Introduction

The Patchworks SFTP connector is used to work with data via files on SFTP servers in process flows. You might work purely in the SFTP environment (for example, copying/moving files between locations), or you might sync data from SFTP files into other connectors, or you might use a combination of both! For example, a process flow might be designed to:

1. Pull files from an SFTP server&#x20;
2. Use the data in those files as the payload for subsequent processing (e.g. sync to Shopify)
3. Move files to a different SFTP server location

<details>

<summary><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FpLMO12yvTCxi9PorCt53%2Ficons8-film.svg?alt=media&#x26;token=e59864bc-2d97-4f98-966c-a083c18e60db" alt="" data-size="line"> Show me</summary>

![](https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2F5m4Nj1UOJ90tf2bEZv7R%2Fsftp%20demo%202.gif?alt=media\&token=59c55aed-ed71-438b-a074-d2fbba1e23d6)

</details>

This guide explains the basics of configuring a connection shape with an SFTP connector.

{% hint style="info" %}
Guidance on this page is for SFTP connections however, they also apply for FTP.
{% endhint %}

## About the Patchworks SFTP connector

### Authentication

When you [install](https://doc.wearepatchworks.com/product-documentation/process-flows/cross-reference-lookups/installing-cross-reference-lookups) the Patchworks SFTP connector from the [Patchworks marketplace](https://doc.wearepatchworks.com/product-documentation/marketplace/the-patchworks-marketplace) and then [add an instance](https://doc.wearepatchworks.com/product-documentation/connectors-and-instances/working-with-instances/adding-an-instance), you'll find that two authentication methods are available:

<table><thead><tr><th width="211.5">Auth method</th><th>Summary</th></tr></thead><tbody><tr><td>User pass</td><td>The instance is authenticated by providing a username and password for the SFTP server.</td></tr><tr><td>Key pass</td><td>The instance is authenticated by providing a private key (RSA <code>.pem</code> format) for the SFTP server.</td></tr></tbody></table>

Further information on these authentication methods can be found on our [SFTP (prebuilt connector)](https://doc.wearepatchworks.com/product-documentation/connectors-and-instances/patchworks-connectors/sftp-prebuilt-connector) page.

### Endpoints

When you add a connection shape and select an SFTP connector, you will see that two endpoints are available:

<div align="left"><figure><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FiEgat43ynCBRKURqQaVm%2Fsftp%20endpoints.png?alt=media&#x26;token=1ab15ae0-23cd-47ef-81b1-6e707a9a74d5" alt="" width="351"><figcaption></figcaption></figure></div>

Here:

* `SFTP GET UserPass` is used to retrieve files from the given server (i.e. to **receive** data)
* `SFTP PUT UserPass` is used to add/update files on the given server (i.e. to **send** data)

{% hint style="info" %}
You may notice that the `PUT UserPass` endpoint has a `GET`  HTTP method - that's because it's not actually used for SFTP. All we're actually doing here is retrieving host information from the connector instance - you'll set the FTP action later in the endpoint configuration, via an `ftp command` settings.
{% endhint %}

{% hint style="warning" %}
[Pre](https://doc.wearepatchworks.com/product-documentation/developer-hub/connector-builder/building-your-own-connector/4-endpoints/endpoint-options/pre-request-script), [Post](https://doc.wearepatchworks.com/product-documentation/developer-hub/connector-builder/building-your-own-connector/4-endpoints/endpoint-options/post-request-script) and [Response](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape/using-connector-shape-response-scripts) scripts are not supported for SFTP endpoints. Any script requirements should be handled via a [script shape](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/advanced-shapes/script-shape) in the flow.
{% endhint %}

## Configuring SFTP endpoints

Having selected either of the two SFTP endpoints, configuration options are displayed. The same options are used for both endpoints but in a different sequence, reflecting usage:

<div align="left"><figure><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FrjgL4PCT6ErsMN0fi2Bk%2Fsftp%20get%20and%20put%20settings.png?alt=media&#x26;token=423c006c-3a7e-4a93-b1bf-a2ed061bfde7" alt="" width="563"><figcaption></figcaption></figure></div>

These fields are summarised below:

<table><thead><tr><th width="150">Option</th><th>Summary</th></tr></thead><tbody><tr><td>FTP command</td><td>A valid FTP command is expected at the start of this field (e.g. <code>get</code>, <code>put</code>, <code>rename</code>, etc.). If required, qualifying path/filename information can follow a given command. </td></tr><tr><td>Root</td><td>This field is only needed if you are specifying a <strong>regular expression</strong> in the subsequent <code>path</code> field. <br><br>If you are NOT defining the <code>path</code> field as a regular expression, the <code>root</code> field isn't important - you can leave it set to <code>/</code>.<br><br>If you are ARE defining the <code>path</code> field as a regular expression, enter a root path that reflects the expected file location as closely as possible - this will optimise performance for expression matching.<br><br>For example, suppose the files that we want to process are in the following SFTP folder: <br><code>/orders/store/year/pending</code> and that our specified <code>path</code> contains a regular expression to retrieve all files for <code>store 1</code> for the current day in 2023. In this case our <code>root</code> would be defined as: <br><code>orders/store1/2023/pending</code><br><br>In this way, any regular expression to match for the <code>path</code> will start in the relevant (<code>2023</code> )folder - rather than checking folders and subfolders for all stores and all years.</td></tr><tr><td>Path</td><td><p>If the name of the file that you want to target is static and known, enter the full path to it here - for example:</p><p><code>store1/2023/pending/20230728.json</code></p><p></p><p><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FKvJZhQaEKfDOWvbHsbYe%2Ftip%20icon.svg?alt=media&#x26;token=975b37c4-e5e3-462c-a32f-cc72a5461cac" alt="" data-size="line"> When specifying a path to a given folder in this way, you don't need a <code>/</code> at the start or at the end.</p><p><br>If the name is variable and therefore unknown, you can specify a <strong>regular expression</strong> as the <code>path</code>. In this case, you enter the required regular expression here, and ensure that the <code>root</code> field contains a path to the relevant folder (see above). </p><p></p></td></tr><tr><td>Original filename</td><td>This field is not currently used. For information about working with original filenames please see the <a href="#using-an-original_filename-variable">Using an {{original_filename}} variable</a> section below.</td></tr><tr><td>Original path</td><td>This field is not currently used. For information about working with original paths please see the <a href="#using-an-original_path-variable">Using an {{original_path}} variable</a> section below.</td></tr></tbody></table>

## Valid FTP commands

Please refer to the [Valid FTP commands](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape/configuring-sftp-connections/valid-ftp-commands) page.

## Using an {{original\_filename}} variable

If you're processing files between SFTP server locations, the `{{original_filename}}` variable is used to reference filenames from a **previous SFTP connection step**. It's most typically used with the `SFTP PUT UserPass` endpoint. \
\
This handles cases where you're taking action with files/data processed by a previous [connection shape](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape) which is configured to use the `SFTP GET UserPass` endpoint and retrieve files matching a regular expression `path`.&#x20;

In this scenario, we can't know the literal name of the file(s) that the `SFTP PUT UserPass` endpoint will receive. So, by setting the `path` field to `{{original_filename}}`, we can refer back to the filename(s) from the previous SFTP connection step.

<details>

<summary><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FuKTXJEY34CVnak6PxrOz%2Fexample%20icon%202.svg?alt=media&#x26;token=7c8af2c5-9519-4757-bea9-172569a023bd" alt="" data-size="line"> Example</summary>

The sample process flow below shows two connection shapes that are configured with SFTP endpoints - the **first** is to `get` files and the **second** is to `put` files:\
\ <img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2F94AovtnK1EhLPwcfccA4%2Fsftp%20main%20eg%201.png?alt=media&#x26;token=40729a07-e05e-4409-9a75-f92622fec276" alt="" data-size="original">\
\
If we look at settings for the **first** SFTP connection, we can see that it's configured to `get` files matching a regular expression, in a `pending` folder:\
\
![](https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FjTmtXJf8qgoHuiC1vgCp%2Fsftp%20main%20eg%202.png?alt=media\&token=bf47bfde-8fa4-477f-a50e-b45645d43a4b)\
\
The regular expression is explained below:

* the `/` at the start of the string denotes the start of the regular expression
* the `^` means that the first part of the line (i.e. filename) must start with the literal string `orders`
* `[0-9]*` matches any sequence of digits
* `\.` is an escaped full stop, so a literal full stop is used instead of the special meaning that a full stop has in regular expressions
* `json` is a literal string to be matched
* the `/` at the end of the string denotes the end of the regular expression

It will match the following files:

![](https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2F3d4YHWMQmty4m2XTc5ut%2Feg1.png?alt=media\&token=8be98731-0dc3-4c0b-84c8-7b56133e0e5c)

These are:

* orders20230717.json
* orders20230718.json
* orders20230719.json

Now if we check settings for the **second** SFTP connection, we can see that it's configured to `put` files into a `ready` folder. By specifying `{{original_filename}}` in the `path` field, we are referencing filenames from the **first** SFTP connection.

![](https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FKzmVAgZWGetzpbImxjQi%2Fsftp%20main%20eg%203.png?alt=media\&token=b45b76bf-eb90-4704-9491-2899c5d686a4)

So, the three files retrieved from the **first** SFTP connection step (from the `pending` folder) are copied to the `ready` folder in the **second** SFTP connection step.

</details>

## Using an {{original\_path}} variable

The `{{original_path}}` variable is used to replicate the path from a **previous** **SFTP connection step**. It's most typically used with the `SFTP PUT UserPass` endpoint.&#x20;

This handles cases where you're taking action with files/data processed by a previous [connection shape](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape) which is configured to use the `SFTP GET UserPass` endpoint to retrieve files matching a regular expression `path` and you want to replicate the source path in the target location.

<details>

<summary><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FuKTXJEY34CVnak6PxrOz%2Fexample%20icon%202.svg?alt=media&#x26;token=7c8af2c5-9519-4757-bea9-172569a023bd" alt="" data-size="line"> Example</summary>

The sample process flow below shows two connection shapes that are configured with SFTP endpoints - the **first** is to `get` files and the **second** is to `put` files:\
\ <img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FlOMzP3pMnCd7QaDj4VPu%2Fsftp%20path%20eg%201.png?alt=media&#x26;token=2537df34-d13e-4437-9b04-c9e6d72ec5f6" alt="" data-size="original">\
\
Our aim is to copy files retrieved from an FTP location in the first connection step, to a second FTP location, using the same folder structure as the source.

If we look at settings for the **first** SFTP connection, we can see that it's configured to `get` files matching a regular expression, in a `store1` folder:\
\
![](https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2Fm6Ae6W6cgpm5Lf6dr6f3%2Fsftp%20path%20eg%202.png?alt=media\&token=ab2c5f57-9ad4-4892-874c-0c0b3a6d6eb0)\
\
The path is added as a regular expression, explained below:

* the `/` at the start of the string denotes the start of the regular expression
* `orders\/2023\/pending\/` is a literal string for the folder structure, with escaped forward slashes &#x20;
* `orders` is a literal string
* `[0-9]*` matches any sequence of digits
* `\.` is an escaped full stop, so a literal full stop is used instead of the special meaning that a full stop has in regular expressions
* `json` is a literal string to be matched
* the `/` at the end of the string denotes the end of the regular expression

It will match the following files:

![](https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2F3d4YHWMQmty4m2XTc5ut%2Feg1.png?alt=media\&token=8be98731-0dc3-4c0b-84c8-7b56133e0e5c)

These are:

* orders20230717.json
* orders20230718.json
* orders20230719.json

Now if we check settings for the **second** SFTP connection, we can see that it's configured to `put` files into a `path` specified as `{{original_path}}`:

![](https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FCxUGPTHlWVKZyxssE6VE%2Fsftp%20path%20eg%203.png?alt=media\&token=8674261d-47d3-4451-8e5d-60af1702a662)

So, the three files retrieved from the **first** SFTP connection step are copied to the `ready` folder in the **second** SFTP connection step and the source folder structure is replicated:

<img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FT5BMG6L7RWkEWJysKpnH%2Fsftp%20path%20eg%204.png?alt=media&#x26;token=6456b5b9-f6e5-4843-a020-3df5e6640eb2" alt="" data-size="original">

</details>

## Using a {{current\_filename}} variable

The `{{current_path}}` variable is used to reference the filename within the **current SFTP connection step**. It's particularly useful when moving files - for example:

`get_and_move:store1/completed_orders/{{current_filename}}`

## Creating SFTP folders dynamically based on timestamps

A fairly common requirement is to create folders on an SFTP server which are named according to the current date. This can be achieved using a [custom script](https://doc.wearepatchworks.com/product-documentation/developer-hub/custom-scripting), as summarised below.

#### Script code

The following four lines of code should be added to your script:

{% code lineNumbers="true" %}

```php
$timestamp = round(microtime(true) * 100);
$meta = $data['meta'];
$meta['original_filename'] = 'FIXED_TEXT' . '_' . $timestamp . '.xml';
$data['meta'] = $meta;
```

{% endcode %}

{% hint style="info" %}
Our example is PHP - you should change as needed for your preferred language.&#x20;
{% endhint %}

#### SFTP connection shape path

The path in your SFTP connection shape should be set to:

{% code lineNumbers="true" %}

```
{{original_filename}}
```

{% endcode %}

#### How it works

The `data` object in the [script shape](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/advanced-shapes/script-shape) contains three items: `payload`, `meta`, and `variables`.

Our [script code](#script-code) creates a timestamp, puts it in to the `meta`, and then puts the `meta` into the `data`.\
The **SFTP shape** always checks if there is an `original_filename` key in the `meta` and if one exists, this is used.

## Syncing SFTP data to another target connector

Much of the information above focuses on scenarios where you are working with files between different SFTP locations. However, another approach is to take the data in files from an SFTP server and sync that data into another Patchworks connector.

When a process flow includes a source connection for an SFTP server (using the `SFTP GET UserPass` endpoint) and a non-SFTP target connector (for example, Shopify), data in the retrieved file(s) is used as the incoming payload for the target connector.

{% hint style="warning" %}
If multiple files are retrieved from the SFTP server (because the required path in settings for the SFTP connector is defined as a regular expression which matches more than one file), then **each matched file is put through subsequent steps in the process flow one at a time, in turn**. So, if you retrieve five files from the source SFTP connection, the process flow will run five times.
{% endhint %}

## More information

For information about working with regular expressions, please see the link below:

{% embed url="<https://www.w3schools.com/php/php_regex.asp>" %}
