get_and_move (FTP command)

Overview

The get_and_move command retrieves the file content, loads it into the flow as a payload, and then moves the file to the specified directory on the remote server.

Need to know

• You can use the get_and_move command to target specific files in one step. In this scenario, the file content is loaded into the flow as a payload.

• You can use the get_and_move command to get & move multiple files (using regex in a filter) or all files in a directory, but this requires multiple steps rather than a single get_and_move operation. In this scenario, content from files is NOT loaded into the flow; instead, payloads will contain the filename.

• Regular expressions are supported when targeting single files to get_and_move.

• When getting and moving all files from a directory, subfolders are not included.

Connection settings

When configuring an SFTP connector, three fields should be updated:

FTP command	Root	Path
FTP command followed by the target directory - i.e. where should the file(s) be moved?	The common root to source and target directories.	The source directory and file(s).

Examples

• Get and move a specific file

• Get & move multiple files

Getting & moving a specific file

• Scenario

• The steps

Scenario

Our process flow is configured as follows:

In this flow, we need to retrieve content from a file named orders.json which is located in /myfiles/folderA on the remote server:

Then we want to move this file to /myfiles/folderA and keep the same filename.

The steps

Connector settings

Our SFTP shape is configured as follows:

• Start looking for the file to get from the root, which is defined as: /myfiles/

• Check the path for the file to get, which is defined as: folderA/orders.json

• Having loaded content from this file, move it to the path (from the root) which is specified immediately after the ftp command. This is defined as: get_and_move:folderB/{{current_filename}}

Payload & SFTP server

When the process flow is run, the payload for the SFTP shape shows the content retrieved from /myfiles/folderA/orders.json :

On our remote server, the file is gone from /myfiles/folderA/:

And now it can be found in /myfiles/folderB/:

Getting & moving multiple files

• Scenario

• The steps

Scenario

Our process flow is configured as below:

In this flow, we need to get and move all files in /myfiles/folderF which start with 'old', on the remote server: We are moving these files to /myfiles/folderG , which is currently empty: We are keeping the same filenames.

The steps

Connector settings for SFTP shape 1

Our first SFTP connector step is configured with a list command, as follows:

• Use the SFTP GET user pass endpoint.

• Use list as the FTP command.

• Look for files in the root, which is defined as: /myfiles/folderF

• Since there's no specific file to target, we leave the path empty

When the flow runs, the SFTP shape outputs a single payload which contains all file names found in /myfiles/folderF, as an array:

Flow control settings

We use a flow control step to extract each file name into its own payload:

Here we create batches of 1, so we get one payload per file name. When the flow runs, this shape outputs a multiple payloads, each containing a single filename. For example:

Filter shape settings

We use a filter shape to extract only the file names that we want to process. To achieve this, we define a single filter rule, as below:

Here we define the field name as 0, so the first value in our payload (bearing in mind we only have one field in each payload).

We set the filter type to string and the operator to regex, then provide our regular expression. Our regular expression is set to /^old.*/i, so only files starting with 'old' will be extracted for onward processing. When the flow runs, this shape outputs three payloads:

Each one contains a single filename. For example:

Connector settings for SFTP shape 2

Our final SFTP connector step is configured to get and move files with the names received from the previous filter shape:

• Since we are updating the remote server (as opposed to retrieving files) the SFTP PUT user pass endpoint is selected.

• The root is defined as /myfiles/, which is the common root for both source (folderF) and target (folderG) directories.

• The path defines which files are retrieved and moved. It's set as folderF/[[payload.0]] which means: look in folderF (from the root) for a filename resolved from the first value in the incoming payload. Keep in mind that this step repeats for each incoming payload from the previous flow control step - i.e. for each file.

• The ftp command includes the get_and_move command, immediately followed by our target directory: copy:folderG/{{current_filename}} .

When the flow runs, this shape outputs a payload for each processed file, each one containing the file name. For example:

On our remote server, all processed files are removed from /myfiles/folderF/:

And now they can be found in /myfiles/folderG/:

In this example we moved selected files from one folder to another. If we needed to copy ALL files, we could simply remove the filter shape.