Flow control shape

Introduction

The flow control shape can be used for cases where you're pulling lots of records from a source connection, but your target connection needs to receive data in small batches. Two common use cases for this shape are:

• A target system can only accept items one at a time

• A target system has a maximum number of records that can be added/updated at one time

The flow control shape takes all received items, splits them into batches of your given number, and sends these batches into the target connection.

A single incoming payload for any process flow shape should not exceed 500MB.

We recommend processing multiple, smaller payloads rather than one single payload (1000 x 0.5MB payloads are more efficient than 1 x 500MB payload!).

For payloads up to 500MB, consider adding a flow control shape to batch data into multiple, smaller payloads. Payloads exceeding 500MB should be batched at source.

Demo

Adding & configuring a flow control shape

Step 1 In your process flow, add the flow control shape in the usual way:

Step 2 Select a source integration and endpoint to determine where the incoming payload to be split originates - for example:

Step 3 Move down to the batch level field and select that data element that you are putting into batches. For example :

The data structure in this dropdown field is pulled from schema associated with the source. If your data is received from a non-connector source (e.g. manual payload, API, webhook, etc.) then you can toggle ON the manual input option and enter the data path manually).

Step 4 In the batch size field, enter the number of items to be included in each batch. For example:

Step 5 By default, the payload format is auto-detected but you can set a specific format here if you prefer:

Step 6

If you're creating batches of one record, you can toggle ON the Do not wrap single records in an array option if you want the output to be this:

{
    "field": "value",
    "field": "value",
    "field": "value"
}

...rather than this:

[{
    "field": "value",
    "field": "value",
    "field": "value"
}]

Step 7 Save the shape. Now when you run this process flow, data will be split into batches of your given size.

If you check the payload for the flow control step after it has run, you'll see that there's one payload for every batch created. For example: