# Flow control shape

## Introduction

The **flow control** shape can be used for cases where you're pulling lots of records from a source connection, but your target connection needs to receive data in small batches. Two common use cases for this shape are:

* A target system can only accept items one at a time
* A target system has a maximum number of records that can be added/updated at one time

The **flow control** shape takes all received items, splits them into batches of your given number, and sends these batches into the target connection.  &#x20;

{% hint style="info" %}
A single incoming payload for [any process flow shape](https://open.gitbook.com/~site/site_dIV1g/~/revisions/9dTvdvRJIRVZQXuBMMnJ/process-flows/building-process-flows/process-flow-shapes) should not exceed 500MB.

We recommend processing multiple, smaller payloads rather than one single payload (1000 x 0.5MB payloads are more efficient than 1 x 500MB payload!).

For payloads up to 500MB, consider adding a[ flow control shape](https://open.gitbook.com/~site/site_dIV1g/~/revisions/9dTvdvRJIRVZQXuBMMnJ/process-flows/building-process-flows/process-flow-shapes/standard-shapes/flow-control-shape) to batch data into multiple, smaller payloads. Payloads exceeding 500MB should be batched at source.
{% endhint %}

## Demo

{% embed url="<https://www.youtube.com/watch?v=NiVxv6lV1zU>" %}

## Adding & configuring a flow control shape

**Step 1**\
In your process flow, add the **flow control** shape in the usual way:

<div align="left"><figure><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FKffoH72gUJzNTAxiQVKm%2Fflow%20control%201.png?alt=media&#x26;token=5442f7f9-1d17-490c-a3ec-94a79dce59a2" alt="" width="375"><figcaption></figcaption></figure></div>

**Step 2**\
Select a **source integration** and **endpoint** to determine where the incoming payload to be split originates - for example:

<div align="left"><figure><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FiQzj0ikgix262lMbjZMU%2Fflow%20control%20settings%20-%20endpoints%20selected.png?alt=media&#x26;token=2c38e090-da3a-4cb7-9c09-03409bc7ce82" alt="" width="355"><figcaption></figcaption></figure></div>

**Step 3**\
Move down to the **batch level** field and select that data element that you are putting into batches. For example :

<div align="left"><figure><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2F89d8ictz9gRjjShIORWT%2Fflow%20control%201.png?alt=media&#x26;token=6b464d76-5db0-4237-a4d2-3b074d30f9b3" alt="" width="357"><figcaption></figcaption></figure></div>

{% hint style="info" %}
The data structure in this dropdown field is pulled from schema associated with the source. If your data is received from a non-connector source (e.g. manual payload, API, webhook, etc.) then you can toggle ON the **manual input** option and enter the data path manually).
{% endhint %}

**Step 4**\
In the **batch size** field, enter the number of items to be included in each batch. For example:

<div align="left"><figure><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FDvt1Bd6GuXvbwIi1US23%2Fflow%20control%203.png?alt=media&#x26;token=7c09ceb9-854f-4f20-8a81-202d1542654d" alt="" width="356"><figcaption></figcaption></figure></div>

**Step 5**\
By default, the **payload format** is **auto**-detected but you can set a specific format here if you prefer:

<div align="left"><figure><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FfWXEvLQSeMVefRlsOGGB%2Fflow%20control%202.png?alt=media&#x26;token=ff63a0a8-fa48-4ba5-8315-b21ee00767c3" alt="" width="352"><figcaption></figcaption></figure></div>

**Step 6**

If you're creating batches of one record, you can toggle ON the  `Do not wrap single records in an array` option if you want the output to be this:

```json
{
    "field": "value",
    "field": "value",
    "field": "value"
}
```

...rather than this:

```json
[{
    "field": "value",
    "field": "value",
    "field": "value"
}]
```

**Step 7**\
Save the shape. Now when you run this process flow, data will be split into batches of your given size.&#x20;

{% hint style="info" %}
If you check the payload for the flow control step after it has run, you'll see that there's one payload for every batch created. For example:

<img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2F4w0Jv9judIGeSMV99w33%2Fflow%20control%20payloads%20example.png?alt=media&#x26;token=c8042b0a-f30a-4db0-96f3-7002a0c318bf" alt="" data-size="original">
{% endhint %}
