> For the complete documentation index, see [llms.txt](https://doc.wearepatchworks.com/product-documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/understanding-how-data-flows-through-shapes.md).

# Understanding how data flows through shapes

## Introduction

Patchworks process flows are incredibly flexible. With a range of shapes for receiving, paginating, manipulating, batching, splitting, caching and sending data, you can build highly complex flows in a matter of a minutes.

With this in mind, it's important to understand how data flows through shapes.

## How data flows

In the simplest of scenarios, a process flow receives a single payload of unpaginated data and this flows all the way through to completion with no manipulation or batching - one payload is received, processed, and completed.

However, if your incoming data is paginated and/or you introduce shapes capable of generating multiple payloads, it's important to understand how these pass through the flow. Essentially, any payloads that a shape outputs are added to a 'bucket' and it's that bucket that is then passed to the next shape.

So, all payloads from one shape are passed to the next shape in the same context - they don't pass down the entire flow individually.

{% hint style="info" %}
If a 'pull' [connection shape](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape.md) is configured to use an endpoint that paginates data, the connection shape outputs each page in its own payload.
{% endhint %}

## Demo

The animation below shows how this works.

{% embed url="<https://youtu.be/-kEHeihSSfY>" %}

<details>

<summary><img src="/files/i4lNQf87gFXIoy2ODqxD" alt="" data-size="line"> Suggested view options</summary>

For best results, view this animation in full screen and adjust settings to 2160p 4k:

<img src="/files/d2x9JphTjy938LlDfplJ" alt="" data-size="original">

</details>

## Shape timeouts and retries

### Timeouts

All shapes (except the [connector shape](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape.md)) have a set timeout of 30 minutes. If processing is not completed within this time, the shape fails.

The timeout for a connection shape is configurable via [connection shape settings](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape.md#configuring-settings-for-a-connection-shape).

### Retries

With some exceptions (detailed below), a further three attempts will be made if a process flow shape fails. Exceptions are summarised below:

| Shape                                                                                                                                       | Default number of retries |
| ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| [Add to cache](/product-documentation/process-flows/building-process-flows/process-flow-shapes/advanced-shapes/cache/add-to-cache-shape.md) | 1                         |
| [Connector](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape.md)             | 1                         |
| [Manual payload](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/manual-payload-shape.md)   | 1                         |
| [Split](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/split-shape.md)                     | 1                         |
| [De-dupe](/product-documentation/process-flows/building-process-flows/process-flow-shapes/advanced-shapes/de-dupe-shape.md)                 | 1                         |

#### Connector retries & operations

You can change the default number of retries for any connector step in the [connector shape settings](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape.md#accessing-connector-shape-settings) (minimum `0`, maximum `2`). However, it's important to be aware of how this impacts your [operations count](/product-documentation/company-management/company-insights/company-insights-overview/about-operations.md).

Each connector request to send or receive data correlates to one operation. So, if a connector request fails and retries, the result is one additional operation for each retry. You can see how this works in the table below:

<table><thead><tr><th width="227.28515625"></th><th data-type="number">Rolling operation count</th></tr></thead><tbody><tr><td>Initial request</td><td>1</td></tr><tr><td>Retry 1</td><td>2</td></tr><tr><td>Retry 2</td><td>3</td></tr></tbody></table>

{% hint style="warning" %}
If a process flow runs on an automatic schedule and is left [unchecked](/product-documentation/process-flows/error-reporting-and-exception-handling/run-logs-and-queue.md#accessing-your-run-logs-and-queue-page) with failing connector requests, this will impact your operations count.
{% endhint %}

{% hint style="info" %}
For more information about operations, please see [About operations](/product-documentation/company-management/company-insights/company-insights-overview/about-operations.md).
{% endhint %}

## Payload size

A single incoming payload for [any process flow shape](https://open.gitbook.com/~site/site_dIV1g/~/revisions/9dTvdvRJIRVZQXuBMMnJ/process-flows/building-process-flows/process-flow-shapes) should not exceed 500MB.

We recommend processing multiple, smaller payloads rather than one single payload (1000 x 0.5MB payloads are more efficient than 1 x 500MB payload!).

For payloads up to 500MB, consider adding a[ flow control shape](https://open.gitbook.com/~site/site_dIV1g/~/revisions/9dTvdvRJIRVZQXuBMMnJ/process-flows/building-process-flows/process-flow-shapes/standard-shapes/flow-control-shape) to batch data into multiple, smaller payloads. Payloads exceeding 500MB should be batched at source.