# Understanding how data flows through shapes ## Introduction Patchworks process flows are incredibly flexible. With a range of shapes for receiving, paginating, manipulating, batching, splitting, caching and sending data, you can build highly complex flows in a matter of a minutes. With this in mind, it's important to understand how data flows through shapes. ## How data flows In the simplest of scenarios, a process flow receives a single payload of unpaginated data and this flows all the way through to completion with no manipulation or batching - one payload is received, processed, and completed. However, if your incoming data is paginated and/or you introduce shapes capable of generating multiple payloads, it's important to understand how these pass through the flow. Essentially, any payloads that a shape outputs are added to a 'bucket' and it's that bucket that is then passed to the next shape. So, all payloads from one shape are passed to the next shape in the same context - they don't pass down the entire flow individually. {% hint style="info" %} If a 'pull' [connection shape](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape.md) is configured to use an endpoint that paginates data, the connection shape outputs each page in its own payload. {% endhint %} ## Demo The animation below shows how this works. {% embed url="" %}
Suggested view options For best results, view this animation in full screen and adjust settings to 2160p 4k:
## Shape timeouts and retries ### Timeouts All shapes (except the [connector shape](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape.md)) have a set timeout of 30 minutes. If processing is not completed within this time, the shape fails. The timeout for a connection shape is configurable via [connection shape settings](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape.md#configuring-settings-for-a-connection-shape). ### Retries With some exceptions (detailed below), a further three attempts will be made if a process flow shape fails. Exceptions are summarised below: | Shape | Default number of retries | | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- | | [Add to cache](/product-documentation/process-flows/building-process-flows/process-flow-shapes/advanced-shapes/cache/add-to-cache-shape.md) | 1 | | [Connector](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape.md) | 1 | | [Manual payload](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/manual-payload-shape.md) | 1 | | [Split](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/split-shape.md) | 1 | | [De-dupe](/product-documentation/process-flows/building-process-flows/process-flow-shapes/advanced-shapes/de-dupe-shape.md) | 1 | #### Connector retries & operations You can change the default number of retries for any connector step in the [connector shape settings](/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape.md#accessing-connector-shape-settings) (minimum `0`, maximum `2`). However, it's important to be aware of how this impacts your [operations count](/product-documentation/company-management/company-insights/company-insights-overview/about-operations.md). Each connector request to send or receive data correlates to one operation. So, if a connector request fails and retries, the result is one additional operation for each retry. You can see how this works in the table below:
Rolling operation count
Initial request1
Retry 12
Retry 23
{% hint style="warning" %} If a process flow runs on an automatic schedule and is left [unchecked](/product-documentation/process-flows/error-reporting-and-exception-handling/run-logs-and-queue.md#accessing-your-run-logs-and-queue-page) with failing connector requests, this will impact your operations count. {% endhint %} {% hint style="info" %} For more information about operations, please see [About operations](/product-documentation/company-management/company-insights/company-insights-overview/about-operations.md). {% endhint %} ## Payload size A single incoming payload for [any process flow shape](https://open.gitbook.com/~site/site_dIV1g/~/revisions/9dTvdvRJIRVZQXuBMMnJ/process-flows/building-process-flows/process-flow-shapes) should not exceed 500MB. We recommend processing multiple, smaller payloads rather than one single payload (1000 x 0.5MB payloads are more efficient than 1 x 500MB payload!). For payloads up to 500MB, consider adding a[ flow control shape](https://open.gitbook.com/~site/site_dIV1g/~/revisions/9dTvdvRJIRVZQXuBMMnJ/process-flows/building-process-flows/process-flow-shapes/standard-shapes/flow-control-shape) to batch data into multiple, smaller payloads. Payloads exceeding 500MB should be batched at source. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/understanding-how-data-flows-through-shapes.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.