Understanding how data flows through shapes

Introduction

Patchworks process flows are incredibly flexible. With a range of shapes for receiving, paginating, manipulating, batching, splitting, caching and sending data, you can build highly complex flows in a matter of a minutes.

With this in mind, it's important to understand how data flows through shapes.

How data flows

In the simplest of scenarios, a process flow receives a single payload of unpaginated data and this flows all the way through to completion with no manipulation or batching - one payload is received, processed, and completed.

However, if your incoming data is paginated and/or you introduce shapes capable of generating multiple payloads, it's important to understand how these pass through the flow. Essentially, any payloads that a shape outputs are added to a 'bucket' and it's that bucket that is then passed to the next shape.

So, all payloads from one shape are passed to the next shape in the same context - they don't pass down the entire flow individually.

If a 'pull' connection shape is configured to use an endpoint that paginates data, the connection shape outputs each page in its own payload.

Demo

The animation below shows how this works.

Suggested view options

For best results, view this animation in full screen and adjust settings to 2160p 4k:

Shape timeouts and retries

Timeouts

All shapes (except the connector shape) have a set timeout of 30 minutes. If processing is not completed within this time, the shape fails.

The timeout for a connection shape is configurable via connection shape settings.

Retries

With some exceptions (detailed below), a further three attempts will be made if a process flow shape fails. Exceptions are summarised below:

Shape	Default number of retries
Add to cache	1
Connector	1
Manual payload	1
Split	1
De-dupe	1

Connector retries & operations

You can change the default number of retries for any connector step in the connector shape settings (minimum 0, maximum 2). However, it's important to be aware of how this impacts your operations count.

Each connector request to send or receive data correlates to one operation. So, if a connector request fails and retries, the result is one additional operation for each retry. You can see how this works in the table below:

	Rolling operation count
Initial request	1
Retry 1	2
Retry 2	3

If a process flow runs on an automatic schedule and is left unchecked with failing connector requests, this will impact your operations count.

For more information about operations, please see About operations.

Payload size

A single incoming payload for any process flow shape should not exceed 500MB.

We recommend processing multiple, smaller payloads rather than one single payload (1000 x 0.5MB payloads are more efficient than 1 x 500MB payload!).

For payloads up to 500MB, consider adding a flow control shape to batch data into multiple, smaller payloads. Payloads exceeding 500MB should be batched at source.