Understanding how data flows through shapes

Introduction

Patchworks process flows are incredibly flexible. With a range of shapes for receiving, paginating, manipulating, batching, splitting, caching and sending data, you can build highly complex flows in a matter of a minutes.

With this in mind, it's important to understand how data flows through shapes.

How data flows

In the simplest of scenarios, a process flow receives a single payload of unpaginated data and this flows all the way through to completion with no manipulation or batching - one payload is received, processed, and completed.

However, if your incoming data is paginated and/or you introduce shapes capable of generating multiple payloads, it's important to understand how these pass through the flow. Essentially, any payloads that a shape outputs are added to a 'bucket' and it's that bucket that is then passed to the next shape.

So, all payloads from one shape are passed to the next shape in the same context - they don't pass down the entire flow individually.

If a 'pull' connection shape is configured to use an endpoint that paginates data, the connection shape outputs each page in its own payload.

Demo

The animation below shows how this works.

Suggested view options

For best results, view this animation in full screen and adjust settings to 2160p 4k:

Shape timeouts and retries

Timeouts

All shapes (except the connection shape) have a set timeout of 30 minutes. If processing is not completed within this time, the shape fails.

The timeout for a connection shape is configurable via connection shape settings.

Retries

With some exceptions (detailed below), a further three attempts will be made if a process flow shape fails. Exceptions are summarised below:

Shape	Number of retries
Add to cache	1
Connection	2
Manual payload	1
Split	1
De-dupe	1

Payload size

A single incoming payload for any process flow shape should not exceed 500MB.

We recommend processing multiple, smaller payloads rather than one single payload (1000 x 0.5MB payloads are more efficient than 1 x 500MB payload!).

For payloads up to 500MB, consider adding a flow control shape to batch data into multiple, smaller payloads. Payloads exceeding 500MB should be batched at source.