Understanding how data flows through shapes
Introduction
Patchworks process flows are incredibly flexible. With a range of shapes for receiving, paginating, manipulating, batching, splitting, caching and sending data, you can build highly complex flows in a matter of a minutes.
With this in mind, it's important to understand how data flows through shapes.
How data flows
In the simplest of scenarios, a process flow receives a single payload of unpaginated data and this flows all the way through to completion with no manipulation or batching - one payload is received, processed, and completed.
However, if your incoming data is paginated and/or you introduce shapes capable of generating multiple payloads, it's important to understand how these pass through the flow. Essentially, any payloads that a shape outputs are added to a 'bucket' and it's that bucket that is then passed to the next shape.
So, all payloads from one shape are passed to the next shape in the same context - they don't pass down the entire flow individually.
If a 'pull' connection shape is configured to use an endpoint that paginates data, the connection shape outputs each page in its own payload.
Demo
The animation below shows how this works.
Shape timeouts and retries
Timeouts
All shapes (except the connection shape) have a set timeout of 30 minutes. If processing is not completed within this time, the shape fails.
The timeout for a connection shape is configurable via connection shape settings.
Retries
With some exceptions (detailed below), a further three attempts will be made if a process flow shape fails. Exceptions are summarised below:
| Shape | Number of retries |
|---|---|
1 | |
2 | |
1 | |
1 | |
1 |
Last updated
