About data usage

Introduction

Data usage is calculated by aggregating the size of payloads that leave each shape in a process flow - these are known as payloads out.

In any process flow, data is received and passed from one shape to the next. Different shapes handle their incoming payload(s) in different ways - in some cases, data simply passes through (data in is the same as data out) but in others, data is manipulated and changed.

Understanding how these payloads are aggregated provides a clearer picture of your overall data usage.

What is a payload out?

A payload is the data generated/processed during the execution of any shapes within a flow. We refer to the payload out when it leaves that shape. This could be via any of the mechanisms below:

Mechanism	Summary
Patchworks API call	Data is received from or sent to an API call.
Connector shape	Data is received from or sent to a Patchworks connector.
Other shapes	Data is processed within any shape - for example, by a custom script (script shape or transform function), mapping payloads from one structure to another (map shape), or routing payloads down multiple branches (route shape).
File transfers	Data moves between systems - for example, CSV files or image files.
Database queries	Data is fetched from or inserted into a database.
Triggers	Data is sent/received via an event, webhook or Patchworks API call.

How data usage is calculated

To calculate data usage, all payload out sizes (from each shape in a process flow) are aggregated. Here's how this works:

Considerations for your data usage

Typically, the size of a payload that goes into a process flow shape is the same as the payload out size - payloads are not modified unless your flow includes actions that are designed to do this.

The table below summarises process flow shapes and their ability to change the size of incoming payloads:

Shape	Change size?	Notes
Add to cache	No	The payload out is always the same as the incoming payload.
Assert	No	The payload out is always the same as the incoming payload.
Connector	Yes	When receiving data, the payload out will reflect the volume of data received from this connection request. When sending data, the payload out will be the same as the incoming payload UNLESS you choose to save response as payload (in which case the payload out is - typically - smaller than the incoming payload).
De-dupe	No	If set to filter or track & filter, a de-dupe shape may reduce the size of the incoming payload by removing duplicate items. If set to track, the incoming payload simply passes through for tracking - the payload out will always be the same as the incoming payload. A de-dupe shape will never increase the size of the payload out.
Filter	Yes	If a filter removes data from an incoming payload, the payload out will be smaller than the incoming payload. A filter will never increase the size of the payload out.
Flow control	No	The incoming payload is batched into multiple, smaller batches but the aggregate size of the payload out for those batches is always equal to the incoming payload size.
Load from cache	Yes	The payload out will reflect the volume of data loaded from the cache.
Map	Yes	A straightforward like-for-like mapping between two systems will not affect the size of the payload out. However, if transform functions are applied the size of the payload out may change slightly - for example, prefix, suffix, format date and script transforms. Typically, any size variations from mapping transformations are small.
Manual payload	No	The payload out is always the same as the incoming payload.
Route	Yes	When an incoming payload hits a route shape, your defined conditions are checked and a payload out is created for each defined route. If your route conditions exclude items in the incoming payload from progressing down any routes then the aggregate size of payloads out will be smaller than the incoming payload.
Run process flow	Yes	If you configure this shape with a manual payload then the payload out is likely to differ from any incoming payload. If no manual payload is specified then the payload out is always the same as the incoming payload.
Script	Yes	A custom script might increase or decrease the size of the payload out.
Set variables	No	The payload out is always the same as the incoming payload.
Split	Yes	The incoming data is split at a defined data element, so only that element progresses to the next step - i.e. the payload out is likely to be smaller than the incoming payload.
Track data	No	The incoming payload simply passes through for tracking - the payload out will always be the same as the incoming payload. A track data shape will never increase the size of the payload out.

Examples

The examples below show how data usage can be affected by different process flow shapes.