About data usage
Last updated
Last updated
Introduction
Data usage is calculated by aggregating the size of payloads that leave each shape in a process flow - these are known as payloads out.
In any process flow, data is received and passed from one shape to the next. Different shapes handle their incoming payload(s) in different ways - in some cases, data simply passes through (data in is the same as data out) but in others, data is manipulated and changed.
Understanding how these payloads are aggregated provides a clearer picture of your overall data usage.
A payload is the data generated/processed during the execution of any shapes within a flow. We refer to the payload out when it leaves that shape. This could be via any of the mechanisms below:
Patchworks API call
Data is received from or sent to an API call.
Connector shape
Data is received from or sent to a Patchworks connector.
Other shapes
Data is processed within any shape - for example, by a custom script (script shape or transform function), mapping payloads from one structure to another (map shape), or routing payloads down multiple branches (route shape).
File transfers
Data moves between systems - for example, CSV files or image files.
Database queries
Data is fetched from or inserted into a database.
Triggers
Data is sent/received via an event, webhook or Patchworks API call.
To calculate data usage, all payload out sizes (from each shape in a process flow) are aggregated. Here's how this works:
Typically, the size of a payload that goes into a process flow shape is the same as the payload out size - payloads are not modified unless your flow includes actions that are designed to do this.
The table below summarises process flow shapes and their ability to change the size of incoming payloads:
Yes
When receiving data, the payload out will reflect the volume of data received from this connection request.
When sending data, the payload out will be the same as the incoming payload UNLESS you choose to save response as payload (in which case the payload out is - typically - smaller than the incoming payload).
No
If set to filter or track & filter, a de-dupe shape may reduce the size of the incoming payload by removing duplicate items.
If set to track, the incoming payload simply passes through for tracking - the payload out will always be the same as the incoming payload.
A de-dupe shape will never increase the size of the payload out.
Yes
If a filter removes data from an incoming payload, the payload out will be smaller than the incoming payload. A filter will never increase the size of the payload out.
No
The incoming payload is batched into multiple, smaller batches but the aggregate size of the payload out for those batches is always equal to the incoming payload size.
Yes
A straightforward like-for-like mapping between two systems will not affect the size of the payload out. However, if transform functions are applied the size of the payload out may change slightly - for example, prefix, suffix, format date and script transforms.
Typically, any size variations from mapping transformations are small.
Yes
When an incoming payload hits a route shape, your defined conditions are checked and a payload out is created for each defined route.
If your route conditions exclude items in the incoming payload from progressing down any routes then the aggregate size of payloads out will be smaller than the incoming payload.
Yes
If you configure this shape with a manual payload then the payload out is likely to differ from any incoming payload. If no manual payload is specified then the payload out is always the same as the incoming payload.
Yes
The incoming data is split at a defined data element, so only that element progresses to the next step - i.e. the payload out is likely to be smaller than the incoming payload.
No
The incoming payload simply passes through for tracking - the payload out will always be the same as the incoming payload.
A track data shape will never increase the size of the payload out.
The examples below show how data usage can be affected by different process flow shapes.
(1) Identify interactions Every time a shape in your process flow performs a manipulation, moves data or sends/receives data between systems, an interaction occurs. This could be an API call, a file upload, or any other data transfer/manipulation.
(2) Measure payload sizes For each interaction, the size of the payload out is measured in megabytes.
Only the actual payload is considered - metadata, headers, and other non-payload data are NOT considered when calculating the payload out size.
(3) Aggregate payload sizes All payload sizes are aggregated to calculate the total data usage.
In the simplest of flows, a connector shape receives a 1MB payload, so the first payload out is 1MB.
The map shape receives this as its incoming payload. There are no field transformations so the data doesn't change - the second payload out is 1MB.
The final connector shape receives this as its incoming payload to be sent into the associated system. We have NOT set the save response as payload option to on, so the payload out is 1MB.
The aggregate total for the payload out size for all three shapes is 3MB.
Here, a connector shape receives a 1MB payload, so the first payload out is 1MB.
The de-dupe shape receives this as its incoming payload and filters out all duplicate records, reducing the payload size. The next payload out is 0.75MB.
The map shape receives this as its incoming payload. There are no field transformations so the data doesn't change - the next payload out is 0.75MB. The flow control shape receives this as its incoming payload and batches it into 10 payloads for onward processing. The next payload out is 10 x 75K.
The second connector receives all 10 payloads to be sent into the associated system. We have NOT set the save response as payload option to on, so the payload out is 10 x 75K. Finally, all 10 payloads pass through the de-dupe shape for tracking only. The payload out is 10 x 75K.
The aggregate total for the payload out size for all shapes is 4.75MB.
Here, a connector shape receives a 1MB payload, so the first payload out is 1MB.
This payload is added to a cache, and the next payload out is 1MB. This payload is passed to a route shape with conditions that send half the payload down one route and half down the other - resulting in 2 x 0.5MB payloads out.
For route 1, the first 0.5MB payload passes through a track data shape and the payload out is 0.5MB. The map shape receives this as its incoming payload and there are no field transformations so the data doesn't change - the payload out is 0.5MB. The final connector receives this as its incoming payload to be sent into the associated system. We have NOT set the save response as payload option to on, so the payload out is 0.5MB.
For route 2, the first 0.5MB payload is received by the map shape - there are no field transformations so the data doesn't change - the payload out is 0.5MB. The final connector receives this as its incoming payload to be sent into the associated system. We have NOT set the save response as payload option to on, so the payload out is 0.5MB.
The aggregate total for the payload out size for all shapes is 5.5MB.
Here, a connector shape receives a 1MB payload, so the first payload out is 1MB.
This payload is added to a cache, and the next payload out is 1MB. Then we load cache data from an existing company cache which is 50MB, so the next payload out is 50MB.
The script shape receives this as its incoming payload and runs - it doesn't do anything that affects the payload size so the next payload out is 50MB.
The map shape receives this as its incoming payload. There are no field transformations so the data doesn't change - the next payload out is 50MB.
The final connector receives this as its incoming payload to be sent into the associated system. We have NOT set the save response as payload option to on, so the payload out is 50MB.
The aggregate total for the payload out size for all shapes is 202MB.
