About data usage
Last updated
Last updated
Introduction
Data usage is calculated by aggregating the size of payloads that leave each shape in a process flow - these are known as payloads out.
In any process flow, data is received and passed from one shape to the next. Different shapes handle their incoming payload(s) in different ways - in some cases, data simply passes through (data in is the same as data out) but in others, data is manipulated and changed.
Understanding how these payloads are aggregated provides a clearer picture of your overall data usage.
A payload is the data generated/processed during the execution of any shapes within a flow. We refer to the payload out when it leaves that shape. This could be via any of the mechanisms below:
Mechanism | Summary |
---|---|
To calculate data usage, all payload out sizes (from each shape in a process flow) are aggregated. Here's how this works:
Typically, the size of a payload that goes into a process flow shape is the same as the payload out size - payloads are not modified unless your flow includes actions that are designed to do this.
The table below summarises process flow shapes and their ability to change the size of incoming payloads:
The examples below show how data usage can be affected by different process flow shapes.
Shape | Change size? | Notes |
---|---|---|
No
The payload out is always the same as the incoming payload.
No
The payload out is always the same as the incoming payload.
Yes
When receiving data, the payload out will reflect the volume of data received from this connection request.
When sending data, the payload out will be the same as the incoming payload UNLESS you choose to save response as payload
(in which case the payload out is - typically - smaller than the incoming payload).
No
If set to filter
or track & filter
, a de-dupe shape may reduce the size of the incoming payload by removing duplicate items.
If set to track
, the incoming payload simply passes through for tracking - the payload out will always be the same as the incoming payload.
A de-dupe shape will never increase the size of the payload out.
Yes
If a filter removes data from an incoming payload, the payload out will be smaller than the incoming payload. A filter will never increase the size of the payload out.
No
The incoming payload is batched into multiple, smaller batches but the aggregate size of the payload out for those batches is always equal to the incoming payload size.
Yes
The payload out will reflect the volume of data loaded from the cache.
Yes
A straightforward like-for-like mapping between two systems will not affect the size of the payload out. However, if transform functions are applied the size of the payload out may change slightly - for example, prefix
, suffix
, format date
and script
transforms.
Typically, any size variations from mapping transformations are small.
No
The payload out is always the same as the incoming payload.
Yes
When an incoming payload hits a route shape, your defined conditions are checked and a payload out is created for each defined route.
If your route conditions exclude items in the incoming payload from progressing down any routes then the aggregate size of payloads out will be smaller than the incoming payload.
Yes
If you configure this shape with a manual payload then the payload out is likely to differ from any incoming payload. If no manual payload is specified then the payload out is always the same as the incoming payload.
Yes
A custom script might increase or decrease the size of the payload out.
No
The payload out is always the same as the incoming payload.
Yes
The incoming data is split at a defined data element, so only that element progresses to the next step - i.e. the payload out is likely to be smaller than the incoming payload.
No
The incoming payload simply passes through for tracking - the payload out will always be the same as the incoming payload.
A track data shape will never increase the size of the payload out.
Patchworks API call
Data is received from or sent to an API call.
Connector shape
Data is received from or sent to a Patchworks connector.
Other shapes
Data is processed within any shape - for example, by a custom script (script shape or transform function), mapping payloads from one structure to another (map shape), or routing payloads down multiple branches (route shape).
File transfers
Data moves between systems - for example, CSV files or image files.
Database queries
Data is fetched from or inserted into a database.
Triggers
Data is sent/received via an event, webhook or Patchworks API call.