About data usage
Introduction
Data usage is calculated by aggregating the size of payloads that leave each shape in a process flow - these are known as payloads out.
In any process flow, data is received and passed from one shape to the next. Different shapes handle their incoming payload(s) in different ways - in some cases, data simply passes through (data in is the same as data out) but in others, data is manipulated and changed.
Understanding how these payloads are aggregated provides a clearer picture of your overall data usage.
What is a payload out?
A payload is the data generated/processed during the execution of any shapes within a flow. We refer to the payload out when it leaves that shape. This could be via any of the mechanisms below:
Mechanism | Summary |
---|---|
Patchworks API call | Data is received from or sent to an API call. |
Connector shape | Data is received from or sent to a Patchworks connector. |
Other shapes | Data is processed within any shape - for example, by a custom script (script shape or transform function), mapping payloads from one structure to another (map shape), or routing payloads down multiple branches (route shape). |
File transfers | Data moves between systems - for example, CSV files or image files. |
Database queries | Data is fetched from or inserted into a database. |
Triggers | Data is sent/received via an event, webhook or Patchworks API call. |
How data usage is calculated
To calculate data usage, all payload out sizes (from each shape in a process flow) are aggregated. Here's how this works:
Considerations for your data usage
Typically, the size of a payload that goes into a process flow shape is the same as the payload out size - payloads are not modified unless your flow includes actions that are designed to do this.
The table below summarises process flow shapes and their ability to change the size of incoming payloads:
Shape | Change size? | Notes |
---|---|---|
No | The payload out is always the same as the incoming payload. | |
No | The payload out is always the same as the incoming payload. | |
Yes | When receiving data, the payload out will reflect the volume of data received from this connection request. When sending data, the payload out will be the same as the incoming payload UNLESS you choose to | |
No | If set to If set to A de-dupe shape will never increase the size of the payload out. | |
Yes | If a filter removes data from an incoming payload, the payload out will be smaller than the incoming payload. A filter will never increase the size of the payload out. | |
No | The incoming payload is batched into multiple, smaller batches but the aggregate size of the payload out for those batches is always equal to the incoming payload size. | |
Yes | The payload out will reflect the volume of data loaded from the cache. | |
Yes | A straightforward like-for-like mapping between two systems will not affect the size of the payload out. However, if transform functions are applied the size of the payload out may change slightly - for example, | |
No | The payload out is always the same as the incoming payload. | |
Yes | When an incoming payload hits a route shape, your defined conditions are checked and a payload out is created for each defined route. If your route conditions exclude items in the incoming payload from progressing down any routes then the aggregate size of payloads out will be smaller than the incoming payload. | |
Yes | If you configure this shape with a manual payload then the payload out is likely to differ from any incoming payload. If no manual payload is specified then the payload out is always the same as the incoming payload. | |
Yes | A custom script might increase or decrease the size of the payload out. | |
No | The payload out is always the same as the incoming payload. | |
Yes | The incoming data is split at a defined data element, so only that element progresses to the next step - i.e. the payload out is likely to be smaller than the incoming payload. | |
No | The incoming payload simply passes through for tracking - the payload out will always be the same as the incoming payload. A track data shape will never increase the size of the payload out. |
Examples
The examples below show how data usage can be affected by different process flow shapes.
Last updated