# Filter shape

## Introduction

When a connector is built, default filters can be applied at the API level, so when a process flow [connection shape](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape) pulls data, the payload received is has already been refined.

However, there may be times where you want to apply additional filters to a payload that's been pulled via a [connection shape](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape) - for example, if the API for a connector does not support particular filters that you need.

{% hint style="info" %}
The **filter** shape works with a source payload. As such, it should be placed AFTER a [connection shape](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/connector-shape) in process flows.
{% endhint %}

{% hint style="info" %}
A single incoming payload for [any process flow shape](https://open.gitbook.com/~site/site_dIV1g/~/revisions/9dTvdvRJIRVZQXuBMMnJ/process-flows/building-process-flows/process-flow-shapes) should not exceed 500MB.

We recommend processing multiple, smaller payloads rather than one single payload (1000 x 0.5MB payloads are more efficient than 1 x 500MB payload!).

For payloads up to 500MB, consider adding a[ flow control shape](https://open.gitbook.com/~site/site_dIV1g/~/revisions/9dTvdvRJIRVZQXuBMMnJ/process-flows/building-process-flows/process-flow-shapes/standard-shapes/flow-control-shape) to batch data into multiple, smaller payloads. Payloads exceeding 500MB should be batched at source.
{% endhint %}

## Need to know

When specifying a filter value, the maximum number of characters is 1024.

## Accessing filter shape settings

To view/update the settings for an existing **filter** shape, click the associated 'cog' icon:

<div align="left"><figure><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2F1eO7NU3poZJptPwYEyDO%2Ffilter%201.png?alt=media&#x26;token=fd459fdc-c408-41dd-a0e7-53bb314ecb5c" alt="" width="307"><figcaption></figcaption></figure></div>

This opens the [options panel ](#configuring-settings-for-a-new-filter-shape)- for example: &#x20;

<div align="left"><figure><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FAKBKTp3AMhhscsQvIRMO%2Ffilter%201.png?alt=media&#x26;token=d8074328-c479-4dc0-891d-501beecd932a" alt="" width="375"><figcaption></figcaption></figure></div>

## Configuring settings for a new filter shape

<details>

<summary><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FpLMO12yvTCxi9PorCt53%2Ficons8-film.svg?alt=media&#x26;token=e59864bc-2d97-4f98-966c-a083c18e60db" alt="" data-size="line"> Show me</summary>

![](https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2Fj0MXYbcb95cr0GZkkjgr%2Fdemo%20filter%20shape.gif?alt=media\&token=2fcb23cc-a6f6-4fc0-a704-099a980087bb)

</details>

Follow the steps below to configure a **filter** shape.

**Step 1**\
Select a **source integration** and **endpoint** to determine where the incoming payload to be filtered originates.

**Step 2**\
Click the **add new filter** button:

<div align="left"><figure><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FHg6CPXo29qma2IE1ZfaA%2Ffilter%203.png?alt=media&#x26;token=d1497cc8-7d2f-4297-b72d-fd4de76cda21" alt="" width="375"><figcaption></figcaption></figure></div>

**Step 3**\
Filter settings are displayed:

<div align="left"><figure><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FXhBumh2J32azDPpzk6jp%2Ffilter%20settings%201.png?alt=media&#x26;token=d166e11d-ee52-4871-8d18-d0e5776cb13c" alt="" width="375"><figcaption></figcaption></figure></div>

From here, you can select a **field** (from the **data schema** associated with the **source endpoint** selected in step 1) - for example:

<div align="left"><figure><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2F0TDVn4wt57ASsZAHnnNx%2Ffilter%205.png?alt=media&#x26;token=704fd9c9-531c-4ad8-a7db-d5283457fec7" alt="" width="375"><figcaption></figcaption></figure></div>

Alternatively, you can toggle the manual input option to ON and add a manual path.&#x20;

<div align="left"><figure><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FGcvMggllHlPyleB9iT0k%2Ffilter%20settings%20-%20manual%20input%201.png?alt=media&#x26;token=58e4160c-1356-46f2-948c-8cd874c557ca" alt="" width="563"><figcaption></figcaption></figure></div>

{% hint style="info" %}
The manual data path field supports [metadata variables](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/dynamic-variables/metadata-variables).
{% endhint %}

**Step 4**\
Use the remaining **operator**, **type** and **value** options to define the required filter.&#x20;

{% hint style="info" %}
Presentation of the **value** field is dependant upon your selected [type](#available-filter-types). For example, if the **type** field is set to **specific** date, you can pick a date for the **value**:

![](https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2Fyle9uXULQAwfGnFT3cW1%2F12%20-%20filter%20-%20date.png?alt=media\&token=d750b912-3f60-43a5-ae8b-9ce318ba2501)
{% endhint %}

{% hint style="info" %}
Available [data types](#available-filter-types) are detailed below. When defining a **value**, you can include [payload](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/dynamic-variables/payload-variables), [flow](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/dynamic-variables/flow-variables), and [metadata](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/dynamic-variables/metadata-variables) variables.
{% endhint %}

**Step 5**\
Use the **keep matching?** toggle option to choose how matched records should be treated:

<div align="left"><figure><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FxzxzuQRKMa8SmtlaVDEl%2F13%20-%20filter%20-%20keep%20matching.png?alt=media&#x26;token=bac29106-802b-4a1b-a3d2-eee626c08127" alt="" width="337"><figcaption></figcaption></figure></div>

Here:

* If the **keep matching?** option is toggled OFF, matched records are removed from the payload before it moves on down the flow for further processing.
* If the **keep matching?** option is toggled ON, matched records remain in the onward payload, and all non-matching records will be removed.

**Step 6**\
Click the **create** button to confirm your settings.

**Step 7**\
The filter is added to the **filter shape** - you can now add more filters if needed:

<div align="left"><figure><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2Fd2tpTbqmCLsHQDrvUdxY%2Ffilter%206.png?alt=media&#x26;token=08c6e242-2f25-4071-b02e-9c1b2fba05a7" alt="" width="375"><figcaption></figcaption></figure></div>

{% hint style="info" %}
Don't forget that when a payload is running you can [click the tick icon associated with any shape and view the payload at that point in time](https://doc.wearepatchworks.com/product-documentation/error-reporting-and-exception-handling/real-time-run-logs#real-time-monitoring) - this is a great way to check that your filter is refining data as expected.
{% endhint %}

## Available data types

When defining a filter, you can choose from the following types:

<table data-full-width="false"><thead><tr><th width="177">Type</th><th>Expected value</th></tr></thead><tbody><tr><td>String</td><td><p>A text string - for example : <code>blue</code>  </p><p><br>Note that when a <code>string</code> type is selected, you can choose the <strong>regex</strong> operator and define a regex value. This provides greater flexibility if you can't achieve the desired results using standard operators. <code>preg_match</code> is used for pattern matching. For an example, see <a href="#using-regex-for-string-type-filters">Using regex for string-type filters</a>.<br><br>You can also match a <code>string</code>-type field according to whether or not the value is one of many items in a given list. For more information, see <a href="filter-shape/using-contains-one-of-many-or-does-not-contain-one-of-many-for-string-filters">Using contains one of many or does not contain one of many for string filters</a>.</p></td></tr><tr><td>String length</td><td><p>A number which represents the expected string length for the received payload. Here, the 'payload' might refer to a targeted field within the incoming payload or the entire payload. <br><br>For example, if you want to ensure that an <code>objectId</code> field is never empty, you would define a filter for <code>objectId</code> <code>greater than</code> <code>0</code>: <br><br><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FPpJ1DNBzK2J0mERBYco2%2Ffilter%201.png?alt=media&#x26;token=bd708e56-9fd9-4b65-bd60-e8a4e0784053" alt=""><br><br>In this case, toggling the <code>keep matching</code> option to OFF means that the ongoing payload will include only items where this field is not empty. Conversely, toggle this option on if you want to pass on empty payloads for any reason.<br><br>You can use the same principle to check for empty payloads (as opposed to a specific field). In this case you would define a filter for <code>*</code> <code>greater than</code> <code>0</code>: </p><p><br><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FIhrpvf7aZWzfWJwzaVuE%2Ffilter%202.png?alt=media&#x26;token=9d051206-0b0f-4efa-bbf3-7803179f1b50" alt=""><br><br>Note that a blank <code>value</code>  is processed as a <code>0</code> value.</p></td></tr><tr><td>Number</td><td>A number - for example: <code>2</code></td></tr><tr><td>Specific date</td><td>A day, month and year, selected from a date picker.</td></tr><tr><td>Dynamic date</td><td><p>Specify a date/time which is relative to a +/- number of units (seconds, minutes, hours, days, months, years). For example:</p><p><br><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FKuXpDYhcYH1SiFHLkr75%2Ffilter%203.png?alt=media&#x26;token=bd78b842-7f0a-40d0-b60a-9047cb4227e1" alt=""><br></p><p>It's important to be aware that relative date/time variables are affected by the <a href="../../../process-flow-settings#understanding-process-flow-settings">use queued at time process flow setting</a>.</p></td></tr><tr><td>Boolean</td><td>A true or false value. For example, if you only want to consider items where an <code>itemRef</code> field is set to <code>true</code>, you would define a filter for <code>itemRef</code> <code>equals</code> <code>true</code>: <br><br><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2Fi5jcSc30GKor0X4kGxnM%2Ffilter%204.png?alt=media&#x26;token=ca2c92a6-962f-4ce6-820f-2f5e012db0d4" alt=""></td></tr><tr><td>Null comparison</td><td><p>A field is <code>null</code> if it has no defined value. Choose whether the give data path <code>is null</code> or <code>not null</code>. For example:</p><p><br><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FBUulutLrnHAbTpC68YqZ%2Ffilter%205.png?alt=media&#x26;token=f807a5cf-f161-4ff3-8e63-c1e269e7020f" alt=""><br></p></td></tr><tr><td>Variable</td><td><p>Designed specifically for cases where you are comparing a variable value as the filter comparison. </p><p><img src="https://2440044887-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLYNcUBVQwSkOMG6KjZfz%2Fuploads%2FgRiynXmMwmLrZjhoRsBm%2Fvariable%20filter%20type.png?alt=media&#x26;token=b490983f-3d52-4a4e-8f99-4332533b2f20" alt=""></p><p></p><p>When selected, a <code>value type</code> field is displayed and the expected type for your variable can be selected. This ensures that a true comparison can be made.  Note that if you are filtering on multiple variables, the <code>value type</code> field should be set to <code>None</code>.</p></td></tr></tbody></table>

## Related information

* [Using regex for string-type filters](#using-regex-for-string-type-filters)
* [Using contains one of many or does not contain one of many for string filters](https://doc.wearepatchworks.com/product-documentation/process-flows/building-process-flows/process-flow-shapes/standard-shapes/filter-shape/using-contains-one-of-many-or-does-not-contain-one-of-many-for-string-filters)
