Listing Unique records within an Array in Azure Data Factory

Problem Statement :

Is it possible to remove the duplicate within an Array in Azure Data Factory.

Prerequisites :

1. Azure Data Factory

Solution :

1. The “union()” function in ADF returns a collection that has all the items from the specified collections. So one can leverage this function to get the unique list from an Array.
2. Let’s say we have a list of values in an Array variable

3. Using Set Variable activity, we can get the unique list from the Array.

@union(variables('DuplicateArray'),variables('DuplicateArray'))

Output :

ADF JSON :

{
    "name": "ReturnUnique",
    "properties": {
        "activities": [
            {
                "name": "Remove Duplicates",
                "type": "SetVariable",
                "dependsOn": [],
                "userProperties": [],
                "typeProperties": {
                    "variableName": "UniqueArray",
                    "value": {
                        "value": "@union(variables('DuplicateArray'),variables('DuplicateArray'))",
                        "type": "Expression"
                    }
                }
            }
        ],
        "variables": {
            "DuplicateArray": {
                "type": "Array",
                "defaultValue": [
                    "A1",
                    "B2",
                    "C3",
                    "A1",
                    "A5",
                    "B2"
                ]
            },
            "UniqueArray": {
                "type": "Array"
            }
        },
        "annotations": []
    }
}