Problem Statement :
As of Dec 17,2022 ; One can identify the dependencies on a Dataset in an Azure Data Factory by logging in to the Azure Data Factory via Portal and accessing the Entity and check the Related tab.
A) Pipeline Dependent on Dataset
B) Dataflow Dependent on Dataset
So how can one get the related / dependent objects on a Dataset in an Azure Data Factory in an automated way.
Solution :
- Open Powershell in admin mode and import the Azure SDK modules by executing the below commands :
Import-Module Az.Accounts Import-Module Az.DataFactory |
2. Install the custom module by executing the below command
Install-Module -Name AzDataFactoryV2DatasetDependencies |
3. Now the environment is set for us to execute the commands to fetch the pipeline details .
To get the list of all Linked Services in an ADF and its corresponding dependent objects, execute the below command: Get-AzDataFactoryV2DatasetDependencies -ResourceGroupName “<<RGName>>” -DataFactoryName “<<ADFName>>” |
To get the details of a specific Linked Services in an ADF and its corresponding dependent objects, execute the below command: Get-AzDataFactoryV2DatasetDependencies -ResourceGroupName “<<RGName>>” -DataFactoryName “<<ADFName>>” -DatasetName“<<DatasetName>>” |
Output Results :
- Overall ADF output
2. Specific Dataset output
This would avoid manually identifying the dependent objects of a Dataset in Azure Data Factory.
Similarly for Synapse, refer the below blog :
Automated Way to Identify the Dependencies on a Dataset in Synapse