Automated Way to Get Latest Pipeline Execution Details of an Azure Data Factory

Problem Statement :

As of Dec 18,2023 ; One can get the list of all execution details of a specific pipeline of an Azure Data Factory via the Powershell Module : Get-AzDataFactoryV2PipelineRun. But this Module, provides all execution details of the pipeline or the execution detail of the specific pipelinerunid.

Similarly, the Monitor section provides all execution details and we have to manually scan for the latest execution per pipeline.

So is there an automated way to Get Latest Pipeline Execution Details of an Azure Data Factory.

Solution :

1. Open Powershell in admin mode and import the Azure SDK modules by executing the below commands :

Import-Module Az.Accounts Import-Module Az.DataFactory

2. Install the custom module by executing the below command

Install-Module -Name AzDataFactoryV2PipelinesLastExecution

3. Now the environment is set for us to execute the commands to fetch the pipeline details .

To get the list of all Pipelines in an ADF and its corresponding last execution details, execute the below command: Get-AzDataFactoryV2PipelinesLastExecution -ResourceGroupName “<<RGName>>” -DataFactoryName “<<ADFName>>”

To get the details of a specific Pipeline in an ADF and its corresponding last execution, execute the below command: Get-AzDataFactoryV2PipelinesLastExecution -ResourceGroupName “<<RGName>>” -DataFactoryName “<<ADFName>>” -PipelineName “<<PipelineName>>”

To get the list of all Pipelines in an ADF and its corresponding last execution details in a specific time range, execute the below command: Get-AzDataFactoryV2PipelinesLastExecution -ResourceGroupName “<<RGName>>” -DataFactoryName “<<ADFName>>” -StartRange “<<StartDate>>” -EndRange “<<EndDate>>”

Output Results :

1. Overall ADF output

2. Specific Pipeline output

3. Specific Range output

Note : In case if no values are provided to StartRange and EndRange ; then default values would be assigned as 1900-01-01 and latest datetime accordingly.

This would avoid manually identifying the latest execution details of a pipeline in Azure Data Factory.

Similarly for Synapse,

Automated Way to Get Latest Pipeline Execution Details of Synapse