DEV Community

Cover image for Azure Data Factory Dynatrace: spot failing pipelines
Span Observer
Span Observer

Posted on Originally published at span.observer

Azure Data Factory Dynatrace: spot failing pipelines

A nightly pipeline fails at 3 a.m., nobody is told, and by 9 a.m. the finance dashboard shows yesterday’s figures. Wiring Azure Data Factory Dynatrace together answers that scenario concretely: which pipeline failed, since when, and whether the fault lies in the activity or in the machine running it. The vendor post, Transforming Azure Data Factory operations with Dynatrace, sets out that ambition; what follows is an operator’s reading, cross-checked against the documentation.

I start from an operations decision: what should happen when a pipeline does not finish? Your answer determines which signals you need. The scope is Dynatrace SaaS; I read nothing about Managed on this topic, so I promise nothing there.

A note on method: span.observer has not connected this integration to a real factory. The facts come from the Dynatrace documentation and the post cited, read on 10 November 2026. Anything I could not read is marked « not verified ».

Azure Data Factory Dynatrace: two sources of signal

To monitor Azure Data Factory with Dynatrace, two streams complement each other. The first is the classic Azure service: Dynatrace queries the Azure Monitor API about every 5 minutes and derives metrics. The second is the factory’s diagnostics, which produce one log per run. The vendor post talks about ingesting logs and metrics from Azure Monitor and correlating them.

The difference is simple: metrics tell you how many (how many runs failed this hour), logs tell you which one and why (which pipeline, which error message). You build an alert on the former and run an investigation in the latter. For Azure logs in general, see sending Azure Monitor logs to Dynatrace.

Bring the Data Factory metrics into Dynatrace

The integration documentation asks for Dynatrace 1.199 or later and an environment ActiveGate 1.195 or later. It lists no permission specific to the service: the general Azure connection model applies (a read-only service principal, an ActiveGate that reaches management.azure.com and login.microsoftonline.com). The custom-role templates on the general page do include the read action Microsoft.DataFactory/*/read.

  1. Check that an ActiveGate with the Azure module is available, under Deployment Status then ActiveGates.
  2. Create or reuse the Azure connection: Settings, Cloud and virtualization, Azure, Connect new instance.
  3. In the connection’s Services section, add Azure Data Factory (V1, V2) with Manage services then Add service.
  4. Open the Dashboards page: a preset dashboard appears there with the recommended metrics.

Two useful details from the page. On a service that is already monitored, the preset dashboard may not show up until you resave the credentials (Save on the instance). And the dashboard cannot be edited: clone it, then edit the copy.

Which metrics to read first

The page lists dozens of metrics for version 2. Not all deserve your attention on day one. These are the ones the documentation flags as recommended.

The seven Azure Data Factory V2 metrics flagged as recommended in the Dynatrace documentation, with their dimensions

Read them layer by layer. Pipeline runs (succeeded or failed) answer « did the job finish? ». Activity runs, with the PipelineName dimension, answer « which step breaks? ». CPU and memory of the integration runtime answer « is the machine coping? ». The FailureType dimension separates failure families; I did not verify its values, so look at them in your own data before building a rule on them.

The optional metrics add queue length and task pick-up delay. They help when a pipeline slows down without failing.

Seven optional Azure Data Factory V2 metrics from the Dynatrace documentation: runtime queue length, pick-up delay, failed triggers, resource count

Mind the cost: monitored services consume Davis data units, and Azure may charge beyond one million API calls a month. Enable the optional ones only for a specific question.

From the late pipeline to the failed pipeline

The vendor post details what logs add: the status and duration of each run, the time spent in the queue, the actual execution time, and a message field with runtime detail, errors included. Separating queue from execution is valuable: a pipeline that is slow because it waits on a saturated runtime is not fixed like one that is slow because of its source query.

For those logs to exist, the factory must send its diagnostic settings to a destination Dynatrace can read. The exact category and field names are not in the pages I read: not verified, so check your Azure portal and your log view before writing a query.

Tie the pipeline to a business stake

The post’s most interesting idea is to step beyond the technical. A Lookup activity reads a business value at the source (the example given is the day’s revenue in a spreadsheet) and passes it on as a custom user property. You can then see, next to pipeline health, whether the day’s figure actually arrived. The failure is no longer « pipeline X is red » but « the sales report is incomplete ».

If that appeals, look at your business events and business KPIs: it is the same logic, giving a technical signal a business meaning.

Decide the reaction: alert, then automatic retry?

Back to the opening decision. For the alert, rely on pipeline failures rather than a duration threshold: see how to create an alert step by step. For recovery, the post describes Workflows that call the Azure Data Factory REST API to rerun failed pipelines automatically, including those owned by other teams whose retry policy is not configured.

One safeguard: rerunning without understanding can double writes. Keep automatic retries for idempotent pipelines, cap the number of attempts and log every rerun. The Workflow details (trigger, authentication, request body) are not in the post: not verified.

In an Azure context, a failure can also come from the network. On that front, SNAT exhaustion on Azure is a cause to rule out before blaming the pipeline.

How to check that it works

  1. In Dashboards, open the preset Data Factory dashboard: pipeline runs should appear.
  2. Deliberately run a test pipeline that fails (missing source): PipelineFailedRuns should rise with the pipeline name.
  3. Check that the log’s error message matches the one in the Azure portal.
  4. If nothing appears, resave the connection credentials and wait a few five-minute cycles.

Best practices

  • Start with the seven recommended metrics and add the rest on demand.
  • Tag your factories in Azure: the connection accepts tag filters, which limits scope and cost.
  • Separate waiting from execution when analysing a slowdown.
  • Give each alert a pipeline owner, not a generic team.
  • Keep automatic retries for replayable jobs.

Frequently asked questions

Does Azure Data Factory Dynatrace monitoring work on Managed?

I only read the general Azure integration documentation and the Data Factory service page, with no Managed detail. Treat the topic as not verified on Managed.

Do I need an agent installed on the factory?

Not for metrics: they come from the Azure Monitor API through an ActiveGate. A self-hosted integration runtime remains a machine to monitor separately.

Are metrics enough?

They are enough to alert. To know why a pipeline fails, you need the error message from the logs.

Conclusion

First decide what a failed pipeline must trigger, then pick the signals: metrics to alert, logs to understand, business values to prioritise. Test on a staging factory before production.

span.observer is an independent community blog, not affiliated with Dynatrace LLC (nor with Microsoft). Brands mentioned belong to their owners. Tools change quickly: check commands and options in the official documentation.


Originally published on span.observer. French version: https://span.observer/fr/azure-data-factory-dynatrace-pipelines/.

Top comments (0)