Airflow has some very useful reporting on individual DAGs. But when your workflows traverse multiple DAGs, Airflow's capabilities fall short of providing the critical capabilities your business demands.
In this short video, we will explain how to overcome these deficiencies and delight your business. Let's start with a simple Airflow example.
Within my Airflow system, today I want to focus on two DAGs. The DAGs have been built to perform separate functions, but together, they ensure that when a customer places an order on a website, they're presented with an accurate delivery date.
The purpose of the supply chain DAG is to calculate how bogged down the warehouses are with orders. It starts in the early hours of the morning, after the warehouse shifts have finished, and publishes latency metrics.
The e-commerce DAG is responsible for updating the retailer's website with delivery expectations. To prevent customer dissatisfaction, it must not run until the latency metrics are ready. Otherwise, it might promise customers a next-day delivery when the warehouse is actually running three days behind.
In this case, the supply chain DAG publishes the data that needs to be consumed by the e-commerce DAG. One way to build this dependency in Airflow is to use an external task sensor.
As we can see when we look at the supply chain DAG, there is no means of identifying that there is any downstream dependency. It appears that the DAG runs in isolation.
When we take a look within the e-commerce DAG, we can now see that there is an upstream external task dependency. This is being monitored by the external task sensor. However, there is no easy way to identify what the upstream task or DAG might be. In order to fully understand where the dependency lies, we must go into the code.
So, how do we get to a view that provides visibility of the entire workflow?
Let me introduce Automation Analytics & Intelligence, a solution that provides just that. On the screen, we're looking at exactly the same execution of the DAGs, but this time, with a view of the entire workflow.
The display can be broken into data panels:
The first: A list of the DAGs and tasks within the workflow.
The second: A timeline or Gantt view. This clearly shows the dependency between the data publish and data ingestion tasks, as well as optional displays of average run times for each task and the critical path of the process. This panel also shows the SLA time, a configurable setting that allows real-time monitoring and signaling.
The third: Shows the detail related to the selected DAG or task.
The fourth: Shows familiar detail related to each task execution, but adds a level of analytics with details such as average duration and the deviation from average.
Bringing our focus to the top of the screen, we can identify that this process breached its deadline. We can also access historical executions from the calendar. Additionally, AAI uses historical data to predict workflow outcomes in the future.
So, let's take a look at a predicted workflow outcome five days from now. Here, we are not only able to achieve visibility of the full workflow, but we're able to understand the predicted service delivery outcome.
We've just scratched the surface of how AAI delivers total visibility across complex DAGs and disparate Airflow environments. Beyond visibility, AAI is your early warning system. It doesn't just show you a failure; it identifies the root cause and empowers you to fix it before the business is impacted.
From predictive insights and proactive fixes to seamless service delivery, this is just the beginning of what Automation Analytics & Intelligence can do for your Airflow workflows.