Step-by-Step Guide to Building and Managing Robust Workflows with Apache Airflow
Sub Category
- Other IT & Software
{inAds}
Objectives
- Understand what Apache Airflow is, its purpose and pros and cons of using Airflow
- Step-by-step guide to installing Airflow
- Launch and navigate the Airflow Web UI and learn about various views: DAG, Grid, Graph, Calendar, Task Duration, Code, Variable and Gantt View
- Understand what a DAG is and how to create a DAG definition file and different methods for DAG creation
- Learn about DAG Run, default_arguments, and DAG arguments and Master scheduling concepts such as depends_on_past, wait_for_downstream, catchup, and backfill
- Use the Airflow CLI for various operations and access a handy cheatsheet for quick reference
- Understand tasks, task instances and Learn the lifecycle of a task
- Master different operators including BashOperator, PostgresOperator, PythonOperator, SqliteOperator, and EmailOperator
- Implement sensors like FileSensor, SQLSensor, TimeDeltaSensor, and TimeSensor
- Apply branching logic with BranchSQLOperator, BranchPythonOperator, BranchDayOfWeekOperator, BranchDateTimeOperator, and ShortCircuitOperator
- Manage DAG dependencies and use TaskGroups ,Utilize TriggerDagRunOperator , ExternalTaskSensor and use hooks such as PostgresHook and SHook
- Manage resources with pools and task priorities
- Learn about different types of executors: SequentialExecutor and LocalExecutor and learn the Transition from SequentialExecutor to LocalExecutor
- Explore the Airflow metadata database and Manage roles and create users with different roles including admin, public, user, and operator roles
- Set and manage task-level and DAG-level SLAs and handle SLA misses
- Address issues like zombie tasks, SIGTERM, and SIGKILL errors
Pre Requisites
- Knowledge of Python
- Rest we will cover to learn Airflow from scratch
- Familiarity with command-line interfaces.
- Understanding of database concepts is a plus but not required
FAQ
- Q. How long do I have access to the course materials?
- A. You can view and review the lecture materials indefinitely, like an on-demand channel.
- Q. Can I take my courses with me wherever I go?
- A. Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don't have an internet connection, some instructors also let their students download course lectures. That's up to the instructor though, so make sure you get on their good side!
{inAds}
Coupon Code(s)