8
(2017) Rise of the Data Engineer
(medium.com)
A community for discussion about data engineering
Icon base by Delapouite under CC BY 3.0 with modifications to add a gradient
I've said this before to other people, but over time, those tools eventually became what Airflow and other orchestration tools are: defining DAGs and running scripts.
When I was using SSIS, eventually, every task was a C# or PowerShell executor instead of using the built-in functionality. So glad for Airflow and other modern tools today.
Definitely. It is much more pleasant to work with better tools for the same functionality.
Airflow got a lot of things right. For example in Luigi a runnable “task” is a python class that gets implicitly executed, whereas in Airflow tasks are made from functions that get called in a more straightforward/imperative manner. This makes DAGs much easier to read and write in Airflow.