Amazon's Airflow offering sucks really hard. As in many services, they made an awful job at designing a good UX and hiding the product's complexity from their users.

Astronomer is cool, but expensive and they won't accept monthly billing. Also, it's even more expensive if you need DAGS to access things within a VPC.

Step Functions is decent for simple use cases and very reliable. For complex stuff, you will hate their JSON based DSL with passion.

Thanks, glad to hear that before I bother trying it.

Have heard astronomer is too expensive.

Step Functions doesn't look great either.

At Grove we make extensive use of KubernetesPodOperator[0]. This allows to both encapsulate the job in containers (k8s pods), and also specify the resources we need for each DAG.

[0] https://airflow.apache.org/docs/apache-airflow-providers-cnc...

It depends on what other tools you are currently using for ETL (or want to use). One example is that we used Spark so we would use the Spark submit operator to submit jobs to clusters. You can also use the K8s Pod operator if you want to utilize containers for your compute.

There are a lot of options. We were adopters before AWS hosted airflow was a thing, so I don't have any experiencing running AWS hosted Airflow.

I haven't looked recently to see if some of the challenges we faced early on are solved now, but most of them stemmed from how DAG updates were handled: changing the start date on a DAG would break you DAG forever until you go update the database by hand. Things like this are(/were?) super painful and could get worse with a managed solution.

Thanks. Requirement at this stage is to coordinate the running of a few singer.io or airbyte taps.

I may consider Prefect over Airflow, generally my experience so far resonates with this blog post: https://towardsdatascience.com/tired-of-airflow-try-this-c51...

If you are on aws, then step functions could be your best option. They are responsible for the workflow definition, and the compute load is completely dependent on the underlying tasks and aws services you are going to use.

Edit: typos

We used Astronomer at my last job. Sure beats trying to run your own instance and the Astronomer team were fantastic.