Skip to main content

Workflow engines

Conclusion, next steps

  • the logic behind mapping the event + configuration to the tasks and their dependencies needs to be done on our side, the tools can help with the orchestration of the dependencies after we define them
  • without much changed needed, we could try utilising Celery Canvas to help us with scheduling tasks based on the dependencies we specify
  • other tools mostly provide support for more complex usecases, but from the ones that were listed, I can see Airflow/Prefect could fit into what we do in Packit - the downside of using these is more complex deployment than we have now
  • as for Airflow/Prefect, Airflow is a more stable tool that has great documentation, provides variety of features and a lot of people already handled the problems we would need to handle; Prefect might be on the other hand more light-weight
  • some of the tools offer features for waiting for an external event (such as message on a messaging bus), we could utilise these (e.g. Airflow's Sensors)
  • some of the solutions could help us get rid of sandcastle (KubernetesExecutor in Airflow)