Skip to main content

Prefect

  • docs: https://docs.prefect.io/2.11.5/

    Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines.

  • provides UI for monitoring flows and tasks, and it also allows you to build custom dashboards: UI

  • workflows are defined as directed flows, where tasks can be arbitrary Python functions or objects

  • provides strong support for dynamic workflows

  • more advanced error handling, including retries, retries with backoff, and more fine-grained control over how errors are managed

  • compared to Airflow it is less mature, more light-weight

  • example of flow:

@task
def get_letters(word: str) -> int:
    # call Python libs, shell out to bash
    return list(word)

@task
def display(letter: str):
    # or talk to Cloud services, or whatever else
    print(f"My letter: '{letter}'")

with Flow("myflow") as flow:
    word = Parameter("word")
    letters = get_letters(word)
    display.map(letters)

Deployment​