Alternatives

Many tools and services exist today for deploying Dask clusters, many of which are commonly used on the cloud. This project aims to provide cloud native plugins and tools for Dask which can often compliment other approaches.

Community tools

Dask has a vibrant ecosystem of community tooling for deploying Dask on various platforms. Many of which can be used on public cloud.

Kubernetes

Kubernetes is an extremely popular project for managing cloud workloads and is part of the broader Cloud Native Computing Foundation (CNCF) ecosystem.

Dask has many options for deploying clusters on Kubernetes.

HPC on Cloud

Many popular HPC scheduling tools are used on the cloud and support features such as elastic scaling. If you are already leveraging HPC tools like SLURM on the cloud then Dask has great integration with HPC schedulers.

Hadoop/Spark/Yarn

Many cloud platforms have popular managed services for running Apache Spark workloads.

If you’re already using a managed map-reduce service like Amazon EMR then check out dask-yarn.

Nebari

Nebari is an open source data science platform which can be run locally or on a cloud platform of your choice. It includes a managed Dask service built on Dask Gateway for managing Dask clusters.

Managed Services

Cloud vendors and third-party companies also offer managed Dask clusters as a service

Coiled

Coiled is a mature managed Dask service that spawns clusters in your cloud account and allows you to manage them via a central control plane.

Saturn Cloud

Saturn Cloud is a managed data science platform with hosted Dask clusters or the option to deploy them in your own AWS account.