Alternatives
Contents
Alternatives¶
Many tools and services exist today for deploying Dask clusters, many of which are commonly used on the cloud. This project aims to provide cloud native plugins and tools for Dask which can often compliment other approaches.
Community tools¶
Dask has a vibrant ecosystem of community tooling for deploying Dask on various platforms. Many of which can be used on public cloud.
Kubernetes¶
Kubernetes is an extremely popular project for managing cloud workloads and is part of the broader Cloud Native Computing Foundation (CNCF) ecosystem.
Dask has many options for deploying clusters on Kubernetes.
HPC on Cloud¶
Many popular HPC scheduling tools are used on the cloud and support features such as elastic scaling. If you are already leveraging HPC tools like SLURM on the cloud then Dask has great integration with HPC schedulers.
Hadoop/Spark/Yarn¶
Many cloud platforms have popular managed services for running Apache Spark workloads.
If you’re already using a managed map-reduce service like Amazon EMR then check out dask-yarn.
Nebari¶
Nebari is an open source data science platform which can be run locally or on a cloud platform of your choice. It includes a managed Dask service built on Dask Gateway for managing Dask clusters.
Managed Services¶
Cloud vendors and third-party companies also offer managed Dask clusters as a service
Coiled¶
Coiled is a mature managed Dask service that spawns clusters in your cloud account and allows you to manage them via a central control plane.
Saturn Cloud¶
Saturn Cloud is a managed data science platform with hosted Dask clusters or the option to deploy them in your own AWS account.