||Cluster running on Digital Ocean droplets.|
To authenticate with DigitalOcean you must first generate a personal access token.
Then you must put this in your Dask configuration at
cloudprovider.digitalocean.token. This can be done by
adding the token to your YAML configuration or exporting an environment variable.
# ~/.config/dask/cloudprovider.yaml cloudprovider: digitalocean: token: "yourtoken"
$ export DASK_CLOUDPROVIDER__DIGITALOCEAN__TOKEN="yourtoken"
DropletCluster(region: str = None, size: str = None, image: str = None, debug: bool = False, **kwargs)¶
Cluster running on Digital Ocean droplets.
VMs in DigitalOcean (DO) are referred to as droplets. This cluster manager constructs a Dask cluster running on VMs.
When configuring your cluster you may find it useful to install the
doctltool for querying the DO API for available options.
- region: str
The DO region to launch you cluster in. A full list can be obtained with
doctl compute region list.
- size: str
The VM size slug. You can get a full list with
doctl compute size list. The default is
s-1vcpu-1gbwhich is 1GB RAM and 1 vCPU
- image: str
The image ID to use for the host OS. This should be a Ubuntu variant. You can list available images with
doctl compute image list --public | grep ubuntu.*x64.
- worker_module: str
The Dask worker module to start on worker VMs.
- n_workers: int
Number of workers to initialise the cluster with. Defaults to
- worker_module: str
The Python module to run for the worker. Defaults to
- worker_options: dict
Params to be passed to the worker class. See
distributed.worker.Workerfor default worker class. If you set
worker_modulethen refer to the docstring for the custom worker class.
- scheduler_options: dict
Params to be passed to the scheduler class. See
- docker_image: string (optional)
The Docker image to run on all instances.
This image must have a valid Python environment and have
daskinstalled in order for the
dask-workercommands to be available. It is recommended the Python environment matches your local environment where
EC2Clusteris being created from.
For GPU instance types the Docker image much have NVIDIA drivers and
By default the
daskdev/dask:latestimage will be used.
- docker_args: string (optional)
Extra command line arguments to pass to Docker.
- env_vars: dict (optional)
Environment variables to be passed to the worker.
- silence_logs: bool
Whether or not we should silence logging when setting up the cluster.
- asynchronous: bool
If this is intended to be used directly within an event loop with async/await
- security : Security or bool, optional
Configures communication security in this cluster. Can be a security object, or True. If True, temporary self-signed credentials will be created automatically. Default is
- debug: bool, optional
More information will be printed when constructing clusters to enable debugging.
Create the cluster.
>>> from dask_cloudprovider.digitalocean import DropletCluster >>> cluster = DropletCluster(n_workers=1) Creating scheduler instance Created droplet dask-38b817c1-scheduler Waiting for scheduler to run Scheduler is running Creating worker instance Created droplet dask-38b817c1-worker-dc95260d
Connect a client.
>>> from dask.distributed import Client >>> client = Client(cluster)
Do some work.
>>> import dask.array as da >>> arr = da.random.random((1000, 1000), chunks=(100, 100)) >>> arr.mean().compute() 0.5001550986751964
Close the cluster
>>> client.close() >>> cluster.close() Terminated droplet dask-38b817c1-worker-dc95260d Terminated droplet dask-38b817c1-scheduler
You can also do this all in one go with context managers to ensure the cluster is created and cleaned up.
>>> with DropletCluster(n_workers=1) as cluster: ... with Client(cluster) as client: ... print(da.random.random((1000, 1000), chunks=(100, 100)).mean().compute()) Creating scheduler instance Created droplet dask-48efe585-scheduler Waiting for scheduler to run Scheduler is running Creating worker instance Created droplet dask-48efe585-worker-5181aaf1 0.5000558682356162 Terminated droplet dask-48efe585-worker-5181aaf1 Terminated droplet dask-48efe585-scheduler
adapt(*args[, minimum, maximum])
Turn on adaptivity
call_async(f, *args, **kwargs)
Run a blocking function in a thread as a coroutine.
Create an instance of this class to represent an existing cluster by name.
get_logs([cluster, scheduler, workers])
Return logs for the cluster, scheduler and workers
Generate tags to be applied to all resources.
Return name and spec for the next worker
scale([n, memory, cores])
Scale cluster to n workers
scale_up([n, memory, cores])
Scale cluster to n workers close get_cloud_init logs render_cloud_init render_process_cloud_init scale_down sync