Configuration

Each cluster manager in Dask Cloudprovider will require some configuration specific to the cloud services you wish to use. Many config options will have sensible defaults and often you can create a cluster with just your authentication credentials configured.

Authentication

All cluster managers assume you have already configured your credentials for the cloud you are using.

For AWS this would mean storing your access key and secret key in ~/.aws/credentials. The AWS CLI can create this for you by running the command aws configure.

See each cluster manager for specific details.

Warning

Most cluster managers also allow passing credentials as keyword arguments, although this would result in credentials being stored in code and is not advised.

Cluster config

Configuration can be passed to a cluster manager via keyword arguments, YAML config or environment variables.

For example the FargateCluster manager for AWS ECS takes a scheduler_mem configuration option to set how much memory to give the scheduler in megabytes. This can be configured in the following ways.

from dask_cloudprovider.aws import FargateCluster

cluster = FargateCluster(
    scheduler_mem=8192
)
# ~/.config/dask/cloudprovider.yaml

cloudprovider:
  ecs:
    scheduler_mem: 8192
$ export DASK_CLOUDPROVIDER__ECS__SCHEDULER_MEM=8192

See each cluster manager and the Dask configuration docs for more information.