API

AzureMLCluster(workspace[, compute_target, …]) Deploy a Dask cluster using Azure ML
ECSCluster([fargate_scheduler, …]) Deploy a Dask cluster using ECS
FargateCluster(**kwargs) Deploy a Dask cluster using Fargate on ECS
class dask_cloudprovider.AzureMLCluster(workspace, compute_target=None, environment_definition=None, experiment_name=None, initial_node_count=None, jupyter=None, jupyter_port=None, dashboard_port=None, scheduler_port=None, scheduler_idle_timeout=None, worker_death_timeout=None, additional_ports=None, admin_username=None, admin_ssh_key=None, datastores=None, code_store=None, vnet_resource_group=None, vnet=None, subnet=None, show_output=False, telemetry_opt_out=None, asynchronous=False, **kwargs)[source]

Deploy a Dask cluster using Azure ML

This creates a dask scheduler and workers on an Azure ML Compute Target.

Parameters:
workspace: azureml.core.Workspace (required)

Azure ML Workspace - see https://aka.ms/azureml/workspace.

vm_size: str (optional)

Azure VM size to be used in the Compute Target - see https://aka.ms/azureml/vmsizes.

datastores: List[Datastore] (optional)

List of Azure ML Datastores to be mounted on the headnode - see https://aka.ms/azureml/data and https://aka.ms/azureml/datastores.

Defaults to []. To mount all datastores in the workspace, set to ws.datastores.values().

environment_definition: azureml.core.Environment (optional)

Azure ML Environment - see https://aka.ms/azureml/environments.

Defaults to the “AzureML-Dask-CPU” or “AzureML-Dask-GPU” curated environment.

scheduler_idle_timeout: int (optional)

Number of idle seconds leading to scheduler shut down.

Defaults to 1200 (20 minutes).

experiment_name: str (optional)

The name of the Azure ML Experiment used to control the cluster.

Defaults to dask-cloudprovider.

initial_node_count: int (optional)

The initial number of nodes for the Dask Cluster.

Defaults to 1.

jupyter: bool (optional)

Flag to start JupyterLab session on the headnode of the cluster.

Defaults to False.

jupyter_port: int (optional)

Port on headnode to use for hosting JupyterLab session.

Defaults to 9000.

dashboard_port: int (optional)

Port on headnode to use for hosting Dask dashboard.

Defaults to 9001.

scheduler_port: int (optional)

Port to map the scheduler port to via SSH-tunnel if machine not on the same VNET.

Defaults to 9002.

worker_death_timeout: int (optional)

Number of seconds to wait for a worker to respond before removing it.

Defaults to 30.

additional_ports: list[tuple[int, int]] (optional)

Additional ports to forward. This requires a list of tuples where the first element is the port to open on the headnode while the second element is the port to map to or forward via the SSH-tunnel.

Defaults to [].

compute_target: azureml.core.ComputeTarget (optional)

Azure ML Compute Target - see https://aka.ms/azureml/computetarget.

admin_username: str (optional)

Username of the admin account for the AzureML Compute. Required for runs that are not on the same VNET. Defaults to empty string. Throws Exception if machine not on the same VNET.

Defaults to "".

admin_ssh_key: str (optional)

Location of the SSH secret key used when creating the AzureML Compute. The key should be passwordless if run from a Jupyter notebook. The id_rsa file needs to have 0700 permissions set. Required for runs that are not on the same VNET. Defaults to empty string. Throws Exception if machine not on the same VNET.

Defaults to "".

vnet: str (optional)

Name of the virtual network.

subnet: str (optional)

Name of the subnet inside the virtual network vnet.

vnet_resource_group: str (optional)

Name of the resource group where the virtual network vnet is located. If not passed, but names for vnet and subnet are passed, vnet_resource_group is assigned with the name of resource group associated with workspace

telemetry_opt_out: bool (optional)

A boolean parameter. Defaults to logging a version of AzureMLCluster with Microsoft. Set this flag to False if you do not want to share this information with Microsoft. Microsoft is not tracking anything else you do in your Dask cluster nor any other information related to your workload.

asynchronous: bool (optional)

Flag to run jobs asynchronously.

**kwargs: dict

Additional keyword arguments.

Attributes:
asynchronous
dashboard_link

Link to Dask dashboard.

jupyter_link

Link to JupyterLab on running on the headnode of the cluster.

observed
plan
requested
scheduler_address

Methods

adapt([Adaptive]) Turn on adaptivity
close() Close the cluster.
get_logs([cluster, scheduler, workers]) Return logs for the cluster, scheduler and workers
scale([workers]) Scale the cluster.
scale_down([workers]) Scale down the number of workers.
scale_up([workers]) Scale up the number of workers.
close_when_disconnect  
logs  
sync  
close()[source]

Close the cluster. All Azure ML Runs corresponding to the scheduler and worker processes will be completed. The Azure ML Compute Target will return to its minimum number of nodes after its idle time before scaledown.

Link to Dask dashboard.

Link to JupyterLab on running on the headnode of the cluster. Set jupyter=True when creating the AzureMLCluster.

scale(workers=1)[source]

Scale the cluster. Scales to a maximum of the workers available in the cluster.

scale_down(workers=1)[source]

Scale down the number of workers. Scales to minimum of 1.

scale_up(workers=1)[source]

Scale up the number of workers.

class dask_cloudprovider.ECSCluster(fargate_scheduler=False, fargate_workers=False, image=None, scheduler_cpu=None, scheduler_mem=None, scheduler_timeout=None, scheduler_extra_args=None, worker_cpu=None, worker_mem=None, worker_gpu=None, worker_extra_args=None, n_workers=None, cluster_arn=None, cluster_name_template=None, execution_role_arn=None, task_role_arn=None, task_role_policies=None, cloudwatch_logs_group=None, cloudwatch_logs_stream_prefix=None, cloudwatch_logs_default_retention=None, vpc=None, subnets=None, security_groups=None, environment=None, tags=None, find_address_timeout=None, skip_cleanup=None, aws_access_key_id=None, aws_secret_access_key=None, region_name=None, platform_version=None, fargate_use_private_ip=False, mount_points=None, volumes=None, mount_volumes_on_scheduler=False, **kwargs)[source]

Deploy a Dask cluster using ECS

This creates a dask scheduler and workers on an ECS cluster. If you do not configure a cluster one will be created for you with sensible defaults.

Parameters:
fargate_scheduler: bool (optional)

Select whether or not to use fargate for the scheduler.

Defaults to False. You must provide an existing cluster.

fargate_workers: bool (optional)

Select whether or not to use fargate for the workers.

Defaults to False. You must provide an existing cluster.

image: str (optional)

The docker image to use for the scheduler and worker tasks.

Defaults to daskdev/dask:latest or rapidsai/rapidsai:latest if worker_gpu is set.

scheduler_cpu: int (optional)

The amount of CPU to request for the scheduler in milli-cpu (1/1024).

Defaults to 1024 (one vCPU).

scheduler_mem: int (optional)

The amount of memory to request for the scheduler in MB.

Defaults to 4096 (4GB).

scheduler_timeout: str (optional)

The scheduler task will exit after this amount of time if there are no clients connected.

Defaults to 5 minutes.

scheduler_extra_args: List[str] (optional)

Any extra command line arguments to pass to dask-scheduler, e.g. ["--tls-cert", "/path/to/cert.pem"]

Defaults to None, no extra command line arguments.

worker_cpu: int (optional)

The amount of CPU to request for worker tasks in milli-cpu (1/1024).

Defaults to 4096 (four vCPUs).

worker_mem: int (optional)

The amount of memory to request for worker tasks in MB.

Defaults to 16384 (16GB).

worker_gpu: int (optional)

The number of GPUs to expose to the worker.

To provide GPUs to workers you need to use a GPU ready docker image that has dask-cuda installed and GPU nodes available in your ECS cluster. Fargate is not supported at this time.

Defaults to None, no GPUs.

worker_extra_args: List[str] (optional)

Any extra command line arguments to pass to dask-worker, e.g. ["--tls-cert", "/path/to/cert.pem"]

Defaults to None, no extra command line arguments.

n_workers: int (optional)

Number of workers to start on cluster creation.

Defaults to None.

cluster_arn: str (optional if fargate is true)

The ARN of an existing ECS cluster to use for launching tasks.

Defaults to None which results in a new cluster being created for you.

cluster_name_template: str (optional)

A template to use for the cluster name if cluster_arn is set to None.

Defaults to 'dask-{uuid}'

execution_role_arn: str (optional)

The ARN of an existing IAM role to use for ECS execution.

This ARN must have sts:AssumeRole allowed for ecs-tasks.amazonaws.com and allow the following permissions:

  • ecr:GetAuthorizationToken
  • ecr:BatchCheckLayerAvailability
  • ecr:GetDownloadUrlForLayer
  • ecr:GetRepositoryPolicy
  • ecr:DescribeRepositories
  • ecr:ListImages
  • ecr:DescribeImages
  • ecr:BatchGetImage
  • logs:*
  • ec2:AuthorizeSecurityGroupIngress
  • ec2:Describe*
  • elasticloadbalancing:DeregisterInstancesFromLoadBalancer
  • elasticloadbalancing:DeregisterTargets
  • elasticloadbalancing:Describe*
  • elasticloadbalancing:RegisterInstancesWithLoadBalancer
  • elasticloadbalancing:RegisterTargets

Defaults to None (one will be created for you).

task_role_arn: str (optional)

The ARN for an existing IAM role for tasks to assume. This defines which AWS resources the dask workers can access directly. Useful if you need to read from S3 or a database without passing credentials around.

Defaults to None (one will be created with S3 read permission only).

task_role_policies: List[str] (optional)

If you do not specify a task_role_arn you may want to list some IAM Policy ARNs to be attached to the role that will be created for you.

E.g if you need your workers to read from S3 you could add arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess.

Default None (no policies will be attached to the role)

cloudwatch_logs_group: str (optional)

The name of an existing cloudwatch log group to place logs into.

Default None (one will be created called dask-ecs)

cloudwatch_logs_stream_prefix: str (optional)

Prefix for log streams.

Defaults to the cluster name.

cloudwatch_logs_default_retention: int (optional)

Retention for logs in days. For use when log group is auto created.

Defaults to 30.

vpc: str (optional)

The ID of the VPC you wish to launch your cluster in.

Defaults to None (your default VPC will be used).

subnets: List[str] (optional)

A list of subnets to use when running your task.

Defaults to None. (all subnets available in your VPC will be used)

security_groups: List[str] (optional)

A list of security group IDs to use when launching tasks.

Defaults to None (one will be created which allows all traffic between tasks and access to ports 8786 and 8787 from anywhere).

environment: dict (optional)

Extra environment variables to pass to the scheduler and worker tasks.

Useful for setting EXTRA_APT_PACKAGES, EXTRA_CONDA_PACKAGES and `EXTRA_PIP_PACKAGES if you’re using the default image.

Defaults to None.

tags: dict (optional)

Tags to apply to all resources created automatically.

Defaults to None. Tags will always include {"createdBy": "dask-cloudprovider"}

find_address_timeout: int

Configurable timeout in seconds for finding the task IP from the cloudwatch logs.

Defaults to 60 seconds.

skip_cleanup: bool (optional)

Skip cleaning up of stale resources. Useful if you have lots of resources and this operation takes a while.

Default False.

platform_version: str (optional)

Version of the AWS Fargate platform to use, e.g. “1.4.0” or “LATEST”. This setting has no effect for the EC2 launch type.

Defaults to None

fargate_use_private_ip: bool (optional)

Whether to use a private IP (if True) or public IP (if False) with Fargate.

Default False.

mount_points: list (optional)

List of mount points as documented here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/efs-volumes.html

Default None.

volumes: list (optional)

List of volumes as documented here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/efs-volumes.html

Default None.

mount_volumes_on_scheduler: bool (optional)

Whether to also mount volumes in the scheduler task. Any volumes and mount points specified will always be mounted in worker tasks. This setting controls whether volumes are also mounted in the scheduler task.

Default False.

**kwargs: dict

Additional keyword arguments to pass to SpecCluster.

Attributes:
asynchronous
dashboard_link
observed
plan
requested
scheduler_address
tags

Methods

adapt(*args[, minimum, maximum]) Turn on adaptivity
get_logs([cluster, scheduler, workers]) Return logs for the cluster, scheduler and workers
new_worker_spec() Return name and spec for the next worker
scale([n, memory, cores]) Scale cluster to n workers
scale_up([n, memory, cores]) Scale cluster to n workers
close  
logs  
scale_down  
sync  
class dask_cloudprovider.FargateCluster(**kwargs)[source]

Deploy a Dask cluster using Fargate on ECS

This creates a dask scheduler and workers on a Fargate powered ECS cluster. If you do not configure a cluster one will be created for you with sensible defaults.

Parameters:
kwargs: dict

Keyword arguments to be passed to ECSCluster.

Attributes:
asynchronous
dashboard_link
observed
plan
requested
scheduler_address
tags

Methods

adapt(*args[, minimum, maximum]) Turn on adaptivity
get_logs([cluster, scheduler, workers]) Return logs for the cluster, scheduler and workers
new_worker_spec() Return name and spec for the next worker
scale([n, memory, cores]) Scale cluster to n workers
scale_up([n, memory, cores]) Scale cluster to n workers
close  
logs  
scale_down  
sync