Contents

Openstack

Openstack

OpenStackCluster([region, size, image, ...])

Cluster running on Openstack VM Instances

Overview

Authentication

To authenticate with the OpenStack Identity service (Keystone)

  1. Get your Authentication URL (auth_url) for OpenStack Identity service (Keystone) and put it in your Dask configuration at cloudprovider.openstack.auth_url.

  2. Get your region and put it in your Dask configuration at cloudprovider.openstack.region.
    $ openstack region list
+-----------+---------------+-------------+
| Region    | Parent Region | Description |
+-----------+---------------+-------------+
| RegionOne | None          |             |
+-----------+---------------+-------------+

  3. Generate an application credential.

    $ openstack application credential create dask --unrestricted
+--------------+----------------------------------------------------------------------------------------+
| Field        | Value                                                                                  |
+--------------+----------------------------------------------------------------------------------------+
| description  | None                                                                                   |
| expires_at   | None                                                                                   |
| id           | 0a0372dbedfb4e82ab66449c3316ef1e                                                       |
| name         | dask                                                                             |
| project_id   | e99b6f4b9bf84a9da27e20c9cbfe887a                                                       |
| roles        | Member anotherrole                                                                     |
| secret       | ArOy6DYcLeLTRlTmfvF1TH1QmRzYbmD91cbVPOHL3ckyRaLXlaq5pTGJqvCvqg6leEvTI1SQeX3QK-3iwmdPxg |
| unrestricted | True                                                                                   |
+--------------+----------------------------------------------------------------------------------------+

    and put application_credential_id and application_credential_secret in your Dask configuration at cloudprovider.openstack.application_credential_id and cloudprovider.openstack.application_credential_secret.

All of this variables can be gathered from either OpenStack RC file or clouds.yaml file.

Example Config File

# ~/.config/dask/cloudprovider.yaml

 cloudprovider:
   openstack:
     region: "RegionOne"
     auth_url: "https://cloud.home.karatosun.xyz:5000"
     application_credential_id: "0a0372dbedfb4e82ab66449c3316ef1e"
     application_credential_secret: "ArOy6DYcLeLTRlTmfvF1TH1QmRzYbmD91cbVPOHL3ckyRaLXlaq5pTGJqvCvqg6leEvTI1SQeX3QK-3iwmdPxg"
     auth_type: "v3applicationcredential"

You can also export them as environment variables.

$ export DASK_CLOUDPROVIDER__APPLICATION_CREDENTIAL_ID="0a0372dbedfb4e82ab66449c3316ef1e"
class dask_cloudprovider.openstack.OpenStackCluster(region: str = None, size: str = None, image: str = None, docker_image: str = None, debug: bool = False, bootstrap: bool = True, worker_threads: int = 2, worker_command: str = None, **kwargs)[source]

Cluster running on Openstack VM Instances

This cluster manager constructs a Dask cluster running on generic Openstack cloud

When configuring your cluster you may find it useful to install the ‘python-openstackclient’ client for querying the Openstack APIs for available options.

https://github.com/openstack/python-openstackclient

Parameters
region: str

The name of the region where resources will be allocated in OpenStack. Typically set to ‘default’ unless specified in your cloud configuration.

List available regions using: openstack region list.

auth_url: str

The authentication URL for the OpenStack Identity service (Keystone). Example: https://cloud.example.com:5000

application_credential_id: str

The application credential id created in OpenStack.

Create application credentials using: openstack application credential create

application_credential_secret: str

The secret associated with the application credential ID for authentication.

auth_type: str

The type of authentication used, typically “v3applicationcredential” for using OpenStack application credentials.

network_id: str

The unique identifier for the internal/private network in OpenStack where the cluster VMs will be connected.

List available networks using: openstack network list

image: str

The OS image name or id to use for the VM. Dask Cloudprovider will boostrap Ubuntu based images automatically. Other images require Docker and for GPUs the NVIDIA Drivers and NVIDIA Docker.

List available images using: openstack image list

keypair_name: str

The name of the SSH keypair used for instance access. Ensure you have created a keypair or use an existing one.

List available keypairs using: openstack keypair list

security_group: str

The security group name that defines firewall rules for instances.

The default is default. Please ensure the follwing accesses are configured:

  • egress 0.0.0.0/0 on all ports for downloading docker images and general data access

  • ingress <internal-cidr>/8 on all ports for internal communication of workers

  • ingress 0.0.0.0/0 on 8786-8787 for external accessibility of the dashboard/scheduler

  • (optional) ingress 0.0.0.0./0 on 22 for ssh access

List available security groups using: openstack security group list

create_floating_ip: bool

Specifies whether to assign a floating IP to each instance, enabling external access. Set to True if external connectivity is needed.

external_network_id: str

The ID of the external network used for assigning floating IPs.

List available external networks using: openstack network list –external

n_workers: int (optional)

Number of workers to initialise the cluster with. Defaults to 0.

worker_module: str

The Python module to run for the worker. Defaults to distributed.cli.dask_worker

worker_options: dict

Params to be passed to the worker class. See distributed.worker.Worker for default worker class. If you set worker_module then refer to the docstring for the custom worker class.

worker_threads: int

The number of threads to use on each worker.

worker_commandstr (optional)

The command workers should run when starting. By default this will be python -m distributed.cli.dask_spec, but you can override it—for example, to dask-cuda-worker on GPU-enabled instances.

Attributes
asynchronous

Are we running in the event loop?

auto_shutdown
bootstrap
called_from_running_loop
command
dashboard_link
docker_image
gpu_instance
loop
name
observed
plan
requested
scheduler_address
scheduler_class
worker_class

Methods

adapt([Adaptive, minimum, maximum, ...])

Turn on adaptivity

call_async(f, *args, **kwargs)

Run a blocking function in a thread as a coroutine.

from_name(name)

Create an instance of this class to represent an existing cluster by name.

get_client()

Return client for the cluster

get_logs([cluster, scheduler, workers])

Return logs for the cluster, scheduler and workers

get_tags()

Generate tags to be applied to all resources.

new_worker_spec()

Return name and spec for the next worker

scale([n, memory, cores])

Scale cluster to n workers

scale_up([n, memory, cores])

Scale cluster to n workers

sync(func, *args[, asynchronous, ...])

Call func with args synchronously or asynchronously depending on the calling context

wait_for_workers(n_workers[, timeout])

Blocking call to wait for n workers before continuing

close

get_cloud_init

logs

render_cloud_init

render_process_cloud_init

scale_down

previous

IBM Cloud

next

Nebius