Nebius

NebiusCluster([bootstrap, image_family, ...])

Cluster running on Nebius AI Cloud instances.

Overview

Authentication

Before creating clusters on Nebius, you must configure your authentication credentials. You can do this using the nebius command line tool.

After obtaining your credentials, add them to your Dask configuration under:

  • cloudprovider.nebius.token

  • cloudprovider.nebius.project_id

You can specify these values by either:

  1. Including the environment variables NB_IAM_TOKEN and NB_PROJECT_ID in your YAML configuration.

    # ~/.config/dask/cloudprovider.yaml
    
    cloudprovider:
      nebius:
          token: "your_iam_token"
          project_id: "your_project_id"
    
  2. Exporting them as environment variables in your shell.

    $ export DASK_CLOUDPROVIDER__NEBIUS__TOKEN=($nebius iam get-access-token)
    $ export DASK_CLOUDPROVIDER__NEBIUS__PROJECT_ID="your_project_id"
    

Dask Configuration

You can change configuration of server_platform, server_preset and image_family. List of all available platforms and presets you can find in Nebius docs.

class dask_cloudprovider.nebius.NebiusCluster(bootstrap: str = None, image_family: str = None, project_id: str = None, disk_size: int = None, server_platform: str = None, server_preset: str = None, docker_image: str = None, debug: bool = False, **kwargs)[source]

Cluster running on Nebius AI Cloud instances.

VMs in Nebius AI Cloud are referred to as instances. This cluster manager constructs a Dask cluster running on VMs.

When configuring your cluster you may find it useful to install the nebius tool for querying the Nebius API for available options.

https://docs.nebius.com/cli/quickstart

Parameters
image_family: str

The image to use for the host OS. This should be a Ubuntu variant. You find list available images here https://docs.nebius.com/compute/storage/manage#parameters-boot.

project_id: str

The Nebius AI Cloud project id. You can find in Nebius AI Cloud console.

server_platform: str

List of all platforms and presets here https://docs.nebius.com/compute/virtual-machines/types/.

server_preset: str

List of all platforms and presets here https://docs.nebius.com/compute/virtual-machines/types/.

n_workers: int

Number of workers to initialise the cluster with. Defaults to 0.

worker_module: str

The Python module to run for the worker. Defaults to distributed.cli.dask_worker

worker_options: dict

Params to be passed to the worker class. See distributed.worker.Worker for default worker class. If you set worker_module then refer to the docstring for the custom worker class.

scheduler_options: dict

Params to be passed to the scheduler class. See distributed.scheduler.Scheduler.

env_vars: dict

Environment variables to be passed to the worker.

extra_bootstrap: list[str] (optional)

Extra commands to be run during the bootstrap phase.

Attributes
asynchronous

Are we running in the event loop?

auto_shutdown
bootstrap
called_from_running_loop
command
dashboard_link
docker_image
gpu_instance
loop
name
observed
plan
requested
scheduler_address
scheduler_class
worker_class

Methods

adapt([Adaptive, minimum, maximum, ...])

Turn on adaptivity

call_async(f, *args, **kwargs)

Run a blocking function in a thread as a coroutine.

from_name(name)

Create an instance of this class to represent an existing cluster by name.

get_client()

Return client for the cluster

get_logs([cluster, scheduler, workers])

Return logs for the cluster, scheduler and workers

get_tags()

Generate tags to be applied to all resources.

new_worker_spec()

Return name and spec for the next worker

scale([n, memory, cores])

Scale cluster to n workers

scale_up([n, memory, cores])

Scale cluster to n workers

sync(func, *args[, asynchronous, ...])

Call func with args synchronously or asynchronously depending on the calling context

wait_for_workers(n_workers[, timeout])

Blocking call to wait for n workers before continuing

close

get_cloud_init

logs

render_cloud_init

render_process_cloud_init

scale_down