queens.schedulers package#

Schedulers.

Modules for scheduling and submitting computational jobs.

Submodules#

queens.schedulers.cluster module#

Cluster scheduler for QUEENS runs.

class Cluster[source]#

Bases: Dask

Cluster scheduler for QUEENS.

__init__(experiment_name, workload_manager, walltime, remote_connection, num_jobs=1, min_jobs=0, num_procs=1, num_nodes=1, queue=None, cluster_internal_address=None, restart_workers=False, allowed_failures=5, verbose=True, experiment_base_dir=None, overwrite_existing_experiment=False)[source]#

Init method for the cluster scheduler.

The total number of cores per job is given by num_procs*num_nodes.

Parameters:

  • experiment_name (str) – Name of the current experiment

  • workload_manager (str) – Workload manager (“pbs” or “slurm”)

  • walltime (str) – Walltime for each worker job. Format (hh:mm:ss)

  • remote_connection (RemoteConnection) – SSH connection to the remote host

  • num_jobs (int, opt) – Maximum number of parallel jobs

  • min_jobs (int, opt) – Minimum number of active workers for the cluster

  • num_procs (int, opt) – Number of processors per job per node

  • num_nodes (int, opt) – Number of cluster nodes per job

  • queue (str, opt) – Destination queue for each worker job

  • cluster_internal_address (str, opt) – Internal address of cluster

  • restart_workers (bool) – If True, restart workers after each finished job. For larger jobs (>1min) this should be set to True in most cases.

  • allowed_failures (int) – Number of allowed failures for a task before an error is raised

  • verbose (bool, opt) – Verbosity of evaluations. Defaults to True.

  • experiment_base_dir (str, Path) – Base directory for the simulation outputs

  • overwrite_existing_experiment (bool) – If True, overwrite experiment directory if it exists already. If False, prompt user for confirmation before overwriting.

copy_files_to_experiment_dir(paths)[source]#

Copy file to experiment directory.

Parameters:

paths (Path, list) – Paths to files or directories that should be copied to experiment directory

local_experiment_dir(experiment_name, experiment_base_dir, overwrite_existing_experiment)[source]#

Get the local experiment directory.

Parameters:

  • experiment_name (str) – name of the current experiment

  • experiment_base_dir (str, Path) – Base directory for the simulation outputs

  • overwrite_existing_experiment (bool) – If true, continue and overwrite experiment directory. If false, prompt user for confirmation before continuing and overwriting.

Returns:

experiment_dir (Path) – Path to local experiment directory.

remote_experiment_dir(experiment_name, experiment_base_dir, overwrite_existing_experiment)[source]#

Get experiment directory on remote host.

Parameters:

  • experiment_name (str) – name of the current experiment

  • experiment_base_dir (str, Path) – Remote base directory for the simulation outputs

  • overwrite_existing_experiment (bool) – If true, continue and overwrite experiment directory. If false, prompt user for confirmation before continuing and overwriting.

Returns:

experiment_dir (Path) – Path to experiment directory on remote host.

restart_worker(worker)[source]#

Restart a worker.

This method retires a dask worker. The Client.adapt method of dask takes cares of submitting new workers subsequently.

Parameters:

worker (str, tuple) – Worker to restart. This can be a worker address, name, or a both.

timedelta_to_str(timedelta_obj)[source]#

Format a timedelta object to str.

This function seems unnecessarily complicated, but unfortunately the datetime library does not

support this formatting for timedeltas. Returns the format HH:MM:SS.

Parameters:

timedelta_obj (datetime.timedelta) – Timedelta object to format

Returns:

str – String of the timedelta object

queens.schedulers.local module#

Local scheduler for QUEENS runs.

class Local[source]#

Bases: Dask

Local scheduler class for QUEENS.

__init__(experiment_name, num_jobs=1, num_procs=1, restart_workers=False, verbose=True, experiment_base_dir=None, overwrite_existing_experiment=False)[source]#

Initialize local scheduler.

Parameters:

  • experiment_name (str) – Name of the current experiment

  • num_jobs (int, opt) – Maximum number of parallel jobs

  • num_procs (int, opt) – Number of processors per job

  • restart_workers (bool) – If True, restart workers after each finished job. Try setting it to True in case you are experiencing memory-leakage warnings.

  • verbose (bool, opt) – Verbosity of evaluations. Defaults to True.

  • experiment_base_dir (str, Path) – Base directory for the simulation outputs

  • overwrite_existing_experiment (bool) – If True, overwrite experiment directory if it exists already. If False, prompt user for confirmation before overwriting.

restart_worker(worker)[source]#

Restart a worker.

Parameters:

worker (str, tuple) – Worker to restart. This can be a worker address, name, or a both.

queens.schedulers.pool module#

Pool scheduler for QUEENS runs.

class Pool[source]#

Bases: Scheduler

Pool scheduler class for QUEENS.

pool#

Multiprocessing pool.

Type:

pathos pool

__init__(experiment_name, num_jobs=1, verbose=True, experiment_base_dir=None, overwrite_existing_experiment=False)[source]#

Initialize Pool.

Parameters:

  • experiment_name (str) – Name of the current experiment

  • num_jobs (int, opt) – Maximum number of parallel jobs

  • verbose (bool, opt) – Verbosity of evaluations. Defaults to True.

  • experiment_base_dir (str, Path) – Base directory for the simulation outputs

  • overwrite_existing_experiment (bool) – If True, overwrite experiment directory if it exists already. If False, prompt user for confirmation before overwriting.

evaluate(samples, function, job_ids=None)[source]#

Submit jobs to driver.

Parameters:

  • samples (np.array) – Array of samples

  • function (Callable) – Callable to evaluate in the scheduler

  • job_ids (lst, opt) – List of job IDs corresponding to samples

Returns:

result_dict (dict) – Dictionary containing results

Return type:

dict