queens.schedulers package#

Schedulers.

Modules for scheduling and submitting computational jobs.

Submodules#

queens.schedulers.cluster module#

Cluster scheduler for QUEENS runs.

class Cluster[source]#

Bases: _BaseCluster

Cluster (remote) scheduler for QUEENS.

__init__(experiment_name, workload_manager, walltime, remote_connection, num_jobs=1, min_jobs=0, num_procs=1, num_nodes=1, queue=None, cluster_internal_address=None, restart_workers=False, allowed_failures=5, verbose=True, experiment_base_dir=None, overwrite_existing_experiment=False, job_script_prologue=None)[source]#

Init method for the remote cluster scheduler.

The total number of cores per job is given by num_procs*num_nodes.

Parameters:

  • experiment_name (str) – Name of the current experiment

  • workload_manager (str) – Workload manager (“pbs” or “slurm”)

  • walltime (str) – Walltime for each worker job. Format (hh:mm:ss)

  • remote_connection (RemoteConnection) – SSH connection to the remote host

  • num_jobs (int, opt) – Maximum number of parallel jobs

  • min_jobs (int, opt) – Minimum number of active workers for the cluster

  • num_procs (int, opt) – Number of processors per job per node

  • num_nodes (int, opt) – Number of cluster nodes per job

  • queue (str, opt) – Destination queue for each worker job

  • cluster_internal_address (str, opt) – Internal address of cluster

  • restart_workers (bool) – If True, restart workers after each finished job. For larger jobs (>1min) this should be set to True in most cases.

  • allowed_failures (int) – Number of allowed failures for a task before an error is raised

  • verbose (bool, opt) – Verbosity of evaluations. Defaults to True.

  • experiment_base_dir (str, Path) – Base directory for the simulation outputs

  • overwrite_existing_experiment (bool) – If True, overwrite experiment directory if it exists already. If False, prompt user for confirmation before overwriting.

  • job_script_prologue (list, opt) – List of commands to be executed before starting a worker.

copy_files_from_experiment_dir(destination=None, verbose=True, exclude=None, filters=None)[source]#

Copy files from remote experiment directory to the local machine.

Parameters:

  • destination (Path) – Path to the local directory where the files from the remote experiment directory should be copied to. If None, the default base directory ~/queens-experiments/ is used.

  • verbose (bool) – True for verbose

  • exclude (str | Sequence | None) – Options to exclude

  • filters (str | None) – Filters for rsync

copy_files_to_experiment_dir(paths)[source]#

Copy file to experiment directory.

Parameters:

paths (Path, list) – Paths to files or directories that should be copied to experiment directory

queens.schedulers.cluster_local module#

Cluster scheduler for QUEENS runs.

class ClusterLocal[source]#

Bases: _BaseCluster

Cluster scheduler with local access (no network connection).

copy_files_to_experiment_dir(paths)[source]#

Copy file to experiment directory.

queens.schedulers.local module#

Local scheduler for QUEENS runs.

class Local[source]#

Bases: Dask

Local scheduler class for QUEENS.

__init__(experiment_name, num_jobs=1, num_procs=1, restart_workers=False, verbose=True, experiment_base_dir=None, overwrite_existing_experiment=False)[source]#

Initialize local scheduler.

Parameters:

  • experiment_name (str) – Name of the current experiment

  • num_jobs (int, opt) – Maximum number of parallel jobs

  • num_procs (int, opt) – Number of processors per job

  • restart_workers (bool) – If True, restart workers after each finished job. Try setting it to True in case you are experiencing memory-leakage warnings.

  • verbose (bool, opt) – Verbosity of evaluations. Defaults to True.

  • experiment_base_dir (str, Path) – Base directory for the simulation outputs

  • overwrite_existing_experiment (bool) – If True, overwrite experiment directory if it exists already. If False, prompt user for confirmation before overwriting.

restart_worker(worker)[source]#

Restart a worker.

Parameters:

worker (str, tuple) – Worker to restart. This can be a worker address, name, or a both.

queens.schedulers.pool module#

Pool scheduler for QUEENS runs.

class Pool[source]#

Bases: Scheduler

Pool scheduler class for QUEENS.

pool#

Multiprocessing pool.

Type:

pathos pool

__init__(experiment_name, num_jobs=1, verbose=True, experiment_base_dir=None, overwrite_existing_experiment=False)[source]#

Initialize Pool.

Parameters:

  • experiment_name (str) – Name of the current experiment

  • num_jobs (int, opt) – Maximum number of parallel jobs

  • verbose (bool, opt) – Verbosity of evaluations. Defaults to True.

  • experiment_base_dir (str, Path) – Base directory for the simulation outputs

  • overwrite_existing_experiment (bool) – If True, overwrite experiment directory if it exists already. If False, prompt user for confirmation before overwriting.

evaluate(samples, function, job_ids=None)[source]#

Submit jobs to driver.

Parameters:

  • samples (np.array) – Array of samples

  • function (Callable) – Callable to evaluate in the scheduler

  • job_ids (lst, opt) – List of job IDs corresponding to samples

Returns:

result_dict (dict) – Dictionary containing results

Return type:

dict