queens.schedulers package#

Schedulers.

Modules for scheduling and submitting computational jobs.

Submodules#

queens.schedulers.cluster_scheduler module#

Cluster scheduler for QUEENS runs.

class ClusterScheduler(experiment_name, workload_manager, walltime, remote_connection, num_jobs=1, min_jobs=0, num_procs=1, num_nodes=1, queue=None, cluster_internal_address=None, restart_workers=False, allowed_failures=5, verbose=True)[source]#

Bases: DaskScheduler

Cluster scheduler for QUEENS.

copy_files_to_experiment_dir(paths)[source]#

Copy file to experiment directory.

Parameters:: paths (Path, list) – paths to files or directories that should be copied to experiment directory

restart_worker(worker)[source]#

Restart a worker.

This method retires a dask worker. The Client.adapt method of dask takes cares of submitting new workers subsequently.

Parameters:: worker (str, tuple) – Worker to restart. This can be a worker address, name, or a both.

timedelta_to_str(timedelta_obj)[source]#

Format a timedelta object to str.

This function seems unnecessarily complicated, but unfortunately the datetime library does not: support this formatting for timedeltas. Returns the format HH:MM:SS.

Parameters:: timedelta_obj (datetime.timedelta) – Timedelta object to format
Returns:: str – String of the timedelta object

queens.schedulers.dask_scheduler module#

QUEENS dask scheduler parent class.

class DaskScheduler(experiment_name, experiment_dir, num_jobs, num_procs, client, restart_workers, verbose=True)[source]#

Bases: Scheduler

Abstract base class for schedulers in QUEENS.

num_procs#

number of processors per job

Type:: int

client#

Dask client that connects to and submits computation to a Dask cluster

Type:: Client

restart_workers#

If true, restart workers after each finished job

Type:: bool

evaluate(samples, driver, job_ids=None)[source]#

Submit jobs to driver.

Parameters:

samples (np.array) – Array of samples
driver (Driver) – Driver object that runs simulation
job_ids (lst, opt) – List of job IDs corresponding to samples

Returns:

result_dict (dict) – Dictionary containing results

abstract restart_worker(worker)[source]#: Restart a worker.

async shutdown_client()[source]#: Shutdown the DASK client.

queens.schedulers.local_scheduler module#

Local scheduler for QUEENS runs.

class LocalScheduler(experiment_name, num_jobs=1, num_procs=1, restart_workers=False, verbose=True)[source]#

Bases: DaskScheduler

Local scheduler class for QUEENS.

restart_worker(worker)[source]#

Restart a worker.

Parameters:: worker (str, tuple) – Worker to restart. This can be a worker address, name, or a both.

queens.schedulers.pool_scheduler module#

Pool scheduler for QUEENS runs.

class PoolScheduler(experiment_name, num_jobs=1, verbose=True)[source]#

Bases: Scheduler

Pool scheduler class for QUEENS.

pool#

Multiprocessing pool.

Type:: pathos pool

evaluate(samples, driver, job_ids=None)[source]#

Submit jobs to driver.

Parameters:

samples (np.array) – Array of samples
driver (Driver) – Driver object that runs simulation
job_ids (lst, opt) – List of job IDs corresponding to samples

Returns:

result_dict (dict) – Dictionary containing results

queens.schedulers.scheduler module#

QUEENS scheduler parent class.

class Scheduler(experiment_name, experiment_dir, num_jobs, verbose=True)[source]#

Bases: object

Abstract base class for schedulers in QUEENS.

experiment_name#

name of the current experiment

Type:: str

experiment_dir#

Path to QUEENS experiment directory.

Type:: Path

num_jobs#

Maximum number of parallel jobs

Type:: int

next_job_id#

Next job ID.

Type:: int

verbose#

Verbosity of evaluations

Type:: bool

copy_files_to_experiment_dir(paths)[source]#

Copy file to experiment directory.

Parameters:: paths (str, Path, list) – paths to files or directories that should be copied to experiment directory

abstract evaluate(samples, driver, job_ids=None)[source]#

Submit jobs to driver.

Parameters:

samples (np.array) – Array of samples
driver (Driver) – Driver object that runs simulation
job_ids (lst, opt) – List of job IDs corresponding to samples

Returns:

result_dict (dict) – Dictionary containing results

get_job_ids(num_samples)[source]#

Get job ids and update next_job_id.

Parameters:: num_samples (int) – Number of samples
Returns:: job_ids (np.array) – Array of job ids