queens.models.reinforcement_learning.utils package#

Utils.

Utility functionality related to the reinforcement learning libraries stable-baselines3 and gymnasium.

Submodules#

queens.models.reinforcement_learning.utils.gymnasium module#

Utility functions for working with gymnasium environments.

This module essentially provides a function to create a gymnasium environment from its name to facilitate the creation of gymnasium environments in QUEENS for users who are not familiar with the package.

create_gym_environment(env_name, env_options=None, seed=None)[source]#

Convenience function to create a gymnasium environment.

Parameters:

  • env_name (str) – Name of the gymnasium environment to create.

  • env_options (dict, optional) – Dictionary of options to pass to the environment.

  • seed (int, optional) – Seed to use for the environment.

Returns:

env (gymnasium.Env) – An instance of the created gymnasium environment.

Raises:

InvalidOptionError – If the provided environment name is not known to gymnasium.

queens.models.reinforcement_learning.utils.stable_baselines3 module#

Utiltiy functions for working with stable-baselines3 agents.

This module provides utility functions for working with stable- baselines3 agents in QUEENS. The idea is to enable a QUEENS user who is not familiar with the stable-baselines3 reinforcement learning library but wants to try out RL for their problem to easily create and use a reinforcement learning agent in QUEENS without first studying the package. If you are familiar with stable-baselines3, you can as well create the agents yourself.

create_sb3_agent(agent_name, policy_name, env, agent_options=None)[source]#

Creates a stable-baselines3 agent based on its name as string.

Looks up whether the provided agent name corresponds to an agent supported by stable-baselines3 and creates an instance of the agent with the provided policy and environment. Options for modifying the agent’s optional parameters can be provided as a dictionary.

Parameters:

  • agent_name (str) – The name of the agent to create.

  • policy_name (str) – The name of the policy to use with the agent.

  • env (gymnasium.Env) – The environment to train the agent on. For a convenience function to create a predefined gymnasium environment, see queens.models.reinforcement_learning.utils.gymnasium.create_gym_environment().

  • agent_options (dict, optional) – A dictionary of optional parameters to pass to the agent.

Returns:

agent (stable_baselines3.BaseAlgorithm) – An instance of the created agent.

Raises:

  • ValueError – If the provided agent name is not supported by stable-baselines3

  • InvalidOptionError – If the provided agent name is not known to stable-baselines3 or the provided policy name is not supported by the chosen agent class.

get_supported_sb3_policies(agent_class)[source]#

Looks up the supported policies for a stable-baselines3 agent class.

Parameters:

agent_class (class) – A stable-baselines3 agent class.

Returns:

list – A list of strings representing the supported policies for the given agent class.

Raises:

ValueError – If the provided class is not a stable-baselines3 agent class or does not provide a policy_aliases attribute.

load_model(agent_name, path, experiment_name, env=None)[source]#

Convenience function for loading a stable-baselines3 agent from file.

Checks whether the agent is of an off-policy type and if so loads its replay buffer as well. The latter is required if a continuation of the training is desired.

Parameters:

  • agent_name (str) – The name of the agent to load.

  • path (str, Path) – The path to the directory containing the agent to load.

  • experiment_name (str) – The name of the QUEENS experiment that was used to train the agent (contained in the filename).

  • env (gymnasium.Env, optional) – The environment on which the agent was trained.

Returns:

agent (stable_baselines3.BaseAlgorithm) – The loaded agent.

make_deterministic(seed, disable_gpu=True)[source]#

Make the random number generation deterministic for an agent training.

This is achieved by setting the random seed for all python libraries that are involved in training and stable-baselines3 agent.

Note

This function should be called before creating the agent. Make sure to also pass a seed to the agent and the environment.

  • For the environment, you wanna invoke the reset() method after creation and pass the seed as parameter, e.g., env.reset(seed=seed). This needs to be done before passing the environment to the agent.

  • For the agent, this can be achieved by adding the entry "seed": seed, to the agent_options dict.

Warning

Since GPU computations can result in non-deterministic computations, this functions modifies the CUDA_VISIBLE_DEVICES environment variable to disable GPU computations. This behavior can be changed by adapting the disable_gpu parameter.

Parameters:

  • seed (int) – The random seed to set for all libraries.

  • disable_gpu (bool, optional) – Flag indicating whether to disable GPU computations. Defaults to True.

save_model(agent, gs)[source]#

Save a (trained) stable-baselines3 agent to a file.

Checks whether the agent is of an off-policy type and if so stores its replay buffer as well. The latter is required if a continuation of the training is desired.

Parameters:

  • agent (stable_baselines3.BaseAlgorithm) – The trained agent to save.

  • gs (GlobalSettings) – The global settings of the QUEENS experiment (needed to retrieve the experiment name and the output directory of the current run).