queens.stochastic_optimizers package

Stochastic optimizers.

Modules containing stochastic optimizers.

Submodules

queens.stochastic_optimizers.adam module

Adam optimizer.

class Adam(learning_rate, optimization_type, rel_l1_change_threshold, rel_l2_change_threshold, clip_by_l2_norm_threshold=inf, clip_by_value_threshold=inf, max_iteration=1000000.0, beta_1=0.9, beta_2=0.999, eps=1e-08, learning_rate_decay=None)

Bases: StochasticOptimizer

Adam stochastic optimizer [1].

References

[1] Kingma and Ba. “Adam: A Method for Stochastic Optimization”. ICLR 2015. 2015.

beta_1

\(\beta_1\) parameter as described in [1].

Type:

float

beta_2

\(\beta_2\) parameter as described in [1].

Type:

float

m

Exponential average of the gradient.

Type:

ExponentialAveragingObject

v

Exponential average of the gradient momentum.

Type:

ExponentialAveragingObject

eps

Nugget term to avoid a division by values close to zero.

Type:

float

scheme_specific_gradient(gradient)

Adam gradient computation.

Parameters:

gradient (np.array) – Gradient

Returns:

gradient (np.array) – Adam gradient

queens.stochastic_optimizers.adamax module

Adamax optimizer.

class Adamax(learning_rate, optimization_type, rel_l1_change_threshold, rel_l2_change_threshold, clip_by_l2_norm_threshold=inf, clip_by_value_threshold=inf, max_iteration=1000000.0, beta_1=0.9, beta_2=0.999, eps=1e-08, learning_rate_decay=None)

Bases: StochasticOptimizer

Adamax stochastic optimizer [1]. eps added to avoid division by zero.

References

[1] Kingma and Ba. “Adam: A Method for Stochastic Optimization”. ICLR 2015. 2015.

beta_1

\(\beta_1\) parameter as described in [1].

Type:

float

beta_2

\(\beta_2\) parameter as described in [1].

Type:

float

m

Exponential average of the gradient.

Type:

ExponentialAveragingObject

u

Maximum gradient momentum.

Type:

np.array

eps

Nugget term to avoid a division by values close to zero.

Type:

float

scheme_specific_gradient(gradient)

Adamax gradient computation.

Parameters:

gradient (np.array) – Gradient

Returns:

gradient (np.array) – Adam gradient

queens.stochastic_optimizers.learning_rate_decay module

Learning rate decay for stochastic optimization.

class DynamicLearningRateDecay(alpha=0.1, rho_min=1.0)

Bases: LearningRateDecay

Dynamic learning rate decay.

alpha

Decay factor

Type:

float

rho_min

Threshold for signal-to-noise ratio

Type:

float

k_min

Minimum number of iterations before learning rate is decreased

Type:

int

k

Iteration number

Type:

int

a

Sum of parameters

Type:

np.array

b

Sum of squared parameters

Type:

np.array

c

Sum of parameters times iteration number

Type:

np.array

class LearningRateDecay

Bases: object

Base class for learning rate decay.

class LogLinearLearningRateDecay(slope)

Bases: LearningRateDecay

Log linear learning rate decay.

slope

Logarithmic slope

Type:

float

iteration

Current iteration

Type:

int

class StepwiseLearningRateDecay(decay_factor, decay_interval)

Bases: LearningRateDecay

Step-wise learning rate decay.

decay_factor

Decay factor

Type:

float

decay_interval

Decay interval

Type:

int

iteration

Iteration number

Type:

int

queens.stochastic_optimizers.rms_prop module

RMSprop optimizer.

class RMSprop(learning_rate, optimization_type, rel_l1_change_threshold, rel_l2_change_threshold, clip_by_l2_norm_threshold=inf, clip_by_value_threshold=inf, max_iteration=1000000.0, beta=0.999, eps=1e-08, learning_rate_decay=None)

Bases: StochasticOptimizer

RMSprop stochastic optimizer [1].

References

[1] Tieleman and Hinton. “Lecture 6.5-rmsprop: Divide the gradient by a running average of

its recent magnitude”. Coursera. 2012.

beta

\(\beta\) parameter as described in [1].

Type:

float

v

Exponential average of the gradient momentum.

Type:

ExponentialAveragingObject

eps

Nugget term to avoid a division by values close to zero.

Type:

float

scheme_specific_gradient(gradient)

Rmsprop gradient computation.

Parameters:

gradient (np.array) – Gradient

Returns:

gradient (np.array) – RMSprop gradient

queens.stochastic_optimizers.sgd module

SGD optimizer.

class SGD(learning_rate, optimization_type, rel_l1_change_threshold, rel_l2_change_threshold, clip_by_l2_norm_threshold=inf, clip_by_value_threshold=inf, max_iteration=1000000.0, learning_rate_decay=None)

Bases: StochasticOptimizer

Stochastic gradient descent optimizer.

scheme_specific_gradient(gradient)

SGD gradient computation.

Parameters:

gradient (np.array) – Gradient

Returns:

gradient (np.array) – SGD gradient