pymc.adagrad#

pymc.adagrad(loss_or_grads=None, params=None, learning_rate=1.0, epsilon=1e-06)[source]#

Adagrad updates.

Scale learning rates by dividing with the square root of accumulated squared gradients. See [1] for further description.

Parameters:
loss_or_grads: symbolic expression or list of expressions

A scalar loss expression, or a list of gradient expressions

params: list of shared variables

The variables to generate update expressions for

learning_rate: float or symbolic scalar

The learning rate controlling the size of update steps

epsilon: float or symbolic scalar

Small value added for numerical stability

Returns:
OrderedDict

A dictionary mapping each parameter to its update expression

Notes

Using step size eta Adagrad calculates the learning rate for feature i at time step t as:

\[\begin{split}\\eta_{t,i} = \\frac{\\eta} {\\sqrt{\\sum^t_{t^\\prime} g^2_{t^\\prime,i}+\\epsilon}} g_{t,i}\end{split}\]

as such the learning rate is monotonically decreasing.

Epsilon is not included in the typical formula, see [2].

Optimizer can be called without both loss_or_grads and params in that case partial function is returned

References

[1]

Duchi, J., Hazan, E., & Singer, Y. (2011): Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12:2121-2159.

[2]

Chris Dyer: Notes on AdaGrad. http://www.ark.cs.cmu.edu/cdyer/adagrad.pdf

Examples

>>> a = pytensor.shared(1.0)
>>> b = a * 2
>>> updates = adagrad(b, [a], learning_rate=0.01)
>>> isinstance(updates, dict)
True
>>> optimizer = adagrad(learning_rate=0.01)
>>> callable(optimizer)
True
>>> updates = optimizer(b, [a])
>>> isinstance(updates, dict)
True