hessQuik.activations

hessQuik Activation Function

class hessQuikActivationFunction(*args: Any, **kwargs: Any)

Bases: Module

Base class for all hessQuik activation functions.

forward(x: torch.Tensor, do_gradient: bool = False, do_Hessian: bool = False, forward_mode: bool = True) → Tuple[torch.Tensor, Optional[torch.Tensor], Optional[torch.Tensor]]

Applies a pointwise activation function to the incoming data.

Parameters:

• x (torch.Tensor) – input into the activation function. \((*)\) where \(*\) means any shape.

• do_gradient (bool, optional) – If set to True, the gradient will be computed during the forward call. Default: False

• do_Hessian (bool, optional) – If set to True, the Hessian will be computed during the forward call. Default: False

• forward_mode (bool, optional) – If set to False, the derivatives will be computed in backward mode. Default: True

Returns:

• sigma (torch.Tensor) - value of activation function at input x, same size as x

• dsigma (torch.Tensor or None) - first derivative of activation function at input x, same size as x

• d2sigma (torch.Tensor or None) - second derivative of activation function at input x, same size as x

backward(do_Hessian: bool = False) → Tuple[torch.Tensor, Optional[torch.Tensor]]

Computes derivatives of activation function evaluated at x in backward mode.

Calls self.compute_derivatives without inputs, stores necessary variables in self.ctx.

Inherited by all subclasses.

compute_derivatives(*args, do_Hessian: bool = False) → Tuple[torch.Tensor, Optional[torch.Tensor]]

Parameters:

• args (torch.Tensor) – variables needed to compute derivatives

• do_Hessian (bool, optional) – If set to True, the Hessian will be computed during the forward call. Default: False

Returns:

• dsigma (torch.Tensor or None) - first derivative of activation function at input x, same size as x

• d2sigma (torch.Tensor or None) - second derivative of activation function at input x, same size as x

AntiTanh

class antiTanhActivation(*args: Any, **kwargs: Any)

Bases: hessQuikActivationFunction

Applies the antiderivative of the hyperbolic tangent activation function to each entry of the incoming data.

Examples:

>>> import hessQuik.activations as act
>>> act_func = act.antiTanhActivation()
>>> x = torch.randn(10, 4)
>>> sigma, dsigma, d2sigma = act_func(x, do_gradient=True, do_Hessian=True)

forward(x, do_gradient=False, do_Hessian=False, forward_mode=True)

Activates each entry of incoming data via

\[\sigma(x) = \ln(\cosh(x))\]

compute_derivatives(*args, do_Hessian=False)

Computes the first and second derivatives of each entry of the incoming data via

\[\begin{split}\begin{align} \sigma'(x) &= \tanh(x)\\ \sigma''(x) &= 1 - \tanh^2(x) \end{align}\end{split}\]

Identity

class identityActivation(*args: Any, **kwargs: Any)

Bases: hessQuikActivationFunction

Applies the identity activation function to each entry of the incoming data.

Examples:

>>> import hessQuik.activations as act
>>> act_func = act.identityActivation()
>>> x = torch.randn(10, 4)
>>> sigma, dsigma, d2sigma = act_func(x, do_gradient=True, do_Hessian=True)

forward(x, do_gradient=False, do_Hessian=False, forward_mode=True)

Activates each entry of incoming data via

\[\sigma(x) = x\]

compute_derivatives(*args, do_Hessian=False)

Computes the first and second derivatives of each entry of the incoming data via

\[\begin{split}\begin{align} \sigma'(x) &= 1\\ \sigma''(x) &= 0 \end{align}\end{split}\]

Quadratic

class quadraticActivation(*args: Any, **kwargs: Any)

Bases: hessQuikActivationFunction

Applies the quadratic activation function to each entry of the incoming data.

Examples:

>>> import hessQuik.activations as act
>>> act_func = act.quadraticActivation()
>>> x = torch.randn(10, 4)
>>> sigma, dsigma, d2sigma = act_func(x, do_gradient=True, do_Hessian=True)

forward(x, do_gradient=False, do_Hessian=False, forward_mode=True)

Activates each entry of incoming data via

\[\sigma(x) = \frac{1}{2}x^2\]

compute_derivatives(*args, do_Hessian=False)

Computes the first and second derivatives of each entry of the incoming data via

\[\begin{split}\begin{align} \sigma'(x) &= x\\ \sigma''(x) &= 1 \end{align}\end{split}\]

Sigmoid

class sigmoidActivation(*args: Any, **kwargs: Any)

Bases: hessQuikActivationFunction

Applies the sigmoid activation function to each entry of the incoming data.

Examples:

>>> import hessQuik.activations as act
>>> act_func = act.sigmoidActivation()
>>> x = torch.randn(10, 4)
>>> sigma, dsigma, d2sigma = act_func(x, do_gradient=True, do_Hessian=True)

forward(x, do_gradient=False, do_Hessian=False, forward_mode=True)

Activates each entry of incoming data via

\[\sigma(x) = \frac{1}{1 + e^{-x}}\]

compute_derivatives(*args, do_Hessian=False)

Computes the first and second derivatives of each entry of the incoming data via

\[\begin{split}\begin{align} \sigma'(x) &= \sigma(x)(1 - \sigma(x))\\ \sigma''(x) &= \sigma'(x)(1 - 2 * \sigma(x)) \end{align}\end{split}\]

Softplus

class softplusActivation(*args: Any, **kwargs: Any)

Bases: hessQuikActivationFunction

Applies the softplus activation function to each entry of the incoming data.

Examples:

>>> import hessQuik.activations as act
>>> act_func = act.softplusActivation()
>>> x = torch.randn(10, 4)
>>> sigma, dsigma, d2sigma = act_func(x, do_gradient=True, do_Hessian=True)

__init__(beta: float = 1.0, threshold: float = 20.0) → None

Parameters:

• beta (float) – parameter affecting steepness of the softplus function. Default: 1.0

• threshold (float) – parameter for numerical stability. Uses identity function when \(\beta x > threshold\)

forward(x, do_gradient=False, do_Hessian=False, forward_mode=True)

Activates each entry of incoming data via

\[\sigma(x) = \frac{1}{\beta}\ln(1 + e^{\beta x})\]

compute_derivatives(*args, do_Hessian=False)

Computes the first and second derivatives of each entry of the incoming data via

\[\begin{split}\begin{align} \sigma'(x) &= \frac{1}{1 + e^{-\beta x}}\\ \sigma''(x) &= \frac{\beta}{2\cosh(\beta x) + 2} \end{align}\end{split}\]

Tanh

class tanhActivation(*args: Any, **kwargs: Any)

Bases: hessQuikActivationFunction

Applies the hyperbolic tangent activation function to each entry of the incoming data.

Examples:

>>> import hessQuik.activations as act
>>> act_func = act.tanhActivation()
>>> x = torch.randn(10, 4)
>>> sigma, dsigma, d2sigma = act_func(x, do_gradient=True, do_Hessian=True)

forward(x, do_gradient=False, do_Hessian=False, forward_mode=True)

Activates each entry of incoming data via

\[\sigma(x) = \tanh(x)\]

compute_derivatives(*args, do_Hessian=False)

Computes the first and second derivatives of each entry of the incoming data via

\[\begin{split}\begin{align} \sigma'(x) &= 1 - \tanh^2(x)\\ \sigma''(x) &= -2\tanh(x) (1 - \tanh^2(x)) \end{align}\end{split}\]