Gaussian Error Linear Unit (GELU)#

API#

class warp_nn.modules.activations.GELU(
*,
approximate: Literal['none', 'tanh'] = 'none',
requires_grad: bool = True,
)[source]#

Bases: Module

Gaussian Error Linear Unit (GELU) activation function.

This class computes the element-wise GELU activation function:

\[\text{GELU}(x) = x \, \Phi(x) = \frac{x}{2} \left(1 + \text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)\]

where \(\Phi(x)\) is the cumulative distribution function of the standard normal distribution. If approximate is "tanh", the function is estimated with:

\[\text{GELU}(x) \approx \frac{x}{2} \left(1 + \tanh\left(\sqrt{2 / \pi} \, (x + 0.044715 \, x^3)\right)\right)\]
Parameters:

  • approximate – The approximation algorithm to use: "none" (exact computation) or "tanh".

  • requires_grad – Whether the cached output arrays of the module require gradients.

Raises:

ValueError – If the approximation algorithm is not supported.

__call__(input: array) → array[source]#

Forward pass of the activation function.

Parameters:

input – The input array, with up to 3 dimensions.

Returns:

The output array, with same shape as the input array.

Raises:

TypeError – If the input array’s data type or number of dimensions is not supported.

property approximate: str[source]#

The approximation algorithm used by the GELU function.