InstanceNorm

class nvtripy.InstanceNorm(num_channels: int, dtype: dtype = float32, eps: float = 1e-05)[source]

Bases: Module

Applies Instance Normalization over a mini-batch of inputs:

\(\text{InstanceNorm}(x) = \Large \frac{x - \mu}{ \sqrt{\sigma^2 + \epsilon}} \normalsize * \gamma + \beta\)

where \(\mu\) is the mean and \(\sigma^2\) is the variance, computed per channel for each instance in a mini-batch. \(\gamma\) and \(\beta\) are learnable parameters of shape (C).

InstanceNorm is similar to LayerNorm, but statistics are computed per channel across spatial dimensions, whereas LayerNorm is computed across all dimensions of a sample.

Parameters:

  • num_channels (int) – Number of channels/features expected in the input

  • dtype (dtype) – The data type to use for the module parameters

  • eps (float) – The epsilon value added to the denominator for numerical stability

Example
1instance_norm = tp.InstanceNorm(3)
2instance_norm.weight = tp.ones((3,))
3instance_norm.bias = tp.zeros((3,))
4
5input_tensor = tp.ones((2, 3, 4, 4))
6output = instance_norm(input_tensor)
Local Variables
>>> instance_norm
InstanceNorm(
    weight: Parameter = (shape=(3,), dtype=float32),
    bias: Parameter = (shape=(3,), dtype=float32),
)
>>> instance_norm.state_dict()
{
    weight: tensor([1, 1, 1], dtype=float32, loc=gpu:0, shape=(3,)),
    bias: tensor([0, 0, 0], dtype=float32, loc=gpu:0, shape=(3,)),
}

>>> input_tensor
tensor(
    [[[[1, 1, 1, 1],
       [1, 1, 1, 1],
       [1, 1, 1, 1],
       [1, 1, 1, 1]],

      [[1, 1, 1, 1],
       [1, 1, 1, 1],
       [1, 1, 1, 1],
       [1, 1, 1, 1]],

      [[1, 1, 1, 1],
       [1, 1, 1, 1],
       [1, 1, 1, 1],
       [1, 1, 1, 1]]],


     [[[1, 1, 1, 1],
       [1, 1, 1, 1],
       [1, 1, 1, 1],
       [1, 1, 1, 1]],

      [[1, 1, 1, 1],
       [1, 1, 1, 1],
       [1, 1, 1, 1],
       [1, 1, 1, 1]],

      [[1, 1, 1, 1],
       [1, 1, 1, 1],
       [1, 1, 1, 1],
       [1, 1, 1, 1]]]], 
    dtype=float32, loc=gpu:0, shape=(2, 3, 4, 4))

>>> output
tensor(
    [[[[0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0]],

      [[0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0]],

      [[0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0]]],


     [[[0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0]],

      [[0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0]],

      [[0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0]]]], 
    dtype=float32, loc=gpu:0, shape=(2, 3, 4, 4))
__call__(*args: Any, **kwargs: Any) → Any

Calls the module with the specified arguments.

Parameters:

  • *args (Any) – Positional arguments to the module.

  • **kwargs (Any) – Keyword arguments to the module.

Returns:

The outputs computed by the module.

Return type:

Any

Example
1class Module(tp.Module):
2    def forward(self, x):
3        return tp.relu(x)
4
5
6module = Module()
7
8input = tp.arange(-3, 3)
9out = module(input)  # Note that we do not call `forward` directly.
Local Variables
>>> module
Module(
)
>>> module.state_dict()
{}

>>> input
tensor([-3, -2, -1, 0, 1, 2], dtype=float32, loc=gpu:0, shape=(6,))

>>> out
tensor([0, 0, 0, 0, 1, 2], dtype=float32, loc=gpu:0, shape=(6,))
initialize_dummy_parameters() → None

Initializes any uninitialized parameters in the module with dummy values. This is useful for debugging and testing purposes.

Example
1linear = tp.Linear(2, 2)
2print(linear.state_dict())
3
4linear.initialize_dummy_parameters()
5print(linear.state_dict())
Output
{'weight': <nvtripy.frontend.module.parameter.DefaultParameter object at 0x7935992e3dc0>, 'bias': <nvtripy.frontend.module.parameter.DefaultParameter object at 0x7935992e3c70>}
{'weight': tensor(
    [[1, 1],
     [1, 1]], 
    dtype=float32, loc=gpu:0, shape=(2, 2)), 'bias': tensor([1, 1], dtype=float32, loc=gpu:0, shape=(2,))}
load_state_dict(state_dict: Dict[str, Tensor], strict: bool = True) → Tuple[Set[str], Set[str]]

Loads parameters from the provided state_dict into the current module. This will recurse over any nested child modules.

Parameters:

  • state_dict (Dict[str, Tensor]) – A dictionary mapping names to parameters.

  • strict (bool) – If True, keys in state_dict must exactly match those in this module. If not, an error will be raised.

Returns:

  • missing_keys: keys that are expected by this module but not provided in state_dict.

  • unexpected_keys: keys that are not expected by this module but provided in state_dict.

Return type:

A tuple of two sets of strings representing

Example
 1class MyModule(tp.Module):
 2    def __init__(self):
 3        super().__init__()
 4        self.param = tp.ones((2,), dtype=tp.float32)
 5
 6
 7module = MyModule()
 8
 9print(f"Before: {module.param}")
10
11module.load_state_dict({"param": tp.zeros((2,), dtype=tp.float32)})
12
13print(f"After: {module.param}")
Output
Before: tensor([1, 1], dtype=float32, loc=gpu:0, shape=(2,))
After: tensor([0, 0], dtype=float32, loc=gpu:0, shape=(2,))

See also

state_dict()

named_children() → Iterator[Tuple[str, Module]]

Returns an iterator over immediate children of this module, yielding tuples containing the name of the child module and the child module itself.

Returns:

An iterator over tuples containing the name of the child module and the child module itself.

Return type:

Iterator[Tuple[str, Module]]

Example
 1class StackedLinear(tp.Module):
 2    def __init__(self):
 3        super().__init__()
 4        self.linear1 = tp.Linear(2, 2)
 5        self.linear2 = tp.Linear(2, 2)
 6
 7
 8stacked_linear = StackedLinear()
 9
10for name, module in stacked_linear.named_children():
11    print(f"{name}: {type(module).__name__}")
Output
linear1: Linear
linear2: Linear
named_parameters() → Iterator[Tuple[str, Tensor]]
Returns:

An iterator over tuples containing the name of a parameter and the parameter itself.

Return type:

Iterator[Tuple[str, Tensor]]

Example
 1class MyModule(tp.Module):
 2    def __init__(self):
 3        super().__init__()
 4        self.alpha = tp.Tensor(1)
 5        self.beta = tp.Tensor(2)
 6
 7
 8linear = MyModule()
 9
10for name, parameter in linear.named_parameters():
11    print(f"{name}: {parameter}")
Output
alpha: tensor(1, dtype=int32, loc=cpu:0, shape=())
beta: tensor(2, dtype=int32, loc=cpu:0, shape=())
state_dict() → Dict[str, Tensor]

Returns a dictionary mapping names to parameters in the module. This will recurse over any nested child modules.

Returns:

A dictionary mapping names to parameters.

Return type:

Dict[str, Tensor]

Example
 1class MyModule(tp.Module):
 2    def __init__(self):
 3        super().__init__()
 4        self.param = tp.ones((2,), dtype=tp.float32)
 5        self.linear1 = tp.Linear(2, 2)
 6        self.linear2 = tp.Linear(2, 2)
 7
 8
 9module = MyModule()
10
11state_dict = module.state_dict()
Local Variables
>>> state_dict
{
    param: tensor([1, 1], dtype=float32, loc=gpu:0, shape=(2,)),
    linear1.weight: <nvtripy.frontend.module.parameter.DefaultParameter object at 0x79359933fd00>,
    linear1.bias: <nvtripy.frontend.module.parameter.DefaultParameter object at 0x79359930d2b0>,
    linear2.weight: <nvtripy.frontend.module.parameter.DefaultParameter object at 0x793599307ac0>,
    linear2.bias: <nvtripy.frontend.module.parameter.DefaultParameter object at 0x7935992c33a0>,
}
num_channels: int

Number of channels/features expected in the input.

dtype: dtype

The data type used to perform the operation.

eps: float

A value added to the denominator for numerical stability.

weight: Tensor

The \(\gamma\) parameter of shape (num_channels).

bias: Tensor

The \(\beta\) parameter of shape (num_channels).

forward(x: Tensor) → Tensor[source]
Parameters:

x (Tensor) – Input tensor of shape [N, C, …] where C is the number of features

Returns:

Normalized tensor of the same shape as input

Return type:

Tensor