losses¶

Losses to be used in seq2seq models

loss¶

class losses.loss.Loss(params, model, name='loss')[source]¶

Bases: object

Abstract class from which all losses must inherit.

__init__(params, model, name='loss')[source]¶

Loss constructor. Note that loss constructors should not modify TensorFlow graph, all graph construction should happen in the self._compute_loss() method.

Parameters:	params (dict) – parameters describing the loss. All supported parameters are listed in `get_required_params()`, `get_optional_params()` functions. model (instance of a class derived from `Model`) – parent model that created this loss. Could be None if no model access is required for the use case. name (str) – name for loss variable scope.

Config parameters:

dtype — data dtype. Could be either tf.float16 or tf.float32.

_cast_types(input_dict)[source]¶

This function performs automatic cast of all inputs to the loss dtype.

Parameters:	input_dict (dict) – dictionary passed to `self._compute_loss()` method.
Returns:	same as input_dict, but with all Tensors cast to the loss dtype.
Return type:	dict

_compute_loss(input_dict)[source]¶

This is the main function which should construct loss graph. Typically, loss will take decoder-produced logits as an input and return a singleton loss tensor.

Parameters:	input_dict (dict) – dictionary containing loss inputs. If the loss is used with `models.encoder_decoder` class, `input_dict` will have the following content: { "decoder_output": dictionary returned from decoder.decode() method "target_tensors": data_layer.input_tensors['target_tensors'] }
Returns:	singleton loss tensor. This tensor will be computed independently for each GPU batch and then averaged (`reduce_mean`) over the number of GPUs (or Horovod workers).

compute_loss(input_dict)[source]¶

Wrapper around self._compute_loss() method. Here name and dtype are set in the variable scope and then self._compute_loss() method is called.

Parameters:	input_dict (dict) – see `self._compute_loss()` docs.
Returns:	see `self._compute_loss()` docs.

static get_optional_params()[source]¶

Static method with description of optional parameters.

Returns:	Dictionary containing all the parameters that can be included into the `params` parameter of the class `__init__()` method.
Return type:	dict

static get_required_params()[source]¶

Static method with description of required parameters.

Returns:	Dictionary containing all the parameters that have to be included into the `params` parameter of the class `__init__()` method.
Return type:	dict

name¶: Loss name.

params¶: Parameters used to construct the loss (dictionary).

ctc_loss¶

class losses.ctc_loss.CTCLoss(params, model, name='ctc_loss')[source]¶

Bases: losses.loss.Loss

Implementation of the CTC loss.

__init__(params, model, name='ctc_loss')[source]¶

CTC loss constructor.

See parent class for arguments description.

Config parameters:

mask_nan (bool) — whether to mask nans in the loss output. Defaults to True.

_compute_loss(input_dict)[source]¶

CTC loss graph construction.

Expects the following inputs:

input_dict = {

}

Parameters:

input_dict (dict) –

input dictionary that has to contain the following fields:

input_dict = {
  "decoder_output": {
    "logits": tensor, shape [batch_size, time length, tgt_vocab_size]
    "src_length": tensor, shape [batch_size]
  },
  "target_tensors": [
    tgt_sequence (shape=[batch_size, time length, num features]),
    tgt_length (shape=[batch_size])
  ]
}

Returns: averaged CTC loss.

static get_optional_params()[source]¶

Static method with description of optional parameters.

Returns:	Dictionary containing all the parameters that can be included into the `params` parameter of the class `__init__()` method.
Return type:	dict

losses.ctc_loss.dense_to_sparse(dense_tensor, sequence_length)[source]¶

sequence_loss¶

class losses.sequence_loss.BasicSampledSequenceLoss(params, model, name='basic_sampled_sequence_loss')[source]¶

Bases: losses.loss.Loss

Basic sampled sequence-to-sequence loss. This is used when the full softmax is computational prohibitive. This one does not use one-hot encodings.

__init__(params, model, name='basic_sampled_sequence_loss')[source]¶

Constructor.

Parameters:

params (dict) –

dictionary with loss parameters. Should contain the following: * tgt_vocab_size: Target vocabulary size * batch_size_per_gpu: Size of the per-worker batch * offset_target_by_one: (default: True). Keep it true for auto-regressive models * average_across_timestep: (default: False). If True, will average

loss across timesteps, else it will sum across timesteps

do_mask: (default: True) whether to mask based on tgt_lengths (which is passed as part of loss_input_dict to compute_loss and has to be not None then)

_compute_loss(input_dict)[source]¶

Computes cross entropy based sequence-to-sequence loss.

Parameters:	input_dict (dict) – inputs to compute loss:: { “logits”: logits tensor of shape [batch_size, T, dim] “target_sequence”: tensor of shape [batch_size, T] “tgt_lengths”: tensor of shape [batch_size] or None }
Returns:	Singleton loss tensor

static get_optional_params()[source]¶

Static method with description of optional parameters.

Returns:	Dictionary containing all the parameters that can be included into the `params` parameter of the class `__init__()` method.
Return type:	dict

static get_required_params()[source]¶

Static method with description of required parameters.

Returns:	Dictionary containing all the parameters that have to be included into the `params` parameter of the class `__init__()` method.
Return type:	dict

class losses.sequence_loss.BasicSequenceLoss(params, model, name='basic_sequence_loss')[source]¶

Bases: losses.loss.Loss

Basic sequence-to-sequence loss. This one does not use one-hot encodings

__init__(params, model, name='basic_sequence_loss')[source]¶

Constructor.

Parameters:

params (dict) –

dictionary with loss parameters. Should contain the following: * tgt_vocab_size: Target vocabulary size * batch_size_per_gpu: Size of the per-worker batch * offset_target_by_one: (default: True). Keep it true for auto-regressive models * average_across_timestep: (default: False). If True, will average

loss across timesteps, else it will sum across timesteps

do_mask: (default: True) whether to mask based on tgt_lengths (which is passed as part of loss_input_dict to compute_loss and has to be not None then)

_compute_loss(input_dict)[source]¶

Computes cross entropy based sequence-to-sequence loss.

Parameters:	input_dict (dict) – inputs to compute loss:: { “logits”: logits tensor of shape [batch_size, T, dim] “target_sequence”: tensor of shape [batch_size, T] “tgt_lengths”: tensor of shape [batch_size] or None }
Returns:	Singleton loss tensor

static get_optional_params()[source]¶

Static method with description of optional parameters.

Returns:	Dictionary containing all the parameters that can be included into the `params` parameter of the class `__init__()` method.
Return type:	dict

static get_required_params()[source]¶

Static method with description of required parameters.

Returns:	Dictionary containing all the parameters that have to be included into the `params` parameter of the class `__init__()` method.
Return type:	dict

class losses.sequence_loss.CrossEntropyWithSmoothing(params, model, name='cross_entropy_with_smoothing')[source]¶

Bases: losses.loss.Loss

Softmax cross entropy loss with label smoothing. This one uses one-hot encodings for labels.

__init__(params, model, name='cross_entropy_with_smoothing')[source]¶

Constructor.

Parameters:

params (dict) –

dictionary with loss parameters. Should contain the following:

tgt_vocab_size: Target vocabulary size

batch_size_per_gpu: Size of the per-worker batch

offset_target_by_one: (default: True). Keep it true for auto-regressive models

do_mask: (default: True) whether to mask based on tgt_lengths (which is passed as part of loss_input_dict to compute_loss and has to be not None then)

_compute_loss(input_dict)[source]¶

Computes cross entropy based sequence-to-sequence loss with label smoothing.

Parameters:	input_dict (dict) – inputs to compute loss:: { “logits”: logits tensor of shape [batch_size, T, dim] “target_sequence”: tensor of shape [batch_size, T] “tgt_lengths”: tensor of shape [batch_size] or None }
Returns:	Singleton loss tensor

static get_optional_params()[source]¶

Static method with description of optional parameters.

Returns:	Dictionary containing all the parameters that can be included into the `params` parameter of the class `__init__()` method.
Return type:	dict

static get_required_params()[source]¶

Static method with description of required parameters.

Returns:	Dictionary containing all the parameters that have to be included into the `params` parameter of the class `__init__()` method.
Return type:	dict

class losses.sequence_loss.PaddedCrossEntropyLossWithSmoothing(params, model, name='padded_cross_entropy_with_smoothing')[source]¶

Bases: losses.loss.Loss

static get_optional_params()[source]¶

Static method with description of optional parameters.

Returns:	Dictionary containing all the parameters that can be included into the `params` parameter of the class `__init__()` method.
Return type:	dict

cross_entropy_loss¶

class losses.cross_entropy_loss.CrossEntropyLoss(params, model, name='cross_entropy_loss')[source]¶

Bases: losses.loss.Loss

Implementation of the usual cross_entropy loss with softmax.

text2speech_loss¶

class losses.text2speech_loss.Text2SpeechLoss(params, model, name='text2speech_loss')[source]¶

Bases: losses.loss.Loss

Default text-to-speech loss.

_compute_loss(input_dict)[source]¶

Computes loss for text-to-speech model.

Parameters:

input_dict (dict) –

“decoder_output”: dictionary containing:
”outputs”: array containing:
mel: mel-spectrogram predicted by the decoder [batch, time, n_mel]

post_net_mel: spectrogram after adding the residual corrections from the post net of shape [batch, time, feats]

mag: mag-spectrogram predicted by the decoder [batch, time, n_mag]
”stop_token_predictions”: stop_token predictions of shape [batch, time, 1]
”target_tensors”: array containing:
- spec: the true spectrogram of shape [batch, time, feats]
- stop_token: the stop_token of shape [batch, time]
- spec_length: the length of specs [batch]

Returns: Singleton loss tensor

static get_optional_params()[source]¶

Static method with description of optional parameters.

Returns:	Dictionary containing all the parameters that can be included into the `params` parameter of the class `__init__()` method.
Return type:	dict

wavenet_loss¶

class losses.wavenet_loss.WavenetLoss(params, model, name='wavenet_loss')[source]¶

Bases: losses.loss.Loss

_compute_loss(input_dict)[source]¶

Computes the cross-entropy loss for WaveNet.

Parameters:

input_dict (dict) –

“decoder_output”: array containing: [ * logits: predicted output signal as logits * outputs: array containing: [
- ground truth signal as encoded labels
- mu-law decoded audio
]

]

get_optional_params()[source]¶

Static method with description of optional parameters.

Returns:	Dictionary containing all the parameters that can be included into the `params` parameter of the class `__init__()` method.
Return type:	dict

get_required_params()[source]¶

Static method with description of required parameters.

Returns:	Dictionary containing all the parameters that have to be included into the `params` parameter of the class `__init__()` method.
Return type:	dict