Internal Helpers Submodule

This module contains helper functions and classes which are used internally in the package or can be useful when debugging or implementing custom functionality.

For debugging, print_tensor_op() and print_tensor_size_op() can be used to print tensors from within the DALI pipeline.

The other functionality is used internally, and may be useful when implementing custom processing steps.

accvlab.dali_pipeline_framework.internal_helpers.check_type(input, expected_type_np, identifier)

Python operator for checking the type of the input.

If the type of the input does not match the expected type, a ValueError is raised.

Note

The check is performed at runtime, i.e. not the graph construction time.

Important

Use the output of this function to tie the dependency on the check op, preventing it from being pruned or remain unused. Note that even though preserve=True is set in the operator performing the check, the operator may be not executed if the output is not used, meaning that no check will be performed.

Parameters:

• input (DataNode) – The input to check the type of.

• expected_type_np (dtype) – The expected type of the input.

• identifier (str) – The identifier of the input.

Returns:

The same input. Use this output to tie the dependency on the check op, preventing it from being not called due to the unused output.

Raises:

ValueError – If the type of the input does not match the expected type.

accvlab.dali_pipeline_framework.internal_helpers.print_tensor_op(tensor_list, name)

Python operator for printing a tensor list.

This operator is used to print a tensor list while running the pipeline. Note that simply adding print statements to the pipeline will not work as those will be executed during the DALI graph construction phase, not the actual execution of the pipeline, and therefore, the printed objects will be DataNode (i.e. “placeholder” objects) instead of the actual data.

Note

The input is of type DataNode, not a tensor list (TensorListCPU or TensorListGPU). This is because this function is called in the DALI pipeline definition, i.e. during the graph construction phase.

Parameters:

• tensor_list (DataNode) – The tensor list to print.

• name (str) – The name of the tensor list (will be shown in the printed output).

accvlab.dali_pipeline_framework.internal_helpers.print_tensor_size_op(tensor_list, name)

Python operator for printing the size of a tensor list.

This operator is used to print the size of a tensor list while running the pipeline. See print_tensor_op() for more details on why this operator is needed.

Note

The input is of type DataNode, not a tensor list (TensorListCPU or TensorListGPU). This is because this function is called in the DALI pipeline definition, i.e. during the graph construction phase.

Parameters:

• tensor_list (DataNode) – The tensor list to print the size of.

• name (str) – The name of the tensor list (will be shown in the printed output).

accvlab.dali_pipeline_framework.internal_helpers.get_mapped(val, mapping, encapsulate=False)

Mapping of original values to numerical values.

Handles nested lists/tuples recursively.

The mapping is a dictionary with the original values (i.e. inputs to be mapped) as keys and the mapped values (i.e. the corresponding output values) as values. The dictionary may have a key None, which is used as a default value if the input value is not in the mapping. If no None key is present, all the values must have a corresponding key in the mapping.

Parameters:

• val (Union[Sequence, Any]) – The value(s) to map.

• mapping (dict) – The mapping to use.

• encapsulate (bool, default: False) – Whether to encapsulate the result in a list.

Returns:

list – The mapped value(s).

accvlab.dali_pipeline_framework.internal_helpers.get_as_data_node(value)

Convert a value to a DALI DataNode constant.

Important

The result is always either an 32-bit integer or a 32-bit float (depending on the type of the input value).

Parameters:

value – The value to convert to a DALI DataNode.

Returns:

DataNode – The value as a DALI DataNode.

Mini-Parser

This module contains the mini-parser for the DALI pipeline. See Parser for more details on the parser and AST and its subclasses for more details on the abstract syntax tree which is used as the output of the parser.

The parser is used internally in the AnnotationConditionEval processing step.

class accvlab.dali_pipeline_framework.internal_helpers.mini_parser.Parser(input_str)

Bases: object

The actual parser.

The input string must start with a variable name, followed by an assignment operator, followed by an expression.

The expression is parsed recursively. The expression can contain variables, literals, and operators.

The operators are:

• Logical operators: or, and, not

• Comparison operators: ==, !=, >, >=, <, <=

• Parentheses: ( and )

• Unary minus: -

• Assignment operator: =

The syntax is similar to Python. However, note that

• Only the operators defined above are supported.

• Comparisons of more than two values are not supported (e.g. a < b < c is not supported).

• Only numeric literals are supported. True and False are not supported (not needed in the current use case).

Example

Some valid statements:

• res_var = -_b1 < 10.5

• res_1_var = -_b1 < 10.5 and -c > -20

• res_3_var = not -_b1 < 10.5

• res_4_var = (-_b1 < 10.5 or a_bool_var) and another_nool_var

• res_5_var = (-_b1 < 10.5 or (-c > -20 and d == 10)) and another_var > 30

• res_7_var = (-_b1 < 10.5 or (-c > -20 and d == 10)) and (another_var > 30 and _e < 40) and f > 50

Parameters:

input_str (str) – Input string to parse.

parse()

Parses the input stream and returns an AST.

See the class docstring for the syntax.

Returns:

AST – The AST of the input stream.

class accvlab.dali_pipeline_framework.internal_helpers.mini_parser.AST

Bases: object

Base class for the AST nodes.

class accvlab.dali_pipeline_framework.internal_helpers.mini_parser.Assignment(variable, expression)

Bases: AST

Represents an assignment of a value to a variable.

Note

The constructor parameters are also the attributes of the class.

Parameters:

• variable (Variable) – Variable to assign the value to.

• expression (AST) – Expression to evaluate and assign the result to the variable.

class accvlab.dali_pipeline_framework.internal_helpers.mini_parser.Literal(value)

Bases: AST

Represents a literal value.

Note

The constructor parameters are also the attributes of the class.

Parameters:

value (str) – Value to represent.

class accvlab.dali_pipeline_framework.internal_helpers.mini_parser.Variable(name)

Bases: AST

Represents a variable (by name).

Note

The constructor parameters are also the attributes of the class.

Parameters:

name (str) – Name of the variable.

class accvlab.dali_pipeline_framework.internal_helpers.mini_parser.Comparison(val1, comparison_type, val2)

Bases: AST

Represents a comparison.

Note

The constructor parameters are also the attributes of the class.

Parameters:

• val1 (Variable) – First value to compare.

• comparison_type (str) – Type of comparison to perform. Supported comparisons are ==, !=, <, >, >=, <=.

• val2 (Literal) – Second value to compare.

class accvlab.dali_pipeline_framework.internal_helpers.mini_parser.Or(*conditions)

Bases: AST

Represents a logical or on a set of operands.

Note

The constructor parameters are also the attributes of the class. The individual conditions are stored as a tuple.

Parameters:

conditions (AST) – Conditions to be combined with or. Note that the number of conditions is variable, i.e. more than two conditions can be combined.

class accvlab.dali_pipeline_framework.internal_helpers.mini_parser.And(*conditions)

Bases: AST

Represents a logical and on a set of operands.

Note

The constructor parameters are also the attributes of the class. The individual conditions are stored as a tuple.

Parameters:

conditions (AST) – Conditions to be combined with and. Note that the number of conditions is variable, i.e. more than two conditions can be combined.

class accvlab.dali_pipeline_framework.internal_helpers.mini_parser.Not(condition)

Bases: AST

Represents a logical not (i.e. negation) for an operand.

Note

The constructor parameters are also the attributes of the class.

Parameters:

condition (AST) – Condition to be negated.

class accvlab.dali_pipeline_framework.internal_helpers.mini_parser.UnaryMinus(value)

Bases: AST

Represents a unary minus.

Note

The constructor parameters are also the attributes of the class.

class accvlab.dali_pipeline_framework.internal_helpers.mini_parser.Lexer(input)

Bases: object

Lexer for the simple parser.

Parameters:

input (str) – Input string to generate tokens from.

next_token()

Returns the next token in the input stream

Return type:

Token

class accvlab.dali_pipeline_framework.internal_helpers.mini_parser.TokenType

Bases: object

Represents the type of a token

LITERAL = 'literal'

VARIABLE = 'variable'

ASSIGNMENT = 'assignment'

COMPARISON = 'comparison'

LOGICAL_OR = 'logical_or'

LOGICAL_AND = 'logical_and'

LOGICAL_NOT = 'logical_not'

MINUS = 'minus'

PARENTHESIS_OPEN = 'parenthesis_open'

PARENTHESIS_CLOSE = 'parenthesis_close'

EOL = 'end_of_line'

class accvlab.dali_pipeline_framework.internal_helpers.mini_parser.Token(type, value)

Bases: object

Represents a token in the input stream.

Note

The constructor parameters become attributes of the class.

Parameters:

• type (TokenType) – Type of the token.

• value (str) – Value of the token.