Executable

class nvtripy.Executable[source]

Bases: object

Represents a compiled executable generated by the compiler.

See also

compile()

input_infos: Dict[str, InputInfo]

Stores metadata, like shapes and data types, for each input to the executable.

load() Executable[source]

Loads a executable from the provided path.

Parameters:

path (str) – The path from which to load the exectuable.

Returns:

The executable object loaded from the file.

Return type:

Executable

Example: Save and load executable
 1import os
 2
 3
 4def add(a, b):
 5    return a + b
 6
 7
 8compiled_add = tp.compile(
 9    add,
10    args=[
11        tp.InputInfo(shape=((1, 2, 3),), dtype=tp.float32),
12        tp.InputInfo(shape=((1, 2, 3),), dtype=tp.float32),
13    ],
14)
15
16
17# Assuming `out_dir` is the directory containing the executable:
18executable_file = os.path.join(out_dir, "executable.json")
19loaded_executable = tp.Executable.load(executable_file)
Local Variables
>>> loaded_executable
Executable(a: nvtripy.Tensor, b: nvtripy.Tensor) -> nvtripy.Tensor
Output
[I] /tmp/tmpj1_xhnt4 does not exist, creating now.
__call__(*args: Tensor, **kwargs: Tensor) Tensor | Sequence[Tensor][source]

Invokes the executable with the specified tensor arguments.

Note

Inputs must be evaluated tensors in GPU memory.

You can use nvtripy.copy() or nvtripy.Tensor.eval() to ensure this.

Parameters:
  • *args (Tensor) – Positional arguments. Must be of type Tensor .

  • **kwargs (Tensor) – Keyword arguments. Must be of type Tensor .

Returns:

The output Tensor s of the compiled function.

Return type:

Tensor | Sequence[Tensor]

Example
 1def add(a, b):
 2    return a + b
 3
 4
 5compiled_add = tp.compile(
 6    add,
 7    args=[
 8        tp.InputInfo((1,), dtype=tp.float32),
 9        tp.InputInfo((1,), dtype=tp.float32),
10    ],
11)
12
13a = tp.ones((1,), dtype=tp.float32).eval()
14b = tp.ones((1,), dtype=tp.float32).eval()
15
16out = compiled_add(a, b)
Local Variables
>>> a
tensor([1], dtype=float32, loc=gpu:0, shape=(1,))

>>> b
tensor([1], dtype=float32, loc=gpu:0, shape=(1,))

>>> out
tensor([2], dtype=float32, loc=gpu:0, shape=(1,))
save(path: str) None[source]

Saves this executable to the provided path.

Parameters:

path (str) – The path at which to save the executable.

Example: Save executable
 1import os
 2
 3
 4def add(a, b):
 5    return a + b
 6
 7
 8compiled_add = tp.compile(
 9    add,
10    args=[
11        tp.InputInfo(shape=((1, 2, 3),), dtype=tp.float32),
12        tp.InputInfo(shape=((1, 2, 3),), dtype=tp.float32),
13    ],
14)
15
16# Assuming `out_dir` is the desired output directory:
17executable_file = os.path.join(out_dir, "executable.json")
18compiled_add.save(executable_file)
Output
[I] /tmp/tmps2n56d3p does not exist, creating now.
property serialized_tensorrt_engine: bytes

The serialized TensorRT engine, as bytes, from the executable.

See also

Refer to the TensorRT developer guide for details on how to work with serialized TensorRT engines.

Example: TensorRT engine
 1def add(a, b):
 2    return a + b
 3
 4
 5compiled_add = tp.compile(
 6    add,
 7    args=[
 8        tp.InputInfo(shape=((1, 2, 3),), dtype=tp.float32),
 9        tp.InputInfo(shape=((1, 2, 3),), dtype=tp.float32),
10    ],
11)
12
13trt_engine = compiled_add.serialized_tensorrt_engine