Skip to main content

Runtime Lifecycle

The native runtime performs one exact load and returns an abstract Task:

auto task = trtmc::load_task("model.bundle", "/opt/trtmc/lib");

Load sequence​

  1. BundleReader validates the format-1 header and all section bounds.
  2. The loader validates the family and backend names as safe DSO tokens.
  3. It loads libtrtmc_backend_<backend>.so from the explicit runtime root.
  4. It loads libtrtmc_model_<family>.so from that same root.
  5. It resolves the single trtmc_create_family factory and passes a FamilyContext containing the read-only bundle reader and abstract backend.
  6. It verifies that the returned ITask::task() matches the bundle header.

There is no current-directory search, environment fallback, registry lookup, strategy switch, sibling-family probe, or load retry.

Ownership after transfer​

The selected family implements one or more interfaces in core/runtime/include/trtmc/task.h. It owns preprocessing, postprocessing, request orchestration, tokenizer/sampler state, family sections, and engine binding. It creates engines only through the abstract IBackend/IEngine contract. The backend owns TensorRT runtime objects, not model policy.

FamilyContext.reader is read-only. A factory may consume its sections before returning or copy the lightweight BundleReader into the pipeline for deferred reads; it must not retain a reference to the temporary factory context.

Legacy audio output contract​

The legacy AudioResult carries interleaved float PCM. num_channels defaults to one; for stereo the buffer order is L0, R0, L1, R1, .... num_samples counts total scalar samples, not frames per channel; zero leaves the count unspecified and consumers use samples.size(). A nonzero count must match the buffer size. The sample rate is in frames per second, so duration is samples.size() / num_channels / sample_rate. Buffers must contain whole frames.

The CLI WAV writer preserves the channel count and writes float32 PCM. Its WAV reader intentionally still averages input channels to mono for existing speech tasks. This contract covers whole-result audio; incremental delivery is described in Multichannel streaming audio below.

The appended field preserves existing three-field aggregate initialization and mono behavior at source level. It changes the C++ struct layout: rebuild clients and family DSOs together rather than mixing binaries compiled against different versions of task.h.

Multichannel streaming audio​

Families can opt into IMultichannelStreamingAudioGeneration without changing the existing mono IStreamingAudioGeneration interface. A family implementing both is dispatched through the multichannel capability by the CLI.

Each AudioChunkView borrows interleaved float PCM (L0, R0, L1, R1, ... for stereo), with an explicit channel count and sample rate. num_samples counts scalar samples, not frames per channel. Chunks must be nonempty whole frames; the sample rate and channel count stay constant within a call. Callbacks are synchronous, ordered, and non-concurrent. Their pointers are valid only during the callback. Normal return ends the stream and reports the sum of delivered scalar samples; callback exceptions must stop generation and propagate.

trtmc generate-audio ... --stream true --output audio.raw writes interleaved float32 samples and reports format, sample_rate, num_channels, and num_samples in its success JSON. Playback duration is num_samples / num_channels / sample_rate. This is raw PCM, not a WAV file. Invalid chunks, format changes, inconsistent totals, and file-write errors fail the command without success JSON. A failed stream can leave a partial output file; callers must not treat file existence alone as success.

This capability does not add HTTP transport, encoded formats, or streaming support to models that do not already produce incremental audio. Existing mono families remain unchanged and report num_channels: 1.

Optional load settings​

Runtime-sized KV capacity is passed directly to compatible families. TensorRT-RTX runtime cache and CUDA graph settings are accepted only when the bundle selects trt_rtx; the standard backend rejects them. TVM-FFI BYOK is an explicit extension DSO and three-part binding, not a general plugin registry.

Teardown​

Applications destroy Task objects before the loaded family and backend libraries leave scope. Families release their streams, buffers, communicators, engines, and family-local state; the loader owns the dynamic-library handles.

FP8 tensor storage​

The native runtime represents linear TensorRT kFP8 I/O as DType::kFloat8E4M3. Each element occupies one byte, so tensor sizes and engine buffer allocations use the encoded E4M3 storage size rather than the size of float. This dtype is distinct from signed INT8 and does not convert or dequantize the stored values. The runtime requires kLINEAR FP8 I/O in the selected optimization profile. Vectorized or otherwise non-linear FP8 formats are rejected before buffer allocation or external binding, because tensor views do not describe their padding or layout. Families continue to own quantization scales, graph construction, and tensor semantics. This storage contract alone does not imply that a GPU supports FP8 arithmetic or that a model configuration has been qualified.