Safety Runtime#
-
class SafetyRuntime : public trt_edgellm::safety::ModelInferenceRunner#
Internal execution owner for one caller-owned current CUDA context and externally serialized requests. Healthy teardown releases every graph borrower before destruction. Terminal failure forbids SDK cleanup, so the application must retain the Runtime and graph handles until containment termination.
Public Functions
-
SafetyRuntime() noexcept#
- explicit SafetyRuntime(
- nvinfer2::safe::ISafeRecorder &recorder
-
~SafetyRuntime() noexcept#
-
SafetyRuntime(SafetyRuntime const&) = delete#
-
SafetyRuntime &operator=(SafetyRuntime const&) = delete#
-
LifecycleState lifecycleState() const noexcept#
-
inline DeclaredResourceManager &declaredResources() noexcept#
Initialization resource preparation shares this runtime’s lifecycle.
-
SafetyStatus initialize(SafetyRuntimeInitSpec const &spec) noexcept#
Initialization only: validate fixed assignments and prepare owned TRT IO and streams.
- SafetyStatus initialize(
- SafetyRuntimeArtifactBundle const &artifacts,
- SafetyRuntimeOptions const &options
Load builder artifacts and prepare the complete fixed model flow, including mandatory warmup.
- bool handleRequest(
- LLMGenerationRequest const &request,
- LLMGenerationResponse &response
One externally serialized request; input and caller-preallocated response storage outlive this call. A fatal failure requires retaining the runtime and its recorders for application containment.
-
SafetyStatus requestStatus() const noexcept#
Last public request result; lifecycleState() separately describes admission/readiness.
-
std::vector<LMStageContext> const &contexts() const noexcept#
Immutable context metadata for caller-owned model-local resource preparation.
- SafetyStatus completeContext(
- char const *stage,
- ResourceSlotId slot,
- std::vector<SafetyModelLocalResourceView> const &views
Initialization only: complete a declared context with stable, non-owning resource views.
- SafetyStatus prepare(
- LMStageWarmupCallback warmup,
- void *userData = nullptr
Initialization only: warm every admitted stage/profile and seal Operational resources.
- SafetyStatus runLMStage(
- LMStageId stage,
- ResourceSlotId slot,
- LMStageOutputs &outputs,
- LMStagePrepareCallback prepare = nullptr,
- void *userData = nullptr
Operational catalog dispatch prepares the selected fixed-profile context before execution.
-
SafetyStatus deinitializeGraphs() noexcept#
Healthy Deinit Preparation: destroy graphs/modules, retaining borrowed IO/streams for caller cleanup.
-
SafetyStatus deinitializeResources() noexcept#
Release IO/streams after graph, model-local object and declared-resource teardown.
-
SafetyStatus deinitialize() noexcept#
Combined teardown when no caller-owned resources require cleanup between the two phases.
-
bool fatal() const noexcept#
Terminal errors prohibit further CUDA/DLI work, including cleanup.
-
nvinfer2::safe::ISafeRecorder &errorRecorder() noexcept#
Runtime recorder shared by every graph and execution component.
-
SafetyRuntime() noexcept#
-
struct SafetyRuntimeArtifactBundle#
Directories produced by the Safety LLM and visual builders.
-
struct SafetyRuntimeOptions#