`mod llm`#

module llm#

LLM lifecycle helpers and managed execution entry points.

Functions

fn llm_call(params: LlmCallParams<'_>) -> Result<LlmHandle>#

Start a manual LLM lifecycle span.

This emits an LLM-start event after applying sanitize-request guardrails to the payload recorded for observability.

Parameters

name: Logical provider or model family name recorded on the span.
request: Raw LlmRequest associated with the span.
parent: Optional explicit parent scope.
attributes: LLM attribute bitflags applied to the span.
data: Optional application payload stored on the returned handle. The emitted start event data is the sanitized request payload.
metadata: Optional JSON metadata recorded on the start event.
model_name: Optional normalized model name recorded separately from the request payload.
annotated_request: Optional normalized request annotation produced by a codec.
timestamp: Optional timestamp recorded as the handle start time and on the emitted start event. When None, the current UTC time is used.

Returns

A Result containing the created LlmHandle.

Errors

Returns an error when the runtime owner check fails or when internal state cannot be read safely.

Notes

Sanitize-request guardrails affect only the emitted start-event payload, not the caller-owned LlmRequest.

fn llm_call_end(params: LlmCallEndParams<'_>) -> Result<()>#

Finish a manual LLM lifecycle span.

This emits an LLM-end event for a handle previously returned by llm_call.

Parameters

handle: LLM handle to close.
response: Raw provider response associated with the end event.
data: Optional application payload retained for compatibility. The emitted end event data is the sanitized response unless it sanitizes to JSON null, in which case this payload is used.
metadata: Optional JSON metadata recorded on the end event.
annotated_response: Optional normalized response annotation produced by a response codec. When omitted and response_codec is supplied, the annotation is decoded from the sanitized end-event payload.
response_codec: Optional response codec used to produce a normalized response annotation from the sanitized end-event payload.
timestamp: Optional timestamp recorded on the emitted end event. When None, the runtime uses the current UTC time, or one microsecond after the handle start time if the current time is not later.

Returns

A Result that is Ok(()) when the end event has been emitted.

Errors

Returns an error when the runtime owner check fails, internal state cannot be read safely, or response codec decoding fails.

Notes

Sanitize-response guardrails affect only the emitted end-event payload, not the caller-owned response value.

async fn llm_call_execute(params: LlmCallExecuteParams) -> Result<Json>#

Execute an LLM call through the managed middleware pipeline.

This runs conditional-execution guardrails, request intercepts, and sanitize-request guardrails, emits the LLM-start event, then runs execution intercepts, the provider callback when it is not replaced, and sanitize-response guardrails in the runtime-defined order.

Parameters

name: Logical provider or model family name recorded on emitted events.
request: Raw LlmRequest passed into the managed pipeline.
func: Provider callback or execution continuation.
parent: Optional explicit parent scope for the emitted LLM span.
attributes: LLM attribute bitflags applied to the managed span.
data: Optional application payload stored on the managed LLM handle. It may be used on failure end events that have no output payload.
metadata: Optional JSON metadata recorded on emitted events.
model_name: Optional normalized model name for observability output.
codec: Optional request codec used to produce annotated request data for intercepts and events.
response_codec: Optional response codec used to attach annotated response data to the end event.

Returns

A Result containing the raw JSON response returned by the callback or an execution intercept.

Errors

Returns FlowError::GuardrailRejected when conditional-execution guardrails block the call, or any error raised by request intercepts, execution intercepts, codecs, or the callback itself.

Notes

The LLM-start event is emitted before execution intercepts run. When execution fails after that point, the runtime still emits an LLM-end event without an output payload.

Response codecs enrich observability output only and do not change the value returned to the caller.

fn llm_conditional_execution(request: &LlmRequest) -> Result<()>#

Run only the LLM conditional-execution guardrail chain.

This evaluates whether an LLM call should be allowed to proceed without invoking request intercepts or execution. Each evaluated guardrail emits an automatic guardrail scope start/end pair for observability.

Parameters

request: Raw LlmRequest to validate.

Returns

A Result that is Ok(()) when all guardrails allow execution.

Errors

Returns FlowError::GuardrailRejected when a guardrail blocks execution, or any error raised by the guardrail chain itself.

Notes

This helper is useful for preflight checks when the caller needs the rejection result without starting an LLM span. Guardrail scopes are still emitted for the conditional checks themselves.

fn llm_request_intercepts(name: &str, request: LlmRequest) -> Result<LlmRequest>#

Run only the LLM request-intercept chain.

This applies the currently active global and scope-local request intercepts without emitting lifecycle events or invoking provider execution.

Parameters

name: Logical provider or model family name used when resolving the intercept chain.
request: Raw LlmRequest to transform.

Returns

A Result containing the transformed LlmRequest.

Errors

Returns any error raised by the request-intercept chain.

Notes

Conditional guardrails, codecs, and execution intercepts are not run by this helper.

async fn llm_stream_call_execute(params: LlmStreamCallExecuteParams) -> Result<LlmJsonStream>#

Execute a streaming LLM call through the managed middleware pipeline.

This runs the same pre-execution middleware as llm_call_execute, emits the LLM-start event, and then wraps the provider stream so chunk callbacks and finalization can emit a single LLM-end event when streaming completes.

Parameters

name: Logical provider or model family name recorded on emitted events.
request: Raw LlmRequest passed into the managed pipeline.
func: Streaming provider callback or execution continuation.
collector: Per-chunk collector callback used to accumulate stream state.
finalizer: Finalizer callback used to construct the completed response.
parent: Optional explicit parent scope for the emitted LLM span.
attributes: LLM attribute bitflags applied to the managed span.
data: Optional application payload stored on the managed LLM handle. It may be used on failure end events that have no output payload.
metadata: Optional JSON metadata recorded on emitted events.
model_name: Optional normalized model name for observability output.
codec: Optional request codec used to produce annotated request data for intercepts and events.
response_codec: Optional response codec used to attach annotated response data to the end event.

Returns

A Result containing a boxed stream of JSON chunks.

Errors

Returns FlowError::GuardrailRejected when conditional-execution guardrails block the call, or any error raised by request intercepts, execution intercepts, stream callbacks, codecs, or the provider callback.

Notes

The LLM-start event is emitted before stream execution intercepts run.

The returned stream emits chunk-level results while the runtime defers the LLM-end event until the collector and finalizer complete.

Structs and Unions

struct CreateLlmHandleParams<'a>#

Builder parameters for NemoRelayContextState::create_llm_handle.

name: &'a str#: Logical provider or model family name.

parent_uuid: Option<uuid::Uuid>#: Optional parent scope UUID.

attributes: LlmAttributes#: LLM attribute bitflags.

data: Option<Json>#: Optional application payload stored on the handle.

metadata: Option<Json>#: Optional metadata stored on the handle.

model_name: Option<String>#: Optional normalized model name stored on the handle.

timestamp: Option<DateTime<Utc>>#: Optional timestamp captured as the handle start time and reused by the emitted start event. When omitted, the current UTC time is used.

struct EndLlmHandleParams<'a>#

Builder parameters for NemoRelayContextState::build_llm_end_event.

handle: &'a LlmHandle#: LLM handle to serialize into the emitted end event.

data: Option<Json>#: Optional data payload merged over the handle data.

metadata: Option<Json>#: Optional metadata payload merged over the handle metadata.

annotated_response: Option<Arc<AnnotatedLlmResponse>>#: Optional normalized response annotation produced by a response codec.

timestamp: Option<DateTime<Utc>>#: Optional timestamp recorded on the emitted end event. When omitted, the runtime records the current UTC time, or one microsecond after the handle start time if the current time is not later.

struct LlmCallEndParams<'a>#

Builder parameters for llm_call_end.

handle: &'a LlmHandle#: LLM handle to close.

response: Json#: Raw provider response associated with the end event.

data: Option<Json>#: Optional application payload retained for compatibility; Agent Trajectory Observability Format (ATOF) data is the response.

metadata: Option<Json>#: Optional JSON metadata recorded on the end event.

annotated_response: Option<Arc<AnnotatedLlmResponse>>#: Optional normalized response annotation produced by a response codec.

response_codec: Option<Arc<dyn LlmResponseCodec>>#: Optional response codec used to produce an annotation from sanitized event data.

timestamp: Option<DateTime<Utc>>#: Optional timestamp recorded on the emitted end event. When omitted, the runtime records the current UTC time, or one microsecond after the handle start time if the current time is not later.

struct LlmCallExecuteParams#

Builder parameters for llm_call_execute.

name: String#: Logical provider or model family name recorded on emitted events.

request: LlmRequest#: Raw request passed into the managed pipeline.

func: LlmExecutionNextFn#: Provider callback or execution continuation.

parent: Option<ScopeHandle>#: Optional explicit parent scope for the emitted LLM span.

attributes: LlmAttributes#: LLM attribute bitflags applied to the managed span.

data: Option<Json>#: Optional application payload stored on the handle but not emitted as Agent Trajectory Observability Format (ATOF) data.

metadata: Option<Json>#: Optional JSON metadata recorded on emitted events.

model_name: Option<String>#: Optional normalized model name for observability output.

codec: Option<Arc<dyn LlmCodec>>#: Optional request codec used to produce annotated request data.

response_codec: Option<Arc<dyn LlmResponseCodec>>#: Optional response codec used to attach annotated response data.

struct LlmCallParams<'a>#

Builder parameters for llm_call.

name: &'a str#: Logical provider or model family name recorded on the span.

request: &'a LlmRequest#: Raw request associated with the span.

parent: Option<&'a ScopeHandle>#: Optional explicit parent scope.

attributes: LlmAttributes#: LLM attribute bitflags applied to the span.

data: Option<Json>#: Optional application payload stored on the handle but not emitted as Agent Trajectory Observability Format (ATOF) data.

metadata: Option<Json>#: Optional JSON metadata recorded on the start event.

model_name: Option<String>#: Optional normalized model name recorded separately from the request payload.

annotated_request: Option<Arc<AnnotatedLlmRequest>>#: Optional normalized request annotation produced by a codec.

timestamp: Option<DateTime<Utc>>#: Optional timestamp captured as the handle start time and reused by the emitted start event. When omitted, the current UTC time is used.

struct LlmHandle#

Runtime-owned handle identifying an active or completed LLM call.

uuid: Uuid#: Unique LLM-call identifier.

started_at: DateTime<Utc>#: Timestamp captured when the LLM handle was created.

name: String#: Provider or logical call name recorded on lifecycle events.

data: Option<Json>#: Optional application payload stored on the handle.

metadata: Option<Json>#: Optional metadata attached to the LLM span.

attributes: LlmAttributes#: LLM behavior flags.

parent_uuid: Option<Uuid>#: UUID of the parent scope, if any.

model_name: Option<String>#: Optional normalized model name for observability.

struct LlmRequest#

JSON-shaped LLM request payload passed through the runtime.

headers: serde_json::Map<String, Json>#: Provider-specific request headers.

content: Json#: Provider-specific request body.

struct LlmStreamCallExecuteParams#

Builder parameters for llm_stream_call_execute.

name: String#: Logical provider or model family name recorded on emitted events.

request: LlmRequest#: Raw request passed into the managed pipeline.

func: LlmStreamExecutionNextFn#: Streaming provider callback or execution continuation.

collector: LlmCollectorFn#: Per-chunk collector callback used to accumulate stream state.

finalizer: LlmFinalizerFn#: Finalizer callback used to construct the completed response.

parent: Option<ScopeHandle>#: Optional explicit parent scope for the emitted LLM span.

attributes: LlmAttributes#: LLM attribute bitflags applied to the managed span.

data: Option<Json>#: Optional application payload stored on the handle but not emitted as Agent Trajectory Observability Format (ATOF) data.

metadata: Option<Json>#: Optional JSON metadata recorded on emitted events.

model_name: Option<String>#: Optional normalized model name for observability output.

codec: Option<Arc<dyn LlmCodec>>#: Optional request codec used to produce annotated request data.

response_codec: Option<Arc<dyn LlmResponseCodec>>#: Optional response codec used to attach annotated response data.

mod llm#

`mod llm`#