mod adaptive_hints_intercept

module adaptive_hints_intercept

AdaptiveHintsIntercept: opt-in LLM request intercept that injects AgentHints from HotCache trie.

This module provides AdaptiveHintsIntercept, which builds AgentHints from the prediction trie in HotCache and injects them into LLM request headers as a request intercept. AdaptiveHintsIntercept is opt-in and synchronously transforms the LlmRequest before it reaches the callable.

Structs and Unions

struct AdaptiveHintsIntercept

Opt-in LLM request intercept that injects AgentHints into request headers from the prediction trie in HotCache.

Constructed via AdaptiveHintsIntercept::new and converted to an LlmRequestInterceptFn via AdaptiveHintsIntercept::into_request_fn for registration with the NeMo Flow runtime.

Implementations

impl AdaptiveHintsIntercept

Functions

fn into_request_fn(self) -> LlmRequestInterceptFn

Converts this intercept into an LlmRequestInterceptFn suitable for registration with register_llm_request_intercept.

The returned closure reads the HotCache trie, builds AgentHints, injects them into the request headers and body, and returns the transformed request.

fn new(hot_cache: Arc<RwLock<HotCache>>, agent_id: String) -> Self

Creates a new AdaptiveHintsIntercept.