Troubleshooting Guide#
Use this page when a NeMo Flow setup, build, or runtime workflow does not behave as expected.
Package Or Build Setup Fails#
Confirm that your environment matches Prerequisites, then rerun the binding-specific setup command from Installation.
If a command worked previously and now fails, check whether a toolchain update changed the active Rust, Python, Node.js, Go, or WebAssembly version. Recreate generated artifacts after switching versions, especially Python virtual environments, Node.js node_modules, Go build caches, and WebAssembly pkg/ output.
Rust Workspace Fails To Build#
Run the narrowest failing build first:
cargo build -p nemo-flow
If the core crate builds but another crate fails, rerun the binding-specific build or test command from Testing and Documentation. Binding crates often depend on generated native artifacts, host toolchain headers, or runtime-specific test setup.
Python Native Module Does Not Import#
If import nemo_flow fails, rebuild the Python environment and native extension:
uv sync
uv run pytest python/tests/test_types.py
Confirm that Python satisfies the supported version from Prerequisites. If multiple Python installations are available, make sure the uv environment is the one running the test or application process.
Node.js Native Addon Does Not Load#
If Node.js reports a missing or incompatible native addon, reinstall dependencies from the Node binding directory so the pretest build can regenerate the local addon:
cd crates/node
npm install
npm test
Use Node.js Getting Started to confirm that the example runs against the generated local build, not a stale package or a globally installed copy.
Go Tests Cannot Find The FFI Library#
The Go binding loads the shared FFI library through CGo. Build the release FFI library before running Go tests, and point the linker and runtime loader at the release target directory:
cargo build --release -p nemo-flow-ffi
cd go/nemo_flow
CGO_LDFLAGS="-L../../target/release" LD_LIBRARY_PATH="../../target/release" go test -race -v ./...
On macOS, use DYLD_LIBRARY_PATH="../../target/release" instead of LD_LIBRARY_PATH when the runtime loader cannot find the library.
WebAssembly Tests Or Package Builds Fail#
Confirm that wasm-pack is installed and matches the minimum version in Prerequisites, then rerun the WebAssembly build or tests:
wasm-pack build crates/wasm
wasm-pack test --node crates/wasm
If generated files appear stale, remove the WebAssembly package output and rebuild it from the repository root.
Scope Stack Is Empty#
A scope-stack error usually means runtime work is executing outside an active scope or on a thread that did not receive the intended scope stack.
Use Scopes and Instrument Applications Code Examples to choose the correct scope stack helper for the binding and concurrency model.
Work Leaks Across Requests#
Unexpected shared scopes, middleware, or events across concurrent requests usually means more than one request is using the same scope stack. Create a fresh scope stack per request or agent, and pass that stack into async tasks, threads, callbacks, or worker boundaries that continue the request.
Use Adding Scopes and Marks and Scopes to verify that the root scope matches the request boundary.
Middleware Does Not Run#
Check whether the middleware was registered globally or scope-locally. Scope-local middleware is visible only to the owning scope and descendant scopes, and it is cleaned up when the owning scope closes.
Use Middleware, Advanced Guide: Add Middleware, and Instrument Applications Code Examples to verify the expected registration family.
Middleware Runs In The Wrong Order#
Middleware runs by priority after the runtime merges global middleware with scope-local middleware from the active scope chain. If the order is surprising, check the priority assigned to each middleware entry and confirm that similarly named global and scope-local entries are registered in the intended registry.
Use Middleware and Instrument Applications Code Examples to compare registration names, priorities, and scope ownership.
Guardrail Rejections Stop Calls#
A guardrail rejection is expected to stop the protected tool or LLM call. Inspect the guardrail result and confirm whether the guardrail was intended to sanitize input, sanitize output, or reject the request completely.
Use Advanced Guide: Add Middleware to verify the guardrail family and expected behavior.
Request Intercept Changes Are Not Visible#
Request intercepts transform the request before execution. If the original value still appears downstream, confirm that the intercept returns the changed value in the binding-specific shape and that a later request intercept is not replacing it.
Use Instrument Applications Code Examples to compare the expected request intercept callback signature for the binding.
Execution Intercept Hangs Or Skips The Original Call#
Execution intercepts use a middleware-chain pattern. An intercept that never calls next intentionally short-circuits the original callable, while an intercept that awaits or calls next incorrectly can hang the request.
Use Middleware to confirm when an execution intercept should call next, replace the result, or stop the chain.
Subscribers Do Not Receive Events#
Confirm that the subscriber is registered before the runtime emits the events you expect. For scope-local subscribers, confirm that the active scope is the owning scope or a descendant scope.
Use Subscribers, Events, and Export Observability Data to verify lifecycle timing and event names.
Events Are Missing Expected Fields#
Managed tool and LLM helpers populate semantic fields such as inputs, outputs, model names, and tool call IDs. Manual lifecycle calls require you to provide the relevant params explicitly.
Use Export Observability Data Code Examples and Instrument Applications Code Examples to verify the emitted payload shape.
ATIF Export Is Empty Or Mixed Across Agents#
An empty ATIF export usually means the exporter subscribed after the relevant events were emitted, or the export filter does not match the active root_uuid. Mixed trajectories usually mean multiple agents share a root scope or the export did not filter by root scope.
Use ATIF Export and Export Observability Data Code Examples to confirm exporter setup, event collection timing, and root-scope filtering.
LLM Stream Output Is Missing The Final Chunk#
When wrapping streamed LLM responses, confirm that the stream wrapper receives the provider’s terminal event and that the application drains the stream until completion. A stream that is dropped early can prevent finalizers, collectors, or subscribers from seeing the completed output.
Use Wrap LLM Calls and Provider Response Codecs to verify the expected stream event format.
Provider Payloads Fail To Convert#
JSON conversion errors usually mean the integration passed a value that cannot be represented in NeMo Flow’s JSON model, such as functions, class instances, handles, or provider-specific streaming objects.
Use Non-Serializable Data, Provider Codecs, and Using Codecs to define explicit conversions for provider-specific payloads.
Plugin Configuration Does Not Validate#
If a plugin fails before runtime execution, validate configuration separately from behavior registration. Check required fields, value types, defaults, and whether the plugin is reading configuration from the expected source.
Use Validate Configuration, Advanced Configuration, and Register Behavior to isolate configuration problems from runtime behavior problems.
Adaptive Optimization Does Not Change Behavior#
Confirm that adaptive optimization is configured for the component you expect and that the runtime path actually reaches that component. If behavior does not change, check whether the configured policy is disabled, scoped too narrowly, or not connected to the call path under test.
Use Configure Adaptive Optimization and Adaptive Components to verify component names and configuration scope.
Third-Party Patch Does Not Apply#
Run the wrapper command from the repository root:
./scripts/apply-patches.sh --check
If the patch still does not apply, confirm that the local checkout is clean and matches the pinned commit in third_party/sources.lock.
Third-Party Integration Behaves Differently From Core APIs#
First reproduce the behavior through the closest core or binding-level API. If the core API behaves correctly, inspect the integration wrapper, codec, or provider adapter that translates provider calls into NeMo Flow calls.
Use Integrate Frameworks, Wrap Tool Calls, and Wrap LLM Calls to compare the integration path with the core runtime path.