Skip to content

Quickstart: retriever CLI

Use retriever ingest and retriever query for product-facing workflows. Use retriever harness run only for benchmark execution.

Quick start

Ingest a PDF locally

retriever ingest ./data/multimodal_test.pdf

Then query the default LanceDB table:

retriever query "What is in this document?"

By default, local ingest auto-detects supported input formats and writes to lancedb/nemo-retriever; retriever query reads from the same table. Use explicit high-level options when a task needs behavior beyond the current ingest defaults.

The plain retriever query examples below apply to local and batch ingest output written to LanceDB. Use retriever query service to query a Retriever service.

Ingest a larger corpus with batch mode

retriever ingest batch ./data/pdf_corpus \
  --profile fast-text \
  --pdf-extract-workers 4 \
  --embed-workers 2

Batch mode exposes Ray runtime and batch tuning flags such as --ray-address, --pdf-extract-workers, --ocr-workers, and --embed-workers.

Ingest through a Retriever service

retriever ingest service ./data/pdf_corpus \
  --service-url http://localhost:7670 \
  --service-concurrency 8

Use --service-api-token or NEMO_RETRIEVER_API_TOKEN when the service requires a bearer token. Service ingest does not expose --lancedb-uri; the service configures its vector database. Query the service with:

retriever query service "What is in this corpus?" \
  --service-url http://localhost:7670

Route ingest to hosted or self-hosted NIM endpoints

export NVIDIA_API_KEY=nvapi-...

retriever ingest ./data/multimodal_test.pdf \
  --page-elements-invoke-url https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-page-elements-v3 \
  --ocr-invoke-url https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-ocr-v2 \
  --table-structure-invoke-url https://ai.api.nvidia.com/v1/cv/nvidia/nemotron-table-structure-v1 \
  --embed-invoke-url https://integrate.api.nvidia.com/v1/embeddings \
  --embed-model-name nvidia/llama-nemotron-embed-1b-v2

NVIDIA_API_KEY is required only when those URLs point at hosted build.nvidia.com endpoints. NGC_API_KEY is used separately when pulling or running self-hosted NIM containers.

For NVIDIA inference hub rerank models that expose the Cohere-style rerank route, pass the full /v1/rerank URL and the model name shown in the hub snippet:

export NGC_INFERENCE_API_KEY=...

retriever query "What is in this document?" \
  --embed-invoke-url https://integrate.api.nvidia.com/v1/embeddings \
  --embed-model-name nvidia/llama-nemotron-embed-1b-v2 \
  --reranker-invoke-url https://inference-api.nvidia.com/v1/rerank \
  --reranker-model-name nvidia/nvidia/llama-3.2-nv-rerankqa-1b-v2 \
  --reranker-api-key-env NGC_INFERENCE_API_KEY

Query result controls

Both retriever query and retriever query service return compact JSON hits with source, page_number, and text. Use --candidate-k, --page-dedup, and --content-types to control how results are selected after vector retrieval:

retriever query "annual revenue by region" \
  --top-k 5 \
  --candidate-k 40 \
  --content-types table

--top-k is the final number of results to return after filtering and deduplication. --candidate-k is the number of raw results to retrieve from LanceDB or the Retriever service before filtering, page deduplication, and final truncation. If omitted, the candidate pool is the same size as --top-k. Set --candidate-k larger than --top-k when page deduplication or content-type filtering might remove too many of the nearest retrieved rows. It must always be greater than or equal to --top-k.

Page deduplication and content-type filtering are applied after vector retrieval, preserving retriever ranking order and truncating the final output to --top-k. Local and batch ingest record the canonical embedding model on the LanceDB table, and non-service query uses that model automatically. Use --embed-model-name only as an explicit override or when querying a legacy or third-party table without model metadata. Endpoint URLs and provider prefixes remain runtime configuration, so continue to pass --embed-invoke-url and --embed-model-provider-prefix when the selected model must be routed remotely. For example, a table can store the canonical model nvidia/llama-nemotron-embed-vl-1b-v2 while a LiteLLM-routed request uses nvidia/nvidia/llama-nemotron-embed-vl-1b-v2. The endpoint and routing prefix are intentionally not persisted on the table.

--content-types accepts comma-separated content types such as text, table, chart, image, and infographic. images is accepted as an alias for captioned image rows emitted by ingest. This option filters by content-type metadata only; it does not filter by source, page, or other metadata predicates. Hits with missing or unknown content-type metadata are excluded while --content-types is active. In service mode, results must include content-type metadata to match this filter. Default display values in the JSON output are not used for content-type matching.

Agentic retrieval

--agentic swaps the single dense pass for an LLM-driven ReAct loop: the agent issues several retrieval sub-queries, fuses the candidates, and selects a final ranking. It searches the same LanceDB table built by retriever ingest, so it is a drop-in alternative to standard retrieval.

By default, agentic retrieval runs the agent LLM in process with local vLLM and nemotron-8b (nvidia/Llama-3.1-Nemotron-Nano-8B-v1). This requires a CUDA GPU host and the local extras installed. Provide --agentic-invoke-url when you want a custom model or a separately hosted OpenAI-compatible endpoint.

# default local vLLM agent LLM: nemotron-8b
retriever query "how does the ingestion pipeline handle tables?" \
  --agentic

# custom/self-hosted model through an OpenAI-compatible endpoint
retriever query "summarize the deployment options" \
  --agentic \
  --agentic-llm-model custom-remote-model \
  --agentic-invoke-url http://localhost:9000/v1/chat/completions \
  --embed-invoke-url http://localhost:8000/v1 \
  --agentic-react-max-steps 5

Unlike the dense path (which returns text-enriched hits), agentic mode returns the agent's ranked document IDs as JSON, each annotated with the source that produced it (final_results, rrf, or selection_agent). It reuses the same --top-k, --lancedb-uri, --table-name, --embed-invoke-url, and --embed-model-name options as standard retrieval. Agentic retrieval uses the selected table's model automatically when --embed-model-name is omitted.

How it works. Each agentic query runs Query -> ReActAgentOperator -> (RRF fusion) -> SelectionAgentOperator -> ranked results:

  • ReActAgentOperator runs the per-query ReAct loop; every retrieve tool call delegates to the standard Retriever, so the agent searches the same vector DB and embedding config as dense retrieval.
  • RRFAggregatorOperator fuses candidates from the loop's multiple searches with reciprocal rank fusion.
  • SelectionAgentOperator runs a final LLM selection pass over the fused set and emits the ranked document IDs.

Agentic-only knobs (apply only with --agentic):

  • --agentic-llm-model — local profile alias/model ID when no invoke URL is provided (nemotron-8b by default; super-49b also supported), or the remote model ID when --agentic-invoke-url is provided.
  • --agentic-invoke-url — OpenAI-compatible chat-completions endpoint for the agent LLM. Providing it routes agent LLM calls to that remote endpoint; omit it to run the in-process local model.
  • --agentic-llm-client (optional) — LLM client that builds the agent LLM. Defaults to callable. It drives the in-process adapter when --agentic-invoke-url is omitted, and the shared chat-completions HTTP client when it is set.
  • --agentic-reasoning-effort (default high) — reasoning_effort forwarded on OpenAI-compatible agentic LLM calls; ignored by the local adapter.
  • --agentic-react-max-steps (default 50) — maximum ReAct loop iterations.
  • --agentic-text-truncation (default 0) — max characters of each candidate shown to the agent; 0 disables truncation.
  • --agentic-temperature (default: unset) — sampling temperature for agent LLM calls; omit to use the endpoint/model default (0.0 = greedy). Local and non-NVIDIA OpenAI-compatible endpoints allow up to 2.0; NVIDIA-hosted endpoints allow up to 1.0.