nv-ingest

NVIDIA-Ingest Helm Charts

[!WARNING] NV-Ingest version 24.08 exposed Redis directly to the client, as such setup for the 24.08 nv-ingest-cli differs.

If using 24.08, refer to this section. However, we strongly recommend upgrading to 24.12+ when available.

Prerequisites

Hardware/Software

Refer to our supported hardware/software configurations here.

Setup Environment

NAMESPACE=nv-ingest
kubectl create namespace ${NAMESPACE}

# Nvidia nemo-microservices NGC repository
helm repo add nemo-microservices https://helm.ngc.nvidia.com/nvidia/nemo-microservices --username='$oauthtoken' --password=<NGC_API_KEY>

# Nvidia NIM NGC repository
helm repo add nvidia-nim https://helm.ngc.nvidia.com/nim/nvidia --username='$oauthtoken' --password=<NGC_API_KEY>

helm upgrade \
    --install \
    nv-ingest \
    https://helm.ngc.nvidia.com/nvidia/nemo-microservices/charts/nv-ingest-0.4.0.tgz \
    -n ${NAMESPACE} \
    --username '$oauthtoken' \
    --password "${NGC_API_KEY}" \
    --set imagePullSecret.create=true \
    --set imagePullSecret.password="${NGC_API_KEY}" \
    --set ngcSecret.create=true \
    --set ngcSecret.password="${NGC_API_KEY}" \
    --set image.repository="nvcr.io/nvidia/nemo-microservices/nv-ingest" \
    --set image.tag="24.12"

Optionally you can create your own versions of the Secrets if you do not want to use the creation via the helm chart.


NAMESPACE=nv-ingest
DOCKER_CONFIG='{"auths":{"nvcr.io":{"username":"$oauthtoken", "password":"'${NGC_API_KEY}'" }}}'
echo -n $DOCKER_CONFIG | base64 -w0
NGC_REGISTRY_PASSWORD=$(echo -n $DOCKER_CONFIG | base64 -w0 )

kubectl apply -n ${NAMESPACE} -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: nvcrimagepullsecret
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: ${NGC_REGISTRY_PASSWORD}
EOF
kubectl create -n ${NAMESPACE} secret generic ngc-api --from-literal=NGC_API_KEY=${NGC_API_KEY}

Alternatively, you can also use an External Secret Store like Vault, the name of the secret name expected for the NGC API is ngc-api and the secret name expected for NVCR is nvcrimagepullsecret.

In this case, make sure to remove the following from your helm commmand:

    --set imagePullSecret.create=true \
    --set imagePullSecret.password="${NGC_API_KEY}" \
    --set ngcSecret.create=true \
    --set ngcSecret.password="${NGC_API_KEY}" \

Minikube Setup

The PVC setup for minikube requires a little bit more configuraiton. Please follow the steps below if you are using minikube.

minikube start --driver docker --container-runtime docker --gpus all --nodes 3
minikube addons enable nvidia-gpu-device-plugin
minikube addons enable storage-provisioner-rancher

Usage

Jobs are submitted via the nv-ingest-cli command.

NV-Ingest CLI Installation

NV-Ingest uses a HTTP/Rest based submission method. By default the Rest service runs on port 7670.

[!TIP] This means that the nv-ingest-cli no longer uses a Redis client so users must use the appropriate version to ensure the client is not still trying to use the RedisClient.

First, build nv-ingest-cli from the source to ensure you have the latest code. More information is provided here.

# Just to be cautious we remove any existing installation
pip uninstall nv-ingest-cli

# Build the wheel from source
~~~~~~~[INSERT STEPS TO BUILD FROM SOURCE]~~~~~~~~~~~~~~~

# Pip install that .whl
pip install that wheel made above

Rest Endpoint Ingress

It is recommended that the end user provide a mechanism for Ingress for the NV-Ingest pod. You can test outside of your Kuberenetes cluster by port-forwarding the NV-Ingest pod to your local environment.

Example:

You can find the name of your NV-Ingest pod you want to forward traffic to by running:

kubectl get pods -n <namespace> --no-headers -o custom-columns=":metadata.name"
kubectl port-forward -n ${NAMESPACE} service/nv-ingest 7670:7670

The output will look similar to the following but with different auto-generated sequences.

nv-ingest-674f6b7477-65nvm
nv-ingest-cached-0
nv-ingest-deplot-0
nv-ingest-etcd-0
nv-ingest-milvus-standalone-7f8ffbdfbc-jpmlj
nv-ingest-minio-7cbd4f5b9d-99hl4
nv-ingest-opentelemetry-collector-7bb59d57fc-4h59q
nv-ingest-paddle-0
nv-ingest-redis-master-0
nv-ingest-redis-replicas-0
nv-ingest-yolox-0
nv-ingest-zipkin-77b5fc459f-ptsj6

kubectl port-forward -n ${NAMESPACE} nv-ingest-674f6b7477-65nvm 7670:7670

Executing Jobs

Here is a sample invocation of a PDF extraction task using the port forward above:

mkdir -p ./processed_docs

nv-ingest-cli \
  --doc /path/to/your/unique.pdf \
  --output_directory ./processed_docs \
  --task='extract:{"document_type": "pdf", "extract_text": true, "extract_images": true, "extract_tables": true}' \
  --client_host=localhost \
  --client_port=7670

You can also use NV-Ingest’s Python client API to interact with the service running in the cluster. Use the same host and port as in the previous nv-ingest-cli example.

Parameters

Deployment parameters

Name Description Value
affinity [default: {}] Affinity settings for deployment. {}
nodeSelector Sets node selectors for the NIM – for example nvidia.com/gpu.present: "true" {}
logLevel Log level of NV-Ingest service. Possible values of the variable are TRACE, DEBUG, INFO, WARNING, ERROR, CRITICAL. DEBUG
extraEnvVarsCM [default: “”] A Config map holding Environment variables to include in the NV-Ingest containerextraEnvVarsCM: “” ""
extraEnvVarsSecret [default: “”] A K8S Secret to map to Environment variables to include in the NV-Ingest container ""
fullnameOverride [default: “”] A name to force the fullname of the NV-Ingest container to have, defaults to the Helm Release Name ""
nameOverride [default: “”] A name to base the objects created by this helm chart ""
image.repository NIM Image Repository ""
image.tag Image tag or version ""
image.pullPolicy Image pull policy ""
podAnnotations Sets additional annotations on the main deployment pods {}
podLabels Specify extra labels to be add to on deployed pods. {}
podSecurityContext Specify privilege and access control settings for pod  
podSecurityContext.fsGroup Specify file system owner group id. 1000
extraVolumes Adds arbitrary additional volumes to the deployment set definition {}
extraVolumeMounts Specify volume mounts to the main container from extraVolumes {}
imagePullSecrets Specify list of secret names that are needed for the main container and any init containers.  
containerSecurityContext Sets privilege and access control settings for container (Only affects the main container, not pod-level) {}
tolerations Specify tolerations for pod assignment. Allows the scheduler to schedule pods with matching taints.  
replicaCount The number of replicas for NV-Ingest when autoscaling is disabled 1
resources.limits."nvidia.com/gpu" Specify number of GPUs to present to the running service.  
resources.limits.memory Specify limit for memory 32Gi
resources.requests.memory Specify request for memory 16Gi
tmpDirSize Specify the amount of space to reserve for temporary storage 8Gi

NIM Configuration

Define additional values to the dependent NIM helm charts by updating the “yolox-nim”, “cached-nim”, “deplot-nim”, and “paddleocr-nim” values. A sane set of configurations are already included in this value file and only the “image.repository” and “image.tag” fields are explicitly called out here.

Name Description Value
yolox-nim.image.repository The repository to override the location of the YOLOX  
yolox-nim.image.tag The tag override for YOLOX  
cached-nim.image.repository The repository to override the location of the Cached Model NIM  
cached-nim.image.tag The tag override for Cached Model NIM  
paddleocr-nim.image.repository The repository to override the location of the Paddle OCR NIM  
paddleocr-nim.image.tag The tag override for Paddle OCR NIM  
deplot-nim.image.repository The repository to override the location of the Deplot NIM  
deplot-nim.image.tag The tag override for Deplot NIM  

Milvus Deployment parameters

NV-Ingest uses Milvus and Minio to store extracted images from a document This chart by default sets up a Milvus standalone instance in the namespace using the Helm chart at found https://artifacthub.io/packages/helm/milvus-helm/milvus

Name Description Value
milvusDeployed Whether to deploy Milvus and Minio from this helm chart true
milvus Find values at https://artifacthub.io/packages/helm/milvus-helm/milvus sane {}

Autoscaling parameters

Values used for creating a Horizontal Pod Autoscaler. If autoscaling is not enabled, the rest are ignored. NVIDIA recommends usage of the custom metrics API, commonly implemented with the prometheus-adapter. Standard metrics of CPU and memory are of limited use in scaling NIM.

Name Description Value
autoscaling.enabled Enables horizontal pod autoscaler. false
autoscaling.minReplicas Specify minimum replicas for autoscaling. 1
autoscaling.maxReplicas Specify maximum replicas for autoscaling. 100
autoscaling.metrics Array of metrics for autoscaling. []

Redis configurations

Include any redis configuration that you’d like with the deployed Redis Find values at https://github.com/bitnami/charts/tree/main/bitnami/redis

Name Description Value
redisDeployed Whether to deploy Redis from this helm chart true
redis Find values at https://github.com/bitnami/charts/tree/main/bitnami/redis sane {}

Environment Variables

Define environment variables as key/value dictionary pairs

Name Description Value
envVars Adds arbitrary environment variables to the main container using key-value pairs, for example NAME: value sane {}
envVars.MESSAGE_CLIENT_HOST Override this value if disabling Redis deployment in this chart. "nv-ingest-redis-master"
envVars.MESSAGE_CLIENT_PORT Override this value if disabling Redis deployment in this chart. "7670"
envVars.NV_INGEST_DEFAULT_TIMEOUT_MS Override the Timeout of the NV-Ingest requests. "1234"
envVars.MINIO_INTERNAL_ADDRESS Override this to the cluster local DNS name of minio "nv-ingest-minio:9000"
envVars.MINIO_PUBLIC_ADDRESS Override this to publicly routable minio address, default assumes port-forwarding "http://localhost:9000"
envVars.MINIO_BUCKET Override this for specific minio bucket to upload extracted images to "nv-ingest"

Open Telemetry

Define environment variables as key/value dictionary pairs for configuring OTEL Deployments A sane set of parameters is set for the deployed version of OpenTelemetry with this Helm Chart. Override any values to the Open Telemetry helm chart by overriding the open-telemetry value.

Name Description Value
otelEnabled Whether to enable OTEL collection true
otelDeployed Whether to deploy OTEL from this helm chart true
opentelemetry-collector Configures the opentelemetry helm chart - see https://github.com/open-telemetry/opentelemetry-helm-charts/blob/main/charts/opentelemetry-collector/values.yaml  
otelEnvVars Adds arbitrary environment variables for configuring OTEL using key-value pairs, for example NAME: value sane {}
otelEnvVars.OTEL_EXPORTER_OTLP_ENDPOINT Default deployment target for GRPC otel - Default “http://-opentelemetry-collector:4317”  
otelEnvVars.OTEL_SERVICE_NAME   "nemo-retrieval-service"
otelEnvVars.OTEL_TRACES_EXPORTER   "otlp"
otelEnvVars.OTEL_METRICS_EXPORTER   "otlp"
otelEnvVars.OTEL_LOGS_EXPORTER   "none"
otelEnvVars.OTEL_PROPAGATORS   "tracecontext baggage"
otelEnvVars.OTEL_RESOURCE_ATTRIBUTES   "deployment.environment=$(NAMESPACE)"
otelEnvVars.OTEL_PYTHON_EXCLUDED_URLS   "health"
zipkinDeployed Whether to deploy Zipkin with OpenTelemetry from this helm chart true

Ingress parameters

Name Description Value
ingress.enabled Enables ingress. false
ingress.className Specify class name for Ingress. ""
ingress.annotations Specify additional annotations for ingress. {}
ingress.hosts Specify list of hosts each containing lists of paths.  
ingress.hosts[0].host Specify name of host. chart-example.local
ingress.hosts[0].paths[0].path Specify ingress path. /
ingress.hosts[0].paths[0].pathType Specify path type. ImplementationSpecific
ingress.tls Specify list of pairs of TLS secretName and hosts. []

Probe parameters

| Name | Description | Value | | ———————————– | —————————————– | ——— | | livenessProbe.enabled | Enables livenessProbe | false | | livenessProbe.httpGet.path | LivenessProbeendpoint path | `/health` | | `livenessProbe.httpGet.port` | `LivenessProbe endpoint port | http | | livenessProbe.initialDelaySeconds | Initial delay seconds for livenessProbe | 120 | | livenessProbe.timeoutSeconds | Timeout seconds for livenessProbe | 20 | | livenessProbe.periodSeconds | Period seconds for livenessProbe | 10 | | livenessProbe.successThreshold | Success threshold for livenessProbe | 1 | | livenessProbe.failureThreshold | Failure threshold for livenessProbe | 20 |

Probe parameters

| Name | Description | Value | | ———————————- | —————————————- | ——— | | startupProbe.enabled | Enables startupProbe | false | | startupProbe.httpGet.path | StartupProbeendpoint path | `/health` | | `startupProbe.httpGet.port` | `StartupProbe endpoint port | http | | startupProbe.initialDelaySeconds | Initial delay seconds for startupProbe | 120 | | startupProbe.timeoutSeconds | Timeout seconds for startupProbe | 10 | | startupProbe.periodSeconds | Period seconds for startupProbe | 30 | | startupProbe.successThreshold | Success threshold for startupProbe | 1 | | startupProbe.failureThreshold | Failure threshold for startupProbe | 220 |

Probe parameters

| Name | Description | Value | | ———————————— | —————————————— | ——— | | readinessProbe.enabled | Enables readinessProbe | false | | readinessProbe.httpGet.path | ReadinessProbeendpoint path | `/health` | | `readinessProbe.httpGet.port` | `ReadinessProbe endpoint port | http | | readinessProbe.initialDelaySeconds | Initial delay seconds for readinessProbe | 120 | | readinessProbe.timeoutSeconds | Timeout seconds for readinessProbe | 10 | | readinessProbe.periodSeconds | Period seconds for readinessProbe | 30 | | readinessProbe.successThreshold | Success threshold for readinessProbe | 1 | | readinessProbe.failureThreshold | Failure threshold for readinessProbe | 220 |

Service parameters

Name Description Value
service.type Specifies the service type for the deployment. ClusterIP
service.name Overrides the default service name ""
service.port Specifies the HTTP Port for the service. 8000
service.nodePort Specifies an optional HTTP Node Port for the service. nil
service.annotations Specify additional annotations to be added to service. {}
service.labels Specifies additional labels to be added to service. {}
serviceAccount Options to specify service account for the deployment.  
serviceAccount.create Specifies whether a service account should be created. true
serviceAccount.annotations Sets annotations to be added to the service account. {}
serviceAccount.name Specifies the name of the service account to use. If it is not set and create is “true”, a name is generated using a “fullname” template. ""

Secret Creation

Manage the creation of secrets used by the helm chart

Name Description Value
ngcSecret.create Specifies whether to create the ngc api secret false
ngcSecret.password The password to use for the NGC Secret ""
imagePullSecret.create Specifies whether to create the NVCR Image Pull secret false
imagePullSecret.password The password to use for the NVCR Image Pull Secret ""

24.08 CLI Setup and Usage

NV-Ingest CLI Installation: 24.08

You can find the Python wheel for the nv-ingest-cli located in our NV-Ingest 24.08 release artifacts. Installation of the nv-ingest-cli goes as follows.

# Just to be cautious we remove any existing installation
pip uninstall nv-ingest-cli

# Download the 24.08 .whl
wget https://github.com/NVIDIA/nv-ingest/releases/download/24.08/nv_ingest_client-24.08-py3-none-any.whl

# Pip install that .whl
pip install nv_ingest_client-24.08-py3-none-any.whl

Access To Redis: 24.08

It is recommended that the end user provide a mechanism for Ingress for the Redis pod. You can test outside of your Kuberenetes cluster by port-forwarding the Redis pod to your local environment.

Example:

kubectl port-forward -n ${NAMESPACE} nv-ingest-redis-master-0 6379:6379

Executing Jobs: 24.08

Here is a sample invocation of a PDF extraction task using the port forward above:

mkdir -p ./processed_docs

nv-ingest-cli \
  --doc /path/to/your/unique.pdf \
  --output_directory ./processed_docs \
  --task='extract:{"document_type": "pdf", "extract_text": true, "extract_images": true, "extract_tables": true}' \
  --client_host=localhost \
  --client_port=6379
This site is open source. Improve this page.