Implementing the Ingest Docs Method

Implementing the Method

  1. Edit the RetrievalAugmentedGeneration/examples/simple_rag_api_catalog/chains.py file and add the following statements after the import statements.

    The following statements use the FAISS vector store for the embeddings that the NVIDIA API Catalog model creates.

    import os
from langchain.vectorstores import FAISS

vector_store_path = "vectorstore.pkl"
vector_store = None

  2. Add the import statements for libraries:

    from langchain.document_loaders import UnstructuredFileLoader
from langchain.text_splitter import CharacterTextSplitter
from RetrievalAugmentedGeneration.common.utils import get_config, get_llm, get_embedding_model

document_embedder = get_embedding_model()
settings = get_config()

  3. Update the ingest_docs method with the following statements:

        def ingest_docs(self, data_dir: str, filename: str):
        """Code to ingest documents"""
        try:
            global vector_store

            raw_documents = UnstructuredFileLoader(data_dir).load()
            if raw_documents:
                text_splitter = CharacterTextSplitter(chunk_size=settings.text_splitter.chunk_size,
                                                      chunk_overlap=settings.text_splitter.chunk_overlap)
                documents = text_splitter.split_documents(raw_documents)
                if vector_store:
                    vector_store.add_documents(documents)
                else:
                    vector_store = FAISS.from_documents(documents, document_embedder)
                    logger.info("Vector store created and saved.")
            else:
                logger.warning(f"No documents in '{DOCS_DIR}' to process.")
        except Exception as e:
            logger.error(f"Failed to ingest document: {e}")
            raise ValueError("Failed to upload document. Upload an unstructured text or PDF file.")

Building and Running with Docker Compose

Using the containers has one additional step this time: exporting your NVIDIA API key as an environment variable.

  1. Build the container for the Chain Server:

    $ docker compose --env-file deploy/compose/compose.env -f deploy/compose/simple-rag-api-catalog.yaml build chain-server

  2. Export your NVIDIA API key in an environment variable:

    $ export NVIDIA_API_KEY=nvapi-...

  3. Run the containers:

    $ docker compose --env-file deploy/compose/compose.env -f deploy/compose/simple-rag-api-catalog.yaml up -d

Verify the Ingest Docs Method Using Curl

You can access the Chain Server with a URL like http://localhost:8081.

  • Confirm the ingest_docs method runs by uploading a sample document, such as the README from the repository:

    $ curl http://localhost:8081/documents -F "file=@README.md"

    Example Output

    {"message":"File uploaded successfully"}

    View the logs for the Chain Server to see the logged message from the method:

    $ docker logs chain-server

    Example Output

    INFO:unstructured:Reading document from string ...
INFO:unstructured:Reading document ...
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /root/nltk_data...
[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.
INFO:unstructured:HTML element instance has no attribute type
INFO:unstructured:HTML element instance has no attribute type
INFO:unstructured:HTML element instance has no attribute type
INFO:unstructured:HTML element instance has no attribute type
INFO:unstructured:HTML element instance has no attribute type
INFO:unstructured:HTML element instance has no attribute type
INFO:unstructured:HTML element instance has no attribute type
INFO:unstructured:HTML element instance has no attribute type
INFO:unstructured:HTML element instance has no attribute type
INFO:unstructured:HTML element instance has no attribute type
INFO:unstructured:HTML element instance has no attribute type
WARNING:langchain.text_splitter:Created a chunk of size 1107, which is longer than the specified 510
WARNING:langchain.text_splitter:Created a chunk of size 701, which is longer than the specified 510
INFO:faiss.loader:Loading faiss with AVX2 support.
INFO:faiss.loader:Successfully loaded faiss with AVX2 support.
INFO:example:Vector store created and saved.

Next Steps