Overview

If you are a customer looking for information on how to adopt the cuDF plugin for your Spark workloads, please go to our User Guide for more information: link.

The cuDF plugin leverages GPUs to accelerate processing via the RAPIDS libraries.

As data scientists shift from using traditional analytics to leveraging AI(DL/ML) applications that better model complex market demands, traditional CPU-based processing can no longer keep up without compromising either speed or cost. The growing adoption of AI in analytics has created the need for a new framework to process data quickly and cost-efficiently with GPUs.

The cuDF plugin combines the power of the cuDF library and the scale of the Spark distributed computing framework. The cuDF plugin also has a built-in accelerated shuffle based on UCX that can be configured to leverage GPU-to-GPU communication and RDMA capabilities.