Type-erased resource wrappers

With the property design depicted in cuda::get_property, a library has flexibility in checking constraints and querying custom properties. However, there is also a cost in providing function templates for a potentially wide range of inputs. Depending on the number of different memory resources, both compile time and binary size might increase considerably.

The type-erased wrappers let you coalesce such APIs into a single function. Both resource_ref and any_resource preserve property constraints while erasing the concrete resource type.

Wrapper	Ownership	Typical use
cuda::mr::resource_ref	Non-owning	Accept references/pointers without extending lifetime
cuda::mr::any_resource	Owning	Store a resource with the object that uses it

Common usage

Both wrappers provide a non-templated API surface for allocation and property queries. Choose which wrapper to use based on ownership, then the usage patterns are the same.

resource_ref is constructible from any non-const reference or pointer to a memory resource that satisfies cuda::mr::resource. any_resource is constructible from a resource object and takes ownership of it.

Properties may be passed to both wrappers just as with cuda::mr::resource_with.

void* do_allocate(cuda::mr::resource_ref<> resource, cuda::stream_ref stream, std::size_t size, std::size_t align) {
    return resource.allocate(stream, size, align);
}

void* do_allocate_owned(cuda::mr::any_resource<> resource, cuda::stream_ref stream, std::size_t size, std::size_t align) {
    return resource.allocate(stream, size, align);
}

my_memory_resource resource;
my_memory_resource* pointer_to_resource = &resource;

void* from_reference = do_allocate(resource, stream, 1337, 256);
void* from_ptr = do_allocate(pointer_to_resource, stream, 1337, 256);
void* from_owned = do_allocate_owned(cuda::mr::any_resource<>{resource}, stream, 1337, 256);

resource_ref

cuda::mr::resource_ref is the non-owning, type-erased wrapper. Prefer it when the caller controls the resource lifetime.

struct required_alignment {
    using value_type = std::size_t;
};
void* do_allocate_with_alignment(cuda::mr::resource_ref<required_alignment> resource, cuda::stream_ref stream, std::size_t size) {
    return resource.allocate(stream, size, get_property(resource, required_alignment{}));
}

However, the type erasure comes with the cost that arbitrary properties cannot be queried from either wrapper:

struct required_alignment {
    using value_type = std::size_t;
};
void* buggy_allocate_with_alignment(cuda::mr::resource_ref<> resource, cuda::stream_ref stream, std::size_t size) {
    if constexpr (cuda::has_property<decltype(resource), required_alignment>) { // BUG: This will always be false
        return resource.allocate(stream, size, get_property(resource, required_alignment{}));
    } else {
        return resource.allocate(stream, size, my_default_alignment);
    }
}

So, choose wisely. If your library has a well-defined set of fixed properties that you expect to always be available, then cuda::mr::{synchronous_}resource_ref is an amazing tool to improve compile times and binary size. If you need a flexible interface then constraining a template argument through cuda::mr::{synchronous_}resource_with is the proper solution.

any_resource

cuda::mr::any_resource is the owning counterpart. It is especially suited for containers that must ensure the resource outlives the container.

cuda::mr::make_any_resource<Resource, Properties...>(args...) constructs an any_resource<Properties...> that owns a newly constructed Resource. Resource must satisfy cuda::mr::resource_with<Resource, Properties...>.

#include <cuda/devices>
#include <cuda/memory_resource>
#include <cuda/stream>

void use_any_resource(cuda::stream_ref stream) {
  // Wrap a device memory resource
  cuda::mr::any_resource<cuda::mr::device_accessible> resource{
    cuda::device_default_memory_pool(cuda::devices[0])
  };

  // Allocate memory
  void* ptr = resource.allocate(stream, 1024, 16);

  // Use memory...

  // Deallocate
  resource.deallocate(stream, ptr, 1024, 16);
}

Synchronous variants

The synchronous wrappers mirror the same ownership split: synchronous_resource_ref is non-owning, while any_synchronous_resource owns the resource instance.

cuda::mr::any_synchronous_resource

cuda::mr::any_synchronous_resource is a type-erased wrapper for resources that satisfy cuda::mr::synchronous_resource. It can be constructed from an any_resource, but only exposes synchronous allocation/deallocation.

cuda::mr::make_any_synchronous_resource<Resource, Properties...>(args...) constructs an any_synchronous_resource<Properties...> that owns a newly constructed Resource. Resource must satisfy cuda::mr::synchronous_resource_with<Resource, Properties...>.

#include <cuda/memory_resource>

void use_sync_resource() {
  cuda::mr::any_synchronous_resource<cuda::mr::host_accessible> resource{
    cuda::mr::legacy_pinned_memory_resource{}
  };

  // Synchronous allocation
  void* ptr = resource.allocate_sync(1024, 16);

  // Use memory...

  // Synchronous deallocation
  resource.deallocate_sync(ptr, 1024, 16);
}

resource_cast

cuda::mr::resource_cast<T>(&resource) returns a pointer to the concrete resource stored in a type-erased wrapper when the stored resource has type T. It returns nullptr when the stored type does not match.

This is similar to pointer-form dynamic_cast<T*> in that callers must check for nullptr before using the result. Unlike dynamic_cast, resource_cast matches the exact type stored in the wrapper.

resource_cast works with any_resource, any_synchronous_resource, resource_ref, and synchronous_resource_ref.

#include <cuda/devices>
#include <cuda/memory_resource>
#include <cuda/stream>

void use_resource_cast(cuda::stream_ref stream) {
  cuda::mr::any_resource<cuda::mr::device_accessible> resource{
    cuda::device_default_memory_pool(cuda::devices[0])
  };

  if (auto* pool = cuda::mr::resource_cast<cuda::device_memory_pool_ref>(&resource)) {
    pool->trim_to(1024 * 1024);

    void* ptr = pool->allocate(stream, 1024, 16);
    pool->deallocate(stream, ptr, 1024, 16);
  }
}

try_get_property

Type-erased wrapper conversions can discard properties from the wrapper’s type. try_get_property(resource, property) can query a property that was present before such a conversion, even when it is not listed in the resulting wrapper’s property set. For stateless properties, it returns bool. For stateful properties, it returns cuda::std::optional<Property::value_type>.

#include <cuda/memory_resource>

bool is_device_accessible(cuda::mr::any_synchronous_resource<> resource) {
  return try_get_property(resource, cuda::mr::device_accessible{});
}

dynamic_resource_cast

cuda::mr::dynamic_resource_cast<Properties...>(resource) casts a type-erased resource wrapper to the same wrapper kind with a different property set. It checks the stored resource at run time and throws if the stored resource does not support the requested properties.

#include <cuda/memory_resource>

cuda::mr::any_resource<cuda::mr::device_accessible>
require_device_accessible(cuda::mr::any_resource<> resource) {
  return cuda::mr::dynamic_resource_cast<cuda::mr::device_accessible>(cuda::std::move(resource));
}

cuda::mr::synchronous_resource_ref

cuda::mr::synchronous_resource_ref provides the same type-erased reference behavior as resource_ref, but only exposes synchronous allocation and deallocation.

#include <cuda/memory_resource>

// indicates that the resource must be host-accessible
void use_sync_ref(
  cuda::mr::synchronous_resource_ref<cuda::mr::host_accessible> resource
) {
  // Synchronous allocation
  void* ptr = resource.allocate_sync(1024, 16);

  // Use memory...

  // Synchronous deallocation
  resource.deallocate_sync(ptr, 1024, 16);
}