StringTensor

class sdm.tensor.StringTensor(data: Tensor, offset: Tensor, valid: Tensor | None, size: Sequence[int], stride: Sequence[int] | None = None, storage_offset: int = 0)

Bases: VarLenTensor

A torch.Tensor for UTF-8 encoded string values.

Parameters:

• data (Tensor) – Flat contiguous uint8 tensor containing all string values.

• offset (Tensor) – One-dimensional offsets into data.

• valid (Tensor | None) – One-dimensional mask indicating valid, non-null string values.

• size (Sequence[int]) – The shape of the tensor.

• stride (Sequence[int] | None) – The stride of the tensor.

• storage_offset (int) – The offset into the logical offset storage.

Return type:

Self

classmethod from_arrow(array: Array | ChunkedArray, *, size: Sequence[int] | None = None, device: device | str | None = None) → Self

Create tensor from a string pyarrow.Array.

import pyarrow as pa
from sdm import StringTensor

array = pa.array(["foo", "bar", "hello world", ""])
tensor = StringTensor.from_arrow(array, size=(2, 2))

Parameters:

• array (Array | ChunkedArray) – The string pyarrow.Array or pyarrow.ChunkedArray.

• size (Sequence[int] | None) – The shape of the tensor.

• device (device | str | None) – The device.

Return type:

Self

to_arrow() → Array

Convert this tensor to a flat pyarrow.Array.

Return type:

Array

classmethod from_cudf(ser: cudf.Series | cudf.Index, *, size: Sequence[int] | None = None, device: torch.device | str | None = None) → Self

Create tensor from a string cudf.Series.

Parameters:

• ser (cudf.Series | cudf.Index) – The string cudf.Series or cudf.Index.

• size (Sequence[int] | None) – The shape of the tensor.

• device (torch.device | str | None) – The device.

Return type:

Self

to_cudf() → cudf.Series

Convert this CUDA tensor to a flat cudf.Series.

Return type:

cudf.Series

classmethod from_list(values: str | Sequence[Any] | None, *, dtype: dtype | None = None, device: device | str | None = None, offset_dtype: dtype = torch.int64) → Self

Create tensor from a rectangular Python list of strings.

from sdm import StringTensor

tensor = StringTensor.from_list([
    ["foo", "bar"],
    ["hello world", ""],
    [None, "xzy"],
])

Parameters:

• values (str | Sequence[Any] | None) – The rectangular Python list of strings.

• dtype (dtype | None) – The dtype of the value tensor.

• device (device | str | None) – The device.

• offset_dtype (dtype) – The dtype of the offset tensor.

Return type:

Self