VarLenTensor

class sdm.tensor.VarLenTensor(data: Tensor, offset: Tensor, valid: Tensor | None, size: Sequence[int], stride: Sequence[int] | None = None, storage_offset: int = 0)

Bases: Tensor

A torch.Tensor for rectangular variable-length values.

Values are stored in a flat contiguous data tensor and indexed by an offset tensor.

import torch
from sdm import VarLenTensor

tensor = VarLenTensor(
    data=torch.tensor([1, 2, 3, 4, 5, 6]),
    offset=torch.tensor([0, 2, 5, 5, 6]),
    valid=None,
    size=(2, 2),
)

Parameters:

• data (Tensor) – Flat contiguous tensor containing all element values.

• offset (Tensor) – One-dimensional offsets into data.

• valid (Tensor | None) – One-dimensional mask indicating valid, non-null element values.

• size (Sequence[int]) – The shape of the tensor.

• stride (Sequence[int] | None) – The stride of the tensor.

• storage_offset (int) – The offset into the logical offset storage.

Return type:

Self

classmethod from_tensor(tensor: Tensor, *, offset_dtype: dtype = torch.int64) → Self

Wrap a dense tensor as fixed-size variable-length elements.

Parameters:

• tensor (Tensor) – The dense tensor.

• offset_dtype (dtype) – The dtype of the offset tensor.

Return type:

Self

classmethod from_arrow(array: Array | ChunkedArray, *, size: Sequence[int] | None = None, device: device | str | None = None) → Self

Create tensor from a list pyarrow.Array.

import pyarrow as pa
from sdm import VarLenTensor

array = pa.array([[1, 2], [3, 4, 5], [], None, [6]])
tensor = VarLenTensor.from_arrow(array)

Parameters:

• array (Array | ChunkedArray) – The list pyarrow.Array or pyarrow.ChunkedArray.

• size (Sequence[int] | None) – The shape of the tensor.

• device (device | str | None) – The device.

Return type:

Self

to_arrow() → Array

Convert this tensor to a flat pyarrow.Array.

Return type:

Array

classmethod from_list(values: Sequence[Any], *, dtype: dtype | None = None, device: device | str | None = None, offset_dtype: dtype = torch.int64) → Self

Create tensor from a rectangular Python list.

from sdm import VarLenTensor

tensor = VarLenTensor.from_list([
    [[1, 2], [3, 4, 5]],
    [[], [6]],
    [[7, 8], None],
])

Parameters:

• values (Sequence[Any]) – The rectangular Python list.

• dtype (dtype | None) – The dtype of the value tensor.

• device (device | str | None) – The device.

• offset_dtype (dtype) – The dtype of the offset tensor.

Return type:

Self

property data_offset: tuple[Tensor, Tensor]

Return contiguous data and normalized offsets.

Returns:

(data, offset) tuple.

property valid: Tensor | None

Return the logical validity mask.

Returns:

Boolean mask with shape self.size() indicating valid, non-null tensor elements, or None when all elements are valid.

property is_nullable: bool

Whether this tensor has a validity mask.