Semantic Segmentation

To run the example, please install Open3D with pip install open3d-python.

cd /path/to/MinkowskiEngine
python -m examples.indoor

Segmentation of a hotel room

When you run the example, you will see a hotel room and semantic segmentation of the room. You can interactively rotate the visualization when you run the example. First, we load the data.

def load_file(file_name):
    pcd = o3d.read_point_cloud(file_name)
    coords = np.array(pcd.points)
    colors = np.array(pcd.colors)
    return coords, colors, pcd

You can provide a quantized coordinates that ensures there would be only one point per voxel, or you can use the new MinkowskiEngine.TensorField that does not require quantized coordinates to process point clouds. However, since it does the quanization in the main training process instead of delegating the quantization to the data loading processes, it could slow down the training. Next, you should create a batched coordinates by calling MinkowskiEngine.utils.batched_coordinates.

# Create a batch, this process is done in a data loader during training in parallel.
in_field = ME.TensorField(
    features=torch.from_numpy(colors).float(),
    coordinates=ME.utils.batched_coordinates([coords / voxel_size], dtype=torch.float32),
    quantization_mode=ME.SparseTensorQuantizationMode.UNWEIGHTED_AVERAGE,
    minkowski_algorithm=ME.MinkowskiAlgorithm.SPEED_OPTIMIZED,
    device=device,
)

Finally, we feed-forward the sparse tensor into the network and get the predictions.

# Convert to a sparse tensor
sinput = in_field.sparse()
# Output sparse tensor
soutput = model(sinput)
# get the prediction on the input tensor field
out_field = soutput.slice(in_field)

After doing some post-processing. We can color the labels and visualize the input and the prediction side-by-side.

The weights are downloaded automatically once you run the example and the weights are currently the top-ranking algorithm on the Scannet 3D segmentation benchmark.

Please refer to the complete indoor segmentation example for more detail.