`ncore.sensors` Package#

Package exposing methods related to NCore’s sensor types

class ncore.sensors.BivariateWindshieldModel( windshield_distortion_parameters: BivariateWindshieldModelParameters, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, )#

Bases: ExternalDistortionModel[BivariateWindshieldModelParameters]

Implements an external distortion caused by a vehicle’s windshield. The model is only applicable for cameras where the whole area of interest is projected through the windshield.

The distortion is computed on spherical phi/theta angle-based representations of a sensor ray with direction=[x,y,z] such that phi = asin(x/(x^2+y+2+z^2)) and theta = asin(y/(x^2+y^2+z^2)).

Phi and theta are then deflected via distortion polynomials before the transformed ray is re-constructed with direction [sin(phi’), sin(theta’), 1-(x^2+y^2)] The distortion on phi and theta are computed via separate polynomials of order N in both phi and theta, e.g. phi’ = c0 + c1*phi + c2*phi^2 + (c3 + c4*phi)*theta + c5*theta^2

static compute_poly_order( poly_coeffs: Tensor, ) → int#: Computes the order of a bivariate polynomial give it’s array of coefficients

distort_camera_rays( camera_rays: Tensor, ) → Tensor#: Applies distortion to camera rays in forward direction, from external to internal

static distort_rays( camera_rays: Tensor, phi_poly: Tensor, theta_poly: Tensor, order_phi: int, order_theta: int, poly2d_eval_func, ) → Tensor#: Applies distortion to rays in forward direction, from external to internal

get_parameters() → BivariateWindshieldModelParameters#: Returns the parameters specific to the current windshield distortion model instance

horizontal_poly: Tensor#: Polynomial used for horizontal component of distortion in forward direction

horizontal_poly_inverse: Tensor#: Polynomial used for horizontal component of distortion in backward direction

order_phi: int#: Order of the distortion polynomial on phi

order_theta: int#: Order of the distortion polynomial on theta

static poly_eval_2d( coefficients: Tensor, x: Tensor, y: Tensor, order: int, ) → Tensor#: The bivariate polynomial, provided as a single-dimension tensor [c0, c1, c2…cn] is evaluated as: c0*x^0 +c1*x^1 + c2*x^2 + (c3*x^0 + c4*x^1)y^1 + (c5*x^0)y^2 In essence, each coefficient to y is a polynomial evaluation of increasing degree.

reference_poly: ReferencePolynomial#: Reference polynomial used for the distortion model

undistort_camera_rays( camera_rays: Tensor, ) → Tensor#: Applies distortion to camera rays in backward direction, from external to internal

vertical_poly: Tensor#: Polynomial used for vertical component of distortion in forward direction

vertical_poly_inverse: Tensor#: Polynomial used for vertical component of distortion in backward direction

class ncore.sensors.CameraModel( camera_model_parameters: CameraModelParameters, device: str | device, dtype: dtype, )#

Bases: BaseModel, ABC, Generic[CameraModelParametersT_co]

Base class for all camera models

Generic in the concrete camera model parameter type the model round-trips through get_parameters(), so that a CameraModel remains statically useful without having to narrow it to a concrete model type. Plain (unparameterized) CameraModel annotations remain valid and behave as before.

class CameraRaysReturn( camera_rays: Tensor, valid_flag: Tensor, )#

Bases: object

Contains

camera rays [float] (n,3), undefined (model-specific, possibly non-finite) for invalid rays
valid_flag [bool] (n,), whether the model has a well-defined ray for each image point within its calibrated field of view (independent of the image bounds)

classmethod all_valid( camera_rays: Tensor, ) → CameraRaysReturn#: Camera rays that are all valid (e.g., precomputed rays known to be within the calibrated field of view): a valid flag of True for each ray (on the rays’ device)

camera_rays: Tensor#

valid_flag: Tensor#

class ImagePointsReturn( image_points: Tensor, valid_flag: Tensor, jacobians: Tensor | None = None, )#

Bases: object

Contains

image point coordinates [float] (n,2)
valid_flag [bool] (n,)
[optional] Jacobians of the projection [float] (n,2,3)

image_points: Tensor#

jacobians: Tensor | None = None#

valid_flag: Tensor#

class PixelsReturn( pixels: Tensor, valid_flag: Tensor, )#

Bases: object

Contains

pixel indices [int] (n,2)
valid_flag [bool] (n,)

pixels: Tensor#

valid_flag: Tensor#

class WorldPointsToImagePointsReturn( image_points: Tensor, T_world_sensors: Tensor | None = None, valid_indices: Tensor | None = None, timestamps_us: Tensor | None = None, )#

Bases: object

Contains

image point coordinates of the valid projections [float] (n,2)
[optional] world-to-sensor poses of valid projections [float] (n,4,4)
[optional] indices of the valid projections relative to the input points [int] (n,)
[optional] timestamps of the valid projections [int] (n,)

T_world_sensors: Tensor | None = None#

image_points: Tensor#

timestamps_us: Tensor | None = None#

valid_indices: Tensor | None = None#

class WorldPointsToPixelsReturn( pixels: Tensor, T_world_sensors: Tensor | None = None, valid_indices: Tensor | None = None, timestamps_us: Tensor | None = None, )#

Bases: object

Contains

pixel indices of the valid projections [int] (n,2)
[optional] world-to-sensor poses of valid projections [float] (n,4,4)
[optional] indices of the valid projections relative to the input points [int] (n,)
[optional] timestamps of the valid projections [int] (n,)

T_world_sensors: Tensor | None = None#

pixels: Tensor#

timestamps_us: Tensor | None = None#

valid_indices: Tensor | None = None#

class WorldRaysReturn( world_rays: Tensor, valid_flag: Tensor, T_sensor_worlds: Tensor | None = None, timestamps_us: Tensor | None = None, )#

Bases: object

Contains

rays [point, direction] in the world coordinate frame, represented by 3d start of ray points and 3d ray directions [float] (n,6)
valid_flag [bool] (n,), the valid flag of the corresponding camera rays
[optional] sensor-to-worlds poses of the returned rays [float] (n,4,4)
[optional] timestamps of the returned rays [int] (n,)

T_sensor_worlds: Tensor | None = None#

timestamps_us: Tensor | None = None#

valid_flag: Tensor#

world_rays: Tensor#

camera_rays_to_image_points( cam_rays: Tensor | ndarray | CameraRaysReturn, return_jacobians: bool = False, ) → ImagePointsReturn#

For each camera ray, computes the corresponding image point coordinates and a valid flag. Optionally, the Jacobians of the per-ray transformations can be computed as well

If the camera rays are given as CameraRaysReturn, their valid flag is combined with the validity of the projection, i.e., invalid camera rays result in invalid image points.

camera_rays_to_pixels( cam_rays: Tensor | ndarray | CameraRaysReturn, ) → PixelsReturn#: For each camera ray, computes the corresponding pixel index and a valid flag (combined with the valid flag of camera rays given as CameraRaysReturn, see camera_rays_to_image_points)

external_distortion: ExternalDistortionModel | None#: Source of distortion external to the camera (e.g. windshield). Can be empty (None) if no such source exists.

static from_parameters( cam_model_parameters: CameraModelParameters, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, ) → CameraModel#: Initialize a generic camera model class from camera model parameters

Deprecated since version Prefer: the module-level camera_model_from_parameters(), which dispatches on the parameter type (and can be extended for out-of-tree models via register_camera_model()) instead of being a closed dispatch table inherited by every camera model.

abstractmethod get_parameters() → CameraModelParametersT_co#: Returns the camera model parameters specific to the current camera model instance

image_points_relative_frame_times( image_points: Tensor | ndarray, ) → Tensor#: Convenience wrapper for image_points_relative_frame_times_kernel with the camera’s resolution and shutter type + tensor conversion

static image_points_relative_frame_times_kernel( image_points: Tensor, resolution: Tensor, shutter_type: ShutterType, ) → Tensor#: Get relative frame-times based on the image point coordinates and rolling shutter type

image_points_to_camera_rays( image_points: Tensor | ndarray, ) → CameraRaysReturn#

Computes camera rays for each image point, and a valid flag

A camera ray is valid if the model has a well-defined ray for the image point within its calibrated field of view. Validity is independent of the image bounds: image points outside of the image can be valid, and image points inside of the image can be invalid (e.g., for a calibrated field of view smaller than the image). Invalid camera rays are model-specific and undefined (they may be extrapolated or not even finite) and need to be masked with the valid flag before use.

image_points_to_pixels( image_points: Tensor | ndarray, ) → Tensor#: Given continuous image point coordinates, computes the corresponding pixel indices.

image_points_to_world_rays_mean_pose( image_points: Tensor | ndarray, T_sensor_world_start: Tensor | ndarray, T_sensor_world_end: Tensor | ndarray, start_timestamp_us: int | None = None, end_timestamp_us: int | None = None, camera_rays: CameraRaysReturn | None = None, return_T_sensor_worlds: bool = False, return_timestamps: bool = False, ) → WorldRaysReturn#

Unprojects image points to world rays using the mean pose of the sensor between the start and end poses (not compensating for potential sensor-motion).

Can optionally re-use known camera rays associated with image points.

For each image point returns 3d world rays [point, direction], represented by 3d start of ray points and 3d ray directions in the world frame

image_points_to_world_rays_shutter_pose( image_points: Tensor | ndarray, T_sensor_world_start: Tensor | ndarray, T_sensor_world_end: Tensor | ndarray, start_timestamp_us: int | None = None, end_timestamp_us: int | None = None, camera_rays: CameraRaysReturn | None = None, return_T_sensor_worlds: bool = False, return_timestamps: bool = False, ) → WorldRaysReturn#

Unprojects image points to world rays using rolling-shutter compensation of sensor motion.

Can optionally re-use known camera rays associated with image points.

For each image point returns 3d world rays [point, direction], represented by 3d start of ray points and 3d ray directions in the world frame

image_points_to_world_rays_static_pose( image_points: Tensor | ndarray, T_sensor_world: Tensor | ndarray, timestamp_us: int | None = None, camera_rays: CameraRaysReturn | None = None, return_T_sensor_worlds: bool = False, return_timestamps: bool = False, ) → WorldRaysReturn#

Unprojects image points to world rays using a using a fixed sensor pose (not compensating for potential sensor-motion).

Can optionally re-use known camera rays associated with image points.

For each image point returns 3d world rays [point, direction], represented by 3d start of ray points and 3d ray directions in the world frame

pixels_to_camera_rays( pixel_idxs: Tensor | ndarray, ) → CameraRaysReturn#: For each pixel index computes its corresponding camera ray and a valid flag (see image_points_to_camera_rays)

pixels_to_image_points( pixel_idxs: Tensor | ndarray, ) → Tensor#: Given integer-based pixels indices, computes corresponding continuous image point coordinates representing the center of each pixel.

pixels_to_world_rays_mean_pose( pixel_idxs: Tensor | ndarray, T_sensor_world_start: Tensor | ndarray, T_sensor_world_end: Tensor | ndarray, start_timestamp_us: int | None = None, end_timestamp_us: int | None = None, camera_rays: CameraRaysReturn | None = None, return_T_sensor_worlds: bool = False, return_timestamps: bool = False, ) → WorldRaysReturn#

pixels_to_world_rays_shutter_pose( pixel_idxs: Tensor | ndarray, T_sensor_world_start: Tensor | ndarray, T_sensor_world_end: Tensor | ndarray, start_timestamp_us: int | None = None, end_timestamp_us: int | None = None, camera_rays: CameraRaysReturn | None = None, return_T_sensor_worlds: bool = False, return_timestamps: bool = False, ) → WorldRaysReturn#

pixels_to_world_rays_static_pose( pixel_idxs: Tensor | ndarray, T_sensor_world: Tensor | ndarray, timestamp_us: int | None = None, camera_rays: CameraRaysReturn | None = None, return_T_sensor_worlds: bool = False, return_timestamps: bool = False, ) → WorldRaysReturn#

resolution: Tensor#: Width and height of the image in pixels (int32, [2,])

shutter_type: ShutterType#: Shutter type of the camera’s imaging sensor

world_points_to_image_points_mean_pose( world_points: Tensor | ndarray, T_world_sensor_start: Tensor | ndarray, T_world_sensor_end: Tensor | ndarray, start_timestamp_us: int | None = None, end_timestamp_us: int | None = None, return_T_world_sensors: bool = False, return_valid_indices: bool = False, return_timestamps: bool = False, return_all_projections: bool = False, ) → WorldPointsToImagePointsReturn#: Projects world points to corresponding image point coordinates using the mean pose of the sensor between the start and end poses (not compensating for potential sensor-motion).

world_points_to_image_points_shutter_pose( world_points: Tensor | ndarray, T_world_sensor_start: Tensor | ndarray, T_world_sensor_end: Tensor | ndarray, start_timestamp_us: int | None = None, end_timestamp_us: int | None = None, max_iterations: int = 10, stop_t_delta: float = 0.0001, return_T_world_sensors: bool = False, return_valid_indices: bool = False, return_timestamps: bool = False, return_all_projections: bool = False, ) → WorldPointsToImagePointsReturn#: Projects world points to corresponding image point coordinates using rolling-shutter compensation of sensor motion

world_points_to_image_points_static_pose( world_points: Tensor | ndarray, T_world_sensor: Tensor | ndarray, timestamp_us: int | None = None, return_T_world_sensors: bool = False, return_valid_indices: bool = False, return_timestamps: bool = False, return_all_projections: bool = False, ) → WorldPointsToImagePointsReturn#: Projects world points to corresponding image point coordinates using a fixed sensor pose (not compensating for potential sensor-motion).

world_points_to_pixels_mean_pose( world_points: Tensor | ndarray, T_world_sensor_start: Tensor | ndarray, T_world_sensor_end: Tensor | ndarray, start_timestamp_us: int | None = None, end_timestamp_us: int | None = None, return_T_world_sensors: bool = False, return_valid_indices: bool = False, return_timestamps: bool = False, return_all_projections: bool = False, ) → WorldPointsToPixelsReturn#: Projects world points to corresponding pixel indices using the mean pose of the sensor between the start and end poses (not compensating for potential sensor-motion).

world_points_to_pixels_shutter_pose( world_points: Tensor | ndarray, T_world_sensor_start: Tensor | ndarray, T_world_sensor_end: Tensor | ndarray, start_timestamp_us: int | None = None, end_timestamp_us: int | None = None, max_iterations: int = 10, stop_t_delta: float = 0.0001, return_T_world_sensors: bool = False, return_valid_indices: bool = False, return_timestamps: bool = False, return_all_projections: bool = False, ) → WorldPointsToPixelsReturn#: Projects world points to corresponding pixel indices using rolling-shutter compensation of sensor motion

world_points_to_pixels_static_pose( world_points: Tensor | ndarray, T_world_sensor: Tensor | ndarray, timestamp_us: int | None = None, return_T_world_sensors: bool = False, return_valid_indices: bool = False, return_timestamps: bool = False, return_all_projections: bool = False, ) → WorldPointsToPixelsReturn#: Projects world points to corresponding pixel indices using a fixed sensor pose (not compensating for potential sensor-motion).

class ncore.sensors.ExternalDistortionModel( device: str | device, dtype: dtype, )#

Bases: BaseModel, ABC, Generic[ExternalDistortionParametersT_co]

Base class for distortion effects from external causes to the camera

Generic in the concrete parameter type the model round-trips through get_parameters(), so that an ExternalDistortionModel remains statically useful without having to narrow it to a concrete model type. Plain (unparameterized) annotations remain valid and behave as before.

abstractmethod distort_camera_rays( camera_rays: Tensor, ) → Tensor#: Applies distortion to camera rays in forward direction, from external to internal

static from_parameters( external_distortion_parameters: ExternalDistortionParameters, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, ) → ExternalDistortionModel#: Initialize a generic external distortion model from parameters

Deprecated since version Prefer: the module-level external_distortion_model_from_parameters(), which dispatches on the parameter type (and can be extended for out-of-tree models via register_external_distortion_model()) instead of being a closed dispatch table inherited by every external distortion model.

abstractmethod get_parameters() → ExternalDistortionParametersT_co#: Returns the parameters specific to the concrete distortion model

abstractmethod undistort_camera_rays( camera_rays: Tensor, ) → Tensor#: Applies distortion to camera rays in backward direction, from internal to external

class ncore.sensors.FThetaCameraModel( camera_model_parameters: FThetaCameraModelParameters, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, newton_iterations: int = 3, min_2d_norm: float = 1e-06, )#

Bases: CameraModel[FThetaCameraModelParameters]

Camera model for F-Theta lenses

A: Tensor#

Ainv: Tensor#

bw_poly: Tensor#

dbw_poly: Tensor#

dfw_poly: Tensor#

static eval_tangential_distortion( xy: Tensor, coeffs: Tensor, ) → Tensor#

Evaluates the tangential distortion offsets (in pixels)

For normalized coordinates xy = [x, y] (x = X/Z, y = Y/Z, shape [N,2]) and coefficients coeffs = [t0, t1] (shape [2,]) returns the offsets [N,2] given by T(x, y) = (2*t0*x*y + t1*(r2 + 2*x^2), 2*t1*x*y + t0*(r2 + 2*y^2)) with r2 = x^2 + y^2

fw_poly: Tensor#

get_parameters() → FThetaCameraModelParameters#: Returns the camera model parameters specific to the current camera model instance

max_angle: float#

min_2d_norm: Tensor#

newton_iterations: int#

principal_point: Tensor#

reference_poly: PolynomialType#

tangential_coeffs: Tensor#

tangential_iterations: int#

class ncore.sensors.IdealPinholeCameraModel( camera_model_parameters: IdealPinholeCameraModelParameters, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, )#

Bases: PinholeCameraModel[IdealPinholeCameraModelParameters]

Camera model for an ideal (distortion-free) pinhole camera

get_parameters() → IdealPinholeCameraModelParameters#: Returns the camera model parameters specific to the current camera model instance

class ncore.sensors.LidarModel(device: str | device, dtype: dtype)#

Bases: BaseModel, ABC, Generic[LidarModelParametersT_co]

Base class for all lidar models

Generic in the concrete lidar model parameter type the model round-trips through get_parameters(), so that a LidarModel remains statically useful without having to narrow it to a concrete model type. Plain (unparameterized) LidarModel annotations remain valid and behave as before.

class SensorAnglesReturn( sensor_angles: Tensor, valid_flag: Tensor, )#

Bases: object

Contains

sensor angles [float] (n,2)
valid_flag [bool] (n,)

sensor_angles: Tensor#

valid_flag: Tensor#

class SensorPointsReturn( sensor_points: Tensor, valid_flag: Tensor, )#

Bases: object

Contains

sensor points [float] (n,3)
valid_flag [bool] (n,), the valid flag of the corresponding sensor rays

sensor_points: Tensor#

valid_flag: Tensor#

class SensorRaysReturn( sensor_rays: Tensor, valid_flag: Tensor, )#

Bases: object

Contains

sensor rays [float] (n,3)
valid_flag [bool] (n,)

classmethod all_valid( sensor_rays: Tensor, ) → SensorRaysReturn#: Sensor rays that are all valid (e.g., rays of the elements of a structured lidar model): a valid flag of True for each ray (on the rays’ device)

sensor_rays: Tensor#

valid_flag: Tensor#

class WorldPointsToSensorAnglesReturn( sensor_angles: Tensor, T_world_sensors: Tensor | None = None, valid_indices: Tensor | None = None, timestamps_us: Tensor | None = None, )#

Bases: object

Contains

sensor angles of the valid projections [float] (n,2)
[optional] world-to-sensor poses of valid projections [float] (n,4,4)
[optional] indices of the valid projections relative to the input points [int] (n,)
[optional] timestamps of the valid projections [int] (n,)

T_world_sensors: Tensor | None = None#

sensor_angles: Tensor#

timestamps_us: Tensor | None = None#

valid_indices: Tensor | None = None#

class WorldRaysReturn( world_rays: Tensor, valid_flag: Tensor, T_sensor_worlds: Tensor | None = None, timestamps_us: Tensor | None = None, )#

Bases: object

Contains

rays [point, direction] in the world coordinate frame, represented by 3d start of ray points and 3d ray directions [float] (n,6)
valid_flag [bool] (n,), the valid flag of the corresponding sensor rays
[optional] sensor-to-worlds poses of the returned rays [float] (n,4,4)
[optional] timestamps of the returned rays [int] (n,)

T_sensor_worlds: Tensor | None = None#

timestamps_us: Tensor | None = None#

valid_flag: Tensor#

world_rays: Tensor#

abstractmethod get_parameters() → LidarModelParametersT_co#: Returns the lidar model parameters specific to the current lidar model instance

static maybe_from_parameters( lidar_model_parameters: LidarModelParameters | None, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, ) → LidarModel | None#: Initialize a generic lidar model from parameters, if available

Deprecated since version Prefer: the module-level maybe_lidar_model_from_parameters(), which dispatches on the parameter type (and can be extended for out-of-tree models via register_lidar_model()) instead of being a closed dispatch table inherited by every lidar model.

abstractmethod sensor_angles_to_sensor_rays( sensor_angles: Tensor | ndarray, ) → SensorRaysReturn#: Lidar model-specific implementation of elevation/azimuth angles to sensor rays

abstractmethod sensor_rays_to_sensor_angles( sensor_rays: Tensor | ndarray, normalized: bool = True, ) → SensorAnglesReturn#: Lidar model-specific implementation of sensor_rays_to_sensor_angles

class ncore.sensors.OpenCVFisheyeCameraModel( camera_model_parameters: OpenCVFisheyeCameraModelParameters, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, newton_iterations: int = 20, min_2d_norm: float = 1e-06, )#

Bases: CameraModel[OpenCVFisheyeCameraModelParameters]

Camera model for OpenCV fisheye cameras

approx_backward_poly: Tensor#

dforward_poly: Tensor#

focal_length: Tensor#

forward_poly: Tensor#

get_parameters() → OpenCVFisheyeCameraModelParameters#: Returns the camera model parameters specific to the current camera model instance

inverse_max_angle: float#

max_angle: float#

min_2d_norm: Tensor#

newton_iterations: int#

principal_point: Tensor#

class ncore.sensors.OpenCVPinholeCameraModel( camera_model_parameters: OpenCVPinholeCameraModelParameters, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, max_normalized_radius_squared: float | None = None, )#

Bases: PinholeCameraModel[OpenCVPinholeCameraModelParameters]

Camera model for OpenCV pinhole cameras

static compute_max_normalized_radius_squared( radial_coeffs: ndarray, ) → float#

Computes the largest squared radius of normalized coordinates up to which the radial distortion is valid

With s = r^2 the squared radius of the normalized coordinates (x/z, y/z) and d(s) = N(s) / D(s) the rational radial distortion factor (N(s) = 1 + k1 s + k2 s^2 + k3 s^3, D(s) = 1 + k4 s + k5 s^2 + k6 s^3), the distorted radius r d(s) is a one-to-one function of the radius r up to its first extremum (beyond it, distinct rays project to the same image points). The bound is the smallest positive root of either the derivative of the distorted radius (whose sign is the one of the polynomial N D + 2 s (N' D - N D')) or of D (a pole of the distortion factor), and math.inf if the distortion has neither.

Only the radial distortion is taken into account (tangential and thin-prism distortions are not).

Parameters:: radial_coeffs – the radial distortion coefficients [k1, k2, k3, k4, k5, k6]
Returns:: the exclusive bound on the squared radius of the normalized coordinates of valid camera rays

get_parameters() → OpenCVPinholeCameraModelParameters#: Returns the camera model parameters specific to the current camera model instance

max_normalized_radius_squared: float#: Exclusive bound on the squared radius of the normalized coordinates of valid camera rays

radial_coeffs: Tensor#

tangential_coeffs: Tensor#

thin_prism_coeffs: Tensor#

class ncore.sensors.PinholeCameraModel( camera_model_parameters: PinholeCameraModelParameters, device: str | device, dtype: dtype, )#

Bases: CameraModel[PinholeCameraModelParametersT_co], ABC

Abstract base for pinhole-family camera models

Holds the shared principal point and focal length and implements the closed-form (distortion-free) ideal pinhole projection / unprojection used directly by IdealPinholeCameraModel and as the linear core of OpenCVPinholeCameraModel.

focal_length: Tensor#

principal_point: Tensor#

class ncore.sensors.Rectificator( source: CameraModel, target: CameraModel, )#

Bases: object

Maps imagery from a source camera into a target camera’s image domain

Rectification is a generic “from-camera -> to-camera” intrinsic remap within a shared camera frame. The most common use is mapping a distorted source camera to a distortion-free (ideal pinhole) target, but the target may itself be distorted.

On construction a backward sample map (target -> source direction) is built once: for every target pixel, the corresponding ray is unprojected with the target model and re-projected with the source model, giving the source image coordinate to sample from. Resampling a source image into the target domain is then a single gather / interpolation (apply()). The raw map (sample_map, valid_mask) is exposed so callers can run their own (e.g. GPU or OpenCV) resampling.

apply( image: Tensor | ndarray, mode: Literal['bilinear', 'nearest', 'bicubic'] = 'bilinear', padding_mode: Literal['zeros', 'border', 'reflection'] = 'zeros', ) → Tensor#

Resample a source image into the target camera domain

Parameters:

image – source image, either [H_src, W_src], [H_src, W_src, C] or [N, H_src, W_src, C] (channels-last, as commonly stored). Integer images are converted to float for interpolation.
mode – interpolation mode passed to grid_sample, one of "bilinear", "nearest" or "bicubic".
padding_mode – out-of-bounds sampling mode passed to grid_sample, one of "zeros", "border" or "reflection".

Returns:

the rectified image in the target domain, channels-last, with the same number of leading/trailing dimensions as the (broadcast) input. Invalid target pixels are set to zero.

property sample_map: Tensor#

Source image coordinates to sample for each target pixel, shape [H, W, 2]

sample_map[v, u] is the continuous source image point (x, y) that target pixel (u, v) maps to. Invalid entries (see valid_mask) hold whatever the source projection produced and should not be relied upon.

source_points_to_target( points: Tensor | ndarray, ) → ImagePointsReturn#

Map continuous source image points into the target camera domain

Parameters:: points – source image points, shape [n, 2].
Returns:: target image points and a valid flag (see ImagePointsReturn), invalid for source points without a valid ray in the source camera.

target_points_to_source( points: Tensor | ndarray, ) → ImagePointsReturn#

Map continuous target image points back into the source camera domain

Parameters:: points – target image points, shape [n, 2].
Returns:: source image points and a valid flag (see ImagePointsReturn), invalid for target points without a valid ray in the target camera.

property valid_mask: Tensor#

Boolean mask of valid target pixels, shape [H, W]

A target pixel is valid when it has a valid ray within the target’s calibrated field of view, and this ray projects to a point inside the source image.

class ncore.sensors.RowOffsetStructuredSpinningLidarModel( parameters: RowOffsetStructuredSpinningLidarModelParameters, angles_to_columns_map_resolution_factor: int = 4, angles_to_columns_map_dtype: dtype = torch.int16, angles_to_columns_map_init: bool = False, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, fov_eps_factor: float = 4.0, )#

Bases: StructuredLidarModel[RowOffsetStructuredSpinningLidarModelParameters]

Represents a structured spinning lidar model that is using a per-row azimuth-offset (compatible with, e.g., Hesai P128 sensors)

angles_to_columns_map: Tensor | None#

angles_to_columns_map_dtype: dtype#

angles_to_columns_map_resolution_factor: int#

column_azimuths_rad: Tensor#

elements_to_sensor_angles( elements: Tensor | ndarray, ) → Tensor#: Retrieves the elevation and azimuth angles for elements in the structured lidar model. Elements are given as (row, column) indices.

elements_to_world_rays_shutter_pose( elements: Tensor | ndarray, T_sensor_world_start: Tensor | ndarray, T_sensor_world_end: Tensor | ndarray, start_timestamp_us: int | None = None, end_timestamp_us: int | None = None, sensor_rays: SensorRaysReturn | None = None, return_T_sensor_worlds: bool = False, return_timestamps: bool = False, ) → WorldRaysReturn#

Unprojects elements to world rays using rolling-shutter compensation of sensor motion.

Can optionally re-use known sensor rays associated with elements (as returned by elements_to_sensor_rays). The valid flag of the sensor rays is forwarded to the world rays.

For each element returns 3d world rays [point, direction], represented by 3d start of ray points and 3d ray directions in the world frame

fov_eps_rad: float#

fov_horiz: FOV#

fov_vert: FOV#

get_parameters() → RowOffsetStructuredSpinningLidarModelParameters#: Returns the lidar model parameters specific to the current lidar model instance

n_columns: int#

n_rows: int#

row_azimuth_offsets_rad: Tensor#

row_elevations_rad: Tensor#

sensor_angles_relative_frame_times( sensor_angles: Tensor | ndarray, ) → Tensor#

Get relative frame-times of sensor angle coordinates using internal angle to column mapping.

All sensor angles need to be in the FOV of the sensor.

sensor_angles_to_sensor_rays( sensor_angles: Tensor | ndarray, ) → SensorRaysReturn#: Computes the sensor rays for elevation/azimuth angles.

sensor_rays_to_sensor_angles( sensor_rays: Tensor | ndarray, normalized: bool = True, ) → SensorAnglesReturn#: Computes the elevation and azimuth angles for normalized 3d sensor rays.

spinning_direction: Literal['cw', 'ccw']#

spinning_frequency_hz: float#

world_points_to_sensor_angles_shutter_pose( world_points: Tensor | ndarray, T_world_sensor_start: Tensor | ndarray, T_world_sensor_end: Tensor | ndarray, start_timestamp_us: int | None = None, end_timestamp_us: int | None = None, max_iterations: int = 10, stop_t_delta: float = 0.0001, stop_t_delta_rate: float = 1e-06, return_T_world_sensors: bool = False, return_valid_indices: bool = False, return_timestamps: bool = False, return_all_projections: bool = False, ) → WorldPointsToSensorAnglesReturn#: Projects world points to corresponding sensor angle coordinates using rolling-shutter compensation of sensor motion

class ncore.sensors.StructuredLidarModel(device: str | device, dtype: dtype)#

Bases: LidarModel[StructuredLidarModelParametersT_co], ABC

abstractmethod elements_to_sensor_angles( elements: Tensor | ndarray, ) → Tensor#: Lidar model-specific implementation of elements_to_sensor_angles

elements_to_sensor_points( elements: Tensor | ndarray, element_distances: Tensor | ndarray, ) → SensorPointsReturn#: Computes 3d sensor points and a valid flag (of the sensor rays) for elements in the structured lidar model. Elements are given as (row, column) indices.

elements_to_sensor_rays( elements: Tensor | ndarray, ) → SensorRaysReturn#: Computes normalized 3d sensor ray directions and a valid flag for elements in the structured lidar model. Elements are given as (row, column) indices.

static maybe_from_parameters( lidar_model_parameters: LidarModelParameters | None, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, ) → StructuredLidarModel | None#: Initialize a structured lidar model from parameters, if available

Deprecated since version Prefer: the module-level maybe_lidar_model_from_parameters(). This shadowed the base’s static method with a narrowed return type, so which closed dispatch table ran depended only on the class name the caller happened to spell. The parameter type stays as wide as the base’s: narrowing it here would be a contravariance violation, so the structured-ness of the result is enforced at runtime instead.

ncore.sensors.camera_model_from_parameters( cam_model_parameters: FThetaCameraModelParameters, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, ) → FThetaCameraModel#

ncore.sensors.camera_model_from_parameters( cam_model_parameters: IdealPinholeCameraModelParameters, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, ) → IdealPinholeCameraModel

ncore.sensors.camera_model_from_parameters( cam_model_parameters: OpenCVPinholeCameraModelParameters, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, ) → OpenCVPinholeCameraModel

ncore.sensors.camera_model_from_parameters( cam_model_parameters: OpenCVFisheyeCameraModelParameters, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, ) → OpenCVFisheyeCameraModel

ncore.sensors.camera_model_from_parameters( cam_model_parameters: CameraModelParametersT, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, ) → CameraModel[CameraModelParametersT]

Initializes the camera model corresponding to the given camera model parameters

Dispatches on the runtime type of cam_model_parameters. Out-of-tree camera models can participate by registering their factory via register_camera_model().

Parameters:

cam_model_parameters – the camera model parameters to build the camera model from.
device – the device to instantiate the camera model on.
dtype – the floating point type to instantiate the camera model with.

Returns:

the camera model corresponding to cam_model_parameters.

Raises:

TypeError – if no camera model is registered for the given parameter type.

ncore.sensors.external_distortion_model_from_parameters( external_distortion_parameters: BivariateWindshieldModelParameters, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, ) → BivariateWindshieldModel#

ncore.sensors.external_distortion_model_from_parameters( external_distortion_parameters: ExternalDistortionParametersT, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, ) → ExternalDistortionModel[ExternalDistortionParametersT]

Initializes the external distortion model corresponding to the given parameters

Dispatches on the runtime type of external_distortion_parameters. Out-of-tree models can participate by registering their factory via register_external_distortion_model().

Parameters:

external_distortion_parameters – the parameters to build the distortion model from.
device – the device to instantiate the model on.
dtype – the floating point type to instantiate the model with.

Returns:

the external distortion model corresponding to external_distortion_parameters.

Raises:

TypeError – if no model is registered for the given parameter type.

ncore.sensors.lidar_model_from_parameters( lidar_model_parameters: RowOffsetStructuredSpinningLidarModelParameters, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, ) → RowOffsetStructuredSpinningLidarModel#

ncore.sensors.lidar_model_from_parameters( lidar_model_parameters: LidarModelParametersT, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, ) → LidarModel[LidarModelParametersT]

Initializes the lidar model corresponding to the given lidar model parameters

Dispatches on the runtime type of lidar_model_parameters. Out-of-tree lidar models can participate by registering their factory via register_lidar_model().

Parameters:

lidar_model_parameters – the lidar model parameters to build the lidar model from.
device – the device to instantiate the lidar model on.
dtype – the floating point type to instantiate the lidar model with.

Returns:

the lidar model corresponding to lidar_model_parameters.

Raises:

TypeError – if no lidar model is registered for the given parameter type.

ncore.sensors.maybe_lidar_model_from_parameters( lidar_model_parameters: LidarModelParameters | None, device: str | device = device(type='cuda'), dtype: dtype = torch.float32, ) → LidarModel | None#

Initializes the lidar model corresponding to the given parameters, if any

Convenience wrapper around lidar_model_from_parameters() for the common case of an optional lidar model.

Parameters:

lidar_model_parameters – the lidar model parameters to build from, or None.
device – the device to instantiate the lidar model on.
dtype – the floating point type to instantiate the lidar model with.

Returns:

the corresponding lidar model, or None if no parameters were given.

ncore.sensors.register_camera_model(cls, func=None)#

generic_func.register(cls, func) -> func

Registers a new implementation for the given cls on a generic_func.

ncore.sensors.register_external_distortion_model(cls, func=None)#

generic_func.register(cls, func) -> func

Registers a new implementation for the given cls on a generic_func.

ncore.sensors.register_lidar_model(cls, func=None)#

generic_func.register(cls, func) -> func

Registers a new implementation for the given cls on a generic_func.

ncore.sensors Package#

`ncore.sensors` Package#