Vision

Vision provides dependency-light camera geometry helpers for RGB-D observations, Cartesian/pixel alignment-error helpers for caller-owned correction loops, and appliance-side fiducial tag detection (ArUco, AprilTag, QR).

Tag detection is topic-driven: the appliance subscribes to the camera inside ROS and streams back compact TagFrame results, so image bytes never cross the SDK boundary. A rigid TagLayout or a CharucoBoard can be solved as one pose per frame, and the Calibration solvers turn those poses into a hand-eye mounting or camera intrinsics. Supplying a CameraInfo topic and the printed tag size adds a per-tag pose in the camera optical frame. Corners are ordered top-left, top-right, bottom-right, bottom-left in the tag’s own orientation; the tag frame is centred on the tag with +x right, +y up and +z toward the camera.

Python

Camera geometry and position/pixel alignment-error helpers.

Types

class olo.vision.Pixel

Bases: object

Pixel coordinate, zero-based from the top-left of the image.

__init__(x, y)

Parameters:

• x (float)

• y (float)

Return type:

None

class olo.vision.BoundingBox

Bases: object

Half-open image bounding box with XYXY pixel coordinates.

__init__(minimum, maximum)

Parameters:

• minimum (Pixel)

• maximum (Pixel)

Return type:

None

property center: Pixel

Return the box centre pixel.

classmethod from_xywh(x, y, width, height)

Create a half-open bounding box from x, y, width, height.

Return type:

BoundingBox

Parameters:

• x (float)

• y (float)

• width (float)

• height (float)

classmethod from_xyxy(x_min, y_min, x_max, y_max)

Create a half-open bounding box from x_min, y_min, x_max, y_max.

Return type:

BoundingBox

Parameters:

• x_min (float)

• y_min (float)

• x_max (float)

• y_max (float)

property height: float

Return the box height in pixels.

to_xywh()

Return x, y, width, height.

Return type:

tuple[float, float, float, float]

to_xyxy()

Return x_min, y_min, x_max, y_max.

Return type:

tuple[float, float, float, float]

property width: float

Return the box width in pixels.

class olo.vision.CameraIntrinsics

Bases: object

Pinhole camera calibration parsed from sensor_msgs/CameraInfo.

__init__(width, height, fx, fy, cx, cy, frame_id='', stamp=None, distortion_model='', distortion=())

Parameters:

• width (int)

• height (int)

• fx (float)

• fy (float)

• cx (float)

• cy (float)

• frame_id (str)

• stamp (datetime | None)

• distortion_model (str)

• distortion (tuple[float, ...])

Return type:

None

class olo.vision.PointObservation

Bases: object

Stamped 3D point observation in the camera optical frame.

__init__(point, frame_id, pixel, depth_metres, stamp=None)

Parameters:

• point (Point)

• frame_id (str)

• pixel (Pixel)

• depth_metres (float)

• stamp (datetime | None)

Return type:

None

class olo.vision.AlignmentError

Bases: object

Image-space alignment error.

__init__(x_px, y_px, distance_px, converged)

Parameters:

• x_px (float)

• y_px (float)

• distance_px (float)

• converged (bool)

Return type:

None

class olo.vision.PositionError

Bases: object

Cartesian position error.

__init__(vector, distance, converged, frame_id='')

Parameters:

• vector (Vector)

• distance (float)

• converged (bool)

• frame_id (str)

Return type:

None

Camera geometry helpers

olo.vision.camera_info_to_intrinsics(message)

Parse a sensor_msgs/CameraInfo message into pinhole intrinsics.

Return type:

CameraIntrinsics

Parameters:

message (dict[str, Any])

olo.vision.sample_depth(image, pixel, *, window_size=1, min_depth_metres=None, max_depth_metres=None)

Sample a depth image at a pixel, optionally using a median window.

Return type:

float | None

Parameters:

• image (Image)

• pixel (Pixel)

• window_size (int)

• min_depth_metres (float | None)

• max_depth_metres (float | None)

olo.vision.sample_depth_region(image, region, *, aggregation='median', min_valid_pixels=1, min_depth_metres=None, max_depth_metres=None)

Sample a depth image region and aggregate valid depths in metres.

Return type:

float

Parameters:

• image (Image)

• region (BoundingBox | ndarray)

• aggregation (Literal['median', 'trimmed_mean'])

• min_valid_pixels (int)

• min_depth_metres (float | None)

• max_depth_metres (float | None)

olo.vision.sample_depth_point(depth_image, pixel, intrinsics, *, window_size=1, min_depth_metres=None, max_depth_metres=None)

Sample and deproject a depth point, preserving the depth image observation stamp.

Return type:

PointObservation | None

Parameters:

• depth_image (Image)

• pixel (Pixel)

• intrinsics (CameraIntrinsics)

• window_size (int)

• min_depth_metres (float | None)

• max_depth_metres (float | None)

olo.vision.deproject_pixel(pixel, depth_metres, intrinsics)

Deproject a pixel and depth into the ROS optical camera frame.

Return type:

Point

Parameters:

• pixel (Pixel)

• depth_metres (float)

• intrinsics (CameraIntrinsics)

olo.vision.deproject_point(pixel, depth_metres, intrinsics)

Deproject a pixel and scalar depth into the ROS optical camera frame.

Return type:

Point

Parameters:

• pixel (Pixel)

• depth_metres (float)

• intrinsics (CameraIntrinsics)

olo.vision.deproject(depth_image, pixel, intrinsics, *, window_size=1, min_depth_metres=None, max_depth_metres=None)

Sample and deproject a depth point, preserving frame metadata.

Return type:

PointObservation | None

Parameters:

• depth_image (Image)

• pixel (Pixel)

• intrinsics (CameraIntrinsics)

• window_size (int)

• min_depth_metres (float | None)

• max_depth_metres (float | None)

olo.vision.deproject_region(depth_image, region, intrinsics, *, aggregation='median', min_valid_pixels=1, min_depth_metres=None, max_depth_metres=None)

Deproject a depth region into an aggregated camera-frame observation.

Return type:

PointObservation

Parameters:

• depth_image (Image)

• region (BoundingBox | ndarray)

• intrinsics (CameraIntrinsics)

• aggregation (Literal['median', 'trimmed_mean'])

• min_valid_pixels (int)

• min_depth_metres (float | None)

• max_depth_metres (float | None)

olo.vision.deproject_to_frame(observation, *, frames, target_frame, timeout=5.0)

Transform a stamped point observation into a target frame.

Return type:

PointObservation

Parameters:

• observation (PointObservation)

• frames (FrameLookup)

• target_frame (str)

• timeout (float)

olo.vision.project_point(point, intrinsics)

Project a camera-frame point into pixel coordinates.

Return type:

Pixel

Parameters:

• point (Point)

• intrinsics (CameraIntrinsics)

olo.vision.project(point, intrinsics)

Project a camera-frame point into pixel coordinates.

Return type:

Pixel

Parameters:

• point (Point)

• intrinsics (CameraIntrinsics)

Control helpers

olo.vision.pixel_alignment_error(current, target, *, tolerance_px)

Return signed pixel error and convergence against a tolerance.

Return type:

AlignmentError

Parameters:

• current (Pixel)

• target (Pixel)

• tolerance_px (float)

olo.vision.position_error(current, target, *, tolerance_m)

Return Cartesian target - current error and convergence.

Return type:

PositionError

Parameters:

• current (Point | PointObservation)

• target (Point | PointObservation)

• tolerance_m (float)

Tag detection service

class olo.vision.Vision

Bases: VisionHandle

Sync wrapper for the Vision gRPC service (topics are taken as fully qualified).

__init__(session)

Bind to a channel session.

Return type:

None

property calibration: Calibration

Return the calibration solvers (hand-eye, intrinsics).

charuco(board, *, params=None)

Return a detector that solves a ChArUco board pose and reports its chessboard corners.

Return type:

TagDetector

Parameters:

• board (CharucoBoard)

• params (DetectorParams | None)

detect_tags(image, family, *, intrinsics=None, size_m=None, sizes=None, params=None, timeout=30.0)

Detect tags of one family in an image.

Return type:

TagFrame

Parameters:

• image (Image)

• family (str)

• intrinsics (CameraIntrinsics | None)

• size_m (float | None)

• sizes (Mapping[int, float] | None)

• params (DetectorParams | None)

• timeout (float)

handle(namespace=None)

Return a namespace-scoped handle; None resolves the appliance default robot.

Return type:

VisionHandle

Parameters:

namespace (str | None)

subscribe_tags(image_topic, family, *, camera_info_topic=None, size_m=None, sizes=None, params=None, timeout=None)

Stream detections of one family from a camera topic on this robot.

Return type:

TagStream

Parameters:

• image_topic (TopicInfo)

• family (str)

• camera_info_topic (TopicInfo | None)

• size_m (float | None)

• sizes (Mapping[int, float] | None)

• params (DetectorParams | None)

• timeout (float | None)

tags(family, *, size_m=None, sizes=None, params=None, target=None)

Return a detector for one tag family whose topics resolve against this robot.

Return type:

TagDetector

Parameters:

• family (str)

• size_m (float | None)

• sizes (Mapping[int, float] | None)

• params (DetectorParams | None)

• target (TagLayout | CharucoBoard | None)

class olo.vision.VisionHandle

Bases: object

Namespace-scoped Vision handle; relative topic names resolve against the robot.

__init__(vision, namespace)

Bind to a vision client and resolved robot namespace.

Parameters:

• vision (Vision)

• namespace (str)

Return type:

None

charuco(board, *, params=None)

Return a detector that solves a ChArUco board pose and reports its chessboard corners.

Return type:

TagDetector

Parameters:

• board (CharucoBoard)

• params (DetectorParams | None)

detect_tags(image, family, *, intrinsics=None, size_m=None, sizes=None, params=None, timeout=30.0)

Detect tags of one family in an image.

Return type:

TagFrame

Parameters:

• image (Image)

• family (str)

• intrinsics (CameraIntrinsics | None)

• size_m (float | None)

• sizes (Mapping[int, float] | None)

• params (DetectorParams | None)

• timeout (float)

subscribe_tags(image_topic, family, *, camera_info_topic=None, size_m=None, sizes=None, params=None, timeout=None)

Stream detections of one family from a camera topic on this robot.

Return type:

TagStream

Parameters:

• image_topic (TopicInfo)

• family (str)

• camera_info_topic (TopicInfo | None)

• size_m (float | None)

• sizes (Mapping[int, float] | None)

• params (DetectorParams | None)

• timeout (float | None)

tags(family, *, size_m=None, sizes=None, params=None, target=None)

Return a detector for one tag family whose topics resolve against this robot.

Return type:

TagDetector

Parameters:

• family (str)

• size_m (float | None)

• sizes (Mapping[int, float] | None)

• params (DetectorParams | None)

• target (TagLayout | CharucoBoard | None)

class olo.vision.TagDetector

Bases: object

A tag family plus physical sizes, bound to the appliance detector.

Holds configuration only; detection runs on the appliance. size_m is the default edge length and sizes overrides it per marker id. Without a size, tags are reported with corners but no pose. A target (a TagLayout or CharucoBoard) is additionally solved as one pose per frame, reported on TagFrame.target_transform.

__init__(vision, family, *, size_m=None, sizes=None, params=None, target=None, namespace=None)

Validate the family and capture sizes; nothing is sent until a call is made.

Parameters:

• vision (Vision)

• family (str)

• size_m (float | None)

• sizes (Mapping[int, float] | None)

• params (DetectorParams | None)

• target (TagLayout | CharucoBoard | None)

• namespace (str | None)

Return type:

None

detect(image, *, intrinsics=None, timeout=30.0)

Detect tags in an image the caller already holds.

Return type:

TagFrame

Parameters:

• image (Image)

• intrinsics (CameraIntrinsics | None)

• timeout (float)

subscribe(image_topic, camera_info_topic=None, *, timeout=None)

Stream detections from a camera topic; poses need camera_info_topic and a size.

Return type:

TagStream

Parameters:

• image_topic (TopicInfo)

• camera_info_topic (TopicInfo | None)

• timeout (float | None)

class olo.vision.TagStream

Bases: object

Context-managed sync iterator over a Vision.SubscribeTags server stream.

__init__(stream)

Wrap a SubscribeTags server stream.

Return type:

None

close()

Cancel the stream and mark it closed.

Return type:

None

Tag detection types

olo.vision.TagFamily

alias of Literal[‘aruco_4x4_50’, ‘aruco_4x4_100’, ‘aruco_4x4_250’, ‘aruco_4x4_1000’, ‘aruco_5x5_50’, ‘aruco_5x5_100’, ‘aruco_5x5_250’, ‘aruco_5x5_1000’, ‘aruco_6x6_50’, ‘aruco_6x6_100’, ‘aruco_6x6_250’, ‘aruco_6x6_1000’, ‘aruco_7x7_50’, ‘aruco_7x7_100’, ‘aruco_7x7_250’, ‘aruco_7x7_1000’, ‘aruco_original’, ‘apriltag_16h5’, ‘apriltag_25h9’, ‘apriltag_36h10’, ‘apriltag_36h11’, ‘qr_code’]

olo.vision.TAG_FAMILIES = ('aruco_4x4_50', 'aruco_4x4_100', 'aruco_4x4_250', 'aruco_4x4_1000', 'aruco_5x5_50', 'aruco_5x5_100', 'aruco_5x5_250', 'aruco_5x5_1000', 'aruco_6x6_50', 'aruco_6x6_100', 'aruco_6x6_250', 'aruco_6x6_1000', 'aruco_7x7_50', 'aruco_7x7_100', 'aruco_7x7_250', 'aruco_7x7_1000', 'aruco_original', 'apriltag_16h5', 'apriltag_25h9', 'apriltag_36h10', 'apriltag_36h11', 'qr_code')

Built-in immutable sequence.

If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable’s items.

If the argument is a tuple, the return value is the same object.

class olo.vision.TagFrame

Bases: object

Detections from one source image, stamped with that image’s header.

__init__(frame_id, stamp, width, height, tags=(), qr_codes=(), target_transform=None, target_reprojection_error_px=0.0, charuco_corners=())

Parameters:

• frame_id (str)

• stamp (datetime | None)

• width (int)

• height (int)

• tags (tuple[TagDetection, ...])

• qr_codes (tuple[QrCodeDetection, ...])

• target_transform (Transform | None)

• target_reprojection_error_px (float)

• charuco_corners (tuple[CharucoCorner, ...])

Return type:

None

charuco_corners: tuple[CharucoCorner, ...]

Interpolated chessboard corners; only for ChArUco targets.

property detections: tuple[TagDetection | QrCodeDetection, ...]

Return every detection regardless of kind.

target_to_frame(frames, target_frame, *, timeout=5.0)

Return the target pose expressed in target_frame using a TF lookup at the image stamp.

Return type:

Transform

Parameters:

• frames (FrameLookup)

• target_frame (str)

• timeout (float)

target_transform: Transform | None

Pose of the requested layout or board in the camera optical frame, when solved.

class olo.vision.TagDetection

Bases: _Detection

One decoded ArUco or AprilTag marker.

__init__(*, corners, transform=None, reprojection_error_px=0.0, family, id)

Parameters:

• corners (tuple[Pixel, Pixel, Pixel, Pixel])

• transform (Transform | None)

• reprojection_error_px (float)

• family (str)

• id (int)

Return type:

None

property box: BoundingBox

Return the axis-aligned box around the corners, ready for the depth helpers.

property center: Pixel

Return the mean of the four corners.

to_frame(frames, target_frame, *, timeout=5.0)

Return the tag pose expressed in target_frame using a TF lookup at the image stamp.

Return type:

Transform

Parameters:

• frames (FrameLookup)

• target_frame (str)

• timeout (float)

class olo.vision.QrCodeDetection

Bases: _Detection

One decoded QR code.

__init__(*, corners, transform=None, reprojection_error_px=0.0, data)

Parameters:

• corners (tuple[Pixel, Pixel, Pixel, Pixel])

• transform (Transform | None)

• reprojection_error_px (float)

• data (str)

Return type:

None

property box: BoundingBox

Return the axis-aligned box around the corners, ready for the depth helpers.

property center: Pixel

Return the mean of the four corners.

to_frame(frames, target_frame, *, timeout=5.0)

Return the tag pose expressed in target_frame using a TF lookup at the image stamp.

Return type:

Transform

Parameters:

• frames (FrameLookup)

• target_frame (str)

• timeout (float)

class olo.vision.DetectorParams

Bases: object

Optional detector tuning; None keeps the appliance default.

__init__(min_marker_perimeter_rate=None, max_marker_perimeter_rate=None, error_correction_rate=None)

Parameters:

• min_marker_perimeter_rate (float | None)

• max_marker_perimeter_rate (float | None)

• error_correction_rate (float | None)

Return type:

None

to_proto()

Encode as the wire message, leaving unset fields at zero.

Return type:

DetectorParams

Targets (layouts and boards)

class olo.vision.TagLayout

Bases: object

Rigid arrangement of markers solved as one pose.

markers maps a marker id to its four corners (TL, TR, BR, BL) in the layout’s body frame, in metres. A grid board, a cube or a docking jig are all layouts; use grid_layout() for the common planar grid.

__init__(markers, frame_name='tag_layout')

Parameters:

• markers (Mapping[int, tuple[Point, Point, Point, Point]])

• frame_name (str)

Return type:

None

to_proto()

Encode as the wire message.

Return type:

TagLayout

olo.vision.grid_layout(columns, rows, marker_m, gap_m, ids, *, frame_name='tag_layout')

Planar grid of markers, row-major from the top-left, centred on the board.

The layout frame has X right, Y up and Z toward the viewer, matching the per-tag convention, so a single-marker grid coincides with that tag’s frame.

Return type:

TagLayout

Parameters:

• columns (int)

• rows (int)

• marker_m (float)

• gap_m (float)

• ids (Sequence[int])

• frame_name (str)

class olo.vision.CharucoBoard

Bases: object

Chessboard with markers in the white squares; its interpolated corners give precise poses.

__init__(squares, square_m, marker_m, family, frame_name='charuco_board')

Parameters:

• squares (tuple[int, int])

• square_m (float)

• marker_m (float)

• family (str)

• frame_name (str)

Return type:

None

to_proto()

Encode as the wire message (the family travels on the request).

Return type:

CharucoBoard

class olo.vision.CharucoCorner

Bases: object

One interpolated chessboard corner of a ChArUco board.

__init__(id, pixel)

Parameters:

• id (int)

• pixel (Pixel)

Return type:

None

Calibration

class olo.vision.Calibration

Bases: object

Thin wrappers over the Vision calibration RPCs.

__init__(vision)

Bind to a vision client.

Parameters:

vision (Vision)

Return type:

None

calibrate_intrinsics(frames, board, *, timeout=30.0)

Calibrate pinhole intrinsics from ChArUco frames captured across varied views.

Return type:

IntrinsicsResult

Parameters:

• frames (Sequence[TagFrame])

• board (CharucoBoard)

• timeout (float)

solve_hand_eye(samples, *, mounting='eye_in_hand', timeout=30.0)

Solve the camera mounting with Tsai-Lenz from at least three varied samples.

Eye-in-hand (camera on the arm, target fixed) returns link -> camera; eye-to-hand (camera fixed, target on the arm) returns base -> camera. Both are ready for frames.publish_static once given frame ids.

Return type:

HandEyeResult

Parameters:

• samples (Sequence[HandEyeSample])

• mounting (Literal['eye_in_hand', 'eye_to_hand'])

• timeout (float)

class olo.vision.HandEyeSample

Bases: object

One synchronised observation: the arm link pose and the target pose at the image stamp.

__init__(base_to_link, camera_to_target)

Parameters:

• base_to_link (Transform)

• camera_to_target (Transform)

Return type:

None

classmethod from_frame(frame, frames, base_frame, link, *, timeout=5.0)

Pair a frame’s target pose with the link pose looked up at that frame’s stamp.

Looking TF up at the image stamp, not at call time, is what makes the pair valid; dwell before capturing so both are still.

Return type:

HandEyeSample

Parameters:

• frame (TagFrame)

• frames (FrameLookup)

• base_frame (str)

• link (str)

• timeout (float)

class olo.vision.HandEyeResult

Bases: object

Solved camera mounting plus per-sample consistency residuals.

__init__(transform, residual_translation_m, residual_rotation_rad)

Parameters:

• transform (Transform)

• residual_translation_m (tuple[float, ...])

• residual_rotation_rad (tuple[float, ...])

Return type:

None

property max_residual_rotation_rad: float

Largest per-sample orientation disagreement.

property max_residual_translation_m: float

Largest per-sample position disagreement; a quick quality gate.

transform: Transform

Eye-in-hand: link -> camera. Eye-to-hand: base -> camera.

class olo.vision.IntrinsicsResult

Bases: object

Calibrated pinhole intrinsics with reprojection statistics.

__init__(intrinsics, rms_reprojection_error_px, per_view_error_px)

Parameters:

• intrinsics (CameraIntrinsics)

• rms_reprojection_error_px (float)

• per_view_error_px (tuple[float, ...])

Return type:

None

TypeScript

Import from olo/vision:

Types

• Pixel Zero-based image pixel coordinate.

• BoundingBox Half-open image bounding box with XYXY pixel coordinates.

• CameraIntrinsics Pinhole camera calibration parsed from sensor_msgs/CameraInfo.

• PointObservation Stamped 3D point observation in an image-derived frame.

• AlignmentError Image-space alignment error.

• PositionError Cartesian position error.

Options

• SampleDepthOptions

• SampleDepthRegionOptions

Camera geometry helpers

• cameraInfoToIntrinsics

• sampleDepth

• sampleDepthRegion

• sampleDepthPoint

• deprojectPixel

• deprojectPoint

• deproject

• deprojectRegion

• deprojectToFrame

• projectPoint

• project

Control helpers

• pixelAlignmentError

• positionError

Tag detection service

• Vision Async wrapper for the Vision gRPC service.

• VisionHandle Namespace-scoped handle; relative topics resolve against the robot.

• TagDetector One tag family plus physical sizes, bound to the appliance detector.

• TagStream Async iterator over streamed tag frames.

Tag detection types

• TagFamily

• TagFrame

• TagDetection

• QrCodeDetection

• DetectorParams

• TagDetectorOptions

• DetectTagsOptions

• SubscribeTagsOptions

Tag helpers

• detectionCenter

• detectionBox

• tagToFrame

• targetToFrame

Targets (layouts and boards)

• TagLayout

• gridLayout

• CharucoBoard

• CharucoCorner

Calibration

• Calibration Tsai-Lenz hand-eye and ChArUco intrinsics solvers.

• handEyeSampleFromFrame

• HandEyeSample

• HandEyeResult

• IntrinsicsResult

• SolveHandEyeOptions