RGB-D Cameras and Depth Sensors

An RGB image records color at image pixels. A depth image adds an estimate of distance for image directions, producing an RGB-D pair when the color and depth streams can be aligned. Camera intrinsics relate pixels to rays, and extrinsic calibration describes how separate color and depth sensors sit relative to one another. With a valid depth value, a pixel can be projected into a 3D point. Without calibration or valid depth, painting a point cloud with color may place texture on the wrong geometry. Depth cameras do not all measure in the same way. A stereo system estimates disparity between two views; given its baseline and calibration, disparity can be converted to depth. The RealSense D435 is documented as a stereo depth camera with a separate RGB camera. Structured-light systems project a known pattern and infer shape from its distortion. Time-of-flight systems estimate distance from the travel or phase of emitted light. These mechanisms have different failure modes, so the label RGB-D says what data are available rather than naming one particular sensor technology. Shiny, transparent, dark or repetitive surfaces can produce missing or unreliable depth depending on the sensor. Occlusion means a background point visible to one imager may be hidden from another. Fast motion can misalign streams captured at different times. Calibration errors and depth noise grow into 3D errors, particularly when reconstructing small details. A zero or invalid depth value should not silently be treated as a real point at the camera origin. KinectFusion showed how a moving commodity depth camera could support real-time indoor surface mapping and tracking. That result depends on estimating camera motion and fusing many noisy frames, not on a single perfect depth picture. Before using an RGB-D map to grasp, measure or navigate, inspect invalid-pixel coverage, stream synchronization, registration and independent dimensions under the intended lighting and material conditions.