Sensor Fusion Explained: Techniques for Combining IMU, GNSS, Cameras, Radar, and LiDAR
Robots and autonomous systems rarely rely on one sensor: an IMU reacts quickly but drifts, GNSS provides a global position but can be interrupted, and cameras or LiDAR describe the surroundings in different ways. Sensor fusion combines these imperfect measurements into a more useful estimate of the system’s motion or environment. This guide explains the architecture, common estimator choices, and practical steps for building and validating a fusion pipeline.
What Is Sensor Fusion?
Sensor fusion is the process of combining measurements or interpretations from multiple sensors to estimate something the system cannot observe directly, such as a robot’s position, velocity, orientation, or the location of an object. A fusion algorithm uses both the measurements and a model of how the system behaves, along with information about measurement uncertainty.
Fusion does not mean averaging every sensor reading. A useful estimator accounts for what each sensor measures, when the measurement was taken, which coordinate frame it uses, and how noisy or biased it is. For example, a mobile robot can use wheel odometry and an IMU for continuous short-term motion estimates, then use camera or LiDAR observations to correct accumulated drift.
Why Systems Fuse Sensor Data
Each sensor has blind spots. Inertial sensors operate at high rates but accumulate error when their acceleration and angular velocity are integrated. GNSS provides an absolute position outdoors but may be inaccurate near buildings or unavailable indoors. Cameras depend on visible texture and lighting; LiDAR can provide geometric depth but may struggle with reflective or sparse surfaces. Wheel encoders are responsive but cannot distinguish actual movement from wheel slip.
Fusion can improve continuity and reduce uncertainty when sensors provide complementary information. It cannot make bad measurements good by itself: a time offset, incorrect coordinate transform, biased sensor, or overconfident uncertainty setting can make the combined estimate worse than an individual input.
How Sensor Fusion Works: Architecture
A common state-estimation pipeline is:
Acquire and timestamp measurements -> calibrate and transform frames -> validate and model uncertainty -> predict the system state -> incorporate available measurements -> publish an estimate and diagnostics.
- Acquire measurements. Drivers produce readings at different rates and in different message formats. Preserve the measurement timestamp rather than treating message arrival time as the time of observation.
- Calibrate and transform. Correct sensor-specific bias or scale errors, and express data in consistent frames. A rigid transform describes where a sensor is mounted relative to the robot; it does not correct an incorrect timestamp.
- Predict. A motion model propagates the previous state forward. An IMU or wheel odometry can provide frequent motion information between slower camera or GNSS updates.
- Update. When a measurement arrives, the estimator compares it with the predicted observation. In a Kalman-style filter this difference is the innovation; the filter weights the correction using predicted and measured uncertainty.
- Publish and monitor. Consumers use the estimated state, while diagnostics track timing, covariance, rejected measurements, and innovation size. A confidence value is part of the output, not an optional decoration.
The robot’s coordinate frames and transforms are part of the estimator’s data contract. ROS conventions are described in REP 103 for units and coordinate conventions and REP 105 for common mobile-robot frames. A typical tree distinguishes a locally continuous odom frame from a globally corrected map frame, with base_link attached to the robot body.
Sensors can be combined at different points in this pipeline:
| Fusion level | What is combined | Typical example | Main trade-off |
|---|---|---|---|
| Raw measurement | Calibrated sensor samples | Accelerometer and gyroscope data for attitude estimation | Retains detail but needs careful modeling and synchronization |
| Feature or state | Sensor-derived features or estimates with uncertainty | Camera features and IMU motion in visual-inertial odometry | Reduces data volume, but depends on each sensor’s front end |
| Decision | Detections or classifications | Matching camera and radar object tracks | Modular and practical, but discards some measurement detail |
Estimator Choices
| Method | Useful when | Limitations to understand |
|---|---|---|
| Complementary filter | Combining fast gyro response with slower accelerometer or compass corrections for orientation | A compact design for a limited state; it is not a general-purpose probabilistic estimator |
| Kalman filter (KF) | The state and measurement models are linear and noise is reasonably modeled as Gaussian | Linear-model assumptions do not fit many robot motion and sensor models |
| Extended Kalman filter (EKF) | A practical approximation is needed for nonlinear motion or measurement models | Linearization can be inaccurate for strongly nonlinear behavior or poor initial estimates |
| Unscented Kalman filter (UKF) | Nonlinear models benefit from propagating representative sigma points | More computation; it still depends on a suitable model and uncertainty assumptions |
| Particle filter | The state estimate can be multi-modal or strongly non-Gaussian, such as global localization | Computation grows with the number of particles and state complexity |
| Factor graph / smoothing | Multiple observations and constraints, including loop closures, should be optimized together | More complex memory and optimization behavior than a simple streaming filter |
No method is universally most accurate. Select one based on the state being estimated, sensor models, compute budget, and how uncertainty or multiple possible hypotheses must be represented. For a mobile robot, a local EKF may estimate continuous motion while a separate SLAM system optimizes poses and maps over time.
Sensors, Components, and Variants
| Sensor | Typical contribution | Common weakness or failure mode |
|---|---|---|
| Accelerometer and gyroscope (IMU) | High-rate acceleration and angular-rate measurements | Bias, vibration, mounting errors, and drift after integration |
| Magnetometer | Heading reference relative to the magnetic field | Magnetic interference and local field distortion |
| GNSS receiver | Global position and sometimes velocity outdoors | Multipath, obstruction, variable quality, and outages |
| Wheel or joint encoders | Relative motion, rotation, or joint position | Slip, backlash, missed counts, and mechanical changes |
| Camera | Visual features, optical flow, depth, or object detections | Lighting, motion blur, occlusion, and texture dependence |
| LiDAR | Ranges and geometric structure for localization or mapping | Sparse or reflective returns, motion distortion, and environmental effects |
| Radar | Range and radial velocity, often robust in poor visibility | Lower angular or spatial detail than many camera and LiDAR systems |
Many practical systems combine an IMU with wheel odometry and add camera, LiDAR, radar, or GNSS data according to the environment and task. Distinguish sensor fusion for state estimation from SLAM: a filter can estimate motion from measurements, while a SLAM system also builds a map and may revise past poses when it detects a loop closure. See the guide to SLAM algorithms for robotics for that mapping and optimization workflow.
Real-World Use Cases
- Mobile robots: Wheel odometry and an IMU provide motion between LiDAR or camera localization updates. A global map estimate can correct long-term drift without making the local motion frame jump.
- Drones: IMU data supports fast attitude control; cameras, barometers, GNSS, or range sensors add position and altitude references appropriate to indoor or outdoor flight.
- Driver-assistance systems: Camera, radar, and sometimes LiDAR contribute complementary object and motion observations. Track association and uncertainty matter because each sensor observes different properties.
- Industrial manipulators: Joint encoders estimate configuration while force/torque or vision sensors support contact detection, grasping, and inspection.
- Phones and wearable devices: Gyroscopes and accelerometers provide responsive motion tracking, while GNSS or other references help limit accumulated position drift.
The right combination follows the environment and required behavior. A sensor that is useful in an open outdoor space may add little in a dark, reflective, or signal-obstructed environment.
Practical Considerations and a ROS 2 Example
Design and validation checklist
- Define the state and operating envelope. Decide whether the system needs orientation, local pose, global position, object tracks, or a map, and specify accuracy, update rate, latency, and expected outages.
- Choose sensors for complementary evidence. Identify what each sensor measures directly, what is derived, and which failures can affect multiple sensors at once.
- Calibrate and document frames. Estimate IMU bias and sensor extrinsics; verify axis directions, units, and frame IDs. Use a rigid mounting transform for each sensor and recalibrate after mounting changes.
- Align time. Use hardware synchronization when available. Otherwise retain acquisition timestamps and account for transport delay. For ROS 2 systems, the
message_filterssynchronization tutorials explain exact and approximate message synchronization. - Set realistic uncertainty. Use measured or documented covariance values. Do not report zero uncertainty for a noisy sensor, and do not fuse a derived estimate and its original source as if they were independent measurements.
- Inspect innovations and failure behavior. Record data, compare estimates against ground truth where available, and test sensor dropouts, delayed messages, motion, and environmental changes. Track error metrics such as position RMSE alongside latency and recovery time.
Example: Fusing wheel odometry and an IMU
The following is a ROS 2 parameter fragment for robot_localization. It selects the forward and yaw rates from wheel odometry and the IMU’s yaw rate for a planar robot. It is illustrative: the node name must match the launch file, topic names and frames must match the drivers, and the selected fields must actually be populated with valid covariance.
ekf_filter_node:
ros__parameters:
frequency: 30.0
sensor_timeout: 0.1
two_d_mode: true
odom_frame: odom
base_link_frame: base_link
world_frame: odom
odom0: /wheel/odometry
odom0_config: [false, false, false, false, false, false, true, false, false, false, false, true, false, false, false]
imu0: /imu/data
imu0_config: [false, false, false, false, false, false, false, false, false, false, false, true, false, false, false]The 15 configuration flags select x, y, z, roll, pitch, yaw, then their linear or angular rates, then accelerations. The flags are interpreted in the sensor message’s frame, so valid transforms and correct signs still matter. This local setup produces a continuous estimate in odom; it does not make that estimate globally drift-free. Review the robot_localization documentation for supported inputs, frame behavior, configuration, and launch details before adapting it.
GNSS needs an extra conversion step in this workflow: navsat_transform_node uses a GNSS fix together with the robot’s orientation and filtered odometry to produce a local nav_msgs/Odometry measurement. Feed that converted measurement to an appropriately configured filter rather than treating latitude and longitude as ordinary Cartesian x and y. Keep the local continuous estimate and globally corrected estimate in the intended frame arrangement, and avoid feeding an estimator’s own output back as an independent observation.
For the platform workflow around those estimators—including sensor selection, calibration, coordinate frames, timing, and validation—see the guide to robotics sensor integration. The ROS 2 architecture and setup guide covers the middleware layer, and a home lab hardware guide can help plan a repeatable test setup.
Common Sensor Fusion Misconceptions
- “More sensors always improve accuracy.” Redundant, correlated, miscalibrated, or delayed inputs can increase error. Add sensors only when their contribution and uncertainty are understood.
- “An EKF automatically removes drift.” It can combine measurements according to a model; it cannot correct an unobservable error without a suitable reference or constraint.
- “Matching update rates is enough to synchronize sensors.” Measurements must represent compatible observation times. Equal rates do not fix transport delay, clock offsets, or motion during exposure.
- “A high-rate IMU gives absolute position.” An IMU measures specific force and angular velocity. Integrating these measurements without external constraints causes error to grow.
- “GPS data can be fused as local Cartesian coordinates directly.” GNSS coordinates require a suitable geographic-to-local conversion and consistent frames.
- “One filter should handle every perception task.” State estimation, object tracking, and map optimization may need different models and algorithms; their outputs can be combined at a higher level.
