CAMERA-RADAR 3D MULTI OBJECT TRACKING ROBUST TO SENSOR-FAILURES

Abstract

As autonomous vehicles navigate dense urban environments, three-dimensional detection and tracking of objects is a critical step to ensure passenger safety, making it a cornerstone of autonomous navigation. Furthermore, a robust tracking pipeline is of the utmost importance to ensure that the vehicle maintains awareness of its surroundings, even under high sensor data degradation. Early research focused on vision-based two dimensional (2D) object detection, but extending this to three-dimensional (3D) detection proved challenging due to difficulties in depth estimation with monocular cameras. To address this, camera data is now fused with range-measuring point cloud systems, enabling detection and tracking methods to achieve significantly higher accuracies. Given concerns over affordability and robustness, radars are particularly well-suited due to their availability and resilience to adverse weather conditions. However, previous methods have not addressed the robustness of camera-radar fusion detection and tracking in scenarios involving sensor failure or data degradation from sources other than weather. In this project, I aim to lay a foundation for safer and more robust three-dimensional multi-object tracking. I propose a versatile, plug-and-play pipeline that leverages filtering methods for their adaptability to new scenarios and detection backbones. My main contributions are the design of the first camera-radar realistic noise synthesizer, along with the first camera-radar noise recognition models, along with my backbone-switching approach. My noise synthesizer allows me to artificially recreate sensor failure and data degradation on an existing large-scale autonomous vehicle dataset, without the need for additional information. Furthermore, my noise recognition module is used to estimate sensor noise levels, enabling dynamic switching between off-the-shelf detection backbones to adapt to data degradation. Average results across all the categories and scenes of the nuScenes validation dataset indicate that, in the event of sensor failure, my tracking method is on average 1.01% more precise compared to using each detection backbone individually, with a small 0.1% tracking accuracy tradeoff. On the car category specifically, my method shows an increase in tracking accuracy of up to 2.6% along with an increase in precision of up to 1.05%. Accuracy and precision are represented by the AMOTA and AMOTP metrics, respectively, of the nuScenes dataset.