Towards Stable 3D Object Detection (ECCV 2024)
Jiabao Wang*, Qiang Meng*, Guochao Liu, Liujiang Yan, Ke Wang, Ming-Ming Cheng, Qibin Hou#
Abstract
In autonomous driving, the temporal stability of 3D object detection greatly impacts the driving safety. However, the detection stability cannot be accessed by existing metrics such as mAP and MOTA, and consequently is less explored by the community. To bridge this gap, this work proposes Stability Index (SI), a new metric that can comprehensively evaluate the stability of 3D detectors in terms of confidence, box localization, extent, and heading. By benchmarking state-of-the-art object detectors on the Waymo Open Dataset, SI reveals interesting properties of object stability that have not been previously discovered by other metrics. To help models improve their stability, we further introduce a general and effective training strategy, called Prediction Consistency Learning (PCL). PCL essentially encourages the prediction consistency of the same objects under different timestamps and augmentations, leading to enhanced detection stability. Furthermore, we examine the effectiveness of PCL with the widely-used CenterPoint, and achieve a remarkable SI of 86.00 for vehicle class, surpassing the baseline by 5.48. We hope our work could serve as a reliable baseline and draw the community's attention to this crucial issue in 3D object detection.
Demos
| Confidence | Localization |
| Extent | Heading |
Main Results
We could not provide pre-trained models due to Waymo Dataset License Agreement. However, one can easily reproduce similar performances with our provided default configurations.
Benckmark on Waymo Open Dataset
| Methods | Vehicle(%) | Pedestrain(%) | Cyclist(%) | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| mAPH_l1 | SI | SI_c | SI_l | SI_e | SI_h | mAPH_l1 | SI | SI_c | SI_l | SI_e | SI_h | mAPH_l1 | SI | SI_c | SI_l | SI_e | SI_h | |
| Second | 72.60 | 81.32 | 90.17 | 84.18 | 91.95 | 92.10 | 59.81 | 62.80 | 83.94 | 69.34 | 87.30 | 67.31 | 61.95 | 66.29 | 81.94 | 75.36 | 85.99 | 81.29 |
| CenterPoint-Pillar | 72.44 | 80.61 | 89.03 | 85.37 | 91.00 | 92.82 | 65.59 | 64.57 | 83.24 | 74.43 | 87.38 | 68.85 | 67.36 | 68.06 | 80.77% | 77.66 | 86.96 | 85.89 |
| PointPillar | 72.84 | 80.84 | 89.58 | 84.40 | 92.26 | 91.60 | 54.64 | 62.03 | 84.65 | 72.08 | 88.83 | 57.93 | 59.51 | 66.14 | 82.15% | 74.85 | 88.00 | 77.36 |
| CenterPoint | 73.73 | 80.52 | 89.04 | 85.33 | 90.70 | 92.88 | 69.50 | 68.40 | 85.74 | 73.28 | 88.58 | 74.95 | 71.04 | 68.40 | 80.31% | 78.47 | 87.37 | 89.79 |
| PartA2Net | 75.02 | 82.86 | 91.43 | 85.38 | 91.68 | 91.69 | 66.16 | 65.08 | 84.59 | 73.61 | 86.71 | 67.04 | 67.90 | 72.73 | 85.94% | 79.34 | 86.97 | 84.29 |
| PV R-CNN | 75.92 | 83.73 | 91.94 | 86.36 | 92.30 | 91.66 | 66.28 | 66.17 | 86.02 | 73.50 | 87.39 | 66.58 | 68.38 | 73.53 | 86.84% | 78.86 | 88.44 | 83.22 |
| Voxel R-CNN | 77.19 | 84.26 | 92.01 | 86.66 | 92.11 | 93.33 | 74.21 | 69.50 | 86.87 | 75.33 | 88.06 | 73.56 | 71.68 | 73.23 | 84.42% | 80.07 | 87.67 | 89.28 |
| VoxelNeXt | 77.84 | 84.82 | 92.88 | 86.28 | 91.59 | 94.17 | 76.24 | 74.74 | 92.67 | 75.65 | 88.03 | 75.80 | 75.59 | 76.48 | 89.98% | 79.24 | 84.90 | 87.76 |
| PV R-CNN++ | 77.88 | 84.49 | 92.06 | 87.22 | 92.38 | 93.18 | 73.99 | 69.27 | 86.75 | 75.27 | 88.09 | 73.24 | 71.84 | 73.05 | 84.23% | 80.29 | 87.66 | 89.19 |
| PV R-CNN++ Res | 78.33 | 85.17 | 92.47 | 87.52 | 92.51 | 93.85 | 75.75 | 70.15 | 87.16 | 75.83 | 87.89 | 74.81 | 72.47 | 73.31 | 84.25% | 80.61 | 87.58 | 89.65 |
| DSVT | 78.82 | 84.90 | 92.51 | 86.90 | 91.53 | 94.76 | 76.81 | 74.58 | 91.88 | 76.47 | 88.71 | 75.92 | 75.44 | 76.20 | 88.22% | 80.48 | 86.11 | 89.88 |
| TransFusion | 79.00 | 82.32 | 89.34 | 86.8 | 92.73 | 95.68 | 76.52 | 69.11 | 84.53 | 75.39 | 89.89 | 78.78 | 70.11 | 70.35 | 80.63% | 79.49 | 90.56 | 91.13 |
Results of PCL
| Methods | Vehicle(%) | Pedestrian(%) | Cyclist(%) | |||
|---|---|---|---|---|---|---|
| mAPH_l1 | SI | mAPH_l1 | SI | mAPH_l1 | SI | |
| Baseline | 73.73 | 80.52 | 69.50 | 68.40 | 71.04 | 68.40 |
| w/o PCL | 73.70 | 80.93 | 69.55 | 68.35 | 71.27 | 68.20 |
| PCL (n=0) | 75.57 | 85.42 | 70.18 | 71.87 | 70.86 | 68.80 |
| PCL (n=4) | 75.26 | 85.83 | 69.56 | 72.76 | 70.65 | 69.22 |
| PCL (n=8) | 75.04 | 85.94 | 68.82 | 72.87 | 70.31 | 69.32 |
| PCL (n=12) | 74.64 | 85.93 | 68.50 | 72.95 | 70.85 | 69.33 |
| PCL (n=16) | 74.54 | 86.00 | 67.82 | 73.14 | 70.25 | 69.16 |
Installation
This repository is implemented on top of OpenPCDet. For package installation and data preparation, please follow the guidance of the INSTALL.md and GETTING_STARTED.md in OpenPCDet.
Training and Evaluation
All our experimental configurations can be found in waymo_stable. Steps to reproduce results in our paper include:
Training Baseline CenterPoint
The first step is to train a baseline CenterPoint model configured with centerpoint_baseline.yaml.
cd tools
bash scripts/dist_train.sh 8 --cfg_file cfgs/waymo_stable/centerpoint_baseline.yaml
Fine-tuning with PCL
We provide the configurations of fine-tuning CenterPoint without PCL (centerpoint_finetune.yaml) and with PCL (centerpoint_PCL_n{X}.yaml). Here, X is the maximum interval between the neighborhood sampling, as described in our paper.
Please do not forget to fill in the correct path to baseline checkpoint in argument PRETRAINED before the fine-tuning step.
MODEL:
NAME: CenterPoint
# change this path to the baseline checkpoint.
PRETRAINED: '../output/waymo_stable/centerpoint_baseline/default/ckpt/checkpoint_epoch_36.pth'