We propose an extended version, named SegNet4D, featuring complete 4D semantic segmentation capability and enhanced motion recognition performance.
This repository contains the implementation of our paper:
InsMOS: Instance-Aware Moving Object Segmentation in LiDAR Data
Neng Wang, Chenghao Shi, Ruibin Guo, Huimin Lu, Zhiqiang Zheng, Xieyuanli Chen
-
Our instance-aware moving object segmentation on the SemanticKITTI sequence 08 and 20, 21.
-
Red points indicate predicted moving points, cyan indicate predicted static instance points and gray points are static background.
-
Green bounding boxes represent cars, blue bounding boxes represent pedestrians, and yellow bounding boxes represent cyclists.
News
- [2023-8-12] Code released!
- [2023-6-22] Our work is accepted for IROS2023
Citation
If you use our code in your work, please star our repo and cite our paper.
title={{InsMOS: Instance-Aware Moving Object Segmentation in LiDAR Data}},
author={Wang, Neng and Shi, Chenghao and Guo, Ruibin and Lu, Huimin and Zheng, Zhiqiang and Chen, Xieyuanli},
booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
pages={7598-7605},
year={2023}
}
Data
1, SemanticKITTI: Download SemanticKITTI dataset from the official website.
2, KITTI-road Dataset: Download the KITTI-road Velodyne point clouds from the official website and MOS label from MotionSeg3D.
3, Instance label: Download the box labels from ondrive or BaiduDisk,code:59t7, and please refer to boundingbox_label_readme about more details of instance label .
Data structure
+-- sequences
+-- 00/
| +-- velodyne/
| | +-- 000000.bin
| | +-- 000001.bin
| | +-- ...
| +-- labels/
| | +-- 000000.label
| | +-- 000001.label
| | +-- ...
| +-- boundingbox_label
| | +-- 000000.npy
| | +-- 000001.npy
| | +-- ...
| +-- calib.txt
| +-- poses.txt
| +-- times.txt
+-- 01/ # 00-10 for training
+-- 08/ # for validation
+-- 11-21/ # 11-21 for testing
# kitti-road
+-- 30 31 32 33 34 40 # for training
+-- 35 36 37 38 39 41 # for testing
Installation
1. Dependencies
system dependencies:
python dependencies:
2. Set up conda environment
conda activate insmos
pip install -r requirements.txt
# insltall pytorch with cuda11.3, avoid using "pip install torch"
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge
# ensure numpy==1.18.1
pip uninstall numpy
pip install numpy==1.18.1
Install MinkowskiEngine :
mkdir ThirdParty
sudo apt-get install libopenblas-dev
git clone https://github.com/NVIDIA/MinkowskiEngine.git
cd MinkowskiEngine
conda activate insmos
python setup.py install --blas_include_dirs=${CONDA_PREFIX}/include --blas=openblas
3. Install InsMOS
git clone https://github.com/nubot-nudt/InsMOS.git
cd InsMOS
# activate conda
conda activate insmos
# install
python setup.py develop
Inference
Run the following command to evaluate the model in SemanticKITTI validation dataset or test dataset. At this moment, a "preb_out" folder will be generated, which contains "bbox_preb", "confidence", and "mos_preb" for storing the predicted bounding boxes, confidence scores for moving points, and labels for moving points, respectively.
We public the model was trained on the SemanticKITTI dataset (N_10_t_0.1_odom.ckpt) and the other model was trained on the Semantic-KITTI and KITTI-road dataset (N_10_t_0.1_odom_road.ckpt). You can download from ondrive or BaiduDisk,code:59t7, and then put the model in "ckpt" folder.
python scripts/predict_mos.py --cfg_file config/config.yaml --data_path /path/to/kitti/sequences --ckpt ./ckpt/N_10_t_0.1_odom.ckpt --split valid
Evaluate
We use the semantickitti-api to evaluate the MOS IOU.
python evaluate_mos.py --dataset /path/to/kitti --predictions ./preb_out/InsMOS/mos_preb --split valid
Refine
Run the following command to refine the network predictions.
Re-evaluate the refinement
Re-evaluate the results of refinement.
Visual
Run the following command to visualize the results of moving object segmentation and instance prediction.
Press key n to show next frame.
Press key b to show last frame.
Press key q to quit display.
python vis_mos_bbox.py
Train
You can set single gpu or multi gpu for training in train.py. We set batch size to 4 for each gpu. During the training process, there may be an increase in GPU memory consumption, so it is advisable not to set the batch size too large initially. We test 4-6 is fine on 3090 GPU.
python scripts/train.py
If the training process is interrupted unexpectedly, you can resume the training using the following command.
Contact
Any question or suggestions are welcome!
Neng Wang: nwang@nudt.edu.cn and Xieyuanli Chen: website
License
This project is free software made available under the MIT License. For details see the LICENSE file.