by Tim Broedermann, Christos Sakaridis, Yuqian Fu, and Luc Van Gool

[Paper] [arXiv]

News:

• [2026-01-26] We are happy to announce DGFusion, which adds local depth guidance to CAFuser. DGFusion was accepted to the IEEE Robotics and Automation Letters.
• [2025-04-11] Code and models are released.
• [2025-01-11] We are happy to announce that CAFuser was accepted in the IEEE Robotics and Automation Letters.

This repository contains the official code for the RA-L 2025 paper CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenes. CAFuser is a condition-aware multimodal fusion architecture designed to enhance robust semantic perception in autonomous driving. It employs a Condition Token (CT), dynamically guiding the fusion of multiple sensor modalities to optimize performance across diverse scenarios. To train the CT, a verbo-visual contrastive loss aligns it with semantic environmental descriptors, enabling direct prediction from RGB features. The Condition-Aware Fusion module uses the CT to adaptively fuse sensor data based on environmental context. Further, CAFuser introduces modality-specific feature adapters, aligning inputs from different sensors into a shared latent space, and integrates these without loss of performance with a single shared backbone. CAFuser ranks first on the public MUSES benchmarks, achieving 59.7 PQ for multimodal panoptic and 78.2 mIoU for semantic segmentation and also sets the new state of the art on DeLiVER.

1. Installation
2. Prepare Datasets
3. Training
4. Evaluation
5. Citation
6. Acknowledgments

• We use Python 3.9.18, PyTorch 2.3.1, and CUDA 11.8.
• We use Detectron2-v0.6.
• For complete installation instructions, please see INSTALL.md.

• CAFuser support two datasets: MUSES and DeLiVER. The datasets are assumed to exist in a directory specified by the environment variable DETECTRON2_DATASETS. Under this directory, detectron2 will look for datasets in the structure described below, if needed.

• You can set the location for builtin datasets by export DETECTRON2_DATASETS=/path/to/datasets. If left unset, the default is ./datasets relative to your current working directory.
• For more details on how to prepare the datasets, please see detectron2's documentation.

MUSES dataset structure:

You need to dowload the following packages from the MUSES dataset:

• RGB_Frame_Camera_trainvaltest
• Panoptic_Annotations_trainval
• Semantic_Annotations_trainval
• Event_Camera_trainvaltest
• Lidar_trainvaltest
• Radar_trainvaltest
• GNSS_trainvaltest

and place them in the following structure:

DeLiVER dataset structure:

You can download the DeLiVER dataset from the following link and place it in the following structure:

• We train CAFuser using 4 NVIDIA TITAN RTX GPUs with 24GB memory each.
• For further details on training, please look at the OneFormer getting started and the getting Started with Detectron2
• We provide a script train_net.py, that is made to train all the configs provided in CAFuser.
• We further provide a dummy script slurm_train.sh to run the training on a SLURM cluster. Adjust the scripts at all places marked with TODO's to your needs.

Download swin tiny in the pretrained folder and convert it to a detectron2 compatible format:

Training CAFuser on MUSES dataset:

Training CAFuser on DeLiVER dataset:

• We provide pre-trained weights for CAFuser on MUSES and DeLiVER datasets. We provide models for both the standard CAFuser (CA²) and CAFuser-CAA variants. The models are trained on the MUSES dataset and the DeLiVER dataset, respectively.
• To evaluate a model's performance, use:

MUSES (on the validation set):

Predict on the test set to upload to the MUSES benchmark for both semantic and panoptic segmentation:

This will create folders under /inference for the semantic and panoptic predictions (e.g. output/cafuser_swin_tiny_bs8_200k_deliver_clde/inference/...).

• For the panoptic predictions, you can zip the labelIds folder under the panoptic folder and upload it to the MUSES benchmark.
• For the semantic predictions, you can zip the labelTrainIds folder under the semantic folder and upload it to the MUSES benchmark.

For better visualization you can further set MODEL.TEST.SAVE_PREDICTIONS.CITYSCAPES_COLORS True to get additional folders with the predictions in the cityscapes colors.

DeLiVER on the test set:

Replace deliver_semantic_test with deliver_semantic_val to evaluate on the validation set.

We provide the following results for the MUSES dataset, with the testing score from the official MUSES Benchmark:

We provide the following results for the DeLiVER dataset:

If you find this project useful in your research, please consider citing:

This project is based on the following open-source projects. We thank their authors for making the source code publicly available.

• OneFormer
• Mask2Former
• GroupViT
• Neighborhood Attention Transformer
• detectron2
• MUSES SDK
• HRFuser

This work was supported by the ETH Future Computing Laboratory (EFCL), financed by a donation from Huawei Technologies.