Dengxin Dai

Senior Researcher
E-Mail: ddai [at] mpi-inf.mpg.de
Phone: +49 681 9325 2104
Address: MPI for Informatics, Saarland Informatics Campus, 66123 Saarbrücken
Room: E1 4 – 604

Dengxin Dai is a Senior Researcher at MPI for Informatics and a Lecturer (external) at ETH Zurich. He is head of the research group Vision for Autonomous Systems. In 2016, he obtained his Ph.D. in Computer Vision at ETH Zurich and has been the group leader of the TRACE-Zurich team working on autonomous driving.  His research interests lie in autonomous driving, robust perception in adverse weather and illumination conditions, sensor fusion, multi-task learning, and object recognition under limited supervision.

He is a member of the ELLIS Society. He has organized the CVPR (2019-2022) Workshop ‘Vision for All Seasons: Bad Weather and Nighttime’, the ICCV (2021) workshop ‘Deep Multi-task Learning’, and the ICCV (2019) workshop ‘Autonomous Driving’. He has been a program committee member of several major computer vision conferences and received multiple outstanding reviewer awards. He is currently Editor of IJCV, was guest editor of IJCV, was/is area chair of WACV 2020, CVPR 2021, CVPR 2022, and ECCV 2022. He has received the Golden Owl Award with ETH Zurich in 2021 for his exceptional teaching.

  • Scale-Aware Domain Adaptive Faster R-CNN

    Authors: Yuhua Chen, Haoran Wang, Wen Li, Christos Sakaridis, Dengxin Dai and Luc Van Gool

    International Journal of Computer Vision

    We present Scale-aware Domain Adaptive Faster R-CNN, a model aiming at improving the cross-domain robustness of object detection.

    Object detection typically assumes that training and test samples are drawn from an identical distribution, which, however, does not always hold in practice. Such a distribution mismatch may lead to a significant performance drop. In this work, we present Scale-aware Domain Adaptive Faster R-CNN, a model aiming at improving the cross-domain robustness of object detection. In particular, our model improves the traditional Faster R-CNN model by tackling the domain shift on two levels: (1) the image-level shift, such as image style, illumination, etc., and (2) the instance-level shift, such as object appearance, size, etc. The two domain adaptation modules are implemented by learning domain classifiers in an adversarial training manner. Moreover, we observe that the large variance in object scales often brings a crucial challenge to cross-domain object detection. Thus, we improve our model by explicitly incorporating the object scale into adversarial training. We evaluate our proposed model on multiple cross-domain scenarios, including object detection in adverse weather, learning from synthetic data, and cross-camera adaptation, where the proposed model outperforms baselines and competing methods by a significant margin. The promising results demonstrate the effectiveness of our proposed model for cross-domain object detection.
    Read More

  • Multi-Scale Interaction for Real-Time LiDAR Data Segmentation on an Embedded Platform

    Authors: Shijie Li, Xieyuanli Chen, Yun Liu, Dengxin Dai, Cyrill Stachniss and Juergen Gall

    IEEE Robotics and Automation Letters

    We propose a projection-based method for semantic segmentation of LiDar data, called Multi-scale Interaction Network (MINet), which is very efficient and accurate.

    Real-time semantic segmentation of LiDAR data is crucial for autonomously driving vehicles and robots, which are usually equipped with an embedded platform and have limited computational resources. Approaches that operate directly on the point cloud use complex spatial aggregation operations, which are very expensive and difficult to deploy on embedded platforms. As an alternative, projection-based methods are more efficient and can run on embedded hardware. However, current projection-based methods either have a low accuracy or require millions of parameters. In this letter, we therefore propose a projection-based method, called Multi-scale Interaction Network (MINet), which is very efficient and accurate. The network uses multiple paths with different scales and balances the computational resources between the scales. Additional dense interactions between the scales avoid redundant computations and make the network highly efficient. The proposed network outperforms point-based, image-based, and projection-based methods in terms of accuracy, number of parameters, and runtime. Moreover, the network processes more than 24 scans, captured by a high-resolution LiDAR sensor with 64 beams, per second on an embedded platform, which is higher than the framerate of the sensor. The network is therefore suitable for robotics applications.
    Read More

  • Binaural SoundNet: Predicting Semantics, Depth and Motion with Binaural Sounds

    Authors: Dengxin Dai, Arun Balajee Vasudevan, Jiri Matas and Luc Van Gool

    IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

    This work develops an approach for scene understanding purely based on binaural sounds.

    Humans can robustly recognize and localize objects by using visual and/or auditory cues. While machines are able to do the same with visual data already, less work has been done with sounds. This work develops an approach for scene understanding purely based on binaural sounds. The considered tasks include predicting the semantic masks of sound-making objects, the motion of sound-making objects, and the depth map of the scene. To this aim, we propose a novel sensor setup and record a new audio-visual dataset of street scenes with eight professional binaural microphones and a 360-degree camera. The co-existence of visual and audio cues is leveraged for supervision transfer. In particular, we employ a cross-modal distillation framework that consists of multiple vision teacher methods and a sound student method -- the student method is trained to generate the same results as the teacher methods do. This way, the auditory system can be trained without using human annotations. To further boost the performance, we propose another novel auxiliary task, coined Spatial Sound Super-Resolution, to increase the directional resolution of sounds. We then formulate the four tasks into one end-to-end trainable multi-tasking network aiming to boost the overall performance. Experimental results show that 1) our method achieves good results for all four tasks, 2) the four tasks are mutually beneficial -- training them together achieves the best performance, 3) the number and orientation of microphones are both important, and 4) features learned from the standard spectrogram and features obtained by the classic signal processing pipeline are complementary for auditory perception tasks.
    Read More

  • DLOW: Domain Flow and Applications

    Authors: Rui Gong, Wen Li, Yuhua Chen, Dengxin Dai and Luc Van Gool

    International Journal of Computer Vision 2021

    We present a domain flow generation (DLOW) model to bridge two different domains by generating a continuous sequence of intermediate domains flowing from one domain to the other.

    In this work, we present a domain flow generation (DLOW) model to bridge two different domains by generating a continuous sequence of intermediate domains flowing from one domain to the other. The benefits of our DLOW model are twofold. First, it is able to transfer source images into a domain flow, which consists of images with smoothly changing distributions from the source to the target domain. The domain flow bridges the gap between source and target domains, thus easing the domain adaptation task. Second, when multiple target domains are provided for training, our DLOW model is also able to generate new styles of images that are unseen in the training data. The new images are shown to be able to mimic different artists to produce a natural blend of multiple art styles. Furthermore, for the semantic segmentation in the adverse weather condition, we take advantage of our DLOW model to generate images with gradually changing fog density, which can be readily used for boosting the segmentation performance when combined with a curriculum learning strategy. We demonstrate the effectiveness of our model on benchmark datasets for different applications, including cross-domain semantic segmentation, style generalization, and foggy scene understanding
    Read More

  • Task Switching Network for Multi-Task Learning

    Authors: Guolei Sun, Thomas Probst, Danda Pani Paudel, Nikola Popović, Menelaos Kanakis, Jagruti Patel, Dengxin Dai and Luc Van Gool

    International Conference on Computer Vision 2021

    We introduce Task Switching Networks (TSNs), a task-conditioned architecture with a single unified encoder/decoder for efficient multi-task learning.

    We introduce Task Switching Networks (TSNs), a task-conditioned architecture with a single unified encoder/decoder for efficient multi-task learning. Multiple tasks are performed by switching between them, performing one task at a time. TSNs have a constant number of parameters irrespective of the number of tasks. This scalable yet conceptually simple approach circumvents the overhead and intricacy of task-specific network components in existing works. In fact, we demonstrate for the first time that multi-tasking can be performed with a single task-conditioned decoder. We achieve this by learning task-specific conditioning parameters through a jointly trained task embedding network, encouraging constructive interaction between tasks. Experiments validate the effectiveness of our approach, achieving state-of-the-art results on two challenging multi-task benchmarks, PASCAL-Context and NYUD. Our analysis of the learned task embeddings further indicates a connection to task relationships studied in the recent literature
     
    Read More

  • DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation

    Authors: Lukas Hoyer , Dengxin Dai and Luc Van Gool

    Computer Vision and Pattern Recognition, 2022

    A novel UDA method, DAFormer, consisting of a Transformer encoder and a multi-level context-aware feature fusion decoder, improve SOTA by 10.8 mIoU for GTA->Cityscapes and 5.4 mIoU for Synthia->Cityscapes

    As acquiring pixel-wise annotations of real-world images for semantic segmentation is a costly process, a model can instead be trained with more accessible synthetic data and adapted to real images without requiring their annotations. This process is studied in unsupervised domain adaptation (UDA). Even though a large number of methods propose new adaptation strategies, they are mostly based on outdated network architectures. As the influence of recent network architectures has not been systematically studied, we first benchmark different network architectures for UDA and then propose a novel UDA method, DAFormer, based on the benchmark results. The DAFormer network consists of a Transformer encoder and a multi-level context-aware feature fusion decoder. It is enabled by three simple but crucial training strategies to stabilize the training and to avoid overfitting DAFormer to the source domain: While the Rare Class Sampling on the source domain improves the quality of pseudo-labels by mitigating the confirmation bias of self-training towards common classes, the Thing-Class ImageNet Feature Distance and a learning rate warmup promote feature transfer from ImageNet pretraining. DAFormer significantly improves the state-of-the-art performance by 10.8 mIoU for GTA->Cityscapes and 5.4 mIoU for Synthia->Cityscapes and enables learning even difficult classes such as train, bus, and truck well.
    Read More

  • CoSSL: Co-Learning of Representation and Classifier for Imbalanced Semi-Supervised Learning

    Authors: Yue Fan, Dengxin Dai and Bernt Schiele

    Preprint

    We propose a novel co-learning framework (CoSSL) with decoupled representation learning and classifier learning for imbalanced SSL.

    In this paper, we propose a novel co-learning framework (CoSSL) with decoupled representation learning and classifier learning for imbalanced SSL. To handle the data imbalance, we devise Tail-class Feature Enhancement (TFE) for classifier learning. Furthermore, the current evaluation protocol for imbalanced SSL focuses only on balanced test sets, which has limited practicality in real-world scenarios. Therefore, we further conduct a comprehensive evaluation under various shifted test distributions. In experiments, we show that our approach outperforms other methods over a large range of shifted distributions, achieving state-of-the-art performance on benchmark datasets ranging from CIFAR-10, CIFAR-100, ImageNet, to Food-101. Our code will be made publicly available.
    Read More

  • Decoupling Zero-Shot Semantic Segmentation

    Authors: Jian Ding, Nan Xue, Gui-Song Xia and Dengxin Dai

    Preprint

    Zero-shot semantic segmentation (ZS3) aims to segment the novel categories that have not been seen in the training. We propose to decouple the ZS3 into two sub-tasks: 1) a class-agnostic grouping task to group the pixels into segments. 2) a zero-shot classification task on segments.

    Zero-shot semantic segmentation (ZS3) aims to segment the novel categories that have not been seen in the training. Existing works formulate ZS3 as a pixel-level zeroshot classification problem, and transfer semantic knowledge from seen classes to unseen ones with the help of language models pre-trained only with texts. While simple, the pixel-level ZS3 formulation shows the limited capability to integrate vision-language models that are often pre-trained with image-text pairs and currently demonstrate great potential for vision tasks. Inspired by the observation that humans often perform segment-level semantic labeling, we propose to decouple the ZS3 into two sub-tasks: 1) a class-agnostic grouping task to group the pixels into segments. 2) a zero-shot classification task on segments. The former sub-task does not involve category information and can be directly transferred to group pixels for unseen classes. The latter subtask performs at segment-level and provides a natural way to leverage large-scale vision-language models pre-trained with image-text pairs (e.g. CLIP) for ZS3. Based on the decoupling formulation, we propose a simple and effective zero-shot semantic segmentation model, called ZegFormer, which outperforms the previous methods on ZS3 standard benchmarks by large margins, e.g., 35 points on the PASCAL VOC and 3 points on the COCO-Stuff in terms of mIoU for unseen classes. Code will be released at https: //github.com/dingjiansw101/ZegFormer.
    Read More

  • Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather

    Authors: Martin Hahner , Christos Sakaridis, Dengxin Dai and Luc Van Gool

    International Conference on Computer Vision, 2021

    This work presents a novel method for LiDAR-based 3D object detection in foggy weather by simulating foggy effects into standard LiDAR data..

    This work addresses the challenging task of LiDAR-based 3D object detection in foggy weather. Collecting and annotating data in such a scenario is very time, labor and cost intensive. In this paper, we tackle this problem by simulating physically accurate fog into clear-weather scenes, so that the abundant existing real datasets captured in clear weather can be repurposed for our task. Our contributions are twofold: 1) We develop a physically valid fog simulation method that is applicable to any LiDAR dataset. This unleashes the acquisition of large-scale foggy training data at no extra cost. These partially synthetic data can be used to improve the robustness of several perception methods, such as 3D object detection and tracking or simultaneous localization and mapping, on real foggy data. 2) Through extensive experiments with several state-of-the-art detection approaches, we show that our fog simulation can be leveraged to significantly improve the performance for 3D object detection in the presence of fog. Thus, we are the first to provide strong 3D object detection baselines on the Seeing Through Fog dataset. Our code is available at this http URL.
    Read More

  • ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding

    Authors: Christos Sakaridis, Dengxin Dai, and Luc Van Gool

    International Conference on Computer Vision, 2021

    ACDC is a large-scale dataset for training and testing semantic segmentation methods for four adverse visual conditions: fog, nighttime, rain, and snow.

    Level 5 autonomy for self-driving cars requires a robust visual perception system that can parse input images under any visual condition. However, existing semantic segmentation datasets are either dominated by images captured under normal conditions or are small in scale. To address this, we introduce ACDC, the Adverse Conditions Dataset with Correspondences for training and testing semantic segmentation methods on adverse visual conditions. ACDC consists of a large set of 4006 images which are equally distributed between four common adverse conditions: fog, nighttime, rain, and snow. Each adverse-condition image comes with a high-quality fine pixel-level semantic annotation, a corresponding image of the same scene taken under normal conditions, and a binary mask that distinguishes between intra-image regions of clear and uncertain semantic content. Thus, ACDC supports both standard semantic segmentation and the newly introduced uncertainty-aware semantic segmentation. A detailed empirical study demonstrates the challenges that the adverse domains of ACDC pose to state-of-the-art supervised and unsupervised approaches and indicates the value of our dataset in steering future progress in the field. Our dataset and benchmark are publicly available.
    Read More

  • Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation

    Authors: Qin Wang, Dengxin Dai, Lukas Hoyer, Luc Van Gool and Olga Fink

    International Conference on Computer Vision, 2021

    A novel method to leverage the guidance from self-supervised depth estimation to bridge the domain gap for semantic segmentation, achieving state-of-the-art performance..

    Domain adaptation for semantic segmentation aims to improve the model performance in the presence of a distribution shift between source and target domain. Leveraging the supervision from auxiliary tasks~(such as depth estimation) has the potential to heal this shift because many visual tasks are closely related to each other. However, such supervision is not always available. In this work, we leverage the guidance from self-supervised depth estimation, which is available on both domains, to bridge the domain gap. On the one hand, we propose to explicitly learn the task feature correlation to strengthen the target semantic predictions with the help of target depth estimation. On the other hand, we use the depth prediction discrepancy from source and target depth decoders to approximate the pixel-wise adaptation difficulty. The adaptation difficulty, inferred from depth, is then used to refine the target semantic segmentation pseudo-labels. The proposed method can be easily implemented into existing segmentation frameworks. We demonstrate the effectiveness of our approach on the benchmark tasks SYNTHIA-to-Cityscapes and GTA-to-Cityscapes, on which we achieve the new state-of-the-art performance of 55.0% and 56.6%, respectively. Our code is available here.
    Read More

  • End-to-End Urban Driving by Imitating a Reinforcement Learning Coach

    Authors: Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu and Luc Van Gool

    International Conference on Computer Vision, 2021

    A novel manner to learn end-to-end driving from a reinforcement learning expert that maps bird's-eye view images to continuous low-level actions, achieving the state-of-the-art perfomrance.

    End-to-end approaches to autonomous driving commonly rely on expert demonstrations. Although humans are good drivers, they are not good coaches for end-to-end algorithms that demand dense on-policy supervision. On the contrary, automated experts that leverage privileged information can efficiently generate large scale on-policy and off-policy demonstrations. However, existing automated experts for urban driving make heavy use of hand-crafted rules and perform suboptimally even on driving simulators, where ground-truth information is available. To address these issues, we train a reinforcement learning expert that maps bird's-eye view images to continuous low-level actions. While setting a new performance upper-bound on CARLA, our expert is also a better coach that provides informative supervision signals for imitation learning agents to learn from. Supervised by our reinforcement learning coach, a baseline end-to-end agent with monocular camera-input achieves expert-level performance. Our end-to-end agent achieves a 78% success rate while generalizing to a new town and new weather on the NoCrash-dense benchmark and state-of-the-art performance on the challenging public routes of the CARLA LeaderBoard.
    Read More

  • mDALU: Multi-Source Domain Adaptation and Label Unification with Partial Datasets

    Authors: Rui Gong, Dengxin Dai, Yuhua Chen, Wen Li and Luc Van Gool

    International Conference on Computer Vision, 2021

    A novel method to combine and reuse existing datasets that belong to different domains, have partial annotations, and/or have different data modalities, to boost the overall performance on the target domain.

    One challenge of object recognition is to generalize to new domains, to more classes and/or to new modalities. This necessitates methods to combine and reuse existing datasets that may belong to different domains, have partial annotations, and/or have different data modalities. This paper formulates this as a multi-source domain adaptation and label unification problem, and proposes a novel method for it. Our method consists of a partially-supervised adaptation stage and a fully-supervised adaptation stage. In the former, partial knowledge is transferred from multiple source domains to the target domain and fused therein. Negative transfer between unmatching label spaces is mitigated via three new modules: domain attention, uncertainty maximization and attention-guided adversarial alignment. In the latter, knowledge is transferred in the unified label space after a label completion process with pseudo-labels. Extensive experiments on three different tasks – image classification, 2D semantic image segmentation, and joint 2D-3D semantic segmentation – show that our method outperforms all competing methods significantly.
    Read More

  • End-to-End Learning of Driving Models with Surround-View Cameras and Route Planners

    Authors: Simon Hecker, Dengxin Dai and Luc Van Gool

    European Conference on Computer Vision, 2018

    We propose a novel end-to-end driving method that can learn how-to-drive directly from surrounding-view videos and route planners.

    For human drivers, having rear and side-view mirrors is vital for safe driving. They deliver a more complete view of what is happening around the car. Human drivers also heavily exploit their mental map for navigation. Nonetheless, several methods have been published that learn driving models with only a front-facing camera and without a route planner. This lack of information renders the self-driving task quite intractable. We investigate the problem in a more realistic setting, which consists of a surround-view camera system with eight cameras, a route planner, and a CAN bus reader. In particular, we develop a sensor setup that provides data for a 360-degree view of the area surrounding the vehicle, the driving route to the destination, and low-level driving maneuvers (e.g. steering angle and speed) by human drivers. With such a sensor setup we collect a new driving dataset, covering diverse driving scenarios and varying weather/illumination conditions. Finally, we learn a novel driving model by integrating information from the surround-view cameras and the route planner. Two route planners are exploited: 1) by representing the planned routes on OpenStreetMap as a stack of GPS coordinates, and 2) by rendering the planned routes on TomTom Go Mobile and recording the progression into a video. Our experiments show that: 1) 360-degree surround-view cameras help avoid failures made with a single front-view camera, in particular for city driving and intersection scenarios; and 2) route planners help the driving task significantly, especially for steering angle prediction.
    Read More
  • Area Chair of WACV 2020, CVPR 2021, CVPR 2022, ECCV 2020.
  • Co-organizer of Workshop DeepMTL: Deep Multi-Task Learning at ICCV’21.
  • Lead Guest Editor of the Special Issue “Computer Vision for All Seasons” of IJCV.
  • Senior Program Committee Member: AAAI 2020 and IJCAI 2019.
  • Lead workshop organizer: Vision for All Seasons at CVPR’19, CVPR’20, and CVPR’21.
  • Lead workshop organizer: Autonomous Driving workshop at ICCV’19.
  • Co-organizer of challenge Learning to Drive at ICCV’19
  • Co-organizer of challenge Nighttime Semantic Image Segmentation at CVPR’20.
  • Jury member for the Pioneer Fellowship Program at ETH Zurich.
  • Examiner of Doctoral Exams: EPFL, NUS and ETH Zurich.
  • Regular Reviewers: CVPR, ICCV, ECCV, NeurIPS, ICLR, ICML, ICRA, IROS, AAAI, IJCV, and PAMI.
  1. The IV workshop “Beyond supervised learning: addressing data scarcity in intelligent transportation systems“, 2022
  2. The “Learning to Understand Aerial Images (LUAI)” Workshop at ICCV, Oct. 2021.
  3. The Saarland Informatics Campus Lecture Series, Oct. 2021.
  4. The “Radar Perception for All-Weather Autonomy” Workshop at ICRA, May 2021.
  5. Zurich High School, “General Introduction to Autonomous Driving”, Nov 2020.
  6. The Robotics and Perception Group at the University of Zurich, Sep. 2020.
  7. The “Transferring and Adapting Source Knowledge in ComputerVision and VISDA Challenge” Workshop at ECCV, 2020.
  8. The “Map-based Localization for Autonomous Driving” Workshop at ECCV, 2020.
  9. The “Commands for Autonomous Vehicle” Workshop at ECCV, 2020.
  10. The “Autonomous Driving Workshop” at CVPR, 2020.
  11. The “Bridging the Gap between Computational Photography and Visual Recognition” Workshop at CVPR, 2020.
  12. The “Physics-Based Vision meets Deep Learning (PBDL)” Workshop at ICCV, 2019.

  13. International VDI Conference – Future of AI in Automotive, Berlin, 2018.

  14. The “ApolloScape: the Vision-based Navigation for Autonomous Driving” Workshop at ECCV, 2018.

  15. Scientifica: Zurich Science Days, 2017.

  1. Lead Lecturer, Deep Learning for Autonomous Driving, 6 ECTS, ETH Zurich, 2020 (80+ students).

  2. Lead Lecturer, Deep Learning for Autonomous Driving, 6 ECTS, ETH Zurich, 2021 (100+ students).
  3. Lead Lecturer, Deep Learning for Autonomous Driving, 6 ECTS, ETH Zurich, 2022 (100+ students).
  4. Tutorial “From Image Restoration to Enhancement and Beyond” at ICCV, 2019.