Loading...
Home2022-10-20T13:50:32+00:00

Highlights

German Pattern Recognition Award 2022

October 10th, 2022|

Dr. Dengxin Dai has received the prestigious German Pattern Recognition Award 2022. The award is annually granted for outstanding, internationally visible research in the fields of pattern recognition, computer vision, and machine learning. It was granted to Dr. Dai this year for his outstanding scientific contributions in the area of […]

Read More

Dengxin Dai receives the Golden Owl Award

December 23rd, 2021|

We are very proud to announce that Dr. Dengxin Dai has received the Golden Owl Award 2021 for exceptional teaching.

The Golden Owl honours excellent teachers. The Owl is awarded by VSETH, ETH Zurich’s student association.
All ETH members with a teaching assignment can be nominated for the Golden Owl. One lecturer […]

Read More

News

  • 01/12/2022:

    Two papers accepted to ICRA’23 and one paper accepted to IJCV.

  • 24/10/2022:

    Dengxin Dai has given an invited talk at the ECCV’22 workshop on 3D Perception for Autonomous Driving

  • 10/10/2022:

    Dengxin Dai has been awarded the German Pattern Recognition Award 2022.

  • 10/10/2022:

    Motion Transformer for Motion Prediction has been accepted by NeurIPS 2022.

  • 10/10/2022:

    We won the 1st place in Waymo Open Dataset Challenge 2022 on Motion Prediction.

  • 10/10/2022:

    Dengxin Dai is Associate Editor of ICRA 2023.

  • 10/08/2022:

    Three papers accepted to ECCV 2022.

  • 13/05/2022:

    Dengxin Dai has joined the IJCV Editorial Board.

  • 13/05/2022:

    Lukas Hoyer has been honoured the ETH Medal for his Outstanding Master’s Theses done with Dengxin Dai. Congratulations!

  • 11/03/2022:

    11 papers (3 Orals) accepted to CVPR 2022. Congratulations to all co-authors!

Scientific Mission

The scientific mission of VAS is to develop robust and scalable perception systems for real-world applications. We focus on deep learning-based perception for autonomous systems such as autonomous driving. We are especially fascinated about scaling existing visual perception models to novel domains (e.g. adverse weather/lighting conditions, low-quality data),  to more data modality (e.g. LiDAR, Radar, Events, Audio, HD Maps), to unseen classes (e.g. rare classes), and to new tasks. The relevant research topics are summarised in the diagram shown on the right.

Publications

  • End-to-End Optimization of LiDAR Beam Configuration for 3D Object Detection and Localization

    Authors: Niclas Vödisch, Ozan Unal, Ke Li, Luc Van Gool, and Dengxin Dai

    ICRA, 2022

    The first end-to-end approach to learn to optimize the LiDAR beam configuration for given applications

    Existing learning methods for LiDAR-based applications use 3D points scanned under a pre-determined beam configuration, e.g., the elevation angles of beams are often evenly distributed. Those fixed configurations are task-agnostic, so simply using them can lead to sub-optimal performance. In this work, we take a new route to learn to optimize the LiDAR beam configuration for a given application. Specifically, we propose a reinforcement learning-based learning-to-optimize (RL-L2O) framework to automatically optimize the beam configuration in an end-to-end manner for different LiDAR-based applications. The optimization is guided by the final performance of the target task and thus our method can be integrated easily with any LiDAR-based application as a simple drop-in module. The method is especially useful when a low-resolution (low-cost) LiDAR is needed, for instance, for system deployment at a massive scale. We use our method to search for the beam configuration of a low-resolution LiDAR for two important tasks: 3D object detection and localization. Experiments show that the proposed RL-L2O method improves the performance in both tasks significantly compared to the baseline methods. We believe that a combination of our method with the recent advances of programmable LiDARs can start a new research direction for LiDAR-based active perception. The code is publicly available at github.com/vniclas/lidar_beam_selection.
    Read More

  • Adiabatic Quantum Computing for Multi Object Tracking

    Authors: Jan-Nico Zaech, Alexander Liniger, Martin Danelljan, Dengxin Dai and Luc Van Gool

    CVPR, 2022

    The first MOT formulation designed to be solved with Adiabatic Quantum Computing.

    Multi-Object Tracking (MOT) is most often approached in the tracking-by-detection paradigm, where object detections are associated through time. The association step naturally leads to discrete optimization problems. As these optimization problems are often NP-hard, they can only be solved exactly for small instances on current hardware. Adiabatic quantum computing (AQC) offers a solution for this, as it has the potential to provide a considerable speedup on a range of NP-hard optimization problems in the near future. However, current MOT formulations are unsuitable for quantum computing due to their scaling properties. In this work, we therefore propose the first MOT formulation designed to be solved with AQC. We employ an Ising model that represents the quantum mechanical system implemented on the AQC. We show that our approach is competitive compared with state-of-the-art optimization-based approaches, even when using of-the-shelf integer programming solvers. Finally, we demonstrate that our MOT problem is already solvable on the current generation of real quantum computers for small examples, and analyze the properties of the measured solutions.
    Read More

  • LiDAR Snowfall Simulation for Robust 3D Object Detection

    Authors: Martin Hahner, Christos Sakaridis, Mario Bijelic, Felix Heide, Fisher Yu Dengxin Dai and Luc Van Gool

    CVPR (Oral), 2022

    A novel simulation approach to simulate snowfall effects into existing LiDAR dataset to train robust LiDAR-based perception methods for adverse weather

    3D object detection is a central task for applications such as autonomous driving, in which the system needs to localize and classify surrounding traffic agents, even in the presence of adverse weather. In this paper, we address the problem of LiDAR-based 3D object detection under snowfall. Due to the difficulty of collecting and annotating training data in this setting, we propose a physically based method to simulate the effect of snowfall on real clear-weather LiDAR point clouds. Our method samples snow particles in 2D space for each LiDAR line and uses the induced geometry to modify the measurement for each LiDAR beam accordingly. Moreover, as snowfall often causes wetness on the ground, we also simulate ground wetness on LiDAR point clouds. We use our simulation to generate partially synthetic snowy LiDAR data and leverage these data for training 3D object detection models that are robust to snowfall. We conduct an extensive evaluation using several state-of-the-art 3D object detection methods and show that our simulation consistently yields significant performance gains on the real snowy STF dataset compared to clear-weather baselines and competing simulation approaches, while not sacrificing performance in clear weather. Our code is available at github.com/SysCV/LiDAR_snow_sim.
    Read More

  • Continual Test-Time Domain Adaptation

    Authors: Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai

    CVPR, 2022

    A novel method for long-term test-time adaptation under continually changing environments.

    Test-time domain adaptation aims to adapt a source pre-trained model to a target domain without using any source data. Existing works mainly consider the case where the target domain is static. However, real-world machine perception systems are running in non-stationary and continually changing environments where the target domain distribution can change over time. Existing methods, which are mostly based on self-training and entropy regularization, can suffer from these non-stationary environments. Due to the distribution shift over time in the target domain, pseudo-labels become unreliable. The noisy pseudo-labels can further lead to error accumulation and catastrophic forgetting. To tackle these issues, we propose a continual test-time adaptation approach (CoTTA) which comprises two parts. Firstly, we propose to reduce the error accumulation by using weight-averaged and augmentation-averaged predictions which are often more accurate. On the other hand, to avoid catastrophic forgetting, we propose to stochastically restore a small part of the neurons to the source pre-trained weights during each iteration to help preserve source knowledge in the long-term. The proposed method enables the long-term adaptation for all parameters in the network. CoTTA is easy to implement and can be readily incorporated in off-the-shelf pre-trained models. We demonstrate the effectiveness of our approach on four classification tasks and a segmentation task for continual test-time adaptation, on which we outperform existing methods. Our code is available at https://qin.ee/cotta.
    Read More

  • ZegFormer: Decoupling Zero-Shot Semantic Segmentation

    Authors: Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai

    CVPR, 2022

    ZegFormer is the first framework that decouple the zero-shot semantic segmentation into: 1) class-agnostic segmentation and 2) segment-level zero-shot classification

    Zero-shot semantic segmentation (ZS3) aims to segment the novel categories that have not been seen in the training. Existing works formulate ZS3 as a pixel-level zeroshot classification problem, and transfer semantic knowledge from seen classes to unseen ones with the help of language models pre-trained only with texts. While simple, the pixel-level ZS3 formulation shows the limited capability to integrate vision-language models that are often pre-trained with image-text pairs and currently demonstrate great potential for vision tasks. Inspired by the observation that humans often perform segment-level semantic labeling, we propose to decouple the ZS3 into two sub-tasks: 1) a classagnostic grouping task to group the pixels into segments. 2) a zero-shot classification task on segments. The former task does not involve category information and can be directly transferred to group pixels for unseen classes. The latter task performs at segment-level and provides a natural way to leverage large-scale vision-language models pre-trained with image-text pairs (e.g. CLIP) for ZS3. Based on the decoupling formulation, we propose a simple and effective zero-shot semantic segmentation model, called ZegFormer, which outperforms the previous methods on ZS3 standard benchmarks by large margins, e.g., 22 points on the PASCAL VOC and 3 points on the COCO-Stuff in terms of mIoU for unseen classes. Code will be released at https: //github.com/dingjiansw101/ZegFormer
    Read More

  • Scribble-Supervised LiDAR Semantic Segmentation

    Authors: Ozan Unal, Dengxin Dai and Luc Van Gool

    CVPR (Oral), 2022

    ScribbleKITTI, the first scribble-annotated dataset for LiDAR semantic segmentation and a novel learning method to reduce the performance gap that arises when using such weak annotations.

    Densely annotating LiDAR point clouds remains too expensive and time-consuming to keep up with the ever growing volume of data. While current literature focuses on fully-supervised performance, developing efficient methods that take advantage of realistic weak supervision have yet to be explored. In this paper, we propose using scribbles to annotate LiDAR point clouds and release ScribbleKITTI, the first scribble-annotated dataset for LiDAR semantic segmentation. Furthermore, we present a pipeline to reduce the performance gap that arises when using such weak annotations. Our pipeline comprises of three stand-alone contributions that can be combined with any LiDAR semantic segmentation model to achieve up to 95.7% of the fully-supervised performance while using only 8% labeled points. Our scribble annotations and code are available at github.com/ouenal/scribblekitti
    Read More

  • HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation

    Authors: Lukas Hoyer, Dengxin Dai and Luc Van Gool

    ECCV, 2022

    A multi-resolution training approach for UDA, that combines the strengths of small high-resolution crops to preserve fine segmentation details and large low-resolution crops to capture long-range context dependencies with a learned scale attention.

    Unsupervised domain adaptation (UDA) aims to adapt a model trained on the source domain (e.g. synthetic data) to the target domain (e.g. real-world data) without requiring further annotations on the target domain. This work focuses on UDA for semantic segmentation as real-world pixel-wise annotations are particularly expensive to acquire. As UDA methods for semantic segmentation are usually GPU memory intensive, most previous methods operate only on downscaled images. We question this design as low-resolution predictions often fail to preserve fine details. The alternative of training with random crops of high-resolution images alleviates this problem but falls short in capturing long-range, domain-robust context information. Therefore, we propose HRDA, a multi-resolution training approach for UDA, that combines the strengths of small high-resolution crops to preserve fine segmentation details and large low-resolution crops to capture long-range context dependencies with a learned scale attention, while maintaining a manageable GPU memory footprint. HRDA enables adapting small objects and preserving fine segmentation details. It significantly improves the state-of-the-art performance by 5.5 mIoU for GTA-to-Cityscapes and 4.9 mIoU for Synthia-to-Cityscapes, resulting in unprecedented 73.8 and 65.8 mIoU, respectively. The implementation is available at this https URL.
    Read More

  • Motion Transformer with Global Intention Localization and Local Movement Refinement

    Authors: Shaoshuai Shi, Li Jiang, Dengxin Dai and Bernt Schiele

    NeurIPS, 2022

    A novel Motion TRansformer (MTR) framework that models motion prediction as the joint optimization of global intention localization and local movement refinement. Won the championship of Waymo Open Dataset Challenge 2022 on Motion Prediction

    Predicting multimodal future behavior of traffic participants is essential for robotic vehicles to make safe decisions. Existing works explore to directly predict future trajectories based on latent features or utilize dense goal candidates to identify agent's destinations, where the former strategy converges slowly since all motion modes are derived from the same feature while the latter strategy has efficiency issue since its performance highly relies on the density of goal candidates. In this paper, we propose Motion TRansformer (MTR) framework that models motion prediction as the joint optimization of global intention localization and local movement refinement. Instead of using goal candidates, MTR incorporates spatial intention priors by adopting a small set of learnable motion query pairs. Each motion query pair takes charge of trajectory prediction and refinement for a specific motion mode, which stabilizes the training process and facilitates better multimodal predictions. Experiments show that MTR achieves state-of-the-art performance on both the marginal and joint motion prediction challenges, ranking 1st on the leaderboards of Waymo Open Motion Dataset. Code will be available at this https URL.
    Read More

  • Binaural SoundNet: Predicting Semantics, Depth and Motion with Binaural Sounds

    Authors: Dengxin Dai, Arun Balajee Vasudevan, Jiri Matas and Luc Van Gool

    IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

    This work develops an approach for scene understanding purely based on binaural sounds.

    Humans can robustly recognize and localize objects by using visual and/or auditory cues. While machines are able to do the same with visual data already, less work has been done with sounds. This work develops an approach for scene understanding purely based on binaural sounds. The considered tasks include predicting the semantic masks of sound-making objects, the motion of sound-making objects, and the depth map of the scene. To this aim, we propose a novel sensor setup and record a new audio-visual dataset of street scenes with eight professional binaural microphones and a 360-degree camera. The co-existence of visual and audio cues is leveraged for supervision transfer. In particular, we employ a cross-modal distillation framework that consists of multiple vision teacher methods and a sound student method -- the student method is trained to generate the same results as the teacher methods do. This way, the auditory system can be trained without using human annotations. To further boost the performance, we propose another novel auxiliary task, coined Spatial Sound Super-Resolution, to increase the directional resolution of sounds. We then formulate the four tasks into one end-to-end trainable multi-tasking network aiming to boost the overall performance. Experimental results show that 1) our method achieves good results for all four tasks, 2) the four tasks are mutually beneficial -- training them together achieves the best performance, 3) the number and orientation of microphones are both important, and 4) features learned from the standard spectrogram and features obtained by the classic signal processing pipeline are complementary for auditory perception tasks.
    Read More

  • DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation

    Authors: Lukas Hoyer , Dengxin Dai and Luc Van Gool

    Computer Vision and Pattern Recognition, 2022

    A novel UDA method, DAFormer, consisting of a Transformer encoder and a multi-level context-aware feature fusion decoder, improve SOTA by 10.8 mIoU for GTA->Cityscapes and 5.4 mIoU for Synthia->Cityscapes

    As acquiring pixel-wise annotations of real-world images for semantic segmentation is a costly process, a model can instead be trained with more accessible synthetic data and adapted to real images without requiring their annotations. This process is studied in unsupervised domain adaptation (UDA). Even though a large number of methods propose new adaptation strategies, they are mostly based on outdated network architectures. As the influence of recent network architectures has not been systematically studied, we first benchmark different network architectures for UDA and then propose a novel UDA method, DAFormer, based on the benchmark results. The DAFormer network consists of a Transformer encoder and a multi-level context-aware feature fusion decoder. It is enabled by three simple but crucial training strategies to stabilize the training and to avoid overfitting DAFormer to the source domain: While the Rare Class Sampling on the source domain improves the quality of pseudo-labels by mitigating the confirmation bias of self-training towards common classes, the Thing-Class ImageNet Feature Distance and a learning rate warmup promote feature transfer from ImageNet pretraining. DAFormer significantly improves the state-of-the-art performance by 10.8 mIoU for GTA->Cityscapes and 5.4 mIoU for Synthia->Cityscapes and enables learning even difficult classes such as train, bus, and truck well.
    Read More

  • Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather

    Authors: Martin Hahner , Christos Sakaridis, Dengxin Dai and Luc Van Gool

    International Conference on Computer Vision, 2021

    This work presents a novel method for LiDAR-based 3D object detection in foggy weather by simulating foggy effects into standard LiDAR data..

    This work addresses the challenging task of LiDAR-based 3D object detection in foggy weather. Collecting and annotating data in such a scenario is very time, labor and cost intensive. In this paper, we tackle this problem by simulating physically accurate fog into clear-weather scenes, so that the abundant existing real datasets captured in clear weather can be repurposed for our task. Our contributions are twofold: 1) We develop a physically valid fog simulation method that is applicable to any LiDAR dataset. This unleashes the acquisition of large-scale foggy training data at no extra cost. These partially synthetic data can be used to improve the robustness of several perception methods, such as 3D object detection and tracking or simultaneous localization and mapping, on real foggy data. 2) Through extensive experiments with several state-of-the-art detection approaches, we show that our fog simulation can be leveraged to significantly improve the performance for 3D object detection in the presence of fog. Thus, we are the first to provide strong 3D object detection baselines on the Seeing Through Fog dataset. Our code is available at this http URL.
    Read More

  • ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding

    Authors: Christos Sakaridis, Dengxin Dai, and Luc Van Gool

    International Conference on Computer Vision, 2021

    ACDC is a large-scale dataset for training and testing semantic segmentation methods for four adverse visual conditions: fog, nighttime, rain, and snow.

    Level 5 autonomy for self-driving cars requires a robust visual perception system that can parse input images under any visual condition. However, existing semantic segmentation datasets are either dominated by images captured under normal conditions or are small in scale. To address this, we introduce ACDC, the Adverse Conditions Dataset with Correspondences for training and testing semantic segmentation methods on adverse visual conditions. ACDC consists of a large set of 4006 images which are equally distributed between four common adverse conditions: fog, nighttime, rain, and snow. Each adverse-condition image comes with a high-quality fine pixel-level semantic annotation, a corresponding image of the same scene taken under normal conditions, and a binary mask that distinguishes between intra-image regions of clear and uncertain semantic content. Thus, ACDC supports both standard semantic segmentation and the newly introduced uncertainty-aware semantic segmentation. A detailed empirical study demonstrates the challenges that the adverse domains of ACDC pose to state-of-the-art supervised and unsupervised approaches and indicates the value of our dataset in steering future progress in the field. Our dataset and benchmark are publicly available.
    Read More
Go to Top