Continual Test-Time Domain Adaptation
Qin Wang, Olga Fink, Luc Van Gool, Dengxin Dai
Test-time domain adaptation aims to adapt a source pre-trained model to a target domain without using any source data. Existing works mainly consider the case where the target domain is static. However, real-world machine perception systems are running in non-stationary and continually changing environments where the target domain distribution can change over time. Existing methods, which are mostly based on self-training and entropy regularization, can suffer from these non-stationary environments. Due to the distribution shift over time in the target domain, pseudo-labels become unreliable. The noisy pseudo-labels can further lead to error accumulation and catastrophic forgetting. To tackle these issues, we propose a continual test-time adaptation approach (CoTTA) which comprises two parts. Firstly, we propose to reduce the error accumulation by using weight-averaged and augmentation-averaged predictions which are often more accurate. On the other hand, to avoid catastrophic forgetting, we propose to stochastically restore a small part of the neurons to the source pre-trained weights during each iteration to help preserve source knowledge in the long-term. The proposed method enables the long-term adaptation for all parameters in the network. CoTTA is easy to implement and can be readily incorporated in off-the-shelf pre-trained models. We demonstrate the effectiveness of our approach on four classification tasks and a segmentation task for continual test-time adaptation, on which we outperform existing methods. Our code is available at https://qin.ee/cotta.
Figure 1. We consider the online continual test-time adaptation scenario. The target data is provided in a sequence and from a continually changing environment. An off-the-shelf source pretrained network is used to initialize the target network. The model is updated online based on the current target data, and the predictions are given in an online fashion. The adaptation of the target network does not rely on any source data. Existing methods often suffer from error accumulation and forgetting which result in performance deterioration over time. Our method enables long-term test-time adaptation under continually changing environments.
Figure 2. An overview of the proposed continual test-time adaptation (CoTTA) approach. CoTTA adapts from an off-the-shelf source pre-trained network. Error accumulation is mitigated by using a teacher model to provide weight-averaged pseudo-labels and using multiple augmentations to average the predictions. Knowledge from the source data is preserved by stochastically restoring a small number of elements of trainable weights.
Table 1. Classification error rate (%) for the standard CIFAR10-to-CIFAR10C online continual test-time adaptation task. Tesults are evaluated on WideResNet-28 with the largest corruption severity level 5. * denotes the requirement on additional domain information.
Figure 3. Classification error rate (%) for the standard CIFAR100-to-CIFAR100C online continual test-time adaptation task. All results are evaluated on the ResNeXt-29 architecture with the largest corruption severity level 5.
Figure 4. Semantic segmentation results (mIoU in %) on the Cityscapes-to-ACDC online continual test-time adaptation task. We evaluate the four test conditions continually for ten times to evaluate the long-term adaptation performance. To save space, we only show the continual adaptation results in the first, fourth, seventh, and last round. Full results can be found in the supplementary material. All results are evaluated based on the Segformer-B5 architecture.