SR-DARTS: Robust Differentiable Neural Architecture Search with Stability Regularization

Abstract

Neural Architecture Search (NAS) has emerged as a powerful tool to find the best neural architectures with minimal manual intervention. However, existing NAS approaches suffer from poor robustness, because the search process is sensitive to the noise and hyperparameter settings. To this end, we propose a robust Differentiable Architecture Search with Stability Regularization (SR-DARTS for short). SR-DARTS introduces a new stability regularization term into the validation loss function to explicitly constrain the architecture parameter. Specifically, the new regularizer drives the mean of the architecture parameters towards zero during the search process, thereby accelerating the convergence of the search algorithm. Consequently, SR-DARTS can reduce the variance error during search. Experimental results demonstrate that SR-DARTS can discover accurate neural network architectures and obtain better performance on benchmark datasets.