Deep learning models in image synthesis have proven their applicability in various image translation areas. However, although the synthesized image may reflect the user’s intention, some of its properties may be different from those of real images. In this study, we introduce an undesirable property that we discovered in the multi-domain label-based image translation techniques: Once the image is translated to one domain, the translated image cannot be adequately translated again to another domain. We refer to this problem as the failure of recursive translation, and analyze this phenomenon from the viewpoint of attribute disentanglement and establish a hypothesis: Unlabeled or unknown attributes that are correlated with the direction of translation hinder the network from learning the correct direction of translation. Based on our hypothesis, we also devise a solution that endows the generator with the power of recursive translation, which is achieved by injecting additive perturbations during model training. Our method is simple and easy to implement on various translation models without requiring much hyperparameter adjustment. Beyond enabling recursive translation, it is worth noting that solving the recursive translation problem improves the disentanglement of single translations, which eventually strengthens its practicability.