A2R2: Robust Unsupervised Neural Machine Translation With Adversarial Attack and Regularization on Representations

Unsupervised neural machine translation (UNMT) has recently achieved significant progress without requirement on any parallel data. The models for UNMT are typically the sequence-to-sequence architecture with an encoder to map sentences in different languages to a shared latent space, and a decoder to generate their corresponding translation. Denoising autoencoding and back-translation are called in every iteration for the models to learn the relationship of sentence pairs in languages or between languages. However, sentences generated by the noise model of autoencoding or the reverse model of back-translation are normally different from those written by humans, which may cause inference bias. In this paper, we propose a regularization method for back-translation to explicitly draw representations of sentence pairs closer in the shared space. To enhance the robustness to sentences after autoencoding or back-translation, the adversarial attack on representations is applied. Experiments on unsupervised English<inline-formula> <tex-math notation="LaTeX">$\leftrightarrow $ </tex-math></inline-formula>French, English<inline-formula> <tex-math notation="LaTeX">$\leftrightarrow $ </tex-math></inline-formula>German and English<inline-formula> <tex-math notation="LaTeX">$\leftrightarrow $ </tex-math></inline-formula>Romanian benchmarks show that our approach outperforms the cross-lingual language model (XLM) baseline by <inline-formula> <tex-math notation="LaTeX">$0.4\sim 1.8$ </tex-math></inline-formula> BLEU scores. Additionally, the boost on noisy test sets in most translation directions is over 5 BLEU scores.

Abstract licence: CC BY-NC-ND

1. 20.

   Attention Is All You Need · 2017

39 more references, whose wording the paper’s licence keeps private.

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.