Investigating the translation capabilities of Large Language Models trained on parallel data only

Overview

• Large Language Models (LLMs) have excelled at a variety of Natural Language Processing (NLP) tasks, including Machine Translation.
• Previous methods relied on iterative processes like instruction fine-tuning or continual pre-training, leaving the challenges of training LLMs solely on parallel data unexplored.
• This work introduces PLUME (Parallel Language Model), a collection of three 2B LLMs with varying vocabulary sizes, trained exclusively on Catalan-centric parallel examples.
• These models perform comparably to previous encoder-decoder architectures on both supervised and zero-shot translation tasks.
• The researchers conduct a thorough investigation into the translation capabilities of LLMs, examining their performance, the impact of prompt elements, and their cross-lingual representation space.

Plain English Explanation

Large language models have become incredibly skilled at a wide range of language-related tasks, including translating between languages. However, previous approaches to training these models for translation often involved complex, iterative processes like fine-tuning or continual pre-training.

In this new research, the authors introduce a set of large language models called PLUME, which were trained solely on parallel data - text that has been translated into multiple languages. These PLUME models, ranging in vocabulary size from 32,000 to 256,000 words, were able to perform just as well as previous specialized translation models on a variety of translation tasks, both when translating between languages they were trained on and when trying to translate between languages they weren't explicitly trained on.

By studying these PLUME models in-depth, the researchers were able to gain insights into how large language models handle translation and multilingual capabilities. They looked at factors like the specific prompts used, and how the models represent linguistic concepts across different languages.

Technical Explanation

The researchers introduce PLUME (Parallel Language Model), a collection of three 2 billion parameter LLMs with vocabulary sizes of 32k, 128k, and 256k. These models were trained exclusively on parallel data - text that has been translated between Catalan and other languages.

Despite this specialized training approach, the PLUME models perform comparably to previous encoder-decoder architectures on 16 supervised translation directions and 56 zero-shot ones. The researchers conduct a thorough investigation into the translation capabilities of these LLMs, probing their performance, the impact of prompt elements, and their cross-lingual representation space.

This work represents a novel paradigm for boosting the translation capabilities of large language models, moving away from the previously dominant iterative fine-tuning or continual pre-training approaches.

Critical Analysis

The paper provides a thorough exploration of training large language models solely on parallel data for translation tasks. While the results demonstrate the impressive capabilities of this approach, there are a few caveats to consider:

• The models were trained on a relatively narrow set of Catalan-centric language pairs, so their performance may not generalize as well to more diverse language combinations.
• The researchers note that the models' cross-lingual representation space could be further improved, suggesting opportunities for continued research in this area.
• As with many large language models, there are open questions around the interpretability and potential biases present in the PLUME models.

Overall, this work represents an interesting and promising direction for enhancing the translation abilities of large language models. However, further research is needed to fully understand the strengths, limitations, and broader implications of this approach.

Conclusion

This research introduces a novel paradigm for training large language models solely on parallel data for machine translation tasks. The resulting PLUME models demonstrate strong performance on both supervised and zero-shot translation, rivaling previous specialized architectures.

By conducting a detailed investigation into the translation capabilities of these models, the researchers have provided valuable insights into how large language models handle multilingualism and cross-lingual representation. This work opens up new avenues for further research and development in the field of machine translation, potentially leading to more efficient and versatile translation systems in the future.