From Human Judgements to Predictive Models: Unravelling Acceptability in Code-Mixed Sentences

Overview

• The paper proposes a dataset called Cline, which contains human acceptability judgments for English-Hindi code-mixed text, to help model the naturalness of code-mixed sentences.
• The dataset is the largest of its kind, with 16,642 sentences, and includes samples from synthetic code-mixed text and online social media.
• The paper finds that popular code-mixing metrics, such as CMI and Number of Switch Points, have low correlation with human acceptability judgments, highlighting the need for the Cline dataset.
• Experiments show that fine-tuned Multilingual Large Language Models (MLLMs) like XLM-Roberta and Bernice outperform simpler models and even ChatGPT in code-mixed acceptability tasks.
• The paper also demonstrates the ability to transfer the model's zero-shot learning from English-Hindi to English-Telugu, suggesting applicability to other code-mixed language pairs.

Plain English Explanation

Code-mixing is the practice of combining words or phrases from different languages within a single sentence or conversation. Computational approaches for analyzing or generating code-mixed sentences often rely on training data that reflects the distribution of acceptable code-mixed sentences, but they do not explicitly model the naturalness or acceptability of these sentences.

The researchers in this paper aimed to address this by constructing a dataset called Cline, which contains human judgments on the acceptability of English-Hindi code-mixed text. The dataset includes both synthetically generated code-mixed sentences and samples collected from online social media, making it the largest of its kind.

The researchers found that popular code-mixing metrics, such as the Code-Mixing Index (CMI) and the Number of Switch Points, have a low correlation with human acceptability judgments. This underscores the need for a dataset like Cline, which can help distinguish natural code-mixed text and enable quality-controlled generation of such text.

To demonstrate the usefulness of the Cline dataset, the researchers conducted experiments using various machine learning models. They found that fine-tuned Multilingual Large Language Models (MLLMs) like XLM-Roberta and Bernice outperformed simpler models and even the popular language model ChatGPT in code-mixed acceptability tasks.

Additionally, the researchers showed that their model's zero-shot learning, trained on the English-Hindi data, could be transferred to the English-Telugu language pair, suggesting the potential for applying this approach to other code-mixed language pairs.

Technical Explanation

The researchers constructed the Cline dataset, which contains 16,642 sentences with human acceptability judgments for English-Hindi code-mixed text. The dataset includes samples from two sources: synthetically generated code-mixed text and samples collected from online social media.

To evaluate the usefulness of the Cline dataset, the researchers conducted experiments using various machine learning models. They found that simple Multilayer Perceptron (MLP) models trained solely on code-mixing metrics were outperformed by fine-tuned pre-trained Multilingual Large Language Models (MLLMs) like XLM-Roberta and Bernice.

The researchers also compared the performance of their fine-tuned MLLMs with the zero-shot and few-shot capabilities of ChatGPT, a popular language model. The results showed that the MLLMs fine-tuned on the larger Cline dataset outperformed ChatGPT, suggesting opportunities for improving code-mixed tasks using these models.

Additionally, the researchers demonstrated the ability to transfer the model's zero-shot learning from English-Hindi to English-Telugu acceptability judgments, which outperformed random baselines. This suggests the potential for applying the Cline dataset and the researchers' approach to other code-mixed language pairs.

Critical Analysis

The researchers acknowledge several limitations and areas for further research in their paper. For example, they note that the Cline dataset is limited to English-Hindi code-mixed text and suggest that expanding the dataset to include other language pairs would be valuable.

Additionally, the researchers highlight the need for more comprehensive metrics to assess the naturalness and acceptability of code-mixed text beyond the traditional code-mixing metrics they investigated. Exploring alternative approaches to model human judgments of code-mixed text could be a fruitful area for future research.

While the researchers demonstrate the superiority of fine-tuned MLLMs over simpler models and ChatGPT, it would be interesting to see how these models perform on more diverse and challenging code-mixed tasks, such as generation or translation, in addition to the acceptability judgments explored in this paper.

Overall, the Cline dataset and the researchers' findings provide a valuable contribution to the field of code-mixed language processing, highlighting the importance of modeling human judgments and the potential of advanced language models in this domain.

Conclusion

This paper presents the Cline dataset, a large-scale collection of human acceptability judgments for English-Hindi code-mixed text. The researchers found that popular code-mixing metrics have limited correlation with human judgments, underscoring the need for datasets like Cline to better understand and model the naturalness of code-mixed language.

Experiments using the Cline dataset demonstrated the superior performance of fine-tuned Multilingual Large Language Models (MLLMs) over simpler models and even the popular ChatGPT language model in code-mixed acceptability tasks. The researchers also showcased the ability to transfer the model's zero-shot learning to other code-mixed language pairs, suggesting the broader applicability of their approach.

The Cline dataset and the researchers' findings offer valuable insights for the development of more reliable and accurate computational approaches to analyzing and generating code-mixed text. This work has the potential to contribute to various applications, from code-mixed language processing in social media to quality-controlled generation of code-mixed content.