Lisbon Computational Linguists at SemEval-2024 Task 2: Using A Mistral 7B Model and Data Augmentation

Overview

• This paper describes the approach taken by a team of computational linguists from Lisbon to participate in SemEval-2024 Task 2, which focuses on safe biomedical natural language processing.
• The team used a large language model called Mistral-7B and applied data augmentation techniques to improve the model's performance on the task.
• The paper provides details on the experiment design, the model architecture, and the key insights gained from the research.

Plain English Explanation

The researchers from Lisbon used a powerful language model called Mistral-7B to tackle a challenging task in natural language processing. The task was to develop systems that can safely and accurately process biomedical text, which is important for applications like medical diagnosis and drug discovery.

To improve the model's performance, the team applied data augmentation techniques. This involves creating new training data by making small, controlled changes to the existing data, such as paraphrasing sentences or replacing words with synonyms. This can help the model learn more robust and generalizable patterns from the data.

The researchers evaluated their approach on the benchmark dataset provided for the SemEval-2024 Task 2 competition. They found that their system, which used the Mistral-7B model and data augmentation, performed well compared to other competing systems.

Technical Explanation

The Lisbon team's approach involved fine-tuning the Mistral-7B model on the task-specific dataset provided for SemEval-2024 Task 2. Mistral-7B is a large language model that has been pre-trained on a vast amount of text data, allowing it to capture general patterns in natural language.

To further improve the model's performance, the researchers applied various data augmentation techniques. This included techniques like back-translation, where sentences are translated to another language and then back to the original language, and synonym replacement, where words in the text are replaced with synonyms.

The team evaluated their system on the SemEval-2024 Task 2 dataset, which contains biomedical text from sources like scientific papers and clinical notes. They measured the model's performance on various metrics, such as F1 score and accuracy.

Critical Analysis

The researchers acknowledged that their approach has some limitations. For example, the Mistral-7B model, while powerful, may still struggle with certain types of biomedical terminology or domain-specific language that is not well-represented in the pre-training data.

Additionally, the data augmentation techniques used, while helpful, may introduce some noise or unintended changes to the text that could impact the model's performance. The researchers suggested that further research is needed to explore more sophisticated data augmentation methods that can better preserve the semantic and syntactic properties of the original text.

It would also be interesting to see how the Lisbon team's approach compares to other state-of-the-art methods, such as ensemble models or hybrid architectures, which may offer additional performance improvements.

Conclusion

The Lisbon team's work demonstrates the power of large language models, like Mistral-7B, and the benefits of data augmentation techniques for improving performance on specialized natural language processing tasks, such as the safe processing of biomedical text.

Their approach provides a solid foundation for further research and development in this important area of natural language processing, with potential applications in fields like healthcare, drug discovery, and scientific research.