Comparative Analysis of ChatGPT, GPT-4, and Microsoft Bing Chatbots for GRE Test

Overview

• This paper examines the performance of three AI chatbots - Bing, ChatGPT, and GPT-4 - in answering questions from the Graduate Record Examination (GRE).
• The study assessed the chatbots' capabilities across different question types, including quantitative reasoning and verbal categories, as well as their ability to handle image-based questions.
• The results show that GPT-4 outperformed the other chatbots, particularly in complex language understanding tasks and image-based questions.
• The paper highlights the ability of these chatbots to potentially score well on the GRE, which raises concerns about fairness in higher education admissions if the test is administered online.

Plain English Explanation

This research paper looked at how well three different AI chatbots - Bing, ChatGPT, and GPT-4 - performed when answering questions from a standardized test called the Graduate Record Examination (GRE). The GRE is a common exam that many graduate school applicants take.

The researchers gave the chatbots 137 math-related questions and 157 verbal questions from the GRE to see how they would do. They also tested the chatbots' abilities to answer questions that involved images.

The results showed that the most advanced chatbot, GPT-4, was the best at answering the GRE questions, especially the ones that required deeper understanding of language and analysis of images. The other chatbots, Bing and ChatGPT, also performed well but not as strongly as GPT-4.

These findings suggest that these AI chatbots are becoming so advanced that they could potentially score very high on the GRE. This raises concerns about fairness in the graduate school admissions process, especially if the GRE is taken online and students have access to these chatbots during the test. The researchers emphasize the importance of ensuring test-takers cannot use these chatbots when taking the GRE, to maintain a level playing field.

Technical Explanation

The paper used a total of 294 questions from the GRE, including 137 quantitative reasoning questions and 157 verbal questions, to assess the performance of Bing, ChatGPT, and GPT-4. The chatbots were tasked with answering these questions, and their responses were evaluated across different skills and question styles.

The researchers also explored the chatbots' proficiency in handling image-based questions, which are a common feature of the GRE. Additionally, the uncertainty levels of each chatbot were measured and reported.

The results showed that GPT-4 outperformed the other chatbots, particularly in complex language understanding tasks and image-based questions. This suggests that the latest advancements in large language models, such as those used in GPT-4, have significantly improved their ability to comprehend and reason with visual and textual information.

The findings highlight the potential for these chatbots to achieve high scores on the GRE, which could have implications for the fairness and integrity of the graduate school admissions process, especially if the test is administered online. The paper emphasizes the need to ensure that test-takers are segregated from these AI resources during the examination to maintain a level playing field.

Critical Analysis

The paper provides valuable insights into the current capabilities of AI chatbots in addressing standardized test questions, particularly the GRE. However, it is important to consider several caveats and limitations:

• The study focuses on a single standardized test, the GRE, and the findings may not necessarily generalize to other types of exams or assessment frameworks. Further research is needed to explore the chatbots' performance on a wider range of standardized tests.

• The paper does not delve into the specific strategies or approaches used by the chatbots to solve the GRE questions. Understanding these underlying mechanisms could provide deeper insights into the strengths and limitations of each chatbot.

• The study does not address the potential for these chatbots to be used for academic dishonesty or unfair advantage in the admissions process. Extensive discussions and policy considerations may be necessary to address these ethical concerns.

• The paper's findings highlight the rapid advancements in large language models, but it is essential to continue monitoring and evaluating these technologies as they evolve, to ensure their responsible and ethical deployment in educational contexts.

Conclusion

This research paper provides a compelling investigation into the capabilities of three prominent AI chatbots - Bing, ChatGPT, and GPT-4 - in answering questions from the Graduate Record Examination (GRE). The results indicate that the most advanced chatbot, GPT-4, is able to perform exceptionally well on the GRE, surpassing the other chatbots in both quantitative reasoning and verbal tasks, as well as in handling image-based questions.

These findings raise important considerations about the potential implications of these chatbots on the fairness and integrity of the graduate school admissions process, especially if the GRE is administered online. The paper emphasizes the need to ensure that test-takers are segregated from these AI resources during the examination to maintain a level playing field and prevent unfair advantages.

As AI technologies continue to advance, it will be crucial for educational institutions, policymakers, and researchers to closely monitor and address the ethical considerations surrounding the use of these chatbots in academic settings. Ongoing collaboration and thoughtful policy decisions will be essential to ensure that the admissions process remains fair and equitable for all applicants.