The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models
2405.04623
0
0
Abstract
Scale the model, scale the data, scale the GPU farms is the reigning sentiment in the world of generative AI today. While model scaling has been extensively studied, data scaling and its downstream impacts on model performance remain under-explored. This is particularly important in the context of multimodal datasets whose main source is the World Wide Web, condensed and packaged as the Common Crawl dump, which is known to exhibit numerous drawbacks. In this paper, we evaluate the downstream impact of dataset scaling on 14 visio-linguistic models (VLMs) trained on the LAION400-M and LAION-2B datasets by measuring racial and gender bias using the Chicago Face Dataset (CFD) as the probe. Our results show that as the training data increased, the probability of a pre-trained CLIP model misclassifying human images as offensive non-human classes such as chimpanzee, gorilla, and orangutan decreased, but misclassifying the same images as human offensive classes such as criminal increased. Furthermore, of the 14 Vision Transformer-based VLMs we evaluated, the probability of predicting an image of a Black man and a Latino man as criminal increases by 65% and 69%, respectively, when the dataset is scaled from 400M to 2B samples for the larger ViT-L models. Conversely, for the smaller base ViT-B models, the probability of predicting an image of a Black man and a Latino man as criminal decreases by 20% and 47%, respectively, when the dataset is scaled from 400M to 2B samples. We ground the model audit results in a qualitative and historical analysis, reflect on our findings and their implications for dataset curation practice, and close with a summary of mitigation mechanisms and ways forward. Content warning: This article contains racially dehumanising and offensive descriptions.
Create account to get full access
Overview
- This paper evaluates how scaling up multimodal AI models, such as CLIP, can lead to concerning racial biases and classification capabilities.
- The researchers audit the racial classification capabilities of several large-scale multimodal models, finding that they can accurately classify race from visual inputs with high accuracy.
- This raises significant ethical concerns, as these models could be used to perpetuate harmful racial stereotypes and discrimination.
- The paper calls for more rigorous auditing and mitigation of such biases as these models continue to grow in scale and deployment.
Plain English Explanation
The paper looks at a concerning side-effect of making AI models that can handle both images and text, known as "multimodal" models. As these models have gotten larger and more powerful, the researchers found that they have developed the ability to accurately classify a person's race just from looking at an image of them.
This is problematic because it means these models could potentially be used to perpetuate harmful racial stereotypes and discrimination. For example, a model could be used to automatically identify someone's race in a surveillance system, or to make biased assumptions about a person's characteristics or abilities based on their appearance.
The researchers audited several large multimodal models, including CLIP, and found that they were alarmingly good at classifying race from visual inputs alone. This suggests that as these models continue to grow in scale, the risk of them being used in unethical or harmful ways also increases.
The paper calls for more thorough evaluation and mitigation of these biases moving forward, to ensure that the benefits of advanced AI don't come at the cost of enabling new forms of racial discrimination and prejudice. It highlights the importance of carefully auditing these models and taking proactive steps to address any problematic capabilities that emerge.
Technical Explanation
The researchers conducted a series of audits on several large-scale multimodal AI models, including CLIP, to evaluate their racial classification capabilities. They found that these models were able to accurately classify the race of individuals from visual inputs alone, with high accuracy rates.
This is concerning, as it suggests that as these models continue to grow in scale and capabilities, they may develop the ability to perpetuate harmful racial stereotypes and enable new forms of biased and discriminatory applications. The researchers argue that the ability to automatically identify someone's race from an image raises significant ethical issues, as it could be used to make unfair assumptions or decisions about individuals based on their appearance.
The paper also examines the potential scalability of these racial classification capabilities, and the risk that they could be amplified as the models grow in scale and sophistication. This could lead to the development of biased and problematic applications that discriminate against individuals based on their race.
Critical Analysis
The paper raises important concerns about the potential for large-scale multimodal models to develop harmful racial classification capabilities. The researchers provide a thorough audit of several models, demonstrating the troubling accuracy with which they can identify an individual's race from visual inputs alone.
However, the paper could have gone further in exploring potential mitigations or solutions to this issue. While it calls for more rigorous auditing and mitigation efforts, it does not delve into specific techniques or approaches that could be employed to address these biases. Additionally, the paper does not address the role of the underlying training data and how data curation and filtering might contribute to the development of these problematic capabilities.
Further research is needed to understand the root causes of these biases and to develop effective strategies for auditing and mitigating them in large-scale multimodal models. The paper's findings underscore the critical importance of proactively addressing these issues to ensure that the development of advanced AI technology does not come at the cost of perpetuating or amplifying harmful racial biases and discrimination.
Conclusion
This paper provides a concerning look at how the scaling of multimodal AI models can lead to the development of troubling racial classification capabilities. As these models grow in scale and sophistication, they are becoming increasingly adept at accurately identifying an individual's race from visual inputs alone, which raises significant ethical concerns.
The researchers call for more rigorous auditing and mitigation efforts to address these biases, as the potential for these models to be used in harmful or discriminatory applications is substantial. The findings emphasize the critical importance of proactively addressing these issues to ensure that the benefits of advanced AI technology are not outweighed by the perpetuation of harmful racial stereotypes and discrimination.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic
Sachin Goyal, Pratyush Maini, Zachary C. Lipton, Aditi Raghunathan, J. Zico Kolter
0
0
Vision-language models (VLMs) are trained for thousands of GPU hours on carefully curated web datasets. In recent times, data curation has gained prominence with several works developing strategies to retain 'high-quality' subsets of 'raw' scraped data. For instance, the LAION public dataset retained only 10% of the total crawled data. However, these strategies are typically developed agnostic of the available compute for training. In this paper, we first demonstrate that making filtering decisions independent of training compute is often suboptimal: the limited high-quality data rapidly loses its utility when repeated, eventually requiring the inclusion of 'unseen' but 'lower-quality' data. To address this quality-quantity tradeoff ($texttt{QQT}$), we introduce neural scaling laws that account for the non-homogeneous nature of web data, an angle ignored in existing literature. Our scaling laws (i) characterize the $textit{differing}$ 'utility' of various quality subsets of web data; (ii) account for how utility diminishes for a data point at its 'nth' repetition; and (iii) formulate the mutual interaction of various data pools when combined, enabling the estimation of model performance on a combination of multiple data pools without ever jointly training on them. Our key message is that data curation $textit{cannot}$ be agnostic of the total compute that a model will be trained for. Our scaling laws allow us to curate the best possible pool for achieving top performance on Datacomp at various compute budgets, carving out a pareto-frontier for data curation. Code is available at https://github.com/locuslab/scaling_laws_data_filtering.
4/11/2024
A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models
Ashutosh Sathe, Prachi Jain, Sunayana Sitaram
0
0
Vision-language models (VLMs) have gained widespread adoption in both industry and academia. In this study, we propose a unified framework for systematically evaluating gender, race, and age biases in VLMs with respect to professions. Our evaluation encompasses all supported inference modes of the recent VLMs, including image-to-text, text-to-text, text-to-image, and image-to-image. Additionally, we propose an automated pipeline to generate high-quality synthetic datasets that intentionally conceal gender, race, and age information across different professional domains, both in generated text and images. The dataset includes action-based descriptions of each profession and serves as a benchmark for evaluating societal biases in vision-language models (VLMs). In our comparative analysis of widely used VLMs, we have identified that varying input-output modalities lead to discernible differences in bias magnitudes and directions. Additionally, we find that VLM models exhibit distinct biases across different bias attributes we investigated. We hope our work will help guide future progress in improving VLMs to learn socially unbiased representations. We will release our data and code.
6/18/2024
📉
No Filter: Cultural and Socioeconomic Diversityin Contrastive Vision-Language Models
Ang'eline Pouget, Lucas Beyer, Emanuele Bugliarello, Xiao Wang, Andreas Peter Steiner, Xiaohua Zhai, Ibrahim Alabdulmohsin
0
0
We study cultural and socioeconomic diversity in contrastive vision-language models (VLMs). Using a broad range of benchmark datasets and evaluation metrics, we bring to attention several important findings. First, the common filtering of training data to English image-text pairs disadvantages communities of lower socioeconomic status and negatively impacts cultural understanding. Notably, this performance gap is not captured by - and even at odds with - the currently popular evaluation metrics derived from the Western-centric ImageNet and COCO datasets. Second, pretraining with global, unfiltered data before fine-tuning on English content can improve cultural understanding without sacrificing performance on said popular benchmarks. Third, we introduce the task of geo-localization as a novel evaluation metric to assess cultural diversity in VLMs. Our work underscores the value of using diverse data to create more inclusive multimodal systems and lays the groundwork for developing VLMs that better represent global perspectives.
5/27/2024
👁️
The Bias of Harmful Label Associations in Vision-Language Models
Caner Hazirbas, Alicia Sun, Yonathan Efroni, Mark Ibrahim
0
0
Despite the remarkable performance of foundation vision-language models, the shared representation space for text and vision can also encode harmful label associations detrimental to fairness. While prior work has uncovered bias in vision-language models' (VLMs) classification performance across geography, work has been limited along the important axis of harmful label associations due to a lack of rich, labeled data. In this work, we investigate harmful label associations in the recently released Casual Conversations datasets containing more than 70,000 videos. We study bias in the frequency of harmful label associations across self-provided labels for age, gender, apparent skin tone, and physical adornments across several leading VLMs. We find that VLMs are $4-7$x more likely to harmfully classify individuals with darker skin tones. We also find scaling transformer encoder model size leads to higher confidence in harmful predictions. Finally, we find improvements on standard vision tasks across VLMs does not address disparities in harmful label associations.
4/17/2024