CapsFusion: Rethinking Image-Text Data at Scale
2310.20550
0
0
📊
Abstract
Large multimodal models demonstrate remarkable generalist ability to perform diverse multimodal tasks in a zero-shot manner. Large-scale web-based image-text pairs contribute fundamentally to this success, but suffer from excessive noise. Recent studies use alternative captions synthesized by captioning models and have achieved notable benchmark performance. However, our experiments reveal significant Scalability Deficiency and World Knowledge Loss issues in models trained with synthetic captions, which have been largely obscured by their initial benchmark success. Upon closer examination, we identify the root cause as the overly-simplified language structure and lack of knowledge details in existing synthetic captions. To provide higher-quality and more scalable multimodal pretraining data, we propose CapsFusion, an advanced framework that leverages large language models to consolidate and refine information from both web-based image-text pairs and synthetic captions. Extensive experiments show that CapsFusion captions exhibit remarkable all-round superiority over existing captions in terms of model performance (e.g., 18.8 and 18.3 improvements in CIDEr score on COCO and NoCaps), sample efficiency (requiring 11-16 times less computation than baselines), world knowledge depth, and scalability. These effectiveness, efficiency and scalability advantages position CapsFusion as a promising candidate for future scaling of LMM training.
Get summaries of the top AI research delivered straight to your inbox:
Overview
- Large multimodal models have the ability to perform diverse multimodal tasks in a zero-shot manner.
- This success is largely due to the use of large-scale web-based image-text pairs, but these pairs can be noisy.
- Recent studies have used alternatively synthesized captions from captioning models, achieving notable benchmark performance.
- However, the paper reveals significant issues with scalability and world knowledge loss in models trained with these synthetic captions.
Plain English Explanation
Large AI models that can handle different types of data, like images and text, have shown remarkable versatility. They can perform a wide variety of tasks without needing to be specifically trained for each one. This is thanks in large part to the use of massive datasets of image-text pairs from the internet.
However, these web-based image-text pairs often contain a lot of noise and inaccuracies. To address this, recent research has explored using captions generated by AI captioning models instead. This has led to some impressive results on benchmark tests.
But the paper shows that models trained on these synthetic captions face significant challenges. They struggle to scale up and end up losing important real-world knowledge. The researchers trace these issues back to the simplified language structure and lack of detailed knowledge in the existing synthetic captions.
To provide higher-quality and more scalable multimodal training data, the researchers propose a new framework called CapsFusion. This leverages large language models to consolidate and refine information from both web-based image-text pairs and synthetic captions.
Technical Explanation
The paper examines the use of synthetic captions generated by AI models as an alternative to noisy web-based image-text pairs for training large multimodal language models.
The researchers find that while models trained on these synthetic captions can achieve notable benchmark performance, they suffer from significant scalability deficiencies and losses in world knowledge. They trace these issues to the simplified language structure and lack of detailed knowledge in existing synthetic captions.
To address these problems, the researchers propose CapsFusion, a framework that leverages large language models to consolidate and refine information from both web-based image-text pairs and synthetic captions. Extensive experiments show that CapsFusion captions exhibit significant improvements over existing captions in terms of model performance, sample efficiency, world knowledge depth, and scalability.
Critical Analysis
The paper provides a thorough analysis of the limitations of existing synthetic caption approaches, which is an important contribution to the field. The researchers raise valid concerns about the scalability and world knowledge issues that have been overlooked in prior research focused solely on benchmark performance.
However, the paper does not delve deeply into the specific architectural choices or training procedures of the CapsFusion framework. More details on these aspects would be helpful for researchers looking to build upon this work.
Additionally, the paper focuses on the use of synthetic captions for multimodal pretraining, but does not explore other potential approaches, such as semantic augmentation of images using language or harnessing the power of large vision-language models. Examining these alternatives could provide a more comprehensive understanding of the design space for improving multimodal training data.
Conclusion
This paper presents a critical analysis of the limitations of existing synthetic caption approaches for training large multimodal language models. It introduces the CapsFusion framework as a promising solution to address the scalability and world knowledge issues identified.
The demonstrated effectiveness, efficiency, and scalability advantages of CapsFusion captions suggest that this approach could be a valuable contribution to the ongoing efforts to scale up multimodal AI systems and unlock their full potential.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
👁️
Retrieval Enhanced Zero-Shot Video Captioning
Yunchuan Ma, Laiyun Qing, Guorong Li, Yuankai Qi, Quan Z. Sheng, Qingming Huang
0
0
Despite the significant progress of fully-supervised video captioning, zero-shot methods remain much less explored. In this paper, we propose to take advantage of existing pre-trained large-scale vision and language models to directly generate captions with test time adaptation. Specifically, we bridge video and text using three key models: a general video understanding model XCLIP, a general image understanding model CLIP, and a text generation model GPT-2, due to their source-code availability. The main challenge is how to enable the text generation model to be sufficiently aware of the content in a given video so as to generate corresponding captions. To address this problem, we propose using learnable tokens as a communication medium between frozen GPT-2 and frozen XCLIP as well as frozen CLIP. Differing from the conventional way to train these tokens with training data, we update these tokens with pseudo-targets of the inference data under several carefully crafted loss functions which enable the tokens to absorb video information catered for GPT-2. This procedure can be done in just a few iterations (we use 16 iterations in the experiments) and does not require ground truth data. Extensive experimental results on three widely used datasets, MSR-VTT, MSVD, and VATEX, show 4% to 20% improvements in terms of the main metric CIDEr compared to the existing state-of-the-art methods.
5/14/2024
🌀
Data-Efficient Multimodal Fusion on a Single GPU
Noel Vouitsis, Zhaoyan Liu, Satya Krishna Gorti, Valentin Villecroze, Jesse C. Cresswell, Guangwei Yu, Gabriel Loaiza-Ganem, Maksims Volkovs
0
0
The goal of multimodal alignment is to learn a single latent space that is shared between multimodal inputs. The most powerful models in this space have been trained using massive datasets of paired inputs and large-scale computational resources, making them prohibitively expensive to train in many practical scenarios. We surmise that existing unimodal encoders pre-trained on large amounts of unimodal data should provide an effective bootstrap to create multimodal models from unimodal ones at much lower costs. We therefore propose FuseMix, a multimodal augmentation scheme that operates on the latent spaces of arbitrary pre-trained unimodal encoders. Using FuseMix for multimodal alignment, we achieve competitive performance -- and in certain cases outperform state-of-the art methods -- in both image-text and audio-text retrieval, with orders of magnitude less compute and data: for example, we outperform CLIP on the Flickr30K text-to-image retrieval task with $sim ! 600times$ fewer GPU days and $sim ! 80times$ fewer image-text pairs. Additionally, we show how our method can be applied to convert pre-trained text-to-image generative models into audio-to-image ones. Code is available at: https://github.com/layer6ai-labs/fusemix.
4/11/2024
MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Models
Nithin Gopalakrishnan Nair, Jeya Maria Jose Valanarasu, Vishal M Patel
0
0
Large diffusion-based Text-to-Image (T2I) models have shown impressive generative powers for text-to-image generation as well as spatially conditioned image generation. For most applications, we can train the model end-toend with paired data to obtain photorealistic generation quality. However, to add an additional task, one often needs to retrain the model from scratch using paired data across all modalities to retain good generation performance. In this paper, we tackle this issue and propose a novel strategy to scale a generative model across new tasks with minimal compute. During our experiments, we discovered that the variance maps of intermediate feature maps of diffusion models capture the intensity of conditioning. Utilizing this prior information, we propose MaxFusion, an efficient strategy to scale up text-to-image generation models to accommodate new modality conditions. Specifically, we combine aligned features of multiple models, hence bringing a compositional effect. Our fusion strategy can be integrated into off-the-shelf models to enhance their generative prowess.
4/16/2024
OmniFusion Technical Report
Elizaveta Goncharova, Anton Razzhigaev, Matvey Mikhalchuk, Maxim Kurkin, Irina Abdullaeva, Matvey Skripkin, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov
0
0
Last year, multimodal architectures served up a revolution in AI-based approaches and solutions, extending the capabilities of large language models (LLM). We propose an textit{OmniFusion} model based on a pretrained LLM and adapters for visual modality. We evaluated and compared several architecture design principles for better text and visual data coupling: MLP and transformer adapters, various CLIP ViT-based encoders (SigLIP, InternVIT, etc.), and their fusing approach, image encoding method (whole image or tiles encoding) and two 7B LLMs (the proprietary one and open-source Mistral). Experiments on 8 visual-language benchmarks show the top score for the best OmniFusion setup in terms of different VQA tasks in comparison with open-source LLaVA-like solutions: VizWiz, Pope, MM-Vet, ScienceQA, MMBench, TextVQA, VQAv2, MMMU. We also propose a variety of situations, where OmniFusion provides highly-detailed answers in different domains: housekeeping, sightseeing, culture, medicine, handwritten and scanned equations recognition, etc. Mistral-based OmniFusion model is an open-source solution with weights, training and inference scripts available at https://github.com/AIRI-Institute/OmniFusion.
4/10/2024