Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving

Introduction

The paper introduces EM-VLM4AD, an efficient and lightweight vision-language model designed for autonomous driving applications. Vision-Language Models (VLMs) have shown promise in solving tasks at the intersection of vision and language, making them useful for autonomous driving. However, current research in this area predominantly uses large models with over one billion parameters, which require expensive hardware and longer inference times. EM-VLM4AD addresses this issue by developing a smaller model that consumes at least 10x less memory and floating point operations (FLOPs) compared to current AD-VLMs. The model is trained on the DriveLM dataset and can respond to questions conditioned on multiple frames. The authors explore two lightweight language model backbones for EM-VLM4AD: a finetuned T5-Base and an 8-bit quantized T5-Large finetuned using LoRA. They compare the model's efficiency and performance on various metrics to the DriveLM dataset baseline, demonstrating stronger performance with superior efficiency using a much smaller model.

Related Research

The paper discusses the development of Vision-Language Models (VLMs) that combine natural language processing and computer vision techniques. These models encode images and text into a combined latent representation and learn correlations between them through cross-modal pre-training tasks. VLMs like CLIP, BLIP-2, and VL-T5 have shown strong performance across various tasks but are computationally expensive to deploy for real-time applications.

Researchers have begun integrating multimodal data to train language models for autonomous driving tasks. Models like DriveGPT4, DriveMLM, and GPT-Driver leverage large language models and visual encoders to generate answers and control signals for driving scenarios. However, these models use billion-parameter LLMs and expensive image encoders, making them unsuitable for real-time inference.

Processing multiple modalities and text is a challenge for most VLMs, which are pre-trained for single-image single-text problems. Techniques like pooling visual encodings of video frames, using QFormer to embed video and LiDAR data, and employing gated attention pooling across individual image embeddings have been explored to consolidate multiple modalities into a single embedding that can be used by a VLM.

The paper introduces a model that responds to multi-view image input and question prompts for autonomous driving tasks. It uses a frozen T5 language model during the first stage of training, allowing the image embedding network to align with the T5 embeddings. The model demonstrates the ability to perform visual question answering for perception, planning, and traffic agent behavior prediction tasks.

Methods

The paper introduces EM-VLM4AD, a model for Visual Question Answering in Autonomous Driving. It consists of a custom image embedding network and a pre-trained T5 language model.

The image embedding network uses a ViT patch projection embedding scheme to create latent image representations. These individual image embeddings from multiple views are combined using gated pooling attention to produce a single embedding. This is then concatenated with a text embedding and input into the language model to generate answer text.

For the language model, the authors use pre-trained T5 models (T5-Base and 8-bit quantized T5-Large) to reduce computational costs. They fine-tune these models to adapt them to the concatenated multi-view image and text embeddings.

EM-VLM4AD is trained on the DriveLM dataset using a two-stage approach. In the first stage, the image patch encoder and language model are frozen, and only the gated pooling attention and projection layer are trained. In the second stage, only the image patch encoder remains frozen while the language model is fine-tuned.

The models are trained for six epochs each, using specific hardware and hyperparameters. EM-VLM4AD outperforms the DriveLM-Agent baseline in all evaluation metrics.

Experiments

The paper analyzes the quantitative, qualitative, and computational performance of EM-VLM4AD using common image captioning metrics such as BLEU-4, ROUGE-L, METEOR, and CIDEr. The model is compared to DriveLM-Agent, the only existing approach on the DriveLM dataset. Both versions of EM-VLM4AD outperform DriveLM-Agent on all metrics, despite having significantly fewer parameters. The T5-Base version of EM-VLM4AD is the top-performing model.

The computational analysis shows that EM-VLM4AD is more efficient than other multimodal LMs for autonomous driving, requiring fewer parameters, FLOPs, and memory. The T5-Base backbone has the least parameters and FLOP count, while the T5-Large backbone has the least memory requirements due to 8-bit quantization.

Qualitative results demonstrate EM-VLM4AD's ability to accurately respond to various questions related to perception, traffic agent behavior, planning, and scene understanding. The model leverages general knowledge from pretrained components and can understand the c-tag format used in DriveLM. However, it occasionally generates grammatically incorrect answers and struggles with behavior-related questions that may require temporal context from multiple frames.

Conclusion

EM-VLM4AD is a compact multi-frame vision-language model designed for Visual Question Answering in autonomous driving applications. It offers advantages in memory efficiency and computational requirements compared to other language models tailored for autonomous driving. EM-VLM4AD outperforms DriveLM-Agent in BLEU-4, METEOR, ROUGE, and CIDEr metrics on a DriveLM test dataset. The model effectively answers various autonomous driving questions and focuses on relevant camera views using a gated pooling attention layer that integrates view embeddings.

Future research aims to develop EM-VLM4AD into a video-language model that can generate responses from multi-view video inputs, enhancing its capability to handle temporal-related questions. Incorporating multimodal retrieval augmented generation will enable the model to extract insights from similar traffic scenarios, providing additional context for improved performance.