Sparse multimodal fusion with modal channel attention

Abstract

The study investigates the capability of masked multimodal transformer architectures to learn robust embedding spaces when modality samples are sparsely aligned. An extension called modal channel attention (MCA) incorporates modal-incomplete channels into the multihead attention mechanism of the masked multimodal transformer model. Two datasets with four modalities each, CMU-MOSEI for multimodal sentiment recognition and TCGA for multiomics, are utilized. The models demonstrate the ability to learn uniform and aligned embedding spaces with only two out of four modalities present in most samples. Even in the absence of modal sparsity, the proposed MCA mechanism enhances the quality of generated embedding spaces, recall metrics, and subsequent performance on downstream tasks.

Introduction

The paper discusses the increasing use of multimodal models in deep learning applications, which combine various data formats such as text, audio, images, and video. It highlights the challenges of training models with incomplete or partially aligned modalities, as datasets often contain missing or sparse modalities. The paper proposes an extension of the masked multimodal transformer model called modal channel attention (MCA), which incorporates modal-incomplete channels in the multihead attention block.

The authors demonstrate that MCA can learn high-quality embedding spaces even when modalities are sparsely aligned, up to a sparsity level of 40%, corresponding to an average of 2.4 modalities per sample. Compared to the previous masked multimodal attention (MMA) model, MCA generally improves the quality of generated embedding spaces, recall metrics, and performance on downstream tasks through linear probing.

The paper concludes by discussing potential applications and future research directions in multimodal learning with incomplete or sparsely aligned modalities.

Related Work

The paper discusses several approaches for multimodal representation learning, including the Everything At Once model, FLAVA, Zhang et al.'s model, Wei et al.'s model, LORRETA, and Zorro.

The Everything At Once model uses a fusion transformer with contrastive loss to generate embeddings for each modality and pair of modalities, which are then pooled and projected into a multimodal embedding space. However, this approach scales exponentially with the number of modalities, making it computationally inefficient.

FLAVA uses multiple loss models for data with missing modalities, allowing for unimodal and multimodal inference but not generating a multimodal fusion embedding space. Zhang et al.'s model projects unimodal encodings into a modality-aligned feature space and performs weight-shared dual attention prediction, improving predictions for unseen modalities.

Wei et al.'s model and LORRETA are trained to generate missing modalities but cannot directly embed higher-order modality combinations or produce a fused embedding space applicable to tasks like retrieval or linear probing.

Zorro is an MMA model that produces unimodal and multimodal outputs trained with a contrastive loss. A similar MMA architecture was applied by Shi et al. for image segmentation in a biomedical application, though the performance with sparsely aligned multimodal data was not directly explored.

Model

The model architecture consists of a transformer encoder with a learnable pooling layer and noise contrastive estimation (NCE) loss between pooled tokens. Trainable transformations are applied to the input data, such as linear transformations or MLPs, to encode tabular data or compress pretrained frozen embeddings. The attention model uses Multi-Modal Attention (MMA) with fusion token blocks attended by unimodal token blocks. Multi-channel attention is introduced, where blocks of fusion attention are attended to by a subset of modalities, maintaining efficiency regardless of the number of modalities. Each fusion channel corresponds to a different set of modalities, with a unique pooling token in the learnable pooling layer. NCE Loss is applied between each pair of unimodal and fusion channel tokens. The fusion channel attended to by all modalities is used for downstream tasks. The learnable pooling layer has a multimodal mask preventing fusion tokens from attending to unimodal tokens. The model uses 5 transformer encoder layers with a hidden size of 512 and 8 attention heads. The feed-forward layers use a feed-forward multiplier of 4 and the GeGLU activation function. There are 88 fusion tokens in both studied models, with the MCA transformer using 11 channels of 8 tokens each, pooled separately.

Methods

The paper discusses the datasets and training procedures used in the experiments. The CMU-MOSEI dataset contains 23,248 samples of aligned multimodal data (text, audio, and video) with a test split of 2,324 samples. Each modality is preprocessed and transformed into token embeddings using linear layers and positional embeddings.

The TCGA dataset consists of tabular data from multiple omics sources like gene expression and DNA methylation. After preprocessing, it has 7,017 multimodal samples with a test split of 707 samples. The tabular data is encoded using trainable MLPs and combined with column index embeddings.

The paper evaluates performance under modal sparsity, where some modalities are randomly dropped from samples during training. Sparsity levels of 0, 0.2, 0.4, 0.6, and 0.8 are used.

Training used 4 Nvidia GPUs, a batch size of 32, cosine learning rate scheduling, and varied epoch numbers for the two datasets. Figures show metrics like embedding space uniformity, alignment between modalities, median rank, and recall at different sparsity levels on the test sets.

Evaluation of Embeddings

The section analyzes the characteristics of embeddings produced by trained multimodal models, focusing on alignment and uniformity metrics. The Multiple Choice Alignment (MCA) model achieves significantly lower uniformity (indicating a more uniform embedding space) than the Multimodal Mixture-of-Experts (MMA) model, while its alignment is increased. As modal sparsity increases beyond 40%, both models' fusion uniformities worsen, indicating less uniform unimodal embedding spaces.

The recall metrics (median rank, R1, R5, and R10) demonstrate the ability of the model to use a unimodal embedding to recall a fused embedding. The MCA architecture shows improved average median rank and recall metrics over MMA for most modal sparsities examined in both datasets studied. The improvement is more significant in the smaller TCGA dataset at low modal sparsity and in the larger CMU-MOSEI dataset at high modal sparsity.

The results highlight the utility of using MCA as a method for retrieving embeddings containing information from multiple modalities, even when only a single modality is available at inference time and training examples contain sparsely aligned modalities.

This section evaluates the performance of the generated embeddings from the proposed models on two downstream tasks: cancer type classification on the TCGA dataset and sentiment analysis on the CMU-MOSEI dataset. Linear probing is used, where a linear layer is trained on the embeddings produced from the training data, and the metrics are calculated on the test data.

For the TCGA dataset, a multiclass cancer type classification task with 32 classes, the class-averaged area under the precision-recall curve (AUPR) is presented. Both MCA and MMA perform well, with MCA showing a slight improvement over MMA as modal sparsity increases up to 0.4.

For the CMU-MOSEI sentiment analysis task, which involves regression to a single value between 0 and 1 for negative and positive sentiment, the correlation coefficient is used as the metric. MCA meets the baseline result of 0.54 when no modal sparsity is present. As modal sparsity increases, the correlation coefficient decreases, and MCA and MMA results become similar, with a minor advantage for MCA.

In general, MCA provides improved results over MMA for both datasets.

Conclusion

The paper discusses masked multimodal transformer architectures, which can learn embedding spaces robust to missing modalities. It presents multichannel attention (MCA) as an extension of the masked multimodal transformer model, incorporating modal-incomplete channels in the multihead attention mechanism.

Using two well-known datasets with four modalities each, the paper demonstrates that both MMA (Masked Multimodal Attention) and MCA can learn high-quality embedding spaces up to a modal sparsity of 0.4, corresponding to 1-2 modalities most frequently missing per sample.

The results show that MCA generally improves the quality of embedding spaces, recall metrics, and subsequent performance on downstream tasks.

The paper suggests future research directions, including extending the architecture to autoencoding or joint embedding predictive architectures for multimodal transformers, exploring multimodal scaling laws with modal sparsity, and investigating foundational model scale datasets for multimodal representation learning with sparsely aligned datasets.

Acknowledgments

This section acknowledges contributions from individuals who provided helpful discussions during the study. These individuals are Clayton Rabideau, Gabrielle Griffin, Callum Birch-Sykes, and Archis Joglekar. Additionally, it states that the results presented in the paper are derived, either wholly or partially, from data generated by the TCGA Research Network, providing the relevant website link.

Additional Figures