You Only Train Once: A Unified Framework for Both Full-Reference and No-Reference Image Quality Assessment

Overview

• Proposes a unified framework for both full reference (FR) and no reference (NR) image quality assessment (IQA) tasks
• Employs an encoder to extract multi-level features and a Hierarchical Attention (HA) module to model spatial distortions at each stage
• Introduces a Semantic Distortion Aware (SDA) module to examine feature correlations between shallow and deep layers of the encoder
• Outperforms existing models on both FR and NR IQA tasks when trained independently, and further enhances performance when trained jointly

Plain English Explanation

Evaluating the quality of images is an important task, known as image quality assessment (IQA). Humans can seamlessly transition between two types of IQA: full reference (FR), where a high-quality reference image is available, and no reference (NR), where no reference is provided. However, existing IQA models are constrained to either FR or NR tasks, requiring two distinct systems.

To address this, the proposed approach aims to unify FR and NR IQA under a single framework. It first uses an encoder to extract multi-level features from input images. Then, a Hierarchical Attention (HA) module is used as a universal adapter for both FR and NR inputs to model spatial distortions at each encoder stage. Additionally, a Semantic Distortion Aware (SDA) module is introduced to examine feature correlations between shallow and deep layers of the encoder, as different distortions can affect image semantics differently.

By adopting HA and SDA, the proposed network can effectively perform both FR and NR IQA. When trained independently on NR or FR IQA tasks, it outperforms existing models and achieves state-of-the-art performance. Moreover, when trained jointly on NR and FR IQA tasks, it further enhances the performance of NR IQA while achieving on-par performance with the state-of-the-art in FR IQA. This means that the model can be trained once to perform both IQA tasks, greatly improving its versatility.

Technical Explanation

The proposed approach first employs an encoder to extract multi-level features from input images. This allows the model to capture information at different levels of abstraction, which is crucial for understanding complex image distortions.

Next, the Hierarchical Attention (HA) module is introduced as a universal adapter for both FR and NR inputs. The HA module models the spatial distortions at each encoder stage, enabling the model to effectively handle both types of IQA tasks.

Furthermore, the Semantic Distortion Aware (SDA) module is proposed to examine the feature correlations between shallow and deep layers of the encoder. This is important because different distortions can contaminate encoder stages and damage image semantic meaning in different ways. The SDA module helps the model understand these complex relationships, which is crucial for accurately assessing image quality.

By combining the HA and SDA modules, the proposed network can effectively perform both FR and NR IQA tasks. When trained independently on either NR or FR IQA tasks, the model outperforms existing state-of-the-art approaches. Moreover, when trained jointly on both NR and FR IQA tasks, the model further enhances its performance on NR IQA while maintaining state-of-the-art results on FR IQA. This demonstrates the versatility and effectiveness of the proposed unified framework.

Critical Analysis

The paper presents a promising approach to unifying FR and NR IQA tasks, which addresses an important limitation of existing models. The use of the HA and SDA modules is well-justified and the experimental results demonstrate the effectiveness of the proposed approach.

However, the paper does not discuss the potential computational complexity or inference time of the unified model, which could be a concern for real-world applications. Additionally, the paper does not explore the model's robustness to different types of image distortions or its performance on diverse datasets.

Further research could investigate the scalability of the unified framework, its generalization to a wider range of IQA tasks, and its practical implications for various image-related applications. Exploring the interpretability of the HA and SDA modules could also provide valuable insights into the model's decision-making process and help identify areas for improvement.

Conclusion

The proposed unified framework for FR and NR IQA tasks represents a significant advancement in the field. By employing an encoder, a Hierarchical Attention module, and a Semantic Distortion Aware module, the model can effectively handle both types of IQA tasks, outperforming existing state-of-the-art approaches.

The ability to train a single model for both FR and NR IQA tasks greatly improves the model's versatility and practical applicability. This research paves the way for more robust and efficient image quality assessment systems, with potential impact on a wide range of image-related applications, such as image compression, restoration, and enhancement.