ArchesWeather: An efficient AI weather forecasting model at 1.5{deg} resolution

Overview

• Designing effective AI-based weather forecasting systems requires embedding physical constraints as inductive priors in the neural network architecture.
• One popular prior is locality, where the atmospheric data is processed with local neural interactions like 3D convolutions or 3D local attention windows as in Pangu-Weather.
• However, some works have achieved great success in weather forecasting without this locality principle, at the cost of a much higher parameter count.
• This paper presents ArchesWeather, a transformer model that combines 2D attention with a column-wise attention-based feature interaction module, demonstrating improved forecasting skill over the locality-based Pangu-Weather approach.

Plain English Explanation

Weather forecasting is a complex task that relies on understanding the physical processes governing the atmosphere. One way to design effective AI-based weather forecasting systems is to incorporate these physical constraints as inductive priors into the neural network architecture. A common prior is the idea of locality, where the atmospheric data is processed using local neural interactions, such as 3D convolutions or 3D local attention windows, as seen in the Pangu-Weather model.

However, the authors of this paper found that this 3D local processing in Pangu-Weather is computationally sub-optimal. Instead, they developed a new model called ArchesWeather, which uses a transformer architecture that combines 2D attention with a column-wise attention-based feature interaction module. This design, they argue, improves the forecasting skill compared to the locality-based approach.

ArchesWeather is trained using a relatively small budget of a few GPU-days and has a lower inference cost than competing methods. The authors show that an ensemble of two of their best ArchesWeather models can achieve competitive performance with the IFS HRES model and outperforms the 1.4-degree 50-member NeuralGCM ensemble for one-day-ahead forecasting.

Technical Explanation

The paper presents ArchesWeather, a transformer-based model that aims to improve upon the locality-based approach used in the Pangu-Weather model. The authors argue that the 3D local processing in Pangu-Weather is computationally sub-optimal and instead design a new architecture that combines 2D attention with a column-wise attention-based feature interaction module.

ArchesWeather is trained at a 1.5-degree resolution and 24-hour lead time, with a training budget of a few GPU-days and a lower inference cost than competing methods. The authors evaluate the performance of ArchesWeather and show that an ensemble of two of their best models achieves competitive RMSE scores with the IFS HRES model and outperforms the 1.4-degree 50-member NeuralGCM ensemble for one-day-ahead forecasting.

The authors make their code and models publicly available at https://github.com/gcouairon/ArchesWeather.

Critical Analysis

The paper presents a compelling approach to improving weather forecasting by rethinking the architectural design of the neural network. The authors' insight that the 3D local processing in Pangu-Weather is computationally sub-optimal and their exploration of an alternative transformer-based architecture with column-wise attention-based feature interaction is a valuable contribution to the field.

However, the paper does not delve into the potential limitations or caveats of the ArchesWeather approach. For example, it would be interesting to understand how the model's performance scales with different resolutions or lead times, or how it might handle extreme weather events that require more complex physical reasoning.

Additionally, the authors' claims about the computational efficiency of ArchesWeather compared to competing methods could be further explored and substantiated, perhaps by providing more detailed analysis or comparisons of inference times and memory usage.

Overall, the paper presents a solid technical contribution, but there may be opportunities for the authors to dig deeper into the nuances and potential limitations of their approach, which could lead to even more valuable insights for the weather forecasting research community.

Conclusion

This paper introduces ArchesWeather, a transformer-based model for weather forecasting that combines 2D attention with a column-wise attention-based feature interaction module. The authors demonstrate that this architecture can outperform the locality-based Pangu-Weather model, while maintaining a lower computational budget and inference cost.

The ArchesWeather approach represents an important advancement in the design of AI-based weather forecasting systems, as it challenges the prevailing assumption that embedding physical constraints as inductive priors is necessary for effective forecasting. The authors' work suggests that alternative architectural choices can lead to improved performance, opening up new avenues for exploration in this critical field.

By making their code and models publicly available, the authors are contributing to the broader research community and enabling further advancements in weather forecasting technology. As the field continues to evolve, the insights and approaches presented in this paper are likely to have a lasting impact on the development of more accurate, efficient, and robust AI-powered weather forecasting systems.