Data driven weather forecasts trained and initialised directly from observations
0
Sign in to get full access
Overview
- This paper presents a novel approach to weather forecasting that directly trains and initializes models from observational data.
- The proposed method eliminates the need for traditional numerical weather prediction models and can produce accurate forecasts more efficiently.
- The researchers demonstrate the effectiveness of their data-driven approach on several weather prediction tasks.
Plain English Explanation
The researchers have developed a new way to forecast the weather that avoids using the complex numerical models traditionally relied upon. Instead, they train their weather prediction models directly on observational data, such as temperature, humidity, and wind measurements.
This data-driven approach allows the models to learn the patterns in the weather data more efficiently, without having to first simulate the underlying physics through numerical models. The researchers show that their models can produce accurate weather forecasts just as well as, or even better than, the traditional numerical methods.
This is significant because the numerical weather prediction models currently used require a lot of computing power and can be slow to run. By bypassing these models and training directly on observations, the new approach can generate weather forecasts more quickly and with fewer computational resources. This could lead to faster, more accessible weather prediction capabilities.
Technical Explanation
The paper introduces a novel architecture that directly learns weather forecasting from observational data, without relying on traditional numerical weather prediction models. The researchers demonstrate the effectiveness of this data-driven approach on a range of forecasting tasks, including predicting temperature, precipitation, and wind speed.
The key innovation is that the models are trained and initialized using only observational data, rather than being built on top of complex physical simulations. This allows the models to learn the underlying patterns in the weather data more efficiently. The researchers show that their end-to-end AI-driven approach can achieve forecast accuracy comparable to or better than traditional numerical weather prediction, while being much faster to run.
Critical Analysis
The paper presents a compelling case for the potential of data-driven weather forecasting models. However, the researchers acknowledge several caveats and areas for further research. For example, the models may struggle to generalize to weather conditions or regions not well-represented in the training data.
Additionally, while the data-driven approach can produce fast forecasts, it may lack the physical interpretability of traditional numerical models. This could make it more difficult to understand the underlying mechanisms driving the weather patterns.
Further research is needed to explore the potential of hybrid machine learning and physics-based modeling approaches, which could combine the strengths of both data-driven and numerical methods.
Conclusion
This paper presents a promising new direction for weather forecasting that avoids the complexity and computational demands of traditional numerical models. By training models directly on observational data, the researchers have demonstrated the potential for end-to-end AI-driven weather prediction that is both accurate and efficient.
While further research is needed to address the limitations, this work represents an important step towards more accessible and responsive weather forecasting capabilities that could benefit a wide range of applications and industries.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
Data driven weather forecasts trained and initialised directly from observations
Anthony McNally, Christian Lessig, Peter Lean, Eulalie Boucher, Mihai Alexe, Ewan Pinnington, Matthew Chantry, Simon Lang, Chris Burrows, Marcin Chrust, Florian Pinault, Ethel Villeneuve, Niels Bormann, Sean Healy
Skilful Machine Learned weather forecasts have challenged our approach to numerical weather prediction, demonstrating competitive performance compared to traditional physics-based approaches. Data-driven systems have been trained to forecast future weather by learning from long historical records of past weather such as the ECMWF ERA5. These datasets have been made freely available to the wider research community, including the commercial sector, which has been a major factor in the rapid rise of ML forecast systems and the levels of accuracy they have achieved. However, historical reanalyses used for training and real-time analyses used for initial conditions are produced by data assimilation, an optimal blending of observations with a physics-based forecast model. As such, many ML forecast systems have an implicit and unquantified dependence on the physics-based models they seek to challenge. Here we propose a new approach, training a neural network to predict future weather purely from historical observations with no dependence on reanalyses. We use raw observations to initialise a model of the atmosphere (in observation space) learned directly from the observations themselves. Forecasts of crucial weather parameters (such as surface temperature and wind) are obtained by predicting weather parameter observations (e.g. SYNOP surface data) at future times and arbitrary locations. We present preliminary results on forecasting observations 12-hours into the future. These already demonstrate successful learning of time evolutions of the physical processes captured in real observations. We argue that this new approach, by staying purely in observation space, avoids many of the challenges of traditional data assimilation, can exploit a wider range of observations and is readily expanded to simultaneous forecasting of the full Earth system (atmosphere, land, ocean and composition).
Read more7/23/2024
0
End-to-end data-driven weather forecasting
Anna Vaughan, Stratis Markou, Will Tebbutt, James Requeima, Wessel P. Bruinsma, Tom R. Andersson, Michael Herzog, Nicholas D. Lane, Matthew Chantry, J. Scott Hosking, Richard E. Turner
Weather forecasting is critical for a range of human activities including transportation, agriculture, industry, as well as the safety of the general public. Machine learning models have the potential to transform the complex weather prediction pipeline, but current approaches still rely on numerical weather prediction (NWP) systems, limiting forecast speed and accuracy. Here we demonstrate that a machine learning model can replace the entire operational NWP pipeline. Aardvark Weather, an end-to-end data-driven weather prediction system, ingests raw observations and outputs global gridded forecasts and local station forecasts. Further, it can be optimised end-to-end to maximise performance over quantities of interest. Global forecasts outperform an operational NWP baseline for multiple variables and lead times. Local station forecasts are skillful up to ten days lead time and achieve comparable and often lower errors than a post-processed global NWP baseline and a state-of-the-art end-to-end forecasting system with input from human forecasters. These forecasts are produced with a remarkably simple neural process model using just 8% of the input data and three orders of magnitude less compute than existing NWP and hybrid AI-NWP methods. We anticipate that Aardvark Weather will be the starting point for a new generation of end-to-end machine learning models for medium-range forecasting that will reduce computational costs by orders of magnitude and enable the rapid and cheap creation of bespoke models for users in a variety of fields, including for the developing world where state-of-the-art local models are not currently available.
Read more7/16/2024
0
Leveraging data-driven weather models for improving numerical weather prediction skill through large-scale spectral nudging
Syed Zahid Husain, Leo Separovic, Jean-Franc{c}ois Caron, Rabah Aider, Mark Buehner, St'ephane Chamberland, Ervig Lapalme, Ron McTaggart-Cowan, Christopher Subich, Paul A. Vaillancourt, Jing Yang, Ayrton Zadra
Operational meteorological forecasting has long relied on physics-based numerical weather prediction (NWP) models. Recently, this landscape is facing disruption by the advent of data-driven artificial intelligence (AI)-based weather models, which offer tremendous computational performance and competitive forecasting skill. However, data-driven models for medium-range forecasting generally suffer from major limitations, including low effective resolution and a narrow range of predicted variables. This study illustrates the relative strengths and weaknesses of these competing paradigms using the GEM (Global Environmental Multiscale) and GraphCast models to represent physics-based and AI-based approaches, respectively. By analyzing global predictions from these two models against observations and analyses in both physical and spectral spaces, this study demonstrates that GraphCast-predicted large scales outperform GEM, particularly for longer lead times. Building on this insight, a hybrid NWP-AI system is proposed, wherein GEM-predicted large-scale state variables are spectrally nudged toward GraphCast predictions, while allowing GEM to freely generate fine-scale details critical for weather extremes. Results indicate that this hybrid approach is capable of leveraging the strengths of GraphCast to enhance the prediction skill of the GEM model. Importantly, trajectories of tropical cyclones are predicted with enhanced accuracy without significant changes in intensity. Furthermore, this new hybrid system ensures that meteorologists have access to a complete set of forecast variables, including those relevant for high-impact weather events.
Read more7/26/2024
🧠
0
Decomposing weather forecasting into advection and convection with neural networks
Mengxuan Chen, Ziqi Yuan, Jinxiao Zhang, Runmin Dong, Haohuan Fu
Operational weather forecasting models have advanced for decades on both the explicit numerical solvers and the empirical physical parameterization schemes. However, the involved high computational costs and uncertainties in these existing schemes are requiring potential improvements through alternative machine learning methods. Previous works use a unified model to learn the dynamics and physics of the atmospheric model. Contrarily, we propose a simple yet effective machine learning model that learns the horizontal movement in the dynamical core and vertical movement in the physical parameterization separately. By replacing the advection with a graph attention network and the convection with a multi-layer perceptron, our model provides a new and efficient perspective to simulate the transition of variables in atmospheric models. We also assess the model's performance over a 5-day iterative forecasting. Under the same input variables and training methods, our model outperforms existing data-driven methods with a significantly-reduced number of parameters with a resolution of 5.625 deg. Overall, this work aims to contribute to the ongoing efforts that leverage machine learning techniques for improving both the accuracy and efficiency of global weather forecasting.
Read more5/13/2024