ORBIT: Oak Ridge Base Foundation Model for Earth System Predictability
2404.14712
0
0
📈
Abstract
Earth system predictability is challenged by the complexity of environmental dynamics and the multitude of variables involved. Current AI foundation models, although advanced by leveraging large and heterogeneous data, are often constrained by their size and data integration, limiting their effectiveness in addressing the full range of Earth system prediction challenges. To overcome these limitations, we introduce the Oak Ridge Base Foundation Model for Earth System Predictability (ORBIT), an advanced vision-transformer model that scales up to 113 billion parameters using a novel hybrid tensor-data orthogonal parallelism technique. As the largest model of its kind, ORBIT surpasses the current climate AI foundation model size by a thousandfold. Performance scaling tests conducted on the Frontier supercomputer have demonstrated that ORBIT achieves 230 to 707 PFLOPS, with scaling efficiency maintained at 78% to 96% across 24,576 AMD GPUs. These breakthroughs establish new advances in AI-driven climate modeling and demonstrate promise to significantly improve the Earth system predictability.
Create account to get full access
Overview
- Complexity of Earth system dynamics poses challenges for accurate prediction
- Current AI foundation models have limitations in addressing the full range of Earth system prediction challenges
- Introduction of the Oak Ridge Base Foundation Model for Earth System Predictability (ORBIT), a large-scale vision-transformer model with 113 billion parameters
Plain English Explanation
Predicting the Earth's complex environmental systems, like weather and climate, is a major challenge. Current AI models used for this task, although advanced, have limitations in the amount of data they can handle and the scope of problems they can solve.
To overcome these limitations, researchers have developed a new, very large AI model called ORBIT. ORBIT uses a novel technical approach to scale up to 113 billion parameters, making it the largest model of its kind. This allows ORBIT to consider a much broader range of factors when predicting Earth systems.
Testing on a powerful supercomputer has shown that ORBIT can perform these predictions extremely quickly, at speeds of up to 707 petaflops (that's over 700 quadrillion calculations per second!). This represents a significant advance in the ability of AI to drive climate modeling and improve Earth system predictability.
Technical Explanation
The researchers introduce ORBIT, a large-scale vision-transformer model for Earth system predictability. ORBIT utilizes a novel hybrid tensor-data orthogonal parallelism technique to scale up to 113 billion parameters, surpassing current climate AI foundation models by over a thousandfold.
Performance testing on the Frontier supercomputer demonstrated that ORBIT can achieve 230 to 707 PFLOPS of computational power, with scaling efficiency maintained at 78% to 96% across 24,576 AMD GPUs. This represents a major breakthrough in the speed and scalability of AI-driven climate modeling, paving the way for significant improvements in Earth system predictability.
Critical Analysis
The paper provides a compelling demonstration of ORBIT's immense scale and capabilities, but does not fully address potential limitations or areas for further research. While the performance scaling results are impressive, the paper lacks a deeper discussion of the real-world implications and challenges of deploying such a large-scale model in practical climate forecasting applications.
Additionally, the paper does not explore potential biases or errors that may arise from training on such a vast and heterogeneous dataset, nor does it consider the environmental impact of the computational resources required to run ORBIT. Further research is needed to understand the limits and tradeoffs of this approach.
Conclusion
The introduction of ORBIT, a 113 billion parameter vision-transformer model for Earth system predictability, represents a significant advancement in the field of AI-driven climate modeling. The model's immense scale and computational power, as demonstrated through rigorous testing on the Frontier supercomputer, hold great promise for improving the accuracy and timeliness of Earth system predictions.
While further research is needed to address potential limitations and real-world challenges, the breakthroughs achieved by ORBIT establish new frontiers in the application of large-scale AI to tackle the complex and pressing issue of environmental predictability.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
📈
Aurora: A Foundation Model of the Atmosphere
Cristian Bodnar, Wessel P. Bruinsma, Ana Lucic, Megan Stanley, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan Weyn, Haiyu Dong, Anna Vaughan, Jayesh K. Gupta, Kit Tambiratnam, Alex Archibald, Elizabeth Heider, Max Welling, Richard E. Turner, Paris Perdikaris
0
0
Deep learning foundation models are revolutionizing many facets of science by leveraging vast amounts of data to learn general-purpose representations that can be adapted to tackle diverse downstream tasks. Foundation models hold the promise to also transform our ability to model our planet and its subsystems by exploiting the vast expanse of Earth system data. Here we introduce Aurora, a large-scale foundation model of the atmosphere trained on over a million hours of diverse weather and climate data. Aurora leverages the strengths of the foundation modelling approach to produce operational forecasts for a wide variety of atmospheric prediction problems, including those with limited training data, heterogeneous variables, and extreme events. In under a minute, Aurora produces 5-day global air pollution predictions and 10-day high-resolution weather forecasts that outperform state-of-the-art classical simulation tools and the best specialized deep learning models. Taken together, these results indicate that foundation models can transform environmental forecasting.
5/29/2024
On the Foundations of Earth and Climate Foundation Models
Xiao Xiang Zhu, Zhitong Xiong, Yi Wang, Adam J. Stewart, Konrad Heidler, Yuanyuan Wang, Zhenghang Yuan, Thomas Dujardin, Qingsong Xu, Yilei Shi
0
0
Foundation models have enormous potential in advancing Earth and climate sciences, however, current approaches may not be optimal as they focus on a few basic features of a desirable Earth and climate foundation model. Crafting the ideal Earth foundation model, we define eleven features which would allow such a foundation model to be beneficial for any geoscientific downstream application in an environmental- and human-centric manner.We further shed light on the way forward to achieve the ideal model and to evaluate Earth foundation models. What comes after foundation models? Energy efficient adaptation, adversarial defenses, and interpretability are among the emerging directions.
5/8/2024
Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation
Zhitong Xiong, Yi Wang, Fahong Zhang, Adam J. Stewart, Joelle Hanna, Damian Borth, Ioannis Papoutsis, Bertrand Le Saux, Gustau Camps-Valls, Xiao Xiang Zhu
0
0
The development of foundation models has revolutionized our ability to interpret the Earth's surface using satellite observational data. Traditional models have been siloed, tailored to specific sensors or data types like optical, radar, and hyperspectral, each with its own unique characteristics. This specialization hinders the potential for a holistic analysis that could benefit from the combined strengths of these diverse data sources. Our novel approach introduces the Dynamic One-For-All (DOFA) model, leveraging the concept of neural plasticity in brain science to integrate various data modalities into a single framework adaptively. This dynamic hypernetwork, adjusting to different wavelengths, enables a single versatile Transformer jointly trained on data from five sensors to excel across 12 distinct Earth observation tasks, including sensors never seen during pretraining. DOFA's innovative design offers a promising leap towards more accurate, efficient, and unified Earth observation analysis, showcasing remarkable adaptability and performance in harnessing the potential of multimodal Earth observation data.
6/10/2024
Evaluating and Benchmarking Foundation Models for Earth Observation and Geospatial AI
Nikolaos Dionelis, Casper Fibaek, Luke Camilleri, Andreas Luyts, Jente Bosmans, Bertrand Le Saux
0
0
When we are primarily interested in solving several problems jointly with a given prescribed high performance accuracy for each target application, then Foundation Models should for most cases be used rather than problem-specific models. We focus on the specific Computer Vision application of Foundation Models for Earth Observation (EO) and geospatial AI. These models can solve important problems we are tackling, including for example land cover classification, crop type mapping, flood segmentation, building density estimation, and road regression segmentation. In this paper, we show that for a limited number of labelled data, Foundation Models achieve improved performance compared to problem-specific models. In this work, we also present our proposed evaluation benchmark for Foundation Models for EO. Benchmarking the generalization performance of Foundation Models is important as it has become difficult to standardize a fair comparison across the many different models that have been proposed recently. We present the results using our evaluation benchmark for EO Foundation Models and show that Foundation Models are label efficient in the downstream tasks and help us solve problems we are tackling in EO and remote sensing.
6/27/2024