Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data

Overview

• Gradient-boosted decision trees (GBDTs) have long dominated in classification and regression tasks on tabular data.
• Recently, deep learning methods have challenged GBDTs, but often require extensive hyperparameter tuning.
• The authors address this discrepancy by:
  • Introducing RealMLP, an improved multilayer perceptron (MLP)
  • Improving default parameters for GBDTs and RealMLP

Plain English Explanation

The paper addresses a discrepancy in the performance of different machine learning models on tabular data. Gradient-boosted decision trees (GBDTs) have traditionally been the go-to approach for classification and regression tasks on this type of data. However, more recently, deep learning methods have started to challenge the dominance of GBDTs, though they often require a lot of hyperparameter tuning to achieve good performance.

The authors of this paper aim to address this discrepancy in a few ways:

1. They introduce a new type of multilayer perceptron (MLP) called RealMLP, which is an improved version of the standard MLP.
2. They also provide improved default parameters for both GBDTs and RealMLP, so that these models can achieve good performance without extensive hyperparameter tuning.

Technical Explanation

The core of the paper is the introduction of RealMLP, an improved MLP architecture, and the optimization of default hyperparameters for both RealMLP and GBDTs.

The authors first tuned RealMLP and the default hyperparameters on a "meta-train" benchmark consisting of 71 classification and 47 regression datasets. They then compared the performance of the tuned RealMLP and default hyperparameter settings to hyperparameter-optimized versions of other models on a disjoint "meta-test" benchmark with 48 classification and 42 regression datasets, as well as the GBDT-friendly benchmark from a previous study.

The benchmark results showed that RealMLP offers a better time-accuracy tradeoff than other neural networks and is competitive with GBDTs. Moreover, the authors found that a combination of RealMLP and GBDTs with improved default parameters can achieve excellent results on medium-sized tabular datasets (1K--500K samples) without the need for extensive hyperparameter tuning.

Critical Analysis

The paper presents a compelling approach to improving the performance of both GBDTs and deep learning models on tabular data. The authors acknowledge that their work is limited to a specific set of datasets and that further research is needed to understand the broader applicability of their methods.

One potential concern is the reliance on meta-learning to tune the default hyperparameters. While this approach has been shown to be effective, it may not generalize well to all types of tabular datasets, and the authors do not provide a clear way to extend the default hyperparameters to new datasets.

Additionally, the authors do not provide a detailed analysis of the inner workings of RealMLP or the reasons behind its improved performance compared to other neural network architectures. A deeper understanding of the model's strengths and weaknesses could help inform future developments in this area.

Conclusion

This paper offers a promising solution to the discrepancy between the performance of GBDTs and deep learning models on tabular data. By introducing RealMLP and optimizing default hyperparameters, the authors have demonstrated that it is possible to achieve excellent results on medium-sized datasets without the need for extensive hyperparameter tuning.

The findings of this research could have significant implications for the broader field of machine learning, particularly in domains where tabular data is prevalent and the time and resources required for model development are limited. As the authors suggest, further research is needed to explore the broader applicability of their methods and to deepen the understanding of the underlying mechanisms driving the performance improvements.