Bake off redux: a review and experimental evaluation of recent time series classification algorithms
0
🏷️
Sign in to get full access
Overview
- In 2017, a study compared 18 time series classification (TSC) algorithms on 85 datasets
- The study found that only 9 algorithms performed significantly better than the benchmarks used
- The study categorized the algorithms into 5 main types based on the features they extract
- This categorization and the availability of code/results has increased the popularity of the TSC field
- Over 6 years later, the dataset has expanded, and many new algorithms have been proposed
- This paper revisits the original study, evaluating how the algorithm categories have advanced and comparing newer algorithms
Plain English Explanation
This research paper looks at a previous study that compared different algorithms for classifying time series data. Time series data is information collected over time, like stock prices or sensor readings. The 2017 study tested 18 different algorithms on 85 time series datasets, and found that only 9 of them performed significantly better than two benchmark algorithms.
The researchers also categorized the algorithms into 5 main types based on the features they use to analyze the data. This helped organize the field and made the results more accessible. Since then, the dataset has grown, and many new algorithms have been developed.
The current paper revisits this original "bake-off" study to see how the different algorithm categories have progressed. They also evaluate some of the newer algorithms that have been proposed, to see how they compare to the previous best-performing ones. This helps advance the time series classification field by identifying the most effective techniques.
Technical Explanation
The 2017 "bake-off" study compared 18 time series classification (TSC) algorithms on 85 datasets from the UCR archive. It found that only 9 of these algorithms performed significantly better than the Dynamic Time Warping (DTW) and Rotation Forest benchmarks.
The study categorized the algorithms into 5 main types based on the features they extract from the time series data: distance-based, interval-based, shapelet-based, dictionary-based, and hybrid. This taxonomy helped organize the field and made the results more accessible.
In the current paper, the researchers revisit this bake-off to see how the different algorithm categories have advanced over the past 6+ years. The UCR archive has now expanded to 112 datasets, and there have been many new algorithms proposed.
The researchers extend the original taxonomy to include 3 new categories: convolution-based, feature-based, and deep learning approaches. They then compare the performance of the newer algorithms against the previous best-in-class for each category, using the expanded UCR dataset.
The paper also introduces 30 new classification datasets that have been added to the UCR archive or reformatted to the TSC format. These are used to further evaluate the top-performing algorithms from each category.
The key finding is that two recently proposed algorithms, Hydra+MultiROCKET and HIVE-COTEv2, significantly outperform the other approaches on both the current and new TSC problems.
Critical Analysis
The paper provides a comprehensive update to the original 2017 bake-off study, expanding the dataset and algorithm taxonomy to reflect the rapid progress in the TSC field. The systematic evaluation approach and clear categorization of algorithms are valuable contributions.
One potential limitation is that the paper only evaluates the algorithms on the UCR archive datasets. While this is a widely-used benchmark, there may be other real-world time series datasets with different characteristics that are not captured. Expanding the evaluation to a more diverse set of datasets could further strengthen the findings.
Additionally, the paper does not delve deeply into the specific architectural details or training procedures of the newer deep learning and feature-based algorithms. More insight into these components could help readers better understand the underlying reasons for the performance differences.
Overall, this paper serves as an important update to the TSC algorithm landscape, highlighting the latest advancements and providing a roadmap for future research. Encouraging readers to think critically about the generalizability and interpretability of these models is a valuable aspect of the analysis.
Conclusion
This research paper provides an updated evaluation of time series classification algorithms, building on a influential 2017 "bake-off" study. By expanding the dataset, taxonomy, and set of algorithms compared, the authors have identified two new top-performing approaches: Hydra+MultiROCKET and HIVE-COTEv2.
The systematic comparison across algorithm categories and extensive dataset testing contributes significantly to the time series classification field. The findings can help guide researchers and practitioners in selecting the most effective TSC techniques for their specific applications.
Looking ahead, continued advancements in areas like unsupervised contrastive learning and anomaly detection for time series data will likely lead to further improvements in classification performance and robustness. This paper lays important groundwork for these future developments in the TSC domain.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
🏷️
0
Bake off redux: a review and experimental evaluation of recent time series classification algorithms
Matthew Middlehurst, Patrick Schafer, Anthony Bagnall
In 2017, a research paper compared 18 Time Series Classification (TSC) algorithms on 85 datasets from the University of California, Riverside (UCR) archive. This study, commonly referred to as a `bake off', identified that only nine algorithms performed significantly better than the Dynamic Time Warping (DTW) and Rotation Forest benchmarks that were used. The study categorised each algorithm by the type of feature they extract from time series data, forming a taxonomy of five main algorithm types. This categorisation of algorithms alongside the provision of code and accessible results for reproducibility has helped fuel an increase in popularity of the TSC field. Over six years have passed since this bake off, the UCR archive has expanded to 112 datasets and there have been a large number of new algorithms proposed. We revisit the bake off, seeing how each of the proposed categories have advanced since the original publication, and evaluate the performance of newer algorithms against the previous best-of-category using an expanded UCR archive. We extend the taxonomy to include three new categories to reflect recent developments. Alongside the originally proposed distance, interval, shapelet, dictionary and hybrid based algorithms, we compare newer convolution and feature based algorithms as well as deep learning approaches. We introduce 30 classification datasets either recently donated to the archive or reformatted to the TSC format, and use these to further evaluate the best performing algorithm from each category. Overall, we find that two recently proposed algorithms, Hydra+MultiROCKET and HIVE-COTEv2, perform significantly better than other approaches on both the current and new TSC problems.
Read more5/9/2024
0
Convolutional and Deep Learning based techniques for Time Series Ordinal Classification
Rafael Ayll'on-Gavil'an, David Guijo-Rubio, Pedro Antonio Guti'errez, Anthony Bagnall, C'esar Herv'as-Mart'inez
Time Series Classification (TSC) covers the supervised learning problem where input data is provided in the form of series of values observed through repeated measurements over time, and whose objective is to predict the category to which they belong. When the class values are ordinal, classifiers that take this into account can perform better than nominal classifiers. Time Series Ordinal Classification (TSOC) is the field covering this gap, yet unexplored in the literature. There are a wide range of time series problems showing an ordered label structure, and TSC techniques that ignore the order relationship discard useful information. Hence, this paper presents a first benchmarking of TSOC methodologies, exploiting the ordering of the target labels to boost the performance of current TSC state-of-the-art. Both convolutional- and deep learning-based methodologies (among the best performing alternatives for nominal TSC) are adapted for TSOC. For the experiments, a selection of 29 ordinal problems from two well-known archives has been made. In this way, this paper contributes to the establishment of the state-of-the-art in TSOC. The results obtained by ordinal versions are found to be significantly better than current nominal TSC techniques in terms of ordinal performance metrics, outlining the importance of considering the ordering of the labels when dealing with this kind of problems.
Read more7/16/2024
0
ECRTime: Ensemble Integration of Classification and Retrieval for Time Series Classification
Fan Zhao, You Chen
Deep learning-based methods for Time Series Classification (TSC) typically utilize deep networks to extract features, which are then processed through a combination of a Fully Connected (FC) layer and a SoftMax function. However, we have observed the phenomenon of inter-class similarity and intra-class inconsistency in the datasets from the UCR archive and further analyzed how this phenomenon adversely affects the FC+SoftMax paradigm. To address the issue, we introduce ECR, which, for the first time to our knowledge, applies deep learning-based retrieval algorithm to the TSC problem and integrates classification and retrieval models. Experimental results on 112 UCR datasets demonstrate that ECR is state-of-the-art(sota) compared to existing deep learning-based methods. Furthermore, we have developed a more precise classifier, ECRTime, which is an ensemble of ECR. ECRTime surpasses the currently most accurate deep learning classifier, InceptionTime, in terms of accuracy, achieving this with reduced training time and comparable scalability.
Read more7/23/2024
0
Time series classification with random convolution kernels based transforms: pooling operators and input representations matter
Mouhamadou Mansour Lo, Gildas Morvan, Mathieu Rossi, Fabrice Morganti, David Mercier
This article presents a new approach based on MiniRocket, called SelF-Rocket, for fast time series classification (TSC). Unlike existing approaches based on random convolution kernels, it dynamically selects the best couple of input representations and pooling operator during the training process. SelF-Rocket achieves state-of-the-art accuracy on the University of California Riverside (UCR) TSC benchmark datasets.
Read more9/4/2024