Two Complementary Perspectives to Continual Learning: Ask Not Only What to Optimize, But Also How

Overview

• Recent research has made significant progress in continual training of deep neural networks, often by adding replay or regularization terms to the loss function.
• However, these approaches still suffer from temporary but substantial forgetting when starting to train on a new task, even with a perfect approximation of the joint loss.
• The paper argues that continual learning strategies should focus not only on the optimization objective, but also on the optimization trajectory.

Plain English Explanation

The paper discusses the challenges of continual learning, where deep neural networks are trained on a series of tasks over time. Existing approaches have made progress by modifying the loss function, such as adding replay or regularization terms to help the model remember previous tasks. However, the paper shows that even with a perfect approximation of the joint loss across all tasks, these models still experience substantial forgetting when starting to learn a new task.

The key insight is that continual learning strategies should consider not just the optimization objective, but also how that objective is optimized. Some previous work has explored altering the optimization trajectory, for example using gradient projection techniques, but this has been positioned as an alternative to improving the objective. The paper argues that optimizing the trajectory should be a complementary approach.

To support this idea, the researchers conducted experiments combining replay-based objectives with gradient projection-based optimization routines. While the initial results did not show clear benefits, the paper still makes a conceptual argument for the importance of the optimization trajectory in continual learning. This opens up a new direction for future research in this area.

Technical Explanation

The paper explores the challenge of continual learning, where deep neural networks are trained on a sequence of tasks over time. Existing approaches have made progress using techniques that add replay or regularization terms to the loss function to approximate the joint loss over all tasks. However, the authors show that even with a perfect approximation of the joint loss, these methods still suffer from temporary but substantial forgetting when starting to train on a new task.

Motivated by this "stability gap," the authors argue that continual learning strategies should focus not only on the optimization objective, but also on the way this objective is optimized. While there is some prior work that alters the optimization trajectory, such as using gradient projection techniques, this has been positioned as an alternative to improving the optimization objective. The authors contend that optimizing the trajectory should be a complementary approach.

To provide empirical support for this proposition, the researchers conducted a series of pre-registered experiments that combined replay-approximated joint objectives with gradient projection-based optimization routines. However, this initial attempt did not clearly demonstrate the benefits of this approach.

Nevertheless, the conceptual arguments in the paper, as well as some of the empirical results, highlight the distinctive importance of the optimization trajectory in continual learning. This opens up a new direction for future research in continual learning across numerous tasks, potentially including provable approaches or weight interpolation techniques.

Critical Analysis

The paper makes a compelling case for the importance of considering the optimization trajectory in addition to the optimization objective for continual learning. While the initial experiments did not clearly demonstrate the benefits of this approach, the conceptual arguments are well-reasoned and open up an interesting new direction for future research.

One potential limitation of the work is the scope of the experiments, which may not have fully captured the benefits of the proposed approach. The authors acknowledge that further investigation is needed to determine the conditions under which optimizing the trajectory can provide clear advantages over solely optimizing the objective.

Additionally, the paper does not delve into the potential computational and implementation challenges of jointly optimizing the objective and trajectory, which could be an important practical consideration for deploying these techniques in real-world applications.

Overall, the paper makes a valuable contribution by highlighting the distinctive importance of the optimization trajectory in continual learning and encouraging researchers to explore this promising area further. Readers are encouraged to think critically about the insights presented and consider how they might be applied or extended in future work.

Conclusion

This paper argues that continual learning strategies should focus not only on the optimization objective, but also on the optimization trajectory. While existing approaches have made progress by modifying the loss function, the authors show that these methods still suffer from substantial forgetting when learning new tasks.

The key insight is that the optimization trajectory, in addition to the objective, plays a crucial role in mitigating forgetting and enabling effective continual learning. This opens up a new direction for future research in this area, potentially leading to more robust and flexible deep neural networks that can continuously adapt to new information without catastrophic forgetting.