Discovering Novel Actions from Open World Egocentric Videos with Object-Grounded Visual Commonsense Reasoning

Overview

• This paper proposes a novel framework called ALGO (Action Learning with Grounded Object recognition) to tackle the challenge of learning to infer labels in an "open world" environment, where the target labels are unknown.
• The key idea is to leverage symbolic knowledge stored in large-scale knowledge bases to infer activities in egocentric videos with limited supervision.
• ALGO uses a two-step approach: first, it uses object-centric vision-language models as a "noisy oracle" to ground objects in the video; second, it discovers plausible activities through an energy-based symbolic pattern theory framework, grounding knowledge-based action (verb) concepts in the video.

Plain English Explanation

In many real-world scenarios, the specific "labels" or categories we want to identify may not be known ahead of time. This makes it challenging for AI systems to reliably recognize and classify objects, actions, and events. This paper explores a novel solution to this "open world" challenge, where the target labels are unknown.

The key idea is to leverage the wealth of information stored in large-scale knowledge bases - things like common sense knowledge about the world, activities, and how objects interact. By tapping into this symbolic knowledge, the researchers developed a two-step framework called ALGO that can infer activities in videos, even when the specific labels are not provided upfront.

First, ALGO uses "vision-language models" - AI systems trained on massive amounts of image-text data - as a kind of "noisy oracle" to identify and ground objects in the video. This provides a starting point for the system to reason about what's happening.

Then, guided by this grounded object information and the prior knowledge from the knowledge base, ALGO discovers plausible activities through an "energy-based" framework. This allows it to learn and ground the relevant action (verb) concepts in the video, even without being explicitly told what to look for.

The researchers show that this neuro-symbolic approach outperforms other methods on several benchmark datasets for activity recognition in videos. It's an important step towards building AI systems that can flexibly adapt and learn in open-ended, real-world environments.

Technical Explanation

The core technical contribution of this paper is the ALGO framework, which combines neural and symbolic approaches to tackle the challenge of open-world activity inference.

The first step is a "neuro-symbolic prompting" approach that uses object-centric vision-language models as a noisy oracle to ground objects in the video. These models are pre-trained on large-scale image-text data, and can provide an initial set of grounded object detections to build upon.

The second step is driven by commonsense knowledge extracted from large-scale knowledge bases. ALGO discovers plausible activities through an "energy-based symbolic pattern theory framework." This allows it to reason about and ground relevant action (verb) concepts in the video, even when the specific target labels are unknown.

The researchers evaluated ALGO on four publicly available datasets for egocentric video activity recognition: EPIC-Kitchens, GTEA Gaze, GTEA Gaze Plus, and Charades-Ego. They show that ALGO outperforms other methods, particularly in the challenging open-world setting where the target labels are not provided.

Furthermore, the researchers demonstrate that ALGO can be extended to zero-shot inference, where the system has to recognize activities without any labeled examples. On the Charades-Ego dataset, ALGO achieves competitive performance in this zero-shot setting, showcasing its flexibility and broad applicability.

Critical Analysis

The ALGO framework represents an important step towards building more versatile and adaptable AI systems that can operate in open-ended, real-world environments. By leveraging both neural and symbolic approaches, it tackles the significant challenge of activity inference when the target labels are unknown.

That said, the paper does acknowledge some limitations. The performance of the neuro-symbolic prompting approach is still dependent on the quality and coverage of the initial object detections provided by the vision-language models. If these models fail to accurately ground certain objects, it can negatively impact the subsequent activity inference.

Additionally, the energy-based symbolic pattern theory framework, while powerful, relies on the availability and coverage of the commonsense knowledge base. If the knowledge base is incomplete or biased, it could lead to suboptimal activity discovery and grounding.

Future research could explore ways to make the ALGO framework more robust to imperfect object detections and knowledge base limitations, perhaps by incorporating additional learning mechanisms or uncertainty modeling. Investigating the scalability and generalization of the approach to larger and more diverse datasets would also be valuable.

Conclusion

The ALGO framework proposed in this paper represents a promising step towards building AI systems that can flexibly operate in open-world environments, where the target labels are unknown. By combining neural and symbolic approaches, it leverages both the power of pre-trained vision-language models and the richness of commonsense knowledge to infer activities in egocentric videos.

The strong performance of ALGO on various benchmarks, as well as its ability to extend to zero-shot inference, highlights its potential to advance the state of the art in activity recognition and open-world learning. As AI systems continue to be deployed in more complex, real-world scenarios, frameworks like ALGO will become increasingly important for achieving true autonomy and versatility.