Event-based vision on FPGAs -- a survey

Overview

• This paper provides a comprehensive survey of event-based vision on Field Programmable Gate Arrays (FPGAs).
• Event-based vision, also known as dynamic vision sensors (DVS), is a novel sensing paradigm that captures only changes in the visual scene, rather than full video frames.
• FPGAs are reconfigurable hardware devices that can efficiently implement event-based vision algorithms in embedded systems.
• The survey covers a wide range of topics, including event cameras, FPGA architectures, event-based processing, and various applications of event-based vision on FPGAs.

Plain English Explanation

Event cameras are a new type of visual sensor that work very differently from traditional cameras. Instead of capturing full video frames like a normal camera, event cameras only register changes in the scene. Whenever something moves or changes, the event camera sends out a small signal called an "event." These events are processed much more efficiently than full video frames, making event cameras well-suited for embedded systems and real-time applications.

FPGAs are a type of specialized computer chip that can be programmed to do specific tasks very quickly. They are often used in embedded systems, like self-driving car sensors or robotics, where speed and efficiency are important. By pairing event cameras with FPGAs, researchers can create vision systems that are fast, low-power, and able to respond instantly to changes in the environment.

This survey paper looks at all the different ways that event cameras and FPGAs can be used together. It covers the technical details of how event cameras work, the architecture of FPGAs, and the various algorithms and applications that have been developed to process event data on FPGA hardware. The goal is to provide a comprehensive overview of this emerging field for researchers and engineers working on embedded vision systems.

Technical Explanation

The paper begins by introducing the concept of event-based vision and dynamic vision sensors (DVS). Unlike traditional cameras that capture full video frames, event cameras only output asynchronous "events" corresponding to local pixel-level brightness changes in the scene. This event-driven sensing paradigm offers advantages in terms of high temporal resolution, low latency, and reduced data bandwidth compared to frame-based cameras.

The authors then discuss the suitability of Field Programmable Gate Arrays (FPGAs) for implementing event-based vision algorithms. FPGAs are reconfigurable hardware devices that can efficiently parallelize event processing tasks in embedded systems. The paper reviews common FPGA architectures and their relevance to event-based vision.

Next, the survey covers various event-based processing techniques that have been mapped onto FPGA platforms. This includes low-level event filtering and feature extraction, as well as higher-level event-driven neural networks and graph-based algorithms. The authors discuss the unique challenges and design considerations for event-based processing on FPGAs.

Finally, the paper examines a range of applications that have been explored for event-based vision on FPGAs, such as object tracking, gesture recognition, and simultaneous localization and mapping (SLAM). The authors highlight the benefits of this approach for low-power, low-latency embedded vision tasks.

Critical Analysis

The survey provides a thorough and well-structured overview of the state-of-the-art in event-based vision on FPGAs. The authors comprehensively cover the relevant background, key technical aspects, and diverse applications of this emerging field.

One potential limitation is the relatively narrow focus on FPGA platforms, as event-based vision algorithms can also be implemented on other hardware accelerators like GPUs or specialized neuromorphic chips. The authors acknowledge this, but a broader discussion of hardware trade-offs and co-design opportunities could have been insightful.

Additionally, while the paper highlights the benefits of event-based vision for embedded systems, it does not delve deeply into the practical challenges of deploying these systems in real-world scenarios. Factors like event camera calibration, noise handling, and integration with other sensors could merit further exploration.

Overall, this survey serves as a valuable reference for researchers and engineers interested in exploring the synergies between event-based vision and reconfigurable hardware. By clearly articulating the state of the art, it can help drive further advancements in this promising field.

Conclusion

This comprehensive survey paper provides a thorough overview of the emerging field of event-based vision on Field Programmable Gate Arrays (FPGAs). The authors explain how event cameras, which only capture changes in a scene rather than full video frames, can be efficiently processed on reconfigurable FPGA hardware for embedded vision applications.

The paper covers the fundamental principles of event-based sensing, the architectural features of FPGAs that make them well-suited for this task, and the various event-processing techniques that have been developed. It also explores a range of real-world applications, from object tracking to gesture recognition, that leverage the low-latency and low-power benefits of this approach.

By synthesizing the latest research in this area, the survey serves as a valuable resource for both academic and industry researchers working on embedded vision systems. It highlights the significant potential of combining event-based sensing with reconfigurable hardware to create highly efficient and responsive visual perception capabilities for a wide range of applications.