What is it about?
We apply reinforcement learning to video compressive sensing to adapt the compression ratio. Specifically, video snapshot compressive imaging (SCI), which captures high-speed video using a low-speed camera, is considered in this work, in which multiple (B) video frames can be reconstructed from a snapshot measurement. One research gap in previous studies is how to adapt B in the video SCI system for different scenes. In this paper, we fill this gap by utilizing reinforcement learning (RL). An RL model, as well as various convolutional neural networks for reconstruction, are learned to achieve adaptive sensing of video SCI systems. Furthermore, the performance of an object detection network using directly the video SCI measurements without reconstruction is also used to perform RL-based adaptive video compressive sensing. Our proposed adaptive SCI method can thus be implemented at low cost and in real-time. Our work takes the technology one step further towards real applications of video SCI.
Featured Image
Photo by Remy Gieling on Unsplash
Why is it important?
As to the connected and autonomous vehicles, since the driving scenes being captured are dynamic and vehicle speed varies over time, it is imperative to realize adaptive video SCI for real-world applications, i.e., automatically determining the optimal B (B refers to the compression ratio) under different application environments. Take measurement-based object detection as an example, when the vehicle is driving under a slow-motion scenario, increasing B can further accelerate inference speed while still guaranteeing a high measurement-based detection accuracy. In contrast, if the driving environment changes rapidly, e.g., a red traffic light or an accident suddenly halts all vehicles, B should be decreased to avoid missing high-speed information. Therefore, it is the right time to take the developments further and make SCI systems suitable for real applications.
Perspectives
This research is to make Video Snapshot Compressive Imaging (SCI) adaptive to different scenes and more suitable for real-world applications, especially in areas such as connected and autonomous vehicles. It was clear that the driving scenes being captured are dynamic, and vehicle speeds vary over time. We recognized that when the vehicle is moving slowly, the compression ratio of camera video images can be increased, speeding up the inference while still maintaining high accuracy. Conversely, in rapidly changing environments, the compression ratio should be reduced to avoid missing essential high-speed information. Being able to successfully devise an RL model that learns this adaptability is one of the highlights of our work. This paper is just the beginning, and we hope that our findings will serve as a foundation for future research in this field, furthering our understanding and real-world applications of adaptive SCI.
Sidi Lu
College of William and Mary
Read the Original
This page is a summary of: Reinforcement Learning for Adaptive Video Compressive Sensing, ACM Transactions on Intelligent Systems and Technology, July 2023, ACM (Association for Computing Machinery),
DOI: 10.1145/3608479.
You can read the full text:
Contributors
The following have contributed to this page
