We are thrilled to share our groundbreaking paper published today in
@Nature: "Low Latency Automotive Vision with Event Cameras."
Paper:
nature.com/articles/s41586-0…
Video:
youtu.be/dwzGhMQCc4Y
Code & Dataset:
github.com/uzh-rpg/dagr
Frame-based sensors such as the RGB cameras used in the automotive industry face a bandwidth–latency trade-off: higher frame rates reduce perceptual latency but increase bandwidth demands, whereas lower frame rates save bandwidth at the cost of missing vital scene dynamics due to increased perceptual latency (see Fig. 1a of the paper). Event cameras have emerged as alternative vision sensors to address this trade-off. Event cameras measure the changes in intensity asynchronously, offering high temporal resolution and sparsity, markedly reducing bandwidth and latency requirements. Despite these advantages, event-camera-based algorithms are either highly efficient but lag behind image-based ones in accuracy or sacrifice the sparsity and efficiency of events to achieve comparable results. To overcome this, we propose a hybrid event- and frame-based object detector based on Deep Asynchronous GNNs, which preserves the advantages of each modality and thus does not suffer from this trade-off. Our method exploits the high temporal resolution and sparsity of events and the rich but low temporal resolution information in standard images to generate efficient, high-rate object detections, reducing perceptual and computational latency. In doing so, it emulates the slow-fast pathways in biological neural networks and uses them to its advantage. We show that using a 20-Hz RGB camera plus an event camera achieves the same latency as a 5,000-Hz camera with the bandwidth of a 50-Hz camera, i.e., an over 100-fold bandwidth reduction, without compromising accuracy. Our approach paves the way for efficient and robust perception in edge-case scenarios by uncovering the potential of event cameras. We release the code and the dataset (DSEC-Detection) to the public.
Kudos to
@DanielGehrig6, who, with this work, also received the UZH Annual Award for the Best PhD thesis!
**Reference**
Daniel Gehrig, Davide Scaramuzza
Low Latency Automotive Vision with Event Cameras
Nature, May 29, 2024.
DOI: 10.1038/s41586-024-07409-w
PDF (Open Access):
nature.com/articles/s41586-0…
Video (Narrated):
youtu.be/dwzGhMQCc4Y
Code & Datasets:
github.com/uzh-rpg/dagr
@UZH_en @UZH_Science @UZHspacehub @ERC_Research @nccrrobotics