-
Generalized Event Cameras
Authors:
Varun Sundar,
Matthew Dutson,
Andrei Ardelean,
Claudio Bruschini,
Edoardo Charbon,
Mohit Gupta
Abstract:
Event cameras capture the world at high time resolution and with minimal bandwidth requirements. However, event streams, which only encode changes in brightness, do not contain sufficient scene information to support a wide variety of downstream tasks. In this work, we design generalized event cameras that inherently preserve scene intensity in a bandwidth-efficient manner. We generalize event cam…
▽ More
Event cameras capture the world at high time resolution and with minimal bandwidth requirements. However, event streams, which only encode changes in brightness, do not contain sufficient scene information to support a wide variety of downstream tasks. In this work, we design generalized event cameras that inherently preserve scene intensity in a bandwidth-efficient manner. We generalize event cameras in terms of when an event is generated and what information is transmitted. To implement our designs, we turn to single-photon sensors that provide digital access to individual photon detections; this modality gives us the flexibility to realize a rich space of generalized event cameras. Our single-photon event cameras are capable of high-speed, high-fidelity imaging at low readout rates. Consequently, these event cameras can support plug-and-play downstream inference, without capturing new event datasets or designing specialized event-vision models. As a practical implication, our designs, which involve lightweight and near-sensor-compatible computations, provide a way to use single-photon sensors without exorbitant bandwidth costs.
△ Less
Submitted 2 July, 2024;
originally announced July 2024.
-
Event Cameras Meet SPADs for High-Speed, Low-Bandwidth Imaging
Authors:
Manasi Muglikar,
Siddharth Somasundaram,
Akshat Dave,
Edoardo Charbon,
Ramesh Raskar,
Davide Scaramuzza
Abstract:
Traditional cameras face a trade-off between low-light performance and high-speed imaging: longer exposure times to capture sufficient light results in motion blur, whereas shorter exposures result in Poisson-corrupted noisy images. While burst photography techniques help mitigate this tradeoff, conventional cameras are fundamentally limited in their sensor noise characteristics. Event cameras and…
▽ More
Traditional cameras face a trade-off between low-light performance and high-speed imaging: longer exposure times to capture sufficient light results in motion blur, whereas shorter exposures result in Poisson-corrupted noisy images. While burst photography techniques help mitigate this tradeoff, conventional cameras are fundamentally limited in their sensor noise characteristics. Event cameras and single-photon avalanche diode (SPAD) sensors have emerged as promising alternatives to conventional cameras due to their desirable properties. SPADs are capable of single-photon sensitivity with microsecond temporal resolution, and event cameras can measure brightness changes up to 1 MHz with low bandwidth requirements. We show that these properties are complementary, and can help achieve low-light, high-speed image reconstruction with low bandwidth requirements. We introduce a sensor fusion framework to combine SPADs with event cameras to improves the reconstruction of high-speed, low-light scenes while reducing the high bandwidth cost associated with using every SPAD frame. Our evaluation, on both synthetic and real sensor data, demonstrates significant enhancements ( > 5 dB PSNR) in reconstructing low-light scenes at high temporal resolution (100 kHz) compared to conventional cameras. Event-SPAD fusion shows great promise for real-world applications, such as robotics or medical imaging.
△ Less
Submitted 17 April, 2024;
originally announced April 2024.
-
SoDaCam: Software-defined Cameras via Single-Photon Imaging
Authors:
Varun Sundar,
Andrei Ardelean,
Tristan Swedish,
Claudio Bruschini,
Edoardo Charbon,
Mohit Gupta
Abstract:
Reinterpretable cameras are defined by their post-processing capabilities that exceed traditional imaging. We present "SoDaCam" that provides reinterpretable cameras at the granularity of photons, from photon-cubes acquired by single-photon devices. Photon-cubes represent the spatio-temporal detections of photons as a sequence of binary frames, at frame-rates as high as 100 kHz. We show that simpl…
▽ More
Reinterpretable cameras are defined by their post-processing capabilities that exceed traditional imaging. We present "SoDaCam" that provides reinterpretable cameras at the granularity of photons, from photon-cubes acquired by single-photon devices. Photon-cubes represent the spatio-temporal detections of photons as a sequence of binary frames, at frame-rates as high as 100 kHz. We show that simple transformations of the photon-cube, or photon-cube projections, provide the functionality of numerous imaging systems including: exposure bracketing, flutter shutter cameras, video compressive systems, event cameras, and even cameras that move during exposure. Our photon-cube projections offer the flexibility of being software-defined constructs that are only limited by what is computable, and shot-noise. We exploit this flexibility to provide new capabilities for the emulated cameras. As an added benefit, our projections provide camera-dependent compression of photon-cubes, which we demonstrate using an implementation of our projections on a novel compute architecture that is designed for single-photon imaging.
△ Less
Submitted 8 September, 2023; v1 submitted 31 August, 2023;
originally announced September 2023.
-
Coupling a Recurrent Neural Network to SPAD TCSPC Systems for Real-time Fluorescence Lifetime Imaging
Authors:
Yang Lin,
Paul Mos,
Andrei Ardelean,
Claudio Bruschini,
Edoardo Charbon
Abstract:
Fluorescence lifetime imaging (FLI) has been receiving increased attention in recent years as a powerful diagnostic technique in biological and medical research. However, existing FLI systems often suffer from a tradeoff between processing speed, accuracy, and robustness. In this paper, we propose a robust approach that enables fast FLI with no degradation of accuracy. The approach is based on a S…
▽ More
Fluorescence lifetime imaging (FLI) has been receiving increased attention in recent years as a powerful diagnostic technique in biological and medical research. However, existing FLI systems often suffer from a tradeoff between processing speed, accuracy, and robustness. In this paper, we propose a robust approach that enables fast FLI with no degradation of accuracy. The approach is based on a SPAD TCSPC system coupled to a recurrent neural network (RNN) that accurately estimates the fluorescence lifetime directly from raw timestamps without building histograms, thereby drastically reducing transfer data volumes and hardware resource utilization, thus enabling FLI acquisition at video rate. We train two variants of the RNN on a synthetic dataset and compare the results to those obtained using center-of-mass method (CMM) and least squares fitting (LS fitting). Results demonstrate that two RNN variants, gated recurrent unit (GRU) and long short-term memory (LSTM), are comparable to CMM and LS fitting in terms of accuracy, while outperforming them in background noise by a large margin. To explore the ultimate limits of the approach, we derived the Cramer-Rao lower bound of the measurement, showing that RNN yields lifetime estimations with near-optimal precision. Moreover, our FLI model, which is purely trained on synthetic datasets, works well with never-seen-before, real-world data. To demonstrate real-time operation, we have built a FLI microscope based on Piccolo, a 32x32 SPAD sensor developed in our lab. Four quantized GRU cores, capable of processing up to 4 million photons per second, are deployed on a Xilinx Kintex-7 FPGA. Powered by the GRU, the FLI setup can retrieve real-time fluorescence lifetime images at up to 10 frames per second. The proposed FLI system is promising and ideally suited for biomedical applications.
△ Less
Submitted 24 July, 2023; v1 submitted 27 June, 2023;
originally announced June 2023.
-
Scalable multi-chip quantum architectures enabled by cryogenic hybrid wireless/quantum-coherent network-in-package
Authors:
Eduard Alarcón,
Sergi Abadal,
Fabio Sebastiano,
Masoud Babaie,
Edoardo Charbon,
Peter Haring Bolívar,
Maurizio Palesi,
Elena Blokhina,
Dirk Leipold,
Bogdan Staszewski,
Artur Garcia-Sáez,
Carmen G. Almudever
Abstract:
The grand challenge of scaling up quantum computers requires a full-stack architectural standpoint. In this position paper, we will present the vision of a new generation of scalable quantum computing architectures featuring distributed quantum cores (Qcores) interconnected via quantum-coherent qubit state transfer links and orchestrated via an integrated wireless interconnect.
The grand challenge of scaling up quantum computers requires a full-stack architectural standpoint. In this position paper, we will present the vision of a new generation of scalable quantum computing architectures featuring distributed quantum cores (Qcores) interconnected via quantum-coherent qubit state transfer links and orchestrated via an integrated wireless interconnect.
△ Less
Submitted 8 April, 2023; v1 submitted 24 March, 2023;
originally announced March 2023.
-
A 3.3 Gbps SPAD-Based Quantum Random Number Generator
Authors:
Pouyan Keshavarzian,
Karthick Ramu,
Duy Tang,
Carlos Weill,
Francesco Gramuglia,
Shyue Seng Tan,
Michelle Tng,
Louis Lim,
Elgin Quek,
Denis Mandich,
Mario Stipčević,
Edoardo Charbon
Abstract:
Quantum random number generators are a burgeoning technology used for a variety of applications, including modern security and encryption systems. Typical methods exploit an entropy source combined with an extraction or bit generation circuit in order to produce a random string. In integrated designs there is often little modelling or analytical description of the entropy source, circuit extractio…
▽ More
Quantum random number generators are a burgeoning technology used for a variety of applications, including modern security and encryption systems. Typical methods exploit an entropy source combined with an extraction or bit generation circuit in order to produce a random string. In integrated designs there is often little modelling or analytical description of the entropy source, circuit extraction and post-processing provided. In this work, we first discuss theory on the quantum random flip-flop (QRFF), which elucidates the role of circuit imperfections that manifest themselves in bias and correlation. Then, a Verilog-AMS model is developed in order to validate the analytical model in simulation. A novel transistor implementation of the QRFF circuit is presented, which enables compensation of the degradation in entropy inherent to the finite non-symmetric transitions of the random flip-flop. Finally, a full system containing two independent arrays of the QRFF circuit is manufactured and tested in a 55 nm Bipolar-CMOS-DMOS (BCD) technology node, demonstrating bit generation statistics that are commensurate to the developed model. The full chip is able to generate 3.3 Gbps of data when operated with an external LED, whereas an individual QRFF can generate 25 Mbps each of random data while maintaining a Shannon entropy bound > 0.997, which is one of the highest per pixel bit generation rates to date. NIST STS is used to benchmark the generated bit strings, thereby validating the QRFF circuit as an excellent candidate for fully-integrated QRNGs.
△ Less
Submitted 11 September, 2022;
originally announced September 2022.
-
A High Speed Integrated Quantum Random Number Generator with on-Chip Real-Time Randomness Extraction
Authors:
Francesco Regazzoni,
Emna Amri,
Samuel Burri,
Davide Rusca,
Hugo Zbinden,
Edoardo Charbon
Abstract:
The security of electronic devices has become a key requisite for the rapidly-expanding pervasive and hyper-connected world. Robust security protocols ensuring secure communication, device's resilience to attacks, authentication control and users privacy need to be implemented. Random Number Generators (RNGs) are the fundamental primitive in most secure protocols but, often, also the weakest one.…
▽ More
The security of electronic devices has become a key requisite for the rapidly-expanding pervasive and hyper-connected world. Robust security protocols ensuring secure communication, device's resilience to attacks, authentication control and users privacy need to be implemented. Random Number Generators (RNGs) are the fundamental primitive in most secure protocols but, often, also the weakest one. Establishing security in billions of devices requires high quality random data generated at a sufficiently high throughput. On the other hand, the RNG should exhibit a high integration level with on-chip extraction to remove, in real time, potential imperfections. We present the first integrated Quantum RNG (QRNG) in a standard CMOS technology node. The QRNG is based on a parallel array of independent Single-Photon Avalanche Diodes (SPADs), homogeneously illuminated by a DC-biased LED, and co-integrated logic circuits for postprocessing. We describe the randomness generation process and we prove the quantum origin of entropy. We show that co-integration of combinational logic, even of high complexity, does not affect the quality of randomness. Our CMOS QRNG can reach up to 400 Mbit/s throughput with low power consumption. Thanks to the use of standard CMOS technology and a modular architecture, our QRNG is suitable for a highly scalable solution.
△ Less
Submitted 11 February, 2021;
originally announced February 2021.
-
Quanta Burst Photography
Authors:
Sizhuo Ma,
Shantanu Gupta,
Arin C. Ulku,
Claudio Bruschini,
Edoardo Charbon,
Mohit Gupta
Abstract:
Single-photon avalanche diodes (SPADs) are an emerging sensor technology capable of detecting individual incident photons, and capturing their time-of-arrival with high timing precision. While these sensors were limited to single-pixel or low-resolution devices in the past, recently, large (up to 1 MPixel) SPAD arrays have been developed. These single-photon cameras (SPCs) are capable of capturing…
▽ More
Single-photon avalanche diodes (SPADs) are an emerging sensor technology capable of detecting individual incident photons, and capturing their time-of-arrival with high timing precision. While these sensors were limited to single-pixel or low-resolution devices in the past, recently, large (up to 1 MPixel) SPAD arrays have been developed. These single-photon cameras (SPCs) are capable of capturing high-speed sequences of binary single-photon images with no read noise. We present quanta burst photography, a computational photography technique that leverages SPCs as passive imaging devices for photography in challenging conditions, including ultra low-light and fast motion. Inspired by recent success of conventional burst photography, we design algorithms that align and merge binary sequences captured by SPCs into intensity images with minimal motion blur and artifacts, high signal-to-noise ratio (SNR), and high dynamic range. We theoretically analyze the SNR and dynamic range of quanta burst photography, and identify the imaging regimes where it provides significant benefits. We demonstrate, via a recently developed SPAD array, that the proposed method is able to generate high-quality images for scenes with challenging lighting, complex geometries, high dynamic range and moving objects. With the ongoing development of SPAD arrays, we envision quanta burst photography finding applications in both consumer and scientific photography.
△ Less
Submitted 21 June, 2020;
originally announced June 2020.
-
A Bit Too Much? High Speed Imaging from Sparse Photon Counts
Authors:
Paramanand Chandramouli,
Samuel Burri,
Claudio Bruschini,
Edoardo Charbon,
Andreas Kolb
Abstract:
Recent advances in photographic sensing technologies have made it possible to achieve light detection in terms of a single photon. Photon counting sensors are being increasingly used in many diverse applications. We address the problem of jointly recovering spatial and temporal scene radiance from very few photon counts. Our ConvNet-based scheme effectively combines spatial and temporal informatio…
▽ More
Recent advances in photographic sensing technologies have made it possible to achieve light detection in terms of a single photon. Photon counting sensors are being increasingly used in many diverse applications. We address the problem of jointly recovering spatial and temporal scene radiance from very few photon counts. Our ConvNet-based scheme effectively combines spatial and temporal information present in measurements to reduce noise. We demonstrate that using our method one can acquire videos at a high frame rate and still achieve good quality signal-to-noise ratio. Experiments show that the proposed scheme performs quite well in different challenging scenarios while the existing approaches are unable to handle them.
△ Less
Submitted 11 May, 2019; v1 submitted 6 November, 2018;
originally announced November 2018.
