Neuromorphic encoding of image pixel data into rate-coded optical spike trains with a photonic VCSEL-neuron

In recent years, the fields of artificial intelligence (AI) and machine learning (ML) have been growing rapidly, fueling the interest and research efforts in unconventional, “beyond von Neumann” computing hardware. Neuromorphic computing is among these promising new approaches, taking inspiration from the extraordinary computational power of the human brain. Neuromorphic (brain-inspired) systems aim at processing information by reproducing the massively parallel architecture and dynamics and signaling observed in the neuronal networks in the brain. Current neuromorphic computing platforms include, among others, Loihi by Intel,¹ SpiNNaker by the University of Manchester,² TrueNorth by IBM,³ Akida by BrainChip,⁴ and BrainScaleS by the University of Heidelberg.⁵ These platforms exhibit a varying degree of biological plausibility and hold great promise for efficient operation of AI algorithms or computational neuroscientific models.⁶ There is also a wide range of novel experimental technologies that can power neuromorphic systems, including memristors^7,8 and nano-oscillators,^9,10 among others. Besides the aforementioned fully electronic approaches, photonic realizations of neuromorphic hardware are attracting increasing interest, given the key inherent advantages of optical systems.¹¹ Examples of these include signaling via optical pulses that have non-interacting bosonic nature and allow for both low-loss, high-speed waveguiding and wavelength-division multiplexing, which allows us to increase communication capacity. Photonic devices can also operate at very high frequencies, can exhibit the nonlinear dynamical responses required for neuromimetic information processing, and offer promising prospects for low power, energy-efficient operation. A wide array of photonic devices is currently being investigated for neuromorphic architectures. These include micropillar lasers,¹² distributed feedback lasers,^13,14 electro-optic modulators,¹⁵ micro-ring resonators with phase change materials,¹⁶ Kerr microcomb sources,¹⁷ quantum-dot lasers,¹⁸ hybrid resonant tunneling diode circuits,^19–21 superconducting Josephson junctions,²² or coherent meshes of Mach–Zehnder modulators (MZM).²³ Among the available photonic technologies for light-powered neuromorphic processing systems, VCSELs (vertical-cavity surface emitting lasers) are gathering significant research interest due to their mature fabrication technology and favorable inherent attributes (such as low-power and high-speed operation, reduced costs, and compactness). Recent experimental^24–26 and numerical^27,28 works have reported that commercially available VCSELs and VCSELs with integrated saturable absorbing regions are capable of eliciting controllable spiking responses and exhibit behavior analogous to the leaky integrate-and-fire (LIF) neuronal model.²⁹ Crucially, VCSELs also exhibit the capability of information representation using spiking rate coding,³⁰ as observed in certain classes of biological neurons. This functionality yields them as a promising solution for interfacing and conversion of data into a suitable spike-based representation, a key challenge in the field of neuromorphic engineering.³¹ 

One of the domains where neuromorphic approaches are attracting growing interest is event-based computer vision systems. These offer many attractive features, such as low power consumption due to the sparse, low redundancy information representation via events (spikes) and high temporal resolution with continuous response, all while not suffering from drawbacks such as motion blur.³² These systems readily yield themselves for combination with deep learning approaches,³³ convolutional neural networks,³⁴ and spiking neural networks³⁵ for fast, efficient computer vision solutions. While there is a growing appeal for photonic neuromorphic computing, the suitability of optical brain-inspired spiking hardware for computer vision remains a mostly unexplored territory. The human eye features two types of photosensitive cells, cones and rods, which convert the incoming visual stimuli into electric signals that are transmitted to neurons in the retina and the inner brain for processing and interpretation. The former have fixed wavelength range operation, whereas the latter are more sensitive to light intensity. In this work, we propose a system using a single VCSEL operating as a spiking photonic neuron, which emulates the operation of a cone cell. We demonstrate that the VCSEL-neuron is able to encode the R/G/B color channel intensity from a spatial area of interest (a camera pixel) into a continuous train of spikes, where the color component intensity dictates the spiking rate. Moreover, using time-division multiplexing (TDM) techniques, we are able to process (at different time instants) complex greyscale (GS) and RGB-color source images with a large number of pixels using just one VCSEL-neuron. Therefore, this work experimentally demonstrates a highly hardware-friendly neuromorphic photonic system for the spiking representation of standard digital images, where the pixel values of each color channel are encoded into rate-coded optical spike trains. Furthermore, our implementation uses an off-the-shelf VCSEL and fiber-optic components operating at the standard telecom wavelength of 1310 nm, therefore making our approach fully compatible with optical communication networks and data center technologies. The reported system offers enticing prospects for operation as an input layer device in future photonic spiking neural network architectures for light-enabled machine learning and artificial intelligence hardware.

The VCSEL-based photonic neuron operates as an analog, coherent optical device. The information input is provided as an intensity-modulated light signal from an external laser source. The input data (R/G/B digital images) are pre-processed prior to injection into the VCSEL-neuron, as shown in Sec. II A. The VCSEL-neuron encodes the pixel values into a spike-based representation, which is later reconstructed following the steps described in Sec. II B. Finally, the optical setup used to operate the VCSEL-neuron is described in Sec. II C. In a fully parallelized realization of a neuromorphic vision system, a single VCSEL-neuron would be used to encode the information of a single color pixel of the input image. For such an implementation, the number of required VCSEL-neurons (3 · N² VCSEL-neurons in the input layer for RGB square images with size N × N) grows very quickly even for small image sizes. Instead, we take the approach of sequentially processing the pixels by means of time-division multiplexing (TDM). This allows us to dramatically reduce the number of required VCSELs to just a single device. Therefore, using this technique, we are able to deliver a neuromorphic photonic system for spike-based image data encoding, benefiting from a hardware-friendly implementation while yielding high operational speeds for image encoding functionalities. For future larger-scale implementations of neuromorphic photonic systems, integrated VCSEL arrays³⁶ offer the promise of network scalability and parallelized operation.

In this work, the VCSEL-neuron encodes pixel information from digital images into spike trains. Multi-channel RGB images can be split into individual color channels, where each channel can be processed independently either serially by implementing time-division multiplexing step or in parallel using individual VCSEL-neurons for each color channel. Here, all the inspected color channels are processed serially to allow for the use of a single VCSEL-neuron for the processing of the complete color information. A single color channel of the image is represented as an N × N array of independent pixel values with standard eight-bit depth of pixel data. The pixels in the single-channel array are serialized to enable processing with a single VCSEL-neuron device, thus critically reducing the required experimental system complexity. The pixel serialization process is realized either on a row-by-row basis or using randomized processing order, where the n pixels are shuffled before injection into the VCSEL-neuron. Each of the n pixels is time multiplexed using a fixed time period T = T_P + T₀, referred to, in this work, as a single cycle. During this cycle, single input pixel information is injected into the VCSEL-neuron in the form of square wave shaped power drop with temporal length T_P, followed by a quiescent “zero” state for the remaining time period T₀ (effectively realizing a return-to-zero coding scheme). In the power drop, the amplitude A_n represents the pixel color intensity. The mapping between each pixel value P_n and its respective drop amplitude A_n is linear across the non-zero values, where the min–max range of the produced waveform corresponds to the available digital-to-analog converter (DAC) range. Pixel values P_n = 0 are assigned as A_n = 0. For eight-bit pixel values, the mapping can be described as

where b sets the baseline value of the linear mapping for non-zero pixel values. In this work, we selected b = 0.4. This mapping was selected to achieve a fully injection-locked dynamical state of the VCSEL-neuron for zero (dark) pixel values and a gradually increasing perturbation depth for all non-zero pixel color values. This can be considered as a basic model of a neuromorphic vision system studied over time period [0, T]. In this model, a single cycle of length T represents the behavior of a photodetector (PD) (or a single-pixel camera) exposed to constant light intensity during [O, T_P], while subject to darkness (P = 0) otherwise. The electric output of the photodetector is then processed by the VCSEL-neuron into a rate-coded train of spikes in the optical domain. Without light exposure, the detector receives no signal and the spiking neuron remains quiescent [Fig. 1(c)].

The temporal evolution of the system plays a crucial role for representation of information in spiking neural networks. Therefore, a 2D pixel array response over a given time period would ideally require a 3D visualization. For simplicity, we instead show the response of each individual pixel (a single cycle) as a 2D temporal map. In each of the n recorded cycles, the produced spikes are counted as individual events that cross a given measurement threshold value, returning the cycle spike counts c_n. Every channel is injected into the VCSEL multiple times under the same set of conditions to obtain an average spike count value $cn̄$⁠, delivering a clearer representation of the system’s response. The average spike count is converted into an average spiking rate as $rn=cn̄TP$⁠. The averaging step is performed over multiple independent encoding trials. This value is normalized by the maximum measured spiking rate in the image max(r_n) and directly converted into a component intensity value for given pixel as $Pn=rnmax(rn)$⁠. In the final step, the reconstructed pixel intensity values $Pn$ are assigned to their original positions in the image pixel array, creating a direct reconstruction of the source image.

The experimental setup of this work is shown in Fig. 1(a). The setup consists of off-the-shelf fiber-optic components, where a commercially available telecom-wavelength (1310 nm) VCSEL is used to implement the photonic spiking neuron. The VCSEL is operated at a constant temperature of 298 K and with an applied bias current of I = 5 mA well above the device’s lasing threshold current (I_thr = 2.9 mA). The VCSEL’s spectrum [shown in Fig. 1(b)] reveals a characteristic two-peak emission that corresponds to the two orthogonal polarizations of the main transverse mode of the device. These are referred to as parallel (the main lasing peak, at λ_y) and orthogonal (the subsidiary attenuated peak, at λ_x) polarized modes. The VCSEL used in this work does not exhibit polarization switching (PS) with applied bias current.

The operation of the VCSEL-neuron relies on coherent injection locking of one of the VCSEL’s polarized modes to an intensity-modulated signal from an external tunable laser source (TL, Santec TSL210V), representing the information (image data) input to the system. The intensity of the tunable laser light represents the image pixel information input to the VCSEL-neuron. We want to emphasize that the VCSEL used in this work as an artificial spiking photonic neuron is a commercially sourced, inexpensive device operating at the standard telecom wavelength of 1310 nm. No modifications of any kind to the factory design were implemented. In this work, we injected the external light signal (encoded with image information) in the subsidiary orthogonally polarized mode to achieve the characteristic high-speed spiking operation. Light intensity modulation is realized with a 10 GHz Mach–Zehnder modulator (MZM) controlled by RF signals generated with a 12 GSa s⁻¹ arbitrary waveform generator (AWG, Keysight M8190A). Amplification of the electric modulation signal is provided by a 10 dBm wide bandwidth amplifier (Mini-Circuits ZX60-14012L-S+). First, the externally injected signal from the TL signal passes through an optical isolator (ISO) to ensure unidirectional coupling. The signal is attenuated with a variable optical attenuator (VOA) and polarization matched to the MZM using a polarization controller (PC). The operation point of the MZM is set to a working point between the quadrature and the peak, where the electric modulation results in optical power drops while preserving the full dynamic range and approximately linear response with high output power. The modulated optical signal is polarization matched to one of the polarized modes of the VCSEL (here, into the orthogonally polarized mode at λ_x) and injected into the laser through a 50:50 optical coupler and an optical circulator (CIRC). The second coupler branch is used to record the average injection power P_inj using an optical power meter (PM). The input signal wavelength is shifted with respect to the λ_x mode by an initial frequency detuning δ_f. The detuning and injection power allow for adjustments of the spiking threshold in the VCSEL-neuron, with higher detuning and lower power values decreasing the threshold distance from the quiescent state (where the VCSEL is injection locked to the external injection). Input stimuli (power drops) that are strong enough to push the system out of the injection locking regime result in the VCSEL-neuron responding with fast (sub-nanosecond) optical spiking responses, including single spikes³⁷ [Fig. 1(d)] and continuous spike trains³⁰ [Fig. 1(e)]. Following the behavior of excitable systems, the spike shape does not depend on the input perturbation and only requires the perturbation (drop) to cross the spiking threshold. However, during continuous modulation, the spiking frequency is a function of modulation amplitude. The detuning and power values used through this work (indicated in the figure captions) were found experimentally based on values used in previous works (see Refs. 30 and 38). The VCSEL output passes through the circulator into a 50:50 coupler for analysis using an optical spectrum analyzer (OSA) and a 13 GHz real-time oscilloscope (RT OSC). For the oscilloscope readout, the optical signal is converted using a 9 GHz amplified photodetector (PD, Thorlabs PDA8GS). The modulated signal injected into the VCSEL-neuron is recorded from the power-meter (PM) branch. The response of the VCSEL-neuron is then recorded and time de-multiplexed to retrieve responses corresponding to individual serialized pixels.

To assess the viability of the VCSEL-neuron for encoding pixel color intensity information into optical spike trains, we investigate its operation using multiple input digital images. We use a single-channel grayscale (GS) image containing a vertical gradient, a two-channel RB image containing a two-color (red-to-blue) diagonal gradient, and a complex, 32 × 32 pixel RGB image.

The source images, individual channels, and corresponding input modulation waveforms for the grayscale (GS) image and red (R) and blue (B) channels in the RB image are shown in Fig. 2. Figure 3 provides the experimentally measured results for the vertical GS gradient image. The input modulation waveform, shown for the GS image in Fig. 2(c), carries the pixel values in the amplitude of negative pulses with temporal duration T_p = T₀ = 60 ns, resulting in a total encoding time of 7.68 µs. For this measurement, the frequency detuning was δ_f = −6.29 GHz and injected power was P_inj = 182.3 μW. Based on the mapping formula shown in Eq. (1), the first eight black pixels encode into zero amplitude power drops (no drops), keeping the VCSEL-neuron in a quiescent state. As the pixel intensity gradually changes from black (0) to white (255), the amplitude of the injected modulated power drops increases, pushing the VCSEL-neuron beyond the spiking activation threshold into the continuous spiking regime. The approximate spiking threshold is illustrated as a red solid line in Fig. 3(a). The response for the GS modulation case is shown in the temporal map in Fig. 3(b), where each row in the map corresponds to the response to a single given pixel in the GS image. The major gridlines in the map in Fig. 3(b) separate groups of eight processed pixels, hence visualizing each individual row of the 8 × 8 pixel image. In the case of the GS image, all eight pixels in a given row have an identical value and should therefore produce equivalent spiking rates. Similar to the stochastic nature of certain biological neurons, where spike firing can be considered as governed by Poisson statistics,³⁹ the individual spiking events can appear at random instants while still correctly representing the input GS image data in the average spiking rate, which increases monotonically with the input pixel intensity.³⁰ This is illustrated in both the increasing density of the events (spikes) in the temporal map of Fig. 3(b) and the increase in the measured spiking rate shown in Fig. 3(c). Specifically, the spiking rates in Fig. 3(c) are obtained by averaging the rates from n_meas = 3 independent oscilloscope readouts taken for the same GS image input under the same operating conditions. The different colors used in Fig. 3(c) highlight the spiking rates produced by the pixels from each individual row in the image. Figure 3(d) shows the relation between the amplitude of the injected optical power drops encoding the pixel GS intensity and the corresponding average spiking rate from the VCSEL-neuron, clearly confirming a monotonically increasing trend. Note that for certain pixels at the start of the rows [see, for instance, rows depicted in purple, pink, and green in Fig. 3(c)], we observed a sudden, higher-than-expected spiking rate that gradually decreases toward a constant value. We believe this transient effect is due to the stateful (memory exhibiting) nature of the VCSEL-neuron, where the response to an incoming perturbation depends on the previous state of the device. This is in agreement with earlier reports on the leaky integrate-and-fire (LIF) functionality in VCSEL-neurons.²⁹ High degree of consistency was observed among the readouts used for the averaging, confirming that the VCSEL-neuron can process input images reliably in a single shot fashion.

To demonstrate the optical spike encoding of multi-channel images with the VCSEL-neuron, we processed the red–blue (RB) diagonal gradient image shown in Fig. 2. The individual red (RB-R) and blue (RB-B) image channels and their corresponding input waveforms are shown, respectively, in the middle (d)–(f) and bottom (g)–(i) rows of Fig. 2. The frequency detuning set for both the RB-R and RB-B channels was equal to δ_f = −5.59 GHz, while the injected power was set to P_inj = 115.6 μW and P_inj = 128.4 μW for the RB-R and RB-B channels, respectively. Each individual pulse (representing single-pixel information) in both modulation waveforms for the RB-R and RB-B channels has a set temporal length T_p = 60 ns with a return-to-zero period of T₀ = 30 ns, resulting in an encoding time of 5.76 µs per a single iteration of a channel. For both RB-R and RB-B cases, the VCSEL-neuron is able to successfully encode the pixel intensity of each color channel into the average spiking rate. Notably, we can again observe the short-term memory of the VCSEL-neuron influencing the continuous spiking responses, with low-to-high transitions in the RB-R channel causing over-firing with higher-than-expected spiking frequency, while high-to-low transitions in the RB-B channel do not exhibit such an effect. Therefore, abrupt, high-amplitude incoming stimuli, arriving after a quiescent or a low spiking rate state, can cause an accentuated response with an increased spiking frequency output. This functionality could hold further promise for spike-based signaling of rapidly changing events.

To verify the resulting encoded images obtained from the VCSEL-neuron, we have directly reconstructed the source images from the average spiking rates shown in Figs. 3(c), 4(c), and 4(g). Figure 5 shows the composites and individual channels for both the GS and RB images, with source images shown in the top row and reconstructed images shown in the bottom row. For all considered cases, both the individual channels and composite images show very good overall agreement, clearly preserving the color gradients.

Furthermore, we have also investigated the processing of larger-scale full-color images using a 32 × 32 pixel RGB image with more complex features. The pixels are processed in a fixed randomized order, where the same order was used for all three color channels. The input waveform pulse parameters were T_p = 53.3 ns and T₀ = 26.6 ns, resulting in 81.9 µs single-channel processing time. The frequency detuning was set to δ_f = −4.9 GHz for RGB-R, δ_f = −4.9 GHz for RGB-G, and δ_f = −5.25 GHz for RGB-B. The injected power was set to P_inj = 120.9 μW for RGB-R, P_inj = 119.6 μW for RGB-G, and P_inj = 121.5 μW for RGB-B. The spike trains were reconstructed using the same methodology as previously used for the GS and RB images in Figs. 3 and 4. Due to the high number of processed pixels (n = 32² = 1024), the modulation waveform and temporal maps are omitted from the depicted results for this case. The response spiking frequencies as a function of input pixel values for the three channels (R/G/B color) are shown as color dots in Fig. 6. The latter also shows the mean spiking rate and the standard deviation for each set of equal pixel values with number of occurrences n_occ > 3. The dashed curve connecting the mean spiking rate values is considered as a tone curve of the system. The mean values in Fig. 6 exhibit a clear, monotonically increasing relation between the source value and the produced spiking rate. The source and reconstructed images for the individual color channels (R, G, and B) and the composite RGB image are included for comparison in Fig. 7, showing again very good overall agreement. The degree of variance observed in the plots in Fig. 6 (and the color noise observed in the reconstruction in Fig. 7) can be attributed to the random pixel addressing process, which distributes the memory-induced transient rate increase effect across the image. The reconstructed composite image, combining all the independently processed R/G/B channels, is in good agreement with the source and preserves the key features from the original image.

This work demonstrates that a VCSEL-based photonic spiking neuron can be used to encode digital image data into optical spike trains with variable spike firing rates, therefore exhibiting the behavior of certain biological neurons at speeds that are multiple orders of magnitude faster. We use this functionality to rate-encode and reconstruct grayscale and RGB color digital images, showing, in all cases, very good agreement between source and reconstructed image data. Use of time-division multiplexing allows us to dramatically reduce system requirements, permitting operation with a single VCSEL and enabling an extremely hardware-friendly implementation. This approach yields a simple neuromorphic photonic platform serving as a data encoder (interface) for spike-based machine vision hardware, which is also able to deliver high speed, thanks to the GHz-rate optical spikes, low timescales (active encoding time of 50–60 ns per pixel is used in this work), and low energy consumption. For future approaches, we envision systems using multiple independent and array-integrated VCSEL-neurons for parallel pixel processing operation for faster operation speeds and enhanced functionalities. Our results further demonstrate that the VCSEL-neuron implementation, built with a commercially available and telecom-compatible 1310 nm VCSEL and off-the-shelf fiber-optic components, can realize spike-rate information encoding of complex analog (image) data streams, yielding itself as a viable option for operation as an input layer device in future photonic spiking neural network hardware architectures.

The authors would like to thank Professor Thorsten Ackemann and Professor Alan Kemp (University of Strathclyde) for lending some of the equipment used in this work. The authors would also like to thank Mason Warren for providing the 32 × 32 RGB image.

The authors acknowledge funding support from the European Commission (Grant No. 828841 ChipAI-H2020-FETOPEN-2018–2020), the Office of Naval Research Global (Grant No. ONRGNICOPN62909-18-1-2027), the UK Engineering and Physical Sciences Research Council Doctoral Training Partnership (Grant No. EP/N509760), and the UKRI Turing AI Acceleration Fellowships Programme (Grant No. EP/V025198/1).

The data that support the findings of this study are openly available in the University of Strathclyde’s KnowledgeBase at http://doi.org/10.15129/b312473f-1d2d-4104-a2d9-0014edf191bd, Ref. 40.