In-sensor reservoir computing system for latent fingerprint recognition with deep ultraviolet photo-synapses and memristor array

1Fingerprint recognition is the preferred technique in biometrics which measures and analyzes the characteristics of human identity–. In most cases, latent fingerprints are not visible to the naked eye and require visualization,. The extraction and recognition of fingerprints play critical roles in personal identification, criminal investigation, and biometric security–. The quasi-zero background on the Earth’s surface and the strong absorption of deep ultraviolet (DUV) waveband by organic residues endow DUV light with a promising capability to detect and recognize latent fingerprints–. Nevertheless, the conventional ex-situ DUV fingerprint recognition systems feature separated sensors, memory, and processor, which seriously deteriorate the latency of decision-making and inevitably increase the overall computing power (Fig. 1a)–. In addition, these systems utilize additional optical filters for charge-coupled devices (CCDs) and complementary metal-oxide-semiconductor (CMOS) image sensors, increasing the complexity of the entire system for latent fingerprint identification–. Therefore, to simplify the system construction and enhance the processing efficiency of the DUV fingerprint recognition system, it is urgent to develop a new principle device and new computing architecture.

2Inspired by the biological sensory systems, neuromorphic architectures bring innovative thoughts to improve the efficiency of DUV fingerprint recognition because of their in-sensor and in-memory computing merits, relying on artificial neural network (ANN) computational models. Among various ANN models, reservoir computing (RC) only needs to train the output layer of the network and is demonstrated to be suitable for processing complex spatiotemporal data with the lowest computational cost. Recently, emerging devices have been developed for RC system to condense or process spatiotemporal information with higher energy efficiency beyond CMOS technology–. However, these RC techniques are based on electrical stimuli and thus require complex circuits to support the information transmission between the sensor and RC system, which induces extra energy consumption and latency. Fortunately, a promising strategy of in-sensor RC based on optoelectronic devices has been proposed for temporal sensory information processing and verified with the assistance of system simulation,. In order to fulfill the in-sensor applications, the optoelectronic devices should be marked by the properties of nonlinearity response, short-term memory (STM), multiple states and stability. Nevertheless, the waveband utilized in above works is not suitable for DUV detection. Ultra-wide bandgap semiconductors with their bandgap immediately corresponding to the DUV region, provide subversive scheme for filter-free DUV sensors–. Especially, gallium oxide is currently regarded as the most promising material when compressively considering the DUV sensitivity, physicochemical stability, workability, availability, and cost–. Moreover, its amorphous counterpart (a-GaOx) exhibits obvious persistent photoconductivity (PPC) effect,, catering to the implementation of DUV in-sensor RC system. Therefore, designing an a-GaOx device with enhanced PPC effect and performing system demonstration is crucial to promote the DUV in-sensor RC system for latent fingerprint recognition.

3In this work, we constructed a DUV in-sensor RC system based on photo-synapses and memristor array, mimicking the biological visual systems, for the highly efficient recognition of latent fingerprints. By Ga-rich component design, we implemented a-GaOx DUV sensors with enhanced PPC effect that illustrates nonlinearly adjustable conductivity, enabling the synaptic features, such as paired-pulse facilitation (PPF) and STM. These features ensure the a-GaOx photo-synapse as an expected DUV photoelectronic reservoir, which maps the complex input vector into a dimensionality-reduced output vector. Besides, the analog memristor array stores the weights of the output layer and enables parallel in-memory computing of feature outputs from the photo-synapses. As a result, we demonstrate the inputting, mapping, featuring, training, and recognition of fingerprint images based on a hardware DUV in-sensor RC system. By adopting a dual-feature strategy, the recognition accuracy of fingerprint images maintains over 90% even under 15% noise level, demonstrating the robust anti-noise characteristics of the system. This DUV in-sensor RC system will make a difference for highly efficient latent fingerprint recognition in the future for criminal investigation and security.

4Inspired by the biological visual systems (Fig. 1b), a DUV in-sensor RC system was configured for high-efficiency latent fingerprint recognition based on photo-synapses and memristor array (Fig. 1c). In the human visual perception system, sense and transmission of external stimuli rely on parallel networks of receptors (retina), neurons, and visual cortex in the brain, making the compact system efficient for solving complex and unstructured real-world problems. The visual signals are received by the retina and then transmitted along neurons and synapses to the visual cortex that carries out memory, learning, and recognition functions for further processing. Therefore, different from the traditional architecture for latent fingerprint recognition, the bio-inspired RC architecture is capable of sensing and processing in parallel so as to ensure high efficiency and low power dissipation.

5The RC system basically consists of a reservoir layer and output readout layer, as shown in the dashed box of Fig. 1c. In an RC system, it is required that the states of the reservoir should be nonlinearly mapped by the temporal inputs and correlate to their former states, the weights of the readout layer are trained for specific learning tasks, and the final outputs are normally based on a linearly weighted combination of the reservoir feature outputs. The PPC effect with nonlinearly tunable characteristics renders the a-GaOx photo-synapses as ideal photoelectronic reservoirs, as shown in the left panel of Fig. 1c. Memristors feature excellent adjustable resistive state characteristics and naturally perform the vector-matrix multiplication in an array structure, making it suitable to serve as readout layer–, as shown in the right panel of Fig. 1c. Therefore, we built an in-sensor RC system for latent fingerprint recognition based on DUV photo-synapse and memristor array. The fabrication processes of the photo-synapse and details of memristor array are shown in the Methods section and Supplementary Information.

6In our previous work, the PPC effect in a-GaOx film has been suppressed by a high-temperature annealing process to implement high responsibility and swift response speed for high-speed DUV sensor application. But, for photo-synapse application, the PPC effect, which ensures the dynamically modulated conductivity, should be taken full advantages to mimic the synaptic plasticity,.

7To verify the mechanism of the PPC effect in the fabricated a-GaOx film, Kelvin probe force microscopy (KPFM), which uncovers the potential distribution of the object surface, has been performed with negative and positive biasing (Supplementary Fig. 1). In the target regions (5 × 5 μm2), only the middle parts (1 × 5 μm2) were injected with extrinsic electrons or holes. After the carrier injection, the dynamic evolution of both the surface potential distributions and the relative height in the target regions were measured every 9 min as presented in Fig. 2a. The recovery of target-region potential in both situations takes a long time, indicating the abundant electron and hole traps inside the a-GaOx film. Since the potential recovery of the hole-injected region lags obviously behind the situation of electron injection, the trapping effect of holes rather than electrons contributes more to the persistent conductivity.

8The typical calculation results demonstrate that the oxygen vacancies, one of the intrinsic deficiencies in Ga2O3, could act as deep donors–, and dominate the trapping phenomenon of nonequilibrium carriers. In view of the potential associations between the oxygen vacancy and hole trapping, we fabricated a-GaOx DUV sensors with different O contents in the film, labeled S1, S2, and S3 (see Methods section). X-ray photoelectric spectroscopy (XPS) characterizations of O1s peak of S1, S2, and S3 are performed as shown in Fig. 2b. The first peak OI is attributed to the lattice oxygen, and the second peak OII is associated with the oxygen vacancies,,. The highest OII/OI ratio demonstrates the most abundant oxygen vacancies in the as-sputtered a-GaOx S1 film with the lowest oxygen content. To testify the effect of this component design, normalized time-dependent photoresponse curves were measured under a light pulse illumination of 25 ms, as shown in Fig. 2c. Obviously, the a-GaOx sample S1 with the highest OII component exhibits the most significant PPC effect. Therefore, the deliberately enlarged PPC effect by Ga-rich design turns the sample S1 into an ideal photo-synapse. All following experiments are based on the sample S1.

9To further estimate the feasibility of nonlinear response characteristics as a photoelectronic reservoir, the PPC effect of this a-GaOx photo-synapse has been studied systematically. Figure 2d shows the time-dependent photoresponse curves of the a-GaOx photo-synapse under different light pulse widths (25 ms, 50 ms, and 100 ms). The decay process of the current after the removal of external light stimuli reveals the enhancement of the PPC effect with increasing the light pulse width. Concurrently, the intrinsic volatile characteristics of the photocurrent imply the STM effect of the device. Obviously, a longer illumination time leads to a higher reinforced PPC effect. Defining the response current (ΔI) as the difference between maximum photocurrent and initial dark current, it always exhibits nonlinear dynamics with the light pulse under various power densities, as shown in Supplementary Fig. 2. Besides, it should be noted that a negative voltage pulse strategy on the back gate was utilized to release the extra photocurrent caused by the PPC effect (Supplementary Fig. 3), to ensure the repeatable operation of the device.

10In a biological neural system, the PPF as one typical function of the synapse, demonstrates the ability to process continuous temporal information. To validate the PPF effect, we applied consecutive light stimuli on the device, exhibited in Fig. 2e. During operation, the bias was set at 1 V, the power density of DUV pulses was fixed at 450 nW/cm2, and the pulse width was fixed at 25 ms. Herein, a time-dependent photoresponse curve is depicted, revealing the real-time modulation of the channel conductivity or synaptic weight. Even though the power density was the same for both stimuli, the ΔI of the latter stimulus was obviously higher than that of the former, indicating the facilitation effect of synaptic plasticity. Taking ΔI1 as the response current of the first stimulus and ΔI2 of the second, the PPF index, as a ratio of ΔI2 to ΔI1, is calculated to be 128.4% in this case. To demonstrate the dependence of PPF on pulse interval, we then tested and extracted the PPF index under various intervals, as shown in Fig. 2f. The results show that the PPF index decreases exponentially with increasing pulse intervals, while it keeps almost independent of the power density of the light pulse (Supplementary Fig. 4). Moreover, the PPF index versus pulse interval curve can be fitted by a double-exponential function:where C1 and C2 are the facilitation ranges, and τ1 and τ2 with fitted values of 28.97 ms and 279.91 ms are the characteristic time constants relative to a rapid and slow relaxation time, respectively. As a result, short pulse interval approaching to rapid relaxation constant exhibits higher PPF index or facilitation efficiency, making the STM generated by different inputs more distinguishable. In addition, the photoresponse of the a-GaOx photo-synapse could always maintain nonlinearity even under multi-bit pulse stimuli (Supplementary Fig. 5), which plays an important role in classification and recognition tasks in biological activities,. Therefore, this nonlinear relationship between conductivity and external stimuli of the photo-synapse has the potential to construct a feature space for nonlinear mapping. The above-mentioned basic neuromorphic characterizations of the a-GaOx device reveal its promising potential for serving as a photoelectronic reservoir.

11Figure 2g visualizes in detail the generating and trapping/detrapping behavior of nonequilibrium carriers inside the a-GaOx photo-synapse. Under DUV light stimulus (left panel), the photogenerated electron-hole pairs increase in the active layer, while the holes drift toward the electrode/GaOx interface and get captured by oxygen-vacancy-related traps. After stimulus removal (middle panel), the energy barrier in the detrapping process prolongs the annihilation of photogenerated current. Thus, during re-stimulus process (right panel), the accumulated trapped holes lay a footstone of increasing free electron concentration and lead to the PPF effect.

12The feature extraction of the original image simplifies the recognition process and improves the efficiency. Owing to the nonlinear PPF of synaptic plasticity, the inputs sequence of the a-GaOx photo-synapse reservoir can be distinctly mapped into feature outputs. To assess the capability of the feature mapping of the reservoir, we perform the measurement of a 4-bit optical stream, which can be mimicked by the corresponding 4-bit inputs within “0000” to “1111”, as shown in Fig. 3a. Each periodical input waveform (25 ms pulse width, 45 ms pulse interval) is considered as one bit, in which the “off” and “on” state of the light pulse denote “0” and “1” in the basic binary image.

13The configuration of the feature space of inputs/outputs is the basis of readout training. Therefore, all the I-t characteristics of all 4-bit inputs of pixel sequences have been measured and sampled for feature values. To illustrate the feature sampling, the I-t curves of three representative inputs of “0001” (in blue), “0011” (in red), and “1101” (in purple) of the a-GaOx reservoir are exhibited in Fig. 3b. Although the last pulses are all “1”, their decay processes after the input sequences are different. Therefore, the final state of the reservoir not only relates to the last input, but also depends on its real-time state, indicating the lateral connections in such an a-GaOx reservoir,. Based on the conspicuous difference, each pixel sequence can be featured by current sampling to realize feature extraction. For accurate feature extraction, sampling point 1 (SMP1) is defined as the average current value in the sampling time (ST) section after a sampling delay (SD) from the last pulse. Obviously, different SD and ST conditions will lead to discrepant SMP1 results, thus the optimal sampling section needs to be determined.

14The SMP1 value has a negative correlation with SD and ST due to the fading effect of response current, as shown in Fig. 3c. Based on a statistical analysis of 100 cycles for all input sequences, the reservoir outputs with SD ranging from 0 to 20 ms and ST from 10 to 30 ms show similar fluctuations for SMP1 evolution, as shown in Supplementary Fig. 6. Apparently, the situation of 0 ms SD and 10 ms ST presents a relatively larger SMP1 value, which will facilitate the distinction of all 16 input sequences. Therefore, this sampling condition is utilized for current sampling and feature space configuration throughout the following experiment, as shown in Fig. 3d. In addition, similar statistical results (100 cycles for each input) based on 20 stochastically selected devices further validate the reliability and repeatability of the a-GaOx reservoir (Supplementary Fig. 7). Since there are potential overlaps between SMP1 distributions of two certain inputs (e.g., “1011” and “1101” in the dashed circle of Fig. 3d), a dual-feature strategy, namely adding an additional SMP2 at the end of the 3rd pulse region (see the dashed square), has been proposed to sharpen the output features. Consequently, the feature space based on nonlinear photoresponse configures the classification process of the reservoir, reducing the dimensionality of raw data from 4-bit digital inputs to 2 analog outputs that serve as the inputs of the linear readout layer,. The energy consumption per pulse operation of the photoelectronic reservoir can be estimated to be E = 20 nA × 1 V × 25 ms = 0.5 nJ, indicating that the reservoir architecture possesses potential energy-efficient characteristic.

15To verify the feasibility of DUV in-sensor RC for fingerprint recognition, we constructed a hardware system, composed of a photo-synapse reservoir layer and a memristor readout layer, as shown in Fig. 4a. The relationship between the mathematical model and the physical hardware of this system has been illustrated (see Supplementary Fig. 8). In such a system for DUV fingerprint recognition, the images are first converted into DUV light pulses. And then, the pulse signals are projected parallelly onto the a-GaOx photo-synapse reservoir layer, generating single or dual feature outputs. After receiving the outputs of reservoirs, a memristor array loading pre-trained weights performs the in parallel in-memory readout process. In the RC system, only the weights in the readout layer need to be trained, reducing the training costs for DUV fingerprint recognition. A photograph of the local hardware test board system with sensor array, memristor array, and data interfaces is shown in Supplementary Fig. 9. To perform the training and recognition of fingerprint images, 40 fingerprints of 5 people (8 for each) are chosen from the Second Fingerprint Verification Competition (FVC 2002) database (Supplementary Fig. 10). Preprocessing of the raw human fingerprint images includes cropping, scaling, binarizing, and rejoining to 20 × 4 pixels in order to adapt the 4-bit light pulse input as shown in Fig. 4b. Correspondingly, the reservoirs (20a-GaOx photo-synapses) generate 20 pairs of feature outputs (SMP1 and SMP2) per image for the training of the memristor array network. By simulating the readout network training as shown in Fig. 4c, the dual-feature strategy successfully achieves 100% accuracy after 100 training epochs, demonstrating a much faster convergence rate than in the single-feature situation (500 epochs). Thus, a dual-feature strategy is employed, and only a dimensionality-reduced 40 × 5 weight matrix needs to be trained for each fingerprint image. As an example, a recognition accuracy for unseen fingerprint images has been simulated to be around 92% based on this dual-feature strategy, where the expanded sample amounts of the fingerprint images ensure the high recognition accuracy (see Supplementary Fig. 11).

16The memristor device with analog conductivity is deliberated to enable the hardware realization of a fully connected readout network (the characteristics of the memristors are shown in Supplementary Fig. 12). As exhibited in Fig. 4d, e, both the colormaps and statistical histogram of the 200 weights obtained by software simulation and memristor hardware verification after the training are highly consistent within tolerable error. Therefore, this hardware DUV in-sensor RC system performs a remarkable accuracy for fingerprint recognition. In addition, the considerable retention performance (more than 150 min, as shown in Supplementary Fig. 13) of the memristor indicates that once the offline training has been completed, the trained weights of the local memristor array can be used for recognition for a long time.

17In the practical DUV fingerprint recognition task, noises are always inevitable. Here, stochastic noises with a varying scale from 1% to 20% were introduced into the input images to mimic the potential optical interference. This generates a new test set to verify the recognition capability of this in-sensor RC system based on the a-GaOx photo-synapse. Three situations, full-precision (double-precision floating-point) simulation, limited-precision (32-bit fixed-point quantization) simulation, and hardware experiment, are considered for comparison. As shown in Fig. 4f, recognition accuracies in all situations remain comparable under ≤3% noise level and deteriorate asynchronously with its increment. The limited resistive states of the memristor device and the amplification of non-ideal factors (e.g., device-to-device and cycle-to-cycle variations, discreteness of operations, etc.) under high-level noise dominate the relatively quick deterioration in the hardware situation. Therefore, the improvement of resistive states and uniformity of the memristor devices could further improve the system robustness. It is noteworthy that the recognition accuracy of the hardware experiment still maintains above 90% under 15% noise level. In summary, the fully-hardware DUV in-sensor RC system based on a-GaOx photo-synapse has promising potential to be competent for high-precision in-situ DUV fingerprint recognition tasks.

18In summary, we proposed a fully-hardware DUV in-sensor RC system composed of a photo-synapse reservoir layer and a memristor readout layer for latent fingerprint recognition. It is found that the oxygen-vacancy-related hole traps dominate the PPC effect and induce the nonlinear neuromorphic features of the a-GaOx DUV photo-synapse for in-senor RC. As a result, the inputs of the reservoir can be nonlinearly mapped to dimensionality-reduced outputs, which constitute the feature space. Acting as the readout network, memristor device array with analog conductivity takes charge of the training of reservoir outputs and parallel in-memory computing. Based on such a hardware system, the high recognition accuracy of DUV fingerprint images nearly matches the simulation results when adopting a dual-feature strategy. The system achieves 100% recognition accuracy after 100 training epochs and maintains 90% accuracy even under 15% background noise level consistent with the anti-noise characteristics of DUV light. This fully-hardware DUV in-sensor RC system provides a prototype for efficient identification and security applications.

19After the first lithography process, the 208 nm a-GaOx thin film was deposited onto the low-resistance p-type Si substrate with 300 nm SiO2 by RF-magnetron sputtering at room temperature. Samples S1, S2, and S3 were fabricated by sequentially increasing oxygen content. Separated a-GaOx films were obtained after the lift-off process. The rapid thermal annealing process of the a-GaOx thin films was performed at 400 °C in ambient N2 for 1 min. After another lithography process, the source and drain electrodes consisting of Ti/Au (20/50 nm) were deposited onto the a-GaOx thin film by electron beam evaporation.

20The relative contents of OI and OII in the a-GaOx film were determined by X-ray photoelectron spectroscopy (XPS, ESCALAB 250Xi). The morphology of the a-GaOx film was characterized by atomic force microscope (AFM, Bruker Dimension Icon). The potential distributions of the samples were obtained by above AFM instrument under KPFM mode. In the target regions (5 × 5 μm2), only the middle parts (1 × 5 μm2) were injected. After the carrier injection, the surface potential of the target region was scanned every 9 min.

21The electronic and photoelectronic characteristics, including I–t photoresponse curves, PPF effects, and reservoir feature outputs of the a-GaOx photo-synapse, were measured at room temperature using an Agilent B1500A Semiconductor Device Analyzer. An LED with a wavelength of 254 nm was used as the monochromatic light source, and the light intensity was calibrated by an optical power meter (S401C and PM100D). A shutter (Thorlab SHB-025T) was used to modulate the pulse waveform of the DUV light source. During all the measurements, the bias was fixed at 1 V. All experiments for RC were performed under the optimal pulse condition of 25 ms width and 45 ms interval.

22Experiments were reproducible. All the error box figures depicted by Origin software are unified with the same standard: box with percentile range from 25 to 75, whisker with outlier coefficient of 1.5, median with solid line and mean with circle dot.

23The above-mentioned 1K-bit array adopts the 0.18 µm standard technology of Semiconductor Manufacturing International Corporation (SMIC). The memristor composed of TiN/TaOx/HfOy/TiN was stacked by the following steps: First, the TiN layer was deposited by physical vapor deposition as the bottom electrode. Next, the HfOy and TaOx layers were successively stacked on the TiN layer by atomic layer deposition. Finally, another TiN layer was deposited as the top electrode using the same process as the bottom layer.

24The current values of the reservoirs are transmitted to trans-impedance amplifiers (TIAs) to convert them into voltage values, which are then fed into the memristor array. Each differential pair in the memristor array represents a single weight of the neural network. Transistors are used for device addressing and crosstalk current suppression. As for the training of the memristor array, we utilized an offline training method to update the weight (conductance) matrix of the array. Once the software simulation is completed, the weights of the whole array (400 memristor devices) are updated by referring to the simulation results, column by column. To SET a selected column, all source lines (in blue, in Fig. 4a) were floated, except the selected one, which was grounded. All word lines (in red) were biased at the same SET voltages. Each bit line (in black) was assigned with a different voltage based on the targeted conductance of the selected memristor. For RESET operation, word lines of selected columns were grounded, all source lines were biased with RESET voltages, and each bit line was biased with a large voltage to allow sufficient RESET current.

25All the reservoir outputs were used to construct the input layer of a fully-connected network: for the single-feature strategy, the network size was 20 × 5; for the dual-feature strategy, the network size was 40 × 5. The fully-connected network was trained by the MATLAB Deep-learning Toolbox, utilizing the Softmax output function and the logistic regression to supervise the learning. The stochastic noise was made by the dot product of the MATLAB randn matrix and the grayscale value throughout the whole image. The final noise was obtained by the product of the noise level and the stochastic noise and then added into the original images.