Real-time laser spot detection and tracking system based on parallel multi-target detection and determination algorithm

Laser is considered one of the greatest inventions of the 20th century, alongside nuclear energy, computers, and semiconductors. It is a special light beam with high energy concentration, good orientation, and some other special characteristics. These properties provide tremendous potential for the application of lasers in industrial^1,2 and military³ fields. Due to its promising applications, laser technology has been proactively researched. In many practical applications of laser technology, the detection and tracking of laser spots play an integral role. For example, in the applications of interactive interfaces,^4–8 bridge deflection measurement,^9–12 laser guided robots,^13–16 and aircraft tooling measurement and assembly,¹⁷ all of them are involved in laser spot detection and tracking technology.

By analyzing the existing laser spot detection and tracking systems, we found that the current detection and tracking systems mostly rely on the personal computer (PC) as the core processor to detect and locate the laser spot and then use the electromagnetic motor to track the spot. By using the PC and the electromagnetic motor, researchers have made lots of practical studies in the field of laser spot detection and tracking. For example, in Ref. 9, considering the difficulties of instrument installation and inefficient measurement in harsh environments that exist in traditional bridge deflection measurement methods such as dial indicator measurement¹⁸ and displacement meter measurement,¹⁹ Xie et al. designed a real-time laser spot positioning system for bridge deflection measurement based on PC. By adopting laser spot tracking and positioning technology, the system can be used to achieve real-time and highly accurate persistent detection. In Ref. 20, taking into account the lack of suitable easy-to-learn and easy-to-use apps to support geography education at that time, Kovárová designed an app based on a PC to interact between the laser pointer and the map. By using a laser pointer instead of finger pointing on the map to interact with the map, which facilitates the education of geography knowledge in schools. Undeniably, the PC has its own unrivaled advantage of copious logic resources and can execute extremely complex algorithms to improve the accuracy of laser spot detection, but precisely the fact that its logic resources are too voluminous to be fully utilized results in a substantial waste of resources. Besides, its high energy consumption and large size will make the whole system relatively bulky without portability or flexibility. Therefore, some researchers are already engaged in researching lightweight, miniaturized spot detection and tracking systems. In the past, advanced RISC machines (ARMs) microprocessors with high performance and low power consumption have been considered as an alternative for PCs to achieve lightweight and miniaturized spot detection and tracking systems. Limited by the low efficiency of serial execution, the real-time detection performance and efficiency of the system based on ARM microprocessors are generally inadequate, which makes the designed system difficult to satisfy the requirements in practical applications.

Since the inception of the field programmable gate array (FPGA) over 30 years ago, it has been widely applied in different fields to implement a myriad of applications.^21–25 Owing to the effective parallel computing capability, tremendous flexibility, and reconfigurability of the FPGA, an increasing number of researchers have taken it as a better platform to build high-performance, lightweight, and miniaturized systems.^26–30 As a result, a number of research projects on laser spot detection systems based on FPGAs have emerged. For example, based on an FPGA system, Sun et al.³¹ improved the circle fitting algorithm and greatly accelerated the speed and spot center positioning accuracy of the system. Considering the importance of spot energy control in laser polishing, Zhang et al.³² designed a laser spot size detection system based on an FPGA and helped achieve laser spot control in laser polishing. Besides the above-mentioned research, there are also some very meaningful studies on the laser spot detection of FPGAs.^33–37 However, all of the aforementioned research is only applicable when the background environment is clean and simple; only in this environment can the laser spot be separated from the background perfectly and accurately. In a complex environment where there are many interferences with partial characteristics of the spot in the background, such as luminous lamps and computer monitors, the laser spot will be unable to be completely separated from the background. In this case, the existing spot detection systems described in the above-mentioned research cannot work well. In practical engineering applications, the background environment where the laser spot is located tends to be complex, with multiple interferences present. Hence, it is crucial and meaningful to research the detection and tracking technology of laser spots in complex environments.

In this paper, we propose a novel parallel multi-target detection and determination algorithm for laser spot detection in complex environments and deploy it on an FPGA. Unlike the current mainstream methods that can only accurately locate laser spots in clean and simple environments, this algorithm can successfully and accurately locate laser spots in more complex environments. This is a highlight of our work. Then, the algorithm is further applied to an FPGA system to build a prototype system for laser spot detection and tracking. The system adopts an FPGA as a substitute for a traditional PC for image data processing, which greatly reduces the bulk, weight, and power consumption of the system while maintaining real-time detection performance, making the system more portable. Besides, it is taken into account that piezoelectric actuators have the advantages of high precision positioning, speedy response, and no electromagnetic interference compared to electromagnetic motors commonly used in current tracking systems. We take full advantage of our existing advanced ultrasonic motor technology to design and machine a two-axis ultrasonic motor platform for laser spot tracking. The use of piezoelectric actuators facilitates the system's development of high-precision wireless communication and control technology. The two advantages of the system described earlier broaden the application of existing laser spot detection and tracking systems. This is the other highlight of our work.

In this section, we first outline the design of the overall system and then describe the algorithm design in detail.

The design of the whole system contains numerous components. To enable the design principles of the system to be better understood, a system architecture diagram is drawn up, as shown in Fig. 1 below. The functional modules inside the dashed box in Fig. 1 are implemented through the FPGA, and the modules outside the dashed box represent the peripheral devices that interact with the FPGA.

As can be seen from Fig. 1, the FPGA is an integral part of the system. Its primary function involves processing the input image data and making the corresponding control command, similar to the function of our human brains. Apart from the FPGA, the devices outside the dotted box in Fig. 1 are also critical. They are used to act as human eyes and arms, working together with the FPGA to perform the spot detection tracking function. These device types and critical parameters are listed in Table I below.

In the system, the OV5640 camera is utilized to capture external images, which are akin to human eyes. Simultaneously with external image capture, the acquired image is imported into the FPGA for further processing to determine the spot coordinates. It is worth noting that in order to assist in the debugging of the system, a LabVIEW control panel was designed. The control panel can not only be used to quickly determine the optimal image parameters to facilitate the debugging of the system but can also be used to visualize the detected coordinates. When the laser spot coordinates are determined, on the one hand, the coordinates are used to add the laser spot markers for the subsequent RGBLCD display, and on the other hand, the coordinates are used to control the two-axis ultrasonic motor stage to track the detected laser spot. The response time of the ultrasonic motor used for spot tracking is exceptionally fast, only needing 2 ms to respond. To ensure the ultrasonic motor tracks the laser spot quickly and accurately, precise speed regulation of the ultrasonic motor is of utmost importance. By using the ultrasonic motor driver we designed, we are able to accurately control the speed by tuning the input voltage of the driver. Consequently, to implement precise control of the input voltage, a 10-bit-wide dual DA converter is used in the system, which can achieve a maximum conversion rate of 125 MSPS (million samples per second). The ultrasonic motor drive control is mainly implemented based on a proportional-integral-differential control feedback adjustment algorithm. Through real-time feedback of the detected laser spot coordinates, the distance of the spot coordinates from the center of the camera field of view is calculated, and finally, the digital distance is converted into an analog voltage output via a dual DA to regulate the ultrasonic motor speed. The implementation of speed feedback adjustment effectively mitigates the adverse impacts of inertia, which improves the tracking accuracy of ultrasonic motors without sacrificing tracking speed.

The algorithm design mainly includes the design of the laser spot detection algorithm and the laser spot tracking algorithm. The laser spot tracking is based on the detection of the laser spot; only after the position of the laser spot is precisely located can the detected spot be tracked according to the spot position. Therefore, we elaborate on the design of the laser spot detection algorithm in advance. It should be noted that since our innovation mainly lies in the spot detection algorithm, this section focuses only briefly on the design of the spot detection algorithm and the implementation of the spot tracking algorithm.

The laser spot detection algorithm is primarily conceived around the proposed multi-target detection and discrimination algorithm. Prior to the operation of multi-target detection and discrimination, several image processing operations need to be conducted as a prelude. The design principle of the laser spot detection algorithm is shown in Fig. 2.

Since the brightness in the center of the laser spot is overly intense and exceeds the acquisition limit of the camera, it will cause the brightness of the spot area to be oversaturated. Ultimately, in the image, it will bring up the problem that the center of the spot color appears white and cannot be captured properly. However, as the spot extends outward from the inside, the brightness of the spot will gradually decrease. When the brightness drops below the camera’s acquisition limit, the color of the area can be captured and displayed normally. Since this area is located in the outer region of the spot, it is like a halo surrounding the spot. Therefore, the area where the color can be captured normally is defined as the spot halo area. It can be perceived that the laser spot captured by the camera is actually a bright white spot with a halo of a specific color. In the detection algorithm, multi-target detection is performed using two separate processing routes to detect the areas of the image that are as bright as the center of the laser spot and the areas that are as colored as the halo of the laser spot in the image, respectively. After detecting the same bright areas as the center of the laser spot, we roughly screen the regions according to the size of the detected areas to reduce the workload later. After a preliminary screening, the detected areas that are as bright as the center of the laser spot are likely to be laser spots in terms of size. Subsequently, a precise screening is carried out by comparing whether the areas detected by processing route 1, which are as bright as the center of the laser spot, intersect with the areas detected by processing route 2, which are the same color as the laser spot halo. The bright area that intersects the detected areas with the same color as the laser spot’s halo is considered to be the location of the laser spot.

After the image binarization was completed, morphological filtering was performed on the binarized image to remove the noise. Then, the multi-target detection module is applied to the morphologically filtered binarized images to detect multiple regions on the image that are as bright as the center of the laser spot and various areas that have the same color as the laser spot halo, respectively. The design principle of the multi-target detection module is shown in Fig. 3. In the multi-target detection algorithm, we define multiple 41-bit registers. The number of registers is defined as N. The next 40 bits of data are used to store the left, right, top, and bottom boundary coordinates of the target, and the highest bit is used as a flag bit to indicate whether the boundary coordinates stored in this register are valid or not. As shown in Fig. 3(a), when the first target appears, the boundary coordinates of that target are recorded in the first register. However, when the later target appears, we will determine whether it falls within the neighborhood of the previous target.

If the target falls within the neighborhood of the existing target, the boundary of the existing target will be extended to surround the target; otherwise, it will be judged as a new target, and the boundary coordinates of the new target will be recorded in the register. The width of this neighborhood is defined as L. As shown in Fig. 3(b), this figure elaborates on the principle of target expansion and the emergence of new objectives. When x₃ − x₂ < L, the boundary of the existing target T₁ will expand and enclose the boundary of the emerging target T₂, resulting in the two targets eventually merging into a new target, and the boundary coordinates of the new target will be rewritten into the register, overwriting the boundary coordinates previously written into the register. When y₃ − y₂ > L, the emerging target T₃ will be considered a new target, and the boundary of this new target will be written to a new register. When multi-target detection operations are performed on both processing routes, the boundary coordinates of each target area, which are as bright as the center of the spot, and each target region, which has the same color as the spot halo, will be determined. After determining the boundary coordinates for each target, a rough screen is performed based on the area to filter out boundary coordinates where the area is obviously unreasonable. Subsequently, by applying the previously described region intersectionality method for target screening, the laser spot location can be accurately determined. Although the color in the center of the laser spot cannot be captured properly due to the high brightness resulting in the color being displayed in white, the color of the laser spot halo around the center of the spot can be captured properly. The principle of area intersectionality is actually finding the special target that simultaneously satisfies the requirements of brightness in the center of the target and the existence of a laser spot halo at the periphery of this target. After the position of the spot has been confirmed, the coordinates of the spot center can be further determined.

Based on the laser spot coordinates, laser spot tracking can be further implemented. The principle of laser spot tracking is actually implemented based on PID regulation with spot coordinate feedback, as shown in Fig. 4 below. In Fig. 4, the fixed coordinates of the center of the field of view are set as the final aim coordinates. Through feedback on the received laser spot coordinates, the error value of the detected spot coordinates from the center coordinates is calculated. Based on the calculated error, the ultrasonic motor drive can be controlled by PID feedback regulation to achieve spot tracking.

To validate the effectiveness of the proposed multi-target detection and determination algorithm, we used Modelsim software to simulate and verify the designed algorithm, as shown in Fig. 6 below.

The following Fig. 5(a) is the image we collected from the actual environment. The resolution of this image is 800 × 480, which is in accordance with the resolution required by the algorithm. In the picture, there are objects such as luminous lamps that are as bright as the laser spot, as well as other objects such as posters that are similar to the laser spot's halo color. All of them are interferences that hamper the detection of the laser spot. Therefore, it is appropriate to validate the effectiveness of the proposed algorithm for laser point detection in complex environments by using this image as an input in the simulation. Using Fig. 5(a) as the input image in the simulation and reading the image data to emulate the camera timing afterward, the image processing results obtained by the proposed algorithm are shown in Fig. 5(b). It can be seen from Fig. 5(b) that the position of the laser spot in the picture is correctly outlined with a red box and marked with a green crosshair at the center. The detection result in Fig. 5(b) illustrates that the proposed algorithm can effectively detect laser spot positions in complex environments. In order to better facilitate the understanding of the proposed algorithm, the intermediate process of algorithm processing is graphically visualized. The images of the intermediate process performed by the algorithm are shown in Figs. 5(c)–5(e). Figure 5(c) shows the binarized image after multi-target detection based on luminance, which is the result of the input image after processing route 1 in Fig. 2. Figure 5(d) shows the binarized image after multi-target detection based on the halo color of the laser spot, which is the result of the input image after processing route 2 in Fig. 2. Taking advantage of FPGA parallel processing, processing routes 1 and 2 are carried out simultaneously. Afterward, the detected targets in Fig. 5(c) are first roughly screened according to the area size, leaving the candidate targets with a relatively suitable area size to reduce the subsequent computation. It should be noted that, considering the effect of distance on the detected spot size, the criteria for rough screening of targets based on area size can be relaxed appropriately. The rough screened Fig. 5(c) is combined with the multi-target result information obtained in Fig. 5(d) to determine the location of the laser spot based on the principle of area intersection. Figure 5(e) shows a schematic representation of the regions intersection determination using Figs. 5(c) and 5(d). Since the binarized image based on luminance fully retains the information of the laser spot center, we finally chose to mark the spot position on this binarized image, as shown in Fig. 5(f). By superimposing the detected spot position information in Fig. 5(f) with the input image, the final output image can be obtained, as shown in Fig. 5(b).

After the effectiveness of the algorithm is verified through simulation, we further evaluate the time consumed by the algorithm to process a frame of an image. It is found that at 50 MHz image processing frequency, the time consumed by the algorithm to process a frame with a resolution of 800 × 480 is only about 7.88 ms, as shown in Fig. 6 below. In the figure, at the time of 16 640 ns, the image data starts to be acquired for processing, and the processing is finally completed at the time of 7893 720 ns. By analyzing the time gap, it is obtained that at a clock frequency of 50 MHz, the algorithm only demands a total time of 7877 080 ns to process a frame of an image, which is what we describe as about 7.88 ms. It is indicated that the algorithm can process more than 100 f/s, reaching the frame rate required for real-time detection. When the image processing is complete, the coordinates of the laser spot in the image are subsequently determined. Therefore, in Fig. 6, the appearance of the coordinates is taken as a sign of the completion of the image processing.

By using the described method, we further applied the proposed laser spot detection algorithm to a physical system to build a prototype laser spot detection and tracking system, as shown in Fig. 7 below. The system can precisely detect and track the laser spot in complex environments.

After the construction of the prototype system, the performance of the system was further evaluated. In complex environments, the detection accuracy of laser spot detection can be affected by both acquisition distance and luminous intensity. Consequently, we tested the correlation between spot detection accuracy and acquisition distance at different luminous intensities, as shown in Fig. 8 below.

As can be seen from Fig. 7, the laser spot detection accuracy is around 90% at different luminous intensities and acquisition distances. Additionally, the variation of the curves in the figure shows that the system has excellent robustness to resist the effects of different luminous intensities and acquisition distances.

Since the proposed multi-target detection and determination algorithm mainly consumes the internal logic resources of the FPGA, the number of detectable targets N has an impact on the consumption of logic resources. In this regard, the consumption of logic resources by the proposed multi-target detection algorithm in the case of different numbers of detectable targets was tested. The logic resources of the FPGA are mainly comprised of look-up tables (LUTs) and flip-flops (FFs). The utilization of logic resources can be gauged by the consumption of LUTs and FFs. As shown in Fig. 9 below, the figure demonstrates the consumption of LUTs and FFs in different numbers of detectable targets N. From the figure, it can be seen that the consumption of logic resources is basically positively related to the number of detectable targets N. It can be used to guide the selection of FPGA devices based on this diagram and the actual requirements.

Based on the evaluation of the physical system, the proposed multi-target detection and determination algorithm for laser spot detection shows excellent detection performance and unique advantages. However, there is still potential for optimization in terms of algorithm performance. For example, at different luminous intensities, the color threshold of the laser spot needs to be manually adjusted by the designed LabVIEW control panel to guarantee excellent spot detection accuracy. In the future, we plan to explore and develop threshold adaptive technology that will automatically select the appropriate color threshold based on the external environment's luminous intensity, eliminating the need for manual adjustment. This development will enhance the algorithm's robustness and usability, making it more adaptable to different environments.

In conclusion, we propose a parallel new multi-target detection and determination algorithm for laser spot detection in complex environments. Through simulation tests, we found that the algorithm can process one frame in 7.88 ms at 50 MHz image processing frequency, which means it can process more than 100 f/s, reaching the frame rate required for real-time detection. It is difficult to achieve such excellent detection performance with central processing units (CPUs) and ARM processors. Then, the algorithm is further applied to a physical system to build a prototype laser spot detection and tracking system based on an FPGA and piezoelectric actuator. Practical tests show that the system maintains ∼90% accuracy in spot detection at different light intensities, which demonstrates that the designed laser spot detection algorithm has excellent robustness. Except for the merits of compact size and exceptional detection performance, the designed system also possesses the advantage of weak electromagnetic interference, which allows the system to develop high-precision wireless communication and control technology. Hence, the designed system has tremendous potential in applications requiring high laser spot detection performance, low power consumption, and weak electromagnetic interference, such as laser-guided intelligent ranging robots.

This work was supported in part by the National Science Foundation of China (Grant Nos. 52075240, U22A20259, and 12227808), the Fund of Prospective Layout of Scientific Research for NUAA (Nanjing University of Aeronautics and Astronautics), and the National Natural Science Foundation of China for Creative Research Groups (Grant No. 51921003).

The authors have no conflicts to disclose.

Jia Cao: Formal analysis (equal); Methodology (equal); Resources (equal); Validation (equal); Writing – original draft (equal); Writing – review & editing (equal). Yang Chen: Writing – original draft (supporting). De Yu: Validation (supporting). Zheng Xu: Writing – review & editing (supporting). Xiaopin Hu: Validation (supporting). Yongjing Liang: Investigation (supporting). Song Pan: Funding acquisition (supporting); Methodology (supporting); Supervision (supporting). Dawei Wu: Funding acquisition (lead); Supervision (equal).

The data that support the findings of this study are available from the corresponding authors upon reasonable request.