Deep learning driven multifeature extraction for quality evaluation of ultrafast laser drilled microhole arrays

Microhole arrays with diameters of tens to hundreds of micrometers have a wide range of applications such as filters,¹ electronic packaging,^2,3 and shadow masks for OLED production.⁴ Using ultrafast lasers is a preferred method for the drilling of microhole arrays due to their unique characteristics such as small thermal damage and high efficiency.^5,6 Since the quality of laser processing is heavily dependent on the processing parameters and equipment status, quality evaluation and monitoring are often used in laser fabrication to obtain a stable and high processing quality.^7–12 

The gray gradient-based image segmentation has been widely used as the vision-based feature extraction method for quality evaluation in laser fabrication.^13–16 It identifies the target objects by the grayscale gradient at the edge of the objects.¹⁷ Those features (e.g., hole profiles) that have a significant grayscale difference from other regions can be obtained. Thereby, evaluation of the drilling quality in terms of the geometric quality of the hole shape was achieved with the gray gradient-based image segmentation. In ultrafast laser drilling, various by-products such as the recast layer, microcracks, and debris inevitably occur due to the complex and intense light-matter interaction. The quality of microhole arrays needs to be evaluated regarding not only the geometric quality but also the surface quality. Therefore, extracting multiple features from a material-dependent background is crucial for achieving a comprehensive quality evaluation of microhole arrays. However, the gray gradient-based method cannot effectively extract the mentioned by-products. Also, the recognition ability of the method to identify the hole geometry is weakened due to the simultaneous presence of the by-products. Hence, a reliable and efficient multifeature extraction method remains a challenge in the quality evaluation of microhole arrays.

Deep learning is a versatile and powerful tool for image processing, and one of its emerging applications is instance segmentation, i.e., recognizing and demarcating each distinct object of interest appearing in an image.¹⁸ Different from the traditional image segmentation method, the deep learning network recognizes the target objects through the learning of the prelabeled training dataset rather than the gray gradient on the edge of a specific object.^19–22 Its recognition accuracy mainly relies on the quality of the training dataset. Therefore, deep learning was widely used in various cases of vision-based quality evaluation and monitoring.^23–25 Although deep learning is a powerful feature extraction tool for quality evaluation, its application in ultrafast laser processing of microhole arrays has been rarely studied.

In this work, we successfully recognized and extracted multiple features by deep learning, thus achieving the quality evaluation of microhole arrays in terms of both geometric and surface qualities. Microhole arrays of various sizes and surface quality are fabricated on different metals and glass using different processing parameters. The images of the microhole arrays are prepared as the dataset by labeling the typical features of microholes to train the deep learning network. The well-trained deep learning network has efficient and powerful recognition ability. Typical features such as the hole profile, recast layer, microcracks, and debris can be recognized and extracted simultaneously from a complex background, thereby the geometric and surface qualities of the microhole are obtained. The deep learning network exhibits high recognition efficiency, achieving one-step recognition of up to 176 microholes in only 0.13 s. We also demonstrate the implementation of the method with a fast quality evaluation of an array of 2300 microholes. This paper extends the quality evaluation of microhole arrays by considering both geometric and surface qualities, thus providing an efficient quality evaluation method for the applications of high-quality microhole arrays drilled with the ultrafast laser.

The microholes were drilled using an ultrafast laser with multibeam processing in glass, copper, stainless steel, and titanium plate [Fig. S1(a)].²⁸ To prepare microholes of different sizes and surface qualities, various processing parameters were employed in the drilling of different materials. All the microholes were fabricated using a 900 fs pulsed laser, and the pulse frequency varied from 100 to 1000 kHz, pulse energy varied from 100 to 200 μJ. Microholes of different sizes were obtained by changing the trepanning radius, and the hole diameters ranged from 60 to 100 μm in this paper.

A deep learning network and a dataset consisting of a training set and a test set were developed for instance segmentation based on the Detectron2 framework.²⁶ After 36 000 iterations of training, the total loss of the network dropped from the initial 2.66 to 0.09. The average precision (AP) of recognition obtained with the test dataset over the hole profiles, recast layers, microcracks, and debris reached 90%, 80%, 70%, and 73%, respectively (see Table S1 and note S1²⁸ for the details of the training process). The well-trained network can simultaneously recognize multiple objects of different types in the image of microhole arrays. The scheme of the feature extraction for microhole arrays by deep learning is shown in Fig. 1. The input can be either an SEM or optical microscope image of microhole arrays, and the output of the deep learning network is an image in which the recognized objects are marked with their classes and confidence levels. A post-processing module was added to extract features such as hole profiles, recast layers, microcracks, and debris from the output of the network.

The dataset consisting of 560 images of microhole arrays was prepared for training the deep learning network. The images in the dataset were obtained either by scanning electron microscope or optical microscope with different scales and backgrounds. There were 4–64 microholes in each image, and features such as the hole profile, recast layers, microcracks, and debris were labeled in each image (see note S2²⁸ for the details on the labeling process). To build the dataset more accurately and efficiently, the images of the microhole arrays were first processed manually with the gray threshold segmentation method. The image annotation tool “LabelMe” was used to label the irregularly shaped features like recast layers, microcracks, and debris on the images afterward. In addition, a self-developed program based on the watershed contour segmentation algorithm was used to label the hole profiles. The hole profiles were obtained with a manually determined threshold in the program, and the coordinates of the points on the hole profiles were organized as a so-called COCO format (a popular data format for object detection and segmentation, it uses a JSON file to store annotations for each image in a dataset). The dataset consisting of all images and their annotations was randomly divided into a training set and a test set with a ratio of 8:2. The average precision was evaluated with the test set every 800 iterations during the training process. The detailed source codes of dataset preparation and deep learning network related processes such as training, prediction, and feature extraction are all given in the supplementary material.²⁸ Both the training and feature extraction procedures were run on a computer with a GPU (GeForce RTX 3070). The probability density was estimated from the frequency distribution by using the kernel density estimation (KDE) method in matlab.

In the drilling of microhole arrays with ultrafast laser, the features that need to be extracted for the quality evaluation vary with the materials, as shown in Figs. 2(a) and 2(b). As for metals, the typical features include the hole profiles and recast layers. For brittle materials such as glass, the microcracks, and debris need to be considered in addition to the hole profiles and recast layers. The features of arrays include hole spacing and all the features of each individual hole, as shown in Fig. 2(c).

To extract the above-mentioned multiple features from an image of microholes, it is necessary to recognize (i.e., segment and identify) each target object in the image. To this end, we compared two traditional grayscale gradient-based segmentation methods and the deep learning methods. Figures 2(d), 2(e), and 2(f) are the recognition results of the same image [see Fig. S1(c)²⁸ for the original image] using methods of threshold segmentation, watershed segmentation, and deep learning, respectively. It can be found that an accurate and complete segmentation of multi-object cannot be achieved through the two traditional image segmentation methods. On the contrary, the deep learning network can recognize the hole profiles, recast layers, microcracks, and debris simultaneously. Even in the case that a large grayscale difference exists between the target object and the background, some unwanted areas would inevitably be segmented with the traditional image segmentation methods [see Figs. S2(a) and S2(b)].²⁸ The unwanted areas remain along with the extracted features, requiring additional removal processing.

To verify the accuracy of the multi-object recognition by the deep learning network, we tested the network with microholes drilled in metal and glass. For ultrafast laser drilling in metal, the front and back sides of the microhole have different features, i.e., the recast layer only appears on the front side. The deep learning network accurately recognized the subtle difference in features between the front side and backside of a microhole. The hole profiles and recast layers are recognized on the front side [Fig. 3(a)], and only the hole profiles are recognized on the backside [Fig. 3(b)]. For ultrafast laser drilling in brittle materials, it tends to have a more complex surface morphology than metals such as copper, stainless steel, and titanium. In addition to the recast layer, irregularly shaped microcracks and debris appear on the front side of the microhole. The well-trained network successfully recognized most of the features except for some tiny debris and cracks [Fig. 3(c)]. Moreover, we found that the deep learning network also has a good ability to recognize irregularly shaped microholes [Figs. S3(a) and S3(b)].²⁸ 

In addition to the recognition accuracy, a broad and fast recognition capability of the deep learning network under various situations is also critical for quality evaluation. Therefore, we tested the network with microhole arrays of different sizes drilled in aluminum, copper, stainless steel, and titanium. The largest and smallest microholes in the images differ in diameter by two times, and the network presents a very good recognition ability, as shown in Fig. 3(d). More importantly, even the image with a background that is not in the dataset can also be accurately recognized, which means that the network does not require a large dataset to develop a broad recognition ability. In addition, we tested the network's ability and speed to recognize images containing a large number of microholes [Figs. S3(c) and S4].²⁸ The network successfully recognized 176 microholes at one step even though the dataset used to train the network only contains 49 microholes in one image at most, and the recognition process is completed in about 0.13 s.

Moreover, we tested the deep learning network with a video of moving microhole arrays since the dynamic recognition of microholes is very practical in industrial applications. Arrays of microholes with varying numbers, sizes, and surface features can be dynamically and accurately recognized (video S1).²⁸ 

Based on the multifeature extraction, a comprehensive quality evaluation model can be established in terms of both geometric and surface quality of microhole arrays. The key features of microhole arrays are shown in Table I. The geometric quality refers to the radius, roundness, inclination angle, smoothness of the hole profile, as well as the spacing among holes in the arrays. The surface quality refers to the area of morphological features such as the recast layer, microcracks, and debris.

The quality evaluation process of an array of 36 microholes is shown in Fig. S6²⁸ and Fig. 4. We extracted the hole profiles and recast layers around each hole by the deep learning network. The raw profiles were fitted with the ellipse fitting model, from which the geometrical features such as the hole spacing, radius, roundness, and smoothness were obtained. In addition, the area of recast layer around each hole was calculated directly with the extracted morphological features. Another quality evaluation case for an array of 168 microholes is shown in Fig. S7.²⁸ 

For large-area microhole arrays, the number of microholes tends to be very large (usually up to tens of thousands). Instead of extracting the features of all the microholes in the arrays, we evaluated the quality of a large-area microhole array through a statistical approach (see Fig. S8).²⁸ To calculate the qualification rate of the microholes in the arrays, we first obtained the frequency distribution of the feature parameters by random sampling, then estimated the probability density of the feature parameters for the population (i.e., all the microholes in the arrays) based on the sample with a so-called kernel density estimation (KDE) method. Finally, the probabilities that the feature parameters of arrays falling into a qualification range were calculated with the probability density of the hole population.

We performed random sampling on an array of 2430 microholes and obtained a sample of 240 microholes. The radius, smoothness, and roundness of the 240 microholes in the sample were extracted by the deep learning network [Figs. S8(b)–S8(d)],²⁸ and the corresponding frequency distributions were obtained. From the obtained frequency distributions, the probability density curves in a continuous form were calculated using the kernel density estimation, as shown in Fig. 5.

The proportion of microholes with their feature parameters falling into a range of interest in a population can be calculated through the aforementioned probability density obtained from the frequency distribution of the sample. To verify the accuracy of the estimation, we completely measured all the 2400 microholes in the population and compared the measurement results with the two probability densities estimated from two samples with a size of 60 and 240, as shown in Figs. 6(a)–6(c).

The three curves in each figure are similar in shape and location, hence the probability density distribution of the population can be estimated from a sample of the population. However, the slight difference among the three curves indicates that the accuracy of the estimate depends on the sample size to some extent; a higher accuracy can be obtained with a larger sample size. To further verify this conclusion, we compared the measurement results with the estimated proportion of microholes falling into a given interval in the population. As shown in Fig. 6(d), when the sample size is 240 (one-tenth of the population), the estimation errors are 1.1%, 1.2%, and 1.2% for the radius, smoothness, and roundness, respectively. When the sample size is 60, which is only one-fortieth of the population, the estimation errors are 3.4%, 10%, and 4.7% for the radius, smoothness, and roundness, respectively.

In this work, we successfully recognized and extracted multiple features of an ultrafast laser drilled microhole by deep learning, thus achieving the quality evaluation of microhole arrays in terms of both geometric and surface quality. The conclusions are summarized as follows.

1. Compared with traditional vision-based feature extraction methods, the deep learning network has more powerful recognition ability under various situations. Typical features of microholes such as the hole profile, recast layer, microcracks, and debris can be recognized and extracted simultaneously by the deep learning network.

2. With the extracted multiple features, the processing quality of the microhole arrays can be evaluated in terms of both geometric and surface qualities. The geometric quality is evaluated with the qualification rate of the radius, roundness, inclination angle, smoothness of the hole profile, and the spacing among holes in the arrays as well. The surface quality is evaluated with the area of morphological features such as the recast layer, microcracks, and debris.

3. The deep learning feature extraction exhibits high recognition efficiency, achieving one-step recognition of up to 176 microholes in only 0.13 s. Thus, quality evaluation of large area microhole areas can be realized through a statistical approach based on the efficient multifeature extraction.

This paper provides an efficient quality evaluation method for the applications of high-quality microhole arrays drilled with ultrafast lasers. The method presented here can also be applied to quality monitoring in other laser micromachining.

This work was supported by the National Natural Science Foundation of China (NNSFC, No. 52075287).

The authors have no conflicts to disclose.

Zhanwen A: Conceptualization (lead); Formal analysis (lead); Investigation (lead); Methodology (lead); Supervision (lead); Writing – original draft (lead). Guisheng Zou: Conceptualization (lead); Formal analysis (lead); Investigation (lead); Methodology (lead); Supervision (lead); Writing – original draft (lead). Wenqiang Li: Software (equal). Yue You: Software (equal). Bin Feng: Methodology (equal); Writing – review & editing (equal). Zimao Sheng: Methodology (equal). Chengjie Du: Methodology (supporting). Yu Xiao: Writing – review & editing (equal). Jinpeng Huo: Methodology (supporting). Lei Liu: Supervision (equal); Writing – review & editing (equal).