Event-based vision in laser welding: An approach for process monitoring

In production, laser welding processes—if not executed properly—are posing a risk of component failure. Consequently, to ensure the reliability of the components, stringent monitoring of the weld quality is needed. One promising option is the in-process monitoring of the welding keyhole, molten pool, or spatters. Monitoring sensors are acoustic sensors,¹ photodiodes,^2,3 x-ray sensors,⁴ charge-coupled device (CCD) cameras,⁵ and high-speed cameras.⁶ The analysis of the sensor signals includes approaches based on machine learning, such as support vector machines,⁷ decision trees (DTs),⁸ random forests,⁹ or convolutional neural networks (CNNs).^10–12 

Event cameras (ECs), which are inspired by biological mechanisms, measure pixelwise changes in brightness, providing advantages such as high temporal resolution, high dynamic range, low power consumption because no routine operation needs to be recorded, and low latency¹³ compared to standard cameras. These properties have led to promising applications in areas such as vibration measurement,¹⁴ drowsiness detection systems,¹⁵ counting of fast moving objects,¹⁶ or image deblurring.¹⁷ Depending on the further evaluation algorithm of an application, the asynchronous event data are processed and often transformed into alternative representation:¹³ For instance, individual events can be directly handled by event-by-event methods like spiking neural networks.^18,19 However, those networks require special hardware and, therefore, are not considered further. Moreover, events can be represented as event packets, whereby a predefined amount of events is collectively processed.²⁰ Additionally, events can be accumulated into images that can be easily processed using computer vision algorithms.²¹ 

As the EC facilitates a more comprehensive understanding of exceptionally swift processes, the advantages of the EC compared to standard cameras for process monitoring in laser welding might be interesting. The adaptability of the EC in different domains emphasizes the potential to provide insights into the dynamic processes of laser welding processes. To the best of our knowledge, no prior publications have investigated the potential of EC for quality monitoring of laser welding processes. Therefore, the present paper includes the following contributions:

•Transformation of the asynchronous events of the EC data to different promising event presentations for in-process quality monitoring of laser welding defects.

•Proposal of different sampling strategies of the event camera data for quality monitoring of laser welding processes.

•Analysis and interpretation of the introduced event representations for different in production occurring laser welding defects.

•Usage of machine learning algorithms including decision trees and convolution neural networks to distinguish the introduced defects.

The explained EC encoding is inspired by the spiking nature of biological visual pathways.²² Compared to standard cameras, the EC offers numerous potential advantages: ECs have an extremely high temporal resolution and low latency¹³ (both in the order of microseconds), which make these sensors react quickly to visual stimuli. Moreover, ECs have a very high dynamic range²³ (120 versus 60 dB of standard cameras) and low power consumption.²⁴ Therefore, ECs are currently used for feature detection and tracking like corners,²⁵ fastmoving objects,¹⁶ optical flow estimation,²⁶ 3D reconstruction,²⁷ pose estimation,²⁸ visual-inertial odometry,²⁹ image reconstruction,²⁰ motion segmentation,³⁰ neuromorphic control,³¹ or image deblurring.¹⁷ However, the spatial resolution of the EC is generally lower than that of standard cameras.

After having seen the advantages and applications of EC in diverse fields, we now look at how the EC can be used for the quality monitoring of laser welding processes.

Based on an industrial welding process, two metal plates were welded together. Therefore, in production occurring scenarios were provoked. The process is observed with the EC, and the aim is to recognize differences in its data. Figure 1 shows the experimental setup. Therefore, a laser beam was directed onto two thin metal plates by a 2D galvanometer scanner. The two metal plates have a dimension of 80 × 80 mm² and a thickness of 75 μm. As a laser, an IPG YLR-2000-WC fiber laser with an infrared wavelength of 1070 nm at a power of 300 W (max. power 2000 W) and a speed of 0.5 m/s was used. A scanner, a Raylase AS Fiber 30, focused the beam to a diameter of about 50 μm on the work piece surface. An event camera (EC) observed the laser welding process coaxially and in-process. As the EC, a Prophesee Metavision Evaluation Kit 4HD (EVK4) was used. The EC data sequence then is transferred and processed on a computer.

Anticipating scenarios that may arise during production, experiments were conducted. Figure 2 schematically shows the front view of the metal plates of five scenarios. In the reference scenario (a), the metal plates were positioned on top of each other. In (b), a thin film of oil was applied between the metal plates, which led to a spattering process. In (c), only one plate was welded. In (d), there is a defocus effect of the laser beam, and in (e) a gap was inserted between the two plates.

Having seen the experimental setup, we look at the resulting EC data. Figure 3 visualizes a sequence of events E [see Eqs. (1) and (2)] captured during the observation of the laser welding process. The events are shown in the three-dimensional space with the $x,y$-location and the time t as axis. The individual events are marked in blue. The position of the laser is given by the gray line. While at the beginning less events occurred, more events were captured afterwards as the process spattered.

One of the key questions of event cameras is how to extract meaningful information from the event data. Therefore, the events are transferred to alternative representations that make it easier to observe changes in the laser welding process. In the following, we present three different promising event representations to monitor laser welding processes, namely, event rates (Sec. IV A), projection on the x, $y$-plane (Sec. IV B), and projection along scanner data (Sec. IV C). Afterward, the resulting representations can be further analyzed, e.g., by classification algorithms.

An event rate transfers a data sequence E into a time series. More precisely, an event rate gives the number of events that occur in equidistant time intervals per length of the intervals. Mathematically speaking, this means that assume having observed a laser process during the time interval $T=[ t s , t e]$⁠, which results in an EC data sequence E. Let us divide T in L equally long subintervals of length $Δt$⁠, so $L=( t e − t s)/ Δt$⁠.

This event representation projects events in the x, $y$-plane, resulting in images of the dimension $X×Y$⁠. In the following, we present three different such x, $y$-projections, namely, binary (Sec. IV B1), density (Sec. IV B2), and temporal (Sec. IV B3).

So P has the entry $1$ if at least one event occurs at position $(x,y)$ and $0$ if there is no event occurrence.

After having seen different event representations, the question arises as to when and how often the representations should be generated during a welding process for quality monitoring purposes. Therefore, in the following, we present two different sampling strategies, namely, constant time period (Sec. V A) and constant event rate (Sec. V B).

Assume having observed a laser process during the time interval T, which results in an EC data sequence E. Let us divide T in equally long subintervals of length $Δt$⁠. The constant time period sampling strategy then generates the event representations for each subinterval of length $Δt$⁠.

Figure 4(a) visualizes this sampling strategy. As in Fig. 3, the events (blue) of a sequence of events E are shown in the three-dimensional space with the $x,y$-location and the time t as axis. When the time $Δt$ has passed, a binary projection P is generated. In the figure, this results in three binary images. A drawback of considering a constant time period as sampling strategy is that even if few or no events occur—meaning that no or few changes are in the laser process—a representation is created. This can be seen in the figure: In the first of the three intervals, there are almost no changes in the process, yet an almost completely black image is created. This drawback is overcome in the next sampling strategy.

Assume having observed a laser process during the time interval T, which results in an EC data sequence E. Let us divide E in subsets with the same amount of events $Δe$⁠. So each entry of the event rate $ε$ has the value $Δe$⁠. This constant event sampling strategy then generates the event representations for each subset with $Δe$ events.

Figure 4(b) visualizes this sampling strategy. When $Δe$ events have occurred, a binary projection P is generated. In the figure, three binary images are shown. Thereby, projections are created after differently long time intervals. Compared to Fig. 4(a), the first of the three projections is created later, as fewer events occur at the beginning. To determine the time of an anomaly during quality monitoring, the time after each $Δe$ has to be remembered.

After gaining understanding about different sampling strategies and event representations, we now look at which differences can be seen in the different event representations for our in production occurring scenarios according to Fig. 1 like gap or spatter (Sec. VI A). Subsequently, to further show the potential of EC data for process monitoring of laser processes, different configurations in which the EC parameters, the process parameters or the hardware was changed were realized (Sec. VI B). Afterward, we present a machine learning pipeline for quality monitoring including DTs and CNNs (Sec. VI C).

The first column of Fig. 5 shows the event rate ε determined during the welding process of the introduced scenarios according to Fig. 2. The event rate $ε$ with $Δt=1 ms$ is shown over the time in ms. The graph line is shown in rainbow colors. The color changes constantly over time. In the first row, the event rate of reference shows minor changes over time. The next two rows show oily processes. Compared to the reference, the event rate shows high peaks due to the spatters. The event rate of one plate shows nearly no events, the event rate of defocus shows tiny rises and falls, and the event rate of the gap shows few small peaks.

Figure 6 shows the projections on the $x,y$-plane determined during the welding process of the introduced scenarios. The first column of Fig. 6 shows the binary projection P, the second column the density projection Q, the third column the time projection $R 1$⁠, and the last column the time projection $R 2$⁠. The images include the whole image field of the EC of $1280×720$ pixels. In the binary projection, differences in the images between the scenarios are visible. While in reference, a regular spatter pattern with most spatters being completely visible in the image, in oil, far more spatters are visible. In one plate, nearly no events occur. The fifth row results from defocusing smaller than 1.5 mm and the sixth row from defocusing greater than 1.5mm. Therefore, the fifth row shows many thin spatters, whereas the sixth only shows few spatters. In the last row, a gap was introduced on one side of the rectangular welding geometry (see Fig. 1). As seen in the event rate (see Fig. 5 first column and last row), few events occur in that scenario. On closer inspection, one can see that few spatters occur on the left-hand side of the image, which is due to the introduced gap. In addition to the information whether or not an event occurred at a certain position in the $x,y$-plane, no further information like how many events or when the events occurred is available. This information is additionally shown in the next columns. The images of the density and temporal projections are shown in rainbow colors. So, for instance, for the density projection, the color map starts with red (no events), goes to yellow (medium number of events), and ends with green (most events). In the temporal projection, the rainbow colors change constantly over time as in Fig. 5. With this additional temporal information, it can be seen that the spatters in the one plate scenario occur at the beginning of the process when the laser beam penetrates. Moreover, the other events in the one plate scenario occur at four time intervals (pink, orange, green, light blue). Those events are due to reflections on a clamping device in the four corners of the welding geometry. At closer look, they are visible in all scenarios.

The second column of Fig. 5 shows the projections along scanner data V determined during the welding process of the introduced scenarios. The rainbow colors refer to the time, as in the event rate of Fig. 5. The horizontal, rainbow-colored lines give the position of the laser. While the event rate of A gives the amount of occurred events, B shows what happened spatially. For example, the event rate of reference shows small rises and falls. In the projection along the scanner data, it is visible that these events are caused by the regular occurrence of small spatters. Moreover, the high peaks in the event rate in the oil scenario are caused by strong ejections with several spatters that fly both forward and backward. So the projections along the scanner data V show the direction of the spatter relative to the global work piece position.

The third column of Fig. 5 shows the time projections $R 2$ determined during the welding process of the introduced scenarios. The rainbow colors give the time as before. Corresponding points among the representations can be seen. For example, the light green serpentine flying spatter in the reference scenario in the projection along the scanner data is also visible in the temporal projection. However, the spatter path appears longer in the $x$-direction in the temporal projection. This is due to the fact that in the temporal projection, the events are shown relative to the laser position, whereas in the projection along the scanner data, the events are shown relative to the work piece. As the laser moves from left to right, the path of the backward flying light green spatter appears longer in the temporal projection than in the projection along the scanner data. If a spatter flies forward relative to the work piece, like the light blue spatter in the gap scenario in the projection along the scanner data, then its path appears shorter in the temporal projection. Moreover, it can be seen in the temporal projections of the oil scenario, that not all spatter paths are completely visible in the EC observation field. This means that some spatter path in the projection along the scanner data are cut off.

To further show the potential of EC data for process monitoring of laser processes, four different configurations I–IV (see Table I) have been realized. Thereby, configuration I describes the previously used configuration according to Sec. III. Compared to configuration I, in configuration II parameters of the EC have been changed: The threshold for triggering an event has been reduced. This means that events are triggered for smaller intensity changes. This is indicated in the table as the sensitivity $σ$⁠. In configuration III, process parameters, namely, the weld speed ν and the weld power $η$⁠, have been increased. In configuration IV, experiments were carried out on a different system with a different EC and scanner as well as process parameters.

Figure 7 shows different event representations of the different scenarios for the four configurations. The first column shows the projections along the scanner data V of the reference scenario. A cut-out in which the welding path first is a curve and then continues in a straight line is shown. The other columns show the time projection $R 1$ of the different categories. The first line shows configuration I. Due to the more sensitive EC parameters in configuration II compared to configuration I in each image, more events are visible. In addition to the visibility of spatters in configuration II, more events are generated along the welding, which give information about the keyhole. In contrast, more events are generated in uninteresting areas as well: Irrelevant events due to reflections on the clamping device in the curve but also on the straight line occur. In configuration III, more events occur, especially around the curve. In configuration IV, similar patterns like in configuration I are visible. In summary, configurations I and IV prove to be particularly suitable because they give the spatter behavior of the process and do not generate any additional irrelevant events.

Figure 8 shows our proposed machine learning pipeline. After performing the experiments according to Sec. III, the received events are transformed either to event rates or to projections on the $x,y$-plane. A subset of transformed events is then used for the supervised training of a decision tree (DT) or a convolution neural network (CNN). DTs are well-suited due to their interpretable decision rules, while CNNs are explored due to their state-of-the-art performance in computer vision and pattern recognition tasks. Therefore, the label is given by the scenarios introduced in Fig. 2.

With T being the time interval in which a complete rectangular geometry according to Fig. 1 is welded, we select $Δt=T$⁠. This means that for the welding of two metal plates, the event rate ε contains one entry, namely, the number of events that occurred during T. The same applies for $ε i$ and $ε o$⁠. For $ε i$ and $ε o$⁠, visually a radius $r=60$ pixels was chosen. The training data instances of the DT, therefore, consist of three values, namely, $ε$⁠, $ε i$⁠, and $ε o$⁠. Those training data instances are visualized by the dark boxes in Fig. 8. Instead of an event rate, for the CNN, one projection on the $x,y$-plane is created per time interval T. Therefore, CNNs were trained for the binary projection P, the density projection Q, and the time projection $R 1$ as grayscale images.

Table I gives the number of metal plate pairs of the dataset. The dataset consists of 249 welded metal plate pairs. The number of metal plate pairs for training, validation, and testing is given by $n train$ (60%), $n val$ (20%), and $n test$ (20%). As given in Table II, the data are imbalanced between the scenarios. To compensate this, DT used Synthetic Minority Oversampling Technique (SMOTE). For the CNN, a data augmentation with balancing factors was incorporated.

The DTs were implemented with the library Scikit-learn.³² The library uses an optimized version of the CART algorithm. To measure the quality of a split, the Gini impurity was used as criterion. A grid search to find good hyperparameters, namely, the maximum depth of the tree, and the minimum number of chunks required to split an internal node was performed.

To teach the CNN invariances and robustness properties, a data augmentation including rotation, flipping, shifting, and shearing was used. The image size was resized from 720 × 1280 pixels to 100 × 177 pixels to accelerate the training process. The models were developed and trained using TensorFlow.³³ A wide range of CNNs were tried: Initially, the networks were kept shallow extended later to deeper networks. The filter size was changed between 3 × 3 to 7 × 7. In addition to convolution layers, max pooling, global average pooling, batch normalization, or dropout have been used. Different activation functions such as ReLU, LeakyReLU, or tanh were tried. The final layer used a softmax activation. During the training of the CNN, the categorical cross-entropy was used as loss function. Moreover, an Adam optimizer with a learning rate $l r=1⋅ 10 − 4$ was used.

Table III gives the weighted classification accuracies of the different event representations as input. Given the class imbalance in the data, standard accuracy can be misleading due to the underrepresentation of some classes. To overcome this issue, the weighted accuracy was use, which adjusts the overall accuracy by accounting for the number of instances in each class. The DT delivers an accuracy of 64.29% for the multiclass classification problem with five classes. In comparison, the CNN with binary projection P provides a higher accuracy of 72.96%. This makes sense, as the CNN processes a complete picture with information about the positions of the events, rather than relying on just three simple features as in the DT. Nevertheless, our results show that a density projection Q does not improve the classification accuracy and a time projection $R 1$ significantly degrades the classification.

This paper, for the first time, investigates the potential of event cameras in laser welding. Event cameras (EC) differ from standard frame cameras: Rather than capturing images at a fixed rate, they asynchronously measure pixelwise brightness changes and output an event sequence that includes time, location, and sign of the brightness changes. To extract meaningful information from the asynchronous events that occur during a laser welding process, we present three promising event representations for in-process quality monitoring of laser welding processes: First, we propose the use of event rates that count the number of events occurring in a predefined time interval. Therefore, spattering processes can be quantified. Second, to also capture spatial information, we propose to transform the events into projections in the EC observation field. Three such projections, namely, binary, density, and temporal, have been analyzed. Third, to also incorporate scanner information, we propose to project the events along the scanner data. In our experiments, we observe the laser process of two metal plates coaxially and in process with an event camera and introduced different in production occurring scenarios. Those scenarios include contamination, leading to a spattering process or a gap between the two plates. Our laser welding experiments show clear differences for the scenarios in the event representations. To further encourage the potential of event cameras for process monitoring in laser welding, different configurations in which the event camera parameters, the process parameters, or the hardware parameters were changed are realized. After having analyzed the differences in the event representation when changing the laser welding process, we then present a machine learning pipeline for quality monitoring. Therefore, event rates were used to train a decision tree (DT) and projections of the events in the event camera observation field—so images—were used to train convolutional neural networks (CNNs). Therefore, the CNN outperformed the DT. Moreover, binary and density representations outperformed time representations across the CNN architectures.

In the context of future research, to improve the classification accuracy, additional features such as spatter count, spatter angle, and spatter speed may be analyzed. In addition to additional features, further classification algorithms could be applied. Moreover, while our analysis classified the laser welding process after the weld of one experiment, other sampling strategies like analyzing the event data after a predefined amount of events has occurred could be applied.

The authors have no conflicts to disclose.

Patricia M. Dold: Conceptualization (equal); Data curation (lead); Formal analysis (lead); Investigation (equal); Methodology (lead); Software (equal); Visualization (lead); Writing – original draft (lead); Writing – review & editing (equal). Praveen Nadkarni: Investigation (equal); Methodology (supporting); Software (equal); Visualization (supporting); Writing – original draft (supporting); Writing – review & editing (equal). Meiko Boley: Investigation (supporting); Project administration (equal); Supervision (equal); Writing – review & editing (equal). Valentin Schorb: Investigation (equal); Writing – review & editing (equal). Lili Wu: Supervision (supporting); Writing – review & editing (equal). Fabian Steinberg: Supervision (supporting); Writing – review & editing (equal). Peter Burggräf: Supervision (supporting); Writing – review & editing (equal). Ralf Mikut: Conceptualization (equal); Project administration (equal); Supervision (equal); Writing – review & editing (equal).