A PC-based control method for high-speed sorting line integrating data reading, image processing, sequence logic control, communication, and HMI

Nowadays, the products in the industry update rapidly, the labor cost gradually increases, the market competition gets fiercer, and health and environmental protection draw more attention. The same challenges also exist in fields such as 3C (computer, communication, and consumer) products, mask production, solid waste sorting, and logistics express sorting, which demand related manufacturing, inspection and sorting of continuity, high-efficiency, and automation. As far as sorting is concerned, high-speed automatic sorting lines are mostly based on various sensors, and the visual sensor is the most widely used because of its remarkable non-destruction and rapid detection.^1–7 The collection of image data is often completed by using a visual sensor plus a dedicated controller and circuit and then read and processed by using a host computer (usually a PC) through a communication interface and protocol.⁴ The vision-based automatic sorting line requires in-line detection, identification, and separation.⁷ For these operations, in-line and parallel control of data reading, material identification, material buffering, position synchronization, and accurate separation are often needed. However, the coordination in the course of the operations, especially at a high speed, is more difficult to be satisfied in terms of the control architecture, real-time, stability, disturbance rejection, algorithm, sequential logic control, scalability, openness, and so on. Therefore, there is an urgent demand for the further research on the architecture and method of the vision-based high-speed automatic sorting control system to resolve these problems in practical applications.

In recent years, a variety of architectures and control methods for vision-based sorting line control systems have been proposed; however, they all present drawbacks especially for high speed. In the security inspection, due to the slow line speed (generally less than 0.3 m/s) and the lack of sequential logic control, it is usually a PC-based image acquisition and processing system. For industrial applications that require sequential logic control, PLCs (programmable logic controllers) are often used as the lower computers,^1–6 and PCs run as the host computers with the human–machine interface (HMI) software. If machine vision is also required, smart cameras with image acquisition and processing are often used, which can send the image processing results to PLCs for judgment during the control process, and then, the corresponding control logic signals are output. Compared with the architecture proposed here, this widely used architecture has at least the following disadvantages: higher camera hardware cost, inflexible development, a complex system, and difficulties in coordination and control resulting from the integration and solidification of image processing programs in smart cameras, as well as additional PLC cost. As for the automatic sorting systems used in modern agriculture, sorting materials such as potatoes, fruits, eggs, mushrooms, and seeds^7–15 are mostly transported in a single column, for which it is convenient to use a detection switch for position synchronization. Until now, the line speed is generally less than 0.5 m/s. In the field of solid waste sorting,^16–19 sorting materials are often characterized with irregularity and inconsistency in terms of size, shape, position, and posture and also usually scattered in multiple columns on the conveyor belt. The detection methods with reflected or transmitted intensity images or spectral images for identification are used, including visible light monochrome or color vision, x-ray transmission (XRT), x-ray fluorescence (XRF), laser-induced breakdown spectroscopy (LIBS), near infrared (NIR) spectroscopy, and far infrared (FIR) spectroscopy. In order to improve the sorting efficiency and quality, it is more urgent and difficult to combine with in-line and high-speed separation, but there has been little previous research work focused on automatic sorting control systems. In such control systems with strong real-time requirements such as motion control and open CNC (computer numerical control), the architecture of PC general OS plus real-time operating system (RTOS) is generally adopted,^20–22 but this RTOS will add additional software and development cost and is not familiar to ordinary engineers and technicians.

The control system architectures and methods above support this study. However, the previous studies mostly focus on identification algorithms and do not fully explain how the control system can effectively solve problems, such as line speed, data buffering, image processing, logic control, real-time, and position synchronization. In industrial applications, separation is not less important than identification with respect to in-line efficiency; on one hand, with the popularity of the research on in-line rapid visual identification, high-efficient recognition algorithms are continuously introduced, which make the recognition quality and speed improve further, and on the other hand, sequential logic control is maturely applied in the field of the industrial production line. However, there is not much research work to combine the two for the low-cost, simple structure, coordinated control, and practical high-speed sorting line control system. Moreover, the PC has the advantages of fast operation speed, popularization of multi-core and multi-thread (MCMT),^23–25 rich display and storage resources, convenient connection to industrial cameras for image acquisition, convenient networking, convenient production management, high familiarity with general OS such as Windows for engineers and technicians, flexible programming, etc., which makes it necessary and possible to realize high-speed sorting line control systems under general OS.

With respect to the communication mode in the field of industrial control, it mainly consists of a communication bus interface and communication protocol. The mode can be classified into two types: parallel communication, such as PCI (Peripheral Component Interconnect), and serial communication, such as UART (Universal Asynchronous Receiver/Transmitter) and Ethernet in which high-speed Ethernet communication has the most application prospect.

The purpose of the present study is to develop a new control technique with respect to the comprehensive performance in real time, high-speed, coordinated work, simplicity, generality, and low cost. In this study, a control system integrated with data reading, image processing, high-speed and frequent sequential logic control with accurate positioning, communication, and HMI for the high-speed automatic sorting line was designed. Taking the DE-XRT solid waste sorting line as an example, the feasible system structure and software architecture were proposed. The system hardware mainly includes line scan image detector, IPC, belt conveyor, and line array of high-speed on–off valves; by means of PC-based general OS Windows 7, the control software was developed in C language with Microsoft Visual Studio 2012; the research on the function, real-time, and stability of the control system was conducted on the basis of multi-core multi-thread (MCMT) of PC, the optimized process-thread settings (OPTS), and four threads named main thread, acquisition thread, processing thread, and PLC-like thread, combined with the first-in first-out (FIFO) stacker, encoder position synchronization, and variable sharing between threads.

The presented control system has the advantages of near real-time, versatility, openness, easy reconfiguration, simple structure, ability to manage and control integration, easy implementation of algorithms, flexible development, convenient usability, and low cost. It contributes to practical application for the continuous production line system of high-speed detection and frequent sequence logic control with accurate positioning.

The rest of this paper is organized as follows: Sec. II is materials and methods in which the hardware structure of the control system is introduced, the relationship between the operation parameters and design principles is analyzed, and the software architecture based on MCMT is proposed. Section III describes the results in which the real-time test by reading the Windows stamp counter is given, and experiments under three conditions are conducted. Section IV presents the discussion, and Sec. V is a summary of this study.

The platform of the dual-energy x-ray transmission (DE-XRT) high-speed sorting line and the structure of system are shown in Fig. 1. The platform consists of a vibration feeding mechanism, belt conveyor, x-ray source, line scan DE-XRT detector of energy integration, incremental encoder, and trolley wheel, control cabinet (including industrial PC-IPC, motion control and I/O card–MCIO card, switches, terminals, DC power supply etc.), line array of gas ejection/blowing valves and a gas supply system, etc.

When the sorting line is working, the main operations are detection, identification, and separation. The conveyor belt usually moves at a constant speed to drive the trolley wheel to rotate so that the incremental encoder value continues to increase, thereby indicating the distance of the belt movement in real-time. The materials to be sorted are first scattered on the conveyor belt by the vibration feeder and then conveyed by the conveyor belt through the detection point. At the detection point, the materials are scanned by a fan-shaped collimated continuous-spectrum x-ray beam emitted from the x-ray source. When scanning one material, the beam is partially absorbed or scattered by the material and partially passes through the material to the detector. The detector continuously collects DE-XRT image data of each material, and the gray value of the image is related to the effective atomic number and thickness of the material. There is an identification buffer area from the detection point to the separation point. In the identification area, each frame of image data and one encoder value are routinely read and processed by the IPC so that the type and the center-of-gravity coordinates of the material are identified. The x-coordinate value is determined by the detector channel number and the gray value of each pixel in the material area. The y-coordinate value (encoder pulse) is the belt moving distance of the material center of gravity passing through the detection point, which can be calculated by the encoder value read once per frame, the line scan frequency of the detector, the line speed, and the gray value of each pixel in the material area. At the same time, to get the real-time encoder value, the IPC continuously reads this value with a very high frequency. The real-time encoder value matches the center-of-gravity y-coordinate value of each identified material, that is, the identified material reaches the separation point, and at that moment, the IPC controls the corresponding high-speed gas valve to release the high-pressured gas for separation according to the type of material.

The detector is connected to the IPC via Gigabit Ethernet so as to carry the acquisition data stream. The IPC and the MCIO card are connected through the PCI bus. The trolley wheel and the belt are in rolling contact with no slipping. The wheel is coaxially fixed to the encoder. The encoder signal is input to the MCIO card; the digital output signals of the MCIO card are responsible for controlling the solenoid valves of the gas ejection, and the MCIO card is also responsible for controlling the belt line and the vibration feeder. The x-ray source is monitored and controlled by the IPC through the EIA-RS-232C serial communication interface.

A LINX-1605-301 line scan x-ray detector of Sens-Tech Company is used. Partial x-ray from the x-ray source passes through the material, enters the low-energy scintillator, and then passes through the middle copper sheet to enter the high-energy scintillator partially. The transmitted x-ray interacts with the scintillators, with transformation and integration to energy intension signals. The integration time is set to 300 ms, 4 subsamples are averaged for each sampling, and each channel signal is converted to 16-bit pixel depth. The channels of low-energy and high-energy are both 320, with 1.6 mm distance between adjacent channels (the detector resolution). A SRB401 x-ray source of Spellman Company is used with the tube voltage adjustment range of 100 kV–220 kV, the tube current adjustment range of 0.5 mA–2 mA at 200 kV, the maximum power of 400 W, the fan beam of 85° × 4°, and the focal size of 0.8 mm × 0.5 mm. An Advantech (Taiwan, China) IPC-610MB-L industrial PC (IPC) acts as the host computer with 8 GB RAM installed, Windows 7, and Microsoft Visual C++ 2012. It has an Intel i7-2600, 4-core, 64-bit, and 3.4 GHz CPU (central processing unit). The polyurethane (PU) belt conveyor is driven by a servo motor through a worm reducer with a 5:1 reduction ratio. A GTS-400-PV-PCI MCIO card of Googoltech (Shenzhen, China) is utilized. This controller can output pulse train to control 4 axes, and the pulses train can be chosen to control the servo motor. This MCIO card can also receive five channels of encoder signals up to 8 MHz after the quadruple frequency; in this paper, one channel of the incremental encoder signal is connected to measure the position forward. A ISC3806-003G-2500BZ1-5L type rotary encoder of Rep Hedss (Wuxi, China) is used with a resolution of 2500 pulses/revolution and 10 000 pulse counts per revolution after the quadruple frequency.

The related operation parameters have an important impact on the control system. The summary of all parameters can be seen in the nomenclature. The main parameters and their interrelationships are as follows:

Assuming that the x-accuracy (pitch) of the line scan detector, i.e., the resolution of the detector, is P_x (mm/pixel), the line scan frequency is F_l (line/s), the number of lines per frame image is L_f (line), the frame frequency is F_f (frame/s), and the acquisition time of one frame of image is T_f (s). For the energy integration detector of this study, the integration time is T_i (s) and the number of subsamples for one sampling is N_s, and then, there are the following equations:

If the line speed is v (m/s), then the y-accuracy of line scan detector P_y (mm/pixel) is given by

Generally,

If the encoder resolution is R_enc (pulse/revolution) and the diameter of the encoder trolley wheel is D_tw (mm), then the calculation equation of the pulse equivalent δ (mm) is determined by

The pulse equivalent value can also be further measured. The logical control scan cycle is T_PLC (s), and the distance quantization error of the separation point caused by scan cycle T_PLC is E_PLC (mm), and then, the E_PLC can be written as

The selection of the x-accuracy of the line scan detector is related to the size and quality requirements of the detected material. The higher line scan frequency can improve the line speed and the y-accuracy of the line scan detector. Under the same y-accuracy of the line scan detector, the faster the line speed, the higher must be the line scan frequency. The mechanical length of the buffer zone D_d2s (mm) is generally at least two images away. The encoder with a high accuracy has a small pulse equivalent. The parameters above need to be balanced selected according to the efficiency, quality, calculation speed, hardware response speed, etc.

If an area array camera is used, it will have similar parameters: the field of view (FOV) is H_FOV * V_FOV, the unit is mm, and the pixel resolution is H_pixel * V_pixel, the unit is pixel; the corresponding horizontal P_H (mm/pixel) and vertical accuracy P_V (mm/pixel) are determined by

where the horizontal accuracy and vertical accuracy are usually equal. There is usually a small amount of overlap between each image snapped continuously by an area array camera,^8–12 which should be considered in image processing.

The operation parameters used in this study are summarized in Table I.

In addition, for ease of explanation, the following global variables need to be defined:

The encoder value V_enc, unit is pulse number, is defined in C language as follows:

The sequence number of each frame image N_frm is defined in C language as follows:

The identified material data structure S_Wp is defined in C language as follows:

where Num is the identified material serial number, x is the x-coordinate value in the unit of the detector channel number, y is the y-coordinate value in the unit of encoder pulse, and class is the identified material type. The FIFO stacker with structure S_Wp as the element is defined; the stacker depth is greater than the number of possible materials in the buffer zone.

In order to achieve high-speed control, the real-time control must be met. To this end, we made full use of the current PC multi-core multi-thread (MCMT) technology.²⁴ Optimized process-thread settings (OPTS) were adopted, and the application process priority was set to real time; four threads were used to set reasonable priorities and were bound to different physical cores of CPU, especially the PLC-like thread monopolizing specific CPU physical core.²⁴ The four threads include main thread, acquisition thread, processing thread, and PLC-like thread [see Table II]; moreover, the first-in first-out (FIFO) stacker, encoder position synchronization, and variable sharing between threads are combined with the above threads and called in a nested way.

MFC dialogue box based on VC++ was used, and DLLs (Dynamic Link Library) were called in the application program. These DLLs are provided by the detector, x-ray source, and MCIO card manufacturer to operate these devices through Ethernet, RS232C, and PCI communication interfaces, respectively. The functions of the main thread include HMI, parameter setting, start/stop x-ray source, start/stop belt line, and start/stop sorting operation. During sorting, the main thread, acquisition thread and PLC-like thread are all turned on, and the processing thread is turned on periodically in the acquisition thread. Besides, the main functions of the other three threads and technologies such as FIFO stacker, encoder position synchronization, and variable sharing between threads are introduced below.

Obviously, for vision-based sorting line control, reading visual data is the first concern and processing the data is the key issue. These two functions are closely related and need to be executed successively. Therefore, the acquisition thread and the processing thread were chosen to achieve the two main functions, respectively. The flowchart of the two threads is shown in Fig. 2.

In the main cycle of the acquisition thread, the following operations are executed circularly: while the lower computer has enough a frame of image data collected, that is, a frame of image data is OK, the data are transmitted to the host computer PC by communication. After that, the pre-operations such as obtaining the current encoder value and the image serial number plus 1 are immediately performed, and then, the processing thread is opened to execute a frame of image processing, and wait for the next frame of image acquisition and data transmission. In the processing thread, the following operations are mainly performed: image pre-processing, image segmentation to identify the range of each piece of material, calculating the center-of-gravity coordinates of each piece of material, sequentially numbering them according to the center-of-gravity y-coordinate value of the material from small to large, identification, and pushing the structure data of each piece of material into the defined structure FIFO stacker according to the number and forming the structure data in sequence. The time sequence of two threads is shown in Fig. 3, and it must be guaranteed that

where T_obp (s) is the pre-operation time before image processing and T_p (s) is the image processing time. For the sake of safety, the blank time must be reserved; otherwise, the whole control will be disordered.

The acquisition interval of two adjacent images is about several hundred milliseconds. For the current general IPC computing power, this amount of time is moderate, so we set the relative priority of the acquisition thread and processing thread to highest, and each thread is bound to a physical core of the CPU.

In order to separate the identified materials in time and accurately, we used the PLC-like thread to simulate industrial PLC for high-speed logic control that requires the highest real-time. For example, according to the quantization error [Eq. (6)] generated by the scan cycle, at the line speed of 1 m/s, the scan cycle that generates quantization error of ±1 mm is 1 ms. Under the same allowable error condition, the faster is the line speed, the smaller is the scan cycle. Moreover, the scan cycle needs to be stable. In this study, the scan cycle T_PLC is preferably less than 100 µs. We set the relative priority of the PLC-like thread to time-critical, and this thread was bound to another independent physical core of the CPU. The outline flow diagram of PLC-like thread is shown in Fig. 4. It mainly completes the following operations: popping out the FIFO stacker, reading the encoder value, comparing and matching, and outputting the corresponding digital signals for blowing separation. This outline flowchart can only demonstrate the sequential logic control of a single column of materials in which the material data to be separated are not more than one at any moment. Based on the idea of this diagram, it is easy to realize the sequential logic control of multiple columns of materials (see the flowchart in Fig. 12 in the  Appendix). In that case, the amount of the materials to be separated at every moment can be changeable.

To adapt to temporarily buffer data between the detection point and separation point, a first-in first-out (FIFO) stacker technology was adopted. The FIFO stacker data communication between the processing thread and PLC-like thread of this system is frequent. Figure 5 illustrates the relationship between the FIFO stacker and the two threads. The FIFO stacker data are generated by the processing thread and consumed by the PLC-like thread. In the processing thread, the data of each material must be pushed onto the FIFO stacker [see Sec. II C 1 and Fig. 2(b) in detail] after the completion of image processing. As mentioned above, the data include the material number, x-coordinate value, y-coordinate value, and type, forming a data buffer for the materials to be separated. In the PLC-like thread, the data are popped from the FIFO stacker, and then, other operations are carried out (see Sec. II C 2, Fig. 4, and the  Appendix in detail). However, FIFO stackers cannot be read and written at the same time. To prevent data conflicts, semaphore technology is used to share memory between threads.^24,25 In the processing thread, the FIFO stacker must be pushed until the push-enable signal Sem_Push is true, and the pop-enable signal Sem_Pop is reset before pushing and set after pushing; similarly, in the PLC-like thread, the FIFO stacker must be popped until the pop-enable signal Sem_Pop is true, and the push-enable signal Sem_Push is reset before popping and set after popping.

For the high-speed sorting line, because detection, identification, and separation cannot be performed in the same place, position synchronization is always a problem. Incremental photoelectric encoders are widely used for position measurement. In this study, the incremental distance of the conveyor belt movement is converted into the number of pulses of the rotary encoder by the trolley wheel. The encoder can send feedback of the position of the conveyor belt in real-time, which is compared and matched with the y-coordinate value of the material identified, so as to synchronize the position during separation and perform y-coordinate calibration for each image to avoid the accumulation position error, as shown in Fig. 6.

In Fig. 6, V_enc − t1 (pulse) is the encoder value read in the acquisition thread when one-frame image acquisition is OK (the moment of t1), which can be regarded as the y-coordinate value of the last line of that frame image. This encoder reading is performed for each frame of image, and by this way, a position alignment for each image can be realized, which can avoid the accumulated error caused by the inaccurate speed and time measurement. For example, Ref. 4 uses the line speed and time to calculate the moving distance of the conveyor belt, which is prone to cumulative error. y_Cg (mm) is the center-of-gravity y-coordinate value of a material, which can be calculated by the encoder calibrating value V_enc − t1, the number of the pixel row of this center of gravity in the frame image and pulse equivalent, as follows:

where L_Cg is the row number the pixel of the material's center of gravity located in the frame of image, and the definitions of L_f, P_y, and δ are the same as those in Sec. II B. In fact, y_Cg (pulse) is the encoder value when the center of gravity of the material passes through the detection point, which is pushed to the FIFO stacker as one of the material structure data. V_enc − t2 (pulse) is the encoder value when the center of gravity of a material reaches the separation point, also the matching value. In the PLC-like thread, when a matching occurs during quick scan and a gas ejection is required, a control command is issued immediately to open some nozzles for separation. In fact, there is the following relationship equation:

where the meaning of D_d2s is the same as in Sec. II B. x_Cg (mm) is the center-of-gravity x-coordinate value. If the material is required for gas ejection when matching, the control system controls the corresponding valve in the valve array to open. This value is also used as one of the material structure data to push the FIFO stacker. The equation is as follows:

where C_Cg is the column number of the pixel of the center of gravity of the material located in the frame and the definition of P_x is the same as in Sec. II B.

The encoder value may overflow because continuous run of the belt line causes continuous increase in the encoder value, and the length of encoder variable (an unsigned integer) is not enough. Therefore, when the value increases to the maximum, the overflow will occur as soon as the encoder increases by 1 again, as shown in Fig. 7. Overflow may lead to control confusion. Suppose that V_enc_1 is the encoder value read in the last PLC scan cycle, V_enc is the value read in the current PLC scan cycle, n is the total bits of the encoder variable (in this study n = 30, encoder overflow will occur about every 3 h at the line speed 2 m/s), the maximum value 2ⁿ – 1, and C1 is the judgment constant, less than 2ⁿ⁻¹ – 1. Therefore, when V_enc_1 is between “2ⁿ − 1 − C1” and “2ⁿ – 1” and V_enc is between 0 and C1, it is judged that an overflow has occurred, as shown in the following equation:

According to the times of overflow N_OV and the current encoder value V_enc, it is easy to calculate the encoder value after the extended length. For example, the bits of the extended encoder value can reach 64, which can ensure long-term (about several million years in this study) operation without overflow again. The calculation equation of the encoder value V_encE after bit-expansion is as follows:

For real-time performance verification, the accurate time measurement is very important. Generally, the “Intel Architecture” Pentium processor PC has a stamp counter, which is a timer based on the CPU frequency. The timer can run continuously and stably without any other disturbances after PC is turned on, which can realize nanosecond level high-precision timing. Its implementation instruction is __asm RDTSC, which returns a 64-bit unsigned integer by the number of CPU clock cycles. When the difference in the stamp counter before and after certain operations or processes is obtained, this difference and the CPU frequency are used to calculate the time of the operations or processes, and the equation is as follows:

where T_t (s) is the test time by Windows stamp counter, D_sc is the difference of the stamp counter before and after certain operations or processes, and f_CPU (Hz) is the main frequency of CPU.

For the convenience of data recording and analysis, the read stamp counter can also be converted to character strings and stored as CSV (Comma-Separated Values) format files, which can be opened for viewing or statistical analysis with Microsoft Excel.

The main HMI is shown in Fig. 8, which includes the parameter setting and control of detector, x-ray source, belt conveyor, and vibration feeder. To start the system, operate in the following order: click “Detector ON” to turn on the detector, click “Conveyor ON” to turn on the belt conveyor, click “X-ray Source ON” to turn on the x-ray source, click “Feeder ON” to turn on the vibration feeder, click “Sorting Start” to start sorting; to stop the system, the corresponding turn off operation in reverse order can be performed. All functions of the laboratory platform were normal after running test (see Separation video.mp4 of the supplementary material). On the basis of laboratory testing, the material separation accuracy of this experimental platform at a line speed of 1 m/s is over 99%. The non-ferrous metal sorting throughput is estimated at 50 ton/h.

The raw data curve of single column of detector is shown in Fig. 9(a) in which the detection condition is without material, the x-ray source voltage is 200 kV, and the current is 2 mA. One frame of the raw high-energy x-ray image and one frame of the raw low-energy x-ray image collected at a time with materials and an attached color image of one of the materials are shown in Fig. 9(b). Since the raw high-energy image and the raw low-energy image are both with 16-bit pixel depth data, the high 8 bits of each pixel of the images are intercepted to obtain the BMP (bitmap) grayscale image. The detection data could be stored as raw data and/or BMP image by HMI configuration for offline analysis; the image could also be displayed in real-time, but it needs a lot of computer resources.

For the whole control system, the real-time assessment is particularly important. The following focuses on the scan cycle of PLC-like thread to verify the real-time performance of the control system.

As mentioned above, the Windows stamp counter is used for time measurement. In order to facilitate the evaluation of the accuracy and rationality of the method, the code execution time for reading the Windows stamp counter and auxiliary test time of saving the CSV File need to be obtained and analyzed through experiments.

As shown by the statistical data in Table III, the time to read Windows stamp counter T_rsc was only about 0.17 µs on average, which was very small and stable, so its impact on real-time test could be ignored.

During the PLC-like thread scan cycle test, a series of operations were required in a control cycle: read the Windows stamp counter, calculate the difference between the current stamp counter and the last stamp counter, convert the difference into a string, store the string as a CSV file to the hard disk, and assign the current stamp counter to the last stamp counter. These operations should be removed during the normal running of the control system. The time consumed by these operations was defined as auxiliary test time T_at. About a 14 µs average time and a small standard deviation can be seen from the statistical data, as shown in Table III. The time value for T_PLC test is equal to the time interval recorded in the CSV file minus the auxiliary test time T_at.

The time of acquisition of one frame of image T_f, that is, the interval between get two images in the acquisition thread, is also shown in Table III. T_f is determined by the lower computer, i.e., the controller of the detector. From the test results, it can be seen that its value is relatively stable with 385 250 µs on average, which is almost the same as set in Table I.

The test result of the operation time before image processing, T_obp, is also shown in Table III. In this study, the operations mainly include obtaining the encoder value, image number plus 1, and thread opening, with a small amount of code and a very small and stable measurement average of about 3 µs. According to Fig. 3 and Eq. (8), there is enough time for image processing in the frame interval. Therefore, it is feasible to take 340 ms as the theoretical frame interval T_f. Each frame of image is different, and so is the processing time T_p of each image. The possible maximum of image processing time must meet Eq. (8), and a certain amount of blank time must be left.

The scan cycle of PLC-like thread has the greatest influence on high-speed sorting and is the most critical factor affecting the performance of control system. The scan cycle must be very small and stable because a long cycle would cause an inaccurate ejection separation for the materials. According to the logic control procedure of the multicolumn material sequence in the  Appendix, the test was conducted with the line speed of 1 m/s and 50 materials of candidate separating and in separating. The disturbance was also set by means of playing the MP4 file using the Windows Media Player software and copying a large file of 2.52 GB from the PC hard disk through the USB (universal serial bus) interface to the flash disk at the same time. The test data under the three conditions of optimized process-thread settings without disturbance (OPTS-NonD), optimized process-thread settings with disturbance (OPTS-D), and default process-thread settings with disturbance (DPTS-D) can be seen in Table IV.

As shown in Table IV, the mean value of T_PLC measured with OPTS-NonD was about 15 µs, under the conditions of which the theoretical ejection error is about ±0.015 mm according to Eq. (6). Among the total measurement values, greater than 100 µs accounted for 0.156% and greater than 1000 µs accounted for 0.069%, this shows a high accuracy. Therefore, OPTS-NonD has the best performance. When the PC-based control system is running, the PC resources should be dedicated to the control, and the tasks unrelated should be closed. The curve of 100 continuous samples of T_PLC with OPTS-NonD is shown in Fig. 10.

Under the same logic control program quantity, Siemens S7-300 PLC (CPU 315-2 PN/DP, order No. 6ES7 315-2EG10-0AB0) was used for the comparative test. As shown in Fig. 11, its average scan cycle was about 3 ms, almost 200 times of our system; hence, its ejection error is ±3 mm, far more than our system, which is usually not allowed in practice.

In the case of artificially added video playing and copying disturbance with OPTS, the average scan cycle was about 16 µs, which was not much different from OPTS-NonD in Sec. III C 1. The standard deviation did not change much either. The percentage of greater than 100 µs increased, which was about 1.7 times of OPTS-NonD. It shows that the OPTS in this system has certain disturbance rejection performance.

In Windows 7 and Microsoft Visual C++ 2012 default process-thread settings (process priority: normal, thread priority: normal, and no thread CPU core assigning) with disturbance (DPTS-D), the average scan cycle was about 35 µs. Compared with OPTS-NonD in Sec. III C 1, about 2.4 times average scan cycle, about 4.6 times standard deviation, and 7.5 times percentage of scan cycle greater than 100 µs all show that the real-time performance of DPTS-D is greatly deteriorated, which proves that the OPTS has better anti-disturbance ability once again.

From the experimental results above, it can be seen that the necessary functions of data reading, image processing, frequent sequential logic control, communication, and HMI in high-speed sorting line have been realized through general OS PC-based programming. Using multi-core multi-thread (MCMT) technology based on PC to optimize process-thread settings (OPTS), multiple tasks are parallel. Setting thread/process priority and binding each thread to its own physical core reduced the real-time fluctuation caused by external disturbance and thread switching and achieved the near real-time performance in practical applications. In terms of cost and convenience of development and use, the system architecture proposed is better than that of PC-based general OS plus real-time operating system (RTOS) and also better than that of PLC plus HMI [see Refs. 20–22].

The important features of the presented system are as follows: near real-time, high-speed, coordinated control, simplicity, generality, and low cost. First, near real-time and high-speed are the remarkable characteristics of this system. The scan cycle of logic control was only 15 µs on average, superior to conventional PLC (ms level), which benefits from the powerful computing ability and speed of PC. It is also superior to 500 µs of Ref. 20 using PC plus Windows OS and RTOS. It can be seen that T_PLC is relatively small and stable, which fully ensures the separation accuracy and real-time control. Statistical data show that the scan cycle of more than 100 µs (blowing separation error 0.1 mm) only accounts for 0.156%, which is far less than the commonly allowed 2% error rate of sorting, and is acceptable in practice. When the line speed is 1 m/s, the theoretical position ejection accuracy can reach ±0.015 mm, better than ±0.3 mm in Ref. 11. Therefore, this control system is particularly suitable for high-speed and frequent logic control with accurate positioning, and an example can be given that the theoretical position accuracy can reach about ±0.05 mm when the line speed increases to 3 m/s. We confirmed that the control performance did not deteriorate even at the line speed of 3 m/s. Second, the coordinated control of detection, identification, and separation can be well realized. To be suitable for the buffered material on the conveyor belt that has been identified but has not reached the separation point, this paper described the method of buffering the material data with the FIFO stacker and synchronizing the encoder position. This method does not need to add additional sensors such as the photoelectric switch to detect the material again before separation, which is superior to the method in Refs. 6, 7, 16, and 18. For the dynamic position synchronization of in-line detection and separation, we used the encoder to correct the position of each frame of the image and match the center of gravity of each material with the encoder value obtained frequently so as to precisely control the separation action. Because of less influence from the line speed fluctuation and zero accumulated error, the method gains obvious advantage over that of calculating the valve action time by the line speed and buffer distance in Ref. 4. The method in the  Appendix can realize multicolumn scattered materials separation superior to the single column performance in Refs. 7, 10, 16, and 18 in terms of production efficiency. Besides, encoder synchronization is also applied to the ejection range, which can avoid the delay operation difficult to accurately control. Third, simplicity, generality, and low cost are other important features. For example, the cost of RTOS such as IntervalZero RTX is about $15 000 and that of Siemens S7-300 PLC about $1,500, which does not include the costs of extra development and/or detection switches. In addition, if the industrial camera is adopted, the smart camera (PLC + HMI) will be about 5 times more expensive than the ordinary camera (structure in this paper). This control system achieves a simple control layer and as few CPUs as possible. Only the general OS is used, and software is also conducive to the popularization and application of the proposed method.

On the other hand, the developed system has some problems that need to be settled in the future: The task scheduling of the operating system could be carried out. The practical real-time and reliability of control should be improved. If the processing cannot be completed within one frame interval, the solutions are to improve the hardware computing ability, optimize the algorithm, or use GPU (graphics processing unit) parallel computing,^26,27 etc. The graphical programming language (such as the industrial PLC ladder diagram) has not been developed in this system. In addition, the low on–off frequency and unstable mechanical delay of the valve will affect separation. The ejection separation force is so small that only light materials can be ejected. For the separation of heavy and irregular materials, other methods can be adopted, such as parallel robot grasping.^28–30 

The method proposed in this paper can be applied to the fields of visible light line scan or area array camera, spectrum, and magnetic measurement to obtain generalization image, especially to those fields that need frequent logic control with accurate positioning at the same time. Practical possible application cases include high-speed sorting systems based on machine vision for biscuits, potatoes, fruits, eggs, etc., defect detection, sorting, and pick-and-place systems for industrial products such as automobile parts and 3C products, high-speed solid waste sorting and separation production line for metal, plastic, glass, ceramics and similar ones, industrial mineral processing production line, and express logistics sorting system.

Taking the dual-energy x-ray transmission high-speed automatic sorting line as an example, aiming at the problems of coordination among in-line high-speed dynamic image processing, material identification, and material separation, a control method based on a PC general OS was proposed, integrating data reading, image processing, sequence logic control, communication, and HMI. In this method, we made full use of the multi-core multi-thread (MCMT) of the current PC, optimized the priority of process and multiple threads, bound the threads with high real-time requirements to run in a unique CPU physical core, and used the FIFO stacker data buffer, encoder real-time position synchronization, variable sharing between threads, and so on to achieve the overall control. Besides, the presented design principle of operating parameters is very helpful to solve such engineering problems. The main findings and the key innovation points are as follows:

1. At present, the high computational speed of PC is the basis of its ability to undertake image processing and logic control at the same time.

2. Due to the MCMT of PC, the priority of process and multiple threads can be set and threads can be bound to specific CPU physical cores by weighing control tasks, which can improve the real-time stability and anti-disturbance of the control system.

3. The high-speed and high-frequent logic control thread can monopolize one of the CPU physical cores, and the scan cycle of this system has achieved 15 µs on average.

4. The FIFO stacker was defined corresponding to the identified but not separated materials. After identification, push the stacker; before separation, pop the stacker. The stacker depth is greater than the number of buffer materials on the machine. The method is simple and convenient.

5. Based on 1 and 2 above, the encoder position synchronization can effectively correct the position of each frame, avoiding the position accumulation error. At the same time, material detection switches are not attached to it. Besides, it also can ensure a position matching and accurate separation in real-time. Moreover, it can control the blowing space range according to the size of the material during the blowing separation so as to avoid the inconvenience and error caused by the conventional delay operation.

6. The above technologies of MCMT, FIFO stacker, encoder position synchronization, and variable sharing between threads realize the integrated coordinated control of data reading, image processing, high-speed sequence logic control, communication, and HMI under a PC general OS with lower cost.

The abundant resources and popularization of PC make the control system similar to that in this study have the advantages of universality, openness, easy reconstruction, low cost, simple structure, integration of management and control, easy algorithm expansion, flexible and convenient use, etc., which is worth further study and has more application potential and competitiveness in the future.

See the supplementary material (Separation video.mp4) for the separation test.

This research was supported by the Jiangsu Provincial Key Research & Development Program of China (Grant No. BE2018722).

The data that support the findings of this study are available from the corresponding author upon reasonable request.

P_x

the x-accuracy (pitch) of the line scan detector, i.e., the

resolution of the detector

F_l

line-scanning rate

L_f

lines of one frame of the image

F_f

frame rate

T_f

the time of acquisition of one frame of the image

T_i

the time of energy integration

N_s

the number of subsamples for one sampling

T_obp

the time of operations before image processing

T_p

the time of processing one frame of image

v

conveyor speed

P_y

the y-accuracy of the line scan detector

R_enc

the resolution of the encoder

D_tw

the diameter of the trolley wheel coaxial with the encoder

δ

pulse equivalent

D_d2s

the distance between the detection point and separation point

T_rsc

the test time of reading the Windows stamp counter once

T_at

auxiliary test time by CSV file saving

T_PLC

scan cycle time of the PLC-like thread

T_t

the time of testing by the Windows stamp counter

D_sc

the difference of the stamp counter before and after certain operations or processes

f_CPU

the main frequency of CPU

V_enc

current encoder value—its range is 0–2ⁿ − 1, where n is the bit number of the variable V_enc

V_enc_1

last encoder value

V_encE

encoder bit-expansion value, the denotable range is 0–18 446 744 073 709 551 615 (2⁶⁴ − 1)

N_OV

overflow numbers of the encoder

N_frm

the sequence number of every frame image

C1

the auxiliary constant for obtaining V_encE—it is less than 2ⁿ⁻¹ − 1 and greater than the incremental value of the encoder in the possible maximum scanning period of the PLC-like thread cycle, it is used to catch the overflow of the encoder, and 200 000 000 is adopted in this paper; as shown in Fig. 7 in detail

C2

the constant related to ejection/blowing delay or offset, the delay from the detection point to separation point actually exists, C2 = D_d2s/δ, and this is also the key reason to use the FIFO stacker, as shown in Fig. 12 

C3

the constant of the half blowing distance range, related to the suitable blowing distance range at certain line speed, it can be experimentally determined; for example, when C3 = 1/δ, the calculated blowing range is about 2 mm, as shown in Fig. 12 in detail

C4

the constant of clusters control—materials on the line belt may not be well-distributed to clusters, and when the distance between the two neighboring clusters is larger than C4, the popping FIFO stacker will pause to reduce the material quantity waiting for separating and in process of separating, which can also shorten the scanning period of the PLC-like thread cycle, as shown in Fig. 12 in detail

FIFO stacker

first-in first-out stacker—its element is in S_Wp type

Sem_Push

semaphore of the push FIFO stacker—true is permitted

Sem_Pop

semaphore of the pop FIFO stacker—true is permitted

WS

the number of candidate separating and in separating materials, whose data have been popped out of the FIFO stacker

S_Wp

the defined structure, which includes the material sequence number, center coordinate value, and material class value—the element of the FIFO stacker is in this data type

P0

the S_Wp type variable, temporary stored data popping FIFO stacker

P1

the S_Wp type array to store the data of waiting for separation and in the process of separating materials, whose size is usually no more than that of the FIFO stacker

PopEnFifo

bool type flag of the FIFO stacker pop enable, enable = true/disable = false

See Fig. 12