Local patterns play an important role in statistical physics as well as in image processing. Two-dimensional ordinal patterns were studied by Ribeiro et al. who determined permutation entropy and complexity in order to classify paintings and images of liquid crystals. Here, we find that the patterns of neighboring pixels come in three types. The statistics of these types, expressed by two parameters, contains the relevant information to describe and distinguish textures. The parameters are most stable and informative for isotropic structures.
Local correlations and transition probabilities have a long history in statistical physics. Around 2000, physicists used local entropy and complexity to describe pattern formation,1,2 while computer scientists became interested in local pattern statistics of textures.3 On ordinary level, the entropy–complexity plane was introduced by Rosso et al.4 for one-dimensional systems and extended to two-dimensional patterns in images by Ribeiro et al.5 Starting from the observation that patterns come in three types, this paper develops a different approach. Two parameters expressing the frequency of types define smoothness and branching structure of an image. It is shown that the parameters consistently describe textures and are well-suited to distinguish different structures.
Ten years ago, Ribeiro et al.5 started the study of ordinal patterns in images. Rosso et al.4 had introduced the entropy–complexity plane to distinguish chaos and noise in one-dimensional signals. This physical background was appropriate for the study of phase transitions of liquid crystals and Ising systems in two dimensions. The methodology was applied by Zunino and Ribeiro6 to the study of textures in images by two-dimensional ordinal patterns on different scales. In the sequel, Sigaki et al.7 predicted physical properties of liquid crystals and studied historical artwork, observing distinct clusters of artistic styles.8 Brazhe9 introduced a multiscale algorithm, and Azami et al.10 suggested a new ordinal entropy concept. Pessa and Ribeiro11 investigated transition probabilities between neighboring ordinal patterns and corresponding networks.
There is a huge demand for fast methods evaluating textures. An unbelievable amount of image data is produced each day, ranging from microscopic pictures in cancer and virus detection to multispectral satellite images. They have to be screened automatically to fix regions where valuable information could be. Computers must find similar phenomena and do some classification before men and women will see the picture. There are many tools for studying local structures.3 Ordinal parameters, such as permutation entropy,12 have shown to be fast, simple, and robust in one dimension. It is tempting to transfer them to two dimensions where quadratic data size requires just these properties.
Here, we study ordinal patterns in images and introduce two new parameters which we call smoothness and branching structure The basic observation is that the 24 patterns can be divided into three types. At least for isotropic textures, the distinction between the three types seems to be the essential information contained in the frequency distribution of the 24 ordinal patterns. Taking two contrasts of frequencies, similar to the one-dimensional case,13 images can be represented in the – -plane, similar to the entropy–complexity plane.
In Sec. II, we give the basic definitions and comment on interpretation, calculation, and relevance of our parameters. There is also a theoretical discussion concerning statistical dependence of and . In Sec. III, we apply our method to the Kylberg–Sintorn rotation database,14–16 which contains 900 unrotated and rotated samples of 25 textures. The results are very promising. Our parameters distinguish many of the textures, show a very small variance within structures, and are rotation-invariant for isotropic textures. We also apply the method to some photos and fractal surfaces.
A. Two-dimensional ordinal patterns
The definition of ordinal patterns was introduced by Ribeiro et al. in 2012.5 We slightly change the notation of permutations, using rank numbers, which are directly linked to the spatial visualization of the patterns (cf.13 Sec. II). Note that the entries of a monochrome image matrix are real numbers, often integers between 0 and 255, which represent light intensities or shades of gray ( , ). There are other applications where the values in a matrix represent certain physical quantities.
Let be a data matrix. The local pattern at position is the matrix
We now replace the values by their rank numbers 1, 2, 3, 4 within this small matrix: 1 denotes the minimum, and 4 denotes the maximum. In general, denotes the number of , which fulfill
An example of the construction of an ordinal pattern is shown in Fig. 1. The smallest of the values is , which is pictured with the smallest height on the left and the darkest gray tone. On the right-hand side of the figure are the ranks that form the ordinal pattern.
Instead of the matrix (1) of immediately neighboring values, one can also take the values , , , and , which are steps apart. The resulting matrix of ranks will then be called an ordinal pattern at position with delay Our initial definition of an ordinal pattern is the case Clearly, is a scale parameter, and we may think of a zoom factor.
In the study of time series in one dimension, is called embedding delay. Zunino and Ribeiro6 studied different delays for the - and -direction.
1. Treatment of ties
As usual in the study of ordinal patterns, we exclude equality of values, which are compared, and enforce this assumption by adding a tiny white noise to the data matrix This is a correct definition even though different randomizations may result in slightly different pattern frequencies. Digitized images are themselves random discretizations of three-dimensional continuous scenes; therefore, we live with random errors anyway.
As an illustration, consider the case that all four values in (1) are equal. Assigning this case to a particular ordinal pattern would result in a large bias toward this pattern. Randomization means assigning each such case to any possible pattern with probability This results in small random deviations and works well in practice.
For the types defined below, however, ties are less troublesome. As mentioned at the end of Sec. II D, they can be treated without randomization.
B. Three types of 2 × 2 patterns
There are different ordinal patterns. Let us arrange them according to shape characteristics. Fixing rank 1 at the upper left corner, we obtain six basic ordinal patterns, shown in the first row of Fig. 2. They form three pairs of patterns according to the ranks, which are diagonally opposite of each other. The patterns of each pair are mapped into each other by reflection at the main diagonal. Rotating the six basic patterns by , , and , we derive the other 18 patterns. In this way, the 24 ordinal patterns are arranged in three sets of eight patterns, which we denote with “type I,” “type II,” and “type III” (Fig. 2).
The three types describe the symmetry classes of patterns with respect to the symmetry group of a square.
When we reverse the order of a pattern—exchange 1 with 4 and 2 with 3—the resulting pattern has the same type.
The type of a pattern is the rank number, which shares a diagonal with 4. For instance, we have type II if 2 is on one diagonal with 4.
(i) was explained above and illustrated in Fig. 2. The symmetry group of the square contains the identity map, three rotations, two reflections at the diagonals, and two reflections at horizontal and vertical symmetry axes. Assertions (ii) and (iii) follow just from the inspection of Fig. 2. (ii) also follows from (iii). Order reversal agrees with rotation for type I and with a horizontal or vertical reflection for types II and III.
Let us turn to the interpretation of types. Figure 3 shows a spatial visualization of the three types. In type I, values are either increasing in both rows or decreasing in both rows and the same for columns. This type represents smoothness when gray values represent a function over a plane region, for example, temperature.
Suppose the gray values of our image over an open connected region all lie on a plane in 3-space with Then, all patterns for grid points in this region are of type I. This also holds if is a function with continuous and non-zero partial derivatives in this region.
For or, more generally, the function is increasing in the -direction throughout the region. For it is decreasing in the region. The same holds for an increase or decrease in the -direction and (more generally ) or respectively.
For type II, the parallel increase or decrease holds for either rows or columns. In the other direction, we have one increase and one decrease. In a smooth surface, that could only happen when one partial derivative becomes zero somewhere inside our small square. Of course, discrete images are not smooth. Type II frequently occurs at tree-like structures of an image.
In type III, both values of one diagonal are larger than both values of the other diagonal. This type can represent a saddle point or an edge or ridge in the diagonal direction. For a checkerboard image, all patterns are of type III. For white noise, all three types occur with the same frequency.
The concept of type of a pattern seems new. However, there is a similar concept in statistics when we consider the interaction of two dichotomic variables (“smoker” and “drinker,” say) on a numeric variable (“blood pressure”). For a table of means of depending on – -combinations, the classification of interaction into “pure ordinal,” “hybrid,” and “pure disordinal” introduced by Leigh and Kinnear19 exactly corresponds to our types I, II, and III.
C. Frequencies and relevance of types
In this paper, we derive other parameters, which summarize the . To this end, we study real texture data of Kylberg and Sintorn.14–16 Their database contains 100 samples for each of 25 carefully prepared textures and each of 9 rotations, altogether more than 20 000 images of pixels. Details are given in Sec. III. Here, we explain why it makes sense to go from patterns to types.
The 24 patterns correspond to the columns of Fig. 4. They were enumerated according to the three blocks of Fig. 2 and rowwise within each block. Therefore, patterns 1–8 are of type I, 9–16 of type II, and 17–24 of type III. The rows in Fig. 4 correspond to the 25 textures in the database. The color in field represents the mean frequency of pattern in 100 unrotated samples of texture .
The picture shows a clear distinction between patterns of different types throughout all textures. The probability of a pattern is about 0.07 for type I, about 0.04 for type II, and almost constant 0.015 for type III. In the structures of rows 1, 3, 13, 17, 18, 20, 22, and 25, the frequencies of patterns of the same type almost coincide. These are isotropic textures, as explained below. In such cases, all the information of the pattern distribution lies in the probabilities of the types.
There are other images, rows 2, 5–8, 11, 12, 15, and 24, where probabilities within type I and/or II show clear differences. These textures have some dominating directions, and differences of the within types can give information about these directions. However, even in these cases, the main part of the information of is contained in .
Thus, from a practical viewpoint, the focus on types is justified. Compressing 24 probabilities to just three parameters is connected with a small loss of information.
D. The two parameters
The vector of type frequencies belongs to the simplex In two-dimensional projection, this is an equilateral triangle with barycentric coordinates as shown in Fig. 5(a). We are looking for an orthogonal coordinate system, which expresses the information of by two parameters. First, let us note that vectors of type frequencies can be anywhere in the simplex.
When a -pattern of numbers is extended periodically to the right and downward, all patterns of resulting matrices have the same type.
The vectors of possible type frequencies of matrices are dense in the simplex
(i) In a periodic arrangement, the immediate right neighbor of the basic pattern is , which is just the reflection of the basic pattern at the vertical symmetry axis. Further right, the basic pattern is repeated. The immediate lower neighbor is a reflection at the horizontal axis, and the immediate diagonal neighbor is a rotation of the basic pattern. By Proposition 1, they all have the same type, as is demonstrated in the upper row of Fig. 6. (ii) Consider a rational point in the simplex and a positive integer Construct a square matrix of size with rows of type I at the top, rows of type II in the middle, and rows of type III at the bottom. Only two border lines of patterns are not controlled. The relative frequency of type is between and for , which converges to for .
The upper row of Fig. 6 shows periodic patterns of pure type. The example for type I is not smooth. The checkerboard pattern is typical for type 3, but even in the case , it may be obscured. For example, since our gray tones are integers, addition of a function with and will give us the impression of an image with a gradient while leaving the order relation between horizontal and vertical neighbors and q unchanged. The lower row of Fig. 6 shows how two types can mix. Usually, all three types do mix locally in quite different ways. From this perspective, there are no general rules for the interpretation of
Let us turn from artificial constructions to real data. For the Kylberg–Sintorn collection of fairly representative everyday textures, we had in Fig. 4. These inequalities determine a triangle, colored in Fig. 5(a), which contains our region of interest. Figure 5(b) shows a close-up of the triangle, with mean frequency vectors for the 25 unrotated textures. We need an orthogonal coordinate system for this region.
For one-dimensional ordinal patterns, similar parameters are discussed in Ref. 13.
The smoothness parameter is defined and denoted in the same way as persistence of one-dimensional order patterns.18 Its maximum is realized for increasing or decreasing functions. The theoretical minimum is assumed for alternating sequences or checkerboard patterns. In reality, rarely assumes negative values.
The parameter does not exist in one dimension and is more difficult to interpret. In some way, it describes how much branching and spiral structures dominate checkerboard-like noise in the image. The theoretical bounds of are 1 and 1, and its range in all images of Sec. III is between 0 and 0.3.
An alternative choice of orthogonal coordinates is shown in Fig. 5(c). We take the same origin point and the parameters and so that the region of interest is in the positive quadrant. In Sec. II F, we show that under certain assumptions, these parameters are uncorrelated. On the other hand, they are difficult to interpret and did not classify textures well. Their values appeared in a narrow strip, such as the entropy–complexity combinations in Sec. III E. Favoring classification strength more than the total lack of correlation, we decided to take and as our parameters. As we shall see, their correlation is positive, but very small.
1. Treatment of ties
Since types are based on horizontal and vertical comparisons, ties on the diagonal of a pattern do not matter. Ties have to be considered in four cases: (1) two equal values in one row or column, (2) two pairs of equal values in rows or columns, (3) three equal values, or (4) four equal values. Randomization, which is our preferred method, will assign each case of (3) or (4) to each of the three types with probability In case (2), we get type I or II with probability . This also holds in case (1) when the equal values represent the maximum or minimum of the pattern. Otherwise, we get type II or III with probability .
This allows abandonment of randomization, determining the equality cases by logical matrix operations and dividing their frequencies deterministically among the types. Moreover, it is possible to change the weights by assigning all instances of case (4) to type I. A constant function is certainly smooth. We can also assign of case (3) to type I and only to the other two types, but such rules must be confirmed by experience. For our data, case (4) had a frequency of 1% or smaller and case (3) of less than 5% for all data except four textures. We tried to change the rules for ties and did not obtain substantial changes.
E. Neighbor dependence of types
For one-dimensional patterns, dependence of successive patterns is a big problem, see Elsinger,20 Weiß,21 and de Sousa and Hlinka.22 For neighboring ordinal patterns, transition probabilities were investigated and corresponding Markov chains constructed by Pessa and Ribeiro.11 Horizontal and vertical neighbors have to be distinguished. Many pairs of patterns cannot occur as neighbors. This creates strong statistical dependence between pattern probabilities
Fortunately, the dependence between neighboring types is much weaker. There are no forbidden pairs of neighbor types. For the white noise model, we now explicitly calculate the dependence. The white noise assumptions imply that types of disjoint squares are independent and that there are no ties (with probability one). Since any two intersecting squares appear within a block, we can restrict our study to a data matrix White noise means that the entries of are independent and identically distributed random numbers, and this implies that they are exchangeable. That is, all orderings between the nine entries of are equally likely.
In our setting, are categorical random variables. Their values I, II, and III for every case are names even though in MATLAB, they will be denoted 1, 2, and 3. Such variables have no mean, but we can study their independence. The following program runs 3 s on a PC:
Now, we can evaluate the dependence of types. For instance, , the probability that is type II is of course as well as for all other types.
Also, , the probability that is type II and is type III turns out to be Thus, the two events and are independent by virtue of the formula
It is convenient to calculate it for all types of and all types of and collect the results in a matrix the covariance matrix of types of , and its diagonal neighbor :
This program returns a zero matrix. Thus, there are no dependencies whatsoever between the type of a pattern and its diagonal neighbor type. Due to symmetry, this also holds for For horizontal neighbors and for vertical neighbors , MATLAB determines another matrix.
The types of an ordinal pattern and one of its diagonal neighbors are statistically independent.
- The dependence of types of horizontal neighbors as well as of vertical neighbors is given by the following covariance matrix of types and :
Thus, there is a tendency of all types, particularly type 3, to favor neighbors of the same type, and a tendency between type 3 and the other two types to avoid neighboring, while types 1 and 2 are neutral to each other. This effect is small, almost negligible: the correlation coefficient (covariance divided by ) between type 3 neighbors is 0.05, and the other correlations are still closer to zero.
There could be dependencies between three or four neighbors, which are not detected by the covariance matrix. This can be checked by establishing the statistics of all possible combinations of the four types inside our matrix. For Fig. 7, we evaluated a combined index as follows: .
A small calculation proved that indeed, the probability of a pattern to have type III increases from 0.333 to 0.367, 0.414, and 0.575 if we assume that one, two in a line, or three neighboring patterns within the block are type III. However, the histogram of all possible combinations of the four types shows for 3,3,3,3, the only frequency that considerably exceeds the mean indicated by the line in Fig. 7. As compensation, the four combinations of one type I and three types III have the smallest frequency. Combinations 1,1,1,1 and 2,2,2,2 have the second highest frequency. On the whole, dependencies caused by two or three neighbors are small and do not change the picture.
F. Correlation of types and parameters
We have dealt with local dependencies of types. What we really want to know, however, is the dependence of and of and for an data matrix. We assume that the data were obtained from white noise so that are random variables. Let be the number of entries of the type matrix.
Although white noise is a special model, the main message is that and are almost uncorrelated and do not contain much common information. This is in sharp contrast to pattern frequencies in the one- and two-dimensional case.11,20–22 The theorem also says that so that the variables and are uncorrelated.
Let be the type matrix obtained from the data matrix. Instead of we consider the absolute frequencies of types in We have , where runs through the places in the matrix and if the entry in at place is and 0 otherwise.
Under the assumption of white noise, this sum contains zeros for all pairs of places , which are not equal and not neighbors in a row or column since the types of such pairs of places are independent. We are left with terms for which and terms for which is one of the four neighbors of Actually, we have less neighbors if is in the first or last row or column. This border effect is neglected for convenience and is the only reason to speak of an asymptotic formula.
III. APPLICATIONS AND EXAMPLES
A. The Kylberg–Sintorn rotation dataset of textures
The Kylberg–Sintorn rotation dataset consists of different textures obtained from bulk solids, such as lentils, grains, and sprinkles as well as more regular structures, such as woven materials and knitwear. The database is publicly available in Ref. 15.
Each texture and rotation is represented by image samples, which were obtained by cutting the original photo into small tiles following a grid. Each tile has a size of pixels, and the gray values were normalized to a mean value of and a standard deviation of . Pictures of all 25 textures can be found in Ref. 15 as well as in the paper of Kylberg and Sintorn14 and the dissertation.16 Figure 9 shows six of the structures.
All photos are also provided under 8 rotations by multiples of
Rotations of the textures were produced with different techniques. Part of the data comprises software rotations of the textures using different interpolation methods, and the goal of research was the comparison of such algorithms. The highlight of the database is a set hardware rotations, implemented by turning the camera. Figure 2 in Ref. 14 shows the sophisticated setup to derive photos of comparable quality for all textures and angles. In our work, we only use these hardware rotations. Thus, we have 900 samples of each structure: 100 unrotated and 800 rotated ones.
For each tile of pixels, we determined the type frequencies , , and of the ordinal patterns and calculated and . As the tiles are pretty small, we used an embedding delay of .
B. Results for unrotated textures
Figure 8 shows the parameter pairs for all 2500 unrotated samples of Kylberg and Sintorn. Each texture is represented by a cluster of 100 points.
It is surprising how small and compact the clusters are, with the exception of sprinkles, canvas, rug, and wheat. Although the images are fairly small, the parameters and are consistently estimated with differences of less than to the cluster mean.
Some of the materials are clearly identified by the two parameters. Especially rice but also lentils, which are overlapping with rug, are separated from all other types of fabric and knitwear. They have many patterns of type III and rather few of type I, resulting in small values of and . In the case of rice, this seems partly due to noise in the photo (see Fig. 9). Some other materials seem to be isolated from the crowd but will not stay in this position when we look at the rotated images (see Sec. III C and Fig. 10). Figure 4 shows that for rice depicted in row , all type I patterns have small frequencies. Some noise, from the camera sensor or from our treatment of equal values, can also be seen in lentils. Of course, our parameters do not allow distinction of effects of the material, light sources, or camera focus settings.
The largest smoothness was obtained for fabric 3 and 4, which have almost the same parameter values. The largest is reached by fabric 5, which has many curves almost parallel to the axes; cf. Fig. 9. This value will decrease when the material is rotated. Already in Fig. 4, fabric 5 in row 8 showed different pattern probabilities within type I, which indicates dominating directions in the texture.
Finally, let us compare rice and sprinkles in Fig. 9. Both materials consist of lengthy kind of cylinders. However, rice grains are shorter and arranged in an irregular way, while sprinkles are longer and form bundles of parallel pieces. This leads to preferential directions in the samples, although there are no preferred directions in the large photo. This seems the reason for the large and irregular size of the sprinkles cluster in Fig. 8. Actually, also, the frequencies of patterns of type I did vary much more for sprinkles than for any other texture.
The results of this experiment were encouraging but must not be overestimated. There are problems with equal gray values, which affect up to 70% of the patterns. The method of adding noise could in such cases create artificial noise. It is a surprise that it works so well.
C. Results for rotated textures
For the six textures in Fig. 9, we now consider the parameters of the rotated photos. Together with the unrotated samples, we plot 900 points for each material.
It turns out that for isotropic textures, as shown in the upper row of Fig. 9, there is no effect of rotation. Figure 10 shows that clusters stay in the same location. Although our parameters are based on square patterns, they are rotation-invariant for isotropic textures. Note that the clusters for rice, lentils, and oatmeal are disjoint according to the change of the -scale.
The non-isotropic textures in the lower row behave differently. Knitwear 3 has clear vertical structures, which apparently did not interact with rotations. Our parameters seem to evaluate the irregularly directed mini-threads in the photo.
The long cylinders of sprinkles already caused rotation effects among the samples in the unrotated case, as discussed above. Rotating the whole picture did not much increase this variation.
However, for some of the materials, we noticed a small shift of one rotation to higher values, e.g., the dark blue dots for sprinkles in Fig. 10. A reason may be that the affected photos were taken with a slight change of focus, which resulted in a softer and smoother image.
Fabric 5 shows very clear rotation effects, which were also seen in other fabrics and canvas. This is due to the clear horizontal and vertical structures. Unrotated pictures show very large values; cf. Fig. 8. Rotations around and are associated with the smallest , followed by and . Nearest to the unrotated values are rotations by and , followed by and . Thus, we expect the smallest change for multiples of and the largest effect for .
Can we test our parameters also with our own archive of photos? There are some difficulties. The pixel structure can only be evaluated if we have a pixel format, such as tiff, png, or bmp. The compressed jpeg format is not appropriate. Next, color photos must be transformed to grayscale images.
The main problem is the high resolution of photos compared to low resolution of graytones, which causes much equality among values of neighboring pixels. Moreover, photos usually contain many objects with further details so that there is no uniform texture at all. In order not to get a mean of many different textures, one has to consider small parts of the photo.
Figure 11 shows four pairs of detail photos with pixels cut out from four larger photos. The delay used for the textures does not give much difference between the parameters of the photos due to little difference of values of neighboring pixels. On this scale, all photos were rather smooth. Only for delay 5, the character of the image becomes visible. Lanterns and tomato remain smooth, while the fur of an animal has parameters near the point of white noise. Fur and trees with their fractal structure involve many type III patterns. Forest was taken from the background with some fog, which smoothes the structure.
E. Fractal surfaces
Fractal surfaces are one of the few model classes of images, which can be generated by a stochastic algorithm. Depending on the Hurst exponent , the resulting surface is rough for small and smooth for larger . Ribeiro et al.5 and Zunino and Ribeiro6 studied ordinal patterns in fractal surfaces and represented them in the entropy–complexity causality plane.
For our study, we also used the midpoint displacement algorithm as described by Peitgen and Saupe.24 We simulated 100 fractal surfaces of size for each Hurst exponent We determined the distributions of patterns and types with embedding delays . Based on these distributions, we calculated entropy and complexity in accordance with Refs. 5 and 6 and our two parameters.
Due to the self-similarity of fractional Brownian motion, the results do not essentially depend on ; see Ref. 6. However, there are numeric effects for small and large . The smallest variation was obtained for , which is very similar to 20 or 50. Figure 12 shows separated point clouds for the parameter values. As expected, large corresponds to large values of and , indicating smoothness. For small , the point of white noise (formally ) is approached, which is for our parameters and in the entropy–complexity plane.
We introduced two very simple ordinal parameters describing smoothness and branching structure in images. They could become part of the big toolbox of image processing. Applications range from virus detection to the analysis of satellite images. Our parameters were tested with the Kylberg–Sintorn rotation database. They showed a small variation in samples of the same texture and proved to be amazingly invariant under rotations for isotropic textures. Many structures can be separated by using just these two parameters.
Of course, the study of patterns expresses specific features of the microstructure of images. Larger patterns will provide more information. A principal obstacle is the number of possible permutations, which vastly increases with the size of the pattern. Our study shows, however, that by grouping into meaningful types, this number can be drastically reduced.
This paper is dedicated to Karsten Keller. We gratefully remember the time when we cooperated with him. With his great exploratory spirit and educational experience, he was a wonderful advisor for the diploma thesis of K.W.
We thank both referees for their helpful comments.
Conflict of Interest
The authors have no conflicts to disclose.
Christoph Bandt: Conceptualization (equal); Methodology (equal); Visualization (equal); Writing – original draft (equal); Writing – review & editing (equal). Katharina Wittfeld: Conceptualization (equal); Methodology (equal); Visualization (equal); Writing – original draft (equal); Writing – review & editing (equal).
The Kylberg–Sintorn rotation dataset is publicly available.15 The eight photo tiles extracted from the photos of K.W. are available from the corresponding author upon reasonable request.