Two new parameters for the ordinal analysis of images

Ten years ago, Ribeiro et al.⁵ started the study of ordinal patterns in images. Rosso et al.⁴ had introduced the entropy–complexity plane to distinguish chaos and noise in one-dimensional signals. This physical background was appropriate for the study of phase transitions of liquid crystals and Ising systems in two dimensions. The methodology was applied by Zunino and Ribeiro⁶ to the study of textures in images by two-dimensional ordinal patterns on different scales. In the sequel, Sigaki et al.⁷ predicted physical properties of liquid crystals and studied historical artwork, observing distinct clusters of artistic styles.⁸ Brazhe⁹ introduced a multiscale algorithm, and Azami et al.¹⁰ suggested a new ordinal entropy concept. Pessa and Ribeiro¹¹ investigated transition probabilities between neighboring ordinal patterns and corresponding networks.

There is a huge demand for fast methods evaluating textures. An unbelievable amount of image data is produced each day, ranging from microscopic pictures in cancer and virus detection to multispectral satellite images. They have to be screened automatically to fix regions where valuable information could be. Computers must find similar phenomena and do some classification before men and women will see the picture. There are many tools for studying local structures.³ Ordinal parameters, such as permutation entropy,¹² have shown to be fast, simple, and robust in one dimension. It is tempting to transfer them to two dimensions where quadratic data size requires just these properties.

Here, we study $2×2$ ordinal patterns in images and introduce two new parameters which we call smoothness $τ$ and branching structure $κ.$ The basic observation is that the 24 patterns can be divided into three types. At least for isotropic textures, the distinction between the three types seems to be the essential information contained in the frequency distribution of the 24 ordinal patterns. Taking two contrasts of frequencies, similar to the one-dimensional case,¹³ images can be represented in the $τ$– $κ$-plane, similar to the entropy–complexity plane.

In Sec. II, we give the basic definitions and comment on interpretation, calculation, and relevance of our parameters. There is also a theoretical discussion concerning statistical dependence of $τ$ and $κ$⁠. In Sec. III, we apply our method to the Kylberg–Sintorn rotation database,^14–16 which contains 900 unrotated and rotated samples of 25 textures. The results are very promising. Our parameters distinguish many of the textures, show a very small variance within structures, and are rotation-invariant for isotropic textures. We also apply the method to some photos and fractal surfaces.

The definition of ordinal $2×2$ patterns was introduced by Ribeiro et al. in 2012.⁵ We slightly change the notation of permutations, using rank numbers, which are directly linked to the spatial visualization of the patterns (cf.¹³ Sec. II). Note that the entries of a monochrome image matrix are real numbers, often integers between 0 and 255, which represent light intensities or shades of gray (⁠ $0= black$⁠, $255= white$⁠). There are other applications where the values in a matrix represent certain physical quantities.

We now replace the values $w k$ by their rank numbers 1, 2, 3, 4 within this small matrix: 1 denotes the minimum, and 4 denotes the maximum. In general, $r k$ denotes the number of $w j$⁠, which fulfill $w j≤ w k.$

An example of the construction of an $2×2$ ordinal pattern is shown in Fig. 1. The smallest of the values is $w 3$⁠, which is pictured with the smallest height on the left and the darkest gray tone. On the right-hand side of the figure are the ranks that form the ordinal pattern.

Instead of the matrix (1) of immediately neighboring values, one can also take the values $x m n$⁠, $x m n + d$⁠, $x m + d n$⁠, and $x m + d n + d$⁠, which are $d$ steps apart. The resulting matrix of ranks will then be called an ordinal pattern at position $(m,n)$ with delay $d.$ Our initial definition of an ordinal pattern is the case $d=1.$ Clearly, $d$ is a scale parameter, and we may think of a zoom factor.

In the study of time series in one dimension, $d$ is called embedding delay. Zunino and Ribeiro⁶ studied different delays for the $x$- and $y$-direction.

As usual in the study of ordinal patterns, we exclude equality of values, which are compared, and enforce this assumption by adding a tiny white noise to the data matrix $X.$ This is a correct definition even though different randomizations may result in slightly different pattern frequencies. Digitized images are themselves random discretizations of three-dimensional continuous scenes; therefore, we live with random errors anyway.

As an illustration, consider the case that all four values in (1) are equal. Assigning this case to a particular ordinal pattern would result in a large bias toward this pattern. Randomization means assigning each such case to any possible pattern with probability $1/24.$ This results in small random deviations and works well in practice.

For the types defined below, however, ties are less troublesome. As mentioned at the end of Sec. II D, they can be treated without randomization.

There are $4!=24$ different $2×2$ ordinal patterns. Let us arrange them according to shape characteristics. Fixing rank 1 at the upper left corner, we obtain six basic ordinal patterns, shown in the first row of Fig. 2. They form three pairs of patterns according to the ranks, which are diagonally opposite of each other. The patterns of each pair are mapped into each other by reflection at the main diagonal. Rotating the six basic patterns by $90 °$⁠, $180 °$⁠, and $270 °$⁠, we derive the other 18 patterns. In this way, the 24 ordinal patterns are arranged in three sets of eight patterns, which we denote with “type I,” “type II,” and “type III” (Fig. 2).

For one-dimensional ordinal patterns, symmetry classes with respect to time and order reversal were studied by Sinn and Keller.¹⁷ They come up in the study of Gaussian processes.¹⁸ 

Let us turn to the interpretation of types. Figure 3 shows a spatial visualization of the three types. In type I, values are either increasing in both rows or decreasing in both rows and the same for columns. This type represents smoothness when gray values represent a function over a plane region, for example, temperature.

For type II, the parallel increase or decrease holds for either rows or columns. In the other direction, we have one increase and one decrease. In a smooth surface, that could only happen when one partial derivative becomes zero somewhere inside our small square. Of course, discrete images are not smooth. Type II frequently occurs at tree-like structures of an image.

In type III, both values of one diagonal are larger than both values of the other diagonal. This type can represent a saddle point or an edge or ridge in the diagonal direction. For a checkerboard image, all $2×2$ patterns are of type III. For white noise, all three types occur with the same frequency.

The concept of type of a $2×2$ pattern seems new. However, there is a similar concept in statistics when we consider the interaction of two dichotomic variables $x,y$ (“smoker” and “drinker,” say) on a numeric variable $z$ (“blood pressure”). For a $2×2$ table of means of $z$ depending on $x$– $y$-combinations, the classification of interaction into “pure ordinal,” “hybrid,” and “pure disordinal” introduced by Leigh and Kinnear¹⁹ exactly corresponds to our types I, II, and III.

In this paper, we derive other parameters, which summarize the $p k$⁠. To this end, we study real texture data of Kylberg and Sintorn.^14–16 Their database contains 100 samples for each of 25 carefully prepared textures and each of 9 rotations, altogether more than 20 000 images of $122×122$ pixels. Details are given in Sec. III. Here, we explain why it makes sense to go from patterns to types.

The 24 patterns correspond to the columns of Fig. 4. They were enumerated according to the three blocks of Fig. 2 and rowwise within each block. Therefore, patterns 1–8 are of type I, 9–16 of type II, and 17–24 of type III. The rows in Fig. 4 correspond to the 25 textures in the database. The color in field $(i,k)$ represents the mean frequency $p k$ of pattern $k$ in 100 unrotated samples of texture $i$⁠.

The picture shows a clear distinction between patterns of different types throughout all textures. The probability of a pattern is about 0.07 for type I, about 0.04 for type II, and almost constant 0.015 for type III. In the structures of rows 1, 3, 13, 17, 18, 20, 22, and 25, the frequencies of patterns of the same type almost coincide. These are isotropic textures, as explained below. In such cases, all the information of the pattern distribution $p$ lies in the probabilities $q 1, q 2, q 3$ of the types.

There are other images, rows 2, 5–8, 11, 12, 15, and 24, where probabilities within type I and/or II show clear differences. These textures have some dominating directions, and differences of the $p k$ within types can give information about these directions. However, even in these cases, the main part of the information of $p$ is contained in $q 1, q 2, q 3$⁠.

Thus, from a practical viewpoint, the focus on types is justified. Compressing 24 probabilities $p k$ to just three parameters $q j$ is connected with a small loss of information.

The vector of type frequencies belongs to the simplex $S={( q 1, q 2, q 3) | q 1+ q 2+ q 3=1}.$ In two-dimensional projection, this is an equilateral triangle with barycentric coordinates $q j,$ as shown in Fig. 5(a). We are looking for an orthogonal coordinate system, which expresses the information of $q j$ by two parameters. First, let us note that vectors of type frequencies can be anywhere in the simplex.

The upper row of Fig. 6 shows periodic patterns of pure type. The example for type I is not smooth. The checkerboard pattern is typical for type 3, but even in the case $q 3=1$⁠, it may be obscured. For example, since our gray tones are integers, addition of a function $ax+by$ with $|a |<1$ and $|b |<1$ will give us the impression of an image with a gradient while leaving the order relation between horizontal and vertical neighbors and q unchanged. The lower row of Fig. 6 shows how two types can mix. Usually, all three types do mix locally in quite different ways. From this perspective, there are no general rules for the interpretation of $q j.$

Let us turn from artificial constructions to real data. For the Kylberg–Sintorn collection of fairly representative everyday textures, we had $q 1> q 2> q 3$ in Fig. 4. These inequalities determine a triangle, colored in Fig. 5(a), which contains our region of interest. Figure 5(b) shows a close-up of the triangle, with mean frequency vectors $q$ for the 25 unrotated textures. We need an orthogonal coordinate system for this region.

For one-dimensional ordinal patterns, similar parameters are discussed in Ref. 13.

The smoothness parameter $τ$ is defined and denoted in the same way as persistence of one-dimensional order patterns.¹⁸ Its maximum $2/3$ is realized for increasing or decreasing functions. The theoretical minimum $−1/3$ is assumed for alternating sequences or checkerboard patterns. In reality, $τ$ rarely assumes negative values.

The parameter $κ$ does not exist in one dimension and is more difficult to interpret. In some way, it describes how much branching and spiral structures dominate checkerboard-like noise in the image. The theoretical bounds of $κ$ are $−$1 and 1, and its range in all images of Sec. III is between 0 and 0.3.

An alternative choice of orthogonal coordinates is shown in Fig. 5(c). We take the same origin point and the parameters $1/3− q 3$ and $q 1− q 2$ so that the region of interest is in the positive quadrant. In Sec. II F, we show that under certain assumptions, these parameters are uncorrelated. On the other hand, they are difficult to interpret and did not classify textures well. Their values appeared in a narrow strip, such as the entropy–complexity combinations in Sec. III E. Favoring classification strength more than the total lack of correlation, we decided to take $τ$ and $κ$ as our parameters. As we shall see, their correlation is positive, but very small.

Since types are based on horizontal and vertical comparisons, ties on the diagonal of a $2×2$ pattern do not matter. Ties have to be considered in four cases: (1) two equal values in one row or column, (2) two pairs of equal values in rows or columns, (3) three equal values, or (4) four equal values. Randomization, which is our preferred method, will assign each case of (3) or (4) to each of the three types with probability $1 3.$ In case (2), we get type I or II with probability $1 2$⁠. This also holds in case (1) when the equal values represent the maximum or minimum of the pattern. Otherwise, we get type II or III with probability $1 2$⁠.

This allows abandonment of randomization, determining the equality cases by logical matrix operations and dividing their frequencies deterministically among the types. Moreover, it is possible to change the weights by assigning all instances of case (4) to type I. A constant function is certainly smooth. We can also assign $2 3$ of case (3) to type I and only $1 6$ to the other two types, but such rules must be confirmed by experience. For our data, case (4) had a frequency of 1% or smaller and case (3) of less than 5% for all data except four textures. We tried to change the rules for ties and did not obtain substantial changes.

For one-dimensional patterns, dependence of successive patterns is a big problem, see Elsinger,²⁰ Weiß,²¹ and de Sousa and Hlinka.²² For $2×2$ neighboring ordinal patterns, transition probabilities were investigated and corresponding Markov chains constructed by Pessa and Ribeiro.¹¹ Horizontal and vertical neighbors have to be distinguished. Many pairs of patterns cannot occur as neighbors. This creates strong statistical dependence between pattern probabilities $p k.$

Fortunately, the dependence between neighboring types is much weaker. There are no forbidden pairs of neighbor types. For the white noise model, we now explicitly calculate the dependence. The white noise assumptions imply that types of disjoint $2×2$ squares are independent and that there are no ties (with probability one). Since any two intersecting $2×2$ squares appear within a $3×3$ block, we can restrict our study to a $3×3$ data matrix $X.$ White noise means that the entries of $X$ are independent and identically distributed random numbers, and this implies that they are exchangeable. That is, all orderings between the nine entries of $X$ are equally likely.

In our setting, $t1,…,t4$ are categorical random variables. Their values I, II, and III for every case are names even though in MATLAB, they will be denoted 1, 2, and 3. Such variables have no mean, but we can study their independence. The following program runs 3 s on a PC:

Now, we can evaluate the dependence of types. For instance, $mean(t1==2)$⁠, the probability that $t1$ is type II is of course $1/3$ as well as for all other types.

Also, $mean(t1==2 \& t4==3)$⁠, the probability that $t1$ is type II and $t4$ is type III turns out to be $1/9.$ Thus, the two events $A={ω |t1(ω)=2}$ and $B={ω |t4(ω)=3}$ are independent by virtue of the formula $P(A∩B)=P(A)P(B).$

It is convenient to calculate it for all types $j=1,2,3$ of $t1$ and all types $k=1,2,3$ of $t4$ and collect the results in a matrix $C,$ the covariance matrix of types of $t1$⁠, and its diagonal neighbor $t4$⁠:

$for j=1:3; for k=1:3;$

$c(j,k)= mean(t1==j \& t4==k)-1/9; end; end.$

This program returns a $3×3$ zero matrix. Thus, there are no dependencies whatsoever between the type of a $2×2$ pattern and its diagonal neighbor type. Due to symmetry, this also holds for $t2,t3.$ For horizontal neighbors $t1,t2,$ and for vertical neighbors $t1,t3$⁠, MATLAB determines another matrix.

Thus, there is a tendency of all types, particularly type 3, to favor neighbors of the same type, and a tendency between type 3 and the other two types to avoid neighboring, while types 1 and 2 are neutral to each other. This effect is small, almost negligible: the correlation coefficient (covariance divided by $2/9$⁠) between type 3 neighbors is 0.05, and the other correlations are still closer to zero.

There could be dependencies between three or four neighbors, which are not detected by the covariance matrix. This can be checked by establishing the statistics of all possible combinations of the four types inside our $3×3$ matrix. For Fig. 7, we evaluated a combined index $tc$ as follows: $tc=t1-1+3*(t2-1)+9*(t3-1)+27*(t4-1); hist (tc,81)$⁠.

A small calculation proved that indeed, the probability of a $2×2$ pattern to have type III increases from 0.333 to 0.367, 0.414, and 0.575 if we assume that one, two in a line, or three neighboring patterns within the $3×3$ block are type III. However, the histogram of all possible combinations of the four types shows for 3,3,3,3, the only frequency that considerably exceeds the mean $9!/81$ indicated by the line in Fig. 7. As compensation, the four combinations of one type I and three types III have the smallest frequency. Combinations 1,1,1,1 and 2,2,2,2 have the second highest frequency. On the whole, dependencies caused by two or three neighbors are small and do not change the picture.

We have dealt with local dependencies of types. What we really want to know, however, is the dependence of $q 1, q 2, q 3$ and of $τ$ and $κ$ for an $M×N$ data matrix. We assume that the data were obtained from white noise so that $q j$ are random variables. Let $U=(M−1)(N−1)$ be the number of entries of the type matrix.

Although white noise is a special model, the main message is that $τ$ and $κ$ are almost uncorrelated and do not contain much common information. This is in sharp contrast to pattern frequencies in the one- and two-dimensional case.^11,20–22 The theorem also says that $Cov( q 1, q 3)= Cov( q 2, q 3)$ so that the variables $1/3− q 3$ and $q 1− q 2$ are uncorrelated.

The Kylberg–Sintorn rotation dataset consists of $25$ different textures obtained from bulk solids, such as lentils, grains, and sprinkles as well as more regular structures, such as woven materials and knitwear. The database is publicly available in Ref. 15.

Each texture and rotation is represented by $100$ image samples, which were obtained by cutting the original photo into small tiles following a $10×10$ grid. Each tile has a size of $122×122$ pixels, and the gray values were normalized to a mean value of $127$ and a standard deviation of $40$⁠. Pictures of all 25 textures can be found in Ref. 15 as well as in the paper of Kylberg and Sintorn¹⁴ and the dissertation.¹⁶  Figure 9 shows six of the structures.

All photos are also provided under 8 rotations by multiples of $40 °.$

Rotations of the textures were produced with different techniques. Part of the data comprises software rotations of the textures using different interpolation methods, and the goal of research was the comparison of such algorithms. The highlight of the database is a set hardware rotations, implemented by turning the camera. Figure 2 in Ref. 14 shows the sophisticated setup to derive photos of comparable quality for all textures and angles. In our work, we only use these hardware rotations. Thus, we have 900 samples of each structure: 100 unrotated and 800 rotated ones.

For each tile of $122×122$ pixels, we determined the type frequencies $q 1$⁠, $q 2$⁠, and $q 3$ of the $2×2$ ordinal patterns and calculated $τ$ and $κ$⁠. As the tiles are pretty small, we used an embedding delay of $d=1$⁠.

Figure 8 shows the parameter pairs $(τ,κ)$ for all 2500 unrotated samples of Kylberg and Sintorn. Each texture is represented by a cluster of 100 points.

It is surprising how small and compact the clusters are, with the exception of sprinkles, canvas, rug, and wheat. Although the images are fairly small, the parameters $τ$ and $κ$ are consistently estimated with differences of less than $±0.02$ to the cluster mean.

Some of the materials are clearly identified by the two parameters. Especially rice but also lentils, which are overlapping with rug, are separated from all other types of fabric and knitwear. They have many patterns of type III and rather few of type I, resulting in small values of $τ$ and $κ$⁠. In the case of rice, this seems partly due to noise in the photo (see Fig. 9). Some other materials seem to be isolated from the crowd but will not stay in this position when we look at the rotated images (see Sec. III C and Fig. 10). Figure 4 shows that for rice depicted in row $16$⁠, all type I patterns have small frequencies. Some noise, from the camera sensor or from our treatment of equal values, can also be seen in lentils. Of course, our parameters do not allow distinction of effects of the material, light sources, or camera focus settings.

The largest smoothness $τ$ was obtained for fabric 3 and 4, which have almost the same parameter values. The largest $κ$ is reached by fabric 5, which has many curves almost parallel to the axes; cf. Fig. 9. This value will decrease when the material is rotated. Already in Fig. 4, fabric 5 in row 8 showed different pattern probabilities within type I, which indicates dominating directions in the texture.

Finally, let us compare rice and sprinkles in Fig. 9. Both materials consist of lengthy kind of cylinders. However, rice grains are shorter and arranged in an irregular way, while sprinkles are longer and form bundles of parallel pieces. This leads to preferential directions in the samples, although there are no preferred directions in the large photo. This seems the reason for the large and irregular size of the sprinkles cluster in Fig. 8. Actually, also, the frequencies of patterns of type I did vary much more for sprinkles than for any other texture.

The results of this experiment were encouraging but must not be overestimated. There are problems with equal gray values, which affect up to 70% of the $2×2$ patterns. The method of adding noise could in such cases create artificial noise. It is a surprise that it works so well.

For the six textures in Fig. 9, we now consider the parameters of the rotated photos. Together with the unrotated samples, we plot 900 points for each material.

It turns out that for isotropic textures, as shown in the upper row of Fig. 9, there is no effect of rotation. Figure 10 shows that clusters stay in the same location. Although our parameters are based on square patterns, they are rotation-invariant for isotropic textures. Note that the clusters for rice, lentils, and oatmeal are disjoint according to the change of the $x$-scale.

The non-isotropic textures in the lower row behave differently. Knitwear 3 has clear vertical structures, which apparently did not interact with rotations. Our parameters seem to evaluate the irregularly directed mini-threads in the photo.

The long cylinders of sprinkles already caused rotation effects among the samples in the unrotated case, as discussed above. Rotating the whole picture did not much increase this variation.

However, for some of the materials, we noticed a small shift of one rotation to higher $τ$ values, e.g., the dark blue dots for sprinkles in Fig. 10. A reason may be that the affected photos were taken with a slight change of focus, which resulted in a softer and smoother image.

Fabric 5 shows very clear rotation effects, which were also seen in other fabrics and canvas. This is due to the clear horizontal and vertical structures. Unrotated pictures show very large $κ$ values; cf. Fig. 8. Rotations around $40 °$ and $320 °$ are associated with the smallest $κ$⁠, followed by $120 °$ and $240 o$⁠. Nearest to the unrotated values are rotations by $80 °$ and $280 °$⁠, followed by $160 °$ and $200 °$⁠. Thus, we expect the smallest change for multiples of $90 °$ and the largest effect for $45 °$⁠.

Can we test our parameters also with our own archive of photos? There are some difficulties. The pixel structure can only be evaluated if we have a pixel format, such as tiff, png, or bmp. The compressed jpeg format is not appropriate. Next, color photos must be transformed to grayscale images.

The main problem is the high resolution of photos compared to low resolution of graytones, which causes much equality among values of neighboring pixels. Moreover, photos usually contain many objects with further details so that there is no uniform texture at all. In order not to get a mean of many different textures, one has to consider small parts of the photo.

Figure 11 shows four pairs of detail photos with $640×640$ pixels cut out from four larger photos. The delay $d=1$ used for the textures does not give much difference between the parameters of the photos due to little difference of values of neighboring pixels. On this scale, all photos were rather smooth. Only for delay 5, the character of the image becomes visible. Lanterns and tomato remain smooth, while the fur of an animal has parameters near $(0,0),$ the point of white noise. Fur and trees with their fractal structure involve many type III patterns. Forest was taken from the background with some fog, which smoothes the structure.

Fractal surfaces are one of the few model classes of images, which can be generated by a stochastic algorithm. Depending on the Hurst exponent $H∈[0,1]$⁠, the resulting surface is rough for small $H$ and smooth for larger $H$⁠. Ribeiro et al.⁵ and Zunino and Ribeiro⁶ studied ordinal patterns in fractal surfaces and represented them in the entropy–complexity causality plane.

For our study, we also used the midpoint displacement algorithm as described by Peitgen and Saupe.²⁴ We simulated 100 fractal surfaces of size $2049×2049$ for each Hurst exponent $H=0.1,0.2,…,0.9.$ We determined the distributions of $2×2$ patterns and types with embedding delays $d=1,…,200$⁠. Based on these distributions, we calculated entropy and complexity in accordance with Refs. 5 and 6 and our two parameters.

Due to the self-similarity of fractional Brownian motion, the results do not essentially depend on $d$⁠; see Ref. 6. However, there are numeric effects for small and large $d$⁠. The smallest variation was obtained for $d=10$⁠, which is very similar to 20 or 50. Figure 12 shows separated point clouds for the parameter values. As expected, large $H$ corresponds to large values of $τ$ and $κ$⁠, indicating smoothness. For small $H$⁠, the point of white noise (formally $H=0$⁠) is approached, which is $(0,0)$ for our parameters and $(1,0)$ in the entropy–complexity plane.

We introduced two very simple ordinal parameters describing smoothness and branching structure in images. They could become part of the big toolbox of image processing. Applications range from virus detection to the analysis of satellite images. Our parameters were tested with the Kylberg–Sintorn rotation database. They showed a small variation in samples of the same texture and proved to be amazingly invariant under rotations for isotropic textures. Many structures can be separated by using just these two parameters.

Of course, the study of $2×2$ patterns expresses specific features of the microstructure of images. Larger patterns will provide more information. A principal obstacle is the number of possible permutations, which vastly increases with the size of the pattern. Our study shows, however, that by grouping into meaningful types, this number can be drastically reduced.

This paper is dedicated to Karsten Keller. We gratefully remember the time when we cooperated with him. With his great exploratory spirit and educational experience, he was a wonderful advisor for the diploma thesis of K.W.

We thank both referees for their helpful comments.

The authors have no conflicts to disclose.

Christoph Bandt: Conceptualization (equal); Methodology (equal); Visualization (equal); Writing – original draft (equal); Writing – review & editing (equal). Katharina Wittfeld: Conceptualization (equal); Methodology (equal); Visualization (equal); Writing – original draft (equal); Writing – review & editing (equal).

The Kylberg–Sintorn rotation dataset is publicly available.¹⁵ The eight photo tiles extracted from the photos of K.W. are available from the corresponding author upon reasonable request.