Spectral simplicity of apparent complexity. I. The nondiagonalizable metadynamics of prediction

Complex systems—that is, many-body systems with strong interactions—are usually observed through low-resolution feature detectors. The consequence is that their hidden structure is, at best, only revealed over time. Since individual observations cannot capture the full resolution of each degree of freedom, let alone a sufficiently full set of them, the measurement time series often appear stochastic and non-Markovian, exhibiting long-range correlations. Empirical challenges aside, restricting to the purely theoretical domain, even finite systems can appear quite complicated. Despite admitting finite descriptions, stochastic processes with sofic support, to take one example, exhibit infinite-range dependencies among the chain of random variables they generate.¹ While such infinite-correlation processes are legion in complex physical and biological systems, even approximately analyzing them is generally appreciated as difficult, if not impossible. Generically, even finite systems lead to uncountably infinite sets of predictive features.² These facts seem to put physical sciences' most basic goal—prediction—out of reach.

We aim to show that this direct, but sobering conclusion is too bleak. Rather, there is a collection of constructive methods that address the hidden structure and the challenges associated with predicting complex systems. This follows up on our recent introduction of a functional calculus that uncovered new relationships among supposedly different complexity measures³ and that demonstrated the need for a generalized spectral theory to answer such questions.⁴ Those efforts yielded elegant, closed-form solutions for complexity measures that, when compared, offered insight into the overall theory of complexity measures. Here, providing the necessary background for and greatly expanding those results, we show that different questions regarding correlation, predictability, and prediction each require their own analytical structures, expressed as various kinds of hidden transition dynamic. The resulting transition dynamic among hidden variables summarizes symmetry breaking, synchronization, and information processing, for example. Each of these metadynamics, though, is built up from the original given system.

The shift in perspective that allows the new level of tractability begins by recognizing that—beyond their ability to generate many sophisticated processes of interest—hidden Markov models can be treated as exact mathematical objects when analyzing the processes they generate. Crucially, and especially when addressing nonlinear processes, most questions that we ask imply a linear transition dynamic over some hidden state space. Speaking simply, something happens, then it evolves linearly in time, then we snapshot a selected characteristic. This broad type of sequential questioning cascades, in the sense that the influence of the initial preparation cascades through state space as time evolves, affecting the final measurement. Alternatively, other, complementary kinds of questioning involve accumulating such cascades. The linear algebra underlying either kind is highlighted in Table I in terms of an appropriate discrete-time transition operator T or a continuous-time generator G of time evolution.

In this way, deploying linear algebra to analyze complex systems relies on identifying an appropriate hidden state space. And, in turn, the latter depends on the genre of the question. Here, we focus on closed-form expressions for a process' complexity measures. This determines what the internal system setup $〈·|$ and the final detection $|·〉$ should be. We show that complexity questions fall into three subgenres and, for each of these, we identify the appropriate linear dynamic and closed-form expressions for several of the key questions in each genre. See Table II. The burden of the following is to explain the table in detail. We return to a much-elaborated version at the end.

Associating observables $x∈A$ with transitions between hidden states $s∈S$⁠, gives a hidden Markov model (HMM) with observation-labeled transition matrices ${T(x):Ti,j(x)=Pr(x,sj|si)}x∈A$⁠. They sum to the row-stochastic state-to-state transition matrix $T=∑x∈AT(x)$⁠. (The continuous-time versions are similarly defined, which we do later on.) Adding measurement symbols $x∈A$ this way—to transitions—can be considered a model of measurement itself.⁵ The efficacy of our choice will become clear.

It is important to note that HMMs, in continuous and discrete time, arise broadly in the sciences, from quantum mechanics,^6,7 statistical mechanics,⁸ and stochastic thermodynamics^9–11 to communication theory,^12,13 information processing,^14–16 computer design,¹⁷ population and evolutionary dynamics,^18,19 and economics. Thus, HMMs appear in the most fundamental physics and in the most applied engineering and social sciences. The breadth suggests that the thorough-going HMM analysis developed here is worth the required effort to learn.

Since complex processes have highly structured, directional transition dynamics—T or G—we encounter the full richness of matrix algebra in analyzing HMMs. We explain how analyzing complex systems induces a nondiagonalizable metadynamics, even if the original dynamic is diagonalizable in its underlying state-space. Normal and diagonalizable restrictions, so familiar in mathematical physics, simply fail us here.

The diversity of nondiagonalizable dynamics presents a technical challenge, though. A new calculus for functions of nondiagonalizable operators—e.g., T^L or e^tG—becomes a necessity if one's goal is an exact analysis of complex processes. Moreover, complexity measures naively and easily lead one to consider illegal operations. Taking the inverse of a singular operator is a particularly central, useful, and fraught example. Fortunately, such illegal operations can be skirted since the complexity measures only extract the excess transient behavior of an infinitely complicated orbit space.

To explain how this arises—how certain modes of behavior, such as excess transients, are selected as relevant, while others are ignored—we apply the meromorphic functional calculus and new results for spectral projection operators recently derived in Ref. 4 to analyze complex processes generated by HMMs.

The following shows that this leads to a simplified spectral theory of weighted directed graphs, that even nondiagonalizable eigenspaces can be manipulated individually, and that, more specifically, the techniques can be applied to the challenges of prediction. The results developed here greatly extend and (finally) explain those announced in Ref. 3. The latter introduced the basic methods and results by narrowly focusing on closed-form expressions for several measures of intrinsic computation, applying them to prototype complex systems.

The meromorphic functional calculus, summarized in detail later, concerns functions of nondiagonalizable operators when poles (or zeros) of the function of interest coincide with poles of the operator's resolvent—poles that appear precisely at the eigenvalues of the transition dynamics. Pole–pole and pole–zero interactions transform the complex-analysis residues within the functional calculus. One notable result is that the negative-one power of a singular operator exists in the meromorphic functional calculus. We derive its form, note that it is the Drazin inverse, and show how widely useful and common it is.

For example, the following gives the first closed-form expressions for many complexity measures in wide use—many of which turn out to be expressed most concisely in terms of a Drazin inverse. Furthermore, spectral decomposition gives insight into the subprocesses of a complex system in terms of the spectral projection operators of the appropriate transition dynamic.

In the following, we emphasize that when we observe processes generated by a source capable of even the simplest computations, much of the predictable structure lies beyond pairwise correlation. We clarify how different measures of complexity quantify and distinguish nuanced aspects of what is predictable and what is necessary for prediction. We then give closed-form solutions for this quantification, resulting in a new level of rigor, tractability, and insight.

Sections II and III briefly review the relevant background in stochastic processes, the HMMs that generate them, and complexity measures. Several classes of HMMs are discussed in Sec. III. Mixed-state presentations (MSPs)—HMM generators of a process that also track distributions induced by observation—are reviewed in Sec. IV. They are key to complexity measures within an information-theoretic framing. Section V then shows how each complexity measure reduces to the linear algebra of an appropriate HMM adapted to the question genre.

To make progress at this point, we summarize the meromorphic functional calculus in Sec. VI. Several of its mathematical implications are discussed in relation to projection operators in Sec. VII and a spectral weighted directed graph theory is presented in Sec. VIII.

With this all set out, the sequel, Part II⁷⁹ finally derives the promised closed-form complexities of a process and outlines common simplifications for special cases. This leads to the discovery of the symmetry collapse index, which indicates the sophistication of finite computational structures hidden in infinite-Markov-order processes. Leveraging the functional calculus, Part II⁷⁹ introduces a novel extension—the complexity measure frequency spectrum and shows how to calculate it in closed form. It provides a suite of examples to ground the theoretical developments and works through in-depth a pedagogical example.

We first describe a system of interest in terms of its observed behavior, following the approach of computational mechanics, as reviewed in Ref. 20. Again, a process is the collection of behaviors that the system produces and their probabilities of occurring. A process's behaviors are described via a bi-infinite chain of random variables, denoted by capital letters $… Xt−2 Xt−1 Xt Xt+1 Xt+2…$⁠. A realization is indicated by lowercase letters $… xt−2 xt−1 xt xt+1 xt+2…$⁠. We assume values x_t belong to a discrete alphabet $A$⁠. We work with blocks $Xt:t′$⁠, where the first index is inclusive and the second exclusive: $Xt:t′=Xt…Xt′−1$⁠. Block realizations $xt:t′$ we often refer to as words w. At each time t, we can speak of the past $X−∞:t=…Xt−2Xt−1$ and the future $Xt:∞=XtXt+1…$⁠.

A process's probabilistic specification is a density over these chains: $P(X−∞:∞)$⁠. Practically, we work with finite blocks and their probability distributions $Pr(Xt:t′)$⁠. To simplify the development, we primarily analyze stationary, ergodic processes: those for which $Pr(Xt:t+L)=Pr(X0:L)$ for all $t∈ℤ, L∈ℤ+$⁠. In such cases, we only need to consider a process's length-L word distributions $Pr(X0:L)$⁠.

A common first step to understand how processes express themselves is to analyze correlations among observables. Pairwise correlation in a sequence of observables is often summarized by the autocorrelation function

where the bar above X_t denotes its complex conjugate, and the angled brackets denote an average over all times $t∈ℤ$⁠. Alternatively, the structure in a stochastic process is often summarized by the power spectral density, also referred to more simply as the power spectrum

where $ω∈ℝ$ is the angular frequency.²¹ Though a basic fact, it is not always sufficiently emphasized in applications that power spectra capture only pairwise correlation. Indeed, it is straightforward to show that the power spectrum $P(ω)$ is the windowed Fourier transform of the autocorrelation function $γ(L)$⁠. That is, power spectra describe how pairwise correlations are distributed across frequencies. Power spectra are common in signal processing, both in technological settings and physical experiments.²² As a physical example, diffraction patterns are the power spectra of a sequence of structure factors.²³ 

Other important measures of observable organization called Green–Kubo coefficients determine transport properties in near-equilibrium thermodynamic systems—but are rather more application-specific.^24,25 These coefficients reflect the idea that dissipation depends on the correlation structure. They usually appear in the form of integrating the autocorrelation of derivatives of observables. A change of observables, however, turns this into an integration of a standard autocorrelation function. Green–Kubo transport coefficients then involve the limit $limω→0P(ω)$ for the process of appropriate observables.

One theme in the following is that, though widely used, correlation functions and power spectra give an impoverished view of a process's structural complexity, since they only consider ensemble averages over pairwise events. Moreover, creating a list of higher-order correlations is an impractical way to summarize complexity, as seen in the connected correlation functions of statistical mechanics.²⁶ 

Information measures, in contrast, can involve all orders of correlation and thus help to go beyond pairwise correlation in understanding, for example, how a process' past behavior affects predicting it at later times. Information theory, as developed for general complex processes,¹ provides a suite of quantities that capture prediction properties using variants of Shannon's entropy $H[·]$ and mutual information $I[ · ;· ]$¹³ applied to sequences. Each measure answers a specific question about a process' predictability. For example:

• How much information is contained in the words generated? The block entropy¹ 

  $H(L)=−∑w∈ALPr(w) log2Pr(w).$
• How random is a process? Its entropy rate¹² 

  $hμ=limL→∞H(L)/L.$

• For dynamical systems with a continuous phase-space $B$⁠, the metric entropy also known as Kolmogorov–Sinai (KS) entropy is the supremum of entropy rates induced by partitioning $B$ into different finite alphabets $A$⁠.²⁷ 

• How is the irreducible randomness $hμ$ approached? Via the myopic entropy rates²⁸ 

  $hμ(L)=H[X0|X1−L…X−1].$
• How much of the future can be predicted? Its excess entropy, which is the past–future mutual information:[Ref. 1, and references therein]

  $E=I[X−∞:0;X0:∞].$

  E has also been investigated in the ergodic theory²⁹ and under the names stored information,³⁰ effective measure complexity,³¹ and predictive information.³² 

• How much information must be extracted to know its predictability and so see its intrinsic randomness $hμ$? Its transient information¹ 

  $T=∑L=0∞[E+hμL−H(L)].$

The spectral approach, our subject, naturally leads to allied, but new information measures. To give a sense, later we introduce the excess entropy spectrum $E(ω)$⁠. It completely, yet concisely, summarizes the structure of myopic entropy reduction, in a way similar to how the power spectrum completely describes autocorrelation. However, while the power spectrum summarizes only pairwise linear correlation, the excess entropy spectrum captures all orders of nonlinear dependency between random variables, making it an incisive probe of hidden structure.

Before leaving the measures related to predictability, we must also point out that they have important refinements—measures that lend a particularly useful, even functional, interpretation. These include the bound, ephemeral, elusive, and related informations.^33,34 Though amenable to the spectral methods of the following, we leave their discussion for another venue. Fortunately, their spectral development is straightforward, but would take us beyond the minimum necessary presentation to make good on the overall discussion of spectral decomposition.

Process predictability measures, as just enumerated, certainly say much about a process' intrinsic information processing. They leave open, though, the question of the structural complexity associated with implementing prediction. This challenge entails a complementary set of measures that directly address the inherent complexity of actually predicting what is predictable. For that matter, how cryptic is a process?

Computational mechanics describes minimal-memory maximal prediction—using the minimal memory necessary to predict everything that is predictable about the future—via a process' hidden, effective or causal states and transitions, as summarized by the process's ϵ-machine.²⁰ A causal state $σ∈S+$ is an equivalence class of histories $x−∞:0$ that all yield the same probability distribution over observable futures $X0:∞$⁠. Therefore, knowing a process's current causal state—that $S0+=σ$⁠, say—is sufficient for maximal prediction.

The computational mechanics framework can also be related to several more recent attempts at describing effective levels of complex systems. For example, if individual histories are taken to be the microstates of a stochastic process, then causal states are the minimal high-level description of a stochastic process that satisfies the informational closure criterion of Ref. 35.

Computational mechanics provides an additional suite of quantities that capture the overhead of prediction, again using variants of Shannon's entropy and mutual information applied to the ϵ-machine. Each also answers a specific question about an observer's burden of prediction. For example:

• How much historical information must be stored for maximal prediction? The Shannon information in the causal states or statistical complexity³⁶ 

  $Cμ=H[S0+].$
• How unpredictable is a causal state upon observing a process for duration L? The myopic causal-state uncertainty¹ 

  $H+(L)=H[S0+|X−L…X−1].$
• How much information must an observer extract to synchronize to—that is, to know with certainty—the causal state? The optimal predictor's synchronization information¹ 

  $S=∑L=0∞H+(L).$

Paralleling the purely informational suite of the previous Sec. II B, we later introduce the optimal synchronization spectrum $S(ω)$⁠. It completely and concisely summarizes the frequency distribution of state-uncertainty reduction, similar to how the power spectrum $P(ω)$ completely describes autocorrelation and the excess entropy spectrum $E(ω)$ the myopic entropy reduction. Helpfully, the above optimal prediction measures can be found from the optimal synchronization spectrum.

The structural complexities monitor an observer's burden in optimally predicting a process. And so, they have practical relevance when an intelligent artificial or biological agent must take advantage of a structured stochastic environment—e.g., a Maxwellian Demon taking advantage of correlated environmental fluctuations,³⁷ prey avoiding easy prediction, or profiting from stock market volatility, come to mind.

Prediction has many natural generalizations. For example, since maximal prediction often requires infinite resources, sub-maximal prediction (i.e., predicting with lower fidelity) is of practical interest. Fortunately, there are principled ways to investigate the tradeoffs between predictive accuracy and computational burden.^2,38–40 As another example, maximal prediction in the presence of noisy or irregular observations can be investigated with a properly generalized framework; see Ref. 41. Blending the existing tools, resource-limited prediction under such observational constraints can also be investigated. There are also many applications where prediction is relevant to the task at hand, but is not necessarily the ultimate objective; this of course has a long history, and Ref. 42 has recently tried to formalize this effort. In all of these settings, information measures similar to those listed above are key to understanding and quantifying the tradeoffs arising in prediction.

Having highlighted the difference between prediction and predictability, we can appreciate that some processes hide more internal information—are more cryptic—than others. It turns out, this can be quantified. The crypticity $χ=Cμ−E$ is the difference between the process's stored information $Cμ$ and the mutual information E shared between past and future observables.⁴³ Operationally, crypticity contrasts predictable information content E with an observer's minimal stored-memory overhead $Cμ$ required to make predictions. To predict what is predictable, therefore, an optimal predictor must account for a process's crypticity.

How does a physical system produce its output process? This depends on many details. Some systems employ vast internal mechanistic redundancy, while others under constraints have optimized internal resources down to a minimally necessary generative structure. Different pressures give rise to different kinds of optimality. For example, minimal state-entropy generators turn out to be distinct from minimal state-set generators.^44–46 The challenge then is to develop ways to monitor differences in the generative mechanism.⁴⁷ 

Any generative model^1,48 $M$ with state-set $S$ has a statistical state complexity (state entropy): $C(M)=H[S]$⁠. Consider the corresponding myopic state-uncertainty given L sequential observations

And so

We also have the asymptotic uncertainty $H≡limL→∞H(L)$⁠. Related, there is the excess synchronization information

Such quantities are relevant even when an observer never fully synchronizes to a generative state; i.e., even when $H>0$⁠. Finite-state ϵ-machines always synchronize^49,50 and so their $H$ vanishes.

Since many different mechanisms can generate a given process, we need useful bounds on the statistical state complexity of possible process generators. For example, the minimal generative complexity $Cg=minC(M)$⁠, where we minimize over all models that generate the process, is the minimal state-information a physical system must store to generate its future.⁴⁶ The predictability and the statistical complexities bound each other

That is, the predictable future information E is less than or equal to the information $Cg$ necessary to produce the future which, in turn, is less than or equal to the information $Cμ$ necessary to predict the future.^1,44–47 Such relationships have been explored even for quantum generators of (classical) stochastic processes [Ref. 51, and references therein].

Up to this point, the development focused on introducing and interpreting various information and complexity measures. It was not constructive in that there was no specification of how to calculate these quantities for a given process. To do so, requires models or, in the vernacular, a presentation of a process. Fortunately, a common mathematical representation describes a wide class of process generators: the edge-labeled hidden Markov models (HMMs), also known as Mealy HMMs.^48,52 Using these as our preferred presentations, we will first classify them and then describe how to calculate the information measures of the processes they generate.

Definition 1. A finite-state, edge-labeled hidden Markov model $M={S,A,{T(x)}x∈A,η0}$ consists of:

• A finite set of hidden states $S={s1,…,sM}$⁠. $St$ is the random variable for the hidden state at time t.

• A finite output alphabet $A$⁠.

• A set of M × M symbol-labeled transition matrices ${T(x)}x∈A$, where $Ti,j(x)=Pr(x,sj|si)$ is the probability of transitioning from state s_i to state s_j and emitting symbol x. The corresponding overall state-to-state transition matrix is the row-stochastic matrix $T=∑x∈AT(x)$⁠.

• An initial distribution over hidden states: $η0=(Pr(S0=s1),Pr(S0=s2),…,Pr(S0=sM))$⁠.

Contrast this with the class-equivalent state-labeled HMMs, also known as Moore HMMs.^11,45,52,75 In the automata theory, a finite-state HMM is called a probabilistic nondeterministic finite automaton.⁷⁶ The information theory¹³ refers to them as finite-state information sources and stochastic process theory defines them as functions of a Markov chain.^53,58,77,78

The dynamics of such finite-state models are governed by transition matrices amenable to the linear algebra of vector spaces. As a result, bra-ket notation is useful. Bras $〈·|$ are row vectors and kets $|·〉$ are column vectors. One benefit of the notation is immediately recognizing the mathematical object type. For example, on the one hand, any expression that forms a closed bra-ket pair—either $〈·|·〉$ or $〈·|·|·〉$—is a scalar quantity and commutes as a unit with anything. On the other hand, when useful, an expression of the ket-bra form $|·〉〈·|$ can be interpreted as a matrix.

T's row-stochasticity means that each of its rows sum to unity. Introducing $|1〉$ as the column vector of all 1s, this can be restated as

This is readily recognized as an eigenequation: $T|λ〉=λ|λ〉$⁠. That is, the all-ones vector $|1〉$ is always a right eigenvector of T associated with the eigenvalue λ of unity.

When the internal Markov transition matrix T is irreducible, the Perron-Frobenius theorem guarantees that there is a unique asymptotic state distribution π determined by

with the further condition that π is normalized in probability: $〈π|1〉=1$⁠. This again is recognized as an eigenequation: the stationary distribution π over the hidden states is T's left eigenvector associated with the eigenvalue of unity. When 1 is the only one of T's eigenvalues on the unit circle—i.e., when the process and presentation lack any deterministic periodicities—then the stationary distribution π is also the asymptotic state distribution for an ensemble of realizations, regardless of the initial distribution.

To describe a stationary process, as done often in the following, the initial hidden-state distribution η₀ is set to the stationary one: $η0=π$⁠. The resulting process generated is then stationary. Choosing an alternative η₀ is useful in many contexts. Note that, starting with η_t the expected state distribution at the next time is simply $〈ηt+1|=〈ηtT|$⁠. However, starting with such alternatives typically yields a nonstationary process.

An HMM $M$ describes a process' behaviors as a formal language $L⊆∪ℓ=1∞Aℓ$ of allowed realizations. Moreover, $M$ succinctly describes a process's word distribution $Pr(w)$ over all words $w∈L$⁠. (Appropriately, $M$ also assigns zero probability to words outside of the process' language: $Pr(w)=0$ for all $w∈Lc$⁠, $L$'s complement.) Specifically, the stationary probability of observing a particular length-L word $w=x0x1…xL−1$ is given by

where $T(w)≡T(x0)T(x1) ⋯T(xL−1)$⁠.

More generally, given a nonstationary state distribution η, the probability of seeing w is

where $St∼η$ means that the random variable $St$ is distributed as η.¹³ And, the state distribution having seen word w starting in η is

These conditional distributions are used often since, for example, most observations induce a nonstationary distribution over hidden states. Tracking such observation-induced distributions is the role of a related model class—the mixed-state presentation, introduced shortly. To get there, we must first introduce several, prerequisite HMM classes. See Fig. 1. A simple example of the general HMM just discussed is shown in Fig. 1(a).

An important class of HMMs consists of those that are unifilar. Unifilarity guarantees that, given a start state and a sequence of observations, there is a unique path through the internal states.⁵³ This, in turn, allows one to directly translate between properties of the internal Markov chain and properties of the observed behavior generated along the sequence of edges traversed. The states of unifilar HMMs are maximally predictive.⁵⁴ 

In contrast, general—that is, nonunifilar—HMMs have an exponentially growing number of possible state paths as a function of observed word length. Thus, nonunifilar process presentations break most all quantitative connections between internal dynamics and observations, rendering them markedly less useful process presentations. While they can be used to generate realizations of a given process, they cannot be used outright to predict a process. Unifilarity is required.

Definition 2. A finite-state, edge-labeled, unifilar HMM (uHMM)⁵⁵ is a finite-state, edge-labeled HMM with the following property:

• Unifilarity: For each state $s∈S$ and each symbol $x∈A$⁠, there is at most one outgoing edge from state s that emits symbol x.

An example is shown in Fig. 1(b).

Minimal models are not only convenient to use, but very often allow for determining essential informational properties, such as a process' memory $Cμ$⁠. A process' minimal state-entropy uHMM is the same as its minimal-state uHMM. And, the latter turns out to be the process' ϵ-machine in computational mechanics.²⁰ Computational mechanics shows how to calculate a process' ϵ-machine from the process' conditional word distributions. Specifically, ϵ-machine states, the process' causal states $σ∈S+$⁠, are equivalence classes of histories that yield the same predictions for the future. Explicitly, two histories $x←$ and $x←$′ map to the same causal state $ϵ(x←)=ϵ(x←′)=σ$ if and only if $Pr(X→|x←)=Pr(X→|x←′)$⁠. Thus, each causal state comes with a prediction of the future $Pr(X→|σ)$—its future morph. In short, a process' ϵ-machine is its minimal size, maximally predictive predictor.

Converting a given uHMM to its corresponding ϵ-machine employs probabilistic variants of well-known state-minimization algorithms in the automata theory.⁵⁶ One can also verify that a given uHMM is minimal by checking that all its states are probabilistically distinct.^49,50

Definition 3. A uHMM's states are probabilistically distinct if for each pair of distinct states $sk,sj∈S$ there exists some finite word $w=x0x1…xL−1$ such that

If this is the case, then the process' uHMM is its ϵ-machine.

An example is shown in Fig. 1(c).

The finite-state presentations in these classes form a hierarchy in terms of the processes they can finitely generate:⁴⁴ Processes(ϵ-machines) = Processes(uHMMs) $⊂$ Processes(HMMs). That is, finite HMMs generate a strictly larger class of stochastic processes than finite uHMMs. The class of processes generated by finite uHMMs, though, is the same as generated by finite ϵ-machines.

Though we concentrate on discrete-time processes, many of the process classifications, properties, and calculational methods carry over easily to continuous time. In this setting transition rates are more appropriate than transition probabilities. Continuous-time HMMs can often be obtained as a discrete-time limit $Δt→0$ of an edge-labeled HMM whose edges operate for a time $Δt$⁠. The most natural continuous-time HMM presentation, though, has a continuous-time generator G of time evolution over hidden states, with observables emitted as deterministic functions of an internal Markov chain: $f:S→A$⁠.

Respecting the continuous-time analog of probability conservation, each row of G sums to zero. Over a finite time interval t, marginalizing over all possible observations, the row-stochastic state-to-state transition dynamic is

Any nontrivial continuous-time process generated by such a continuous-time HMM has an uncountably infinite number of possible realizations within a finite time interval, and most of these have vanishing probability. However, probabilities regarding what state the system is in at any finite set of times can be easily calculated, essentially by bundling measurable sets of trajectories that satisfy certain constraints. For this purpose, we introduce the continuous-time observation matrices

where $δx,f(s)$ is a Kronecker delta, $|δs〉$ the column vector of all 0s except for a 1 at the position for state s, and $〈δs|$ its transpose $(|δs〉)⊤$⁠. These “projectors” sum to the identity: $∑x∈AΓx=I$⁠.

An example is shown in Fig. 1(d).

A given process can be generated by nonunifilar, unifilar, and ϵ-machine HMM presentations. A process' ϵ-machine is unique. However, within either the unifilar or nonunifilar HMM classes, there are an infinite number of presentations that generate the process.

This flexibility suggests that we can create a HMM process generator that, through its embellished structure, answers more refined questions than the information generation (⁠$hμ$⁠) and memory (⁠$Cμ$⁠) calculated from the ϵ-machine. To this end, we introduce the mixed-state presentation (MSP). An MSP tracks important supplementary information in the hidden states and, through well-crafted dynamics, over the hidden states. In particular, an MSP generates a process while tracking the observation-induced distribution over the states of an alternative process generator. Here, we review only that subset of mixed-state theory required by the following.

Consider a HMM presentation $M=(S,A,{T(x)}x∈A,π)$ of some process in statistical equilibrium. A mixed state η can be any state distribution over $S$ and so we work with the simplex $ΔS$ of state distributions. However, $ΔS$ is uncountable and so contains far more mixed states than needed to calculate many complexity measures.

How to efficiently monitor the way in which an observer comes to know the HMM state as it sees successive symbols from the process? This is the problem of observer–state synchronization. To analyze the evolution of the observer's knowledge through a sequence of observation-induced mixed states, we use the set $Rπ$ of mixed states that are induced by all allowed words $w∈L$ from the initial mixed state $η0=π$

The cardinality of $Rπ$ is finite when there are only a finite number of distinct probability distributions over $M$'s states that can be induced by observed sequences, if starting from the stationary distribution π.

If w is the first (in lexicographic order) word that induces a particular distribution over $S$⁠, then we denote this distribution as η^w; a shorthand for Eq. (3). For example, if the two words 010 and 110110 both induce the same distribution η over $S$ and no word shorter than 010 induces that distribution, then the mixed state is denoted $η010$⁠. It corresponds to the distribution

Since a given observed symbol induces a unique updated distribution from a previous distribution, the dynamic over mixed states is unifilar. Transition probabilities among mixed states can be obtained via Eqs. (2) and (3). So, if

and

then

These transition probabilities over the mixed states in $Rπ$ are the matrix elements for the observation-labeled transition matrices ${W(x)}x∈A$ of $M$'s synchronizing MSP (⁠$S$-MSP)

where $δπ$ is the distribution over $Rπ$ peaked at the unique start-(mixed)-state π. The row-stochastic net mixed-state-to-state transition matrix of $S$-MSP(⁠$M$⁠) is $W=∑x∈AW(x)$⁠. If irreducible, then there is a unique stationary probability distribution $〈πW|$ over $S$-MSP(⁠$M$⁠)'s states obtained by solving $〈πW|=〈πW|W$⁠. We use $Rt$ to denote the random variable for the MSP's state at time t.

Figure 2 illustrates how a $S$-MSP relates to the HMM whose state distributions it tracks. Both the original HMM and its MSP generate the same process. However, the MSP's dynamic over mixed-states also tracks how, through the mixed states induced by successive observations, an observer comes to know the original HMM's current state. Importantly, this MSP's dynamic is nondiagonalizable, while the original HMM's dynamic is diagonalizable. The appearance of nondiagonalizability when analyzing particular properties, in this case, observer synchronization, is generic. Constructively working with this unavoidable fact motivates much of the following. Crucially, we find that the burden of predicting a stochastic process is fundamentally dependent on the nondiagonalizable characteristics of its MSP.

That feedforward network structures lead to nondiagonalizability and that it is important to network functionality was also observed in a quite different setting—oscillator synchronization on complex networks with directional coupling dynamics.⁵⁷ Briefly comparing settings reveals the origins of the commonality. In each, nondiagonalizability arises from an irreducible interdependence among elements—an interdependence that can be harnessed for hierarchical control. In Ref. 57, nondiagonalizable network structures allow upstream oscillators to influence downstream oscillators and this enables optimal synchronization among all oscillators. In contrast, while our setting does not concern synchronizing oscillator nodes to each other, it does analyze how an observer's belief state synchronizes to the true state of the system under study. During this kind of synchronization, past states of knowledge feed into future states of knowledge. In short, nondiagonalizability corresponds to intrinsically interdependent updates in the evolution of knowledge.

More generally, we work with distributions over $ΔS$⁠. We must consider a mixed-state dynamic that starts from a general (nonpeaked) distribution over $ΔS$⁠. This may be counterintuitive, since a distribution over distributions should correspond to a single distribution. However, the general MSP theory with a general starting distribution over $ΔS$ allows us to consider a weighted average of behaviors originating from different histories. And, this is distinct from considering the behavior originating from a weighted average of histories. This more general MSP formalism arises in the closed-form solutions for more sophisticated complexity measures, such as the bound information. This anticipates tools needed in a sequel.

With this brief overview of mixed states, we can now turn to use them. Section V shows that tracking distributions over the states of another generator makes the MSP an ideal algebraic object for closed-form complexity expressions involving conditional entropies—measures that require conditional probabilities. Sections II B and II C showed that many of the complexity measures for predictability and predictive burden are indeed framed as conditional entropies. And so, MSPs are central to their closed-form expressions.

Historically, mixed states were already implicit in Ref. 58, introduced in their modern form by Refs. 44 and 45, and have been used recently; e.g., in Refs. 59 and 60. Most of these efforts, however, used mixed-states in the specific context of the synchronizing MSP (⁠$S$-MSP). A greatly extended development of mixed-state dynamics appears in Ref. 41.

The overall strategy, though, is easy to explain. Different information-theoretic questions require different mixed-state dynamics, each of which is a unifilar presentation. Employing the mathematical methods developed here, we find that the desired closed-form solutions are often simple functions of the transition dynamic of an appropriate MSP. Specifically, the spectral properties of the relevant MSP control the form of information-theoretic quantities.

Finally, we note that similar linear-algebraic constructions—whose hidden states track relevant information—that are nevertheless not MSPs are just as important for answering different sets of questions about a process. Since these constructions are not directly about predictability and prediction, we report on them elsewhere.

We are now in a position to identify the hidden linear dynamic appropriate to many of the questions that arise in complex systems—their observation, predictability, prediction, and generation, as outlined in Table II. In part, this section addresses a very practical need for specific calculations. In part, it also lays the foundations for further generalizations, to be discussed at the end. Identifying the linear dynamic means identifying the linear operator A such that a question of interest can be reformulated as either being of the cascading form $〈·|An|·〉$ or as an accumulation of such cascading events via $〈·|(∑nAn)|·〉$⁠; recall Table I. Helpfully, many well-known questions of complexity can be mapped to these archetypal forms. And so, we now proceed to uncover the hidden linear dynamics of the cascading questions approximately in the order they were introduced in Sec. II.

For observable correlation, any HMM transition operator will do as the linear dynamic. We simply observe, let time (or space) evolve forward, and observe again. Let us be concrete.

Recall the familiar autocorrelation function. For a discrete-domain process, it is⁶¹ 

where $L∈ℤ$ and the bar denotes the complex conjugate. The autocorrelation function is symmetric about L = 0, so we can focus on $L≥0$⁠. For L = 0, we simply have

For L > 0, we have

Each “$*$” above is a wildcard symbol denoting indifference to the particular symbol observed in its place. That is, the $*$s denotes marginalizing over the intervening random variables. We develop the consequence of this, explicitly calculating⁶² and finding

The result is the autocorrelation in the cascading form $〈·|Tt|·〉$⁠, which can be made particularly transparent by subsuming time-independent factors on the left and right into the bras and kets. Let us introduce the new row vector

and the column vector

Then, the autocorrelation function for nonzero integer L is simply

Clearly, the autocorrelation function is a direct, albeit filtered, signature of iterates of the transition dynamic of any process presentation.

This result can easily be translated to the continuous-time setting. If the process is represented as a function of a Markov chain and we make the translation that

then the autocorrelation function for any $τ∈ℝ$ is simply

where G is determined from T following Sec. III D. Again, the autocorrelation function is a direct fingerprint of the transition dynamic over the hidden states.

The power spectrum is a modulated accumulation of the autocorrelation function. With some algebra, one can show that it is

Reference 61 shows that for discrete-domain processes, the continuous part of the power spectrum is simply

where Re $(·)$ denotes the real part of its argument and I is the identity matrix. Similarly, for continuous-domain processes, one has

Although useful, these signatures of pairwise correlation are only first-order complexity measures. Common measures of complexity that include higher orders of correlation can also be written in the simple cascading and accumulating forms, but require a more careful choice of representation.

For example, any HMM presentation allows us to calculate using Eq. (1) a process's block entropy

but at a computational cost $O(|S|3L|A|L)$ exponential in L, due to the exponentially growing number of words in $L∩AL$⁠. Consequently, using a general HMM, one can neither directly nor efficiently calculate many key complexity measures, including a process's entropy rate and excess entropy.

These limitations motivate using more specialized HMM classes. To take one example, it has been known for some time that a process' entropy rate $hμ$ can be calculated directly from any of its unifilar presentations.⁵³ Another is that we can calculate the excess entropy directly from a process's uHMM forward $S+$ and reverse $S−$ states:^59,60 $E=I[X←;X→]=I[S+;S−]$⁠.

However, efficient computation of myopic entropy rates $hμ(L)$ remained elusive for some time, and we only recently found their closed-form expression.³ The myopic entropy rates are important because they represent the apparent entropy rate of a process if it is modeled as a finite Markov order-(L – 1) process—a very common approximation. Crucially, the difference $hμ(L)−hμ$ from the process' true entropy rate is the surplus entropy rate incurred by using an order-(L – 1) Markov approximation. Similarly, these surplus entropy rates lead directly to not only an apparent loss of predictability, but errors in the inferred physical properties. These include overestimates of dissipation associated with the surplus entropy rate assigned to a physical thermodynamic system.³⁷ 

Unifilarity, it turns out, is not enough to calculate a process' $hμ(L)$ directly. Rather, the $S$-MSP of any process presentation is what is required. Let us now develop the closed-form expression for the myopic entropy rates, following Ref. 41.

The length-L myopic entropy rate is the expected uncertainty in the Lth random variable $XL−1$⁠, given the preceding L – 1 random variables $X0:L−1$

where, in the second line, we explicitly give the condition $η0=π$ specifying our ignorance of the initial state. That is, without making any observations, we can only assume that the initial distribution η₀ over $M$'s states is the expected asymptotic distribution π. For a mixing ergodic process, for example, even if another distribution $η−N=α$ was known in the distant past, we still have $〈η0|=〈η−N|TN→〈π|$⁠, as $N→∞$⁠.

Assuming an initial probability distribution over $M$'s states, a given observation sequence induces a particular sequence of updated state distributions. That is, the $S$-MSP(⁠$M$⁠) is unifilar regardless of whether $M$ is unifilar or not. Or, in other words, given the $S$-MSP's unique start state—$R0=π$—and a particular realization $X0:L−1=wL−1$ of the last L – 1 random variables, we end up at the particular mixed state $RL−1=ηwL−1∈Rπ$⁠. Moreover, the entropy of the next observation is uniquely determined by $M$'s state distribution, suggesting that Eq. (8) becomes

as proven elsewhere.⁴¹ Intuitively, conditioning on all of the past observation random variables is equivalent to conditioning on the random variable for the state distribution induced by particular observation sequences.

We can now recast Eq. (8) in terms of the $(S$-MSP, finding

Here

is simply the column vector whose ith entry is the entropy of transitioning from the ith state of $S$-MSP. Critically, $|H(WA)〉$ is independent of L.

Notice that taking the logarithm of the sum of the entries of the row vector $〈δη|W(x)$ via $〈δη|W(x)|1〉$ is only permissible since $S$-MSP's unifilarity guarantees that $W(x)$ has at most one nonzero entry per row. (We also use the familiar convention that $0 log20=0$⁠.¹³)

The result is a particularly compact and efficient expression for the length-L myopic entropy rates

Thus, all that is required is computing powers $WL−1$ of the MSP transition dynamic. The computational cost $O(L|Rπ|3)$ is now only linear in L. Moreover, W is very sparse, especially so with a small alphabet $A$⁠. And, this means that the computational cost can be reduced even further via numerical optimization.

With $hμ(L)$ in hand, the hierarchy of complexity measures that derive from it immediately follow, including the entropy rate $hμ$⁠, the excess entropy E, and the transient information T.¹ Specifically, we have

The sequel, Part II,⁷⁹ discusses these in more detail, introducing their closed-form expressions. To prepare for this, we must first review the meromorphic functional calculus, which is needed for working with the above operators.

We saw that correlation measures are easily extended to the continuous-time domain via continuous-time HMMs. Entropy rates (since they are rates) and state entropy (since it depends only on the instantaneous distribution) also carry over rather easily to continuous time. Indeed, the former is well studied for chaotic systems⁶³ and the latter is exemplified by the thermodynamic entropy. Yet, other information-theoretic measures of information transduction are awkward when directly translated to continuous time. At least one approach has been taken recently towards understanding their structure,^64–66 but more work is necessary.

If a process' state-space is known, then the $S$-MSP of the generating model allows one to track the observation-induced distributions over its states. This naturally leads to closed-form solutions to informational questions about how an observer comes to know, or how it synchronizes to, the system's states.

To monitor how an observer's knowledge of a process' internal state changes with increasing measurements, we use the myopic state uncertainty $H(L)=H[S0|X−L:0]$⁠.¹ Expressing it in terms of the $S$-MSP, one finds⁴¹ 

Here, $H[η]$ is the presentation-state uncertainty specified by the mixed state η

where $|δs〉$ is the length-$|S|$ column vector of all zeros except for a 1 at the appropriate index of the presentation-state s.

Continuing, we re-express $H(L)$ in terms of powers of the $S$-MSP transition dynamic

Here, we defined

which is the L-independent length-$|Rπ|$ column vector whose entries are the appropriately indexed entropies of each mixed state.

The forms of Eqs. (9) and (11) demonstrate that $hμ(L+1)$ and $H(L)$ differ only in the type of information being extracted after being evolved by the operator: observable entropy $H(WA)$ or state entropy $H[η]$⁠, as implicated by their respective kets. Each of these entropies decreases as the distributions induced by longer observation sequences converge to the synchronized distribution. If synchronization is achieved, the distributions become δ-functions on a single state and the associated state-entropy vanishes.

Paralleling $hμ(L)$⁠, there is a complementary hierarchy of complexity measures that are built up from functions of $H(L)$⁠. These include the asymptotic state uncertainty $H$ and excess synchronization information $S′$⁠, to mention only two

Compared to the $hμ(L)$ family of measures, $H$ and $S′$ mirror the roles of $hμ$ and E, respectively.

The model state-complexity

also has an analog in the $hμ(L)$ hierarchy—the process' alphabet complexity

We just reviewed the linear underpinnings of synchronizing to any model of a process. However, the myopic state uncertainty of the $ϵ-machine$ has a distinguished role in determining the synchronization cost for maximally predicting a process, regardless of the presentation that generated it. Using the ϵ-machine's $S$-MSP, the ϵ-machine myopic state uncertainty can be written in direct parallel to the myopic state uncertainty of any model

The script $W$ emphasizes that we are now specifically working with the state-to-state transition dynamic of the ϵ-machine's MSP.

Paralleling $H(L)$⁠, an obvious hierarchy of complexity measures is built from functions of $H+(L)$⁠. For example, the ϵ-machine's state-complexity is the statistical complexity $Cμ=H+(0)$⁠. The information that must be obtained to synchronize to the causal state in order to maximally predict—the causal synchronization information—is given in terms of the ϵ-machine's $S$-MSP by $S=∑L=0∞H+(L)$⁠.

An important difference when using ϵ-machine presentations is that they have zero asymptotic state uncertainty

Therefore, $S=S′(ϵ-machine)$⁠. Moreover, we conjecture that $S=minM∑L=0∞H(L)$ for any presentation $M$ that generates the process, even if $Cμ≥Cg$⁠.

Many of the complexity measures use a mixed-state presentation as the appropriate linear dynamic, with particular focus on the $S$-MSP. However, we want to emphasize that this is more a reflection of questions that have become common. It does not indicate the general answer that one expects in the broader approach to finding the hidden linear dynamic. Here, we give a brief overview for how other linear dynamics can appear for different types of complexity questions. These have been uncovered recently and will be reported in more detail in sequels.

First, we found the reverse-time mixed-functional presentation (MFP) of any forward-time generator. The MFP tracks the reverse-time dynamic over linear functionals $|η〉$ of state distributions induced by reverse-time observations

The MFP allows direct calculation of the convergence of the preparation uncertainty$(L)≡H(S0|X0:L)$ via powers of the linear MFP transition dynamic. The preparation uncertainty in turn gives a new perspective on the transient information since

can be interpreted as the predictive advantage of hindsight. Related, the myopic process crypticity χ(L) = ⁺$(L)−H+(L)$ had been previously introduced.⁴³ Since $limL→∞H+(L)=H+=0$⁠, the asymptotic crypticity is χ = ⁺$+H+$ = ⁺. And, this reveals a refined partitioning underlying the sum

Crypticity $χ=H(S0+|X0:∞)$ itself is positive only if the process' cryptic order

is positive. The cryptic order is always less than or equal to its better-known cousin, the Markov order R

since conditioning can never increase entropy. In the case of the cryptic order, we condition on future observations $X0:∞$⁠.

The forward-time cryptic operator presentation gives the forward-time observation-induced dynamic over the operators

Since the reverse causal state $S0−$ at time 0 is a linear combination of forward causal states,^67,68 this presentation allows new calculations of the convergence to crypticity that implicate $Pr(S0+|X−L:∞)$⁠.

In fact, the cryptic operator presentation is a special case of the more general myopic bidirectional dynamic over operators

induced by new observations of either the future or the past. This is key to understanding the interplay between forgetfulness and shortsightedness: $Pr(S0|X−M:0,X0:N)$⁠.

The list of these extensions continues. Detailed bounds on entropy-rate convergence are obtained from the transition dynamic of the so-called possibility machine, beyond the asymptotic result obtained in Ref. 50. And, the importance of post-synchronized monitoring, as quantified by the information lost due to negligence over a duration $ℓ$

can be determined using yet another type of modified MSP.

These examples all find an exact solution via a theory parallel to that outlined in the following, but applied to the linear dynamic appropriate for the corresponding complexity question. Furthermore, they highlight the opportunity, enabled by the full meromorphic functional calculus,⁴ to ask and answer more nuanced and, thus, more probing questions about the structure, predictability, and prediction.

It would seem that we achieved our goal. We identified the appropriate transition dynamic for common complexity questions and, by some standard, gave formulae for their exact solution. In point of fact, the effort so far has all been in preparation. Although we set the framework up appropriately for linear analysis, closed-form expressions for the complexity measures still await the mathematical developments of the following Secs. VI. At the same time, at the level of qualitative understanding and scientific interpretation, so far we failed to answer the simple question:

• What range of possible behaviors do these complexity measures exhibit?

  and the natural follow-up question:

• What mechanisms produce qualitatively different informational signatures?

The following Sec. VI reviews the recently developed functional calculus that allows us to actually decompose arbitrary functions of the nondiagonalizable hidden dynamic to give conclusive answers to these fundamental questions.⁴ We then analyze the range of possible behaviors and identify the internal mechanisms that give rise to qualitatively different contributions to complexity.

The investment in this and the succeeding Secs. VI–VIII allow Part II to express new closed-form solutions for many complexity measures beyond those achieved to date. In addition to obvious calculational advantages, this also gives new insights into possible behaviors of the complexity measures and, moreover, their unexpected similarities with each other. In many ways, the results shed new light on what we were (implicitly) probing with already-familiar complexity measures. Constructively, this suggests extending complexity magnitudes to complexity functions that succinctly capture the organization to all orders of correlation. Just as our intuition for pairwise correlation grows out of power spectra, so too these extensions unveil the workings of both a process' predictability and the burden of prediction for an observer.

Here, we briefly review the spectral decomposition theory from Ref. 4 needed for working with nondiagonalizable linear operators. As will become clear, it goes significantly beyond the spectral theorem for normal operators. Although the linear operator theory (especially as developed in the mathematical literature of functional analysis) already addresses nonnormal operators, it had not delivered the comparable machinery for a tractable spectral decomposition of nonnormal and nondiagonalizable operators. Reference 4 explored this topic and derived new relations that enable the practical analysis of nonnormal and nondiagonalizable systems—in which independent subprocesses (irreducible subspaces) can be directly manipulated.

We restrict our attention to operators that have at most a countably infinite spectrum. Such operators share many features with finite-dimensional square matrices. And so, we review several elementary but essential facts that are used extensively in the following.

Recall that if A is a finite-dimensional square matrix, then A's spectrum is simply its set of eigenvalues

where det$(·)$ is the determinant of its argument.

For reference later, recall that the algebraic multiplicity $aλ$ of eigenvalue λ is the power of the term $(z−λ)$ in the characteristic polynomial det$(zI−A)$⁠. In contrast, the geometric multiplicity $gλ$ is the dimension of the kernel of the transformation $A−λI$ or the number of linearly independent eigenvectors for the eigenvalue. Moreover, $gλ$ is the number of Jordan blocks associated with λ. The algebraic and geometric multiplicities are all equal when the matrix is diagonalizable.

Since there can be multiple subspaces associated with a single eigenvalue, corresponding to different Jordan blocks in the Jordan canonical form, it is structurally important to introduce the index of the eigenvalue to describe the size of its largest-dimension associated subspace.

Definition 4. The index$νλ$ of eigenvalue λ is the size of the largest Jordan block associated with λ.

The index gives information beyond what the algebraic and geometric multiplicities themselves reveal. Nevertheless, for $λ∈ΛA$⁠, it is always true that $νλ−1≤aλ−gλ≤aλ−1$⁠. In the diagonalizable case, $aλ=gλ$ and $νλ=1$ for all $λ∈ΛA$⁠.

The resolvent

defined with the help of the continuous complex variable $z∈ℂ$⁠, captures all of the spectral information about A through the poles of the resolvent's matrix elements. In fact, the resolvent contains more than just the spectrum: the order of each pole gives the index of the corresponding eigenvalue.

Each eigenvalue λ of A has an associated spectral projection operator $Aλ$⁠, which is the residue of the resolvent as $z→λ$

where $Cλ$ is a counterclockwise contour in the complex plane around eigenvalue λ. The residue of the matrix can be calculated elementwise.

The projection operators are orthonormal

and sum to the identity