Appealing to several multivariate information measures—some familiar, some new here—we analyze the information embedded in discrete-valued stochastic time series. We dissect the uncertainty of a single observation to demonstrate how the measures’ asymptotic behavior sheds structural and semantic light on the generating process’s internal information dynamics. The measures scale with the length of time window, which captures both intensive (rates of growth) and subextensive components. We provide interpretations for the components, developing explicit relationships between them. We also identify the informational component shared between the past and the future that is not contained in a single observation. The existence of this component directly motivates the notion of a process’s effective (internal) states and indicates why one must build models.

1.
T. M.
Cover
and
J. A.
Thomas
,
Elements of Information Theory
, 2nd ed. (
Wiley-Interscience
,
New York
,
2006
).
2.
J. P.
Crutchfield
,
C. J.
Ellison
,
R. G.
James
, and
J. R.
Mahoney
,
Chaos
20
(
3
),
037105
(
2010
).
3.
A. J.
Bell
, “
The co-Information lattice
,” in
Proceedings of the Fifth International Workshop on Independent Component Analysis and Blind Signal Separation
, ICA 2003 ed., edited by
N.
Murata
,
S.
Amari
,
A.
Cichocki
, and
S.
Makino
(
Springer
,
New York
,
2003
).
4.
P.
Chanda
,
A.
Zhang
,
D.
Brazeau
,
L.
Sucheston
,
J. L.
Freudenheim
,
C.
Ambrosone
, and
M.
Ramanathan
,
Am. J. Hum. Genet.
81
(
5
),
939
(
2007
).
5.
S.
Watanabe
,
IBM J. Res. Dev.
4
(
1
),
66
(
1960
).
6.
7.
S. A.
Abdallah
and
M. D.
Plumbley
, “
A measure of statistical complexity based on predictive information
,” e-print arXiv:1012.1890v1 (
2010
).
8.
S. A.
Abdallah
and
M. D.
Plumbley
, “
Predictive information, multi-information, and binding information
,” Technical Report No. C4DM-TR10-10 (Centre for Digital Music, Queen Mary University of London,
2010
).
9.
L. M. A.
Bettencourt
,
V.
Gintautas
, and
M. I.
Ham
.
Phys. Rev. Lett.
100
(
23
),
238701
(
2008
).
10.
J. P.
Crutchfield
and
D. P.
Feldman
,
Chaos
13
(
1
),
25
(
2003
).
11.
J. P.
Crutchfield
and
N. H.
Packard
,
Physica D
7
,
201
(
1983
).
12.
K.-
E.
Eriksson
and
K.
Lindgren
,
Phys. Scr.
35
(
3
),
388
(
1987
).
13.
For example, see the conflation of I for both the total correlation (here, T) and its rate (here, ρμ) in Ref. 14.
14.
I.
Erb
and
N.
Ay
,
J. Stat. Phys.
115
(
3/4
),
949
(
2004
).
15.
R. W.
Yeung
,
IEEE Trans. Inf. Theory
37
(
3
),
466
(
1991
).
16.
J. P.
Crutchfield
,
C. J.
Ellison
, and
J. R.
Mahoney
,
Phys. Rev. Lett.
103
(
9
),
094101
(
2009
).
17.
P. L.
Williams
and
R. D.
Beer
, “
Nonnegative decomposition of multivariate information
,” e-print arXiv:1004.2515 (
2010
).
18.
C. J.
Ellison
,
J. R.
Mahoney
, and
J. P.
Crutchfield
,
J. Stat. Phys.
136
(
6
),
1005
(
2009
).
19.
R. C.
Ball
,
M.
Diakonova
, and
R. S.
MacKay
,
Adv. Complex Syst.
13
(
3
),
327
(
2010
).
20.
C. R.
Shalizi
and
J. P.
Crutchfield
,
J. Stat. Phys.
104
,
817
(
2001
).
You do not currently have access to this content.