We analyze a cooperative decision-making model that is based on individual aspiration levels using the framework of a public goods game in static and dynamic networks. Sensitivity to differences in payoff and dynamic aspiration levels modulates individual satisfaction and affects subsequent behavior. The collective outcome of such strategy changes depends on the efficiency with which aspiration levels are updated. Below a threshold learning efficiency, cooperators dominate despite short-term fluctuations in strategy fractions. Categorizing players based on their satisfaction level and the resulting strategy reveal periodic cycling between the different categories. We explain the distinct dynamics in the two phases in terms of differences in the dominant cyclic transitions between different categories of cooperators and defectors. Allowing even a small fraction of nodes to restructure their connections can promote cooperation across almost the entire range of values of learning efficiency. Our work reinforces the usefulness of an internal criterion for strategy updates, together with network restructuring, in ensuring the dominance of altruistic strategies over long time-scales.

1.
G.
Hardin
, “
The tragedy of the commons
,”
Science
162
,
1243
1248
(
1968
).
2.
M. A.
Nowak
,
Evolutionary Dynamics: Exploring the Equations of Life
(
Harvard University Press
,
2006
), pp.
71
92
.
3.
D. G.
Rand
and
M. A.
Nowak
, “
Human cooperation
,”
Trends Cognit. Sci.
17
,
413
425
(
2013
).
4.
S.
Pathak
,
P.
Verma
,
S. K.
Ram
, and
S.
Sengupta
, “
How strategy environment and wealth shape altruistic behaviour: Cooperation rules affecting wealth distribution in dynamic networks
,”
Proc. R. Soc. B
287
,
20202250
(
2020
).
5.
S.
Suri
and
D. J.
Watts
, “
Cooperation and contagion in web-based, networked public goods experiments
,”
PLoS One
6
,
e16836
(
2011
).
6.
F. P.
Santos
,
F. C.
Santos
, and
J. M.
Pacheco
, “
Social norms of cooperation in small-scale societies
,”
PLoS Comput. Biol.
12
,
e1004709
(
2016
).
7.
A.
Nishi
,
H.
Shirado
,
D.
Rand
, and
N. A.
Christakis
, “
Inequality and visibility of wealth in experimental social networks
,”
Nature
526
,
426
429
(
2015
).
8.
P.
Richerson
and
R.
Boyd
,
The Origin and Evolution of Cultures
(
OUP
,
2005
).
9.
M. A.
Nowak
and
K.
Sigmund
, “
Evolution of indirect reciprocity by image scoring
,”
Lett. Nat.
393
,
573
577
(
1998
).
10.
O.
Leimar
and
P.
Hammerstein
, “
Evolution of cooperation through indirect reciprocity
,”
Proc. R. Soc. B
268
,
745
753
(
2001
).
11.
H.
Brandt
and
K.
Sigmund
, “
Indirect reciprocity, image scoring, and moral hazard
,”
Proc. Natl. Acad. Sci.
102
,
2666
2670
(
2005
).
12.
H.
Brandt
and
K.
Sigmund
, “
The good, the bad and the discriminator—Errors in direct and indirect reciprocity
,”
J. Theor. Biol.
239
,
183
194
(
2006
).
13.
H.
Ohtsuki
and
Y.
Iwasa
, “
How should we define goodness? Reputation dynamics in indirect reciprocity
,”
J. Theor. Biol.
231
,
107
120
(
2004
).
14.
H.
Ohtsuki
and
Y.
Iwasa
, “
Global analyses of evolutionary dynamics and exhaustive search for social norms that maintain cooperation by reputation
,”
J. Theor. Biol.
244
,
518
531
(
2007
).
15.
N.
Masuda
and
H.
Ohtsuki
, “
Tag-based indirect reciprocity by incomplete social information
,”
Proc. R. Soc. B
274
,
689
695
(
2007
).
16.
G.
Szabó
and
C.
Töke
, “
Evolutionary prisoner’s dilemma game on a square lattice
,”
Phys. Rev. E
58
,
69
73
(
1998
).
17.
A.
Traulsen
,
J. M.
Pacheco
, and
M. A.
Nowak
, “
Pairwise comparison and selection temperature in evolutionary game dynamics
,”
J. Theor. Biol.
246
,
522
529
(
2007
).
18.
X.
Chen
and
L.
Wang
, “
Promotion of cooperation induced by appropriate payoff aspirations in a small-world networked game
,”
Phys. Rev. E
77
,
017103
(
2008
).
19.
Y.
Liu
,
X.
Chen
,
L.
Wang
,
B.
Li
,
W.
Zhang
, and
H.
Wang
, “
Aspiration-based learning promotes cooperation in spatial prisoner’s dilemma games
,”
Europhys. Lett.
94
,
60002
(
2011
).
20.
J.
Du
,
B.
Wu
,
P.
Altrock
, and
L.
Wang
, “
Aspiration dynamics of multi-player games in finite populations
,”
J. R. Soc. Interf.
11
,
20140077
(
2014
).
21.
L.
Zhou
,
B.
Wu
,
V.
Vasconcelos
, and
L.
Wang
, “
Simple property of heterogeneous aspiration dynamics: Beyond weak selection
,”
Phys. Rev. E
98
,
062124
(
2018
).
22.
J. M.
Pacheco
,
A.
Traulsen
, and
M. A.
Nowak
, “
Coevolution of strategy and structure in complex networks with dynamical linking
,”
Phys. Rev. Lett.
97
,
258103
(
2006
).
23.
J. M.
Pacheco
,
A.
Traulsen
, and
M. A.
Nowak
, “
Active linking in evolutionary games
,”
J. Theor. Biol.
243
,
437
443
(
2006
).
24.
N.
Hanaki
,
A.
Peterhansl
,
P.
Dodds
, and
D.
Watts
, “
Cooperation in evolving social networks
,”
Manage. Sci.
53
,
1036
1050
(
2007
).
25.
F.
Fu
,
C.
Hauert
,
M. A.
Nowak
, and
L.
Wang
, “
Reputation-based partner choice promotes cooperation in social networks
,”
Phys. Rev. E
78
,
026117
(
2008
).
26.
D. G.
Rand
,
S.
Arbesman
, and
N. A.
Christakis
, “
Dynamic social networks promote cooperation in experiments with humans
,”
Proc. Natl. Acad. Sci. U.S.A.
108
,
19193
19198
(
2011
).
27.
J.
Li
,
C.
Zhang
,
Q.
Sun
, and
Z.
Chen
, “
Coevolution between strategy and social networks structure promotes cooperation
,”
Chaos Soliton. Fract.
77
,
253
263
(
2015
).
28.
F.
Fu
,
X.
Chen
,
L.
Liu
, and
L.
Wang
, “
Social dilemmas in an online social network: The structure and evolution of cooperation
,”
Phys. Lett. A
371
,
58
64
(
2007
).
29.
K.
Fehl
,
D. J.
van der Post
, and
D.
Semmann
, “
Co-evolution of behaviour and social network structure promotes human cooperation
,”
Ecol. Lett.
14
,
546
551
(
2011
).
30.
J.
Wang
,
S.
Suri
, and
D. J.
Watts
, “
Cooperation and assortativity with dynamic partner updating
,”
Proc. Natl. Acad. Sci. U.S.A.
109
,
14363
14368
(
2012
).
31.
F. C.
Santos
and
J. M.
Pacheco
, “
A new route to the evolution of cooperation
,”
J. Evol. Biol.
19
,
726
733
(
2006
).
32.
A.
Rapoport
,
A. M.
Chammah
, and
C. J.
Orwant
,
Prisoner’s Dilemma: A Study in Conflict and Cooperation
(
University of Michigan Press
,
1965
).
33.
M. W.
Macy
, “
Chains of cooperation: Threshold effects in collective action
,”
Am. Sociol. Rev.
56
,
730
747
(
1991
).
34.
M. W.
Macy
, “
Natural selection and social learning in prisoner’s dilemma: Coadaptation with genetic algorithms and artificial neural networks
,”
Sociol. Meth. Res.
25
,
103
137
(
1996
).
35.
M.
Posch
,
A.
Pichler
, and
K.
Sigmund
, “
The efficiency of adapting aspiration levels
,”
Proc. R. Soc. B
266
,
1427
(
1999
).
36.
M. W.
Macy
and
A.
Flache
, “
Learning dynamics in social dilemmas
,”
Proc. Natl. Acad. Sci. U.S.A.
99
,
7229
7236
(
2002
).
37.
L.
Izquierdo
,
S.
Izquierdo
,
N.
Gotts
, and
J.
Polhill
, “
Transient and asymptotic dynamics of reinforcement learning in games
,”
Games Econ. Behav.
61
,
259
276
(
2007
).
38.
S. S.
Izquierdo
,
L. R.
Izquierdo
, and
N. M.
Gotts
, “
Reinforcement learning dynamics in social dilemmas
,”
J. Artif. Soc. Social Simul.
11
(2),
1
(
2008
).
39.
R.
Karandikar
,
D.
Ray
,
D.
Mookherjee
, and
F.
Vega-Redondo
, “
Evolving aspirations and cooperation
,”
J. Econ. Theory
80
,
292
331
(
1998
).
40.
R. R.
Bush
and
F.
Mosteller
, “
A mathematical model for simple learning
,”
Psychol. Rev.
58
,
313
323
(
1951
).
41.
N.
Masuda
and
M.
Nakamura
, “
Numerical analysis of a reinforcement learning model with the dynamic aspiration level in the iterated prisoner’s dilemma
,”
J. Theor. Biol.
278
,
55
62
(
2011
).
42.
S.
Tanabe
and
N.
Masuda
, “
Evolution of cooperation facilitated by reinforcement learning with adaptive aspiration levels
,”
J. Theor. Biol.
293
,
151
160
(
2012
).
43.
C. P.
Roca
and
D.
Helbing
, “
Emergence of social cohesion in a model society of greedy, mobile individuals
,”
Proc. Natl. Acad. Sci. U.S.A.
108
,
11370
11374
(
2011
).
44.
E.
Pichler
and
A. M.
Shapiro
, “
Public goods games on adaptive coevolutionary networks
,”
Chaos
27
,
073107
(
2017
).
45.
A.
Pal
and
S.
Sengupta
, (2021), “
ReinforcementLearning-StatDynNet
,”
Github.
https://github.com/anuanupapa/ReinforcementLearning-StatDynNet.

Supplementary Material

You do not currently have access to this content.