Nonlinear shifts and dislocations in financial market structure and composition

Equity markets are highly nonstationary and continually change their behaviors over time. In recent years, many securities and asset classes have exhibited huge gains and/or fluctuations in their returns. In particular, equities exposed to the broad macroeconomic themes of artificial intelligence, decarbonization, and energy prices have benefited from growing investor interest. As investors turn to these market themes, however, other sectors are being overlooked and are experiencing capital outflows. These shifts in market dynamics can yield dislocations and create opportunities for arbitrage with respect to certain sectors. All these reasons make the time-varying study of stock prices more essential than ever. In addition, changes to the twenty-first century landscape of businesses require more nuanced sectors to divide the marketplace than traditional options such as the GICS classification system. In this work, we develop new methodologies to most appropriately identify nonlinear shifts in market behaviors, to reveal suitable market themes for portfolio diversification, and to optimally construct portfolios to perform best over long time horizons, all via a modern and sophisticated sector division.

Modern financial portfolio management is an increasing complex endeavor. Sophisticated investment firms have numerous teams focused on strategic, dynamic, and well-diversified asset selection over a variety of time frames, from periods of nanoseconds to years. In this paper, we look holistically over the long term and address three aspects that remain consistently pertinent to portfolio selection: shifts in market structure and behavior over time, representation of various sectors in strong-performing portfolios over the long-term, and a framework for identifying communities of equities from a universe of potential candidates.

A significant body of literature focuses on the evolution of a financial market structure, where the correlation matrix of individual security returns is used as the primary item of study.^1–4 Studying the nuanced connectedness between various groups of assets provides insights into the interdependencies of certain sectors, and how the market can be decomposed into smaller communities.^5–9 In more recent years, network analysis has been used in various applications in financial markets, such as equities,¹⁰ foreign exchange,¹¹ cryptocurrency transactions,¹² mergers of companies,¹³ and global volatility.¹⁴ Most of these analyses originally derive from the correlation matrix, but nonlinear alternatives also exist.¹⁵ 

Like previous work, this paper focuses on identifying structurally important (and locally similar) communities of stocks, where each independent cluster on the network exhibits local affinity and dissimilarity with global comparisons across the graph. Our analysis also makes use of a modern and nuanced division of our universe of stocks into 60 sectors. The existing GICS (Global Industry Classification Standard) sectors are rather old, limited in number, fail to caption important differences in sector subdivisions, and have numerous companies that are misclassified by today’s understanding. Our use of 60 sectors augments and aids the process of trying to find relevant market communities and themes.

We must also acknowledge the influence of statistical physics and time series analysis on this work, particularly Sec. III. In financial markets, time series analysis has been applied to a wide range of asset classes, such as equities,^16–19 foreign exchange,²⁰ cryptocurrencies,^21–28 and debt-related instruments.²⁹ Such methods have been used more broadly in a variety of other disciplines, including epidemiology,^30–39 environmental sciences,^40–48 crime,^49–51 and other fields.^52,53

Traditionally, the study of time-varying market shifts has rested on regime-switching models. These begin with the observation that financial assets exhibit switching patterns, moving between periods of heightened volatility and ease.^54–56 One then applies the analysis of locally stationary time series^57–60 to partition a long time window into designated and discrete segments. These segments are classified into a small number of distinct regimes, usually two,^61–63 but often different numbers⁶⁴ or with varying assumptions that may yield different levels of flexibility.^65–71 In addition, change point or structural break detection algorithms have been applied for a similar purpose,^72–74 to divide a time interval into periods of fundamentally distinct market behavior. These do not have a specified number or type of regime in mind, nor do they classify and group different segments, but they do divide the entire window of analysis into discrete periods. While these discretized divisions have their place, they are a little contrived; therefore, we instead develop a suite of different methods that reveal market shifts on a continual basis and interpret their different uses and benefits.

The study of portfolio optimization has grown widely in the decades since Markowitz’s mean-variance model.^75,76 While there are many approaches, including statistical mechanics,^77–79 clustering,^80,81 fuzzy sets,^82,83 regularization,^84–86 Bayesian approaches,⁸⁷ and multiobjective optimization,⁸⁸ almost all of these approaches aim to find a unique optimal portfolio. Frequently, NP hard constraints,^89,90 such as portfolio cardinality constraints,⁹¹ are imposed, which can make the selection of a single portfolio a difficult computational problem. In addition to being computationally difficult, finding a single optimal portfolio may be sensitive to changes in the data and may prove inflexible for future decision making. Thus, in this paper, we conduct a series of appropriately chosen sampling experiments to instead identify collections (specifically quantiles) of top-performing portfolios and analyze their composition.

This paper is structured to address the broad topic of dynamic asset allocation and optimal portfolio construction during changing market conditions. Our paper proposes new mathematical techniques to address the three most critical aspects of tactical asset allocation: identifying market regimes, highlighting association between underlying portfolio components (assets, trading strategies, and so on) and optimally constructing portfolios within a sample space (with some degrees of freedom). In Sec. III, we compare a variety of new techniques to identify shifts in behavior in the market, consistently taking into account the structure of the market divided into our 60 sectors. We also contrast the findings and effectiveness of these techniques in various portfolio management contexts. In Sec. IV, we turn to the network structure of the market and use our nuanced 60 sectors and network analysis to identify key clusters of structurally similar equity themes and test for essential ingredients in long-term portfolio diversification. In Sec. V, we conduct an extensive sampling experiment where we compare the long-run performance and portfolio composition of long-only, short-only, and long-short portfolios, taking into account the sector structure of the market. In Sec. VI, we conclude. Throughout the paper, we take an interdisciplinary approach, as often seen in econophysics, borrowing methodologies from a wide range of mathematical foundations.

We analyze the daily closing prices of 268 equities between 2005-01-01 and 2023-12-31. Each equity has been allocated into one of 60 (new) sectors that more closely tie with their respective companies’ roles in the business cycle. We feel that our nuanced sector allocation, which is much richer than the traditional Global Industry Classification Standard (GICS) methodology, is key in producing high quality insights. These sector allocations are done manually based on market consensus and are provided in  Appendix A. Price data have been sourced from Yahoo Finance.⁹² 

The primary objects of study in this section are aggregated sector log returns (averaged over their composite assets) investigated over 30-day periods. We present four primary measures of deviation to identify temporal shifts in a market structure. As we consistently group returns by sector, such changes may identify dislocations in sector pricing and, hence, arbitrage opportunities.

This analysis of 30-day intervals requires comment, specifically around the choice of a 30-day window and alternatives. In general, smoothing out a time series is a balance with respect to signal and noise. Given the high level of noise in financial time series, de-noising these time series often helps us make better predictions as we can more accurately capture the true “signal” in the underlying time series. A 30-day window acts as a low-pass filter; a larger window induces more bias into the estimator but reduces the variance more substantially, while a smaller window yields less bias. In our previous work, we have experimented with several time lags, windows, offsets, and so on. Having applied similar methods to underlying financial time series, a 30-day window tends to generalize best. As we acknowledge in Sec. VI, alternative time windows may be an avenue for future work.

Our second measure is slightly more sophisticated. We retain $R j[t−τ+1:t]$ as sequences without summing them and instead compute Wasserstein distances between these sequences, understanding them as distributions; that is, let $d W$ be the $L 1$ Wasserstein metric. Consider each length-30 sequence $R j[t−τ+1:t]$ of daily log returns as a distribution of real values. The Wasserstein metric $d W( R j[t−τ+1:t], R j[t+1:t+τ])$ is defined informally as the work required to transform the distribution $R j[t−τ+1:t]$ into $R j[t+1:t+τ]$ and is outlined in  Appendix B.

The Wasserstein metric $d W$ offers a way to simultaneously compute aggregated measures of disparity between two sequences (considered as distributions) while allowing individual terms to remain relevant. For example, a strong negative shock $R j(t+1)$ followed by positive shock $R j(t+2)$ in the return sequence $R j[t+1:t+τ]$ could “cancel out” in the computation of $S t$⁠, whereas both terms will remain relevant in the computation of $W t$ below.

Finally, we include one additional measure of deviation that targets not just magnitudes of deviations in returns between 30-day periods, but changes in the ranking order of sector returns. Once again, we utilize the 30-day summed returns (equivalently monthly log returns) $R ¯ j[t−τ+1:t]$⁠. Let $K t$ be the Kendall tau coefficient between the two vectors $( R ¯ j [ t − τ + 1 : t ] ) j$ and $( R ¯ j [ t + 1 : t + τ ] ) j$ in $R n$⁠, where $j$ ranges over $1,…,n=60$ and produces two vectors of dimension 60. The Kendall tau coefficient is computed as follows: a pair $1≤i<j≤n$ is said to be concordant between the two vectors if the order of $( R ¯ i[t−τ+1:t], R ¯ j[t−τ+1:t])$ and $( R ¯ i[t+1:t+τ], R ¯ j[t+1:t+τ])$ is the same and discordant otherwise. Then, the coefficient is defined as the difference between the number of concordant and discordant pairs as a proportion of all $( n 2 )$ pairs. The Kendall tau has an associated $p$-value corresponding to the null hypothesis of a zero tau coefficient between vectors. We plot $K t$ in Fig. 3, again for $t=τ,…,T−τ$⁠. For the sake of robustness, we also include the related Pearson and Spearman correlation coefficients between the same vectors. The Spearman correlation is defined as the Pearson correlation computed between the ranks of the two vectors and is, thus, computed entirely based on the ranks of the sector return values, not their relative magnitudes.

Figure 1 reveals some key similarities between our measures $S t$ and $C t$⁠. First, both measures identify the global financial crisis (GFC) and the COVID-19 market crash as the most significant equity market crises over the full period investigated. We consider values in the top fifth percentile of each respective series as indicative of the most atypical market dynamics and mark these thresholds in the figure. Both measures detect three breaches of this 5% threshold during the GFC and one during COVID-19. Furthermore, both measures broadly correspond in amplitude over the entirety of the period studied: generally, they are relatively high and low during the same times.

And yet, some differences between the quantities exist that distinguish them as estimators of structural market shifts. First, $S t$ may serve as a more effective tool in an online capacity for market turmoil identification. Prior to the GFC and COVID-19 market crash, smaller scale perturbations are detected, which could be used as an indicator for subsequent sell-offs in equity markets. Next, $S t$ produces greater contrast in magnitude when compared to the $C t$ measurement. This is highlighted during the GFC: both $C t$ and $S t$ produce three spikes at approximately the same time, but the spikes of $S t$ exhibit greater variability in amplitude. On the other hand, one potential drawback of the $S t$ measurement is the potential for false positive structural shift identification. During the period investigated, 11 breaches of the 5% threshold are identified using $S t$⁠. By contrast, $C t$ only identifies four structural breaks—three during the GFC and one during the COVID-19 market crisis.

Our more refined measure for time-varying market shifts is shown in Fig. 2 and determines the rolling, cumulative sector-to-sector Wasserstein distance between adjacent length-30 sequences of log return distributions. Figure 2 is quite different from the three other measures we have employed for several reasons. First, our Wasserstein metric approach produces the most abrupt changes around market crises. One could argue that this methodology has a higher signal-to-noise ratio with respect to crisis identification, and given the strong association between the amplitude of the estimator and the actual existence of financial market crises—it may indeed be the most effective tool for regime or crisis identification. This is to be expected, as using the Wasserstein metric rather than the sum in $S t$ allows one to more carefully take into account extreme daily values, even if a sharp positive market shift is directly followed by a sharp negative shift. Indeed, such behavior is quite common during crises, and only using the Wasserstein metric will properly detect these unique patterns. Like our $S t$ methodology, the Wasserstein approach produces significant variability in amplitude. By contrast, $C t$⁠, which captures the evolutionary change in the amount of collective market movement, does not exhibit the same variance in amplitude. This framework may have favorable properties for online regime identification, crisis detection, and be useful as an indicator of market dislocations or required shifts in optimal portfolio composition.

Finally, we display the Kendall tau $K t$ in Fig. 3(a), along with the closely related Spearman correlation coefficient in Fig. 3(b) and the more standard Pearson coefficient in Fig. 3(b). Unlike our other time-varying measures, Fig. 3(a) displays the evolution of change in sector log returns ranks and perhaps best indicates the need for changes in portfolio composition over time. The three figures display some similarity in their evolutionary structure and volatility. However, it is notable that pronounced spikes around the GFC and COVID-19 crash are only clearly visible in Pearson Fig. 3(b) and not the others. Log returns are not normally distributed, particularly during market crises, and therefore, the Pearson coefficient may be distorted by extreme values during these times. Thus, looking only at the ranks (Kendall tau) or correlations between the ranks (Spearman) reveals no particular shifts in rank orderings during the crises. Instead, we see considerable consistency over time in the behaviors of $K t$ and the closely related Spearman coefficient, included for robustness. The magnitudes of the Kendall tau and Spearman coefficient are generally quite low, never exceeding 0.3 and 0.4, respectively, and more frequently close to 0. To complement this, we include a histogram of the associated Kendall tau p-values in Fig. 3(c). Less than half (of 4721) of the $p$-values are less than 0.05, with many distributed between 0.05 and 1, indicating that there are many times $t$ where the Kendall tau $K t$ is not significantly different from zero. This indicates a considerable shift in ordering from one rolling period of returns to another. Both of the above findings indicate that optimal portfolios must switch between market sectors on a near-continual basis.

We then form two networks, one traditional and one somewhat unusual. First, more traditionally, we form a weighted graph using the $n$ sectors as vertices and the associated distances $D i j$ as edge weights. In this graph, two vertices (sectors) have a small edge weight between them if they are highly correlated (high correlation being equivalent to small $D i j$⁠). Then, we pass this to Kruskal’s algorithm to form an associated minimum spanning tree (MST).

On the other hand, a little unusually, we form a weighted graph directly using the correlation coefficients $Ψ i j$ themselves as edge weights. In this graph, two vertices are close if they have a low correlation. Unlike $D i j$⁠, $Ψ i j$ does not satisfy the triangle inequality property of a metric; therefore, there is no transitive property of closeness. However, we once again form a minimum spanning tree on this weighted graph. This tree will reveal the minimal sequence of edges to connect the graph subject to wanting connected sectors to be uncorrelated, which is important in diversification.

Kruskal’s algorithm to produce a minimum spanning tree proceeds as follows. In a weighted graph, one first orders the edges by increasing weight. Then, edges are successively added to the tree if and only if the addition of such an edge would not create a cycle. By design, the algorithm never produces any cycles, but successively adds edges until the graph is connected, producing a connected spanning graph with no cycles, and, hence, a tree. We remark that Kruskal’s algorithm works fine when some edges of the graph have negative weight, which may be the case for the correlation weightings $Ψ i j$⁠. The resulting MST still aims to minimize path lengths with respect to correlation; negative correlation values are fine and indeed are even desired to show assets with a low correlation, which is beneficial for diversification. Concretely, Kruskal’s algorithm finds the exact same MST as if we adjusted all edges by a constant, for example, examining edges $Ψ i j+1$⁠, all of which would be non-negative.

In Figs. 4 and 5, we display the minimum spanning tree using $D$ and $Ψ$ as the edge weights between sectors considered vertices, respectively. In the first network, Fig. 4, we identify sectors that are more closely correlated. Several noteworthy clusters are identified, and such groupings could be used to identify broad themes among the market. First, we see a strong association between software applications, Internet, and software infrastructure companies. These sectors are connected to semiconductors and consumer electronics, with the latter two exhibiting higher centrality on the graph. These nodes are connected to sectors, such as computer hardware, information technologies, electrical equipment, and scientific instruments, which are again more central to the underlying graph. This entire collection of sectors could form a primary “digitization” theme or be broken down into more detailed sub-themes, such as “software businesses” and “computing infrastructure.” We also see a strong association between healthcare plans, medical distribution, biotechnology, and drug manufacturers, forming a “healthcare” theme. One can also see a collection of sectors that includes discount stores, home improvement retail, packaging containers, and restaurants, forming a theme related to “consumer goods and services.” The groupings that are identified on the graph could be used in several contexts. One potential application would be forming new communities of equities based on the exposed themes and re-computing sector attributes, such as expected returns, volatilities, and beta against the larger market. One could use these themes as an alternative to portfolio factors or could diversify a portfolio with respect to these algorithmically determined themes (rather than GICS sectors).

In Fig. 5, we display the minimum spanning tree associated with the correlation matrix $Ψ$⁠, directly. Edges are drawn between equities with a lower correlation, rather than a greater similarity. The structure of this network is striking, almost entirely centered around the gold sector (which contains four assets) as a star-shaped graph. Almost every sector shares an edge only with gold because the gold sector repeatedly exhibits the lowest pairwise correlation relative to anything else. The centrality and importance of the gold sector demonstrate that gold (and gold-related businesses) is a vital asset in diversifying financial portfolios in all weather settings.

In this section, we conduct a series of simulation studies across different sample spaces. We sample over long-only, short-only, and long-short portfolios, uniformly across all stocks or stratified by sectors. This allows us to investigate the composition of strong-performing portfolios across sectors and identify sectors that are disproportionately represented in top-performing portfolios. We complement the previous work that only varied the portfolio size as the determinant of the sample space and did not consider any sector structure.⁹³ 

In Sec. V B, we report numerous results from sampling both uniform and stratified portfolios, either long-only, short-only, or long-short, and for various values of portfolio size $m$⁠, most commonly $m=30$⁠. In every instance, we draw 10 000 samples from each sample space in each experiment. We report both quantiles of the annualized Sharpe ratio across the entire sample space, as well as the composition of portfolios by sector in top 1% performing portfolios, quantified and ordered by a Sharpe ratio.

Table I displays Sharpe ratio quantiles for portfolios of size $m=10,20,30$ with uniform sampling and then $m=30$ with stratified sampling. A short-only trading strategy over such a prolonged period in the market, where equities exhibit survivorship bias and significantly positive cumulative returns, would almost certainly be a losing investment strategy. This is visible in the consistently negative values of the short-only Sharpe ratios. Accordingly, we do not focus on the short-only strategy and contrast the return profiles of the long-only and long-short strategies. The results demonstrate that the long-only portfolio is superior to the long-short portfolio in almost all points of the distribution. This highlights, over the long-term, the degree of a manager skill that is required when incorporating more shorting into a portfolio. Our results indicate that among our collection of equities, short positions are best used when highly targeted and over discrete periods with finite end dates. Indeed, shorting allows limitless downside on portfolio returns. Furthermore, our simulations do not consider transaction costs, stock loan, or margin requirements, which all add to the difficulty of beating a standard long-only portfolio. The inherent asymmetry associated with short positions further adds to the difficulty in long-short managers beating traditional long-only investors. The four parts of the table show that, predictably, the spread of quantiles narrows from $m=10$ to 20 and 30. In addition, a comparison of the third and fourth quarter of the table shows essentially no difference in the Sharpe distribution between stratified and uniform sampling.

Figure 6 shows the kernel density estimation [6(a)] and histogram [6(b)] of the Sharpe ratio distribution for long-short and long-only investment portfolios with $m=30$⁠. Confirming Table I, it is clear that long-only Sharpe ratios are superior in expectation to that of long-short portfolios due to the length of the period. We also visually see that the distribution of outcomes is much narrower for long-only Sharpe ratios—highlighting the increase in the size of the sample space when incorporating more degrees of freedom into the portfolio. However, the increase in the sample space is predominantly negative—further emphasizing the importance of a manager skill when managing portfolios with more degrees of freedom. By way of contrast, Fig. 6(c) repeats this experiment restricting to the period 2007-09-01 to 2009-03-31, comprising the global financial crisis (GFC). Here, we see a clear shift, with short-only outperforming long-short and short-only. This reflects the fact that for investment managers to successfully engage in short-selling, they are better off doing so in focused bear market windows.

In Table II, we quantify this comparison between long-only and long-short distributions. Specifically, we compute $P(X>Y)$⁠, the empirical probability that a randomly chosen long-short portfolio has a greater Sharpe ratio than a randomly chosen long-only portfolio. We compute this for $m=10,20,30$⁠, using uniform and stratified sampling and over the entire period or just the GFC. Confirming Fig. 6, we calculate a relatively small probability (around 7%) that a candidate long-only portfolio across the entire period can beat long-short. However, this increases to around 70% when restricting to the GFC. Curiously, we observe slightly smaller probabilities for stratified sampling than uniform sampling, but only when the full period is considered, but it is possible these differences are simply random.

In Tables III, IV and V, we turn to the second part of our experiments, where we investigate the sector composition among top-performing portfolios (top 1% by the Sharpe ratio). Table III gives the top five most occurring and bottom five least occurring sectors in uniformly sampled long-only, short-only, and long-short portfolios of size $m=30$⁠, ranked by the percentage of high-performing portfolios they appear in. In long-only portfolios, we see primarily the largest sectors represented the most: semiconductors, oil and gas, and REITs (real estate investment trusts). The smallest sectors (by a number of constituent stocks) are represented the least, such as credit and casinos.

Thus, we complement this with Table IV, where we adjust the percentage of representation in top-performing portfolios by the size of each sector. For example, biotechnology is represented 2.02 times more often in top-performing portfolios relative to the number of biotechnology stocks in our equity universe. Results are expected to be relatively similar to Table V, which reports percentages of occurrence in top-performing portfolios under stratified sampling. Stratified sampling is specifically designed to correct for the difference in sizes of sectors; therefore, results are expected to be similar. In both Tables IV and V, the top and bottom sectors, respectively, represent the most and least frequently represented sectors in top portfolios adjusted for their different sizes.

We see two main themes in these aforementioned tables. First, biotechnology, medical distribution, and medical instruments are among the most represented sectors (adjusted for size) in top-performing long-only portfolios. This indicates the significant growth of the pharmaceutical industry over the past two decades, especially around the COVID-19 pandemic.⁹⁷ Second, (physical) retail and department stores are among the most represented in short-only portfolios and least represented among long-only portfolios, in stark contrast to Internet retail. This reflects the growth in online shopping and the decline of brick-and-mortar retail in the past 20 years. Furthermore, numerous other findings can be observed, such as a poor representation (even after adjusting for sector size) of credit-related companies and casinos and a strong performance of restaurants and producers of non-alcoholic beverages.

Our paper makes several contributions to the field. First, in Sec. III, we introduce new techniques to identify structural shifts in the market structure relative to our original sector decomposition, which may identify market dislocations between sector pricing. The Wasserstein metric may prove especially useful for online change point detection, given its association with empirically significant financial crises and clarity in the amplitude of its measurements. The four techniques we present each have respective benefits and drawbacks and could be used by investment managers in different contexts. Based on the individual investor’s portfolio management goals and preferences, one of these methods may be preferable to the others. While the metrics $S t, C t, W t$ all reveal financial crises (including signals in advance), the Kendall tau coefficient $K t$ has a complementary finding that optimal portfolios must switch between market sectors on a near-continual basis, an observation that is unchanged during crises.

In Sec. IV, we study the market’s connectedness and underlying network structure relative to our new sectors, building from both the correlation matrix and the transformed distance matrix. We capture two key findings. First, we identify communities of sectors that behave similarly over time. Such communities could be thought of by investment managers as broad themes to diversify across when managing a collection of investments. Their differing market centrality may also be of note and significant when diversifying across the revealed themes. Second, the gold-related sector is identified as an essential group among a portfolio of equity sectors. This is primarily due to gold’s long-term (structurally) depressed correlation with almost every sectors, as reflected in the minimum spanning tree of the untransformed correlation matrix.

Finally, in Sec. V, we conduct an extensive sampling experiment, employing uniform sampling across all stocks and stratified sampling adjusted by sector sizes and varying portfolio sizes and the period of analysis. We compare the distributional performance of equally weighted long-only, short-only, and long-short investment portfolios by the Sharpe ratio and investigate the sector composition of the top 1% performing portfolios. We show in quantitative detail that the incorporation of shorting is generally unwise across a long period, but may show positive results over select bear market periods. Regarding sector composition of top performing portfolios, we carefully control for sector size in two different ways and are able to identify the sectors that were most and least represented in successful portfolios. We see that medical-adjacent industries, such as biotechnology, medical instruments, and distribution performed well, as did Internet retail, while long positions in physical retail were poorly represented in top performing portfolios.

This section is motivated by the study of investment managers along the spectrum of sophistication, which is generally correlated with management and performance fees. Typically, investors who have more discretion as to how they manage their portfolio by way of a wider degree of instruments (equity, credit, options, and others) and direction (both long and short) to use, charge higher fees for managing clients’ money. Mathematically, this is equivalent to making investment decisions from a wider universe of potential decisions (a larger decision space). The specific exposures that such sophisticated portfolios can capture may deviate meaningfully from the index in both positive and negative directions and, therefore, make the matter of an investor’s skill level extremely important. With the increased fees these portfolio managers command, asset owners should see a commensurate increase in returns and risk-adjusted returns. The general approach in this section could be used by asset allocators to assess fee justifiability among potential fund managers.

Overall, our proposed methodologies are most helpful to a Chief Investment Officer or a Head of Quantitative Research type role, where one is managing a portfolio of underlying investors or trading strategies. Our paper demonstrates concrete use cases for managing equity hedge funds, where one can systematically identify shifts in equity market dynamics, clusters of businesses that are likely to respond similarly to exogenous shocks and make appropriate inference as to how portfolio returns are likely to be impacted. These ideas are at the very core of quantitative investment and portfolio management, and although they may seem independent, the three methodologies we present are inter-connected. Our regime identification would be used in an online (real-time) setting to identify changes in market dynamics. As market dynamics change, portfolio managers are likely to change their underlying portfolio composition. To understand which equities are likely to behave similarly and dissimilarly, one must be able to determine their inherent similarity. Our network analysis is a parsimonious framework for identifying clusters of equities that are likely to behave similarly. Finally, our portfolio simulation section provides portfolio managers with an approach for thinking about how the distribution of risk and return is likely to change as the underlying composition changes.

In particular, the methods in this paper, such as the 30-day smoothing window and long-term analyses of portfolio performance, are designed for investors who make portfolio decisions relatively infrequently, that is, investors who are building trading strategies for mid-long term asset allocation. This includes large pensions and endowments, superannuation funds, and focused investment groups who specialize in security selection rather than exploiting market inefficiencies in very short-term periods. This approach is not uncommon in trading, where momentum and trend following strategies can work based on smoothing out the underlying time series. CTA and managed futures strategies have worked in this way for many years.

There are a variety of opportunities for future research. First, one could extend our work in detecting the market structure and see if this can be done in an online setting where machine learning algorithms may predict the probability of future dislocations in the market. Furthermore, many parameters in our modeling approach could be optimized based on the specific properties of an underlying time series. For instance, a time series with more substantial short-term variance may require a tighter window, while a more slow-moving time series may benefit from a larger window. This could be estimated based on the specific time series properties that are presented. Second, it would be interesting to explore the market’s inherent connectedness between a mixture of asset classes. That is, how would the market’s network diagram look after fixed income, cryptocurrency, currencies, and other options are incorporated into the graph? After including assets with longer-term structurally depressed correlation with equities, alternative assets may be seen as a more suitable diversification strategy than gold. Finally, in our portfolio sampling section, we highlight the required level of a manager skill (by quantile of the distribution) in order to justify the existence of long-only vs long-short equity managers. This is due to the enlarged space of outcomes as investors manage portfolios from larger decision spaces. It would be interesting to test this on a wider range of periods than simply the entire window and the GFC. There may be numerous periods other than just market crises where shorting has its place in expanding the possible space of returns in both positive and negative directions. Similarly, with an even larger sample space of asset classes, such as a portfolio that can invest across the capital structure and in different markets, there may be an even higher benchmark on an investment manager skill.

The authors have no conflicts to disclose.

Nick James: Conceptualization (equal); Data curation (equal); Formal analysis (equal); Investigation (equal); Methodology (equal); Software (equal); Validation (equal); Visualization (equal); Writing – original draft (equal); Writing – review & editing (equal). Max Menzies: Conceptualization (equal); Data curation (equal); Formal analysis (equal); Investigation (equal); Methodology (equal); Software (equal); Validation (equal); Visualization (equal); Writing – original draft (equal); Writing – review & editing (equal).

The data that support the findings of this study are openly available in Ref. 92.