top of page

Pattern Recognition in Sectorial ETFs

Updated: Nov 25, 2020

In the great bhavacakra of the modern global economy sector rotation strategies attempt to capture excess returns by investing in sectorially themed baskets and attempting to capture the spotlight of that given sector. In this strategy industrial sectors are assumed to be in rotation, like in the wheel of life; what is below will be above, and vice-versa, given enough time. The axis of the bhavacakra houses three animals: a pig, a bird and a snake, representing ignorance, greed and hate. Quantitative analysis will help us, at least, against ignorance and help us contemplate the great wheel from a more elevated position. The wheel is held by Yama, or Mara, so that we can keep bankruptcy in mind, always, as we check if this karmic cycle of the sectors really happens, or has been recently stagnant.

What are the relations among sectors and the market? As a whole, there should be some discernible pattern in pricing or returns relations. We will investigate these relations using the following ETF list as proxies for industry sectors:

These ETFs have been selected without an exhaustive research and come from this post. We are adding IGE, iShares North American Natural Resources ETF, due to its inherent interest for being at the very basis of the wheel, resource gathering operations. SPY and QQQ are added to our analysis as possible instruments in a sector rotation strategy or a possible predictable variables. We will have a look at the overall market and the sectors individually as, when aggregated, the sectors have to lead to the emergence of the whole market:

import matplotlib.dates as mdates
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np

self = QuantBook()
instruments = ['QQQ', 'SPY']

# Use these, or other sector ETFs for simplicity:
sector_ETF = ['IYM', 'XHB', 'FSTA', 'TPYP', 'KBWB', 'IXJ', 'XLI', 'KBWY', 'TDIV', 'IXP', 'RYU', 'IGE']
ETF_symbols = {etf: str(self.AddEquity(etf).Symbol.ID) for etf in sector_ETF}
pred_symbols = {symbol : str(self.AddEquity(symbol).Symbol.ID) for symbol in instruments}
all_symbols = {**ETF_symbols, **pred_symbols}
inv_symbols = dict(map(reversed,all_symbols.items()))

start =  datetime(2010, 1, 1)
end =

history = self.History(self.Securities.Keys, start, end, 

The symbols are stored in dictionaries for easy access by ticker or symbol reference internal to Quantconnect´s systems. The description of symbol objects is here. Dictionary comprehension in Python makes it very "code efficient", also splatting the dictionaries (**, double splat operator) can be used to naively join them. The inverted dictionary will come handy to simplify dataframe headers.

Let´s also define some handy plotting functions, for formatting purposes, specially to obtain diagonal heatmaps from seaborn (plot_hm):

def plot_df(df, color='blue', size=(16, 7), legend='Close Price', y_label='Price in USD'):'dark_background')
    plt.rcParams["figure.figsize"] = size
    ax = df.plot()
    x = 0.01
    y = 0.01
    plt.text(x, y, '', fontsize=15, transform=ax.transAxes)
    plt.legend(ncol=int(len(df.columns) / 2))
    date_form = mdates.DateFormatter("%m-%Y")
    _ = plt.xticks(rotation=45)
def plot_hm(df, title='Title'):
    mask = np.triu(np.ones_like(corr, dtype=bool))
    cmap = sns.color_palette("RdBu")
    ax = sns.heatmap(df, mask=mask, vmax=.3, center=0, cmap=cmap, annot=True,
                     square=True, linewidths=0, cbar_kws={"shrink": .5})
    _ = plt.setp(ax.get_xticklabels(), rotation=45)

We can now take a look at the price levels in time split in two segments:

price_history = history['close'].unstack(level=0)
new_cols = [inv_symbols[col] for col in price_history.columns]
price_history.columns = new_cols

It is apparent that disturbances are shared, market and sectors follow similar overall patterns with local differences in movement. We can check the values for correlation and covariance of prices looking for sectors with special behaviour but there is too much "blue":

The cycling nature is not apparent from the correlation and covariance values, with some anomalies: (North American) natural resources ETF IGE to the market, staples to healthcare and staples to technology and energy. Looking at price, correlations and covariances are relatively high in the past 10 years, showing weaks signals of rotation in limited sectors. For a somewhat pareidolic analysis we can investigate the pairplot for the prices in these assets:


We need to zoom in to specific areas for patterns that catch the eye. Positive correlations are represented by 'x=y' like function plots, IGE, the second column shows a faintly inverse correlation, and then there are the diverging, or channel like structures. Although difficult to interpret for the human eye, correlation and covariance patterns appear very distinctively. Technology (TDIV, weighted for dividends more than technology) shows a perfect fit to SPY. In general, the messier the pairplot for a given pair, the more risk diversification offered by holding that given pair, the higher the sectorial rotation effect could be. These clusters are maybe too well ordered, too appealing to the human eye to represent sectorial rotations.

If this is the overall stock market what we are looking to diversify, this partial pairplot gives us the information we need:

cols = price_history.columns
_=sns.pairplot(pd.DataFrame(price_history), y_vars=instruments, x_vars=cols)

We need to zoom in here, mostly linear relationship, very few "clefts" in the pairs to show possible pricing divergences:

We can look at the same analysis for the daily returns for the same instruments, for the sake of completeness of the analysis. Correlation and covariance of both prices and returns are worth investigating, both contain complementary information, returns will generally constitute a stationary times series and prices will not, so that different insights can be gained from looking at both together. Alternatively, fractional differentiation could be used and obtain correlations and covariances for those partially differentiated time series, in an attempt to condense information. First, for the returns characteristics, uncentering the color scale:

Correlation and covariance of results are positive all across the matrix. This is not unexpected and leads to a completely different galaxy of pairplots (masked by some outlier cases, mind the axis values, these could be different) showing the strong relationship in the returns pairs:


And for the market:

We now have an intuition that the different sectors, in general, have had a 10 year period of relatively high price correlation with a few exceptions, with also a high correlation of returns. Then, when do sectors rotate? Conventional wisdom puts the recovery to recession cycle at 5 or 6 years period in which sectorial patterns should emerge, so our 10 year correlation and covariance should have been more red and less blue, our plairplots more cloudy and less linear, less correlation should be expected if there is a case for strong sectorial rotation, now and in the future. Otherwise the economy itself does not make a lot of sense and the market, in this case the US market, is concentrating investment efforts all across sectors without regards to production-consumption cycles. This is a situation in which, apparently, all sectors and all trades have been the same sector and the same trade, with the exception of the "forced-in" natural resources, the base of the pyramid, that could be the only one still "alarmed" by economic cycles.

We have no solution to sectorial rotation from correlation and covariance alone, we are blinded and cannot see the bhavacakra in the sectors with this methods, nor how to effectively construct a rotating portfolio. Some sectors are even "The Market" by themselves. We will have to analyze the capacity to look forward that these sectorial ETFs provide with other tools, their predictive power for example, and check if these sectors, while moving along their "karmic" lines, predict the price levels of one another, or the market in general. This will be analyzed in our next publication.

Remember that information in does not constitute financial advice, we do not hold positions in any of the companies or assets that we mention in our posts at the time of posting. If you are in need of algorithmic model development, deployment, verification or validation do not hesitate and contact us. We will be also glad to help you with your predictive machine learning or artificial intelligence challenges.

22 views0 comments

Recent Posts

See All


bottom of page