 that an investor has and the need to allocate that budget between competing investments. This budget choice could be on a cash basis or on a volatility basis, where a certain percentage of their target volatility is allocated to an investment. Correlation and the risk budget are linked as the risk budget is a choice between investing a proportion of your wealth in different investments, each with a varying degree of risk. This is why the correlation amongst those investment choices is an important input.  

Correlation is a simple measure that explains the relationship between two variables. It normalises the covariance of the two variables, i.e how they move together, by scaling using the standard deviation of the variables. 

If your portfolio consists of two stocks which have a negative correlation, then your overall portfolio volatility will be lower than the weighted average of their volatilities. When one stock rises in price, most likely the other will fall and in aggregate your portfolio volatility will be reduced. Correlation has the following properties

bounded between -1 and 1 and can never be outside that range

represents how strong the relationship is between two sets of data   

can and does change through time and depends on the window used for the correlation calculation

Correlation is a critical component when you consider two or more stocks in your portfolio. Diversification of a stock portfolio only benefits the investor when there is less than perfect correlation between the stocks. We saw this in our post on the 

 where we illustrated the risk versus return benefit of combining stocks with less than perfect correlation.   

When it comes to portfolio management and multiple strategies competing for a risk budget, i.e inclusion in the overall portfolio, then the importance of correlation shifts from the stock level to the strategy level. We will illustrate these concepts in the following sections. 

Correlation moves from stocks to strategies

We have seen the power of modifying correlation between stock A versus stock B. Changing correlation can either reduce the portfolio risk whilst generating the same amount of return or the correlation change improves the returns for the same amount of risk - both methods increase the risk-adjusted returns of the portfolio. We now shift that logic to the strategy level instead of the stock level. We are considering the impact of strategy correlation when combining two strategies together into one final portfolio. 

An investor has thousands of stocks to choose from to form their portfolio and can choose from various investment strategies, each of which generates a set of stocks to invest in. The set of stocks investors can choose from is normally called the 

of stocks. In quantitative terminology, when we consider all stocks and want to rank all stocks against each other we refer to this as the 

In this cross-sectional world where we are attempting to come up with quantitative equity trading signals, we must move away from any toy examples on a small set of 3-5 stocks and instead think about ranking between thousands of stocks. Therefore, our effort is focused on developing strategies that are based on sound mathematical and financial concepts that are applicable across all stocks so we can generate a final score (ranking) for each stock, which represents a stocks future forecasted return or

Imagine we develop two investment strategies, call them strategy A and strategy B. Strategy A is a value strategy where it focuses on a mathematical rules-based method to identify the top 50 value stocks. Strategy B is a momentum strategy that focuses on identifying the top 50 stocks based on some new momentum metrics. The correlation concept we applied to stock A and stock B can likewise be applied at the strategy level so our goal here is to hopefully identify low or even negative correlation between momentum and value and therefore combine the two strategies into a final portfolio. This final portfolio would then provide higher risk-adjusted returns to an end investor, rather than investment in purely strategy A or strategy B independently.

To illustrate the strategy correlation concept we mention above, we can look at equity style factors and their correlation. Equity style factors in general have some overall correlation principles but the correlation is time-varying and does move around over short time horizons. Two factors that have historically tended to have negative correlation are 

Therefore an end investor if they wanted to increase their risk-adjusted returns could consider investing in two strategies, say for example 

, to really benefit from this negative correlation to increase their overall risk-adjusted returns of their final portfolio.

Decrease portfolio volatility by equal-weighting value and momentum 

An investor if they wished to reduce overall portfolio volatility could allocate 50% of their investment to value and 50% to momentum and 

Why does the correlation among assets matter so much. We describe how lower correlation helps lower portfolio risk and increase risk-adjusted returns.

2. Correlation within your Stock Portfolio

In this post we discuss percentiles. Quick reminder, they are 

 refers to a figure out of 100. Therefore 20%, refers to literally 20 out of 100. Factors that are naturally represented using percentages include 

dividend yield, gross margin, change in return-on-equity since last quarter 

 in relation to others but grouped on the 0 to 100 scale 

Imagine you have 700 stocks and want to choose the top 25% based on certain factors. However, the data from your chosen factors all have various ways of being measured, some are measured by percentage such as dividend yield, others are more hard coded like price-to-earnings such as 9.5. 

Choosing percentile simplifies this for you

, and precludes you from having detailed knowledge of the individual metrics and their actual amounts. Instead using percentiles you can just get the top 20% of all stocks or the bottom 20% of all stocks based on the factor of your choice. You let the data do the work for you rather than you entering lots of very nuanced hardcoded values for percentages or multiples.

Compared to hardcoded values, percentiles are a robust metric to compare all stocks against each other today and through history. For example, let us consider a simple investing strategy for the last 12 years that may have invested in companies with a price-to-book less than 0.65 (which was the 10th percentile) back in 2010. 

In the figure below, if we use a hardcoded value, the universe of stocks we can choose from varies wildly from a high of 700 to a low of 100 companies. However using a percentile-based method ensures that we are consistently using the bottom 10% price-to-book companies where the overall number stays very constant. If you noticed some drift upwards from 2020 onwards, this is because the overall number increases even for the percentiles as our universe size has also increased with time. This is for US stocks where our universe has grown by 1000 companies (from 4000 to 5000 stocks) in the last 2 years and thus the number of companies in the percentile metric increased from 400 to 500. 

Let us define percentiles so it is clear how we are using them

Percentiles are simple percentage calculations that tell us the numeric value at which "x" percent of our data is below.

Or with a slightly more mathematic language,

One common use of percentiles is to generate graphs that use the median and the first and third quartile percentiles to visually represent the data. This simply means that the data in question is separated via the 25th, 50th and 75th percentile and is a very quick way to visualise the distribution in percentage terms through time as per the below graph. A user can quickly see the four areas of the graph and how the three quartiles lines help separate the areas.  

Why use percentiles for data outlier control 

One of the most important aspects from a practical, 

, use case for percentiles is in the area of 

. When onboarding either a critical dataset like price and fundamental data, or onboarding a new or alternative dataset, the cleaning, scrubbing and outlier control of that data is crucial. We wrote a detailed post on 

 but a quick illustration of it in action is found in the next graph. 

If we apply our outlier control at the 99.5th percentile for example we get a much more stable and controllable piece of fundamental data to use for our quantitative models. As much as there is some predictive power (alpha) in the tails, that is also where data errors lie so it is prudent to apply this type of data pruning at the data ingestion stage of the pipeline. 

Using percentiles as part of a large stock screener strategy

how can we use all these percentile metrics for a simple stock screening strategy ?

 Most stock screeners out there allow users to hardcode fixed values for various metrics such as P/E or P/B but as we have seen above these are not the most robust methods for cross-sectional analysis through time. Let's start with a toy example of a user choosing 5 metrics that they wish to create a stock screen with. Instead of hardcoding values, a more robust method would be to use percentiles such as 

Payout ratio between 30th and 50th percentile 

In the above, you can see we have also augmented the screens with a fixed dividend yield and also used percentile band for the payout ratio. 

A quick note of caution for users when using percentiles in their own analysis and work. When using percentiles as part of a quantitative investing strategy and judging their efficacy, we must ensure that the percentile is a 

 metric and not calculated on the full-sample and then applied with lookahead historically (you would be surprised how many times this has been seen). A quick code snippet for calculating a daily percentile value across the cross-section of stocks is as follows:

method returns a single value that has lookahead if it was applied historically as it uses all historical data in its calculation. The 

 method calculates a percentile on a per day basis and so only uses information that is available at that 

How are percentiles calculated in python and in which ways and why are percentiles used in quantitative investing strategies.

3. Explaining percentiles and use cases

We won't cover all datasets but we can at least cover the 3 main types of datasets out there that are used by quantitative investment firms. Each of these as we shall see need varying degrees of outlier control. 

Price and volume data by its very nature is very robust and prone to the fewest errors. It is not error free though as no system ever is. There are also most likely additional 3rd party data vendors involved in delivery this data to end users. We cover this is more detail in a separate post, however a quick example of a count of a strange data error that would be need to be scrubbed is given below. 

Datasets from fundamental data providers (balance sheet, income statement and detailed analyst estimate data) contain orders of magnitude more data than simple daily price and volume data. Fundamental balance sheet style data tends to contain upwards of 500 items per company per reporting period. Errors can easily be introduced when that data is ingested by the data provider prior to deliver to quantitative investment firms. 

For detailed analyst estimates data, all analyst updates to all forecasted items for every company with analyst coverage are all stored by the data provider and then distributed to clients. As a result, these datasets quickly grow into enormous datasets with more outliers introduced. Consequently, the data cleaning and outlier control is more involved and a lot more quality assurance (QA) of the data needs to be undertaken by the quantitative team. Errors can occur easily in such circumstances, a few examples of which are: 

Fundamental data provider misread a record

Analyst estimate erroneously entered or format was incorrect (entered as pounds not pence for example)

Corporate actions were not applied correctly 

A simple example of this can be found if we look at dividend yield. Let us assume you want to build a strategy for a yield factor and you either want to use overall dividend yield metric in some form or a growth metric on the dividend yield. Clearly, we better investigate our raw dividend yield data from our data provider. 

What we would like to see is data that aligns with our economic rationale: so a good prior may be that our data should contain maximum dividend yields of maybe 50-60% and mostly these would be poor illiquid companies where their price is driving that metric higher.  

What is clear from the graph is the raw dividend data contains some extremely large and unrealistic dividend yields. A much safer thing to do, than to allow this type of data to feed through to your quantitative model, is to truncate all the outliers to instead equal the 99.9th percentile as we show in the following graph. This gives a much more reasonable and realistic dividend yield to be used in our model. 

Some users may decide to not even truncate their outliers to the 99.9th percentile and instead remove the outliers altogether from their model. This is also a valid method and during the research phase of the model build, both methods can be tested to see if that strategy you built is susceptible to these data outliers. 

Combining outlier control with histograms is probably the single best way to get a visual representation of a data metric and should be the first step in analysing your data. After that, users can then also calculate descriptive statistics from their data to explain its distribution in further detail. An example of this is shown now for 

 (P/E) where we first outlier control extremely large unrealistic values and then   plot the histogram.

We implement our usual truncating technique here (also termed 

) and ensure that our P/E data matrix has the appropriate outlier control. This allows us to then plot the histogram of P/E's for our US stock universe in a controlled and stable manner.  

It turns out that the P/E of this universe of stocks has a median of 19 and a mean of 39, indicating its naturally 

Alternative data (sometimes referred to as 

) has been a data source increasingly utilised by quantitative trading firms. By it's very definition, it represents data from non-standard sources and as such the quality of that data, while it is high quality by some providers, can be of low quality by other providers. Alternative datasets have improved over the years, however they can be less robust than standard datasets. Some of the potential issues for these datasets are:

Point-in-time issues where, especially in the early days, the rigidity needed to ensure fully accurate point-in-time data was not fully understood or could just not be guaranteed by the data provider

Short history so difficult to use for long-term backtests

Overall, alternative data will grow and will be an increasing source of alpha for quantitative strategies going forward. Naturally, a lot of the issues raised above will filter out and be fixed purely by time (historic data is naturally growing) and expertise (they are aware of what quantitative firms criteria are) in the alternative data field.  

How are percentiles are used to control for outliers in your datasets.

4. Controlling for outliers in your data

The efficient frontier is a risk versus return plot that illustrates which portfolio is optimal to hold for your given level of risk. This was inspired by academic work from Harry Markowitz in the 1950s.

A portfolio is just a weighted combination of a set of stocks. By definition, each stock in your portfolio has an expected return and expected risk level associated with it. These expected return and expected risks are key to deciding which stocks you want to hold in your portfolio. The risk and return can be estimated via complex mathematical models and this is what the alpha-generating hedge fund industry focuses on, where alpha refers to future expected returns. 

For this article, what can we do if we do not have access to these sophisticated estimation methods for risk and return ? 

We can use a very simple heuristic which is that we expect future risk and return to equal the past risk and return - so let's go with that model assumption for this article. 

What do we mean when we say Portfolio A dominates Portfolio B ?

The concept of domination relates to the comparison of two portfolios where one portfolio is clearly a better choice than the other. This better choice, assuming A dominates B, will either be based on 

Portfolio A has a greater return than Portfolio B, with both Portfolio A having the same or less risk than portfolio B

Portfolio A has the same return as Portfolio B but it has lower risk 

Mathematically we can write this as a mean-variance condition

Essentially, any portfolio that does not sit on the efficient frontier is dominated by a portfolio that does sit on the efficient frontier.

Choosing a portfolio from a set of N stocks 

There are essentially an infinite number of ways to combine a set of N stocks into a portfolio P if we are allowed to use any weight combinations for our N stocks, no matter how small each weight is. If we take a simple two stock portfolio of stock A and stock B, then the two limits in this case are 100% weight in stock A and 0% in stock B or 100% in stock B and 0% in stock A - but in between these limits we can also invest x% in stock A and y% in stock B. 

It is easy to show a risk-return graph of all the possible portfolios of stock A and B, if we have an assumption for the correlation between stock A and B. To make things simple, we can again suggest that the future correlation between stock A and B is equal to the past correlation. For a specific example, let us take Amazon and Walmart as our two stocks and calculate the average annual return, risk and correlation.   

The average annual return, risk and correlation between Amazon and Walmart, all figures in percentages, are given by:

We can quickly look at the potential portfolios assuming we shift our portfolio in steps from 100% invested in Walmart to 100% invested in Amazon.

Investigating further to test the impact of the correlation parameter, we can vary it from a very unrealistic -70% all the way to 70%. Each point on each line is a potential portfolio for an investor to invest in, under the fixed correlation assumption for that line plot. This hopefully illustrates why a low to negative correlation is so desired from a diversification perspective and allows the investor to obtain superior risk-adjusted returns. 

Clearly the left-most blue line is our preferred portfolio (even though it is based on a non-existent -70% correlation). 

This extreme example was shown to really illustrate the power of negative correlation, or even small positive correlation, and why it is so important at both the stock or portfolio level but also at the strategy level. If an investor can combine one strategy (or portfolio) with another that has low correlation or even negative correlation then the final combined portfolio will offer much better risk-adjusted returns than either of the investments independently.  

Example of the efficient frontier for US stocks 

Let us see a toy example and assume we have a small set of 6 US stocks, covering the technology sector, food and clothing. We can calculate the average annual return and the volatility of each of these stocks. Let us now assume that out of these 6 stocks we want a portfolio of 5 stocks only, then the question is 

What weight should each stock in our portfolio have to ensure we are maximising our return for a given risk level i.e we are on the efficient frontier ?

We show in the table below the returns and risk for each stock where our goal is to obtain the optimal weights i.e the set of weights that lie on the efficient frontier. These sets of weights will be the vector of weights in the last column 

One simple way to do this is to create a large set of random portfolio weights via a Monte Carlo simulation and then calculate the portfolio risk and return. If we plot all of those simulations, which in our case is 10,000 we get a plot like below. For each risk point, we have highlighted the best portfolio return and it is clear to see, a la Markowitz, that there is a frontier where portfolios on that frontier 

Additionally, we have also highlighted with a red start the optimal sharpe ratio portfolio, which is the best risk-adjusted return portfolio a user could obtain theoretically. 

The efficient frontier shows the power of diversification when choosing amongst a portfolio of non-perfectly correlated stocks.

1. Efficient Frontier

The information that most investors are familiar with for stocks is price and volume data. If we go one step further, we can easily obtain company fundamental data such as net income, return on equity, free cashflow and lots more data. This type of financial company data can be used to inform investors decisions about which stock to invest in. 

The question is then: how do we use all this information to figure out why we should invest in stock A, compared to stock B, compared to stock C, ..., compared to stock Z ?

This can be easily achieved if we can combine all these stock-specific datasets together to obtain a final 

 or ranking. This final stock score can then be used to rank all of our potential investable stocks in descending order. Then we can choose the N stocks with the highest scores to form our investable portfolio, where N is whatever size of portfolio the end investor wants.

Now we have all these different datasets to combine, the nuance is how to both clean that data and also standardize and normalize the datasets so stocks in each dataset can be compared against each other efficiently. This may involve taking price, volume and fundamental data and then fitting some linear model to the data. The final stage would be then to transform that linear model into a final ranking or distribution for each stock.  

Terminology-check: Standardization versus Normalization

For anyone who has worked with data, one of the biggest issue is to ensure the data is cleaned and scrubbed and outliers removed. This ensures that the data passed to our financial models is as robust and error-free as we can make it, so our models can make more accurate predictions. The data cleaning and outlier detection then naturally leads to the question 

How do I standardize my data or normalize my data to then detect outliers and remove them to clean my data ?

The standardized or normalized data is a form of 

. We are taking data in raw form and modifying the representation of that data, as we will see below with some examples. Once the data has been modified, it can be interpreted more easily but the overall properties of the data should remain intact. This means that data which was an outlier in its raw form, will still be an outlier in the new transformed data but the hope is that it will be easier for us to identify those outliers.  

 are used interchangeably by some people in the data science or investing industries. This can lead to some confusion (or misinterpretation) at best or misusing one method in place of the other at worst. It is best shown via a simple example. 

The easiest and probably the most commonly used standardization methods is to squash the data inside the minimum-maximum range. In formula this is given by

This essentially transforms your data into a [0,1] range, which we can think of as a form of percentage so 0% to 100%. This is a standardization technique for raw data where all raw data (of different types) is transformed into a 

. This technique can also be referred to as 

. However, this is not really normalization in the sense of data normalization or a normal distribution, as we try to explain in the next section. 

Normalization in the statistics, data science or mathematical professions refers to the 

, which means the distribution of the data can be explained by two parameters - the mean and standard deviation. The standard normal random variable is one of the most common distributions that quantitative researchers use and is a special case of the normal distribution - with a mean of zero and standard deviation of one.  

One method where normalization is frequently used is to 

, also referred to as a standard score, which is given by the following formula where mu is the mean or average of the data and sigma the volatility

The z-score subtracts the average value from each data point and then normalizes the data by dividing by the standard deviation. If the distribution of the raw data is a normal distribution then calculating a z-score turns the data into a standard normal random distribution. The z-score tells the user how many standard deviations away from the average value your data point is. We can illustrate a comparison between the standard normal distribution and other normal distributions with different parameter to visually show their impact

standard normal distribution, mean = 0 and standard deviation = 1  

normal distribution, mean = 0 and standard deviation = 2  

normal distribution, mean = 0 and standard deviation = 3  

normal distribution, mean = 2 and standard deviation = 1  

It is hopefully clear the impact of the larger standard deviation in how it creates fatter tails with more change of data coming from those extreme events. The last example with a positive mean of two instead of zero is used to illustrate the 

of the distribution but leaving the shape unchanged. 

If anyone wants to test and play with some data, here is some example python code to illustrate normal distributions by varying parameters mu and sigma, the mean and standard deviation.

How to normalize a quantitative trading signal to ensure you distill all information down into one final number for every stock.


5. Normalization Methods for Financial Data

Start your deep dive into the data platform, exploring concepts, datasets and python code.  

Enterprise

Personal

Quant data enthusiasts

Correlation within your Stock Portfolio

Explaining percentiles and use cases

Controlling for outliers in your data

Efficient Frontier

Normalization Methods for Financial Data