Hierarchical Risk Parity in Python: A Robust Alternative to Mean-Variance Optimization

Traditional mean-variance optimization is elegant on paper. Give it expected returns and a covariance matrix, and it returns the portfolio with the highest expected Sharpe ratio for a given risk budget. The problem is that those inputs are estimates. When they are wrong—especially when they are noisy—the optimizer can produce extreme, unstable weights that look brilliant in-sample and fail out-of-sample.

##The In-Sample vs. Out-of-Sample Trap in Portfolio Optimization

In-sample performance is a poor guide to future results. An optimizer can exploit historical noise, concentrating capital in assets that happened to have high returns and low correlations during the estimation window. That is not a forecast; it is curve-fitting. The walk-forward test is the only backtest that matters because it repeatedly re-estimates the model on past data and evaluates it on unseen future data. HRP is designed with this reality in mind.

Why Matrix Inversion Destabilises Mean-Variance Portfolios

Mean-variance optimization requires the inverse of the covariance matrix. When assets are highly correlated, the covariance matrix becomes ill-conditioned. Small changes in estimated correlations can cause large changes in the inverse, and therefore large changes in portfolio weights. Traditional risk parity methods can suffer from the same problem when they solve for equal risk contributions using the full covariance matrix. In high-correlation regimes, the inversion step becomes a source of instability rather than a source of insight.

Hierarchical Risk Parity (HRP): A Robust Portfolio Construction Method

Hierarchical Risk Parity (HRP) was introduced by Marcos López de Prado in 2016. It replaces matrix inversion with hierarchical clustering, using only the covariance matrix to build a tree of assets and allocate risk top-down. HRP is not a return-forecasting model. It is a portfolio construction method that tries to make the allocation process more robust.

Origins in Graph Theory and Machine Learning

HRP borrows from graph theory and machine learning. Assets are treated as nodes in a graph, and correlations are converted into distances. Hierarchical clustering then groups similar assets into clusters. The result is a dendrogram—a tree that shows which assets are most alike. This structure allows the allocator to diversify across clusters rather than across individual assets alone.

The Three Concerns Addressed: Instability, Concentration, and Underperformance

De Prado frames HRP as a response to three common flaws of quadratic optimizers:

Instability: Small input changes can produce large weight changes.
Concentration: Optimizers often pile into a few assets or clusters.
Out-of-sample underperformance: In-sample optimality rarely translates into future performance.

HRP addresses these by avoiding covariance inversion, by allocating risk across the cluster tree, and by focusing on covariance structure rather than noisy return forecasts.

How HRP Works: Tree Clustering, Quasi-Diagonalisation, and Recursive Bisection

The HRP algorithm has three main stages: tree clustering, quasi-diagonalisation, and recursive bisection. Each stage is mechanical and transparent.

Step 1: Computing the Distance Matrix from Correlations

The first step is to transform the correlation matrix into a distance matrix. A common formula is:

d_{i,j} = sqrt(0.5 * (1 - rho_{i,j}))

where rho_{i,j} is the correlation between assets i and j. This distance is zero when two assets are perfectly correlated, larger when they are uncorrelated, and largest when they are negatively correlated. The result is a proper metric that can be used by clustering algorithms.

Step 2: Hierarchical Clustering with `scipy.cluster.hierarchy`

Using the condensed distance matrix, scipy.cluster.hierarchy.linkage builds a hierarchical clustering. Single linkage is a common choice because it tends to form long chains and reveals the connectedness of assets. The output is a linkage matrix that describes how assets are merged into clusters. A dendrogram can visualise this structure.

Step 3: Quasi-Diagonalisation (Matrix Reordering)

Once the tree is built, HRP reorders the covariance matrix so that similar assets are adjacent. This is called quasi-diagonalisation. In Python, scipy.cluster.hierarchy.leaves_list returns the order of assets at the leaves of the dendrogram. The covariance matrix is then reordered according to that sequence. Large correlations cluster near the diagonal, making the block structure visible.

Step 4: Recursive Bisection and Inverse-Variance Weighting

Finally, HRP allocates capital top-down. The ordered list of assets is split into two halves. The variance of each half is computed using inverse-variance weights within that cluster. Capital is then allocated between the two halves inversely to their variance: the lower-variance cluster receives more weight. This process repeats recursively within each half until each asset has a final weight. No matrix inversion is required.

Implementing Hierarchical Risk Parity in Python: A Step-by-Step Guide

The following implementation uses only numpy, pandas, and scipy. It is designed to be readable and easy to adapt.

Loading and Preparing Financial Time Series Data

Assume you have a pandas.DataFrame of asset returns, with dates as the index and assets as columns. For example:

import pandas as pd

# prices: DataFrame of adjusted close prices, columns are assets
returns = prices.pct_change().dropna()

Returns should be aligned and free of missing values. For cross-sectional or walk-forward work, you may compute returns within each period and then pass the resulting matrix to the allocator.

Building the Correlation and Covariance Matrices

corr = returns.corr()
cov = returns.cov()

The correlation matrix is used for clustering. The covariance matrix is used for risk allocation.

Generating Linkage Matrices and Dendrograms

import numpy as np
from scipy.cluster.hierarchy import linkage, dendrogram, leaves_list
from scipy.spatial.distance import squareform

def correlation_distance(corr):
    return np.sqrt(np.clip((1.0 - corr) / 2.0, 0.0, 1.0))

def build_linkage(corr, method='single'):
    dist = correlation_distance(corr)
    condensed = squareform(dist, checks=False)
    return linkage(condensed, method=method)

def quasi_diagonal_order(link):
    return list(leaves_list(link))

To visualise the tree:

import matplotlib.pyplot as plt

link = build_linkage(corr)
dendrogram(link, labels=list(corr.columns))
plt.show()

Translating the Algorithm into Production-Ready Code

The core HRP functions can be encapsulated in a class. This mirrors the way AlphaNova participants structure logic in a single Predictor class: pure Python, self-contained, and easy to test locally.

class HRPAllocator:
    def __init__(self, linkage_method='single'):
        self.linkage_method = linkage_method

    def fit(self, returns):
        self.corr_ = returns.corr()
        self.cov_ = returns.cov()
        self.link_ = build_linkage(self.corr_, method=self.linkage_method)
        self.sort_ix_ = quasi_diagonal_order(self.link_)
        self.weights_ = self._recursive_bisection(self.cov_, self.sort_ix_)
        return self

    def _cluster_variance(self, cov, cluster_items):
        cov_sub = cov.loc[cluster_items, cluster_items]
        inv_var = 1.0 / np.diag(cov_sub)
        inv_var /= inv_var.sum()
        w = inv_var.reshape(-1, 1)
        return float(w.T @ cov_sub @ w)

    def _recursive_bisection(self, cov, sort_ix):
        weights = pd.Series(1.0, index=sort_ix)
        cluster_items = [sort_ix]

        while cluster_items:
            next_items = []
            for items in cluster_items:
                if len(items) > 1:
                    mid = len(items) // 2
                    next_items.append(items[:mid])
                    next_items.append(items[mid:])
            cluster_items = next_items

            for i in range(0, len(cluster_items), 2):
                left = cluster_items[i]
                right = cluster_items[i + 1]
                var_left = self._cluster_variance(cov, left)
                var_right = self._cluster_variance(cov, right)
                alpha = 1.0 - var_left / (var_left + var_right)
                weights[left] *= alpha
                weights[right] *= 1.0 - alpha

        return weights

    def weights(self):
        return self.weights_

Usage:

allocator = HRPAllocator(linkage_method='single').fit(returns)
weights = allocator.weights()

This code avoids matrix inversion entirely. It can handle a covariance matrix that is ill-conditioned or even singular.

HRP in Practice: Handling Ill-Conditioned and Singular Matrices

Quadratic optimizers require a positive-definite covariance matrix. When assets are highly correlated, or when the number of assets is large relative to the number of observations, the covariance matrix may be ill-conditioned or singular. In those cases, matrix inversion fails or produces unstable results. HRP sidesteps this problem because it never inverts the covariance matrix.

Portfolios with Highly Correlated Assets

In markets, correlations often rise during stress. Assets that looked independent can become highly correlated. A mean-variance optimizer may respond by taking extreme long or short positions. HRP, by contrast, groups correlated assets into the same cluster and allocates risk across clusters. This tends to produce more diversified weights and less turnover.

The Advantage of Skipping Covariance Inversion

The ability to work with an ill-degenerated or singular covariance matrix is a major technical advantage. Real-world datasets—especially in walk-forward cross-sectional challenges where the asset universe changes over time—can easily produce such matrices. HRP remains computable where quadratic optimizers break down.

From Portfolio Construction to Signal Ranking: Lessons for AlphaNova Competitions

AlphaNova hosts walk-forward, cross-sectional signal forecasting competitions. Participants receive obfuscated financial data, submit a pure Python Predictor class, and are evaluated out-of-sample using the Sharpe ratio. A greedy quality selection process admits only genuinely uncorrelated, overfit-filtered signals. This structure rewards robust, uncorrelated alpha.

Overfitting Resistance in Walk-Forward Analysis

HRP's philosophy aligns with AlphaNova's emphasis on out-of-sample testing. Both avoid relying on a single historical fit. The walk-forward test repeatedly re-estimates and evaluates, which helps filter out strategies that only worked in the past. HRP's resistance to concentration and instability is a portfolio-level analogue of that discipline.

Why Robust Allocation Complements Pure Alpha Signals

A strong alpha signal still needs a robust allocation layer. If a signal is concentrated in a few correlated assets, its realized Sharpe ratio can suffer. HRP can be used to allocate across assets or across signals in a way that respects covariance structure. AlphaNova's use of geometric fingerprinting to measure signal uniqueness—described in From Signals to Cities: Compression and the Geometry of Novelty—echoes this idea: uncorrelated building blocks are more valuable than redundant ones.

Evaluating HRP Portfolios: Monte Carlo Evidence vs. CLA and Risk Parity

De Prado's Monte Carlo experiments compared HRP with the Critical Line Algorithm (CLA) and traditional Risk Parity methods. The findings were notable: HRP delivered lower out-of-sample variance than CLA and traditional Risk Parity, even though minimum-variance is CLA's explicit objective.

Monte Carlo Evidence: HRP vs. Minimum-Variance and Risk Parity

The experiments suggest that avoiding return forecasts and focusing on the covariance structure can improve robustness. HRP does not try to find the mathematically optimal in-sample portfolio. Instead, it builds a diversified allocation that is less sensitive to estimation error.

Interpreting Variance Reduction Without Return Forecasting

HRP makes no return forecasts. It only uses the covariance matrix to allocate risk. This is a deliberate trade-off: by dropping return forecasts, HRP avoids a major source of overfitting. For a practical guide to evaluating the resulting return stream, see How to Compute the Sharpe Ratio from a Pandas Series of Returns. The Sharpe ratio remains the key metric in AlphaNova competitions, and robust allocation can help a signal's out-of-sample risk-adjusted performance.

Integrating HRP into a Quantitative Research Workflow

HRP is not a stand-alone trading strategy. It is a portfolio construction tool that can sit inside a broader signal generation pipeline. A clean research workflow makes it easier to test, iterate, and deploy.

Pure Python, Reproducible Environments, and Local Testing

AlphaNova's workflow is a good example. Participants write a single Predictor class in pure Python. A local runner is provided for testing before submission. This encourages reproducible environments and rapid iteration. You can embed HRP logic inside your Predictor to translate signals into portfolio weights, then test the full pipeline locally.

Owning Your Intellectual Property

AlphaNova competitions are free to enter. Participants retain full intellectual property ownership. That matters for researchers who want to build on their work. You can experiment with HRP, refine your implementation, and keep the resulting IP while competing for cash prizes.

Conclusion: Advancing Portfolio Construction with Machine Learning Techniques

HRP applies modern mathematics to an old problem. By using hierarchical clustering and recursive bisection instead of matrix inversion, it avoids a major source of instability in traditional portfolio optimization. The result is a robust allocation method that can handle highly correlated assets and ill-conditioned covariance matrices. Platforms like AlphaNova aim to democratise access to institutional-grade quantitative research, enabling data scientists and quants to test ideas like HRP in high-quality, free-to-enter competitions where performance alone determines earnings.

Next Steps

If you want to apply these principles, start by implementing the HRP functions above on a dataset you understand. Test the allocator across different linkage methods and look at the stability of the resulting weights. Then consider how HRP might complement your own alpha signals in a walk-forward, cross-sectional workflow.

How to Implement Hierarchical Risk Parity (HRP) Clustering in Python