Mean-Variance Optimization in Practice: Reverse Optimization and Implied Expected Returns

The fact that mean-variance optimizers are highly sensitive to changes in expected returns […] is well known in investment practice1, with a couple of practical solutions already described in this blog, for example using near efficient portfolios or subset resampling-based efficient portfolios. In this blog post, I will introduce another approach originally described in Sharpe2 and known as reverse optimization3, which consists in trying to improve the robustness of expected returns estimates by turning the mean-variance optimization problem around. In more detail, instead of optimizing portfolio weights using a set of expected returns and a covariance matrix4, reverse optimization starts with the weights of a given portfolio5 and solves for the corresponding set of [implied] expected returns3 that makes it mean–variance efficient. That approach, illustrated in Figure 1 taken from Two Sigma6, has been popularized by the Black-Litterman methodology and has empirically been shown to improve out-of-sample performance of portfolios7, c.f. for example Ardia and Boudt4 or Ni et al.7. Figure 1. Differences between traditional and reverse mean-variance optimization. Source: Two Sigma. As a main example of usage, I will show how reverse optimization allows to analyze the hidden assumption made by index investors in the MSCI World w.r.t. future U.S. over ex-U.S. equities performance. Reverse mean-variance optimization and implied expected returns Let be: $n$, the number of assets in a universe of assets $\mu \in \mathbb{R}^{n}$, the vector of expected asset arithmetic returns at a given holding horizon (one day, one month…) $\Sigma \in \mathcal{M}(\mathbb{R}^{n \times n})$, the covariance matrix of the asset arithmetic returns at the holding horizon $\lambda$, 0 < \lambda < +\infty, a risk tolerance8 parameter Under the Markowitz’s mean-variance framework, a mean–variance efficient portfolio is defined as a vector of portfolio weights $w^* \in \mathbb{R}^{n}$ that maximizes the mean–variance utility function [w^* = \operatorname{argmax} \mu {}^t w - \frac{\lambda}{2} w {}^t \Sigma w \newline \textrm{s.t. } w \in C] , where $C \subset \mathbb{R}^{n}$ is the set of investment constraints on the portfolio weights (budget constraint, no short-sale constraint, sector exposure constraints…). Reverse unconstrained mean-variance optimization When no constraints are imposed on the mean-variance optimization problem or when no linear constraint is binding at optimality12, the first-order optimality condition associated to the mean–variance utility function maximisation problem above is [\frac{\partial \mathcal{L}}{\partial w} = \mu - \lambda \Sigma w = 0] , where $\mathcal{L}(w) = \mu {}^t w - \frac{\lambda}{2} w {}^t \Sigma w$ is the Lagrangian of the mean–variance utility function maximisation problem. Now, assuming that the solution to this problem $w^* \in \mathbb{R}^{n}$ is known but $\mu$ is not, it is then possible to isolate $\mu$, which leads to the classical1 definition of the vector of implied expected returns $\mu_{impl} \in \mathbb{R}^{n}$ as [\mu_{impl} = \lambda \Sigma w^] Ardia and Boudt4 comments that definition as follows: We thus see that the implied expected return for [a] stock is proportional to the covariance between the stock’s return and the return on the mean–variance efficient portfolio. Ceteris paribus, this covariance will tend to be higher for stocks that have: (i) a relatively larger weight in the efficient portfolio proxy, (ii) a high volatility, and (iii) a high correlation with the stocks in the efficient portfolio. The propagation of the weights bets to implied returns bets thus interacts with the variance and correlation properties of the individual stock returns. Reverse budget-constrained mean-variance optimization Herold1 notes that in the literature, implied returns are regularly computed according to [the relationship established in the previous sub-section, which] will often result in unreasonable values1. Indeed, the assumption that no constraints are imposed on the mean-variance optimization problem or that no constraints are binding at optimality is not realistic, because in practice the budget constraint is relevant to the investor1. So, let be $C = \{ w \in \mathbb{R}^{n}, \sum_{i=1}^n w_i = 1 \}$ representing the budget constraint. The Lagrangian of the mean–variance utility function maximisation problem then becomes19 [\mathcal{L}(w) = \mu {}^t w - \frac{\lambda}{2} w {}^t \Sigma w - \gamma \left( w {}^t 1_n - 1 \right)] , where: $\gamma \in \mathbb{R}$ is the Lagrange multiplier associated to the budget constraint $\sum_{i=1}^n w_i = 1$ $1_n$ is a $n$-dimentional vector of 1s and the first-order optimality condition becomes19 [\frac{\partial \mathcal{L}}{\partial w} = \mu - \lambda \Sigma w - \gamma 1_n = 0] Consequently, and again assuming that the solution to this problem $w^*$ is known but $\mu$ is not, the vector of implied expected returns $\mu_{impl}$ turns into a two-parameter family that depends on a […] multiplicative factor and an additive component1: [\mu_{impl} = \lambda \Sigma w^ + \gamma 1_n] Herold1 argues10 that this definition - although involving an additional parameter $\gamma$ - results in sensible and more realistic values for the implied returns1 than the common procedure of determining the multiplier only and implicitly setting the additive constant to zero1. As a side note, under the additional assumption that the marginal investor’s objective is to maximize risk-adjusted returns (or Sharpe Ratio)6, it can be shown11 that $\gamma$ can be set to 0, which justifies the widespread use of the definition of $\mu_{impl}$ established in the previous sub-section. Reverse linearly-constrained mean-variance optimization Beyond the budget constraint analyzed in the previous sub-section, the standard mean-variance portfolio optimization problem originally introduced in Markowitz12 involves general linear constraints on the portfolio weights $w$ modelled through the set $C = \{ w \in \mathbb{R}^{n}, Ax = b, Cx \leq d \}$, where: $A \in \mathcal{M}(\mathbb{R}^{n_e \times n})$ and $b \in \mathbb{R}^{n_e}$ represent linear equality constraints, like the budget constraint $\sum_{i=1}^n w_i = 1$ $C \in \mathcal{M}(\mathbb{R}^{n_i \times n})$ and $d \in \mathbb{R}^{n_i}$ represent linear inequality constraints like nonnegativity constraints13 $w_i \geq 0, i=1..n$ or sector constraints While that formulation is apparently much more complex than that of the previous sub-section, Sharpe2 actually notes that the solution, once determined, can he characterized as the solution to a [linear equality-constrained] problem2, so that only the case $C = \{ w \in \mathbb{R}^{n}, Ax = b \}$ needs to be analyzed. As Sharpe2 puts it: Inequality constraints not binding in the solution are simply omitted from the expression of the problem; those that were binding are included as equalities. Thus, assuming that the solution to this problem $w^*$ is known14, the Lagrangian of the mean–variance utility function maximisation problem becomes2 [\mathcal{L}(w) = \mu {}^t w - \frac{\lambda}{2} w {}^t \Sigma w - \langle {\Gamma, A' w - b'} \rangle] , where: $n_b \geq 0$ represents the number of binding inequality and bound constraints at optimality turned into linear equality constraints $A’ \in \mathcal{M}(\mathbb{R}^{(n_e + n_b) \times n})$ and $b \in \mathbb{R}^{n_e + n_b}$ represent the original linear equality constraints $(A,b)$ plus the binding inequality and bound constraints at optimality turned into linear equality constraints $\Gamma \in \mathbb{R}^{n_e + n_b}$ are the Lagrange multipliers associated to the $n_e + n_b$ equality constraints and the first-order optimality condition becomes2 [\frac{\partial \mathcal{L}}{\partial w} = \mu - \lambda \Sigma w - A' {}^t \Gamma = 0] This time, the vector of implied expected returns $\mu_{impl}$ turns into a $n_e + n_b + 1$-parameter family that depends on a multiplicative factor and several additive components: [\mu_{impl} = \lambda \Sigma w^* + A' {}^t \Gamma] One important consequence of that result is that for a budget and nonnegativity-constrained mean-variance optimization problem: If there are no asset with a null weight in the solution $w^*$, then the definition of $\mu_{impl}$ established in the previous sub-section remains valid15. Otherwise, the definition of $\mu_{impl}$ established in the previous sub-section provides an upper bound for the implied expected return of any asset whose weight is null2. Reverse non-linearly-constrained mean-variance optimization Non-linear constraints imposed on the portfolio weights (tracking error constraints, turnover constraints, cardinality constraints…) are out of scope of this blog post. Nevertheles, the interested reader is refered to Bertsimas et al.16, which shows that reverse optimization can be characterized as an inverse optimization problem, with applications to the Black-Litterman model. Reverse optimization in practice From the previous section, using reverse optimization within a universe of assets requires: The vector of a reference mean–variance efficient portfolio weights $w^*$ An estimate of the covariance matrix $\Sigma$ of the asset arithmetic returns at a given holding horizon An estimate of the risk tolerance \lambda > 0 An estimate of the Lagrange multiplier $\gamma$ associated to the budget constraint or an estimate of the Lagrange multipliers $\Gamma$ associated to the equality constraints binding at optimality for $w^*$ Unfortunately, most of these quantities cannot be directly observed and so [it is needed to] resort to estimating them instead17. Choosing a reference mean–variance efficient portfolio $w^*$ Ardia and Boudt4 notes that traditionally, the standard choice [for a reference mean-variance efficient portfolio] has been to use the market capitalization weighted portfolio4 because of the result that under the capital asset pricing model (CAPM) the market capitalization portfolio is mean–variance efficient4. Nevertheless, for almost all [investors], the investable universe does not correspond to all tradable assets and hence, the CAPM conditions [under which that portfolio is mean–variance efficient] are by construction not verified4. As a consequence, Ardia and Boudt4 argues that the market capitalization weighted portfolio is only one possible proxy for a mean–variance efficient portfolio and that other proxies may lead to more accurate [implied] expected returns2. Al-Thani et al.3 even goes further and states that the most neutral and economically defensible starting point for reverse optimization and implied-return extraction […] should be chosen from [a risk-based portfolio]3. Examples of such risk-based portfolios - some empirically demonstrated in Ardia and Boudt4 to provide a small forecast accuracy gain […] compared with the market capitalization weighted portfolio4 - are the following: The equally weighted portfolio The inverse volatility weighted portfolio The equal-risk-contribution portfolio The maximum diversification portfolio The minimum variance portfolio The conviction-parity portfolio One practical remark about these alternative reference mean–variance efficient portfolios is that their asset weights $w^*$ might not be easily observable (unlike the market capitalization portfolio) nor computable (unlike the risk-based portfolios above). In such a case, and provided historical portfolio returns are available, it should be possible to infer $w^*$ through an index tracking optimization procedure. Estimating the asset covariance matrix $\Sigma$ As noted in Shumway et al.17, the covariance matrix of returns can be reasonably estimated using historical return data17 because risk tends to be strongly persistent through time6. This is what is done for example in Herold1, in Ardia and Boudt4 or in Bevan and Winkelmann18. Depending on the choice of the mean–variance efficient portfolio, though, the primary goal when estimating the asset covariance matrix might not be to find the best estimate of the true covariance matrix17 but simply to find the best estimate of [whatever} estimate of the covariance matrix17 was originally used to produce that portfolio. In other words, in the specific context of reverse optimization, the problem of estimating the asset covariance matrix is slightly different from the problem of forecasting the asset covariance matrix for the next holding period19. Estimating the risk tolerance parameter $\lambda$ The risk tolerance parameter $\lambda$ represents the magnitude of the trade-off between expected return and variance2 and acts as a scaling factor for the reverse optimization estimate of [implied expected] returns20. Two Sigma6 explains this as follows: Crucially, the implied returns that come from reverse optimization are only relative return levels - not absolute return expectations. That is, reverse optimization may tell us that investors expect twice the return from [asset] A as from [asset] B, but provides no guidance on whether those expected returns are 2% versus 1% annually or 20% versus 10%. In practice, $\lambda$ can be estimated by several different methods: Using a sensible fixed value, or a range of sensible fixed values, like in Ardia and Boudt4 which uses $\lambda \in \{1, 2.4, 5 \}$. Using the risk premium of the chosen reference mean–variance efficient portfolio, through the formula1 [\lambda = \frac{\mu_p}{\sigma_p^2}] , where: $\mu_p$ is the risk premium of the chosen reference mean–variance efficient portfolio, that is, the expected portfolio return over the risk free rate21 $\sigma_p$ is the standard deviation of the chosen reference mean–variance efficient portfolio returns over the risk free rate A couple of remarks: The portfolio risk premium $\mu_p$ and the portfolio standard deviation $\sigma_p$ might both be estimated from a set of historical portfolio returns. In that case, those historical returns must be arithmetic returns so as to make $\lambda$ invariant w.r.t. time, since both numerator and denominator [are then] linear in time22. The portfolio risk premium $\mu_p$ might be estimated from a set of historical portfolio returns and the portfolio standard deviation might be estimated through the formula $\sigma_p = \sqrt{ w^* {}^t \Sigma w^*}$. Same remark as above, since $\Sigma$ is supposed to be the covariance matrix of the asset arithmetic returns. In addition, for internal consistency, $\Sigma$ should then be the covariance matrix of the asset arithmetic excess returns over the risk free rate23. In both cases, due to the constraint \lambda > 0, the estimated portfolio risk premium $\mu_p$ must be stritly positive for the estimation of the risk tolerance parameter to make sense. This might not always be possible, depending on the exact set of historical portfolio returns used. In both cases also, it is important to keep in mind that one concern when inferring the risk tolerance parameter from market data is the extent of the sampling variation24. Using the Shape Ratio of the chosen reference mean–variance efficient portfolio, through the formula18 [\lambda = \frac{ \text{SR}_p }{\sigma_p}] , where: $\text{SR}_p$ is the Sharpe Ratio of the chosen reference mean–variance efficient portfolio $\sigma_p = \sqrt{ w^* {}^t \Sigma w^*}$ is the standard deviation of the chosen reference mean–variance efficient portfolio returns over the risk free rate Three remarks: The periodicity of the Sharpe Ratio must be consistent with the holding horizon of the asset covariance matrix $\Sigma$. For example, using an annualized Sharpe Ratio and an asset covariance matrix at a daily horizon would not be consistent… For internal consistency, $\Sigma$ should be the covariance matrix of the asset arithmetic excess returns over the risk free rate23. Examples of annualized Sharpe Ratios used in the litterature are ranging from 0.20-0.30125 to 1.018. Ultimately, this depends on the chosen reference mean–variance efficient portfolio as well as on the holding horizon. Using the target expected return of a reference asset. It is not uncommon626, especially when computing long-term capital market assumptions (CMAs), to set the risk tolerance parameter to a value that makes the implied expected return of an asset equal to a prespecified value1. When there is no Lagrange multiplier to consider, this is simply done through the formula [\lambda = \frac{ \left( \Sigma w^* \right)i}{ \mu{target,i} }] , where $1 \leq i \leq n$ is the index of the reference asset whose target expected return $\mu_{target,i}$ is known. A couple of examples: When the universe of assets contains an equity index, it is possible to use a27 forecast of the equity risk premium - to which the risk free rate must be added - as the target expected return. Such a forecast is readily available on Aswath Damodaran’s website, who maintains estimates of the historical implied equity risk premiums for different countries with plenty of details in his yearly-updated paper Equity Risk Premiums (ERP): Determinants, Estimation and Implications28. Such a forecast can also be directly extracted from the equity index historical returns data, c.f. the three procedures in Merton29. When the universe of assets contains government bonds, it is possible to use the27 forecast produced by the Bogle model for bonds as the target expected return. When the universe of assets contains a riskless asset with a non-null weight in $w^*$, it is possible to use the risk free rate as the target expected return, c.f. Sharpe2. One general remark: The periodicity of the target implied expected return $\mu_{target,i}$ must be consistent with the holding horizon of the asset covariance matrix $\Sigma$. Estimating the Lagrange multiplier $\gamma$ Like the risk tolerance parameter, the Lagrange multiplier $\gamma$ associated to the budget constraint can be estimated by several different methods30: If $\lambda$ is already known Using the risk premium $\mu_p$ of the chosen reference mean–variance efficient portfolio, through the formula [\gamma = \mu_p - \lambda \sigma_p^2] Using the Shape Ratio $\text{SR}_p$ of the chosen reference mean–variance efficient portfolio, through the formula [\gamma = \text{SR}p \sigma_p - \lambda \sigma_p^2] To be noted that if $\lambda$ has been estimated using the same information as for $\gamma$, the formulas above will lead to $\gamma = 0$. In other words, two independent pieces of information are required in order to properly estimate $\gamma$. If $\lambda$ is not already known, using the target expected returns or risk premiums of two reference assets to simultaneously determine the two parameters $\lambda$ and $\gamma$. That method is proposed in Herold1 and consists in solving the following system of linear equations [\begin{cases} \mu{target,i} = \lambda \left( \Sigma w^* \right)i + \gamma \ \mu{target,j} = \lambda \left( \Sigma w^* \right)_j + \gamma \end{cases}] , where where $1 \leq i \leq n$ and $1 \leq j \leq n$, $j \ne i$, are the indexes of the two reference assets whose target expected returns $\mu_{target,i}$ $\mu_{target,j}$ are known. Estimating the Lagrange multipliers $\Gamma$ In case of general linear constraints, the estimation of several Lagrange multipliers $\Gamma$ will require yet additional independent pieces of information. Unfortunately, assuming that these pieces of information are available might partially defeat the purpose of finding implied returns31, which explains why practical guidelines for general linear constraints are difficult to find in the litterature. What can be said, though, is that Herold’s method1 is particularly suited to this case. Caveats In the previous sub-sections, several different procedures to estimate the risk tolerance parameter $\lambda$ and the Lagrange multiplier $\gamma$ have been described. Best and Grauer9 shows that these procedures lead to differences a