Factor Covariance Matrix Forecasts

Factor Covariance Matrix Forecasts#

In this tutorial we are going to create several covariance matrices of factor returns. This covariance matrix follows from a standard factor model. We will also tie out the numbers against a pandas/numpy-based reimplementation. More specifically, we will:

  • Create a basic risk model.

  • Extract the factor returns.

  • Use the covariance matrix forecast report to compute the covariance matrix.

  • Use pandas to extract the factor volatility and correlation matrix time-series.

  • Replicate the volatility forecast.

  • Replicate the correlation forecast.

Throughout this notebook we work with a randomly generated dataset. The results should generalize to real data, but for legal reasons we do not show any real data on our public API. Bayesline clients can run this notebook on real data.

Imports & Setup#

For this tutorial notebook, you will need to import the following packages.

import pandas as pd
import polars as pl
import numpy as np

from bayesline.api.equity import (
    FactorCovarianceReportSettings,
    ExposureSettings,
    CategoricalExposureGroupSettings,
    ContinuousExposureGroupSettings,
    FactorRiskModelSettings,
    ModelConstructionSettings,
    UniverseSettings,
)
from bayesline.apiclient import BayeslineApiClient

We will also need to have a Bayesline API client configured.

bln = BayeslineApiClient.new_client(
    endpoint="https://[ENDPOINT]",
    api_key="[API-KEY]",
)

Creating the covariance matrix forecasts#

Let’s first set up a basic risk model and use it to generate the forecasts. We choose to run with mostly default settings. The steps involved are:

  1. Creating the settings of the risk model.

  2. Loading the report model engine.

  3. Running the engine to generate the covariance report.

The first step is creating the risk model settings.

factorriskmodel_settings = FactorRiskModelSettings(
    universe=UniverseSettings(),
    exposures=ExposureSettings(
        exposures=[
            ContinuousExposureGroupSettings(hierarchy="market"),
            CategoricalExposureGroupSettings(hierarchy="trbc"),
            ContinuousExposureGroupSettings(hierarchy="style"),
        ]
    ),
    modelconstruction=ModelConstructionSettings(
        estimation_universe=None,
        zero_sum_constraints={"trbc": "mcap_weighted"},
    ),
)

Next, we create the report engine from the report settings. We run with the defaults here.

report_settings = FactorCovarianceReportSettings(
    factor_model_settings=factorriskmodel_settings
)
report_engine = bln.equity.reports.load(
    report_settings.with_dataset("bayesline/Bayesline-US-All-1y")
)

Let’s see what these settings really are by printing them out.

print(report_settings.factor_cov_settings.model_dump_json(indent=2))

{
  "halflife_vol": 60.0,
  "halflife_cor": 120.0,
  "halflife_vra": null,
  "nw_lags_vol": 0,
  "nw_lags_vol_halflife_override": null,
  "nw_lags_cor": 0,
  "shrink_cor_method": null,
  "shrink_cor_length": null,
  "combine_standardized": false
}

The different settings that jointly make up the covariance matrix are:

  1. halflife_factor_vol The halflife of the factor volatility. The default is a 60-day halflife.

  2. halflife_factor_vra The halflife of the cross-sectional factor volatility adjustment. The default is to not do any adjustment.

  3. halflife_factor_cor The halflife of the factor correlation. The default is a 120-day halflife.

  4. nw_lags_factor_vol The overlap or Newey-West lags to incluce on the factor volatility forecast. The default is zero, meaning no autocorrelation correction is performed.

  5. nw_lags_factor_cor The overlap or Newey-West lags to incluce on the factor correlation forecast. The default is zero, meaning no autocorrelation correction is performed.

Now we get the actual time-series of the covariance matrices.

# generate the report data
report = report_engine.calculate(start_date=None, end_date=None)

# get the covariance matrix (and skip the first date)
df_report = report.get_covariance().filter(pl.col("date") > pl.col("date").min())

# convert to pandas
df_vcov = df_report.to_pandas().set_index(["date", "factor"]).rename(columns=lambda c: c.split("^")[0])
df_vcov
Academic & Educational Services Basic Materials Consumer Cyclicals Consumer Non-Cyclicals Dividend Energy Financials Government Activity Growth Healthcare ... Institutions, Associations & Organizations Leverage Market Momentum Real Estate Size Technology Utilities Value Volatility
date factor
2025-04-01 Academic & Educational Services 0.028461 -0.001975 0.007310 0.003482 -0.003814 0.013186 0.003410 -0.008398 0.004290 -0.061013 ... 0.018185 -0.000225 0.001207 0.004089 -0.003952 0.001755 0.008957 0.006069 -0.003784 0.006459
Basic Materials -0.001975 0.000137 -0.000507 -0.000242 0.000265 -0.000915 -0.000237 0.000583 -0.000298 0.004234 ... -0.001262 0.000016 -0.000084 -0.000284 0.000274 -0.000122 -0.000622 -0.000421 0.000263 -0.000448
Consumer Cyclicals 0.007310 -0.000507 0.001878 0.000894 -0.000980 0.003387 0.000876 -0.002157 0.001102 -0.015672 ... 0.004671 -0.000058 0.000310 0.001050 -0.001015 0.000451 0.002301 0.001559 -0.000972 0.001659
Consumer Non-Cyclicals 0.003482 -0.000242 0.000894 0.000426 -0.000467 0.001613 0.000417 -0.001027 0.000525 -0.007464 ... 0.002225 -0.000028 0.000148 0.000500 -0.000484 0.000215 0.001096 0.000742 -0.000463 0.000790
Dividend -0.003814 0.000265 -0.000980 -0.000467 0.000511 -0.001767 -0.000457 0.001126 -0.000575 0.008177 ... -0.002437 0.000030 -0.000162 -0.000548 0.000530 -0.000235 -0.001200 -0.000813 0.000507 -0.000866
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2026-03-31 Size -0.000408 -0.000009 0.000112 -0.001085 -0.000074 -0.000341 0.000731 -0.002446 0.000065 -0.000573 ... 0.003500 -0.000023 0.001415 0.000011 -0.000320 0.000663 -0.000032 -0.001118 0.000196 0.001740
Technology -0.001537 -0.004048 -0.002443 -0.003422 -0.000196 -0.002747 -0.000773 -0.002704 0.000123 -0.002570 ... 0.001855 -0.000808 0.001157 0.000667 -0.002062 -0.000032 0.003295 -0.001451 -0.000566 0.001996
Utilities -0.002903 0.003570 -0.002790 0.006001 0.000260 0.005314 -0.001584 0.010334 -0.000296 0.000663 ... -0.001439 -0.000973 -0.004867 0.003461 0.005367 -0.001118 -0.001451 0.016901 -0.001056 -0.002930
Value 0.000100 0.001364 0.001202 0.000088 0.000154 0.001082 -0.000131 0.000415 0.000086 0.000152 ... 0.002268 0.000365 0.001313 -0.000328 -0.000358 0.000196 -0.000566 -0.001056 0.001118 0.000716
Volatility -0.008443 -0.002458 -0.002833 -0.006751 -0.000023 -0.005901 0.004123 -0.036713 -0.000102 -0.007210 ... 0.028142 -0.001643 0.014878 0.003621 -0.002065 0.001740 0.001996 -0.002930 0.000716 0.023099

5271 rows × 21 columns

For downstream comparisons, we split these into factor volatilities and correlations.

# calculate actual factor volatilities from vcovs
df_vol = df_vcov.groupby(level="date").apply(
    lambda df: pd.Series(np.diag(df) ** 0.5, df.index.droplevel("date"))
)
df_vol.columns.name = None
df_vol.tail()
Academic & Educational Services Basic Materials Consumer Cyclicals Consumer Non-Cyclicals Dividend Energy Financials Government Activity Growth Healthcare ... Institutions, Associations & Organizations Leverage Market Momentum Real Estate Size Technology Utilities Value Volatility
date
2026-03-25 0.187684 0.135533 0.084404 0.101847 0.026099 0.171107 0.059623 0.558476 0.016555 0.111548 ... 0.219636 0.029571 0.119714 0.067182 0.088935 0.024066 0.057046 0.129037 0.034140 0.142146
2026-03-26 0.186673 0.135342 0.083893 0.101263 0.025962 0.175770 0.059410 0.562395 0.016781 0.113629 ... 0.227715 0.029849 0.119927 0.068960 0.088547 0.024209 0.057490 0.128775 0.033951 0.144664
2026-03-27 0.185756 0.137121 0.083399 0.103873 0.025808 0.178650 0.059235 0.560397 0.016834 0.113983 ... 0.226444 0.029669 0.121502 0.069202 0.088067 0.024372 0.057378 0.129958 0.033746 0.144584
2026-03-30 0.184752 0.137265 0.083120 0.103240 0.025731 0.178317 0.059636 0.563394 0.016762 0.113979 ... 0.229290 0.029593 0.121534 0.071026 0.087889 0.024379 0.057571 0.129206 0.033583 0.146394
2026-03-31 0.184494 0.136686 0.082649 0.104374 0.025733 0.186215 0.060048 0.572618 0.016700 0.113353 ... 0.236841 0.029946 0.124351 0.071614 0.087416 0.025741 0.057403 0.130003 0.033433 0.151984

5 rows × 21 columns

# calculate actual factor volatilities from vcovs
df_vol = df_vcov.groupby(level="date").apply(
    lambda df: pd.Series(np.diag(df) ** 0.5, df.index.droplevel("date"))
)
df_vol.columns.name = None
df_vol.tail()
Academic & Educational Services Basic Materials Consumer Cyclicals Consumer Non-Cyclicals Dividend Energy Financials Government Activity Growth Healthcare ... Institutions, Associations & Organizations Leverage Market Momentum Real Estate Size Technology Utilities Value Volatility
date
2026-03-25 0.187684 0.135533 0.084404 0.101847 0.026099 0.171107 0.059623 0.558476 0.016555 0.111548 ... 0.219636 0.029571 0.119714 0.067182 0.088935 0.024066 0.057046 0.129037 0.034140 0.142146
2026-03-26 0.186673 0.135342 0.083893 0.101263 0.025962 0.175770 0.059410 0.562395 0.016781 0.113629 ... 0.227715 0.029849 0.119927 0.068960 0.088547 0.024209 0.057490 0.128775 0.033951 0.144664
2026-03-27 0.185756 0.137121 0.083399 0.103873 0.025808 0.178650 0.059235 0.560397 0.016834 0.113983 ... 0.226444 0.029669 0.121502 0.069202 0.088067 0.024372 0.057378 0.129958 0.033746 0.144584
2026-03-30 0.184752 0.137265 0.083120 0.103240 0.025731 0.178317 0.059636 0.563394 0.016762 0.113979 ... 0.229290 0.029593 0.121534 0.071026 0.087889 0.024379 0.057571 0.129206 0.033583 0.146394
2026-03-31 0.184494 0.136686 0.082649 0.104374 0.025733 0.186215 0.060048 0.572618 0.016700 0.113353 ... 0.236841 0.029946 0.124351 0.071614 0.087416 0.025741 0.057403 0.130003 0.033433 0.151984

5 rows × 21 columns

# calculate actual factor correlations from vcovs
df_cor = df_vcov.groupby(level="date").apply(
    lambda df: df.droplevel("date")
    / np.outer(np.diag(df) ** 0.5, np.diag(df) ** 0.5)
)
df_cor.tail()
Academic & Educational Services Basic Materials Consumer Cyclicals Consumer Non-Cyclicals Dividend Energy Financials Government Activity Growth Healthcare ... Institutions, Associations & Organizations Leverage Market Momentum Real Estate Size Technology Utilities Value Volatility
date factor
2026-03-31 Size -0.085841 -0.002684 0.052847 -0.403941 -0.111629 -0.071182 0.472602 -0.165917 0.151280 -0.196416 ... 0.574063 -0.030011 0.442030 0.006216 -0.142327 1.000000 -0.021525 -0.334196 0.227721 0.444806
Technology -0.145151 -0.515929 -0.514876 -0.571166 -0.132626 -0.256960 -0.224223 -0.082265 0.128251 -0.395049 ... 0.136438 -0.470045 0.162046 0.162310 -0.411016 -0.021525 1.000000 -0.194446 -0.294997 0.228783
Utilities -0.121029 0.200927 -0.259636 0.442244 0.077573 0.219515 -0.202845 0.138821 -0.136157 0.044973 ... -0.046746 -0.249991 -0.301050 0.371753 0.472253 -0.334196 -0.194446 1.000000 -0.242867 -0.148294
Value 0.016262 0.298450 0.434840 0.025358 0.179505 0.173794 -0.065180 0.021673 0.154796 0.040146 ... 0.286446 0.364942 0.315728 -0.137116 -0.122327 0.227721 -0.294997 -0.242867 1.000000 0.140896
Volatility -0.301098 -0.118324 -0.225568 -0.425596 -0.005789 -0.208490 0.451712 -0.421851 -0.040011 -0.418486 ... 0.781808 -0.361023 0.787197 0.332704 -0.155426 0.444806 0.228783 -0.148294 0.140896 1.000000

5 rows × 21 columns

Manually replicating the covariance forecasts#

We can also estimated the risk model and get the returns directly. From these returns we can construct the covariance forecasts. Bayesline returns dataframes in polars, but they can be easily converted to pandas dataframes. We also remove the factor group (market, style, industry, etc.) for convenience.

risk_model = bln.equity.riskmodels.load(
    factorriskmodel_settings.with_dataset("bayesline/Bayesline-US-All-1y")
).get_model()
df_factor_returns = risk_model.fret().tail(-1).to_pandas().set_index("date").rename(columns=lambda c: c.split(".")[1])
df_factor_returns.tail()
Market Dividend Growth Leverage Momentum Size Value Volatility Academic & Educational Services Basic Materials ... Consumer Non-Cyclicals Energy Financials Government Activity Healthcare Industrials Institutions, Associations & Organizations Real Estate Technology Utilities
date
2026-03-25 0.005545 0.000347 -0.002265 -0.001562 0.002639 0.001209 0.000531 0.005782 0.013582 0.010119 ... 0.003309 -0.007811 -0.000147 -0.070527 0.007364 0.000178 0.004781 -0.003222 -0.001333 -0.001449
2026-03-26 -0.008572 0.000614 -0.001881 0.002976 -0.009838 -0.002133 0.000665 -0.017761 0.004051 0.007486 ... 0.001583 0.025360 -0.002418 0.051660 0.014214 -0.003017 -0.036999 0.002994 -0.005429 0.006636
2026-03-27 -0.013455 -0.000276 0.001305 -0.000166 0.005455 -0.002215 0.000204 -0.008686 0.005188 0.015191 ... 0.014673 0.021339 -0.002691 0.022885 -0.008802 0.000996 0.004197 0.001861 -0.002984 0.012870
2026-03-30 -0.007819 0.001160 -0.000572 0.001426 -0.010121 0.001568 -0.000970 -0.015963 0.003955 0.009354 ... 0.000108 -0.009378 0.005429 0.048425 0.007160 -0.004746 -0.025031 0.004538 -0.004506 0.001859
2026-03-31 0.016875 0.001635 -0.000666 -0.003213 0.006885 0.004965 -0.001093 0.025092 -0.010212 -0.004790 ... -0.010916 -0.032651 0.005498 -0.068412 0.002255 0.000748 0.036847 -0.001887 0.002617 -0.011560

5 rows × 21 columns

From these returns we can run standard pandas functions to get the EWMAs.

# calculate expected factor volatilities using the ewma
df_vol_tieout = (
    pd.DataFrame(df_factor_returns**2)
    .ewm(halflife=report_settings.factor_cov_settings.halflife_vol)
    .mean()
    .astype(np.float32)
    ** 0.5
    * 252**0.5
)
df_vol_tieout.tail()
Market Dividend Growth Leverage Momentum Size Value Volatility Academic & Educational Services Basic Materials ... Consumer Non-Cyclicals Energy Financials Government Activity Healthcare Industrials Institutions, Associations & Organizations Real Estate Technology Utilities
date
2026-03-25 0.119714 0.026099 0.016555 0.029571 0.067182 0.024066 0.034140 0.142146 0.187684 0.135533 ... 0.101847 0.171107 0.059623 0.558476 0.111548 0.067597 0.219636 0.088935 0.057046 0.129037
2026-03-26 0.119927 0.025962 0.016781 0.029849 0.068960 0.024209 0.033951 0.144664 0.186673 0.135342 ... 0.101263 0.175770 0.059410 0.562395 0.113629 0.067391 0.227715 0.088547 0.057490 0.128775
2026-03-27 0.121502 0.025808 0.016834 0.029669 0.069202 0.024372 0.033746 0.144584 0.185756 0.137121 ... 0.103873 0.178649 0.059235 0.560397 0.113983 0.067003 0.226444 0.088067 0.057378 0.129958
2026-03-30 0.121534 0.025731 0.016762 0.029593 0.071026 0.024379 0.033583 0.146394 0.184752 0.137265 ... 0.103240 0.178317 0.059636 0.563394 0.113979 0.067110 0.229290 0.087889 0.057571 0.129206
2026-03-31 0.124351 0.025733 0.016700 0.029946 0.071614 0.025741 0.033433 0.151984 0.184494 0.136686 ... 0.104374 0.186215 0.060048 0.572618 0.113353 0.066714 0.236841 0.087416 0.057403 0.130003

5 rows × 21 columns

pd.testing.assert_frame_equal(df_vol, df_vol_tieout, check_column_type=False, check_categorical=False, check_like=True)

The correlation are a bit more involved. We need to create the outer products and then run EWMAs on each cell in the outer product matrix.

# calculate the ewma on the outer product (vcov with mean zero)
df_factor_returns_outer = df_factor_returns.groupby("date").apply(
    lambda df: pd.DataFrame(np.outer(df, df), df.columns, df.columns)
)
df_factor_returns_outer.index.names = ["date", "factor"]
df_cor_tieout = (
    pd.DataFrame(df_factor_returns_outer)
    .unstack()
    .ewm(halflife=report_settings.factor_cov_settings.halflife_cor)
    .mean()
    .stack(future_stack=True)
    .reindex(df_factor_returns_outer.columns, axis=1)
    .groupby("date")
    .apply(
        lambda df: df.droplevel("date")
        / np.outer(np.diag(df) ** 0.5, np.diag(df) ** 0.5)
    )
    .astype(np.float32)
)

df_cor_tieout
Market Dividend Growth Leverage Momentum Size Value Volatility Academic & Educational Services Basic Materials ... Consumer Non-Cyclicals Energy Financials Government Activity Healthcare Industrials Institutions, Associations & Organizations Real Estate Technology Utilities
date factor
2025-04-01 Market 1.000000 -1.000000 1.000000 -1.000000 1.000000 1.000000 -1.000000 1.000000 1.000000 -1.000000 ... 1.000000 1.000000 1.000000 -1.000000 -1.000000 1.000000 1.000000 -1.000000 1.000000 1.000000
Dividend -1.000000 1.000000 -1.000000 1.000000 -1.000000 -1.000000 1.000000 -1.000000 -1.000000 1.000000 ... -1.000000 -1.000000 -1.000000 1.000000 1.000000 -1.000000 -1.000000 1.000000 -1.000000 -1.000000
Growth 1.000000 -1.000000 1.000000 -1.000000 1.000000 1.000000 -1.000000 1.000000 1.000000 -1.000000 ... 1.000000 1.000000 1.000000 -1.000000 -1.000000 1.000000 1.000000 -1.000000 1.000000 1.000000
Leverage -1.000000 1.000000 -1.000000 1.000000 -1.000000 -1.000000 1.000000 -1.000000 -1.000000 1.000000 ... -1.000000 -1.000000 -1.000000 1.000000 1.000000 -1.000000 -1.000000 1.000000 -1.000000 -1.000000
Momentum 1.000000 -1.000000 1.000000 -1.000000 1.000000 1.000000 -1.000000 1.000000 1.000000 -1.000000 ... 1.000000 1.000000 1.000000 -1.000000 -1.000000 1.000000 1.000000 -1.000000 1.000000 1.000000
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2026-03-31 Industrials 0.226267 -0.025949 0.042911 0.205076 0.119979 0.169108 0.317757 0.185423 0.044676 0.375895 ... 0.114218 0.069578 0.114124 -0.058272 -0.078380 1.000000 0.210333 0.056403 -0.482192 0.034718
Institutions, Associations & Organizations 0.413368 -0.009360 0.261116 -0.340772 0.606009 0.574063 0.286446 0.781808 -0.292448 -0.006832 ... -0.255683 -0.004063 0.344633 -0.243807 -0.393242 0.210333 1.000000 -0.175821 0.136438 -0.046746
Real Estate -0.111979 0.101780 -0.076714 0.050771 0.009966 -0.142327 -0.122327 -0.155426 -0.015587 0.171827 ... 0.300349 0.081663 0.033409 0.047162 0.137231 0.056403 -0.175821 1.000000 -0.411016 0.472253
Technology 0.162046 -0.132626 0.128251 -0.470045 0.162310 -0.021525 -0.294996 0.228783 -0.145151 -0.515929 ... -0.571166 -0.256960 -0.224223 -0.082265 -0.395049 -0.482192 0.136438 -0.411016 1.000000 -0.194446
Utilities -0.301050 0.077573 -0.136157 -0.249991 0.371753 -0.334196 -0.242868 -0.148294 -0.121028 0.200927 ... 0.442244 0.219515 -0.202845 0.138822 0.044974 0.034718 -0.046746 0.472253 -0.194446 1.000000

5271 rows × 21 columns

pd.testing.assert_frame_equal(df_cor, df_cor_tieout, check_index_type=False, check_categorical=False, check_like=True, atol=1e-5)