Factor Covariance Matrix Forecasts

Factor Covariance Matrix Forecasts#

In this tutorial we are going to create several covariance matrices of factor returns. This covariance matrix follows from a standard factor model. We will also tie out the numbers against a pandas/numpy-based reimplementation. More specifically, we will:

  • Create a basic risk model.

  • Extract the factor returns.

  • Use the covariance matrix forecast report to compute the covariance matrix.

  • Use pandas to extract the factor volatility and correlation matrix time-series.

  • Replicate the volatility forecast.

  • Replicate the correlation forecast.

Throughout this notebook we work with a randomly generated dataset. The results should generalize to real data, but for legal reasons we do not show any real data on our public API. Bayesline clients can run this notebook on real data.

Imports & Setup#

For this tutorial notebook, you will need to import the following packages.

import pandas as pd
import polars as pl
import numpy as np

from bayesline.api.equity import (
    ReportSettings,
    FactorCovarianceReportSettings,
    ExposureSettings,
    CategoricalExposureGroupSettings,
    ContinuousExposureGroupSettings,
    FactorRiskModelSettings,
    ModelConstructionSettings,
    ReportSettings,
    UniverseSettings,
)
from bayesline.apiclient import BayeslineApiClient

We will also need to have a Bayesline API client configured.

bln = BayeslineApiClient.new_client(
    endpoint="https://[ENDPOINT]",
    api_key="[API-KEY]",
)

Creating the covariance matrix forecasts#

Let’s first set up a basic risk model and use it to generate the forecasts. We choose to run with mostly default settings. The steps involved are:

  1. Creating the settings of the risk model.

  2. Loading the report model engine.

  3. Running the engine to generate the covariance report.

The first step is creating the risk model settings.

factorriskmodel_settings = FactorRiskModelSettings(
    universe=UniverseSettings(dataset="Bayesline-US-All-1y"),
    exposures=ExposureSettings(
        exposures=[
            ContinuousExposureGroupSettings(hierarchy="market"),
            CategoricalExposureGroupSettings(hierarchy="trbc"),
            CategoricalExposureGroupSettings(hierarchy="continent"),
            ContinuousExposureGroupSettings(hierarchy="style"),
        ]
    ),
    modelconstruction=ModelConstructionSettings(
        estimation_universe=None,
        zero_sum_constraints={"trbc": "mcap_weighted", "continent": "mcap_weighted"},
    ),
)

Next, we create the report engine from the report settings. We run with the defaults here.

report_settings = ReportSettings(
    report=FactorCovarianceReportSettings(),
    risk_model=factorriskmodel_settings,
)
report_engine = bln.equity.portfolioreport.load(report_settings)

Let’s see what these settings really are by printing them out.

print(report_settings.report.model_dump_json(indent=2))

{
  "report_type": "Factor Covariance report",
  "measures": [
    {
      "type": "FactorCovariance"
    }
  ],
  "halflife_factor_vol": 42,
  "halflife_factor_vra": null,
  "halflife_factor_cor": 126,
  "shrink_factor_cor_method": null,
  "shrink_factor_cor_length": 1008,
  "shrink_factor_cor_standardized": false,
  "nw_lags_factor_vol": 0,
  "nw_lags_factor_vol_halflife_override": null,
  "nw_lags_factor_cor": 0
}

The different settings that jointly make up the covariance matrix are:

  1. halflife_factor_vol The halflife of the factor volatility. The default is a 42-day halflife.

  2. halflife_factor_vra The halflife of the cross-sectional factor volatility adjustment. The default is to not do any adjustment.

  3. halflife_factor_cor The halflife of the factor correlation. The default is a 126-day halflife.

  4. nw_lags_factor_vol The overlap or Newey-West lags to incluce on the factor volatility forecast. The default is zero, meaning no autocorrelation correction is performed.

  5. nw_lags_factor_cor The overlap or Newey-West lags to incluce on the factor correlation forecast. The default is zero, meaning no autocorrelation correction is performed.

Now we get the actual time-series of the covariance matrices.

# generate the report data
order = {
    "date": ["date"],
    "factor": ["factor_group", "factor"],
    "factor_col": ["factor_group_col", "factor_col"],
}
report = report_engine.get_report(order=order)

# massage the data into a more usable format
df_report = (
    report.get_data(
        [],
        expand=("date", "factor"),
        pivot_cols=("factor_col",),
        value_cols=("FactorCovariance",)
    )
).with_columns(pl.col("date").cast(pl.Date))  # string to date

# convert to pandas
df_vcov = df_report.to_pandas().set_index(["date", "factor"]).rename(columns=lambda c: c.split("^")[0])
df_vcov
Market Energy Basic Materials Industrials Consumer Cyclicals Consumer Non-Cyclicals Financials Healthcare Technology Utilities ... Asia Europe Oceania Size Value Growth Volatility Momentum Dividend Leverage
date factor
2024-08-27 Market 0.008549 0.007347 0.002354 0.002585 0.001155 0.000515 0.000188 0.000235 -0.002596 0.009968 ... 0.0 0.0 0.0 -0.001120 0.001911 -0.001005 0.004580 -0.004461 0.002833 -0.001222
Energy 0.007347 0.006315 0.002024 0.002222 0.000993 0.000443 0.000161 0.000202 -0.002232 0.008568 ... 0.0 0.0 0.0 -0.000963 0.001642 -0.000864 0.003936 -0.003834 0.002435 -0.001050
Basic Materials 0.002354 0.002024 0.000648 0.000712 0.000318 0.000142 0.000052 0.000065 -0.000715 0.002745 ... 0.0 0.0 0.0 -0.000308 0.000526 -0.000277 0.001261 -0.001229 0.000780 -0.000336
Industrials 0.002585 0.002222 0.000712 0.000782 0.000349 0.000156 0.000057 0.000071 -0.000785 0.003014 ... 0.0 0.0 0.0 -0.000339 0.000578 -0.000304 0.001385 -0.001349 0.000857 -0.000369
Consumer Cyclicals 0.001155 0.000993 0.000318 0.000349 0.000156 0.000070 0.000025 0.000032 -0.000351 0.001347 ... 0.0 0.0 0.0 -0.000151 0.000258 -0.000136 0.000619 -0.000603 0.000383 -0.000165
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2025-08-25 Growth 0.000211 0.000137 -0.000168 0.000039 -0.000155 -0.000261 0.000139 -0.000551 0.000170 -0.000278 ... 0.0 0.0 0.0 0.000135 -0.000047 0.000260 0.000197 0.000133 -0.000044 -0.000015
Volatility 0.017282 -0.000875 -0.002093 0.000434 -0.001165 -0.006964 0.004359 -0.005585 0.001023 -0.004869 ... 0.0 0.0 0.0 0.001830 0.000648 0.000197 0.012445 0.001006 -0.000604 -0.000865
Momentum -0.001328 0.000668 -0.001548 -0.000355 -0.002242 -0.000612 0.000611 -0.001446 0.000729 0.001895 ... 0.0 0.0 0.0 0.000023 -0.000888 0.000133 0.001006 0.003371 -0.000375 -0.000507
Dividend -0.000546 0.000055 0.000258 -0.000081 0.000065 0.000493 -0.000289 0.000603 -0.000126 0.000305 ... 0.0 0.0 0.0 -0.000118 0.000074 -0.000044 -0.000604 -0.000375 0.000497 0.000136
Leverage -0.000515 0.000029 0.000426 0.000329 0.000777 0.000641 -0.000032 0.000340 -0.000528 -0.000042 ... 0.0 0.0 0.0 0.000047 0.000256 -0.000015 -0.000865 -0.000507 0.000136 0.000572

6474 rows × 26 columns

For downstream comparisons, we split these into factor volatilities and correlations.

# calculate actual factor volatilities from vcovs
df_vol = df_vcov.groupby(level="date").apply(
    lambda df: pd.Series(np.diag(df) ** 0.5, df.index.droplevel("date"))
)
df_vol.columns.name = None
df_vol.tail()
Market Energy Basic Materials Industrials Consumer Cyclicals Consumer Non-Cyclicals Financials Healthcare Technology Utilities ... Asia Europe Oceania Size Value Growth Volatility Momentum Dividend Leverage
date
2025-08-19 0.174478 0.161339 0.092217 0.052369 0.076123 0.085978 0.055699 0.114562 0.048106 0.116367 ... 0.0 0.0 0.0 0.031683 0.029726 0.016493 0.110163 0.058340 0.022914 0.024584
2025-08-20 0.173223 0.161692 0.091460 0.052181 0.076143 0.086372 0.055279 0.114801 0.047997 0.115407 ... 0.0 0.0 0.0 0.031418 0.029517 0.016369 0.109364 0.058225 0.022732 0.024397
2025-08-21 0.171782 0.160867 0.091433 0.051746 0.075585 0.085692 0.054947 0.114128 0.047597 0.114540 ... 0.0 0.0 0.0 0.031267 0.029270 0.016392 0.108454 0.057814 0.022549 0.024258
2025-08-22 0.180060 0.159554 0.090673 0.051503 0.075659 0.086504 0.055919 0.115678 0.047266 0.113831 ... 0.0 0.0 0.0 0.031239 0.031298 0.016255 0.112492 0.058330 0.022457 0.024074
2025-08-25 0.179187 0.159279 0.090122 0.051098 0.075124 0.086540 0.055482 0.115959 0.046934 0.113455 ... 0.0 0.0 0.0 0.030989 0.031140 0.016135 0.111557 0.058057 0.022287 0.023911

5 rows × 26 columns

# calculate actual factor correlations from vcovs
df_cor = df_vcov.groupby(level="date").apply(
    lambda df: df.droplevel("date")
    / np.outer(np.diag(df) ** 0.5, np.diag(df) ** 0.5)
)
df_cor.tail()
Market Energy Basic Materials Industrials Consumer Cyclicals Consumer Non-Cyclicals Financials Healthcare Technology Utilities ... Asia Europe Oceania Size Value Growth Volatility Momentum Dividend Leverage
date factor
2025-08-25 Growth 0.072930 0.053423 -0.115818 0.047327 -0.127851 -0.186615 0.155763 -0.294290 0.224344 -0.151667 ... NaN NaN NaN 0.269275 -0.092818 1.000000 0.109421 0.141630 -0.122407 -0.038837
Volatility 0.864562 -0.049251 -0.208170 0.076127 -0.139029 -0.721397 0.704250 -0.431754 0.195327 -0.384719 ... NaN NaN NaN 0.529328 0.186631 0.109421 1.000000 0.155344 -0.242822 -0.324273
Momentum -0.127632 0.072282 -0.295866 -0.119687 -0.514055 -0.121782 0.189530 -0.214781 0.267664 0.287649 ... NaN NaN NaN 0.012715 -0.491260 0.141630 0.155344 1.000000 -0.290121 -0.365039
Dividend -0.136759 0.015583 0.128541 -0.071216 0.038839 0.255597 -0.233936 0.233220 -0.120000 0.120484 ... NaN NaN NaN -0.171221 0.106988 -0.122407 -0.242822 -0.290121 1.000000 0.255712
Leverage -0.120253 0.007557 0.197648 0.269557 0.432760 0.309850 -0.024378 0.122760 -0.470407 -0.015581 ... NaN NaN NaN 0.062999 0.343915 -0.038837 -0.324273 -0.365039 0.255712 1.000000

5 rows × 26 columns

Manually replicating the covariance forecasts#

We can also estimated the risk model and get the returns directly. From these returns we can construct the covariance forecasts. Bayesline returns dataframes in polars, but they can be easily converted to pandas dataframes. We also remove the factor group (market, style, industry, etc.) for convenience.

risk_model = bln.equity.riskmodels.load(factorriskmodel_settings).get_model()
df_factor_returns = risk_model.fret().to_pandas().set_index("date").rename(columns=lambda c: c.split(".")[1])
df_factor_returns.tail()
Market Energy Basic Materials Industrials Consumer Cyclicals Consumer Non-Cyclicals Financials Healthcare Technology Utilities ... Asia Europe Oceania Size Value Growth Volatility Momentum Dividend Leverage
date
2025-08-19 -0.010030 -0.002211 0.003790 0.004671 0.007428 0.003076 -0.002137 0.004506 -0.004272 0.003795 ... 0.0 0.0 0.0 0.001249 0.001424 0.001119 -0.013432 -0.006619 -0.000567 0.004158
2025-08-20 -0.004105 0.011421 0.000776 -0.002489 -0.004869 0.006748 0.001097 0.008071 -0.002586 0.000871 ... 0.0 0.0 0.0 0.000048 -0.000745 -0.000332 -0.002525 0.003210 0.000316 0.000465
2025-08-21 0.000757 0.006351 0.005661 0.000192 -0.001684 -0.001305 -0.001847 0.003947 0.000147 -0.002310 ... 0.0 0.0 0.0 -0.001286 0.000062 -0.001115 0.000428 0.001444 0.000265 -0.000868
2025-08-22 0.028486 -0.001532 0.000407 0.002156 0.005032 -0.007904 0.006137 -0.011687 -0.001228 -0.003669 ... 0.0 0.0 0.0 0.001864 0.005716 -0.000019 0.016106 -0.005248 -0.001016 0.000465
2025-08-25 -0.007338 0.008952 0.002981 -0.000784 0.001873 -0.005584 0.000900 -0.008283 0.001188 -0.005571 ... 0.0 0.0 0.0 0.000403 0.001239 -0.000353 -0.000519 0.002438 -0.000440 0.000653

5 rows × 26 columns

From these returns we can run standard pandas functions to get the EWMAs.

# calculate expected factor volatilities using the ewma
df_vol_tieout = (
    pd.DataFrame(df_factor_returns**2)
    .ewm(halflife=report_settings.report.halflife_factor_vol)
    .mean()
    .astype(np.float32)
    ** 0.5
    * 252**0.5
)
df_vol_tieout.tail()
Market Energy Basic Materials Industrials Consumer Cyclicals Consumer Non-Cyclicals Financials Healthcare Technology Utilities ... Asia Europe Oceania Size Value Growth Volatility Momentum Dividend Leverage
date
2025-08-19 0.174478 0.161339 0.092217 0.052369 0.076123 0.085978 0.055699 0.114562 0.048106 0.116367 ... 0.0 0.0 0.0 0.031683 0.029726 0.016493 0.110163 0.058340 0.022915 0.024584
2025-08-20 0.173223 0.161692 0.091460 0.052181 0.076143 0.086372 0.055279 0.114801 0.047997 0.115407 ... 0.0 0.0 0.0 0.031418 0.029517 0.016369 0.109364 0.058225 0.022732 0.024397
2025-08-21 0.171782 0.160867 0.091433 0.051746 0.075585 0.085692 0.054947 0.114128 0.047597 0.114540 ... 0.0 0.0 0.0 0.031267 0.029270 0.016392 0.108454 0.057814 0.022549 0.024258
2025-08-22 0.180060 0.159554 0.090673 0.051503 0.075659 0.086504 0.055919 0.115678 0.047266 0.113831 ... 0.0 0.0 0.0 0.031239 0.031298 0.016255 0.112492 0.058330 0.022457 0.024074
2025-08-25 0.179187 0.159279 0.090122 0.051098 0.075124 0.086540 0.055482 0.115959 0.046934 0.113455 ... 0.0 0.0 0.0 0.030989 0.031140 0.016135 0.111557 0.058057 0.022287 0.023911

5 rows × 26 columns

pd.testing.assert_frame_equal(df_vol, df_vol_tieout, check_column_type=False, check_categorical=False, check_like=True)

The correlation are a bit more involved. We need to create the outer products and then run EWMAs on each cell in the outer product matrix.

# calculate the ewma on the outer product (vcov with mean zero)
df_factor_returns_outer = df_factor_returns.groupby("date").apply(
    lambda df: pd.DataFrame(np.outer(df, df), df.columns, df.columns)
)
df_factor_returns_outer.index.names = ["date", "factor"]
df_cor_tieout = (
    pd.DataFrame(df_factor_returns_outer)
    .unstack()
    .ewm(halflife=report_settings.report.halflife_factor_cor)
    .mean()
    .stack(future_stack=True)
    .reindex(df_factor_returns_outer.columns, axis=1)
    .groupby("date")
    .apply(
        lambda df: df.droplevel("date")
        / np.outer(np.diag(df) ** 0.5, np.diag(df) ** 0.5)
    )
    .astype(np.float32)
)

df_cor_tieout
Market Energy Basic Materials Industrials Consumer Cyclicals Consumer Non-Cyclicals Financials Healthcare Technology Utilities ... Asia Europe Oceania Size Value Growth Volatility Momentum Dividend Leverage
date factor
2024-08-27 Market 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 -1.000000 1.000000 ... NaN NaN NaN -1.000000 1.000000 -1.000000 1.000000 -1.000000 1.000000 -1.000000
Energy 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 -1.000000 1.000000 ... NaN NaN NaN -1.000000 1.000000 -1.000000 1.000000 -1.000000 1.000000 -1.000000
Basic Materials 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 -1.000000 1.000000 ... NaN NaN NaN -1.000000 1.000000 -1.000000 1.000000 -1.000000 1.000000 -1.000000
Industrials 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 -1.000000 1.000000 ... NaN NaN NaN -1.000000 1.000000 -1.000000 1.000000 -1.000000 1.000000 -1.000000
Consumer Cyclicals 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 -1.000000 1.000000 ... NaN NaN NaN -1.000000 1.000000 -1.000000 1.000000 -1.000000 1.000000 -1.000000
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2025-08-25 Growth 0.072930 0.053423 -0.115818 0.047327 -0.127851 -0.186615 0.155763 -0.294290 0.224344 -0.151667 ... NaN NaN NaN 0.269276 -0.092818 1.000000 0.109421 0.141630 -0.122407 -0.038837
Volatility 0.864562 -0.049251 -0.208170 0.076127 -0.139029 -0.721398 0.704250 -0.431754 0.195327 -0.384719 ... NaN NaN NaN 0.529328 0.186631 0.109421 1.000000 0.155344 -0.242822 -0.324273
Momentum -0.127632 0.072282 -0.295866 -0.119687 -0.514055 -0.121782 0.189530 -0.214781 0.267665 0.287649 ... NaN NaN NaN 0.012715 -0.491259 0.141630 0.155344 1.000000 -0.290121 -0.365039
Dividend -0.136758 0.015583 0.128541 -0.071216 0.038839 0.255597 -0.233936 0.233220 -0.120000 0.120484 ... NaN NaN NaN -0.171222 0.106988 -0.122407 -0.242822 -0.290121 1.000000 0.255712
Leverage -0.120253 0.007557 0.197648 0.269557 0.432760 0.309850 -0.024378 0.122760 -0.470408 -0.015581 ... NaN NaN NaN 0.062999 0.343915 -0.038837 -0.324273 -0.365039 0.255712 1.000000

6474 rows × 26 columns

pd.testing.assert_frame_equal(df_cor, df_cor_tieout, check_index_type=False, check_categorical=False, check_like=True, atol=1e-6)