Factor Covariance Matrix Forecasts

Factor Covariance Matrix Forecasts#

In this tutorial we are going to create several covariance matrices of factor returns. This covariance matrix follows from a standard factor model. We will also tie out the numbers against a pandas/numpy-based reimplementation. More specifically, we will:

  • Create a basic risk model.

  • Extract the factor returns.

  • Use the covariance matrix forecast report to compute the covariance matrix.

  • Use pandas to extract the factor volatility and correlation matrix time-series.

  • Replicate the volatility forecast.

  • Replicate the correlation forecast.

Throughout this notebook we work with a randomly generated dataset. The results should generalize to real data, but for legal reasons we do not show any real data on our public API. Bayesline clients can run this notebook on real data.

Imports & Setup#

For this tutorial notebook, you will need to import the following packages.

import pandas as pd
import polars as pl
import numpy as np

from bayesline.api.equity import (
    FactorCovarianceReportSettings,
    ExposureSettings,
    CategoricalExposureGroupSettings,
    ContinuousExposureGroupSettings,
    FactorRiskModelSettings,
    ModelConstructionSettings,
    UniverseSettings,
)
from bayesline.apiclient import BayeslineApiClient

We will also need to have a Bayesline API client configured.

bln = BayeslineApiClient.new_client(
    endpoint="https://[ENDPOINT]",
    api_key="[API-KEY]",
)

Creating the covariance matrix forecasts#

Let’s first set up a basic risk model and use it to generate the forecasts. We choose to run with mostly default settings. The steps involved are:

  1. Creating the settings of the risk model.

  2. Loading the report model engine.

  3. Running the engine to generate the covariance report.

The first step is creating the risk model settings.

factorriskmodel_settings = FactorRiskModelSettings(
    universe=UniverseSettings(),
    exposures=ExposureSettings(
        exposures=[
            ContinuousExposureGroupSettings(hierarchy="market"),
            CategoricalExposureGroupSettings(hierarchy="trbc"),
            ContinuousExposureGroupSettings(hierarchy="style"),
        ]
    ),
    modelconstruction=ModelConstructionSettings(
        estimation_universe=None,
        zero_sum_constraints={"trbc": "mcap_weighted"},
    ),
)

Next, we create the report engine from the report settings. We run with the defaults here.

report_settings = FactorCovarianceReportSettings(
    factor_model_settings=factorriskmodel_settings
)
report_engine = bln.equity.reports.load(
    report_settings.with_dataset("bayesline/Bayesline-US-All-1y")
)

Let’s see what these settings really are by printing them out.

print(report_settings.factor_cov_settings.model_dump_json(indent=2))

{
  "halflife_vol": 60.0,
  "halflife_cor": 120.0,
  "halflife_vra": null,
  "nw_lags_vol": 0,
  "nw_lags_vol_halflife_override": null,
  "nw_lags_cor": 0,
  "shrink_cor_method": null,
  "shrink_cor_length": null,
  "combine_standardized": false
}

The different settings that jointly make up the covariance matrix are:

  1. halflife_factor_vol The halflife of the factor volatility. The default is a 60-day halflife.

  2. halflife_factor_vra The halflife of the cross-sectional factor volatility adjustment. The default is to not do any adjustment.

  3. halflife_factor_cor The halflife of the factor correlation. The default is a 120-day halflife.

  4. nw_lags_factor_vol The overlap or Newey-West lags to incluce on the factor volatility forecast. The default is zero, meaning no autocorrelation correction is performed.

  5. nw_lags_factor_cor The overlap or Newey-West lags to incluce on the factor correlation forecast. The default is zero, meaning no autocorrelation correction is performed.

Now we get the actual time-series of the covariance matrices.

# generate the report data
report = report_engine.calculate(start_date=None, end_date=None)

# get the covariance matrix (and skip the first date)
df_report = report.get_covariance().filter(pl.col("date") > pl.col("date").min())

# convert to pandas
df_vcov = df_report.to_pandas().set_index(["date", "factor"]).rename(columns=lambda c: c.split("^")[0])
df_vcov
Academic & Educational Services Basic Materials Consumer Cyclicals Consumer Non-Cyclicals Dividend Energy Financials Government Activity Growth Healthcare ... Institutions, Associations & Organizations Leverage Market Momentum Real Estate Size Technology Utilities Value Volatility
date factor
2025-04-01 Academic & Educational Services 0.029966 -0.002397 0.007511 0.003567 -0.003908 0.013510 0.003488 -0.008661 0.004479 -0.062701 ... 0.018827 -0.000213 0.001271 0.004183 -0.004209 0.001777 0.009248 0.006246 -0.003862 0.006650
Basic Materials -0.002397 0.000192 -0.000601 -0.000285 0.000313 -0.001081 -0.000279 0.000693 -0.000358 0.005015 ... -0.001506 0.000017 -0.000102 -0.000335 0.000337 -0.000142 -0.000740 -0.000500 0.000309 -0.000532
Consumer Cyclicals 0.007511 -0.000601 0.001883 0.000894 -0.000980 0.003386 0.000874 -0.002171 0.001123 -0.015716 ... 0.004719 -0.000053 0.000319 0.001048 -0.001055 0.000446 0.002318 0.001566 -0.000968 0.001667
Consumer Non-Cyclicals 0.003567 -0.000285 0.000894 0.000425 -0.000465 0.001608 0.000415 -0.001031 0.000533 -0.007464 ... 0.002241 -0.000025 0.000151 0.000498 -0.000501 0.000212 0.001101 0.000744 -0.000460 0.000792
Dividend -0.003908 0.000313 -0.000980 -0.000465 0.000510 -0.001762 -0.000455 0.001130 -0.000584 0.008177 ... -0.002455 0.000028 -0.000166 -0.000545 0.000549 -0.000232 -0.001206 -0.000815 0.000504 -0.000867
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2026-03-31 Size -0.000416 -0.000027 0.000110 -0.001081 -0.000074 -0.000338 0.000728 -0.002423 0.000064 -0.000572 ... 0.003468 -0.000020 0.001408 0.000008 -0.000319 0.000658 -0.000029 -0.001117 0.000194 0.001727
Technology -0.001592 -0.003989 -0.002406 -0.003404 -0.000201 -0.002714 -0.000762 -0.002654 0.000126 -0.002552 ... 0.001815 -0.000801 0.001161 0.000645 -0.002039 -0.000029 0.003253 -0.001454 -0.000563 0.001975
Utilities -0.002772 0.003643 -0.002818 0.006006 0.000266 0.005321 -0.001576 0.010350 -0.000291 0.000662 ... -0.001393 -0.000992 -0.004899 0.003477 0.005372 -0.001117 -0.001454 0.016975 -0.001069 -0.002930
Value 0.000130 0.001376 0.001224 0.000094 0.000155 0.001080 -0.000145 0.000409 0.000085 0.000148 ... 0.002241 0.000370 0.001333 -0.000333 -0.000357 0.000194 -0.000563 -0.001069 0.001127 0.000708
Volatility -0.008483 -0.002486 -0.002836 -0.006738 -0.000010 -0.005903 0.004154 -0.036847 -0.000093 -0.007229 ... 0.028196 -0.001636 0.014895 0.003633 -0.002074 0.001727 0.001975 -0.002930 0.000708 0.023168

5271 rows × 21 columns

For downstream comparisons, we split these into factor volatilities and correlations.

# calculate actual factor volatilities from vcovs
df_vol = df_vcov.groupby(level="date").apply(
    lambda df: pd.Series(np.diag(df) ** 0.5, df.index.droplevel("date"))
)
df_vol.columns.name = None
df_vol.tail()
Academic & Educational Services Basic Materials Consumer Cyclicals Consumer Non-Cyclicals Dividend Energy Financials Government Activity Growth Healthcare ... Institutions, Associations & Organizations Leverage Market Momentum Real Estate Size Technology Utilities Value Volatility
date
2026-03-25 0.187725 0.135723 0.084518 0.101834 0.026181 0.171109 0.059769 0.558746 0.016477 0.111581 ... 0.219731 0.029652 0.119914 0.067208 0.088902 0.023988 0.056709 0.129350 0.034274 0.142350
2026-03-26 0.186729 0.135498 0.084006 0.101249 0.026044 0.175761 0.059556 0.562685 0.016709 0.113658 ... 0.227805 0.029929 0.120131 0.068976 0.088513 0.024126 0.057143 0.129082 0.034084 0.144876
2026-03-27 0.185819 0.137364 0.083513 0.103850 0.025891 0.178639 0.059380 0.560700 0.016760 0.114019 ... 0.226535 0.029748 0.121707 0.069221 0.088033 0.024283 0.057027 0.130254 0.033878 0.144801
2026-03-30 0.184813 0.137546 0.083225 0.103217 0.025810 0.178309 0.059767 0.563712 0.016689 0.114020 ... 0.229350 0.029669 0.121738 0.071040 0.087866 0.024296 0.057213 0.129501 0.033712 0.146612
2026-03-31 0.184630 0.136953 0.082755 0.104334 0.025810 0.186179 0.060200 0.572984 0.016630 0.113394 ... 0.236829 0.030019 0.124546 0.071623 0.087384 0.025655 0.057034 0.130287 0.033566 0.152210

5 rows × 21 columns

# calculate actual factor volatilities from vcovs
df_vol = df_vcov.groupby(level="date").apply(
    lambda df: pd.Series(np.diag(df) ** 0.5, df.index.droplevel("date"))
)
df_vol.columns.name = None
df_vol.tail()
Academic & Educational Services Basic Materials Consumer Cyclicals Consumer Non-Cyclicals Dividend Energy Financials Government Activity Growth Healthcare ... Institutions, Associations & Organizations Leverage Market Momentum Real Estate Size Technology Utilities Value Volatility
date
2026-03-25 0.187725 0.135723 0.084518 0.101834 0.026181 0.171109 0.059769 0.558746 0.016477 0.111581 ... 0.219731 0.029652 0.119914 0.067208 0.088902 0.023988 0.056709 0.129350 0.034274 0.142350
2026-03-26 0.186729 0.135498 0.084006 0.101249 0.026044 0.175761 0.059556 0.562685 0.016709 0.113658 ... 0.227805 0.029929 0.120131 0.068976 0.088513 0.024126 0.057143 0.129082 0.034084 0.144876
2026-03-27 0.185819 0.137364 0.083513 0.103850 0.025891 0.178639 0.059380 0.560700 0.016760 0.114019 ... 0.226535 0.029748 0.121707 0.069221 0.088033 0.024283 0.057027 0.130254 0.033878 0.144801
2026-03-30 0.184813 0.137546 0.083225 0.103217 0.025810 0.178309 0.059767 0.563712 0.016689 0.114020 ... 0.229350 0.029669 0.121738 0.071040 0.087866 0.024296 0.057213 0.129501 0.033712 0.146612
2026-03-31 0.184630 0.136953 0.082755 0.104334 0.025810 0.186179 0.060200 0.572984 0.016630 0.113394 ... 0.236829 0.030019 0.124546 0.071623 0.087384 0.025655 0.057034 0.130287 0.033566 0.152210

5 rows × 21 columns

# calculate actual factor correlations from vcovs
df_cor = df_vcov.groupby(level="date").apply(
    lambda df: df.droplevel("date")
    / np.outer(np.diag(df) ** 0.5, np.diag(df) ** 0.5)
)
df_cor.tail()
Academic & Educational Services Basic Materials Consumer Cyclicals Consumer Non-Cyclicals Dividend Energy Financials Government Activity Growth Healthcare ... Institutions, Associations & Organizations Leverage Market Momentum Real Estate Size Technology Utilities Value Volatility
date factor
2026-03-31 Size -0.087767 -0.007812 0.051913 -0.403710 -0.111696 -0.070753 0.471439 -0.164838 0.150275 -0.196580 ... 0.570746 -0.026066 0.440511 0.004331 -0.142113 1.000000 -0.020136 -0.334070 0.225717 0.442306
Technology -0.151158 -0.510701 -0.509806 -0.571969 -0.136784 -0.255635 -0.221979 -0.081213 0.132799 -0.394590 ... 0.134354 -0.467682 0.163508 0.158012 -0.409212 -0.020136 1.000000 -0.195636 -0.294188 0.227499
Utilities -0.115229 0.204157 -0.261373 0.441839 0.079028 0.219362 -0.200980 0.138650 -0.134463 0.044824 ... -0.045137 -0.253754 -0.301886 0.372597 0.471882 -0.334070 -0.195636 1.000000 -0.244381 -0.147736
Value 0.020946 0.299401 0.440687 0.026834 0.179429 0.172757 -0.071787 0.021291 0.152658 0.038806 ... 0.281907 0.367435 0.318979 -0.138385 -0.121592 0.225717 -0.294188 -0.244381 1.000000 0.138647
Volatility -0.301855 -0.119242 -0.225170 -0.424275 -0.002648 -0.208306 0.453329 -0.422491 -0.036698 -0.418838 ... 0.782194 -0.358040 0.785705 0.333288 -0.155940 0.442306 0.227499 -0.147736 0.138647 1.000000

5 rows × 21 columns

Manually replicating the covariance forecasts#

We can also estimated the risk model and get the returns directly. From these returns we can construct the covariance forecasts. Bayesline returns dataframes in polars, but they can be easily converted to pandas dataframes. We also remove the factor group (market, style, industry, etc.) for convenience.

risk_model = bln.equity.riskmodels.load(
    factorriskmodel_settings.with_dataset("bayesline/Bayesline-US-All-1y")
).get_model()
df_factor_returns = risk_model.fret().tail(-1).to_pandas().set_index("date").rename(columns=lambda c: c.split(".")[1])
df_factor_returns.tail()
Market Dividend Growth Leverage Momentum Size Value Volatility Academic & Educational Services Basic Materials ... Consumer Non-Cyclicals Energy Financials Government Activity Healthcare Industrials Institutions, Associations & Organizations Real Estate Technology Utilities
date
2026-03-25 0.005580 0.000363 -0.002220 -0.001572 0.002646 0.001181 0.000537 0.005788 0.013241 0.010212 ... 0.003293 -0.007817 -0.000165 -0.070588 0.007417 0.000180 0.004814 -0.003258 -0.001323 -0.001459
2026-03-26 -0.008602 0.000615 -0.001893 0.002980 -0.009817 -0.002108 0.000657 -0.017798 0.004285 0.007291 ... 0.001567 0.025334 -0.002422 0.051750 0.014207 -0.003142 -0.036999 0.002984 -0.005370 0.006618
2026-03-27 -0.013475 -0.000330 0.001293 -0.000141 0.005472 -0.002188 0.000218 -0.008729 0.005280 0.015455 ... 0.014650 0.021335 -0.002698 0.023007 -0.008833 0.000926 0.004219 0.001851 -0.002942 0.012854
2026-03-30 -0.007827 0.001134 -0.000577 0.001407 -0.010111 0.001593 -0.000949 -0.015981 0.003921 0.009553 ... 0.000090 -0.009390 0.005387 0.048501 0.007191 -0.004815 -0.024947 0.004605 -0.004455 0.001864
2026-03-31 0.016869 0.001627 -0.000683 -0.003213 0.006868 0.004951 -0.001138 0.025127 -0.010651 -0.004681 ... -0.010863 -0.032599 0.005578 -0.068569 0.002256 0.000796 0.036700 -0.001743 0.002512 -0.011541

5 rows × 21 columns

From these returns we can run standard pandas functions to get the EWMAs.

# calculate expected factor volatilities using the ewma
df_vol_tieout = (
    pd.DataFrame(df_factor_returns**2)
    .ewm(halflife=report_settings.factor_cov_settings.halflife_vol)
    .mean()
    .astype(np.float32)
    ** 0.5
    * 252**0.5
)
df_vol_tieout.tail()
Market Dividend Growth Leverage Momentum Size Value Volatility Academic & Educational Services Basic Materials ... Consumer Non-Cyclicals Energy Financials Government Activity Healthcare Industrials Institutions, Associations & Organizations Real Estate Technology Utilities
date
2026-03-25 0.119914 0.026181 0.016477 0.029652 0.067208 0.023988 0.034274 0.142350 0.187725 0.135723 ... 0.101834 0.171109 0.059769 0.558746 0.111581 0.067564 0.219731 0.088902 0.056709 0.129350
2026-03-26 0.120131 0.026044 0.016709 0.029929 0.068976 0.024126 0.034084 0.144876 0.186729 0.135498 ... 0.101249 0.175761 0.059556 0.562685 0.113658 0.067376 0.227805 0.088513 0.057143 0.129082
2026-03-27 0.121707 0.025891 0.016760 0.029748 0.069221 0.024283 0.033878 0.144801 0.185819 0.137364 ... 0.103850 0.178639 0.059380 0.560700 0.114019 0.066984 0.226535 0.088033 0.057027 0.130254
2026-03-30 0.121738 0.025810 0.016689 0.029669 0.071040 0.024296 0.033712 0.146612 0.184813 0.137546 ... 0.103217 0.178309 0.059767 0.563712 0.114020 0.067107 0.229350 0.087866 0.057213 0.129501
2026-03-31 0.124546 0.025810 0.016630 0.030019 0.071623 0.025655 0.033566 0.152210 0.184630 0.136953 ... 0.104334 0.186179 0.060200 0.572983 0.113394 0.066713 0.236829 0.087384 0.057034 0.130287

5 rows × 21 columns

pd.testing.assert_frame_equal(df_vol, df_vol_tieout, check_column_type=False, check_categorical=False, check_like=True)

The correlation are a bit more involved. We need to create the outer products and then run EWMAs on each cell in the outer product matrix.

# calculate the ewma on the outer product (vcov with mean zero)
df_factor_returns_outer = df_factor_returns.groupby("date").apply(
    lambda df: pd.DataFrame(np.outer(df, df), df.columns, df.columns)
)
df_factor_returns_outer.index.names = ["date", "factor"]
df_cor_tieout = (
    pd.DataFrame(df_factor_returns_outer)
    .unstack()
    .ewm(halflife=report_settings.factor_cov_settings.halflife_cor)
    .mean()
    .stack(future_stack=True)
    .reindex(df_factor_returns_outer.columns, axis=1)
    .groupby("date")
    .apply(
        lambda df: df.droplevel("date")
        / np.outer(np.diag(df) ** 0.5, np.diag(df) ** 0.5)
    )
    .astype(np.float32)
)

df_cor_tieout
Market Dividend Growth Leverage Momentum Size Value Volatility Academic & Educational Services Basic Materials ... Consumer Non-Cyclicals Energy Financials Government Activity Healthcare Industrials Institutions, Associations & Organizations Real Estate Technology Utilities
date factor
2025-04-01 Market 1.000000 -1.000000 1.000000 -1.000000 1.000000 1.000000 -1.000000 1.000000 1.000000 -1.000000 ... 1.000000 1.000000 1.000000 -1.000000 -1.000000 1.000000 1.000000 -1.000000 1.000000 1.000000
Dividend -1.000000 1.000000 -1.000000 1.000000 -1.000000 -1.000000 1.000000 -1.000000 -1.000000 1.000000 ... -1.000000 -1.000000 -1.000000 1.000000 1.000000 -1.000000 -1.000000 1.000000 -1.000000 -1.000000
Growth 1.000000 -1.000000 1.000000 -1.000000 1.000000 1.000000 -1.000000 1.000000 1.000000 -1.000000 ... 1.000000 1.000000 1.000000 -1.000000 -1.000000 1.000000 1.000000 -1.000000 1.000000 1.000000
Leverage -1.000000 1.000000 -1.000000 1.000000 -1.000000 -1.000000 1.000000 -1.000000 -1.000000 1.000000 ... -1.000000 -1.000000 -1.000000 1.000000 1.000000 -1.000000 -1.000000 1.000000 -1.000000 -1.000000
Momentum 1.000000 -1.000000 1.000000 -1.000000 1.000000 1.000000 -1.000000 1.000000 1.000000 -1.000000 ... 1.000000 1.000000 1.000000 -1.000000 -1.000000 1.000000 1.000000 -1.000000 1.000000 1.000000
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2026-03-31 Industrials 0.228479 -0.025782 0.039055 0.202773 0.124681 0.167971 0.316804 0.188615 0.044724 0.373740 ... 0.112988 0.068361 0.111934 -0.059869 -0.082290 1.000000 0.212385 0.054179 -0.479444 0.035091
Institutions, Associations & Organizations 0.412659 -0.006597 0.264721 -0.339985 0.607500 0.570746 0.281907 0.782194 -0.298033 -0.006043 ... -0.252321 -0.003293 0.346594 -0.244793 -0.394259 0.212385 1.000000 -0.174273 0.134354 -0.045137
Real Estate -0.113487 0.107564 -0.076570 0.047704 0.011607 -0.142113 -0.121591 -0.155940 -0.009027 0.169025 ... 0.298918 0.080334 0.030914 0.048033 0.138668 0.054179 -0.174273 1.000000 -0.409212 0.471882
Technology 0.163508 -0.136784 0.132799 -0.467683 0.158012 -0.020136 -0.294188 0.227499 -0.151158 -0.510701 ... -0.571968 -0.255635 -0.221979 -0.081213 -0.394590 -0.479444 0.134354 -0.409212 1.000000 -0.195636
Utilities -0.301886 0.079028 -0.134463 -0.253754 0.372597 -0.334071 -0.244381 -0.147736 -0.115229 0.204157 ... 0.441839 0.219362 -0.200980 0.138650 0.044825 0.035091 -0.045137 0.471882 -0.195636 1.000000

5271 rows × 21 columns

pd.testing.assert_frame_equal(df_cor, df_cor_tieout, check_index_type=False, check_categorical=False, check_like=True, atol=1e-5)