{ "cells": [ { "cell_type": "markdown", "id": "4b0aca25", "metadata": {}, "source": [ "# Idiosyncratic volatility and correlation forecasting\n", "\n", "Use this notebook to extract a volatility forecast report for the idiosyncratic returns. The notebook also shows how to compute idiosyncratic correlations. Getting these correlations out of the system directly is not available yet." ] }, { "cell_type": "code", "execution_count": 2, "id": "4b9e63e4", "metadata": {}, "outputs": [], "source": [ "import datetime as dt\n", "from itertools import combinations_with_replacement\n", "\n", "import polars as pl\n", "\n", "from bayesline.api.equity import (\n", " ReportSettings,\n", " ExposureSettings,\n", " FactorRiskModelSettings,\n", " ModelConstructionSettings,\n", " ReportSettings,\n", " UniverseSettings,\n", " CategoricalExposureGroupSettings,\n", " ContinuousExposureGroupSettings,\n", " PortfolioHierarchySettings,\n", " IdiosyncraticVolatilityReportSettings,\n", " IdiosyncraticReturnReportSettings,\n", ")\n", "from bayesline.apiclient import BayeslineApiClient" ] }, { "cell_type": "code", "execution_count": null, "id": "451ed0a5", "metadata": { "tags": [ "skip-execution" ] }, "outputs": [], "source": [ "bln = BayeslineApiClient.new_client(\n", " endpoint=\"https://[ENDPOINT]\",\n", " api_key=\"[API-KEY]\",\n", ")" ] }, { "cell_type": "markdown", "id": "9e180008", "metadata": {}, "source": [ "We begin by specifying a standard factor model that we can compute the idiosyncratic returns in reference to." ] }, { "cell_type": "code", "execution_count": 3, "id": "863a6a08", "metadata": {}, "outputs": [], "source": [ "factorriskmodel_settings = FactorRiskModelSettings(\n", " universe=UniverseSettings(dataset=\"Bayesline-US-All-1y\"),\n", " exposures=ExposureSettings(\n", " exposures=[\n", " ContinuousExposureGroupSettings(hierarchy=\"market\"),\n", " CategoricalExposureGroupSettings(hierarchy=\"trbc\"),\n", " ContinuousExposureGroupSettings(hierarchy=\"style\"),\n", " ]\n", " ),\n", " modelconstruction=ModelConstructionSettings(\n", " estimation_universe=None,\n", " zero_sum_constraints={\"trbc\": \"mcap_weighted\"}\n", " ),\n", ")" ] }, { "cell_type": "markdown", "id": "ce2b6c9d", "metadata": {}, "source": [ "Idiosyncratic risk calculations need a portfolio. This is necessary because we need to make sure we deliver the risk numbers at the asset level in the right ID space. The portfolio can be entirely synthetic, and the weights don't really matter. Here we consider a portfolio of six stocks. The date is the earliest date for which we want to compute output." ] }, { "cell_type": "code", "execution_count": 4, "id": "5ab67d76", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "UploadCommitResult(version=1, committed_names=[])" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "portfolios_loader = bln.equity.portfolios\n", "uploader = portfolios_loader.uploader\n", "demo_portfolio_dataset = uploader.create_or_replace_dataset(\"Demo-Portfolio\")\n", "\n", "df = pl.DataFrame({\n", " \"portfolio_id\": [\"Test-Portfolio\"]*6,\n", " \"asset_id\": [\n", " \"02079K107\", # Alphabet\n", " \"2592345\", # Microsoft\n", " \"67066G10\", # NVIDIA\n", " \"30303M102\", # Meta\n", " \"57636Q104\", # Mastercard\n", " \"92826C839\", # Visa\n", " ],\n", " \"asset_id_type\": [\"cusip9\", \"sedol7\", \"cusip8\", \"cusip9\", \"cusip9\", \"cusip9\"],\n", " \"date\": [dt.date(2025, 1, 1)]*6, # earliest date to calculate things for\n", " \"value\": [100]*6 # not important, we just need the unique identifiers\n", "})\n", "\n", "demo_portfolio_dataset.fast_commit(df, mode=\"append\")" ] }, { "cell_type": "code", "execution_count": 5, "id": "134b7555", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (6, 5)
dateportfolio_idasset_idasset_id_typevalue
datestrstrstrf32
2025-01-01"Test-Portfolio""02079K107""cusip9"100.0
2025-01-01"Test-Portfolio""2592345""sedol7"100.0
2025-01-01"Test-Portfolio""30303M102""cusip9"100.0
2025-01-01"Test-Portfolio""57636Q104""cusip9"100.0
2025-01-01"Test-Portfolio""67066G10""cusip8"100.0
2025-01-01"Test-Portfolio""92826C839""cusip9"100.0
" ], "text/plain": [ "shape: (6, 5)\n", "┌────────────┬────────────────┬───────────┬───────────────┬───────┐\n", "│ date ┆ portfolio_id ┆ asset_id ┆ asset_id_type ┆ value │\n", "│ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ str ┆ str ┆ f32 │\n", "╞════════════╪════════════════╪═══════════╪═══════════════╪═══════╡\n", "│ 2025-01-01 ┆ Test-Portfolio ┆ 02079K107 ┆ cusip9 ┆ 100.0 │\n", "│ 2025-01-01 ┆ Test-Portfolio ┆ 2592345 ┆ sedol7 ┆ 100.0 │\n", "│ 2025-01-01 ┆ Test-Portfolio ┆ 30303M102 ┆ cusip9 ┆ 100.0 │\n", "│ 2025-01-01 ┆ Test-Portfolio ┆ 57636Q104 ┆ cusip9 ┆ 100.0 │\n", "│ 2025-01-01 ┆ Test-Portfolio ┆ 67066G10 ┆ cusip8 ┆ 100.0 │\n", "│ 2025-01-01 ┆ Test-Portfolio ┆ 92826C839 ┆ cusip9 ┆ 100.0 │\n", "└────────────┴────────────────┴───────────┴───────────────┴───────┘" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "demo_portfolio_dataset.get_data().collect()" ] }, { "cell_type": "markdown", "id": "1dff3910", "metadata": {}, "source": [ "After uploading the portfolio, we put it in a hierarchy of just this portfolio. This is necessary to pass to the report api." ] }, { "cell_type": "code", "execution_count": 6, "id": "50a56d6c", "metadata": {}, "outputs": [], "source": [ "ph_settings = PortfolioHierarchySettings.from_source(\n", " source=\"Demo-Portfolio\",\n", " portfolio_ids=[\"Test-Portfolio\"],\n", ")\n", "ph_loader = bln.equity.portfoliohierarchies\n", "ph_api = ph_loader.load(ph_settings)" ] }, { "cell_type": "markdown", "id": "f6ade72e", "metadata": {}, "source": [ "We can quickly take a look at the (drifted) portfolio constituents." ] }, { "cell_type": "code", "execution_count": 7, "id": "18b3801f", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (1_185, 6)
dateportfolio_idinput_asset_idinput_asset_id_typevaluevalue_bench
datestrstrstrf32f32
2025-01-01"Test-Portfolio""02079K107""cusip9"100.0null
2025-01-01"Test-Portfolio""30303M102""cusip9"100.0null
2025-01-01"Test-Portfolio""57636Q104""cusip9"100.0null
2025-01-01"Test-Portfolio""67066G10""cusip8"100.0null
2025-01-01"Test-Portfolio""92826C839""cusip9"100.0null
………………
2025-08-25"Test-Portfolio""02079K107""cusip9"110.090897null
2025-08-25"Test-Portfolio""30303M102""cusip9"128.864685null
2025-08-25"Test-Portfolio""57636Q104""cusip9"113.143509null
2025-08-25"Test-Portfolio""67066G10""cusip8"133.917923null
2025-08-25"Test-Portfolio""92826C839""cusip9"110.935143null
" ], "text/plain": [ "shape: (1_185, 6)\n", "┌────────────┬────────────────┬────────────────┬─────────────────────┬────────────┬─────────────┐\n", "│ date ┆ portfolio_id ┆ input_asset_id ┆ input_asset_id_type ┆ value ┆ value_bench │\n", "│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ str ┆ str ┆ f32 ┆ f32 │\n", "╞════════════╪════════════════╪════════════════╪═════════════════════╪════════════╪═════════════╡\n", "│ 2025-01-01 ┆ Test-Portfolio ┆ 02079K107 ┆ cusip9 ┆ 100.0 ┆ null │\n", "│ 2025-01-01 ┆ Test-Portfolio ┆ 30303M102 ┆ cusip9 ┆ 100.0 ┆ null │\n", "│ 2025-01-01 ┆ Test-Portfolio ┆ 57636Q104 ┆ cusip9 ┆ 100.0 ┆ null │\n", "│ 2025-01-01 ┆ Test-Portfolio ┆ 67066G10 ┆ cusip8 ┆ 100.0 ┆ null │\n", "│ 2025-01-01 ┆ Test-Portfolio ┆ 92826C839 ┆ cusip9 ┆ 100.0 ┆ null │\n", "│ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n", "│ 2025-08-25 ┆ Test-Portfolio ┆ 02079K107 ┆ cusip9 ┆ 110.090897 ┆ null │\n", "│ 2025-08-25 ┆ Test-Portfolio ┆ 30303M102 ┆ cusip9 ┆ 128.864685 ┆ null │\n", "│ 2025-08-25 ┆ Test-Portfolio ┆ 57636Q104 ┆ cusip9 ┆ 113.143509 ┆ null │\n", "│ 2025-08-25 ┆ Test-Portfolio ┆ 67066G10 ┆ cusip8 ┆ 133.917923 ┆ null │\n", "│ 2025-08-25 ┆ Test-Portfolio ┆ 92826C839 ┆ cusip9 ┆ 110.935143 ┆ null │\n", "└────────────┴────────────────┴────────────────┴─────────────────────┴────────────┴─────────────┘" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ph_api.get(None, None)" ] }, { "cell_type": "markdown", "id": "8b4a4102", "metadata": {}, "source": [ "## Getting the idiosyncratic volatility forecasts\n", "\n", "For the idiosyncratic volatility, we can directly query the output. Below we extract a dataframe with the sqrt-diagonal of the idiosyncratic risk matrix. We run with default settings here, but underneath `IdiosyncraticVolatilityReportSettings` many different options are available." ] }, { "cell_type": "code", "execution_count": 8, "id": "376a7092", "metadata": {}, "outputs": [], "source": [ "report_settings = ReportSettings(\n", " report=IdiosyncraticVolatilityReportSettings(\n", " # we can be flexible with the settings here, e.g. half-life\n", " ),\n", " risk_model=factorriskmodel_settings,\n", ")\n", "report_engine = bln.equity.portfolioreport.load(\n", " report_settings, hierarchy_ref_or_settings=ph_settings,\n", ")\n", "order = {\"date\": [\"date\"], \"asset\": [\"input_asset_id\"]}\n", "report = report_engine.get_report(\n", " order, date_start=dt.date(2025, 1, 2), date_end=dt.date(2025, 1, 31)\n", ")\n", "idio_vol_df = (\n", " report.get_data([], expand=(\"date\", \"input_asset_id\"), value_cols=report.metric_cols)\n", " .rename({\"input_asset_id\": \"asset_id\", \"IdiosyncraticVolatility\": \"idio_vol\"})\n", " .with_columns(pl.col(\"date\").str.to_date())\n", ")" ] }, { "cell_type": "code", "execution_count": 9, "id": "903760b0", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (100, 3)
dateasset_ididio_vol
datestrf32
2025-01-02"02079K107"0.246846
2025-01-02"30303M102"0.20427
2025-01-02"57636Q104"0.143023
2025-01-02"67066G10"0.314196
2025-01-02"92826C839"0.14864
………
2025-01-31"02079K107"0.229065
2025-01-31"30303M102"0.229109
2025-01-31"57636Q104"0.16807
2025-01-31"67066G10"0.452878
2025-01-31"92826C839"0.14307
" ], "text/plain": [ "shape: (100, 3)\n", "┌────────────┬───────────┬──────────┐\n", "│ date ┆ asset_id ┆ idio_vol │\n", "│ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ f32 │\n", "╞════════════╪═══════════╪══════════╡\n", "│ 2025-01-02 ┆ 02079K107 ┆ 0.246846 │\n", "│ 2025-01-02 ┆ 30303M102 ┆ 0.20427 │\n", "│ 2025-01-02 ┆ 57636Q104 ┆ 0.143023 │\n", "│ 2025-01-02 ┆ 67066G10 ┆ 0.314196 │\n", "│ 2025-01-02 ┆ 92826C839 ┆ 0.14864 │\n", "│ … ┆ … ┆ … │\n", "│ 2025-01-31 ┆ 02079K107 ┆ 0.229065 │\n", "│ 2025-01-31 ┆ 30303M102 ┆ 0.229109 │\n", "│ 2025-01-31 ┆ 57636Q104 ┆ 0.16807 │\n", "│ 2025-01-31 ┆ 67066G10 ┆ 0.452878 │\n", "│ 2025-01-31 ┆ 92826C839 ┆ 0.14307 │\n", "└────────────┴───────────┴──────────┘" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "idio_vol_df" ] }, { "cell_type": "markdown", "id": "10d19a19", "metadata": {}, "source": [ "## Computing the idiosyncratic correlations\n", "\n", "Sometimes it is necessary to allow for density in the idiosyncratic risk matrix. Factor models may not be able to explain co-movement in smaller clusters of highly similar assets. We are working on integrations, but for now it is only possible to manually compute these off-diagonal correlations from the idiosyncratic return time-series as a post-processing step. In the code below, we will extract the idiosyncratic returns, and subsequently compute the correlation matrix for two groups within our portfolio of six assets.\n", "\n", "First, we run a very similar report as above to extract the idiosyncratic returns time-series." ] }, { "cell_type": "code", "execution_count": 10, "id": "6084d310", "metadata": {}, "outputs": [], "source": [ "report_settings = ReportSettings(\n", " report=IdiosyncraticReturnReportSettings(),\n", " risk_model=factorriskmodel_settings,\n", ")\n", "report_engine = bln.equity.portfolioreport.load(\n", " report_settings, hierarchy_ref_or_settings=ph_settings,\n", ")\n", "order = {\"date\": [\"date\"], \"asset\": [\"input_asset_id\"]}\n", "report = report_engine.get_report(\n", " order, date_start=dt.date(2025, 1, 2), date_end=dt.date(2025, 1, 31)\n", ")\n", "idio_ret_df = (\n", " report.get_data([], expand=(\"date\", \"input_asset_id\"), value_cols=report.metric_cols)\n", " .rename({\"input_asset_id\": \"asset_id\", \"IdiosyncraticReturn\": \"idio_ret\"})\n", " .with_columns(pl.col(\"date\").str.to_date(), pl.col(\"idio_ret\").fill_nan(None))\n", ")" ] }, { "cell_type": "code", "execution_count": 11, "id": "b3b08986", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (100, 3)
dateasset_ididio_ret
datestrf32
2025-01-02"02079K107"0.002278
2025-01-02"30303M102"0.023586
2025-01-02"57636Q104"-0.003064
2025-01-02"67066G10"0.025151
2025-01-02"92826C839"0.002294
………
2025-01-31"02079K107"0.015925
2025-01-31"30303M102"0.003946
2025-01-31"57636Q104"-0.01843
2025-01-31"67066G10"-0.035743
2025-01-31"92826C839"0.001274
" ], "text/plain": [ "shape: (100, 3)\n", "┌────────────┬───────────┬───────────┐\n", "│ date ┆ asset_id ┆ idio_ret │\n", "│ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ f32 │\n", "╞════════════╪═══════════╪═══════════╡\n", "│ 2025-01-02 ┆ 02079K107 ┆ 0.002278 │\n", "│ 2025-01-02 ┆ 30303M102 ┆ 0.023586 │\n", "│ 2025-01-02 ┆ 57636Q104 ┆ -0.003064 │\n", "│ 2025-01-02 ┆ 67066G10 ┆ 0.025151 │\n", "│ 2025-01-02 ┆ 92826C839 ┆ 0.002294 │\n", "│ … ┆ … ┆ … │\n", "│ 2025-01-31 ┆ 02079K107 ┆ 0.015925 │\n", "│ 2025-01-31 ┆ 30303M102 ┆ 0.003946 │\n", "│ 2025-01-31 ┆ 57636Q104 ┆ -0.01843 │\n", "│ 2025-01-31 ┆ 67066G10 ┆ -0.035743 │\n", "│ 2025-01-31 ┆ 92826C839 ┆ 0.001274 │\n", "└────────────┴───────────┴───────────┘" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "idio_ret_df" ] }, { "cell_type": "markdown", "id": "005a3a21", "metadata": {}, "source": [ "Next, we define the groups of similar assets. These need to be mutually exclusive. I.e. we cannot have one asset that is part of multiple groups. Not all assets have to be part of a group.\n", "\n", "In the example below, we put Alphabet, Microsoft and Meta in a group, and Mastercast and Visa in a separate group. NVIDIA is not part of a group." ] }, { "cell_type": "code", "execution_count": 12, "id": "c4c6150f", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (5, 6)
asset_id02079K107259234530303M10257636Q10492826C839
stri32i32i32i32i32
"02079K107"111nullnull
"2592345"null11nullnull
"30303M102"nullnull1nullnull
"57636Q104"nullnullnull11
"92826C839"nullnullnullnull1
" ], "text/plain": [ "shape: (5, 6)\n", "┌───────────┬───────────┬─────────┬───────────┬───────────┬───────────┐\n", "│ asset_id ┆ 02079K107 ┆ 2592345 ┆ 30303M102 ┆ 57636Q104 ┆ 92826C839 │\n", "│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n", "│ str ┆ i32 ┆ i32 ┆ i32 ┆ i32 ┆ i32 │\n", "╞═══════════╪═══════════╪═════════╪═══════════╪═══════════╪═══════════╡\n", "│ 02079K107 ┆ 1 ┆ 1 ┆ 1 ┆ null ┆ null │\n", "│ 2592345 ┆ null ┆ 1 ┆ 1 ┆ null ┆ null │\n", "│ 30303M102 ┆ null ┆ null ┆ 1 ┆ null ┆ null │\n", "│ 57636Q104 ┆ null ┆ null ┆ null ┆ 1 ┆ 1 │\n", "│ 92826C839 ┆ null ┆ null ┆ null ┆ null ┆ 1 │\n", "└───────────┴───────────┴─────────┴───────────┴───────────┴───────────┘" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "groups = [\n", " [\"02079K107\", \"2592345\", \"30303M102\"], # Alphabet, Microsoft, Meta\n", " [\"57636Q104\", \"92826C839\"], # Mastercard, Visa\n", "]\n", "\n", "# create a dataframe with all combinations\n", "df_offdiag = pl.DataFrame(\n", " [\n", " (left, right)\n", " for group in groups\n", " for left, right in combinations_with_replacement(group, 2)\n", " ], \n", " schema=[\"asset_id\", \"asset_id_right\"],\n", " orient=\"row\",\n", ")\n", "\n", "# just for display, this is in realistic scenarios a very large dataframe\n", "(\n", " df_offdiag.sort(\"asset_id\", \"asset_id_right\")\n", " .with_columns(pl.lit(1))\n", " .pivot(\"asset_id_right\", index=\"asset_id\", maintain_order=True, sort_columns=True)\n", ")" ] }, { "cell_type": "code", "execution_count": 13, "id": "e1a36f1c", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (120, 5)
dateasset_ididio_retasset_id_rightidio_ret_right
datestrf32strf32
2025-01-02"02079K107"0.002278"02079K107"0.002278
2025-01-02"02079K107"0.002278"30303M102"0.023586
2025-01-02"30303M102"0.023586"30303M102"0.023586
2025-01-02"57636Q104"-0.003064"57636Q104"-0.003064
2025-01-02"57636Q104"-0.003064"92826C839"0.002294
……………
2025-01-31"02079K107"0.015925"30303M102"0.003946
2025-01-31"30303M102"0.003946"30303M102"0.003946
2025-01-31"57636Q104"-0.01843"57636Q104"-0.01843
2025-01-31"57636Q104"-0.01843"92826C839"0.001274
2025-01-31"92826C839"0.001274"92826C839"0.001274
" ], "text/plain": [ "shape: (120, 5)\n", "┌────────────┬───────────┬───────────┬────────────────┬────────────────┐\n", "│ date ┆ asset_id ┆ idio_ret ┆ asset_id_right ┆ idio_ret_right │\n", "│ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ f32 ┆ str ┆ f32 │\n", "╞════════════╪═══════════╪═══════════╪════════════════╪════════════════╡\n", "│ 2025-01-02 ┆ 02079K107 ┆ 0.002278 ┆ 02079K107 ┆ 0.002278 │\n", "│ 2025-01-02 ┆ 02079K107 ┆ 0.002278 ┆ 30303M102 ┆ 0.023586 │\n", "│ 2025-01-02 ┆ 30303M102 ┆ 0.023586 ┆ 30303M102 ┆ 0.023586 │\n", "│ 2025-01-02 ┆ 57636Q104 ┆ -0.003064 ┆ 57636Q104 ┆ -0.003064 │\n", "│ 2025-01-02 ┆ 57636Q104 ┆ -0.003064 ┆ 92826C839 ┆ 0.002294 │\n", "│ … ┆ … ┆ … ┆ … ┆ … │\n", "│ 2025-01-31 ┆ 02079K107 ┆ 0.015925 ┆ 30303M102 ┆ 0.003946 │\n", "│ 2025-01-31 ┆ 30303M102 ┆ 0.003946 ┆ 30303M102 ┆ 0.003946 │\n", "│ 2025-01-31 ┆ 57636Q104 ┆ -0.01843 ┆ 57636Q104 ┆ -0.01843 │\n", "│ 2025-01-31 ┆ 57636Q104 ┆ -0.01843 ┆ 92826C839 ┆ 0.001274 │\n", "│ 2025-01-31 ┆ 92826C839 ┆ 0.001274 ┆ 92826C839 ┆ 0.001274 │\n", "└────────────┴───────────┴───────────┴────────────────┴────────────────┘" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# join the time series such that we have each combination that we need to compute\n", "idio_ret_df_joined = (\n", " idio_ret_df.join(df_offdiag, on=\"asset_id\")\n", " .join(idio_ret_df, left_on=(\"date\", \"asset_id_right\"), right_on=(\"date\", \"asset_id\"))\n", ")\n", "idio_ret_df_joined" ] }, { "cell_type": "markdown", "id": "3ceef638", "metadata": {}, "source": [ "We compute the covariance matrix first, and then standardize into the correlation matrix. The computation of the covariance matrix relies on computing a rolling mean to correct for autocorrelation, and subsequently an exponentially weighted moving average. We then divide by the standard deviations to obtain the correlations." ] }, { "cell_type": "code", "execution_count": 14, "id": "04afcfaa", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (120, 6)
dateasset_ididio_retasset_id_rightidio_ret_rightidio_vcov
datestrf32strf32f32
2025-01-02"02079K107"0.002278"02079K107"0.0022780.000005
2025-01-02"02079K107"0.002278"30303M102"0.0235860.000054
2025-01-02"30303M102"0.023586"30303M102"0.0235860.000556
2025-01-02"57636Q104"-0.003064"57636Q104"-0.0030640.000009
2025-01-02"57636Q104"-0.003064"92826C839"0.002294-0.000007
………………
2025-01-31"02079K107"0.015502"30303M102"0.0066210.000004
2025-01-31"30303M102"0.011291"30303M102"0.0060590.000112
2025-01-31"57636Q104"0.009226"57636Q104"0.0054710.000021
2025-01-31"57636Q104"0.004305"92826C839"0.0065550.000022
2025-01-31"92826C839"0.006503"92826C839"0.0054970.000023
" ], "text/plain": [ "shape: (120, 6)\n", "┌────────────┬───────────┬───────────┬────────────────┬────────────────┬───────────┐\n", "│ date ┆ asset_id ┆ idio_ret ┆ asset_id_right ┆ idio_ret_right ┆ idio_vcov │\n", "│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ f32 ┆ str ┆ f32 ┆ f32 │\n", "╞════════════╪═══════════╪═══════════╪════════════════╪════════════════╪═══════════╡\n", "│ 2025-01-02 ┆ 02079K107 ┆ 0.002278 ┆ 02079K107 ┆ 0.002278 ┆ 0.000005 │\n", "│ 2025-01-02 ┆ 02079K107 ┆ 0.002278 ┆ 30303M102 ┆ 0.023586 ┆ 0.000054 │\n", "│ 2025-01-02 ┆ 30303M102 ┆ 0.023586 ┆ 30303M102 ┆ 0.023586 ┆ 0.000556 │\n", "│ 2025-01-02 ┆ 57636Q104 ┆ -0.003064 ┆ 57636Q104 ┆ -0.003064 ┆ 0.000009 │\n", "│ 2025-01-02 ┆ 57636Q104 ┆ -0.003064 ┆ 92826C839 ┆ 0.002294 ┆ -0.000007 │\n", "│ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n", "│ 2025-01-31 ┆ 02079K107 ┆ 0.015502 ┆ 30303M102 ┆ 0.006621 ┆ 0.000004 │\n", "│ 2025-01-31 ┆ 30303M102 ┆ 0.011291 ┆ 30303M102 ┆ 0.006059 ┆ 0.000112 │\n", "│ 2025-01-31 ┆ 57636Q104 ┆ 0.009226 ┆ 57636Q104 ┆ 0.005471 ┆ 0.000021 │\n", "│ 2025-01-31 ┆ 57636Q104 ┆ 0.004305 ┆ 92826C839 ┆ 0.006555 ┆ 0.000022 │\n", "│ 2025-01-31 ┆ 92826C839 ┆ 0.006503 ┆ 92826C839 ┆ 0.005497 ┆ 0.000023 │\n", "└────────────┴───────────┴───────────┴────────────────┴────────────────┴───────────┘" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# compute the covariance matrix by first using a rolling mean (for overlap),\n", "# and then an exponential weighted moving average (for smoothing)\n", "overlap_window = 5\n", "half_life = 126\n", "\n", "idio_vcov_df = (\n", " idio_ret_df_joined\n", " .with_columns(\n", " pl.col(\"idio_ret\").rolling_mean(window_size=overlap_window, min_samples=1).over(\"asset_id\"),\n", " pl.col(\"idio_ret_right\").rolling_mean(window_size=overlap_window, min_samples=1).over(\"asset_id_right\"),\n", " )\n", " .with_columns(\n", " (pl.col(\"idio_ret\") * pl.col(\"idio_ret_right\"))\n", " .ewm_mean(half_life=half_life)\n", " .over((\"asset_id\", \"asset_id_right\"))\n", " .alias(\"idio_vcov\")\n", " )\n", ")\n", "idio_vcov_df" ] }, { "cell_type": "code", "execution_count": 15, "id": "47e085ab", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (80, 3)
dateasset_ididio_var
datestrf32
2025-01-02"02079K107"0.000005
2025-01-02"30303M102"0.000556
2025-01-02"57636Q104"0.000009
2025-01-02"92826C839"0.000005
2025-01-03"02079K107"0.000003
………
2025-01-30"92826C839"0.000022
2025-01-31"02079K107"0.00001
2025-01-31"30303M102"0.000112
2025-01-31"57636Q104"0.000021
2025-01-31"92826C839"0.000023
" ], "text/plain": [ "shape: (80, 3)\n", "┌────────────┬───────────┬──────────┐\n", "│ date ┆ asset_id ┆ idio_var │\n", "│ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ f32 │\n", "╞════════════╪═══════════╪══════════╡\n", "│ 2025-01-02 ┆ 02079K107 ┆ 0.000005 │\n", "│ 2025-01-02 ┆ 30303M102 ┆ 0.000556 │\n", "│ 2025-01-02 ┆ 57636Q104 ┆ 0.000009 │\n", "│ 2025-01-02 ┆ 92826C839 ┆ 0.000005 │\n", "│ 2025-01-03 ┆ 02079K107 ┆ 0.000003 │\n", "│ … ┆ … ┆ … │\n", "│ 2025-01-30 ┆ 92826C839 ┆ 0.000022 │\n", "│ 2025-01-31 ┆ 02079K107 ┆ 0.00001 │\n", "│ 2025-01-31 ┆ 30303M102 ┆ 0.000112 │\n", "│ 2025-01-31 ┆ 57636Q104 ┆ 0.000021 │\n", "│ 2025-01-31 ┆ 92826C839 ┆ 0.000023 │\n", "└────────────┴───────────┴──────────┘" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# to translate the covariance matrix to a correlation matrix, \n", "# we need to select the variance of the idiosyncratic returns\n", "idio_var_df = (\n", " idio_vcov_df.filter(pl.col(\"asset_id\") == pl.col(\"asset_id_right\"))\n", " .select(\"date\", \"asset_id\", pl.col(\"idio_vcov\").alias(\"idio_var\"))\n", ")\n", "idio_var_df" ] }, { "cell_type": "code", "execution_count": 16, "id": "8ff6ce1d", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (120, 4)
dateasset_idasset_id_rightidio_corr
datestrstrf32
2025-01-02"02079K107""02079K107"1.0
2025-01-02"02079K107""30303M102"1.0
2025-01-02"30303M102""30303M102"1.0
2025-01-02"57636Q104""57636Q104"1.0
2025-01-02"57636Q104""92826C839"-1.0
…………
2025-01-31"02079K107""30303M102"0.117286
2025-01-31"30303M102""30303M102"1.0
2025-01-31"57636Q104""57636Q104"1.0
2025-01-31"57636Q104""92826C839"0.98339
2025-01-31"92826C839""92826C839"1.0
" ], "text/plain": [ "shape: (120, 4)\n", "┌────────────┬───────────┬────────────────┬───────────┐\n", "│ date ┆ asset_id ┆ asset_id_right ┆ idio_corr │\n", "│ --- ┆ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ str ┆ f32 │\n", "╞════════════╪═══════════╪════════════════╪═══════════╡\n", "│ 2025-01-02 ┆ 02079K107 ┆ 02079K107 ┆ 1.0 │\n", "│ 2025-01-02 ┆ 02079K107 ┆ 30303M102 ┆ 1.0 │\n", "│ 2025-01-02 ┆ 30303M102 ┆ 30303M102 ┆ 1.0 │\n", "│ 2025-01-02 ┆ 57636Q104 ┆ 57636Q104 ┆ 1.0 │\n", "│ 2025-01-02 ┆ 57636Q104 ┆ 92826C839 ┆ -1.0 │\n", "│ … ┆ … ┆ … ┆ … │\n", "│ 2025-01-31 ┆ 02079K107 ┆ 30303M102 ┆ 0.117286 │\n", "│ 2025-01-31 ┆ 30303M102 ┆ 30303M102 ┆ 1.0 │\n", "│ 2025-01-31 ┆ 57636Q104 ┆ 57636Q104 ┆ 1.0 │\n", "│ 2025-01-31 ┆ 57636Q104 ┆ 92826C839 ┆ 0.98339 │\n", "│ 2025-01-31 ┆ 92826C839 ┆ 92826C839 ┆ 1.0 │\n", "└────────────┴───────────┴────────────────┴───────────┘" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# by joining twice and normalizing, we get the correlation matrix\n", "idio_corr_df = (\n", " idio_vcov_df.join(idio_var_df, on=(\"date\", \"asset_id\"))\n", " .join(idio_var_df, left_on=(\"date\", \"asset_id_right\"), right_on=(\"date\", \"asset_id\"))\n", " .select(\n", " \"date\",\n", " \"asset_id\",\n", " \"asset_id_right\",\n", " (pl.col(\"idio_vcov\") / (pl.col(\"idio_var\") * pl.col(\"idio_var_right\")).sqrt()).alias(\"idio_corr\"))\n", ")\n", "idio_corr_df\n" ] }, { "cell_type": "code", "execution_count": 17, "id": "9fb7a037", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (76, 6)
dateasset_id02079K10730303M10257636Q10492826C839
datestrf32f32f32f32
2025-01-03"02079K107"1.00.662149nullnull
2025-01-03"30303M102"null1.0nullnull
2025-01-03"57636Q104"nullnull1.0-0.173034
2025-01-03"92826C839"nullnullnull1.0
2025-01-06"02079K107"1.00.8804nullnull
………………
2025-01-30"92826C839"nullnullnull1.0
2025-01-31"02079K107"1.00.117286nullnull
2025-01-31"30303M102"null1.0nullnull
2025-01-31"57636Q104"nullnull1.00.98339
2025-01-31"92826C839"nullnullnull1.0
" ], "text/plain": [ "shape: (76, 6)\n", "┌────────────┬───────────┬───────────┬───────────┬───────────┬───────────┐\n", "│ date ┆ asset_id ┆ 02079K107 ┆ 30303M102 ┆ 57636Q104 ┆ 92826C839 │\n", "│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ f32 ┆ f32 ┆ f32 ┆ f32 │\n", "╞════════════╪═══════════╪═══════════╪═══════════╪═══════════╪═══════════╡\n", "│ 2025-01-03 ┆ 02079K107 ┆ 1.0 ┆ 0.662149 ┆ null ┆ null │\n", "│ 2025-01-03 ┆ 30303M102 ┆ null ┆ 1.0 ┆ null ┆ null │\n", "│ 2025-01-03 ┆ 57636Q104 ┆ null ┆ null ┆ 1.0 ┆ -0.173034 │\n", "│ 2025-01-03 ┆ 92826C839 ┆ null ┆ null ┆ null ┆ 1.0 │\n", "│ 2025-01-06 ┆ 02079K107 ┆ 1.0 ┆ 0.8804 ┆ null ┆ null │\n", "│ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n", "│ 2025-01-30 ┆ 92826C839 ┆ null ┆ null ┆ null ┆ 1.0 │\n", "│ 2025-01-31 ┆ 02079K107 ┆ 1.0 ┆ 0.117286 ┆ null ┆ null │\n", "│ 2025-01-31 ┆ 30303M102 ┆ null ┆ 1.0 ┆ null ┆ null │\n", "│ 2025-01-31 ┆ 57636Q104 ┆ null ┆ null ┆ 1.0 ┆ 0.98339 │\n", "│ 2025-01-31 ┆ 92826C839 ┆ null ┆ null ┆ null ┆ 1.0 │\n", "└────────────┴───────────┴───────────┴───────────┴───────────┴───────────┘" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# for small portfolios, the dataframe is small enough to pivot and display\n", "(\n", " idio_corr_df.pivot(\"asset_id_right\", index=(\"date\", \"asset_id\"), maintain_order=True, sort_columns=True)\n", " .filter(pl.col(\"date\") > pl.col(\"date\").min())\n", ")" ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.14" } }, "nbformat": 4, "nbformat_minor": 5 }