{ "cells": [ { "cell_type": "markdown", "id": "4b0aca25", "metadata": {}, "source": [ "# Idiosyncratic volatility and correlation forecasting\n", "\n", "Use this notebook to extract a volatility forecast report for the idiosyncratic returns. The notebook also shows how to compute idiosyncratic correlations. Getting these correlations out of the system directly is not available yet." ] }, { "cell_type": "code", "execution_count": 2, "id": "4b9e63e4", "metadata": {}, "outputs": [], "source": [ "import datetime as dt\n", "from itertools import combinations_with_replacement\n", "\n", "import polars as pl\n", "\n", "from bayesline.api.equity import (\n", " ReportSettings,\n", " ExposureSettings,\n", " FactorRiskModelSettings,\n", " ModelConstructionSettings,\n", " ReportSettings,\n", " UniverseSettings,\n", " CategoricalExposureGroupSettings,\n", " ContinuousExposureGroupSettings,\n", " PortfolioHierarchySettings,\n", " IdiosyncraticVolatilityReportSettings,\n", " IdiosyncraticReturnReportSettings,\n", " IdioReportSettingsV2,\n", ")\n", "from bayesline.apiclient import BayeslineApiClient" ] }, { "cell_type": "code", "execution_count": null, "id": "451ed0a5", "metadata": { "tags": [ "skip-execution" ] }, "outputs": [], "source": [ "bln = BayeslineApiClient.new_client(\n", " endpoint=\"https://[ENDPOINT]\",\n", " api_key=\"[API-KEY]\",\n", ")" ] }, { "cell_type": "markdown", "id": "9e180008", "metadata": {}, "source": [ "We begin by specifying a standard factor model that we can compute the idiosyncratic returns in reference to." ] }, { "cell_type": "code", "execution_count": 3, "id": "863a6a08", "metadata": {}, "outputs": [], "source": [ "factorriskmodel_settings = FactorRiskModelSettings(\n", " universe=UniverseSettings(dataset=\"Bayesline-US-All-1y\"),\n", " exposures=ExposureSettings(\n", " exposures=[\n", " ContinuousExposureGroupSettings(hierarchy=\"market\"),\n", " CategoricalExposureGroupSettings(hierarchy=\"trbc\"),\n", " ContinuousExposureGroupSettings(hierarchy=\"style\"),\n", " ]\n", " ),\n", " modelconstruction=ModelConstructionSettings(\n", " estimation_universe=None,\n", " zero_sum_constraints={\"trbc\": \"mcap_weighted\"}\n", " ),\n", ")" ] }, { "cell_type": "markdown", "id": "8b4a4102", "metadata": {}, "source": [ "## Getting the idiosyncratic volatility forecasts\n", "\n", "For the idiosyncratic volatility, we can directly query the output. Below we extract a dataframe with the sqrt-diagonal of the idiosyncratic risk matrix. We run with default settings here, but underneath `IdioReportSettingsV2` many different options are available." ] }, { "cell_type": "code", "execution_count": 4, "id": "4be054b6", "metadata": {}, "outputs": [], "source": [ "report_settings = IdioReportSettingsV2(\n", " factor_model_settings=factorriskmodel_settings\n", ")\n", "\n", "report_engine = bln.equity.reports.load(report_settings)\n", "report = report_engine.calculate(start_date=\"2026-01-02\", end_date=\"2026-01-30\")\n", "\n", "idio_vol_df = report.accessor.get_data(\n", " [], expand=(\"date\", \"asset_id\"), value_cols=(\"idio_vol\",)\n", ").with_columns(pl.col(pl.Float32).fill_nan(None))\n" ] }, { "cell_type": "code", "execution_count": 5, "id": "2a2aff94", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (171_242, 3)
dateasset_ididio_vol
datestrf32
2026-01-05"IC000B1557"0.007989
2026-01-05"IC0010CEFE"0.326402
2026-01-05"IC0021AFB7"0.05656
2026-01-05"IC002CE8B9"1.21583
2026-01-05"IC002DC646"0.114718
………
2026-01-30"ICFFE60191"0.60148
2026-01-30"ICFFE938FD"0.188128
2026-01-30"ICFFE94AED"0.273419
2026-01-30"ICFFEBBB38"0.072164
2026-01-30"ICFFF2F5AD"1.396709
" ], "text/plain": [ "shape: (171_242, 3)\n", "┌────────────┬────────────┬──────────┐\n", "│ date ┆ asset_id ┆ idio_vol │\n", "│ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ f32 │\n", "╞════════════╪════════════╪══════════╡\n", "│ 2026-01-05 ┆ IC000B1557 ┆ 0.007989 │\n", "│ 2026-01-05 ┆ IC0010CEFE ┆ 0.326402 │\n", "│ 2026-01-05 ┆ IC0021AFB7 ┆ 0.05656 │\n", "│ 2026-01-05 ┆ IC002CE8B9 ┆ 1.21583 │\n", "│ 2026-01-05 ┆ IC002DC646 ┆ 0.114718 │\n", "│ … ┆ … ┆ … │\n", "│ 2026-01-30 ┆ ICFFE60191 ┆ 0.60148 │\n", "│ 2026-01-30 ┆ ICFFE938FD ┆ 0.188128 │\n", "│ 2026-01-30 ┆ ICFFE94AED ┆ 0.273419 │\n", "│ 2026-01-30 ┆ ICFFEBBB38 ┆ 0.072164 │\n", "│ 2026-01-30 ┆ ICFFF2F5AD ┆ 1.396709 │\n", "└────────────┴────────────┴──────────┘" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "idio_vol_df.drop_nulls()" ] }, { "cell_type": "markdown", "id": "10d19a19", "metadata": {}, "source": [ "## Computing the idiosyncratic correlations\n", "\n", "Sometimes it is necessary to allow for density in the idiosyncratic risk matrix. Factor models may not be able to explain co-movement in smaller clusters of highly similar assets. We are working on integrations, but for now it is only possible to manually compute these off-diagonal correlations from the idiosyncratic return time-series as a post-processing step. In the code below, we will extract the idiosyncratic returns, and subsequently compute the correlation matrix for two groups within our portfolio of six assets.\n", "\n", "First, we run a very similar report as above to extract the idiosyncratic returns time-series." ] }, { "cell_type": "code", "execution_count": 6, "id": "a857ca35", "metadata": {}, "outputs": [], "source": [ "idio_ret_df = report.accessor.get_data(\n", " [], expand=(\"date\", \"asset_id\"), value_cols=(\"idio_return\",)\n", ").with_columns(pl.col(pl.Float32).fill_nan(None))" ] }, { "cell_type": "code", "execution_count": 7, "id": "b3b08986", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (170_962, 3)
dateasset_ididio_return
datestrf32
2026-01-05"IC000B1557"0.000503
2026-01-05"IC0010CEFE"-0.020561
2026-01-05"IC0021AFB7"-0.003563
2026-01-05"IC002CE8B9"0.07659
2026-01-05"IC002DC646"0.007227
………
2026-01-30"ICFFE60191"0.018608
2026-01-30"ICFFE938FD"0.010133
2026-01-30"ICFFE94AED"0.00897
2026-01-30"ICFFEBBB38"-0.004432
2026-01-30"ICFFF2F5AD"-0.047053
" ], "text/plain": [ "shape: (170_962, 3)\n", "┌────────────┬────────────┬─────────────┐\n", "│ date ┆ asset_id ┆ idio_return │\n", "│ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ f32 │\n", "╞════════════╪════════════╪═════════════╡\n", "│ 2026-01-05 ┆ IC000B1557 ┆ 0.000503 │\n", "│ 2026-01-05 ┆ IC0010CEFE ┆ -0.020561 │\n", "│ 2026-01-05 ┆ IC0021AFB7 ┆ -0.003563 │\n", "│ 2026-01-05 ┆ IC002CE8B9 ┆ 0.07659 │\n", "│ 2026-01-05 ┆ IC002DC646 ┆ 0.007227 │\n", "│ … ┆ … ┆ … │\n", "│ 2026-01-30 ┆ ICFFE60191 ┆ 0.018608 │\n", "│ 2026-01-30 ┆ ICFFE938FD ┆ 0.010133 │\n", "│ 2026-01-30 ┆ ICFFE94AED ┆ 0.00897 │\n", "│ 2026-01-30 ┆ ICFFEBBB38 ┆ -0.004432 │\n", "│ 2026-01-30 ┆ ICFFF2F5AD ┆ -0.047053 │\n", "└────────────┴────────────┴─────────────┘" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "idio_ret_df.drop_nulls()" ] }, { "cell_type": "markdown", "id": "005a3a21", "metadata": {}, "source": [ "Next, we define the groups of similar assets. These need to be mutually exclusive. I.e. we cannot have one asset that is part of multiple groups. Not all assets have to be part of a group.\n", "\n", "In the example below, we put Apple, Microsoft and Alphabet in a group, and Mastercast and Visa in a separate group. NVIDIA is not part of a group." ] }, { "cell_type": "code", "execution_count": 8, "id": "c4c6150f", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (5, 6)
asset_idIC28E9776FIC63213253IC83A1B819ICA17F00B9ICF982536B
stri32i32i32i32i32
"IC28E9776F"11nullnullnull
"IC63213253"null1nullnullnull
"IC83A1B819"nullnull111
"ICA17F00B9"nullnullnull11
"ICF982536B"nullnullnullnull1
" ], "text/plain": [ "shape: (5, 6)\n", "┌────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐\n", "│ asset_id ┆ IC28E9776F ┆ IC63213253 ┆ IC83A1B819 ┆ ICA17F00B9 ┆ ICF982536B │\n", "│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n", "│ str ┆ i32 ┆ i32 ┆ i32 ┆ i32 ┆ i32 │\n", "╞════════════╪════════════╪════════════╪════════════╪════════════╪════════════╡\n", "│ IC28E9776F ┆ 1 ┆ 1 ┆ null ┆ null ┆ null │\n", "│ IC63213253 ┆ null ┆ 1 ┆ null ┆ null ┆ null │\n", "│ IC83A1B819 ┆ null ┆ null ┆ 1 ┆ 1 ┆ 1 │\n", "│ ICA17F00B9 ┆ null ┆ null ┆ null ┆ 1 ┆ 1 │\n", "│ ICF982536B ┆ null ┆ null ┆ null ┆ null ┆ 1 │\n", "└────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "groups = [\n", " sorted([\"IC83A1B819\", \"ICF982536B\", \"ICA17F00B9\"]), # Apple, Microsoft, Alphabet\n", " sorted([\"IC63213253\", \"IC28E9776F\"]), # Mastercard, Visa\n", "]\n", "\n", "# create a dataframe with all combinations\n", "df_offdiag = pl.DataFrame(\n", " [\n", " (left, right)\n", " for group in groups\n", " for left, right in combinations_with_replacement(group, 2)\n", " ], \n", " schema=[\"asset_id\", \"asset_id_right\"],\n", " orient=\"row\",\n", ")\n", "\n", "# just for display, this is in realistic scenarios a very large dataframe\n", "(\n", " df_offdiag.sort(\"asset_id\", \"asset_id_right\")\n", " .with_columns(pl.lit(1))\n", " .pivot(\"asset_id_right\", index=\"asset_id\", maintain_order=True, sort_columns=True)\n", ")" ] }, { "cell_type": "code", "execution_count": 9, "id": "e1a36f1c", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (180, 5)
dateasset_ididio_returnasset_id_rightidio_return_right
datestrf32strf32
2026-01-02"IC28E9776F"null"IC28E9776F"null
2026-01-02"IC28E9776F"null"IC63213253"null
2026-01-02"IC63213253"null"IC63213253"null
2026-01-02"IC83A1B819"null"IC83A1B819"null
2026-01-02"IC83A1B819"null"ICA17F00B9"null
……………
2026-01-30"IC83A1B819"0.012707"ICA17F00B9"0.021744
2026-01-30"ICA17F00B9"0.021744"ICA17F00B9"0.021744
2026-01-30"IC83A1B819"0.012707"ICF982536B"-0.00373
2026-01-30"ICA17F00B9"0.021744"ICF982536B"-0.00373
2026-01-30"ICF982536B"-0.00373"ICF982536B"-0.00373
" ], "text/plain": [ "shape: (180, 5)\n", "┌────────────┬────────────┬─────────────┬────────────────┬───────────────────┐\n", "│ date ┆ asset_id ┆ idio_return ┆ asset_id_right ┆ idio_return_right │\n", "│ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ f32 ┆ str ┆ f32 │\n", "╞════════════╪════════════╪═════════════╪════════════════╪═══════════════════╡\n", "│ 2026-01-02 ┆ IC28E9776F ┆ null ┆ IC28E9776F ┆ null │\n", "│ 2026-01-02 ┆ IC28E9776F ┆ null ┆ IC63213253 ┆ null │\n", "│ 2026-01-02 ┆ IC63213253 ┆ null ┆ IC63213253 ┆ null │\n", "│ 2026-01-02 ┆ IC83A1B819 ┆ null ┆ IC83A1B819 ┆ null │\n", "│ 2026-01-02 ┆ IC83A1B819 ┆ null ┆ ICA17F00B9 ┆ null │\n", "│ … ┆ … ┆ … ┆ … ┆ … │\n", "│ 2026-01-30 ┆ IC83A1B819 ┆ 0.012707 ┆ ICA17F00B9 ┆ 0.021744 │\n", "│ 2026-01-30 ┆ ICA17F00B9 ┆ 0.021744 ┆ ICA17F00B9 ┆ 0.021744 │\n", "│ 2026-01-30 ┆ IC83A1B819 ┆ 0.012707 ┆ ICF982536B ┆ -0.00373 │\n", "│ 2026-01-30 ┆ ICA17F00B9 ┆ 0.021744 ┆ ICF982536B ┆ -0.00373 │\n", "│ 2026-01-30 ┆ ICF982536B ┆ -0.00373 ┆ ICF982536B ┆ -0.00373 │\n", "└────────────┴────────────┴─────────────┴────────────────┴───────────────────┘" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# join the time series such that we have each combination that we need to compute\n", "idio_ret_df_joined = (\n", " idio_ret_df\n", " .join(df_offdiag, on=\"asset_id\")\n", " .join(idio_ret_df, left_on=(\"date\", \"asset_id_right\"), right_on=(\"date\", \"asset_id\"))\n", ")\n", "idio_ret_df_joined" ] }, { "cell_type": "markdown", "id": "3ceef638", "metadata": {}, "source": [ "We compute the covariance matrix first, and then standardize into the correlation matrix. The computation of the covariance matrix relies on computing a rolling mean to correct for autocorrelation, and subsequently an exponentially weighted moving average. We then divide by the standard deviations to obtain the correlations." ] }, { "cell_type": "code", "execution_count": 10, "id": "04afcfaa", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (180, 6)
dateasset_ididio_returnasset_id_rightidio_return_rightidio_vcov
datestrf32strf32f32
2026-01-02"IC28E9776F"null"IC28E9776F"nullnull
2026-01-02"IC28E9776F"null"IC63213253"nullnull
2026-01-02"IC63213253"null"IC63213253"nullnull
2026-01-02"IC83A1B819"null"IC83A1B819"nullnull
2026-01-02"IC83A1B819"null"ICA17F00B9"nullnull
………………
2026-01-30"IC83A1B819"0.007446"ICA17F00B9"0.0049050.000025
2026-01-30"ICA17F00B9"0.004905"ICA17F00B9"0.0049050.000036
2026-01-30"IC83A1B819"0.007446"ICF982536B"-0.014498-8.7480e-7
2026-01-30"ICA17F00B9"0.004905"ICF982536B"-0.014498-0.000007
2026-01-30"ICF982536B"-0.014498"ICF982536B"-0.0144980.00004
" ], "text/plain": [ "shape: (180, 6)\n", "┌────────────┬────────────┬─────────────┬────────────────┬───────────────────┬────────────┐\n", "│ date ┆ asset_id ┆ idio_return ┆ asset_id_right ┆ idio_return_right ┆ idio_vcov │\n", "│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ f32 ┆ str ┆ f32 ┆ f32 │\n", "╞════════════╪════════════╪═════════════╪════════════════╪═══════════════════╪════════════╡\n", "│ 2026-01-02 ┆ IC28E9776F ┆ null ┆ IC28E9776F ┆ null ┆ null │\n", "│ 2026-01-02 ┆ IC28E9776F ┆ null ┆ IC63213253 ┆ null ┆ null │\n", "│ 2026-01-02 ┆ IC63213253 ┆ null ┆ IC63213253 ┆ null ┆ null │\n", "│ 2026-01-02 ┆ IC83A1B819 ┆ null ┆ IC83A1B819 ┆ null ┆ null │\n", "│ 2026-01-02 ┆ IC83A1B819 ┆ null ┆ ICA17F00B9 ┆ null ┆ null │\n", "│ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n", "│ 2026-01-30 ┆ IC83A1B819 ┆ 0.007446 ┆ ICA17F00B9 ┆ 0.004905 ┆ 0.000025 │\n", "│ 2026-01-30 ┆ ICA17F00B9 ┆ 0.004905 ┆ ICA17F00B9 ┆ 0.004905 ┆ 0.000036 │\n", "│ 2026-01-30 ┆ IC83A1B819 ┆ 0.007446 ┆ ICF982536B ┆ -0.014498 ┆ -8.7480e-7 │\n", "│ 2026-01-30 ┆ ICA17F00B9 ┆ 0.004905 ┆ ICF982536B ┆ -0.014498 ┆ -0.000007 │\n", "│ 2026-01-30 ┆ ICF982536B ┆ -0.014498 ┆ ICF982536B ┆ -0.014498 ┆ 0.00004 │\n", "└────────────┴────────────┴─────────────┴────────────────┴───────────────────┴────────────┘" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# compute the covariance matrix by first using a rolling mean (for overlap),\n", "# and then an exponential weighted moving average (for smoothing)\n", "overlap_window = 5\n", "half_life = 126\n", "\n", "# compute rolling means\n", "idio_ret_smoothed = (\n", " idio_ret_df.with_columns(pl.col(pl.Float32).fill_nan(None))\n", " .with_columns(\n", " pl.col(\"idio_return\").rolling_mean(window_size=overlap_window, min_samples=1).over(\"asset_id\"),\n", " )\n", ")\n", "# then join\n", "idio_ret_df_joined = (\n", " idio_ret_smoothed.join(df_offdiag, on=\"asset_id\")\n", " .join(idio_ret_smoothed, left_on=(\"date\", \"asset_id_right\"), right_on=(\"date\", \"asset_id\"))\n", ") \n", "# then EWMA of the product \n", "idio_vcov_df = ( \n", " idio_ret_df_joined \n", " .with_columns( \n", " (pl.col(\"idio_return\") * pl.col(\"idio_return_right\"))\n", " .ewm_mean(half_life=half_life)\n", " .over((\"asset_id\", \"asset_id_right\")) \n", " .alias(\"idio_vcov\")\n", " ) \n", ") \n", "idio_vcov_df" ] }, { "cell_type": "code", "execution_count": 11, "id": "47e085ab", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (100, 3)
dateasset_ididio_var
datestrf32
2026-01-02"IC28E9776F"null
2026-01-02"IC63213253"null
2026-01-02"IC83A1B819"null
2026-01-02"ICA17F00B9"null
2026-01-02"ICF982536B"null
………
2026-01-30"IC28E9776F"0.000079
2026-01-30"IC63213253"0.000035
2026-01-30"IC83A1B819"0.000101
2026-01-30"ICA17F00B9"0.000036
2026-01-30"ICF982536B"0.00004
" ], "text/plain": [ "shape: (100, 3)\n", "┌────────────┬────────────┬──────────┐\n", "│ date ┆ asset_id ┆ idio_var │\n", "│ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ f32 │\n", "╞════════════╪════════════╪══════════╡\n", "│ 2026-01-02 ┆ IC28E9776F ┆ null │\n", "│ 2026-01-02 ┆ IC63213253 ┆ null │\n", "│ 2026-01-02 ┆ IC83A1B819 ┆ null │\n", "│ 2026-01-02 ┆ ICA17F00B9 ┆ null │\n", "│ 2026-01-02 ┆ ICF982536B ┆ null │\n", "│ … ┆ … ┆ … │\n", "│ 2026-01-30 ┆ IC28E9776F ┆ 0.000079 │\n", "│ 2026-01-30 ┆ IC63213253 ┆ 0.000035 │\n", "│ 2026-01-30 ┆ IC83A1B819 ┆ 0.000101 │\n", "│ 2026-01-30 ┆ ICA17F00B9 ┆ 0.000036 │\n", "│ 2026-01-30 ┆ ICF982536B ┆ 0.00004 │\n", "└────────────┴────────────┴──────────┘" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# to translate the covariance matrix to a correlation matrix, \n", "# we need to select the variance of the idiosyncratic returns\n", "idio_var_df = (\n", " idio_vcov_df.filter(pl.col(\"asset_id\") == pl.col(\"asset_id_right\"))\n", " .select(\"date\", \"asset_id\", pl.col(\"idio_vcov\").alias(\"idio_var\"))\n", ")\n", "idio_var_df" ] }, { "cell_type": "code", "execution_count": 12, "id": "8ff6ce1d", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (180, 4)
dateasset_idasset_id_rightidio_corr
datestrstrf32
2026-01-02"IC28E9776F""IC28E9776F"null
2026-01-02"IC28E9776F""IC63213253"null
2026-01-02"IC63213253""IC63213253"null
2026-01-02"IC83A1B819""IC83A1B819"null
2026-01-02"IC83A1B819""ICA17F00B9"null
…………
2026-01-30"IC83A1B819""ICA17F00B9"0.408816
2026-01-30"ICA17F00B9""ICA17F00B9"1.0
2026-01-30"IC83A1B819""ICF982536B"-0.013701
2026-01-30"ICA17F00B9""ICF982536B"-0.174511
2026-01-30"ICF982536B""ICF982536B"1.0
" ], "text/plain": [ "shape: (180, 4)\n", "┌────────────┬────────────┬────────────────┬───────────┐\n", "│ date ┆ asset_id ┆ asset_id_right ┆ idio_corr │\n", "│ --- ┆ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ str ┆ f32 │\n", "╞════════════╪════════════╪════════════════╪═══════════╡\n", "│ 2026-01-02 ┆ IC28E9776F ┆ IC28E9776F ┆ null │\n", "│ 2026-01-02 ┆ IC28E9776F ┆ IC63213253 ┆ null │\n", "│ 2026-01-02 ┆ IC63213253 ┆ IC63213253 ┆ null │\n", "│ 2026-01-02 ┆ IC83A1B819 ┆ IC83A1B819 ┆ null │\n", "│ 2026-01-02 ┆ IC83A1B819 ┆ ICA17F00B9 ┆ null │\n", "│ … ┆ … ┆ … ┆ … │\n", "│ 2026-01-30 ┆ IC83A1B819 ┆ ICA17F00B9 ┆ 0.408816 │\n", "│ 2026-01-30 ┆ ICA17F00B9 ┆ ICA17F00B9 ┆ 1.0 │\n", "│ 2026-01-30 ┆ IC83A1B819 ┆ ICF982536B ┆ -0.013701 │\n", "│ 2026-01-30 ┆ ICA17F00B9 ┆ ICF982536B ┆ -0.174511 │\n", "│ 2026-01-30 ┆ ICF982536B ┆ ICF982536B ┆ 1.0 │\n", "└────────────┴────────────┴────────────────┴───────────┘" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# by joining twice and normalizing, we get the correlation matrix\n", "idio_corr_df = (\n", " idio_vcov_df.join(idio_var_df, on=(\"date\", \"asset_id\"))\n", " .join(idio_var_df, left_on=(\"date\", \"asset_id_right\"), right_on=(\"date\", \"asset_id\"))\n", " .select(\n", " \"date\",\n", " \"asset_id\",\n", " \"asset_id_right\",\n", " (pl.col(\"idio_vcov\") / (pl.col(\"idio_var\") * pl.col(\"idio_var_right\")).sqrt()).alias(\"idio_corr\"))\n", ")\n", "idio_corr_df\n" ] }, { "cell_type": "code", "execution_count": 13, "id": "9fb7a037", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (95, 7)
dateasset_idIC28E9776FIC63213253IC83A1B819ICA17F00B9ICF982536B
datestrf32f32f32f32f32
2026-01-05"IC28E9776F"1.01.0nullnullnull
2026-01-05"IC63213253"null1.0nullnullnull
2026-01-05"IC83A1B819"nullnull1.01.01.0
2026-01-05"ICA17F00B9"nullnullnull1.01.0
2026-01-05"ICF982536B"nullnullnullnull1.0
…………………
2026-01-30"IC28E9776F"1.00.638806nullnullnull
2026-01-30"IC63213253"null1.0nullnullnull
2026-01-30"IC83A1B819"nullnull1.00.408816-0.013701
2026-01-30"ICA17F00B9"nullnullnull1.0-0.174511
2026-01-30"ICF982536B"nullnullnullnull1.0
" ], "text/plain": [ "shape: (95, 7)\n", "┌────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐\n", "│ date ┆ asset_id ┆ IC28E9776F ┆ IC63213253 ┆ IC83A1B819 ┆ ICA17F00B9 ┆ ICF982536B │\n", "│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ f32 ┆ f32 ┆ f32 ┆ f32 ┆ f32 │\n", "╞════════════╪════════════╪════════════╪════════════╪════════════╪════════════╪════════════╡\n", "│ 2026-01-05 ┆ IC28E9776F ┆ 1.0 ┆ 1.0 ┆ null ┆ null ┆ null │\n", "│ 2026-01-05 ┆ IC63213253 ┆ null ┆ 1.0 ┆ null ┆ null ┆ null │\n", "│ 2026-01-05 ┆ IC83A1B819 ┆ null ┆ null ┆ 1.0 ┆ 1.0 ┆ 1.0 │\n", "│ 2026-01-05 ┆ ICA17F00B9 ┆ null ┆ null ┆ null ┆ 1.0 ┆ 1.0 │\n", "│ 2026-01-05 ┆ ICF982536B ┆ null ┆ null ┆ null ┆ null ┆ 1.0 │\n", "│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n", "│ 2026-01-30 ┆ IC28E9776F ┆ 1.0 ┆ 0.638806 ┆ null ┆ null ┆ null │\n", "│ 2026-01-30 ┆ IC63213253 ┆ null ┆ 1.0 ┆ null ┆ null ┆ null │\n", "│ 2026-01-30 ┆ IC83A1B819 ┆ null ┆ null ┆ 1.0 ┆ 0.408816 ┆ -0.013701 │\n", "│ 2026-01-30 ┆ ICA17F00B9 ┆ null ┆ null ┆ null ┆ 1.0 ┆ -0.174511 │\n", "│ 2026-01-30 ┆ ICF982536B ┆ null ┆ null ┆ null ┆ null ┆ 1.0 │\n", "└────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# for small portfolios, the dataframe is small enough to pivot and display\n", "(\n", " idio_corr_df.pivot(\"asset_id_right\", index=(\"date\", \"asset_id\"), maintain_order=True, sort_columns=True)\n", " .filter(pl.col(\"date\") > pl.col(\"date\").min())\n", ")" ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.15" } }, "nbformat": 4, "nbformat_minor": 5 }