{ "cells": [ { "cell_type": "markdown", "id": "bedb01fa", "metadata": {}, "source": [ "# Exposure Upload Commit Modes\n", "\n", "In this tutorial we are going to demonstrate the usage of the different **commit modes** for the **Exposure Uploader**.\n", "\n", "As a prerequisite see the [Uploaders Tutorial](https://docs.bayesline.com/0.12.1/notebooks/tutorial_uploaders.html) for a comprehensive overview of the **Data Uploaders API**.\n", "\n", "Specifically, we will explore:\n", "* [Initial exposure upload](#initial-exposure-upload)\n", "* [Overwriting exposures](#overwriting-exposures)\n", "* [Appending arbitrary records](#appending-arbitrary-records)\n", "* [Appending new exposures as they arrive](#appending-new-exposures)\n", "* [Adding new factors](#adding-new-factors)\n", "* [Replacing factors](#replacing-factors)\n", "\n", "For this tutorial we will use a very small dataframe which makes it easier to point out the behavior of the different upload modes. We test our implementation on global datasets with 20 years of history to ensure they scale in a performant way.\n", "\n", "For simplicity we'll be using the [`fast_commit`](https://docs.bayesline.com/0.12.1/_autosummary/bayesline.api.equity.UploaderApi.fast_commit.html#bayesline.api.equity.UploaderApi.fast_commit) functionality and skip the [staging step](https://docs.bayesline.com/0.12.1/notebooks/tutorial_uploaders.html#staging-data)." ] }, { "cell_type": "markdown", "id": "10159433", "metadata": {}, "source": [ "## Imports & Setup\n", "\n", "For this tutorial notebook, you will need to import the following packages." ] }, { "cell_type": "code", "execution_count": 2, "id": "2e0f35e7", "metadata": {}, "outputs": [], "source": [ "import io\n", "\n", "import pandas as pd\n", "import polars as pl\n", "\n", "from bayesline.apiclient import BayeslineApiClient" ] }, { "cell_type": "markdown", "id": "8793821c", "metadata": {}, "source": [ "We will also need to have a Bayesline API client configured." ] }, { "cell_type": "code", "execution_count": null, "id": "164df105", "metadata": { "tags": [ "skip-execution" ] }, "outputs": [], "source": [ "bln = BayeslineApiClient.new_client(\n", " endpoint=\"https://[ENDPOINT]\",\n", " api_key=\"[API-KEY]\",\n", ")" ] }, { "cell_type": "code", "execution_count": 4, "id": "dc5ea9c9", "metadata": {}, "outputs": [], "source": [ "uploaders = bln.equity.uploaders\n", "exposure_uploaders = uploaders.get_data_type(\"exposures\")\n", "\n", "my_exposure_dataset = exposure_uploaders.create_or_replace_dataset(\"Test-Exposure-Modes\")" ] }, { "cell_type": "markdown", "id": "fb728069", "metadata": {}, "source": [ "Finally we are adding a helper function below to help us print out the data changes between versions." ] }, { "cell_type": "code", "execution_count": 5, "id": "de243f98", "metadata": {}, "outputs": [], "source": [ "def highlight_changes(old: pl.DataFrame, new: pl.DataFrame):\n", " cols = [\"date\", \"asset_id\", \"asset_id_type\", \"factor_group\", \"factor\", \"exposure\"]\n", " unchanged = pl.concat([old, new]).group_by(cols).len().filter(pl.col(\"len\") > 1).drop(\"len\").to_pandas()\n", " dropped = old.join(new, on=cols[:-1], how=\"anti\").to_pandas()\n", " added = new.join(old, on=cols[:-1], how=\"anti\").to_pandas()\n", "\n", " df = pd.concat([added, dropped, unchanged], ignore_index=True)\n", " mask = pd.concat([\n", " pd.DataFrame({col: [\"background-color: #d4edda\"] * len(added) for col in cols}),\n", " pd.DataFrame({col: [\"background-color: #f8d7da; text-decoration: line-through\"] * len(dropped) for col in cols}),\n", " pd.DataFrame({col: [\"\"] * len(unchanged) for col in cols}),\n", " ], ignore_index=True)\n", " return df.style.apply(lambda _: mask, axis=None)" ] }, { "cell_type": "markdown", "id": "7abbadaa", "metadata": {}, "source": [ "## Availabe Commit Modes\n", "\n", "Each uploader has different commit modes that are tailored to the type of data that is being handled. For example for exposures a common use case is to add a new set of factors, but leaving any existing factors untouched. For this use case the exposures uploader provides a dedicated commit mode which we'll explore later.\n", "\n", "Below we show how to obtain the available commit modes together with a description of their behavior." ] }, { "cell_type": "code", "execution_count": 6, "id": "5b1e98ec", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'append': 'Appends new factor/date combinations to the existing data. Collisions will be ignored.',\n", " 'append_factor': 'Appends new factors to the existing data. Collisions with existing factors will be ignored.',\n", " 'overwrite': 'Overwrites the entire dataset with the new data.',\n", " 'overwrite_factor': 'Overwrites every factor present in the incoming data.',\n", " 'append_from': 'Appends new factor/date combinations to the existing data but only after the last date in the existing data. Collisions will be ignored.',\n", " 'overwrite_from': 'Overwrites the entire dataset with the new data but only after the last date in the existing data.'}" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "my_exposure_dataset.get_commit_modes()" ] }, { "cell_type": "markdown", "id": "c4a60927", "metadata": {}, "source": [ "## Initial Exposure Upload\n", "\n", "The initial exposure upload is applied against a new exposure dataset. For this we can use the `append` mode but `overwrite` or `upsert` would work just the same.\n", "\n", "Below we create our starting dataframe which we'll modify as we go through this tutorial. Recall that we can get examples of the required input format from the API as well." ] }, { "cell_type": "code", "execution_count": 7, "id": "5424d890", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (3, 9)
dateasset_idasset_id_typestyle^momentum_6style^momentum_12style^growthmarket^marketindustry^consumerindustry^tech
datestrstrf64f64f64f64f64f64
2025-01-06"GOOG""cusip9"-0.3-0.21.21.0null1.0
2025-01-06"AAPL""cusip9"0.10.51.11.01.0null
2025-01-07"GOOG""cusip9"-0.28-0.191.211.0null1.0
" ], "text/plain": [ "shape: (3, 9)\n", "┌───────────┬──────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐\n", "│ date ┆ asset_id ┆ asset_id_ ┆ style^mom ┆ … ┆ style^gro ┆ market^ma ┆ industry^ ┆ industry^ │\n", "│ --- ┆ --- ┆ type ┆ entum_6 ┆ ┆ wth ┆ rket ┆ consumer ┆ tech │\n", "│ date ┆ str ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n", "│ ┆ ┆ str ┆ f64 ┆ ┆ f64 ┆ f64 ┆ f64 ┆ f64 │\n", "╞═══════════╪══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡\n", "│ 2025-01-0 ┆ GOOG ┆ cusip9 ┆ -0.3 ┆ … ┆ 1.2 ┆ 1.0 ┆ null ┆ 1.0 │\n", "│ 6 ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n", "│ 2025-01-0 ┆ AAPL ┆ cusip9 ┆ 0.1 ┆ … ┆ 1.1 ┆ 1.0 ┆ 1.0 ┆ null │\n", "│ 6 ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n", "│ 2025-01-0 ┆ GOOG ┆ cusip9 ┆ -0.28 ┆ … ┆ 1.21 ┆ 1.0 ┆ null ┆ 1.0 │\n", "│ 7 ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n", "└───────────┴──────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "my_exposure_dataset.get_parser(\"Wide-Format\").get_examples()[0]" ] }, { "cell_type": "code", "execution_count": 8, "id": "2bfc10b8", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (4, 4)
dateasset_idasset_id_typestyle^momentum
datestrstrf64
2025-01-01"AAPL""ticker"1.0
2025-01-01"GOOG""ticker"2.0
2025-01-02"AAPL""ticker"1.5
2025-01-02"GOOG""ticker"2.5
" ], "text/plain": [ "shape: (4, 4)\n", "┌────────────┬──────────┬───────────────┬────────────────┐\n", "│ date ┆ asset_id ┆ asset_id_type ┆ style^momentum │\n", "│ --- ┆ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ str ┆ f64 │\n", "╞════════════╪══════════╪═══════════════╪════════════════╡\n", "│ 2025-01-01 ┆ AAPL ┆ ticker ┆ 1.0 │\n", "│ 2025-01-01 ┆ GOOG ┆ ticker ┆ 2.0 │\n", "│ 2025-01-02 ┆ AAPL ┆ ticker ┆ 1.5 │\n", "│ 2025-01-02 ┆ GOOG ┆ ticker ┆ 2.5 │\n", "└────────────┴──────────┴───────────────┴────────────────┘" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_csv = \"\"\"\n", "date\tasset_id\tasset_id_type\tstyle^momentum\n", "2025-01-01\tAAPL\tticker\t1\n", "2025-01-01\tGOOG\tticker\t2\n", "2025-01-02\tAAPL\tticker\t1.5\n", "2025-01-02\tGOOG\tticker\t2.5\n", "\"\"\"\n", "\n", "exposures_df = pl.read_csv(io.StringIO(data_csv.strip()), separator=\"\\t\", try_parse_dates=True)\n", "\n", "exposures_df" ] }, { "cell_type": "code", "execution_count": 9, "id": "48217c7d", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "UploadCommitResult(version=1, committed_names=[])" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "my_exposure_dataset.fast_commit(exposures_df, mode=\"append\")" ] }, { "cell_type": "markdown", "id": "ab34e721", "metadata": {}, "source": [ "If we now confirm what was commited it's (unsurprisingly) exactly the dataset as we uploaded it (after parsing)." ] }, { "cell_type": "code", "execution_count": 10, "id": "8dbc3783", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
 dateasset_idasset_id_typefactor_groupfactorexposure
02025-01-01 00:00:00AAPLtickerstylemomentum1.000000
12025-01-01 00:00:00GOOGtickerstylemomentum2.000000
22025-01-02 00:00:00AAPLtickerstylemomentum1.500000
32025-01-02 00:00:00GOOGtickerstylemomentum2.500000
\n" ], "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "highlight_changes(\n", " pl.DataFrame(schema=my_exposure_dataset.get_schema()),\n", " my_exposure_dataset.get_data(version=1).collect()\n", ")" ] }, { "cell_type": "markdown", "id": "953efe3d", "metadata": {}, "source": [ "## Overwriting exposures\n", "\n", "Next up we say 'we made a mistake and would like to `overwrite` the entire dataset'. Note that the easiest way to do this is to delete the dataset entirely, which would also delete the entire version history." ] }, { "cell_type": "code", "execution_count": 12, "id": "1182c0ef", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "UploadCommitResult(version=2, committed_names=[])" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_csv = \"\"\"\n", "date\tasset_id\tasset_id_type\tstyle^value\n", "2025-01-03\tTSLA\tticker\t8\n", "2025-01-03\tAMZN\tticker\t9\n", "2025-01-04\tTSLA\tticker\t10\n", "2025-01-04\tAMZN\tticker\t11\n", "\"\"\"\n", "\n", "exposures_df = pl.read_csv(io.StringIO(data_csv.strip()), separator=\"\\t\", try_parse_dates=True)\n", "\n", "my_exposure_dataset.fast_commit(exposures_df, mode=\"overwrite\")" ] }, { "cell_type": "markdown", "id": "c86a0db2", "metadata": {}, "source": [ "As expected the data has been completely overridden." ] }, { "cell_type": "code", "execution_count": 13, "id": "f005c01c", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
 dateasset_idasset_id_typefactor_groupfactorexposure
02025-01-03 00:00:00TSLAtickerstylevalue8.000000
12025-01-03 00:00:00AMZNtickerstylevalue9.000000
22025-01-04 00:00:00TSLAtickerstylevalue10.000000
32025-01-04 00:00:00AMZNtickerstylevalue11.000000
42025-01-01 00:00:00AAPLtickerstylemomentum1.000000
52025-01-01 00:00:00GOOGtickerstylemomentum2.000000
62025-01-02 00:00:00AAPLtickerstylemomentum1.500000
72025-01-02 00:00:00GOOGtickerstylemomentum2.500000
\n" ], "text/plain": [ "" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "highlight_changes(\n", " my_exposure_dataset.get_data(version=1).collect(),\n", " my_exposure_dataset.get_data(version=2).collect()\n", ")" ] }, { "cell_type": "markdown", "id": "78d71a19", "metadata": {}, "source": [ "Note that we can still recover the previous version." ] }, { "cell_type": "code", "execution_count": 15, "id": "a3dc7932", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "shape: (4, 6)
dateasset_idasset_id_typefactor_groupfactorexposure
datestrstrstrstrf32
2025-01-01"AAPL""ticker""style""momentum"1.0
2025-01-01"GOOG""ticker""style""momentum"2.0
2025-01-02"AAPL""ticker""style""momentum"1.5
2025-01-02"GOOG""ticker""style""momentum"2.5
" ], "text/plain": [ "shape: (4, 6)\n", "┌────────────┬──────────┬───────────────┬──────────────┬──────────┬──────────┐\n", "│ date ┆ asset_id ┆ asset_id_type ┆ factor_group ┆ factor ┆ exposure │\n", "│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n", "│ date ┆ str ┆ str ┆ str ┆ str ┆ f32 │\n", "╞════════════╪══════════╪═══════════════╪══════════════╪══════════╪══════════╡\n", "│ 2025-01-01 ┆ AAPL ┆ ticker ┆ style ┆ momentum ┆ 1.0 │\n", "│ 2025-01-01 ┆ GOOG ┆ ticker ┆ style ┆ momentum ┆ 2.0 │\n", "│ 2025-01-02 ┆ AAPL ┆ ticker ┆ style ┆ momentum ┆ 1.5 │\n", "│ 2025-01-02 ┆ GOOG ┆ ticker ┆ style ┆ momentum ┆ 2.5 │\n", "└────────────┴──────────┴───────────────┴──────────────┴──────────┴──────────┘" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "my_exposure_dataset.get_data(version=1).collect()" ] }, { "cell_type": "markdown", "id": "c5092e67", "metadata": {}, "source": [ "## Appending Arbitrary Records\n", "\n", "`append` means **don't overwrite existing records**. It's purpose is to prevent accidentally changing already uploaded data. For exposures a record already exists if for two records the `date`, `asset_id`, `asset_id_type`, `factor_group` and `factor` match. This means that any upload that uses the `append` mode will ignore (i.e. throw away) any record that already exists in the version that is appended to. \n", "\n", "To show this with an example, we append below dataframe. Note how the record for `TSLA` on `2025-01-04` already exists with value `10.0` and note how after the commit this value stays unchanged. \n", "\n", "`append` is the most generalized form of adding new data without overwriting existing data. In this example we append time (a new date), a new factor and we bring in a new asset as well. Later we'll see how `append_from` and `append_factor` adds more protections to prevent accidentally overwriting existing data.\n", "\n", "Also note that we're using long format for this upload, an alternative option for formatting the input data prior to ingestion." ] }, { "cell_type": "code", "execution_count": 17, "id": "29884b4d", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "UploadCommitResult(version=3, committed_names=[])" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_csv = \"\"\"\n", "date\tasset_id\tasset_id_type\tfactor_group\tfactor\texposure\n", "2025-01-03\tXOM\tticker\tstyle\tgrowth\t50\n", "2025-01-04\tXOM\tticker\tstyle\tgrowth\t51\n", "2025-01-05\tXOM\tticker\tstyle\tgrowth\t51\n", "2025-01-03\tXOM\tticker\tstyle\tvalue\t12\n", "2025-01-04\tXOM\tticker\tstyle\tvalue\t13\n", "2025-01-05\tXOM\tticker\tstyle\tvalue\t14\n", "2025-01-04\tTSLA\tticker\tstyle\tvalue\t9999\n", "\n", "\"\"\"\n", "\n", "exposures_df = pl.read_csv(io.StringIO(data_csv.strip()), separator=\"\\t\", try_parse_dates=True)\n", "\n", "my_exposure_dataset.fast_commit(exposures_df, mode=\"append\")" ] }, { "cell_type": "code", "execution_count": 18, "id": "68b56648", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
 dateasset_idasset_id_typefactor_groupfactorexposure
02025-01-03 00:00:00XOMtickerstylegrowth50.000000
12025-01-03 00:00:00XOMtickerstylevalue12.000000
22025-01-04 00:00:00XOMtickerstylegrowth51.000000
32025-01-05 00:00:00XOMtickerstylegrowth51.000000
42025-01-04 00:00:00XOMtickerstylevalue13.000000
52025-01-05 00:00:00XOMtickerstylevalue14.000000
62025-01-04 00:00:00TSLAtickerstylevalue10.000000
72025-01-03 00:00:00TSLAtickerstylevalue8.000000
82025-01-03 00:00:00AMZNtickerstylevalue9.000000
92025-01-04 00:00:00AMZNtickerstylevalue11.000000
\n" ], "text/plain": [ "" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "highlight_changes(\n", " my_exposure_dataset.get_data(version=2).collect(),\n", " my_exposure_dataset.get_data(version=3).collect()\n", ")" ] }, { "cell_type": "markdown", "id": "51eb47d0", "metadata": {}, "source": [ "## Appending New Exposures\n", "\n", "Day over day new exposures arrive which need to be appended to the existing exposure upload. For this use case we may want to ensure that we don't accidentally add any data for dates that already exist.\n", "\n", "Consider the example where our last upload was on `2025-01-05` and we are now appending exposures `2025-01-06` and `2025-01-07`. If in our input data some records from `2025-01-05` and before slipped in which **does not** already exist in the uploaded exposures (e.g. exposures with new assets were added), using `append` would insert these new records and therefore introduce new data into historical dates. This may or may not be desired depending on use case.\n", "\n", "The commit mode `append_from` was designed to **only** append data that is strictly **after** the last date of the existing data, therefore ensuring no historical data accidentally slips in.\n", "\n", "In the example below we add data for the date `2025-01-06` but also add a new historical record for asset `MSFT` for date `2025-01-05`. Note how this record won't make it through to the committed data." ] }, { "cell_type": "code", "execution_count": 20, "id": "69309697", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "UploadCommitResult(version=4, committed_names=[])" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_csv = \"\"\"\n", "date\tasset_id\tasset_id_type\tstyle^value\tstyle^growth\n", "2025-01-05\tMSFT\tticker\t1000\t2000\n", "2025-01-06\tAMZN\tticker\t40\t70\n", "2025-01-06\tTSLA\tticker\t50\t80\n", "2025-01-06\tXOM\tticker\t60\t90\n", "\"\"\"\n", "\n", "exposures_df = pl.read_csv(io.StringIO(data_csv.strip()), separator=\"\\t\", try_parse_dates=True)\n", "\n", "my_exposure_dataset.fast_commit(exposures_df, mode=\"append_from\")" ] }, { "cell_type": "code", "execution_count": 21, "id": "c67ee4d1", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
 dateasset_idasset_id_typefactor_groupfactorexposure
02025-01-06 00:00:00AMZNtickerstylevalue40.000000
12025-01-06 00:00:00AMZNtickerstylegrowth70.000000
22025-01-06 00:00:00TSLAtickerstylevalue50.000000
32025-01-06 00:00:00TSLAtickerstylegrowth80.000000
42025-01-06 00:00:00XOMtickerstylevalue60.000000
52025-01-06 00:00:00XOMtickerstylegrowth90.000000
62025-01-04 00:00:00XOMtickerstylegrowth51.000000
72025-01-03 00:00:00XOMtickerstylevalue12.000000
82025-01-04 00:00:00AMZNtickerstylevalue11.000000
92025-01-05 00:00:00XOMtickerstylegrowth51.000000
102025-01-03 00:00:00TSLAtickerstylevalue8.000000
112025-01-03 00:00:00XOMtickerstylegrowth50.000000
122025-01-04 00:00:00TSLAtickerstylevalue10.000000
132025-01-04 00:00:00XOMtickerstylevalue13.000000
142025-01-03 00:00:00AMZNtickerstylevalue9.000000
152025-01-05 00:00:00XOMtickerstylevalue14.000000
\n" ], "text/plain": [ "" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "highlight_changes(\n", " my_exposure_dataset.get_data(version=3).collect(),\n", " my_exposure_dataset.get_data(version=4).collect()\n", ")" ] }, { "cell_type": "markdown", "id": "689c4e51", "metadata": {}, "source": [ "# Adding New Factors\n", "\n", "Adding new factors works similarly to appending new exposures, only that here we ensure that **no existing factors** are edited. Any incoming record that contains a factor name that already exists will be discarded.\n", "\n", "Below example uses the `append_factor` mode but the input dataframe also contains values for the existing factor *value*. Note how these records do not make it into the committed dataframe. \n", "\n", "One possible use case for this mode is if the incoming data comes from a single spreadsheet and an additional factor was added to this spreadsheet. We want to ignore all factors that we have already captured." ] }, { "cell_type": "code", "execution_count": 23, "id": "fc10fd57", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "UploadCommitResult(version=5, committed_names=[])" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_csv = \"\"\"\n", "date\tasset_id\tasset_id_type\tstyle^value\tstyle^momentum\n", "2025-01-03\tJPM\tticker\t1000\t-5\n", "2025-01-03\tMSFT\tticker\t1000\t-10\n", "2025-01-04\tMSFT\tticker\t2000\t-20\n", "2025-01-05\tMSFT\tticker\t3000\t-30\n", "2025-01-06\tMSFT\tticker\t4000\t-40\n", "\"\"\"\n", "\n", "exposures_df = pl.read_csv(io.StringIO(data_csv.strip()), separator=\"\\t\", try_parse_dates=True)\n", "\n", "my_exposure_dataset.fast_commit(exposures_df, mode=\"append_factor\")" ] }, { "cell_type": "code", "execution_count": 24, "id": "43de0973", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
 dateasset_idasset_id_typefactor_groupfactorexposure
02025-01-03 00:00:00JPMtickerstylemomentum-5.000000
12025-01-03 00:00:00MSFTtickerstylemomentum-10.000000
22025-01-04 00:00:00MSFTtickerstylemomentum-20.000000
32025-01-05 00:00:00MSFTtickerstylemomentum-30.000000
42025-01-06 00:00:00MSFTtickerstylemomentum-40.000000
52025-01-03 00:00:00TSLAtickerstylevalue8.000000
62025-01-03 00:00:00AMZNtickerstylevalue9.000000
72025-01-06 00:00:00XOMtickerstylegrowth90.000000
82025-01-05 00:00:00XOMtickerstylevalue14.000000
92025-01-05 00:00:00XOMtickerstylegrowth51.000000
102025-01-04 00:00:00XOMtickerstylevalue13.000000
112025-01-06 00:00:00AMZNtickerstylegrowth70.000000
122025-01-04 00:00:00AMZNtickerstylevalue11.000000
132025-01-03 00:00:00XOMtickerstylevalue12.000000
142025-01-04 00:00:00TSLAtickerstylevalue10.000000
152025-01-06 00:00:00XOMtickerstylevalue60.000000
162025-01-04 00:00:00XOMtickerstylegrowth51.000000
172025-01-03 00:00:00XOMtickerstylegrowth50.000000
182025-01-06 00:00:00TSLAtickerstylevalue50.000000
192025-01-06 00:00:00TSLAtickerstylegrowth80.000000
202025-01-06 00:00:00AMZNtickerstylevalue40.000000
\n" ], "text/plain": [ "" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "highlight_changes(\n", " my_exposure_dataset.get_data(version=4).collect(),\n", " my_exposure_dataset.get_data(version=5).collect()\n", ")" ] }, { "cell_type": "markdown", "id": "47215b05", "metadata": {}, "source": [ "## Replacing Factors\n", "\n", "Replacing factors is the inverse of the behavior of appending factors. Any factor that has at least one record in the incoming dataframe will be wiped from the existing dataset and fully replaced with the incoming factor data.\n", "\n", "In below example we fully replace the growth and value factors." ] }, { "cell_type": "code", "execution_count": 25, "id": "83eaab88", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "UploadCommitResult(version=6, committed_names=[])" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_csv = \"\"\"\n", "date\tasset_id\tasset_id_type\tstyle^value\tstyle^growth\n", "2025-01-05\tAVGO\tticker\t1000\t2000\n", "\"\"\"\n", "\n", "exposures_df = pl.read_csv(io.StringIO(data_csv.strip()), separator=\"\\t\", try_parse_dates=True)\n", "\n", "my_exposure_dataset.fast_commit(exposures_df, mode=\"overwrite_factor\")" ] }, { "cell_type": "code", "execution_count": 26, "id": "69520e24", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
 dateasset_idasset_id_typefactor_groupfactorexposure
02025-01-05 00:00:00AVGOtickerstylevalue1000.000000
12025-01-05 00:00:00AVGOtickerstylegrowth2000.000000
22025-01-03 00:00:00TSLAtickerstylevalue8.000000
32025-01-03 00:00:00AMZNtickerstylevalue9.000000
42025-01-04 00:00:00TSLAtickerstylevalue10.000000
52025-01-04 00:00:00AMZNtickerstylevalue11.000000
62025-01-03 00:00:00XOMtickerstylegrowth50.000000
72025-01-03 00:00:00XOMtickerstylevalue12.000000
82025-01-04 00:00:00XOMtickerstylegrowth51.000000
92025-01-05 00:00:00XOMtickerstylegrowth51.000000
102025-01-04 00:00:00XOMtickerstylevalue13.000000
112025-01-05 00:00:00XOMtickerstylevalue14.000000
122025-01-06 00:00:00AMZNtickerstylevalue40.000000
132025-01-06 00:00:00AMZNtickerstylegrowth70.000000
142025-01-06 00:00:00TSLAtickerstylevalue50.000000
152025-01-06 00:00:00TSLAtickerstylegrowth80.000000
162025-01-06 00:00:00XOMtickerstylevalue60.000000
172025-01-06 00:00:00XOMtickerstylegrowth90.000000
182025-01-06 00:00:00MSFTtickerstylemomentum-40.000000
192025-01-04 00:00:00MSFTtickerstylemomentum-20.000000
202025-01-05 00:00:00MSFTtickerstylemomentum-30.000000
212025-01-03 00:00:00JPMtickerstylemomentum-5.000000
222025-01-03 00:00:00MSFTtickerstylemomentum-10.000000
\n" ], "text/plain": [ "" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "highlight_changes(\n", " my_exposure_dataset.get_data(version=5).collect(),\n", " my_exposure_dataset.get_data(version=6).collect()\n", ")" ] }, { "cell_type": "markdown", "id": "c472cbc7", "metadata": {}, "source": [ "## Housekeeping" ] }, { "cell_type": "code", "execution_count": 27, "id": "4569c490", "metadata": {}, "outputs": [], "source": [ "my_exposure_dataset.destroy()" ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.15" } }, "nbformat": 4, "nbformat_minor": 5 }