diff --git a/notebooks/README.md b/notebooks/README.md index c18a50c2..e125004a 100644 --- a/notebooks/README.md +++ b/notebooks/README.md @@ -1,3 +1,14 @@ # Python API Notebooks -This directory contains Colab notebooks that use the Python API. +This directory contains Colab notebooks that use the Data Commons Python client +and SDMX APIs. + +## SDMX notebooks + +These notebooks use SDMX to retrieve statistical observations and the Data Commons +Python client to retrieve place names. Each notebook installs its dependencies +and includes an **Open in Colab** link. + +- [Analyzing census data](analyzing_census_data_sdmx.ipynb) +- [Analyzing income distribution](analyzing_income_distribution_sdmx.ipynb) +- [Analyzing obesity prevalence](analyzing_obesity_prevalence_sdmx.ipynb) diff --git a/notebooks/analyzing_census_data_sdmx.ipynb b/notebooks/analyzing_census_data_sdmx.ipynb new file mode 100644 index 00000000..48c9cd82 --- /dev/null +++ b/notebooks/analyzing_census_data_sdmx.ipynb @@ -0,0 +1,476 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Copyright 2026 Google LLC.\n", + "\n", + "SPDX-License-Identifier: Apache-2.0\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\"Open\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Analyzing Census Data with Data Commons SDMX\n", + "\n", + "Every year, the American Community Survey (published by the US Census) reports thousands of variables about demographics, economics, housing, and more. This information, stored in Data Commons, is available to everyone for data science projects, education, and exploration. This tutorial introduces the Data Commons graph and two of its tools to help integrate its data into data science projects: the knowledge graph [browser](https://datacommons.org/browser) and the SDMX APIs used from Python.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## What is Data Commons?\n", + "\n", + "Data Commons is an open knowledge repository that combines data from public datasets using mapped common entities. It contains statements about real-world objects such as:\n", + "\n", + "- [Santa Clara County](https://datacommons.org/browser/geoId/06085) is contained in the [State of California](https://datacommons.org/browser/geoId/06).\n", + "- The latitude of [Berkeley, CA](https://datacommons.org/browser/geoId/0606000) is 37.8703.\n", + "- [The population of Maryland](https://datacommons.org/browser/geoId/24) was [6.26 M in 2024](https://datacommons.org/browser/geoId/24?statVar=Count_Person).\n", + "\n", + "In the graph, [entities](https://docs.datacommons.org/data_model.html#entity) like Santa Clara County are represented by nodes. Every node has a type corresponding to what the node represents. For example, California is a [State](https://datacommons.org/browser/State). Relations between entities are represented by edges between these nodes. The statement \"Santa Clara County is contained in the State of California\" is represented by two nodes connected by an edge labeled [containedInPlace](https://datacommons.org/browser/containedInPlace). Data Commons closely follows the [Schema.org data model](https://schema.org/docs/datamodel.html) and uses Schema.org schema to provide a common set of types and properties.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Data Commons knowledge graph browser\n", + "\n", + "The [Data Commons browser](https://datacommons.org/browser) provides a way to explore the data in a human-readable format. It is the best way to explore what is in Data Commons. Searching in the browser for an entity like [Mountain View](https://datacommons.org/browser/geoId/0649670) takes you to a page about the entity, including properties such as [containedInPlace](https://datacommons.org/browser/containedInPlace) and [timezone](https://datacommons.org/browser/timezone).\n", + "\n", + "An important property for all entities is the **DCID** (Data Commons identifier). This is a unique identifier assigned to each entity in the knowledge graph. The DCID is listed at the top of a browser page next to \"About:\" and in the list of properties.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## SDMX APIs from Python\n", + "\n", + "The Data Commons SDMX APIs provide statistical observations in a standard format. This notebook uses:\n", + "\n", + "- **SDMX Availability** to confirm that the requested statistical variables are available for a selected geography and source facet.\n", + "- **SDMX Data** to retrieve observation values as SDMX-CSV.\n", + "- **The Data Commons Python client** to retrieve human-readable names for the place DCIDs returned by SDMX.\n", + "\n", + "pySDMX reads Availability responses. Pandas reads the SDMX-CSV data into DataFrames, which can be used with data-processing, analytical, and visualization packages such as Pandas, NumPy, SciPy, and Matplotlib.\n", + "\n", + "## Set up environment\n", + "\n", + "### Install libraries\n", + "\n", + "Install pySDMX for SDMX metadata and the Data Commons Python client for place names.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install \"pysdmx[data,json]==1.19.0\" \"datacommons-client==2.1.6\" --quiet\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Configure the API key\n", + "\n", + "Obtain your own Data Commons API key from [apikeys.datacommons.org](https://apikeys.datacommons.org/) and set `DC_API_KEY` below to your key.\n", + "\n", + "Set the key once below. The notebook sends it in the `X-Api-Key` header for SDMX requests and configures the Data Commons Python client with the same key. After changing the key, rerun this cell and the data cells below; the installation, imports, and helper definitions do not need to be rerun.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from datacommons_client import DataCommonsClient\n", + "\n", + "# Set your Data Commons API key from https://apikeys.datacommons.org/.\n", + "DC_API_KEY = \"\"\n", + "dc_client = DataCommonsClient(api_key=DC_API_KEY)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Import dependencies\n", + "\n", + "Import the libraries and define the helpers required for data manipulation, analysis, and plotting.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "import matplotlib.patches as mpatches\n", + "\n", + "import io\n", + "from concurrent.futures import ThreadPoolExecutor\n", + "\n", + "import pandas as pd\n", + "import requests\n", + "from pysdmx.io import read_sdmx\n", + "\n", + "API_ROOT = \"https://api.datacommons.org\"\n", + "SDMX_DATA_URL = f\"{API_ROOT}/sdmx/v3/data/dataflow/DC/DF_OBS/1.0.0/*\"\n", + "SDMX_AVAILABILITY_URL = (\n", + " f\"{API_ROOT}/sdmx/v3/availability/dataflow/DC/DF_OBS/1.0.0/*\"\n", + ")\n", + "\n", + "\n", + "def _headers(accept=None):\n", + " if not DC_API_KEY:\n", + " raise ValueError(\"Set DC_API_KEY before requesting Data Commons.\")\n", + " headers = {\"X-Api-Key\": DC_API_KEY}\n", + " if accept:\n", + " headers[\"Accept\"] = accept\n", + " return headers\n", + "\n", + "\n", + "def sdmx_data(constraints):\n", + " response = requests.get(\n", + " SDMX_DATA_URL,\n", + " params=constraints,\n", + " headers=_headers(\"application/vnd.sdmx.data+csv;version=2.0.0\"),\n", + " timeout=120,\n", + " )\n", + " response.raise_for_status()\n", + " frame = pd.read_csv(io.BytesIO(response.content), low_memory=False)\n", + " frame[\"OBS_VALUE\"] = pd.to_numeric(frame[\"OBS_VALUE\"], errors=\"coerce\")\n", + " return frame\n", + "\n", + "\n", + "def availability_values(component, constraints):\n", + " response = requests.get(\n", + " f\"{SDMX_AVAILABILITY_URL}/{component}\",\n", + " params=constraints,\n", + " headers=_headers(),\n", + " timeout=120,\n", + " )\n", + " response.raise_for_status()\n", + " message = read_sdmx(io.BytesIO(response.content))\n", + " if not message.structures:\n", + " raise ValueError(\"The SDMX Availability request returned no constraint.\")\n", + " return {\n", + " value.value\n", + " for region in message.structures[0].cube_regions\n", + " for key_value in region.key_values\n", + " if key_value.id == component\n", + " for value in key_value.values\n", + " }\n", + "\n", + "\n", + "def require_variables(label, variables, constraints):\n", + " constraints = {\n", + " key: value for key, value in constraints.items()\n", + " if not (key == \"c[TIME_PERIOD]\" and value == \"LATEST\")\n", + " }\n", + " available = availability_values(\n", + " \"variableMeasured\",\n", + " {**constraints, \"c[variableMeasured]\": \",\".join(variables)},\n", + " )\n", + " missing = sorted(set(variables) - available)\n", + " if missing:\n", + " raise ValueError(f\"{label} is missing variables: {', '.join(missing)}\")\n", + " print(f\"{label}: all {len(variables)} variables are available.\")\n", + "\n", + "\n", + "# The client follows response pagination within each input batch.\n", + "def _fetch_name_batch(nodes):\n", + " return {\n", + " dcid: name.value\n", + " for dcid, name in dc_client.node.fetch_entity_names(\n", + " entity_dcids=nodes\n", + " ).items()\n", + " }\n", + "\n", + "\n", + "def fetch_node_names(dcids, batch_size=500):\n", + " unique_dcids = list(dict.fromkeys(dcids))\n", + " if not unique_dcids:\n", + " return {}\n", + " batches = [\n", + " unique_dcids[index:index + batch_size]\n", + " for index in range(0, len(unique_dcids), batch_size)\n", + " ]\n", + " names = {dcid: dcid for dcid in unique_dcids}\n", + " with ThreadPoolExecutor(max_workers=min(8, len(batches))) as executor:\n", + " for batch_names in executor.map(_fetch_name_batch, batches):\n", + " names.update(batch_names)\n", + " return names\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Example: Median age vs. population by state, county, and city\n", + "\n", + "For this exercise, we will compare median ages and population counts for US states, counties, and cities.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Querying administrative areas with SDMX\n", + "\n", + "The DCID for the United States is `country/USA`. The SDMX constraint `observationAbout.containedInPlace+` selects places directly or transitively contained in the United States. The `observationAbout.typeOf` constraint limits each request to states, counties, or cities.\n", + "\n", + "The SDMX responses contain place DCIDs. The Data Commons Python client supplies their human-readable names.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Querying statistics from Data Commons\n", + "\n", + "Data Commons has a large corpus of statistical data, which can be queried and joined with other statistics. For example, we can query the median income of women living in Berkeley, California or the number of individuals who are insured in Maryland.\n", + "\n", + "Before we explore how to do this, we need to understand how Data Commons stores statistical data. In particular, there are two types of entities: [StatisticalVariable](https://datacommons.org/browser/StatisticalVariable) and [StatVarObservation](https://datacommons.org/browser/StatVarObservation).\n", + "\n", + "A StatisticalVariable represents any type of statistical metric that can be measured at a place and time. Some examples include median income, median income of females, number of high school graduates, unemployment rate, and prevalence of diabetes—essentially anything you might call a metric, statistic, or measure. A StatVarObservation represents an actual measurement of a StatisticalVariable in a given place and time.\n", + "\n", + "One example of a StatVarObservation is the median age of people in San Antonio, Texas in 2024. The statistical metric, time, and place are median age, 2024, and San Antonio. You can explore statistical variables in the [Statistical Variable Explorer](https://datacommons.org/tools/statvar).\n", + "\n", + "We are going to retrieve the total count and median age of populations—[Count_Person](https://datacommons.org/browser/Count_Person) and [Median_Age_Person](https://datacommons.org/browser/Median_Age_Person)—for US states, counties, and cities. Availability first confirms that both variables exist under the selected Census ACS 5-year facet. SDMX Data then returns their latest observations.\n", + "\n", + "**Note:** The nationwide city query may take a minute.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "VARIABLES = [\"Count_Person\", \"Median_Age_Person\"]\n", + "PROVENANCE = \"dc/base/CensusACS5YearSurvey\"\n", + "MEASUREMENT_METHOD = \"CensusACS5yrSurvey\"\n", + "\n", + "\n", + "def fetch_geography(geography_type):\n", + " constraints = {\n", + " \"c[observationAbout.containedInPlace+]\": \"country/USA\",\n", + " \"c[observationAbout.typeOf]\": geography_type,\n", + " \"c[provenance]\": PROVENANCE,\n", + " \"c[measurementMethod]\": MEASUREMENT_METHOD,\n", + " \"c[TIME_PERIOD]\": \"LATEST\",\n", + " }\n", + " require_variables(geography_type, VARIABLES, constraints)\n", + " rows = sdmx_data({\n", + " **constraints,\n", + " \"c[variableMeasured]\": \",\".join(VARIABLES),\n", + " }).dropna(subset=[\"OBS_VALUE\"])\n", + "\n", + " complete = (\n", + " rows.groupby(\"observationAbout\")[\"variableMeasured\"].nunique()\n", + " == len(VARIABLES)\n", + " )\n", + " rows = rows[rows[\"observationAbout\"].isin(complete[complete].index)]\n", + " names = fetch_node_names(rows[\"observationAbout\"].unique())\n", + " return (\n", + " rows.rename(columns={\n", + " \"observationAbout\": \"entity\",\n", + " \"variableMeasured\": \"variable\",\n", + " \"OBS_VALUE\": \"value\",\n", + " })\n", + " .assign(entity_name=lambda frame: frame[\"entity\"].map(names))\n", + " )\n", + "\n", + "\n", + "df_state = fetch_geography(\"State\")\n", + "df_county = fetch_geography(\"County\")\n", + "df_city = fetch_geography(\"City\")\n", + "\n", + "print(\"states:\", df_state[\"entity\"].nunique())\n", + "print(\"counties:\", df_county[\"entity\"].nunique())\n", + "print(\"cities:\", df_city[\"entity\"].nunique())\n", + "df_city.head(5)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can view the data we have queried.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Cleaning and plotting the data\n", + "\n", + "Great! It looks like we have all the data we need. Before we finish, let's do some post-processing.\n", + "\n", + "We're interested in just the statistical values, so we'll select a single data point for each place-statistical-variable pair. The pivot uses each place's unique DCID and human-readable name as its row index, keeping places with identical names separate.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def filter_to_stats_only(frame):\n", + " frame = frame.pivot_table(\n", + " index=[\"entity\", \"entity_name\"],\n", + " columns=\"variable\",\n", + " values=\"value\",\n", + " aggfunc=\"first\",\n", + " )\n", + " frame = frame.rename_axis(None, axis=1)\n", + " return frame\n", + "\n", + "\n", + "df_state = filter_to_stats_only(df_state)\n", + "df_county = filter_to_stats_only(df_county)\n", + "df_city = filter_to_stats_only(df_city)\n", + "df_city = df_city[df_city[\"Median_Age_Person\"] >= 1]\n", + "df_city.head(5)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Finally, let's visualize our results.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def plot_data(title, table):\n", + " \"\"\"Generate a scatter plot comparing median age and population.\"\"\"\n", + " plt.figure(figsize=(12, 8))\n", + " plt.title(title)\n", + " plt.xlabel(\"Median Age in Years\")\n", + " plt.ylabel(\"Population Count (log scale)\")\n", + " axis = plt.gca()\n", + " axis.set_yscale(\"log\")\n", + " axis.scatter(\n", + " table[\"Median_Age_Person\"],\n", + " table[\"Count_Person\"],\n", + " alpha=0.7,\n", + " )\n", + "\n", + "\n", + "plot_data(\"Median Age vs. Population Count for States\", df_state)\n", + "plot_data(\"Median Age vs. Population Count for Counties\", df_county)\n", + "plot_data(\"Median Age vs. Population Count for Cities\", df_city)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can also plot each administrative-area granularity on the same plot to see how they relate.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def plot_all_data(state_table, county_table, city_table):\n", + " plt.figure(figsize=(12, 8))\n", + " plt.title(\"Median Age vs. Population Count\")\n", + " plt.xlabel(\"Median Age in Years\")\n", + " plt.ylabel(\"Population Count (log scale)\")\n", + "\n", + " state_color = \"#ffa600\"\n", + " county_color = \"#bc5090\"\n", + " city_color = \"#003f5c\"\n", + "\n", + " axis = plt.gca()\n", + " axis.set_yscale(\"log\")\n", + " axis.scatter(\n", + " state_table[\"Median_Age_Person\"],\n", + " state_table[\"Count_Person\"],\n", + " color=state_color,\n", + " alpha=0.75,\n", + " )\n", + " axis.scatter(\n", + " county_table[\"Median_Age_Person\"],\n", + " county_table[\"Count_Person\"],\n", + " color=county_color,\n", + " alpha=0.5,\n", + " )\n", + " axis.scatter(\n", + " city_table[\"Median_Age_Person\"],\n", + " city_table[\"Count_Person\"],\n", + " color=city_color,\n", + " alpha=0.4,\n", + " )\n", + "\n", + " plt.legend(handles=[\n", + " mpatches.Patch(color=state_color, label=\"States\"),\n", + " mpatches.Patch(color=county_color, label=\"Counties\"),\n", + " mpatches.Patch(color=city_color, label=\"Cities\"),\n", + " ])\n", + "\n", + "\n", + "plot_all_data(df_state, df_county, df_city)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## What's next\n", + "\n", + "Congratulations—you've completed the Data Commons task using SDMX. You can now explore other Data Commons notebooks for ideas about querying and joining data from the graph.\n" + ] + } + ], + "metadata": { + "colab": { + "provenance": [], + "toc_visible": true + }, + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.11" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/notebooks/analyzing_income_distribution_sdmx.ipynb b/notebooks/analyzing_income_distribution_sdmx.ipynb new file mode 100644 index 00000000..3dd7ce27 --- /dev/null +++ b/notebooks/analyzing_income_distribution_sdmx.ipynb @@ -0,0 +1,385 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Copyright 2026 Google LLC.\n", + "\n", + "SPDX-License-Identifier: Apache-2.0\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\"Open\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Analyzing Income Distribution with Data Commons SDMX\n", + "\n", + "The American Community Survey (published by the US Census) annually reports the number of households in a given income bracket at the state level. We can use this information, stored in Data Commons, to visualize disparity in income for each state in the US. Our goal for this tutorial is to generate a plot that visualizes the total number of households across a given set of income brackets for a selected state.\n", + "\n", + "Before we begin, we'll set up our notebook.\n", + "\n", + "## Set up environment\n", + "\n", + "### Install libraries\n", + "\n", + "Install pySDMX for SDMX metadata and the Data Commons Python client for place names.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install \"pysdmx[data,json]==1.19.0\" \"datacommons-client==2.1.6\" --quiet\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Configure the API key\n", + "\n", + "Obtain your own Data Commons API key from [apikeys.datacommons.org](https://apikeys.datacommons.org/) and set `DC_API_KEY` below to your key.\n", + "\n", + "The notebook sends this key in the `X-Api-Key` header for SDMX requests and configures the Data Commons Python client with the same key. After changing the key, rerun this cell and the data cells below; the installation, imports, and helper definitions do not need to be rerun.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from datacommons_client import DataCommonsClient\n", + "\n", + "# Set your Data Commons API key from https://apikeys.datacommons.org/.\n", + "DC_API_KEY = \"\"\n", + "dc_client = DataCommonsClient(api_key=DC_API_KEY)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Import dependencies\n", + "\n", + "Import the libraries and define the helpers required for data manipulation, analysis, and plotting.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "\n", + "import io\n", + "from concurrent.futures import ThreadPoolExecutor\n", + "\n", + "import pandas as pd\n", + "import requests\n", + "from pysdmx.io import read_sdmx\n", + "\n", + "API_ROOT = \"https://api.datacommons.org\"\n", + "SDMX_DATA_URL = f\"{API_ROOT}/sdmx/v3/data/dataflow/DC/DF_OBS/1.0.0/*\"\n", + "SDMX_AVAILABILITY_URL = (\n", + " f\"{API_ROOT}/sdmx/v3/availability/dataflow/DC/DF_OBS/1.0.0/*\"\n", + ")\n", + "\n", + "\n", + "def _headers(accept=None):\n", + " if not DC_API_KEY:\n", + " raise ValueError(\"Set DC_API_KEY before requesting Data Commons.\")\n", + " headers = {\"X-Api-Key\": DC_API_KEY}\n", + " if accept:\n", + " headers[\"Accept\"] = accept\n", + " return headers\n", + "\n", + "\n", + "def sdmx_data(constraints):\n", + " response = requests.get(\n", + " SDMX_DATA_URL,\n", + " params=constraints,\n", + " headers=_headers(\"application/vnd.sdmx.data+csv;version=2.0.0\"),\n", + " timeout=120,\n", + " )\n", + " response.raise_for_status()\n", + " frame = pd.read_csv(io.BytesIO(response.content), low_memory=False)\n", + " frame[\"OBS_VALUE\"] = pd.to_numeric(frame[\"OBS_VALUE\"], errors=\"coerce\")\n", + " return frame\n", + "\n", + "\n", + "def availability_values(component, constraints):\n", + " response = requests.get(\n", + " f\"{SDMX_AVAILABILITY_URL}/{component}\",\n", + " params=constraints,\n", + " headers=_headers(),\n", + " timeout=120,\n", + " )\n", + " response.raise_for_status()\n", + " message = read_sdmx(io.BytesIO(response.content))\n", + " if not message.structures:\n", + " raise ValueError(\"The SDMX Availability request returned no constraint.\")\n", + " return {\n", + " value.value\n", + " for region in message.structures[0].cube_regions\n", + " for key_value in region.key_values\n", + " if key_value.id == component\n", + " for value in key_value.values\n", + " }\n", + "\n", + "\n", + "def require_variables(label, variables, constraints):\n", + " constraints = {\n", + " key: value for key, value in constraints.items()\n", + " if not (key == \"c[TIME_PERIOD]\" and value == \"LATEST\")\n", + " }\n", + " available = availability_values(\n", + " \"variableMeasured\",\n", + " {**constraints, \"c[variableMeasured]\": \",\".join(variables)},\n", + " )\n", + " missing = sorted(set(variables) - available)\n", + " if missing:\n", + " raise ValueError(f\"{label} is missing variables: {', '.join(missing)}\")\n", + " print(f\"{label}: all {len(variables)} variables are available.\")\n", + "\n", + "\n", + "# The client follows response pagination within each input batch.\n", + "def _fetch_name_batch(nodes):\n", + " return {\n", + " dcid: name.value\n", + " for dcid, name in dc_client.node.fetch_entity_names(\n", + " entity_dcids=nodes\n", + " ).items()\n", + " }\n", + "\n", + "\n", + "def fetch_node_names(dcids, batch_size=500):\n", + " unique_dcids = list(dict.fromkeys(dcids))\n", + " if not unique_dcids:\n", + " return {}\n", + " batches = [\n", + " unique_dcids[index:index + batch_size]\n", + " for index in range(0, len(unique_dcids), batch_size)\n", + " ]\n", + " names = {dcid: dcid for dcid in unique_dcids}\n", + " with ThreadPoolExecutor(max_workers=min(8, len(batches))) as executor:\n", + " for batch_names in executor.map(_fetch_name_batch, batches):\n", + " names.update(batch_names)\n", + " return names\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Getting the data\n", + "\n", + "The Data Commons graph identifies 16 different income brackets. The list of these variables can be found under the **Economy > Household income** category in the [Statistical Variable Explorer](https://datacommons.org/tools/statvar).\n", + "\n", + "SDMX Availability confirms that all 16 variables occur for US states under the Census ACS 5-year facet. It does not guarantee that every state has every bracket. SDMX Data then returns the latest values as SDMX-CSV, which Pandas reads into a DataFrame and filters to states containing all 16 brackets. The Data Commons Python client supplies names for the state DCIDs returned by SDMX.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "income_brackets = [\n", + " \"Count_Household_IncomeOfUpto10000USDollar\",\n", + " \"Count_Household_IncomeOf10000To14999USDollar\",\n", + " \"Count_Household_IncomeOf15000To19999USDollar\",\n", + " \"Count_Household_IncomeOf20000To24999USDollar\",\n", + " \"Count_Household_IncomeOf25000To29999USDollar\",\n", + " \"Count_Household_IncomeOf30000To34999USDollar\",\n", + " \"Count_Household_IncomeOf35000To39999USDollar\",\n", + " \"Count_Household_IncomeOf40000To44999USDollar\",\n", + " \"Count_Household_IncomeOf45000To49999USDollar\",\n", + " \"Count_Household_IncomeOf50000To59999USDollar\",\n", + " \"Count_Household_IncomeOf60000To74999USDollar\",\n", + " \"Count_Household_IncomeOf75000To99999USDollar\",\n", + " \"Count_Household_IncomeOf100000To124999USDollar\",\n", + " \"Count_Household_IncomeOf125000To149999USDollar\",\n", + " \"Count_Household_IncomeOf150000To199999USDollar\",\n", + " \"Count_Household_IncomeOf200000OrMoreUSDollar\",\n", + "]\n", + "\n", + "income_constraints = {\n", + " \"c[observationAbout.containedInPlace+]\": \"country/USA\",\n", + " \"c[observationAbout.typeOf]\": \"State\",\n", + " \"c[provenance]\": \"dc/base/CensusACS5YearSurvey\",\n", + " \"c[measurementMethod]\": \"CensusACS5yrSurvey\",\n", + " \"c[TIME_PERIOD]\": \"LATEST\",\n", + "}\n", + "\n", + "require_variables(\"US state income\", income_brackets, income_constraints)\n", + "income_rows = sdmx_data({\n", + " **income_constraints,\n", + " \"c[variableMeasured]\": \",\".join(income_brackets),\n", + "}).dropna(subset=[\"OBS_VALUE\"])\n", + "\n", + "# Availability checks the slice as a whole; this verifies completeness per state.\n", + "complete = (\n", + " income_rows.groupby(\"observationAbout\")[\"variableMeasured\"].nunique()\n", + " == len(income_brackets)\n", + ")\n", + "income_rows = income_rows[\n", + " income_rows[\"observationAbout\"].isin(complete[complete].index)\n", + "]\n", + "names = fetch_node_names(income_rows[\"observationAbout\"].unique())\n", + "\n", + "data = (\n", + " income_rows.rename(columns={\n", + " \"observationAbout\": \"entity\",\n", + " \"variableMeasured\": \"variable\",\n", + " \"OBS_VALUE\": \"value\",\n", + " })\n", + " .assign(entity_name=lambda frame: frame[\"entity\"].map(names))\n", + " [[\"entity\", \"entity_name\", \"variable\", \"value\"]]\n", + ")\n", + "print(f\"Complete states: {data['entity'].nunique()}\")\n", + "data.head()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Analyzing the data\n", + "\n", + "Let's plot our data as a histogram. Notice that the income ranges as tabulated by the US Census are not equal. At the low end, the range is 0–9,999, whereas toward the top, the range 150,000–199,999 is five times as broad. We make the width of each column correspond to its range, which gives us an idea of total earnings, not just the number of households in that group.\n", + "\n", + "First we provide code for generating the plot.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "label_to_range = {\n", + " \"Count_Household_IncomeOfUpto10000USDollar\": [0, 9999],\n", + " \"Count_Household_IncomeOf10000To14999USDollar\": [10000, 14999],\n", + " \"Count_Household_IncomeOf15000To19999USDollar\": [15000, 19999],\n", + " \"Count_Household_IncomeOf20000To24999USDollar\": [20000, 24999],\n", + " \"Count_Household_IncomeOf25000To29999USDollar\": [25000, 29999],\n", + " \"Count_Household_IncomeOf30000To34999USDollar\": [30000, 34999],\n", + " \"Count_Household_IncomeOf35000To39999USDollar\": [35000, 39999],\n", + " \"Count_Household_IncomeOf40000To44999USDollar\": [40000, 44999],\n", + " \"Count_Household_IncomeOf45000To49999USDollar\": [45000, 49999],\n", + " \"Count_Household_IncomeOf50000To59999USDollar\": [50000, 59999],\n", + " \"Count_Household_IncomeOf60000To74999USDollar\": [60000, 74999],\n", + " \"Count_Household_IncomeOf75000To99999USDollar\": [75000, 99999],\n", + " \"Count_Household_IncomeOf100000To124999USDollar\": [100000, 124999],\n", + " \"Count_Household_IncomeOf125000To149999USDollar\": [125000, 149999],\n", + " \"Count_Household_IncomeOf150000To199999USDollar\": [150000, 199999],\n", + " \"Count_Household_IncomeOf200000OrMoreUSDollar\": [200000, 300000],\n", + "}\n", + "\n", + "\n", + "def plot_income(frame, state_name):\n", + " selected = frame.loc[frame[\"entity_name\"] == state_name]\n", + " if selected.empty:\n", + " print(f\"{state_name} does not have sufficient income data.\")\n", + " return None\n", + "\n", + " values = (\n", + " selected.groupby(\"variable\")[\"value\"].first().reindex(income_brackets)\n", + " )\n", + " if values.isna().any():\n", + " print(f\"{state_name} does not have all income brackets.\")\n", + " return None\n", + "\n", + " widths_without_interval = [\n", + " int((label_to_range[bracket][1] - label_to_range[bracket][0]) / 18)\n", + " for bracket in income_brackets\n", + " ]\n", + " positions = []\n", + " total = 0\n", + " for width in widths_without_interval:\n", + " positions.append(total + width // 2)\n", + " total += width\n", + " widths = [width - 50 for width in widths_without_interval]\n", + "\n", + " plt.figure(figsize=(12, 10))\n", + " plt.xticks(positions, income_brackets, rotation=90)\n", + " plt.grid(True)\n", + " plt.bar(positions, values.values, widths, color=\"b\", alpha=0.3)\n", + " plt.title(f\"Household income distribution: {state_name}\")\n", + " return selected\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can then call this code with a state to plot the income bracket sizes. Changing the state uses data already held in the DataFrame and does not make another API request.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "#@title Enter state to plot { run: \"auto\" }\n", + "state_name = \"Washington\" #@param [\"Missouri\", \"Arkansas\", \"Arizona\", \"Ohio\", \"Connecticut\", \"Vermont\", \"Illinois\", \"South Dakota\", \"Iowa\", \"Oklahoma\", \"Kansas\", \"Washington\", \"Oregon\", \"Hawaii\", \"Minnesota\", \"Idaho\", \"Alaska\", \"Colorado\", \"Delaware\", \"Alabama\", \"North Dakota\", \"Michigan\", \"California\", \"Indiana\", \"Kentucky\", \"Nebraska\", \"Louisiana\", \"New Jersey\", \"Rhode Island\", \"Utah\", \"Nevada\", \"South Carolina\", \"Wisconsin\", \"New York\", \"North Carolina\", \"New Hampshire\", \"Georgia\", \"Pennsylvania\", \"West Virginia\", \"Maine\", \"Mississippi\", \"Montana\", \"Tennessee\", \"New Mexico\", \"Massachusetts\", \"Wyoming\", \"Maryland\", \"Florida\", \"Texas\", \"Virginia\"]\n", + "result = plot_income(data, state_name)\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can also display the raw table of values.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "result\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This is only the beginning! What else can you analyze? For example, you could try computing a measure of income disparity in each state (see [Gini coefficient](https://en.wikipedia.org/wiki/Gini_coefficient)).\n", + "\n", + "You could then expand the DataFrame to include more information and analyze how attributes such as education level, crime, or even weather affect income disparity.\n" + ] + } + ], + "metadata": { + "colab": { + "provenance": [], + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/notebooks/analyzing_obesity_prevalence_sdmx.ipynb b/notebooks/analyzing_obesity_prevalence_sdmx.ipynb new file mode 100644 index 00000000..c6df2164 --- /dev/null +++ b/notebooks/analyzing_obesity_prevalence_sdmx.ipynb @@ -0,0 +1,561 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Copyright 2026 Google LLC.\n", + "\n", + "SPDX-License-Identifier: Apache-2.0\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\"Open\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Notebook Version** - 2.0.0 (SDMX port)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Case Study: Predicting Obesity Prevalence in US Counties\n", + "\n", + "**Objective:** This notebook demonstrates how to use Data Commons SDMX APIs to build a linear regression model predicting the prevalence of obesity in US counties.\n", + "\n", + "**Background:** Obesity prevalence is known to correlate with various health and socioeconomic factors [[1]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3198075/)[[2]](https://www.ncbi.nlm.nih.gov/pubmed/26562758). Data for these factors often reside in separate datasets from different government agencies:\n", + "\n", + "- The Centers for Disease Control (CDC) provides health-condition prevalence data, such as obesity and high blood pressure.\n", + "- The US Bureau of Labor Statistics (BLS) provides unemployment rates.\n", + "- The US Census Bureau provides poverty data and population counts.\n", + "\n", + "Data Commons aggregates these diverse datasets into a unified knowledge graph, simplifying data access and analysis.\n", + "\n", + "**Approach:** This notebook uses Data Commons to retrieve 2021 observations for the following variables for US counties:\n", + "\n", + "- [Percentage of Adult Population That Is Obese](https://datacommons.org/tools/statvar#sv=Percent_Person_Obesity) (CDC)—target variable.\n", + "- [Percentage of Adult Population With High Blood Pressure](https://datacommons.org/tools/statvar#sv=Percent_Person_WithHighBloodPressure) (CDC)—predictor variable.\n", + "- [Unemployment Rate of a Population](https://datacommons.org/tools/statvar#sv=UnemploymentRate_Person) (BLS)—predictor variable.\n", + "- [Population Below Poverty Level Status in Past Year](https://datacommons.org/tools/statvar#sv=Count_Person_BelowPovertyLevelInThePast12Months) (Census)—used to calculate poverty rate.\n", + "- [Total Population](https://datacommons.org/tools/statvar#sv=Count_Person) (Census)—used to calculate poverty rate.\n", + "\n", + "A linear regression model will be trained using high blood pressure prevalence, unemployment rate, and the calculated poverty rate to predict obesity prevalence.\n", + "\n", + "*Note:* The US Census also provides unemployment statistics. Using BLS data here is for demonstration purposes. Comparing results using Census unemployment data could be a potential extension.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Set up environment\n", + "\n", + "### 1.1. Install libraries\n", + "\n", + "Install pySDMX for SDMX metadata and the Data Commons Python client for place names.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install \"pysdmx[data,json]==1.19.0\" \"datacommons-client==2.1.6\" --quiet\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1.2. Configure the API key\n", + "\n", + "Obtain your own Data Commons API key from [apikeys.datacommons.org](https://apikeys.datacommons.org/) and set `DC_API_KEY` below to your key.\n", + "\n", + "The notebook sends this key in the `X-Api-Key` header for SDMX requests and configures the Data Commons Python client with the same key. After changing the key, rerun this cell and the data cells below; the installation, imports, and helper definitions do not need to be rerun.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from datacommons_client import DataCommonsClient\n", + "\n", + "# Set your Data Commons API key from https://apikeys.datacommons.org/.\n", + "DC_API_KEY = \"\"\n", + "dc_client = DataCommonsClient(api_key=DC_API_KEY)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1.3. Import dependencies\n", + "\n", + "Import the libraries and define the helpers required for data manipulation, modeling, and plotting.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "from sklearn.linear_model import LinearRegression\n", + "from sklearn.model_selection import train_test_split\n", + "\n", + "import io\n", + "from concurrent.futures import ThreadPoolExecutor\n", + "\n", + "import pandas as pd\n", + "import requests\n", + "from pysdmx.io import read_sdmx\n", + "\n", + "API_ROOT = \"https://api.datacommons.org\"\n", + "SDMX_DATA_URL = f\"{API_ROOT}/sdmx/v3/data/dataflow/DC/DF_OBS/1.0.0/*\"\n", + "SDMX_AVAILABILITY_URL = (\n", + " f\"{API_ROOT}/sdmx/v3/availability/dataflow/DC/DF_OBS/1.0.0/*\"\n", + ")\n", + "\n", + "\n", + "def _headers(accept=None):\n", + " if not DC_API_KEY:\n", + " raise ValueError(\"Set DC_API_KEY before requesting Data Commons.\")\n", + " headers = {\"X-Api-Key\": DC_API_KEY}\n", + " if accept:\n", + " headers[\"Accept\"] = accept\n", + " return headers\n", + "\n", + "\n", + "def sdmx_data(constraints):\n", + " response = requests.get(\n", + " SDMX_DATA_URL,\n", + " params=constraints,\n", + " headers=_headers(\"application/vnd.sdmx.data+csv;version=2.0.0\"),\n", + " timeout=120,\n", + " )\n", + " response.raise_for_status()\n", + " frame = pd.read_csv(io.BytesIO(response.content), low_memory=False)\n", + " frame[\"OBS_VALUE\"] = pd.to_numeric(frame[\"OBS_VALUE\"], errors=\"coerce\")\n", + " return frame\n", + "\n", + "\n", + "def availability_values(component, constraints):\n", + " response = requests.get(\n", + " f\"{SDMX_AVAILABILITY_URL}/{component}\",\n", + " params=constraints,\n", + " headers=_headers(),\n", + " timeout=120,\n", + " )\n", + " response.raise_for_status()\n", + " message = read_sdmx(io.BytesIO(response.content))\n", + " if not message.structures:\n", + " raise ValueError(\"The SDMX Availability request returned no constraint.\")\n", + " return {\n", + " value.value\n", + " for region in message.structures[0].cube_regions\n", + " for key_value in region.key_values\n", + " if key_value.id == component\n", + " for value in key_value.values\n", + " }\n", + "\n", + "\n", + "def require_variables(label, variables, constraints):\n", + " constraints = {\n", + " key: value for key, value in constraints.items()\n", + " if not (key == \"c[TIME_PERIOD]\" and value == \"LATEST\")\n", + " }\n", + " available = availability_values(\n", + " \"variableMeasured\",\n", + " {**constraints, \"c[variableMeasured]\": \",\".join(variables)},\n", + " )\n", + " missing = sorted(set(variables) - available)\n", + " if missing:\n", + " raise ValueError(f\"{label} is missing variables: {', '.join(missing)}\")\n", + " print(f\"{label}: all {len(variables)} variables are available.\")\n", + "\n", + "\n", + "# The client follows response pagination within each input batch.\n", + "def _fetch_name_batch(nodes):\n", + " return {\n", + " dcid: name.value\n", + " for dcid, name in dc_client.node.fetch_entity_names(\n", + " entity_dcids=nodes\n", + " ).items()\n", + " }\n", + "\n", + "\n", + "def fetch_node_names(dcids, batch_size=500):\n", + " unique_dcids = list(dict.fromkeys(dcids))\n", + " if not unique_dcids:\n", + " return {}\n", + " batches = [\n", + " unique_dcids[index:index + batch_size]\n", + " for index in range(0, len(unique_dcids), batch_size)\n", + " ]\n", + " names = {dcid: dcid for dcid in unique_dcids}\n", + " with ThreadPoolExecutor(max_workers=min(8, len(batches))) as executor:\n", + " for batch_names in executor.map(_fetch_name_batch, batches):\n", + " names.update(batch_names)\n", + " return names\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Data acquisition\n", + "\n", + "Fetch statistical observations for the specified variables for all US counties in 2021 using SDMX Data.\n", + "\n", + "The variables come from three source facets:\n", + "\n", + "- CDC500 with `AgeAdjustedPrevalence` for obesity and high blood pressure.\n", + "- BLS LAUS with `BLSSeasonallyUnadjusted` for unemployment.\n", + "- Census ACS 5-year with `CensusACS5yrSurvey` for poverty counts and total population.\n", + "\n", + "SDMX Availability first confirms that the expected variables are available under each selected facet. The three SDMX Data requests then run in parallel. The Data Commons Python client supplies county names for the DCIDs returned by SDMX.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "SOURCE_GROUPS = [\n", + " {\n", + " \"label\": \"CDC health\",\n", + " \"variables\": [\n", + " \"Percent_Person_Obesity\",\n", + " \"Percent_Person_WithHighBloodPressure\",\n", + " ],\n", + " \"provenance\": \"dc/base/CDC500\",\n", + " \"measurement_method\": \"AgeAdjustedPrevalence\",\n", + " },\n", + " {\n", + " \"label\": \"BLS unemployment\",\n", + " \"variables\": [\"UnemploymentRate_Person\"],\n", + " \"provenance\": \"dc/base/BLS_LAUS\",\n", + " \"measurement_method\": \"BLSSeasonallyUnadjusted\",\n", + " },\n", + " {\n", + " \"label\": \"Census poverty and population\",\n", + " \"variables\": [\n", + " \"Count_Person_BelowPovertyLevelInThePast12Months\",\n", + " \"Count_Person\",\n", + " ],\n", + " \"provenance\": \"dc/base/CensusACS5YearSurvey\",\n", + " \"measurement_method\": \"CensusACS5yrSurvey\",\n", + " },\n", + "]\n", + "\n", + "\n", + "def group_constraints(group):\n", + " return {\n", + " \"c[observationAbout.containedInPlace+]\": \"country/USA\",\n", + " \"c[observationAbout.typeOf]\": \"County\",\n", + " \"c[provenance]\": group[\"provenance\"],\n", + " \"c[measurementMethod]\": group[\"measurement_method\"],\n", + " \"c[TIME_PERIOD]\": \"2021\",\n", + " }\n", + "\n", + "\n", + "for group in SOURCE_GROUPS:\n", + " require_variables(\n", + " group[\"label\"],\n", + " group[\"variables\"],\n", + " group_constraints(group),\n", + " )\n", + "\n", + "\n", + "def fetch_group(group):\n", + " return sdmx_data({\n", + " **group_constraints(group),\n", + " \"c[variableMeasured]\": \",\".join(group[\"variables\"]),\n", + " })\n", + "\n", + "\n", + "with ThreadPoolExecutor(max_workers=3) as executor:\n", + " group_frames = list(executor.map(fetch_group, SOURCE_GROUPS))\n", + "\n", + "observations = pd.concat(group_frames, ignore_index=True)\n", + "observations = observations.dropna(subset=[\"OBS_VALUE\"])\n", + "names = fetch_node_names(observations[\"observationAbout\"].unique())\n", + "\n", + "us_county_observations_df = (\n", + " observations.rename(columns={\n", + " \"observationAbout\": \"entity\",\n", + " \"variableMeasured\": \"variable\",\n", + " \"OBS_VALUE\": \"value\",\n", + " })\n", + " .assign(entity_name=lambda frame: frame[\"entity\"].map(names))\n", + ")\n", + "us_county_observations_df.head(5)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. Data preparation\n", + "\n", + "Process the fetched data for modeling:\n", + "\n", + "1. **Select source facets:** The SDMX requests already constrain each group to its relevant provenance and measurement method.\n", + "2. **Select columns:** Keep only the essential columns: `entity`, `entity_name`, `variable`, and `value`.\n", + "3. **Pivot:** Reshape the DataFrame so each variable becomes a column, indexed by county `entity` and `entity_name`.\n", + "4. **Calculate poverty rate:** Compute the poverty-rate percentage using the population count and the count of people below the poverty level.\n", + "5. **Handle missing values:** Drop counties with any missing value among the selected variables.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "filtered_df = us_county_observations_df[\n", + " [\"entity\", \"entity_name\", \"variable\", \"value\"]\n", + "]\n", + "\n", + "pivoted_df = filtered_df.pivot_table(\n", + " index=[\"entity\", \"entity_name\"],\n", + " columns=\"variable\",\n", + " values=\"value\",\n", + ")\n", + "pivoted_df = pivoted_df[pivoted_df[\"Count_Person\"] > 0]\n", + "pivoted_df[\"PovertyRate\"] = (\n", + " pivoted_df[\"Count_Person_BelowPovertyLevelInThePast12Months\"]\n", + " / pivoted_df[\"Count_Person\"]\n", + " * 100\n", + ")\n", + "pivoted_df.dropna(inplace=True)\n", + "print(f\"Complete counties: {len(pivoted_df):,}\")\n", + "pivoted_df.head(5)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. Exploratory data analysis\n", + "\n", + "Visualize the relationships between the target variable (obesity prevalence) and the predictor variables (high blood pressure prevalence, unemployment rate, and poverty rate) using scatter plots. This helps assess potential correlations.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "pivoted_df.plot(\n", + " kind=\"scatter\",\n", + " x=\"Percent_Person_Obesity\",\n", + " y=\"Percent_Person_WithHighBloodPressure\",\n", + " grid=True,\n", + " figsize=(10, 6),\n", + " title=\"Obesity vs. High Blood Pressure in US Counties\",\n", + ")\n", + "plt.xlabel(\"Obesity Prevalence (Age-Adjusted %)\")\n", + "plt.ylabel(\"High Blood Pressure Prevalence (Age-Adjusted %)\")\n", + "plt.show()\n", + "\n", + "pivoted_df.plot(\n", + " kind=\"scatter\",\n", + " x=\"UnemploymentRate_Person\",\n", + " y=\"Percent_Person_Obesity\",\n", + " grid=True,\n", + " figsize=(10, 6),\n", + " title=\"Unemployment Rate vs. Obesity Prevalence in US Counties\",\n", + ")\n", + "plt.xlabel(\"Unemployment Rate\")\n", + "plt.ylabel(\"Obesity Prevalence (Age-Adjusted %)\")\n", + "plt.show()\n", + "\n", + "pivoted_df.plot(\n", + " kind=\"scatter\",\n", + " x=\"PovertyRate\",\n", + " y=\"Percent_Person_Obesity\",\n", + " grid=True,\n", + " figsize=(10, 6),\n", + " title=\"Poverty Rate vs. Obesity Prevalence in US Counties\",\n", + ")\n", + "plt.xlabel(\"Poverty Rate\")\n", + "plt.ylabel(\"Obesity Prevalence (Age-Adjusted %)\")\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "*Observation:* The scatter plots suggest positive correlations between obesity prevalence and each of the predictor variables.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 5. Model training\n", + "\n", + "Train a linear regression model to predict obesity prevalence based on the selected predictors.\n", + "\n", + "The model follows the form:\n", + "\n", + "$$f_\\theta(x) = \\theta_0 + \\theta_1 (\\text{high blood pressure}) + \\theta_2 (\\text{unemployment}) + \\theta_3 (\\text{poverty rate})$$\n", + "\n", + "### 5.1. Prepare features and target variable\n", + "\n", + "Define the feature matrix `X` (predictors) and the target vector `Y` (obesity prevalence).\n", + "\n", + "### 5.2. Split data\n", + "\n", + "Split the data into training and testing sets: 80% training and 20% testing. A fixed random state makes repeated runs comparable.\n", + "\n", + "### 5.3. Train linear regression model\n", + "\n", + "Instantiate and train scikit-learn's [LinearRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) model using the training data.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "features = [\n", + " \"Percent_Person_WithHighBloodPressure\",\n", + " \"UnemploymentRate_Person\",\n", + " \"PovertyRate\",\n", + "]\n", + "target = \"Percent_Person_Obesity\"\n", + "\n", + "x_train, x_test, y_train, y_test = train_test_split(\n", + " pivoted_df[features],\n", + " pivoted_df[[target]],\n", + " test_size=0.2,\n", + " random_state=42,\n", + ")\n", + "\n", + "model = LinearRegression(fit_intercept=True)\n", + "model.fit(x_train, y_train)\n", + "\n", + "print(f\"Training set size: {len(x_train):,} samples\")\n", + "print(f\"Test set size: {len(x_test):,} samples\")\n", + "print(f\"Model intercept: {model.intercept_}\")\n", + "print(f\"Model coefficients: {model.coef_}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 6. Model evaluation\n", + "\n", + "Assess the performance of the trained model using Mean Squared Error (MSE) and residual analysis.\n", + "\n", + "### 6.1. Calculate Mean Squared Error\n", + "\n", + "Define a function for MSE and calculate it for both the training and test sets. Lower MSE indicates a better fit.\n", + "\n", + "### 6.2. Analyze residuals\n", + "\n", + "Calculate and plot the residual proportion, `(predicted - actual) / actual`, for the test set. Residuals ideally should be scattered around zero.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def mse(y_true, y_pred):\n", + " \"\"\"Compute mean squared error.\"\"\"\n", + " return np.mean((y_pred - y_true) ** 2)\n", + "\n", + "\n", + "train_pred = model.predict(x_train)\n", + "test_pred = model.predict(x_test)\n", + "train_mse = mse(y_train.to_numpy(), train_pred)\n", + "test_mse = mse(y_test.to_numpy(), test_pred)\n", + "\n", + "print(f\"Training MSE: {train_mse:.4f}\")\n", + "print(f\"Test MSE: {test_mse:.4f}\")\n", + "\n", + "residual_proportion = (\n", + " (test_pred.ravel() - y_test[target].to_numpy())\n", + " / y_test[target].to_numpy()\n", + ")\n", + "plt.figure(figsize=(10, 6))\n", + "plt.title(\"Residual Proportions Plot\")\n", + "plt.xlabel(\"Test Data Row Index\")\n", + "plt.ylabel(\"residual / actual obesity prevalence\")\n", + "plt.scatter(range(len(residual_proportion)), residual_proportion)\n", + "plt.show()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The printed MSE values compare model fit on the training and test sets. The residual plot shows the direction and relative size of prediction errors across test counties.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 7. Conclusion and next steps\n", + "\n", + "This notebook demonstrated the use of Data Commons SDMX APIs to acquire data from multiple sources—CDC, BLS, and Census—and build a simple linear regression model to predict obesity prevalence in US counties. Data Commons streamlines the data gathering and integration process.\n", + "\n", + "The resulting model, using high blood pressure prevalence, unemployment rate, and poverty rate, provides a baseline prediction.\n", + "\n", + "**Potential improvements and further exploration:**\n", + "\n", + "- **Add more variables:** Include other variables known or hypothesized to correlate with obesity, such as:\n", + " - `Percent_Person_WithHighCholesterol`\n", + " - `Percent_Person_WithDiabetes`\n", + " - educational attainment levels\n", + " - access to healthy food outlets\n", + " - physical inactivity rates\n", + "- **Feature engineering:** Create new features from existing ones.\n", + "- **Model selection:** Experiment with different regression models, such as Ridge, Lasso, or tree-based models.\n", + "- **Geographic analysis:** Explore spatial patterns in obesity prevalence and model errors.\n", + "- **Alternative data sources:** Compare model performance using Census unemployment data instead of BLS data.\n", + "\n", + "Data Commons provides access to a wide range of variables, enabling exploration of correlations with university counts, crime rates, or environmental factors, potentially leading to more comprehensive models.\n" + ] + } + ], + "metadata": { + "colab": { + "provenance": [], + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}