{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Visualization Examples\n", "\n", "Examples of the main visualization functions provided by **ER-Evaluation**. For more information on the use and meaning of each, please refer to the {doc}`User Guide ` and {doc}`API Documentation `." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "To get started, we first collect all relevant data from our {doc}`toy subset of PatentsView data `:\n", "\n", "- **predictions:** A dictionary of predicted disambiguations indexed by time.\n", "- **reference:** A benchmark dataset containing 401 disambiguated inventors (our \"ground truth\" sample).\n", "- **names:** The names on each record, that are used to compute the homonymy rate and name variation rate statistics." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "from er_evaluation.datasets import load_pv_disambiguations, load_pv_data\n", "\n", "predictions, reference = load_pv_disambiguations()\n", "\n", "pv_data = load_pv_data()\n", "pv_data.set_index(\"mention_id\", inplace=True)\n", "names = pv_data[\"raw_inventor_name_first\"] + \" \" + pv_data[\"raw_inventor_name_last\"]" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Summary Statistics\n", "\n", "Visualization of relevant disambiguation summaries and their evolution over time. Refer to the {doc}`Summary Statistics <02-summary_statistics.html>` page for more information." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "hovertemplate": "prediction=%{x}
average_cluster_size=%{y}", "legendgroup": "", "line": { "color": "#636efa", "dash": "solid", "shape": "spline" }, "marker": { "symbol": "circle" }, "mode": "lines+markers", "name": "", "orientation": "v", "showlegend": false, "type": "scatter", "x": [ "2017-08-08T00:00:00", "2017-10-03T00:00:00", "2017-12-26T00:00:00", "2018-05-28T00:00:00", "2018-11-27T00:00:00", "2019-03-12T00:00:00", "2019-08-20T00:00:00", "2019-10-08T00:00:00", "2019-12-31T00:00:00", "2020-03-31T00:00:00", "2020-06-30T00:00:00", "2020-09-29T00:00:00", "2020-12-29T00:00:00", "2021-12-30T00:00:00", "2022-06-30T00:00:00" ], "xaxis": "x", "y": [ 7.523881392045454, 7.21922400067041, 5.596246819338423, 6.443396890170971, 6.979681013764474, 7.234293290123002, 7.477880620945995, 7.538751540636555, 7.608161223468877, 7.647505270555165, 7.777762305924946, 6.206911879097677, 9.96020159888773, 8.08859736383984, 10.787704984247515 ], "yaxis": "y" }, { "hovertemplate": "prediction=%{x}
matching_rate=%{y}", "legendgroup": "", "line": { "color": "#636efa", "dash": "solid", "shape": "spline" }, "marker": { "symbol": "circle" }, "mode": "lines+markers", "name": "", "orientation": "v", "showlegend": false, "type": "scatter", "x": [ "2017-08-08T00:00:00", "2017-10-03T00:00:00", "2017-12-26T00:00:00", "2018-05-28T00:00:00", "2018-11-27T00:00:00", "2019-03-12T00:00:00", "2019-08-20T00:00:00", "2019-10-08T00:00:00", "2019-12-31T00:00:00", "2020-03-31T00:00:00", "2020-06-30T00:00:00", "2020-09-29T00:00:00", "2020-12-29T00:00:00", "2021-12-30T00:00:00", "2022-06-30T00:00:00" ], "xaxis": "x2", "y": [ 0.939586307803042, 0.9359931280253521, 0.9055619337751356, 0.923629613452719, 0.9315511592478974, 0.9356953777234779, 0.9383064987719777, 0.9389605893327692, 0.9398465165647514, 0.940031610674116, 0.9411164422801489, 0.9077449154969922, 0.9675106000593254, 0.9431116782093361, 0.9713346462884058 ], "yaxis": "y2" }, { "hovertemplate": "prediction=%{x}
H0=%{y}", "legendgroup": "", "line": { "color": "#636efa", "dash": "solid", "shape": "spline" }, "marker": { "symbol": "circle" }, "mode": "lines+markers", "name": "", "orientation": "v", "showlegend": false, "type": "scatter", "x": [ "2017-08-08T00:00:00", "2017-10-03T00:00:00", "2017-12-26T00:00:00", "2018-05-28T00:00:00", "2018-11-27T00:00:00", "2019-03-12T00:00:00", "2019-08-20T00:00:00", "2019-10-08T00:00:00", "2019-12-31T00:00:00", "2020-03-31T00:00:00", "2020-06-30T00:00:00", "2020-09-29T00:00:00", "2020-12-29T00:00:00", "2021-12-30T00:00:00", "2022-06-30T00:00:00" ], "xaxis": "x3", "y": [ 171, 168, 166, 179, 175, 181, 189, 195, 194, 193, 200, 198, 193, 207, 210 ], "yaxis": "y3" }, { "hovertemplate": "prediction=%{x}
H1=%{y}", "legendgroup": "", "line": { "color": "#636efa", "dash": "solid", "shape": "spline" }, "marker": { "symbol": "circle" }, "mode": "lines+markers", "name": "", "orientation": "v", "showlegend": false, "type": "scatter", "x": [ "2017-08-08T00:00:00", "2017-10-03T00:00:00", "2017-12-26T00:00:00", "2018-05-28T00:00:00", "2018-11-27T00:00:00", "2019-03-12T00:00:00", "2019-08-20T00:00:00", "2019-10-08T00:00:00", "2019-12-31T00:00:00", "2020-03-31T00:00:00", "2020-06-30T00:00:00", "2020-09-29T00:00:00", "2020-12-29T00:00:00", "2021-12-30T00:00:00", "2022-06-30T00:00:00" ], "xaxis": "x4", "y": [ 9.595842527724987, 9.232390077753198, 7.486484051752622, 8.6005526077469, 9.268208213114955, 9.646602178980453, 9.896394599509955, 9.927226750925023, 10.03874125853286, 10.043638848656709, 10.10141839893877, 7.0752844357686575, 16.374402931686124, 10.98773626263287, 17.594994005746447 ], "yaxis": "y4" }, { "hovertemplate": "prediction=%{x}
name_variation_rate=%{y}", "legendgroup": "", "line": { "color": "#636efa", "dash": "solid", "shape": "spline" }, "marker": { "symbol": "circle" }, "mode": "lines+markers", "name": "", "orientation": "v", "showlegend": false, "type": "scatter", "x": [ "2017-08-08T00:00:00", "2017-10-03T00:00:00", "2017-12-26T00:00:00", "2018-05-28T00:00:00", "2018-11-27T00:00:00", "2019-03-12T00:00:00", "2019-08-20T00:00:00", "2019-10-08T00:00:00", "2019-12-31T00:00:00", "2020-03-31T00:00:00", "2020-06-30T00:00:00", "2020-09-29T00:00:00", "2020-12-29T00:00:00", "2021-12-30T00:00:00", "2022-06-30T00:00:00" ], "xaxis": "x5", "y": [ 0.14710582386363635, 0.1465683398977625, 0.09790076335877862, 0.1165130514317878, 0.12621076396475128, 0.13103041909110996, 0.13431965600174914, 0.1341260059450446, 0.1340670335167584, 0.13408292340126493, 0.13604400194945346, 0.10656739637737526, 0.24070212026416407, 0.18453121114150708, 0.24670813474432507 ], "yaxis": "y5" }, { "hovertemplate": "prediction=%{x}
homonymy_rate=%{y}", "legendgroup": "", "line": { "color": "#636efa", "dash": "solid", "shape": "spline" }, "marker": { "symbol": "circle" }, "mode": "lines+markers", "name": "", "orientation": "v", "showlegend": false, "type": "scatter", "x": [ "2017-08-08T00:00:00", "2017-10-03T00:00:00", "2017-12-26T00:00:00", "2018-05-28T00:00:00", "2018-11-27T00:00:00", "2019-03-12T00:00:00", "2019-08-20T00:00:00", "2019-10-08T00:00:00", "2019-12-31T00:00:00", "2020-03-31T00:00:00", "2020-06-30T00:00:00", "2020-09-29T00:00:00", "2020-12-29T00:00:00", "2021-12-30T00:00:00", "2022-06-30T00:00:00" ], "xaxis": "x6", "y": [ 0.6199396306818182, 0.6384815218302187, 0.6878498727735369, 0.6382185323295575, 0.6061466754060156, 0.5941666052883553, 0.5773631659500037, 0.5765243239324295, 0.5696419638390624, 0.567111735769501, 0.5599108821276892, 0.6717413045893988, 0.4348279457768509, 0.49807261875155434, 0.19767347927942483 ], "yaxis": "y6" } ], "layout": { "annotations": [ { "font": { "size": 14 }, "showarrow": false, "text": "Average Cluster Size", "x": 0.225, "xanchor": "center", "xref": "paper", "y": 1, "yanchor": "bottom", "yref": "paper" }, { "font": { "size": 14 }, "showarrow": false, "text": "Matching Rate", "x": 0.775, "xanchor": "center", "xref": "paper", "y": 1, "yanchor": "bottom", "yref": "paper" }, { "font": { "size": 14 }, "showarrow": false, "text": "Number of distinct cluster sizes", "x": 0.225, "xanchor": "center", "xref": "paper", "y": 0.6111111111111112, "yanchor": "bottom", "yref": "paper" }, { "font": { "size": 14 }, "showarrow": false, "text": "Entropy", "x": 0.775, "xanchor": "center", "xref": "paper", "y": 0.6111111111111112, "yanchor": "bottom", "yref": "paper" }, { "font": { "size": 14 }, "showarrow": false, "text": "Name Variation Rate", "x": 0.225, "xanchor": "center", "xref": "paper", "y": 0.22222222222222224, "yanchor": "bottom", "yref": "paper" }, { "font": { "size": 14 }, "showarrow": false, "text": "Homonymy Rate", "x": 0.775, "xanchor": "center", "xref": "paper", "y": 0.22222222222222224, "yanchor": "bottom", "yref": "paper" } ], "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Disambiguation Summary Statistics" }, "xaxis": { "anchor": "y", "domain": [ 0, 0.45 ], "matches": "x5", "showticklabels": false }, "xaxis2": { "anchor": "y2", "domain": [ 0.55, 1 ], "matches": "x6", "showticklabels": false }, "xaxis3": { "anchor": "y3", "domain": [ 0, 0.45 ], "matches": "x5", "showticklabels": false }, "xaxis4": { "anchor": "y4", "domain": [ 0.55, 1 ], "matches": "x6", "showticklabels": false }, "xaxis5": { "anchor": "y5", "domain": [ 0, 0.45 ] }, "xaxis6": { "anchor": "y6", "domain": [ 0.55, 1 ] }, "yaxis": { "anchor": "x", "domain": [ 0.7777777777777778, 1 ] }, "yaxis2": { "anchor": "x2", "domain": [ 0.7777777777777778, 1 ] }, "yaxis3": { "anchor": "x3", "domain": [ 0.3888888888888889, 0.6111111111111112 ] }, "yaxis4": { "anchor": "x4", "domain": [ 0.3888888888888889, 0.6111111111111112 ] }, "yaxis5": { "anchor": "x5", "domain": [ 0, 0.22222222222222224 ] }, "yaxis6": { "anchor": "x6", "domain": [ 0, 0.22222222222222224 ] } } } }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from er_evaluation.plots import plot_summaries\n", "\n", "plot_summaries(predictions, names)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Performance Estimates\n", "\n", "Visualize estimates for key metrics over time with uncertainty quantification (+/- 1 standard deviation). Note that estimates for f-score, b-cubed metrics, and cluster metrics can also be computed and visualized in the same way. Refer to the {doc}`Performance Estimation <03-estimating_performance.html>` page for more information.\n", "\n", "**What is the difference between performance estimates and performance metrics?** Performance metrics naively computed on benchmark datasets do not account for sample size and biases, leading to [over-optimistic results and poor performance](https://www.valires.com/post/common-pitfalls-to-avoid-when-estimating-er-performance-metrics). In contrast, our statistical estimators account for the smaller size of the benchmark dataset and provide uncertainty quantification." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "error_y": { "array": [ 0.2152523904052803, 0.2284648845356206, 0.05396008154266888, 0.06762140841603512, 0.0641799010705886, 0.07095188803819781, 0.06855633851104145, 0.08001541453302838, 0.07513925473462715, 0.07646197630520299, 0.0794556989462446, 0.08091626415776206, 0.03812943865295822, 0.03723981444067029, 0.03477773119450892 ] }, "hovertemplate": "estimator=Pairwise precision
prediction=%{x}
value=%{y}", "legendgroup": "Pairwise precision", "line": { "color": "#636efa", "dash": "solid", "shape": "spline" }, "marker": { "symbol": "circle" }, "mode": "lines+markers", "name": "Pairwise precision", "orientation": "v", "showlegend": true, "type": "scatter", "x": [ "2017-08-08T00:00:00", "2017-10-03T00:00:00", "2017-12-26T00:00:00", "2018-05-28T00:00:00", "2018-11-27T00:00:00", "2019-03-12T00:00:00", "2019-08-20T00:00:00", "2019-10-08T00:00:00", "2019-12-31T00:00:00", "2020-03-31T00:00:00", "2020-06-30T00:00:00", "2020-09-29T00:00:00", "2020-12-29T00:00:00", "2021-12-30T00:00:00", "2022-06-30T00:00:00" ], "xaxis": "x", "y": [ 0.5682905929738009, 0.5614532661533171, 0.9177507115333738, 0.878487421180332, 0.876664044828819, 0.871523346373805, 0.8563773235431499, 0.8513799572091044, 0.8653184769196983, 0.8675474079481227, 0.8589447378261852, 0.8637717722052076, 0.9087541920717165, 0.9131787709880134, 0.8833020990513186 ], "yaxis": "y" }, { "error_y": { "array": [ 0.01815964133564645, 0.01895614311202958, 0.02690691861377593, 0.023424393355402976, 0.018081110982682582, 0.017499659502393926, 0.016901175387470806, 0.016721816409361122, 0.015236804944936547, 0.016349374562466146, 0.02184091145328047, 0.014231509859813599, 0.014585344985919273, 0.017568488829289367, 0.014474559485576568 ] }, "hovertemplate": "estimator=Pairwise recall
prediction=%{x}
value=%{y}", "legendgroup": "Pairwise recall", "line": { "color": "#EF553B", "dash": "solid", "shape": "spline" }, "marker": { "symbol": "diamond" }, "mode": "lines+markers", "name": "Pairwise recall", "orientation": "v", "showlegend": true, "type": "scatter", "x": [ "2017-08-08T00:00:00", "2017-10-03T00:00:00", "2017-12-26T00:00:00", "2018-05-28T00:00:00", "2018-11-27T00:00:00", "2019-03-12T00:00:00", "2019-08-20T00:00:00", "2019-10-08T00:00:00", "2019-12-31T00:00:00", "2020-03-31T00:00:00", "2020-06-30T00:00:00", "2020-09-29T00:00:00", "2020-12-29T00:00:00", "2021-12-30T00:00:00", "2022-06-30T00:00:00" ], "xaxis": "x", "y": [ 0.9610920825379785, 0.9541269644954581, 0.9166311090252167, 0.942536009220433, 0.9570321976537539, 0.9601746830047234, 0.962040886261, 0.9627747512149712, 0.9653411752888007, 0.9622775797659693, 0.9514561547643773, 0.8966542126263876, 0.9761365985333893, 0.9622075140463208, 0.9770477584915793 ], "yaxis": "y" } ], "layout": { "legend": { "title": { "text": "estimator" }, "tracegroupgap": 0 }, "margin": { "t": 60 }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Performance estimates" }, "xaxis": { "anchor": "y", "domain": [ 0, 1 ], "title": { "text": "prediction" } }, "yaxis": { "anchor": "x", "domain": [ 0, 1 ], "title": { "text": "value" } } } } }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from er_evaluation.plots import plot_estimates\n", "\n", "plot_estimates(predictions, {\"sample\": reference, \"weights\": \"cluster_size\"})" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Disambiguation Similarity\n", "\n", "Pairwise precision and pairwise F-score between pairs of predicted disambiguations (computed on their inner join). This is used to characterize changes in disambiguations over time. For a given timestamp on the y axis and a timestamp on the x axis, the pairwise precision is defined as:\n", "\n", "- **pairwise precision:** the proportion of predicted links at time x that were also present at time y." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "coloraxis": "coloraxis", "hovertemplate": "x: %{x}
y: %{y}
color: %{z}", "name": "0", "type": "heatmap", "x": [ "2017-08-08T00:00:00", "2017-10-03T00:00:00", "2017-12-26T00:00:00", "2018-05-28T00:00:00", "2018-11-27T00:00:00", "2019-03-12T00:00:00", "2019-08-20T00:00:00", "2019-10-08T00:00:00", "2019-12-31T00:00:00", "2020-03-31T00:00:00", "2020-06-30T00:00:00", "2020-09-29T00:00:00", "2020-12-29T00:00:00", "2021-12-30T00:00:00", "2022-06-30T00:00:00" ], "xaxis": "x", "y": [ "2017-08-08T00:00:00", "2017-10-03T00:00:00", "2017-12-26T00:00:00", "2018-05-28T00:00:00", "2018-11-27T00:00:00", "2019-03-12T00:00:00", "2019-08-20T00:00:00", "2019-10-08T00:00:00", "2019-12-31T00:00:00", "2020-03-31T00:00:00", "2020-06-30T00:00:00", "2020-09-29T00:00:00", "2020-12-29T00:00:00", "2021-12-30T00:00:00", "2022-06-30T00:00:00" ], "yaxis": "y", "z": [ [ 1, 0.6525274724429703, 0.3250507488686197, 0.36349018927831706, 0.37273616599106174, 0.3742724414835651, 0.37448595904179577, 0.37691500963071844, 0.3766456135278368, 0.37626176893786034, 0.3731822972156721, 0.3506724713708968, 0.3239037007616784, 0.32459846587762575, 0.324046344844698 ], [ 0.6273042158116686, 1, 0.28480602422026136, 0.3184098719431503, 0.32535641234239276, 0.3262521516933762, 0.3263616019922797, 0.32895583467224376, 0.32979684887377536, 0.32916549595910444, 0.3447244955928487, 0.3056998967566228, 0.28506910767682503, 0.28512489890061676, 0.2849104995055784 ], [ 0.9497235928473775, 0.8778094794775608, 1, 0.9761839481379201, 0.9886322040677304, 0.9912747025378443, 0.98836367901929, 0.9910015842977858, 0.9935992108190155, 0.9923325498997271, 0.9847357642858859, 0.9238653756736093, 0.8586498277906466, 0.8594069977627624, 0.8585293448412805 ], [ 0.9350495468630541, 0.8602905830928236, 0.8503812583526266, 1, 0.9694845425714077, 0.9779403160579587, 0.9750166297355407, 0.9814141784052334, 0.9858956954376589, 0.9849052069311307, 0.9697763676079931, 0.9140855648173447, 0.8167536181297734, 0.8175524176416817, 0.8169607566851537 ], [ 0.9230518498488148, 0.8461263823347028, 0.8288683308598529, 0.9324031093072849, 1, 0.9737098916164392, 0.9764599412235333, 0.9783659715406452, 0.9776876056453254, 0.9748496479484259, 0.9626334723009485, 0.9106249121871863, 0.7920767879705147, 0.7937625399238655, 0.7939863752623586 ], [ 0.9199441162155905, 0.8418702277126674, 0.824614994832895, 0.9312872921854651, 0.9605003334537159, 1, 0.9691472292445875, 0.9781498498410397, 0.977570728316872, 0.9720294798615737, 0.9583655034385937, 0.9086404788295067, 0.7713564776817584, 0.7731956894086354, 0.7735050471585717 ], [ 0.9126332885654496, 0.8352237383456349, 0.8153167542410524, 0.9218979304703167, 0.9566608843011639, 0.9627767991738972, 1, 0.978823986205347, 0.9698408837826666, 0.9686228359965404, 0.9541604817221939, 0.9056239822789752, 0.7559899528508186, 0.7581106298881966, 0.7583571187847443 ], [ 0.9075650609187638, 0.831788179837836, 0.8075328829457822, 0.9162670460358159, 0.9469344465390277, 0.9599842411746364, 0.9663197958010702, 1, 0.9664175739562639, 0.9638182637997467, 0.9517979974243332, 0.897972420907049, 0.7439065394944286, 0.7458419511460253, 0.7459647713232671 ], [ 0.9156733831379793, 0.8418742646620825, 0.8175026308598405, 0.9289696608292004, 0.9540563477288126, 0.9672302147152756, 0.9655086748947601, 0.974373259875595, 1, 0.9745728613725836, 0.9568974457299, 0.9065231040334685, 0.7458986411472791, 0.7482959072028118, 0.7478887256947208 ], [ 0.9179880167023231, 0.8432696275975917, 0.8192397331598736, 0.9312261156705547, 0.9549964756411048, 0.965310274910345, 0.9670534046920342, 0.9744549388472102, 0.9771606638682065, 1, 0.9566938829054761, 0.9088051424192047, 0.7388079195342425, 0.7409856917202847, 0.7410604897081804 ], [ 0.8696789470210168, 0.8444424390917649, 0.778738951393115, 0.8796695379926462, 0.9068494731027247, 0.9171786703335534, 0.9214258781399758, 0.932331830265746, 0.9322563926140888, 0.9308915855473274, 1, 0.8716115877460611, 0.7110331815968829, 0.7129280128818065, 0.7133900761421454 ], [ 0.9177565466591175, 0.8401636328950821, 0.8183122349664356, 0.926520480173765, 0.9553942923524048, 0.9667240682769569, 0.9682522209802141, 0.972906018673216, 0.973656460944045, 0.9732536253494721, 0.955909865365984, 1, 0.7281275080237336, 0.7298938559682516, 0.7309129925049666 ], [ 0.8816722763186171, 0.8204471638813385, 0.8038015441540542, 0.8900959484516764, 0.9109013358079391, 0.9114025221449308, 0.9144502307531357, 0.9179560332597069, 0.9217931847771931, 0.9199041929895154, 0.9173621890085725, 0.8565731516913149, 1, 0.9880720840221988, 0.9847812480420137 ], [ 0.8827513614778024, 0.819697297334112, 0.8035474422750251, 0.8896486108006428, 0.9111959641528953, 0.9117028319690136, 0.9149520551118749, 0.9182331030432296, 0.9226321766705349, 0.9203871144488979, 0.9176833674753919, 0.8566687806930876, 0.9858254277137327, 1, 0.9845948641798082 ], [ 0.8802316076483998, 0.8180955497159603, 0.8016050740582635, 0.8876172476659263, 0.9098612356176181, 0.9104114836082003, 0.9136176183141833, 0.9166084984475191, 0.9201286660846082, 0.9184277549526337, 0.9157766776187458, 0.8555280414754297, 0.9796026506173756, 0.9175653933008744, 1 ] ] }, { "coloraxis": "coloraxis", "hovertemplate": "x: %{x}
y: %{y}
color: %{z}", "name": "1", "type": "heatmap", "x": [ "2017-08-08T00:00:00", "2017-10-03T00:00:00", "2017-12-26T00:00:00", "2018-05-28T00:00:00", "2018-11-27T00:00:00", "2019-03-12T00:00:00", "2019-08-20T00:00:00", "2019-10-08T00:00:00", "2019-12-31T00:00:00", "2020-03-31T00:00:00", "2020-06-30T00:00:00", "2020-09-29T00:00:00", "2020-12-29T00:00:00", "2021-12-30T00:00:00", "2022-06-30T00:00:00" ], "xaxis": "x2", "y": [ "2017-08-08T00:00:00", "2017-10-03T00:00:00", "2017-12-26T00:00:00", "2018-05-28T00:00:00", "2018-11-27T00:00:00", "2019-03-12T00:00:00", "2019-08-20T00:00:00", "2019-10-08T00:00:00", "2019-12-31T00:00:00", "2020-03-31T00:00:00", "2020-06-30T00:00:00", "2020-09-29T00:00:00", "2020-12-29T00:00:00", "2021-12-30T00:00:00", "2022-06-30T00:00:00" ], "yaxis": "y2", "z": [ [ 1, 0.6396672908679624, 0.4843341365934274, 0.5234823815000352, 0.5310356374929155, 0.5320743709484906, 0.5310593450564083, 0.5326277947315519, 0.5337449407969953, 0.5337513652472036, 0.5222606928881004, 0.5074496905370499, 0.4737601255960541, 0.47465901132352845, 0.4737043213067892 ], [ 0.6396672908679624, 1, 0.43007413384337706, 0.46479156468659183, 0.4699900765005218, 0.4702623253866739, 0.4693326401470681, 0.4714589462424243, 0.4739341551155932, 0.47350213179106915, 0.48958642455074314, 0.44828712876962545, 0.42312202347594274, 0.4230836596657993, 0.42263414942713534 ], [ 0.4843341365934274, 0.43007413384337706, 1, 0.9089503416044139, 0.9017284001543, 0.9002969562466066, 0.8935390681498177, 0.8899094022905394, 0.8969898325671715, 0.8975167714673664, 0.869705802596691, 0.8678912365287855, 0.8303208973365298, 0.8305390434025768, 0.8290912726318587 ], [ 0.5234823815000352, 0.46479156468659183, 0.9089503416044139, 1, 0.9505823343729113, 0.9540438080073016, 0.9477135470242698, 0.947722366226072, 0.9565865159200818, 0.9573137700278977, 0.9225279059706877, 0.920261018091532, 0.8518490447129484, 0.8520781800081971, 0.8508246104888665 ], [ 0.5310356374929155, 0.4699900765005218, 0.9017284001543, 0.9505823343729113, 1, 0.9670600056421988, 0.9664590216206674, 0.9623936410895442, 0.9657274346659225, 0.9648209426553568, 0.9339091959019337, 0.9324725506157981, 0.8473435966681466, 0.8484349867107551, 0.847983610560877 ], [ 0.5320743709484906, 0.4702623253866739, 0.9002969562466066, 0.9540438080073016, 0.9670600056421988, 1, 0.9659515111100809, 0.9689819148298962, 0.9723729812420877, 0.9686582253989845, 0.9373198567426986, 0.9367827942044553, 0.8355518993562235, 0.8367562684115937, 0.8363928552223487 ], [ 0.5310593450564083, 0.4693326401470681, 0.8935390681498177, 0.9477135470242698, 0.9664590216206674, 0.9659515111100809, 1, 0.9725316999440404, 0.967669930593492, 0.9678374841029139, 0.9375075214579225, 0.9358915286820335, 0.8277042106829341, 0.8291797851176553, 0.8287785805877635 ], [ 0.5326277947315519, 0.4714589462424243, 0.8899094022905394, 0.947722366226072, 0.9623936410895442, 0.9689819148298962, 0.9725316999440404, 1, 0.9703791109499944, 0.969107415846449, 0.9419643550464162, 0.9339385760402795, 0.821817047096215, 0.8231080292400599, 0.8225293421584172 ], [ 0.5337449407969953, 0.4739341551155932, 0.8969898325671715, 0.9565865159200818, 0.9657274346659225, 0.9723729812420877, 0.967669930593492, 0.9703791109499944, 1, 0.975865047037384, 0.9444162171988855, 0.938891256642068, 0.8245699514213126, 0.8263693552337398, 0.8251159237843733 ], [ 0.5337513652472036, 0.47350213179106915, 0.8975167714673664, 0.9573137700278977, 0.9648209426553568, 0.9686582253989845, 0.9678374841029139, 0.969107415846449, 0.975865047037384, 1, 0.9436163823313259, 0.9399259095871629, 0.8194701152321582, 0.82100017541867, 0.8202655535964881 ], [ 0.5222606928881004, 0.48958642455074314, 0.869705802596691, 0.9225279059706877, 0.9339091959019337, 0.9373198567426986, 0.9375075214579225, 0.9419643550464162, 0.9444162171988855, 0.9436163823313259, 1, 0.9118165087200062, 0.8011260259047817, 0.8024501576649108, 0.8020124303021742 ], [ 0.5074496905370499, 0.44828712876962545, 0.8678912365287855, 0.920261018091532, 0.9324725506157981, 0.9367827942044553, 0.9358915286820335, 0.9339385760402795, 0.938891256642068, 0.9399259095871629, 0.9118165087200062, 1, 0.787144840960919, 0.7882163176910452, 0.7883262567884973 ], [ 0.4737601255960541, 0.42312202347594274, 0.8303208973365298, 0.8518490447129484, 0.8473435966681466, 0.8355518993562235, 0.8277042106829341, 0.821817047096215, 0.8245699514213126, 0.8194701152321582, 0.8011260259047817, 0.787144840960919, 1, 0.9869474773151184, 0.9821851233036555 ], [ 0.47465901132352845, 0.4230836596657993, 0.8305390434025768, 0.8520781800081971, 0.8484349867107551, 0.8367562684115937, 0.8291797851176553, 0.8231080292400599, 0.8263693552337398, 0.82100017541867, 0.8024501576649108, 0.7882163176910452, 0.9869474773151184, 1, 0.9498991162707977 ], [ 0.4737043213067892, 0.42263414942713534, 0.8290912726318587, 0.8508246104888665, 0.847983610560877, 0.8363928552223487, 0.8287785805877635, 0.8225293421584172, 0.8251159237843733, 0.8202655535964881, 0.8020124303021742, 0.7883262567884973, 0.9821851233036555, 0.9498991162707977, 1 ] ] } ], "layout": { "annotations": [ { "font": { "size": 14 }, "showarrow": false, "text": "Pairwise precision", "x": 0.245, "xanchor": "center", "xref": "paper", "y": 1, "yanchor": "bottom", "yref": "paper" }, { "font": { "size": 14 }, "showarrow": false, "text": "Pairwise F1", "x": 0.755, "xanchor": "center", "xref": "paper", "y": 1, "yanchor": "bottom", "yref": "paper" } ], "coloraxis": { "colorscale": [ [ 0, "rgb(247,251,255)" ], [ 0.125, "rgb(222,235,247)" ], [ 0.25, "rgb(198,219,239)" ], [ 0.375, "rgb(158,202,225)" ], [ 0.5, "rgb(107,174,214)" ], [ 0.625, "rgb(66,146,198)" ], [ 0.75, "rgb(33,113,181)" ], [ 0.875, "rgb(8,81,156)" ], [ 1, "rgb(8,48,107)" ] ] }, "margin": { "t": 60 }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Disambiguation Similarity" }, "xaxis": { "anchor": "y", "constrain": "domain", "domain": [ 0, 0.49 ], "scaleanchor": "y", "title": { "text": "Prediction" } }, "xaxis2": { "anchor": "y2", "domain": [ 0.51, 1 ], "matches": "x", "title": { "text": "Prediction" } }, "yaxis": { "anchor": "x", "autorange": "reversed", "constrain": "domain", "domain": [ 0, 1 ], "title": { "text": "Reference" } }, "yaxis2": { "anchor": "x2", "domain": [ 0, 1 ], "matches": "y", "showticklabels": false, "title": { "text": "Reference" } } } } }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from er_evaluation.plots import plot_comparison\n", "\n", "plot_comparison(predictions, color_continuous_scale=\"Blues\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "The heatmap helps identify changes in the disambiguation results. For instance, you can notice significant changes to the disambiguation algorithm in December 2017 and December 2020." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Cluster Error Metrics\n", "\n", "A key element of our evaluation framework is the definition of record-level and cluster-level error metrics. See the {doc}`Error Analysis <04-error_analysis.html>` page for more information.\n", "\n", "These metrics are used to estimate performance metrics and to perform error analysis. The function below is used to display a scatter plot of any pair of two such metrics, with point size corresponding to sampling weights. See the {py:module}`er_evaluation.error_analysis` module for more information." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "hovertemplate": "expected_relative_extra=%{x}
expected_relative_missing=%{y}
weight=%{marker.size}", "legendgroup": "", "marker": { "color": "#636efa", "opacity": 0.5, "size": [ 1, 1, 0.006289308176100629, 0.047619047619047616, 0.02857142857142857, 0.3333333333333333, 0.043478260869565216, 0.016129032258064516, 0.012195121951219513, 0.3333333333333333, 0.005952380952380952, 0.5, 0.5, 0.1, 0.006097560975609756, 0.07692307692307693, 1, 0.25, 0.1111111111111111, 0.25, 0.07692307692307693, 0.2, 0.16666666666666666, 0.3333333333333333, 0.0045871559633027525, 0.001392757660167131, 0.005494505494505495, 0.017543859649122806, 0.3333333333333333, 0.5, 0.5, 0.3333333333333333, 0.3333333333333333, 0.5, 0.5, 1, 0.03125, 0.16666666666666666, 0.14285714285714285, 0.015384615384615385, 0.3333333333333333, 0.06666666666666667, 0.25, 0.034482758620689655, 0.2, 0.16666666666666666, 0.08333333333333333, 0.015873015873015872, 0.2, 0.1111111111111111, 1, 0.05555555555555555, 0.5, 1, 0.25, 0.09090909090909091, 0.01818181818181818, 0.1, 0.2, 0.09090909090909091, 0.058823529411764705, 0.019230769230769232, 0.01282051282051282, 0.011363636363636364, 0.045454545454545456, 0.0625, 0.125, 0.0136986301369863, 0.02127659574468085, 0.25, 0.1, 0.25, 0.5, 0.005263157894736842, 0.03571428571428571, 0.013333333333333334, 1, 0.09090909090909091, 0.09090909090909091, 0.5, 0.029411764705882353, 0.5, 0.5, 0.5, 1, 0.14285714285714285, 0.25, 0.25, 0.03225806451612903, 1, 0.25, 0.25, 0.1111111111111111, 0.3333333333333333, 1, 0.16666666666666666, 0.058823529411764705, 0.07142857142857142, 0.0625, 0.09090909090909091, 0.041666666666666664, 0.041666666666666664, 1, 0.3333333333333333, 0.05263157894736842, 0.01098901098901099, 0.06666666666666667, 0.008130081300813009, 1, 0.015873015873015872, 1, 0.25, 0.004524886877828055, 0.015384615384615385, 0.012987012987012988, 0.010101010101010102, 0.009345794392523364, 0.015873015873015872, 0.16666666666666666, 0.3333333333333333, 0.043478260869565216, 0.02702702702702703, 0.3333333333333333, 0.029411764705882353, 0.011764705882352941, 0.010869565217391304, 0.04, 0.009174311926605505, 0.2, 0.5, 0.037037037037037035, 0.25, 0.3333333333333333, 0.16666666666666666, 0.3333333333333333, 0.025, 0.2, 0.125, 0.03571428571428571, 0.5, 0.06666666666666667, 0.047619047619047616, 0.027777777777777776, 0.043478260869565216, 0.125, 0.03571428571428571, 0.022222222222222223, 0.03225806451612903, 0.3333333333333333, 0.01639344262295082, 0.04, 0.022222222222222223, 0.25, 0.01, 0.015625, 0.5, 0.1, 0.5, 0.058823529411764705, 0.07692307692307693, 0.09090909090909091, 0.5, 0.02631578947368421, 0.25, 0.09090909090909091, 0.038461538461538464, 0.0009398496240601503, 0.5, 1, 0.3333333333333333, 0.3333333333333333, 0.125, 0.1111111111111111, 0.25, 0.125, 0.16666666666666666, 0.2, 0.5, 0.5, 0.02702702702702703, 0.022222222222222223, 0.05, 0.058823529411764705, 0.03225806451612903, 0.3333333333333333, 0.1111111111111111, 0.023809523809523808, 0.0024096385542168677, 0.041666666666666664, 0.030303030303030304, 0.047619047619047616, 0.07692307692307693, 0.14285714285714285, 0.09090909090909091, 0.04, 0.3333333333333333, 0.16666666666666666, 0.02631578947368421, 0.015625, 0.3333333333333333, 0.09090909090909091, 0.02631578947368421, 1, 0.5, 0.05555555555555555, 0.125, 0.041666666666666664, 0.125, 1, 1, 1, 0.004761904761904762, 1, 0.1, 0.004201680672268907, 0.008695652173913044, 0.017241379310344827, 1, 0.09090909090909091, 0.1111111111111111, 0.1, 0.022222222222222223, 0.14285714285714285, 1, 0.0036101083032490976, 0.02857142857142857, 0.01639344262295082, 0.16666666666666666, 0.008064516129032258, 0.017241379310344827, 0.027777777777777776, 0.009900990099009901, 0.25, 0.25, 0.1111111111111111, 1, 0.2, 0.1, 0.5, 0.07142857142857142, 0.010101010101010102, 0.010526315789473684, 0.2, 0.023809523809523808, 0.3333333333333333, 0.125, 0.5, 0.045454545454545456, 1, 0.011764705882352941, 0.14285714285714285, 0.08333333333333333, 0.029411764705882353, 0.03571428571428571, 0.125, 0.013333333333333334, 1, 0.006211180124223602, 0.5, 0.07142857142857142, 0.2, 0.25, 0.08333333333333333, 0.008771929824561403, 0.14285714285714285, 0.013513513513513514, 0.2, 0.25, 0.058823529411764705, 0.011627906976744186, 0.043478260869565216, 0.027777777777777776, 0.047619047619047616, 0.08333333333333333, 1, 0.1, 1, 0.0031746031746031746, 0.0625, 0.07692307692307693, 0.5, 0.5, 0.06666666666666667, 0.05, 0.02857142857142857, 0.03125, 0.16666666666666666, 0.015625, 0.018518518518518517, 0.5, 0.022222222222222223, 0.2, 0.125, 0.25, 0.3333333333333333, 0.25, 0.14285714285714285, 0.007692307692307693, 0.012195121951219513, 0.3333333333333333, 0.018867924528301886, 0.125, 0.018518518518518517, 1, 1, 1, 0.3333333333333333, 0.045454545454545456, 0.1111111111111111, 0.08333333333333333, 0.1111111111111111, 0.3333333333333333, 0.014925373134328358, 0.1, 0.06666666666666667, 0.5, 0.09090909090909091, 0.013888888888888888, 0.019230769230769232, 0.058823529411764705, 0.3333333333333333, 0.04, 0.25, 0.008928571428571428, 0.05, 0.3333333333333333, 0.058823529411764705, 0.14285714285714285, 0.043478260869565216, 0.010526315789473684, 0.05, 0.02, 0.3333333333333333, 0.14285714285714285, 0.08333333333333333, 0.07692307692307693, 0.3333333333333333, 0.09090909090909091, 0.5, 0.03125, 0.017543859649122806, 0.3333333333333333, 0.02, 1, 0.5, 0.05, 0.03571428571428571, 0.1, 0.25, 0.2, 0.030303030303030304, 0.017857142857142856, 0.14285714285714285, 0.14285714285714285, 0.029411764705882353, 0.03125, 0.0625, 0.02040816326530612, 0.07692307692307693, 0.3333333333333333, 0.2, 0.5, 0.01639344262295082, 0.16666666666666666, 0.008, 0.013157894736842105, 0.0625, 0.008849557522123894, 0.012195121951219513, 0.010869565217391304, 0.02857142857142857, 0.3333333333333333, 1, 0.3333333333333333, 0.008620689655172414, 0.5, 0.07142857142857142, 0.015625, 0.019230769230769232, 0.018867924528301886, 0.013888888888888888, 0.5, 0.1, 0.06666666666666667, 0.05263157894736842, 0.125, 1, 0.07692307692307693, 0.023255813953488372, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 ], "sizemode": "area", "sizeref": 0.0025, "symbol": "circle" }, "mode": "markers", "name": "", "orientation": "v", "showlegend": false, "type": "scatter", "x": [ 0, 0, 0.009868421052631578, 0, 0.4857142857142857, 0, 0.46511627906976744, 0, 0, 0, 0.011764705882352941, 0, 0, 0, 0.029585798816568046, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.008868324657798342, 0.016894087069525665, 0, 0, 0, 0.8125, 0.4000000000000001, 0, 0, 0, 0, 0.25, 0.878048780487805, 0.015151515151515152, 0, 0, 0, 0, 0, 0, 0.7446808510638298, 0, 0.16666666666666666, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.08333333333333333, 0.05555555555555555, 0.018860946745562133, 0, 0.06382978723404255, 0, 0.4482758620689655, 0, 0, 0.0784313725490196, 0, 0, 0, 0, 0, 0, 0.006666666666666667, 0, 0, 0.3888888888888889, 0, 0, 0, 0, 0.5, 0, 0, 0, 0, 0, 0, 0, 0.2, 0, 0, 0, 0.25, 0, 0, 0, 0, 0, 0.14285714285714285, 0, 0, 0, 0, 0, 0.22641509433962265, 0, 0.045454545454545456, 0, 0, 0.008968244262361908, 0, 0.1046511627906977, 0, 0, 0, 0, 0.9473684210526315, 0, 0.2127659574468085, 0, 0, 0, 0, 0, 0, 0.4444444444444445, 0.5, 0, 0, 0.7272727272727272, 0, 0, 0.22065217391304345, 0.16666666666666666, 0, 0, 0, 0, 0, 0, 0, 0.7435897435897436, 0, 0, 0, 0.4000000000000001, 0, 0, 0, 0, 0.047549019607843135, 0, 0, 0, 0.5, 0, 0, 0, 0, 0, 0, 0.725, 0, 0, 0, 0, 0.9361702127659575, 0.9142857142857143, 0, 0, 0, 0, 0.14285714285714285, 0, 0.3333333333333333, 0, 0, 0.23728813559322035, 0, 0, 0, 0, 0.6086956521739131, 0, 0.10367170626349892, 0, 0, 0, 0, 0.24489795918367346, 0, 0.32432432432432434, 0, 0.4000000000000001, 0.07317073170731707, 0, 0, 0, 0, 0, 0.8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0041841004184100415, 0, 0.18309859154929578, 0, 0.7708333333333335, 0, 0, 0, 0.36363636363636365, 0, 0, 0.6982758620689655, 0.031746031746031744, 0, 0, 0.3313953488372093, 0, 0, 0, 0, 0, 0.75, 0, 0, 0, 0.22222222222222218, 0, 0, 0, 0, 0, 0.6190476190476191, 0, 0, 0.75, 0.6659454586432274, 0, 0.14285714285714285, 0, 0, 0.4666666666666667, 0, 0, 0.030120481927710843, 0, 0.391304347826087, 0, 0, 0.14285714285714285, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.07692307692307693, 0, 0, 0, 0, 0, 0.07142857142857142, 0.3333333333333333, 0, 0.16666666666666666, 0, 0, 0, 0, 0.07246376811594203, 0, 0.3333333333333333, 0, 0, 0, 0.125, 0, 0, 0, 0.029850746268656716, 0, 0.88, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.4000000000000001, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.42857142857142855, 0.23809523809523808, 0, 0, 0, 0, 0, 0.5050438596491228, 0.4117647058823529, 0.05181818181818182, 0, 0, 0, 0, 0, 0.15384615384615385, 0, 0, 0.09523809523809523, 0, 0.0196078431372549, 0, 0, 0, 0, 0.23076923076923078, 0.2, 0, 0.05714285714285714, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.5399159663865546, 0, 0, 0, 0, 0, 0, 0.0423728813559322, 0, 0, 0.14634146341463414, 0, 0, 0, 0, 0, 0, 0.09859154929577464, 0.11716524216524217, 0.03354297693920335, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], "xaxis": "x", "y": [ 0, 0, 0.5389542936288089, 0, 0.05551020408163265, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.14201183431952663, 0, 0, 0, 0, 0, 0.10632773819587006, 0.608187134502924, 0, 0.5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.5246913580246912, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.03772189349112426, 0.09730440499671268, 0, 0, 0, 0, 0, 0, 0, 0.18, 0.375, 0, 0, 0, 0.15217777777777777, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.24489795918367346, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.1171875, 0.24489795918367346, 0.2777777777777778, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.009008824553141828, 0, 0, 0, 0.018516901039392086, 0, 0, 0, 0.46880907372400754, 0, 0, 0, 0, 0.3553875236294896, 0, 0.15284908677720732, 0, 0, 0, 0, 0, 0, 0, 0.1875, 0, 0, 0, 0, 0, 0, 0, 0, 0.21875, 0.06887755102040816, 0, 0, 0, 0.03224939532383768, 0, 0, 0, 0.05859999999999999, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.057314927355983945, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.4897959183673469, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.10493827160493827, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.03388822829964328, 0.10493827160493827, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.019997959391898787, 0, 0, 0, 0, 0, 0, 0, 0, 0.04622837370242215, 0.6530612244897959, 0, 0, 0, 0, 0.16924444444444445, 0, 0, 0, 0, 0, 0.375, 0, 0.2570021545090797, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.03635116598079561, 0, 0.04345679012345679, 0, 0, 0.375, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.10609567901234568, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.020831024930747923, 0, 0.47119999999999995, 0, 0, 0, 0, 0, 0, 0, 0.119140625, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.32, 0, 0, 0, 0, 0.12965668371073777, 0, 0, 0, 0, 0, 0, 0, 0, 0.03403686087990487, 0, 0, 0, 0.1782544378698225, 0.553933784264863, 0, 0, 0, 0, 0, 0, 0, 0, 0.04542996214169821, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], "yaxis": "y" }, { "alignmentgroup": "True", "bingroup": "x", "hovertemplate": "expected_relative_extra=%{x}
count=%{y}", "legendgroup": "", "marker": { "color": "#636efa" }, "name": "", "offsetgroup": "", "opacity": 0.5, "showlegend": false, "type": "histogram", "x": [ 0, 0, 0.009868421052631578, 0, 0.4857142857142857, 0, 0.46511627906976744, 0, 0, 0, 0.011764705882352941, 0, 0, 0, 0.029585798816568046, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.008868324657798342, 0.016894087069525665, 0, 0, 0, 0.8125, 0.4000000000000001, 0, 0, 0, 0, 0.25, 0.878048780487805, 0.015151515151515152, 0, 0, 0, 0, 0, 0, 0.7446808510638298, 0, 0.16666666666666666, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.08333333333333333, 0.05555555555555555, 0.018860946745562133, 0, 0.06382978723404255, 0, 0.4482758620689655, 0, 0, 0.0784313725490196, 0, 0, 0, 0, 0, 0, 0.006666666666666667, 0, 0, 0.3888888888888889, 0, 0, 0, 0, 0.5, 0, 0, 0, 0, 0, 0, 0, 0.2, 0, 0, 0, 0.25, 0, 0, 0, 0, 0, 0.14285714285714285, 0, 0, 0, 0, 0, 0.22641509433962265, 0, 0.045454545454545456, 0, 0, 0.008968244262361908, 0, 0.1046511627906977, 0, 0, 0, 0, 0.9473684210526315, 0, 0.2127659574468085, 0, 0, 0, 0, 0, 0, 0.4444444444444445, 0.5, 0, 0, 0.7272727272727272, 0, 0, 0.22065217391304345, 0.16666666666666666, 0, 0, 0, 0, 0, 0, 0, 0.7435897435897436, 0, 0, 0, 0.4000000000000001, 0, 0, 0, 0, 0.047549019607843135, 0, 0, 0, 0.5, 0, 0, 0, 0, 0, 0, 0.725, 0, 0, 0, 0, 0.9361702127659575, 0.9142857142857143, 0, 0, 0, 0, 0.14285714285714285, 0, 0.3333333333333333, 0, 0, 0.23728813559322035, 0, 0, 0, 0, 0.6086956521739131, 0, 0.10367170626349892, 0, 0, 0, 0, 0.24489795918367346, 0, 0.32432432432432434, 0, 0.4000000000000001, 0.07317073170731707, 0, 0, 0, 0, 0, 0.8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0041841004184100415, 0, 0.18309859154929578, 0, 0.7708333333333335, 0, 0, 0, 0.36363636363636365, 0, 0, 0.6982758620689655, 0.031746031746031744, 0, 0, 0.3313953488372093, 0, 0, 0, 0, 0, 0.75, 0, 0, 0, 0.22222222222222218, 0, 0, 0, 0, 0, 0.6190476190476191, 0, 0, 0.75, 0.6659454586432274, 0, 0.14285714285714285, 0, 0, 0.4666666666666667, 0, 0, 0.030120481927710843, 0, 0.391304347826087, 0, 0, 0.14285714285714285, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.07692307692307693, 0, 0, 0, 0, 0, 0.07142857142857142, 0.3333333333333333, 0, 0.16666666666666666, 0, 0, 0, 0, 0.07246376811594203, 0, 0.3333333333333333, 0, 0, 0, 0.125, 0, 0, 0, 0.029850746268656716, 0, 0.88, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.4000000000000001, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.42857142857142855, 0.23809523809523808, 0, 0, 0, 0, 0, 0.5050438596491228, 0.4117647058823529, 0.05181818181818182, 0, 0, 0, 0, 0, 0.15384615384615385, 0, 0, 0.09523809523809523, 0, 0.0196078431372549, 0, 0, 0, 0, 0.23076923076923078, 0.2, 0, 0.05714285714285714, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.5399159663865546, 0, 0, 0, 0, 0, 0, 0.0423728813559322, 0, 0, 0.14634146341463414, 0, 0, 0, 0, 0, 0, 0.09859154929577464, 0.11716524216524217, 0.03354297693920335, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], "xaxis": "x3", "yaxis": "y3" }, { "alignmentgroup": "True", "bingroup": "y", "hovertemplate": "expected_relative_missing=%{y}
count=%{x}", "legendgroup": "", "marker": { "color": "#636efa" }, "name": "", "offsetgroup": "", "opacity": 0.5, "showlegend": false, "type": "histogram", "xaxis": "x2", "y": [ 0, 0, 0.5389542936288089, 0, 0.05551020408163265, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.14201183431952663, 0, 0, 0, 0, 0, 0.10632773819587006, 0.608187134502924, 0, 0.5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.5246913580246912, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.03772189349112426, 0.09730440499671268, 0, 0, 0, 0, 0, 0, 0, 0.18, 0.375, 0, 0, 0, 0.15217777777777777, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.24489795918367346, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.1171875, 0.24489795918367346, 0.2777777777777778, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.009008824553141828, 0, 0, 0, 0.018516901039392086, 0, 0, 0, 0.46880907372400754, 0, 0, 0, 0, 0.3553875236294896, 0, 0.15284908677720732, 0, 0, 0, 0, 0, 0, 0, 0.1875, 0, 0, 0, 0, 0, 0, 0, 0, 0.21875, 0.06887755102040816, 0, 0, 0, 0.03224939532383768, 0, 0, 0, 0.05859999999999999, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.057314927355983945, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.4897959183673469, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.10493827160493827, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.03388822829964328, 0.10493827160493827, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.019997959391898787, 0, 0, 0, 0, 0, 0, 0, 0, 0.04622837370242215, 0.6530612244897959, 0, 0, 0, 0, 0.16924444444444445, 0, 0, 0, 0, 0, 0.375, 0, 0.2570021545090797, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.03635116598079561, 0, 0.04345679012345679, 0, 0, 0.375, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.10609567901234568, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.020831024930747923, 0, 0.47119999999999995, 0, 0, 0, 0, 0, 0, 0, 0.119140625, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.32, 0, 0, 0, 0, 0.12965668371073777, 0, 0, 0, 0, 0, 0, 0, 0, 0.03403686087990487, 0, 0, 0, 0.1782544378698225, 0.553933784264863, 0, 0, 0, 0, 0, 0, 0, 0, 0.04542996214169821, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], "yaxis": "y2" } ], "layout": { "barmode": "overlay", "legend": { "itemsizing": "constant", "tracegroupgap": 0 }, "margin": { "t": 60 }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Cluster-Wise Error Metrics" }, "xaxis": { "anchor": "y", "domain": [ 0, 0.7363 ], "title": { "text": "expected_relative_extra" } }, "xaxis2": { "anchor": "y2", "domain": [ 0.7413, 1 ], "matches": "x2", "showgrid": true, "showline": false, "showticklabels": false, "ticks": "" }, "xaxis3": { "anchor": "y3", "domain": [ 0, 0.7363 ], "matches": "x", "showgrid": true, "showticklabels": false }, "xaxis4": { "anchor": "y4", "domain": [ 0.7413, 1 ], "matches": "x2", "showgrid": true, "showline": false, "showticklabels": false, "ticks": "" }, "yaxis": { "anchor": "x", "domain": [ 0, 0.7326 ], "title": { "text": "expected_relative_missing" } }, "yaxis2": { "anchor": "x2", "domain": [ 0, 0.7326 ], "matches": "y", "showgrid": true, "showticklabels": false }, "yaxis3": { "anchor": "x3", "domain": [ 0.7426, 1 ], "matches": "y3", "showgrid": true, "showline": false, "showticklabels": false, "ticks": "" }, "yaxis4": { "anchor": "x4", "domain": [ 0.7426, 1 ], "matches": "y3", "showgrid": true, "showline": false, "showticklabels": false, "ticks": "" } } } }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from er_evaluation.plots import plot_cluster_errors\n", "\n", "plot_cluster_errors(predictions[pd.Timestamp('2021-12-30')], reference, weights=\"cluster_size\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Fairness Analysis\n", "\n", "The {py:func}`er_evaluation.plot_performance_disparities` function is used identify subgroups in the data with the largest performance disparity compared to full-data performance. You can use any performance metric estimator provided by the package (see the {py:module}`er_evaluation.estimators` module)." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "cliponaxis": false, "hovertemplate": [ "%{y}: %{x} (group size: 3)", "baseline: %{x} (group size: 401)" ], "legendgroup": "Positive differences", "line": { "color": "limegreen", "width": 8 }, "marker": { "color": [ "white", "#222222" ], "line": { "color": [ "limegreen", "limegreen" ], "width": [ 2, 2 ] }, "size": 6, "symbol": 0 }, "mode": "lines+text+markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.9926537417442056, 0.9371681204404455 ], "xaxis": "x", "y": [ "D", "D" ], "yaxis": "y" }, { "cliponaxis": false, "error_x": { "array": [ 0.0039031333266229067, 0.010706518560063265 ], "color": "black", "thickness": 0.75 }, "hoverinfo": "skip", "legendgroup": "Positive differences", "marker": { "color": [ "white", "#222222" ], "size": 6, "symbol": 0 }, "mode": "markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.9926537417442056, 0.9371681204404455 ], "xaxis": "x", "y": [ "D", "D" ], "yaxis": "y" }, { "cliponaxis": false, "hovertemplate": [ "%{y}: %{x} (group size: 37)", "baseline: %{x} (group size: 401)" ], "legendgroup": "Positive differences", "line": { "color": "limegreen", "width": 8 }, "marker": { "color": [ "white", "#222222" ], "line": { "color": [ "limegreen", "limegreen" ], "width": [ 2, 2 ] }, "size": 6, "symbol": 0 }, "mode": "lines+text+markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.9811196039835679, 0.9371681204404455 ], "xaxis": "x", "y": [ "C", "C" ], "yaxis": "y" }, { "cliponaxis": false, "error_x": { "array": [ 0.013582384231800899, 0.010706518560063265 ], "color": "black", "thickness": 0.75 }, "hoverinfo": "skip", "legendgroup": "Positive differences", "marker": { "color": [ "white", "#222222" ], "size": 6, "symbol": 0 }, "mode": "markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.9811196039835679, 0.9371681204404455 ], "xaxis": "x", "y": [ "C", "C" ], "yaxis": "y" }, { "cliponaxis": false, "hovertemplate": [ "%{y}: %{x} (group size: 43)", "baseline: %{x} (group size: 401)" ], "legendgroup": "Positive differences", "line": { "color": "limegreen", "width": 8 }, "marker": { "color": [ "white", "#222222" ], "line": { "color": [ "limegreen", "limegreen" ], "width": [ 2, 2 ] }, "size": 6, "symbol": 0 }, "mode": "lines+text+markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.961920666529861, 0.9371681204404455 ], "xaxis": "x", "y": [ "B", "B" ], "yaxis": "y" }, { "cliponaxis": false, "error_x": { "array": [ 0.015485989945913534, 0.010706518560063265 ], "color": "black", "thickness": 0.75 }, "hoverinfo": "skip", "legendgroup": "Positive differences", "marker": { "color": [ "white", "#222222" ], "size": 6, "symbol": 0 }, "mode": "markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.961920666529861, 0.9371681204404455 ], "xaxis": "x", "y": [ "B", "B" ], "yaxis": "y" }, { "cliponaxis": false, "hovertemplate": [ "%{y}: %{x} (group size: 22)", "baseline: %{x} (group size: 401)" ], "legendgroup": "Positive differences", "line": { "color": "limegreen", "width": 8 }, "marker": { "color": [ "white", "#222222" ], "line": { "color": [ "limegreen", "limegreen" ], "width": [ 2, 2 ] }, "size": 6, "symbol": 0 }, "mode": "lines+text+markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.9598632139437405, 0.9371681204404455 ], "xaxis": "x", "y": [ "None", "None" ], "yaxis": "y" }, { "cliponaxis": false, "error_x": { "array": [ 0.019624562942953002, 0.010706518560063265 ], "color": "black", "thickness": 0.75 }, "hoverinfo": "skip", "legendgroup": "Positive differences", "marker": { "color": [ "white", "#222222" ], "size": 6, "symbol": 0 }, "mode": "markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.9598632139437405, 0.9371681204404455 ], "xaxis": "x", "y": [ "None", "None" ], "yaxis": "y" }, { "cliponaxis": false, "hovertemplate": [ "%{y}: %{x} (group size: 18)", "baseline: %{x} (group size: 401)" ], "legendgroup": "Positive differences", "line": { "color": "limegreen", "width": 8 }, "marker": { "color": [ "white", "#222222" ], "line": { "color": [ "limegreen", "limegreen" ], "width": [ 2, 2 ] }, "size": 6, "symbol": 0 }, "mode": "lines+text+markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.9477516633300012, 0.9371681204404455 ], "xaxis": "x", "y": [ "F", "F" ], "yaxis": "y" }, { "cliponaxis": false, "error_x": { "array": [ 0.025462716970043608, 0.010706518560063265 ], "color": "black", "thickness": 0.75 }, "hoverinfo": "skip", "legendgroup": "Positive differences", "marker": { "color": [ "white", "#222222" ], "size": 6, "symbol": 0 }, "mode": "markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.9477516633300012, 0.9371681204404455 ], "xaxis": "x", "y": [ "F", "F" ], "yaxis": "y" }, { "cliponaxis": false, "hovertemplate": [ "%{y}: %{x} (group size: 89)", "baseline: %{x} (group size: 401)" ], "legendgroup": "Positive differences", "line": { "color": "limegreen", "width": 8 }, "marker": { "color": [ "white", "#222222" ], "line": { "color": [ "limegreen", "limegreen" ], "width": [ 2, 2 ] }, "size": 6, "symbol": 0 }, "mode": "lines+text+markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.9408635789822435, 0.9371681204404455 ], "xaxis": "x", "y": [ "H", "H" ], "yaxis": "y" }, { "cliponaxis": false, "error_x": { "array": [ 0.01618974650471241, 0.010706518560063265 ], "color": "black", "thickness": 0.75 }, "hoverinfo": "skip", "legendgroup": "Positive differences", "marker": { "color": [ "white", "#222222" ], "size": 6, "symbol": 0 }, "mode": "markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.9408635789822435, 0.9371681204404455 ], "xaxis": "x", "y": [ "H", "H" ], "yaxis": "y" }, { "cliponaxis": false, "hovertemplate": [ "%{y}: %{x} (group size: 102)", "baseline: %{x} (group size: 401)" ], "legendgroup": "Negative differences", "line": { "color": "orangered", "width": 8 }, "marker": { "color": [ "white", "#222222" ], "line": { "color": [ "orangered", "orangered" ], "width": [ 2, 2 ] }, "size": 6, "symbol": 0 }, "mode": "lines+text+markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.9335188813828104, 0.9371681204404455 ], "xaxis": "x", "y": [ "G", "G" ], "yaxis": "y" }, { "cliponaxis": false, "error_x": { "array": [ 0.018208311512484726, 0.010706518560063265 ], "color": "black", "thickness": 0.75 }, "hoverinfo": "skip", "legendgroup": "Negative differences", "marker": { "color": [ "white", "#222222" ], "size": 6, "symbol": 0 }, "mode": "markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.9335188813828104, 0.9371681204404455 ], "xaxis": "x", "y": [ "G", "G" ], "yaxis": "y" }, { "cliponaxis": false, "hovertemplate": [ "%{y}: %{x} (group size: 50)", "baseline: %{x} (group size: 401)" ], "legendgroup": "Negative differences", "line": { "color": "orangered", "width": 8 }, "marker": { "color": [ "white", "#222222" ], "line": { "color": [ "orangered", "orangered" ], "width": [ 2, 2 ] }, "size": 6, "symbol": 0 }, "mode": "lines+text+markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.914767116281536, 0.9371681204404455 ], "xaxis": "x", "y": [ "A", "A" ], "yaxis": "y" }, { "cliponaxis": false, "error_x": { "array": [ 0.03713200137081081, 0.010706518560063265 ], "color": "black", "thickness": 0.75 }, "hoverinfo": "skip", "legendgroup": "Negative differences", "marker": { "color": [ "white", "#222222" ], "size": 6, "symbol": 0 }, "mode": "markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.914767116281536, 0.9371681204404455 ], "xaxis": "x", "y": [ "A", "A" ], "yaxis": "y" }, { "cliponaxis": false, "hovertemplate": [ "%{y}: %{x} (group size: 6)", "baseline: %{x} (group size: 401)" ], "legendgroup": "Negative differences", "line": { "color": "orangered", "width": 8 }, "marker": { "color": [ "white", "#222222" ], "line": { "color": [ "orangered", "orangered" ], "width": [ 2, 2 ] }, "size": 6, "symbol": 0 }, "mode": "lines+text+markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.881404664723032, 0.9371681204404455 ], "xaxis": "x", "y": [ "E", "E" ], "yaxis": "y" }, { "cliponaxis": false, "error_x": { "array": [ 0.07961177144390381, 0.010706518560063265 ], "color": "black", "thickness": 0.75 }, "hoverinfo": "skip", "legendgroup": "Negative differences", "marker": { "color": [ "white", "#222222" ], "size": 6, "symbol": 0 }, "mode": "markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.881404664723032, 0.9371681204404455 ], "xaxis": "x", "y": [ "E", "E" ], "yaxis": "y" }, { "cliponaxis": false, "hovertemplate": [ "%{y}: %{x} (group size: 31)", "baseline: %{x} (group size: 401)" ], "legendgroup": "Negative differences", "line": { "color": "orangered", "width": 8 }, "marker": { "color": [ "white", "#222222" ], "line": { "color": [ "orangered", "orangered" ], "width": [ 2, 2 ] }, "size": 6, "symbol": 0 }, "mode": "lines+text+markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.8321934765194068, 0.9371681204404455 ], "xaxis": "x", "y": [ "Y", "Y" ], "yaxis": "y" }, { "cliponaxis": false, "error_x": { "array": [ 0.09739884432111112, 0.010706518560063265 ], "color": "black", "thickness": 0.75 }, "hoverinfo": "skip", "legendgroup": "Negative differences", "marker": { "color": [ "white", "#222222" ], "size": 6, "symbol": 0 }, "mode": "markers", "opacity": 1, "showlegend": false, "type": "scatter", "x": [ 0.8321934765194068, 0.9371681204404455 ], "xaxis": "x", "y": [ "Y", "Y" ], "yaxis": "y" }, { "legendgroup": "Negative differences", "legendgrouptitle": { "text": "Negative differences" }, "marker": { "color": "white", "line": { "color": "orangered", "width": 2 }, "size": 6, "symbol": 0 }, "mode": "markers", "name": "subgroup score", "type": "scatter", "x": [ null ], "y": [ null ] }, { "legendgroup": "Negative differences", "legendgrouptitle": { "text": "Negative differences" }, "marker": { "color": "#222222", "line": { "color": "orangered", "width": 2 }, "size": 6, "symbol": 0 }, "mode": "markers", "name": "baseline score", "type": "scatter", "x": [ null ], "y": [ null ] }, { "legendgroup": "Positive differences", "legendgrouptitle": { "text": "Positive differences" }, "marker": { "color": "white", "line": { "color": "limegreen", "width": 2 }, "size": 6, "symbol": 0 }, "mode": "markers", "name": "subgroup score", "type": "scatter", "x": [ null ], "y": [ null ] }, { "legendgroup": "Positive differences", "legendgrouptitle": { "text": "Positive differences" }, "marker": { "color": "#222222", "line": { "color": "limegreen", "width": 2 }, "size": 6, "symbol": 0 }, "mode": "markers", "name": "baseline score", "type": "scatter", "x": [ null ], "y": [ null ] } ], "layout": { "height": 400, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "title": { "text": "Largest performance differences
(Showing 10/10 cpc_section categories)" }, "xaxis": { "anchor": "y", "domain": [ 0, 1 ], "title": { "text": "Pairwise F-score score" } }, "yaxis": { "anchor": "x", "domain": [ 0, 1 ], "tickmode": "linear", "title": { "text": "cpc_section" } } } } }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from er_evaluation.plots import plot_performance_disparities\n", "\n", "protected_feature = pv_data['cpc_section'].apply(lambda x: x[0] if isinstance(x, np.ndarray) and len(x) > 0 else 'None')\n", "protected_feature = pd.concat([reference, protected_feature], join=\"inner\", axis=1).groupby(\"unique_id\").agg(\"first\")[\"cpc_section\"]\n", "\n", "plot_performance_disparities(\n", " prediction=predictions[pd.Timestamp('2021-12-30')],\n", " reference=reference,\n", " weights=\"cluster_size\",\n", " protected_feature=protected_feature,\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Error Analysis with Decision Trees\n", "\n", "In order to identify combinations of features leading to performance disparities, we recommend doing error analysis with decision trees.\n", "\n", "For this, we first need to define features associated with each cluster and choose an error metric to target. Here, we use an error indicator representing whether or not a given inventor in our reference dataset is associated with a prediction error. Any other error metric from the {py:module}`er_evaluation.error_analysis` module can be used." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "from statistics import mode\n", "from er_evaluation.error_analysis import error_indicator\n", "from er_evaluation.summary import cluster_sizes\n", "\n", "def flatten_mode(x):\n", " return mode(np.concatenate(x.apply(lambda x: np.unique(x)).values))\n", "\n", "pv_data = load_pv_data()\n", "features_df = (\n", " pv_data.merge(pv_data[\"block\"].value_counts().rename(\"block_size\"), left_on=\"block\", right_index=True)\n", " .assign(num_coauthors=pv_data[\"coinventor_sequence\"].apply(len))\n", " .assign(\n", " year_first=pv_data[\"filing_date\"].apply(lambda x: float(str(x).split(\"-\")[0]) if isinstance(x, str) else np.nan)\n", " )\n", " .assign(\n", " year_last=pv_data[\"filing_date\"].apply(lambda x: float(str(x).split(\"-\")[0]) if isinstance(x, str) else np.nan)\n", " )\n", " .merge(reference.rename(\"reference\"), left_on=\"mention_id\", right_index=True)\n", " .groupby(\"reference\")\n", " .agg(\n", " {\n", " \"raw_inventor_name_first\": mode,\n", " \"raw_inventor_name_last\": mode,\n", " \"patent_id\": \"count\",\n", " \"raw_country\": mode,\n", " \"patent_type\": mode,\n", " \"num_coauthors\": \"mean\",\n", " \"block_size\": \"mean\",\n", " \"cpc_section\": flatten_mode,\n", " \"year_first\": min,\n", " \"year_last\": max,\n", " }\n", " )\n", " .rename(\n", " columns={\n", " \"raw_inventor_name_first\": \"name_first\",\n", " \"raw_inventor_name_last\": \"name_last\",\n", " \"patent_id\": \"prolificness\",\n", " \"raw_country\": \"country\",\n", " \"num_coauthors\": \"avg_coauthors\",\n", " }\n", " )\n", ")\n", "\n", "numerical_features = [\n", " \"prolificness\",\n", " \"avg_coauthors\",\n", " \"block_size\",\n", " \"year_first\",\n", " \"year_last\",\n", "]\n", "categorical_features = [\"country\", \"patent_type\", \"cpc_section\"]" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "pred = predictions[pd.Timestamp(\"2021-12-30 00:00:00\")]\n", "y = error_indicator(pred, reference)\n", "weights = 1/cluster_sizes(reference.dropna())\n", "weights = len(y) * weights / weights.sum()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Afterwards, the function below fits and displays a decision tree modeling the chosen error metric as a function of provided features, with node size corresponding to the (weighted) number of samples in each node." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "hoverinfo": "none", "line": { "color": "black", "width": 1 }, "mode": "lines", "type": "scatter", "x": [ 0, -1.75, null, -1.75, -2.25, null, -1.75, -1.25, null, -1.25, -1.75, null, -1.25, -0.75, null, 0, 1.75, null, 1.75, 0.75, null, 0.75, 0.25, null, 0.75, 1.25, null, 1.75, 2.75, null, 2.75, 2.25, null, 2.75, 3.25, null ], "y": [ 0, 1, null, 1, 2, null, 1, 2, null, 2, 3, null, 2, 3, null, 0, 1, null, 1, 2, null, 2, 3, null, 2, 3, null, 1, 2, null, 2, 3, null, 2, 3, null ] }, { "hoverinfo": "text", "marker": { "color": [ 0.17123766985704647, 0.013513513513513506, 0, 0.2, 0, 1, 0.2738020444987036, 0.10526315789473684, 0.05555555555555556, 1, 0.3582964241528579, 0.2553012854275382, 0.5613329916655765 ], "colorbar": { "title": { "text": "Avg. Error" } }, "colorscale": [ [ 0, "rgb(247,251,255)" ], [ 0.125, "rgb(222,235,247)" ], [ 0.25, "rgb(198,219,239)" ], [ 0.375, "rgb(158,202,225)" ], [ 0.5, "rgb(107,174,214)" ], [ 0.625, "rgb(66,146,198)" ], [ 0.75, "rgb(33,113,181)" ], [ 0.875, "rgb(8,81,156)" ], [ 1, "rgb(8,48,107)" ] ], "line": { "color": "grey", "width": 1 }, "showscale": true, "size": [ 67.72098281548365, 42.51028379526563, 41.049010218904485, 11.05002209962943, 9.883440227058545, 4.941720113529272, 52.71629051005966, 30.462808666593283, 29.650320681175632, 6.988647606005008, 43.02353510909506, 35.04370467083151, 24.95923348636886 ] }, "mode": "markers", "text": [ "Avg. Error: 0.17
Size: 401.0000000000003", "Avg. Error: 0.01
Size: 158.01047428265144", "Avg. Error: 0.00
Size: 147.33409088517502", "Avg. Error: 0.20
Size: 10.676383397476442", "Avg. Error: 0.00
Size: 8.541106717981155", "Avg. Error: 1.00
Size: 2.1352766794952887", "Avg. Error: 0.27
Size: 242.9895257173487", "Avg. Error: 0.11
Size: 81.14051382082097", "Avg. Error: 0.06
Size: 76.86996046183039", "Avg. Error: 1.00
Size: 4.270553358990577", "Avg. Error: 0.36
Size: 161.84901189652774", "Avg. Error: 0.26
Size: 107.37863809848498", "Avg. Error: 0.56
Size: 54.47037379804275" ], "textfont": { "size": 10 }, "type": "scatter", "x": [ 0, -1.75, -2.25, -1.25, -1.75, -0.75, 1.75, 0.75, 0.25, 1.25, 2.75, 2.25, 3.25 ], "y": [ 0, 1, 2, 2, 3, 3, 1, 2, 3, 3, 2, 3, 3 ] } ], "layout": { "annotations": [ { "bgcolor": "rgba(255, 255, 255, 0.75)", "borderpad": 1, "font": { "size": 10 }, "showarrow": false, "text": "", "x": 0, "xanchor": "center", "xref": "x", "y": 0, "yanchor": "top", "yref": "y" }, { "bgcolor": "rgba(255, 255, 255, 0.75)", "borderpad": 1, "font": { "size": 10 }, "showarrow": false, "text": "block_size <= 3.29", "x": -1.75, "xanchor": "center", "xref": "x", "y": 1, "yanchor": "bottom", "yref": "y" }, { "bgcolor": "rgba(255, 255, 255, 0.75)", "borderpad": 1, "font": { "size": 10 }, "showarrow": false, "text": "cpc_section_E <= 0.50", "x": -2.25, "xanchor": "center", "xref": "x", "y": 2, "yanchor": "top", "yref": "y" }, { "bgcolor": "rgba(255, 255, 255, 0.75)", "borderpad": 1, "font": { "size": 10 }, "showarrow": false, "text": "cpc_section_E > 0.50", "x": -1.25, "xanchor": "center", "xref": "x", "y": 2, "yanchor": "bottom", "yref": "y" }, { "bgcolor": "rgba(255, 255, 255, 0.75)", "borderpad": 1, "font": { "size": 10 }, "showarrow": false, "text": "block_size <= 1.50", "x": -1.75, "xanchor": "center", "xref": "x", "y": 3, "yanchor": "top", "yref": "y" }, { "bgcolor": "rgba(255, 255, 255, 0.75)", "borderpad": 1, "font": { "size": 10 }, "showarrow": false, "text": "block_size > 1.50", "x": -0.75, "xanchor": "center", "xref": "x", "y": 3, "yanchor": "bottom", "yref": "y" }, { "bgcolor": "rgba(255, 255, 255, 0.75)", "borderpad": 1, "font": { "size": 10 }, "showarrow": false, "text": "block_size > 3.29", "x": 1.75, "xanchor": "center", "xref": "x", "y": 1, "yanchor": "top", "yref": "y" }, { "bgcolor": "rgba(255, 255, 255, 0.75)", "borderpad": 1, "font": { "size": 10 }, "showarrow": false, "text": "prolificness <= 1.50", "x": 0.75, "xanchor": "center", "xref": "x", "y": 2, "yanchor": "bottom", "yref": "y" }, { "bgcolor": "rgba(255, 255, 255, 0.75)", "borderpad": 1, "font": { "size": 10 }, "showarrow": false, "text": "country_DE <= 0.50", "x": 0.25, "xanchor": "center", "xref": "x", "y": 3, "yanchor": "top", "yref": "y" }, { "bgcolor": "rgba(255, 255, 255, 0.75)", "borderpad": 1, "font": { "size": 10 }, "showarrow": false, "text": "country_DE > 0.50", "x": 1.25, "xanchor": "center", "xref": "x", "y": 3, "yanchor": "bottom", "yref": "y" }, { "bgcolor": "rgba(255, 255, 255, 0.75)", "borderpad": 1, "font": { "size": 10 }, "showarrow": false, "text": "prolificness > 1.50", "x": 2.75, "xanchor": "center", "xref": "x", "y": 2, "yanchor": "top", "yref": "y" }, { "bgcolor": "rgba(255, 255, 255, 0.75)", "borderpad": 1, "font": { "size": 10 }, "showarrow": false, "text": "block_size <= 87.50", "x": 2.25, "xanchor": "center", "xref": "x", "y": 3, "yanchor": "bottom", "yref": "y" }, { "bgcolor": "rgba(255, 255, 255, 0.75)", "borderpad": 1, "font": { "size": 10 }, "showarrow": false, "text": "block_size > 87.50", "x": 3.25, "xanchor": "center", "xref": "x", "y": 3, "yanchor": "top", "yref": "y" } ], "margin": { "b": 0, "l": 0, "r": 0, "t": 0 }, "plot_bgcolor": "white", "showlegend": false, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "xaxis": { "showgrid": false, "showticklabels": false, "zeroline": false }, "yaxis": { "autorange": "reversed", "showgrid": false, "showticklabels": false, "zeroline": false } } } }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from er_evaluation.plots import make_dt_regressor_plot\n", "\n", "make_dt_regressor_plot(\n", " y,\n", " weights,\n", " features_df,\n", " numerical_features,\n", " categorical_features,\n", " max_depth=3,\n", " type=\"tree\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "In addition to the standard tree representation, you can use a sunburst chart or a treemap to visualize the tree with more focus on leaf nodes, where the arc angle or block size corresponds to the (weighted) number of samples in each node." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "branchvalues": "total", "customdata": [ "", "block_size ≤ 3.29", "block_size ≤ 3.29
cpc_section_E ≤ 0.50", "block_size ≤ 3.29
cpc_section_E > 0.50", "block_size ≤ 3.29
cpc_section_E > 0.50
block_size ≤ 1.50", "block_size ≤ 3.29
cpc_section_E > 0.50
block_size > 1.50", "block_size > 3.29", "block_size > 3.29
prolificness ≤ 1.50", "block_size > 3.29
prolificness ≤ 1.50
country_DE ≤ 0.50", "block_size > 3.29
prolificness ≤ 1.50
country_DE > 0.50", "block_size > 3.29
prolificness > 1.50", "block_size > 3.29
prolificness > 1.50
block_size ≤ 87.50", "block_size > 3.29
prolificness > 1.50
block_size > 87.50" ], "hovertemplate": "Avg. Error: %{color:.2f}
Size: %{value}
Path:
%{customdata}", "ids": [ "node_0", "node_1", "node_2", "node_3", "node_4", "node_5", "node_6", "node_7", "node_8", "node_9", "node_10", "node_11", "node_12" ], "labels": [ "", "block_size ≤ 3.29", "cpc_section_E ≤ 0.50", "cpc_section_E > 0.50", "block_size ≤ 1.50", "block_size > 1.50", "block_size > 3.29", "prolificness ≤ 1.50", "country_DE ≤ 0.50", "country_DE > 0.50", "prolificness > 1.50", "block_size ≤ 87.50", "block_size > 87.50" ], "marker": { "colorbar": { "title": { "text": "Avg. Error" } }, "colors": [ 0.17123766985704647, 0.013513513513513506, 0, 0.2, 0, 1, 0.2738020444987036, 0.10526315789473684, 0.05555555555555556, 1, 0.3582964241528579, 0.2553012854275382, 0.5613329916655765 ], "colorscale": [ [ 0, "rgb(247,251,255)" ], [ 0.125, "rgb(222,235,247)" ], [ 0.25, "rgb(198,219,239)" ], [ 0.375, "rgb(158,202,225)" ], [ 0.5, "rgb(107,174,214)" ], [ 0.625, "rgb(66,146,198)" ], [ 0.75, "rgb(33,113,181)" ], [ 0.875, "rgb(8,81,156)" ], [ 1, "rgb(8,48,107)" ] ] }, "parents": [ "", "node_0", "node_1", "node_1", "node_3", "node_3", "node_0", "node_6", "node_7", "node_7", "node_6", "node_10", "node_10" ], "type": "sunburst", "values": [ 401, 158.0104742826514, 147.33409088517493, 10.676383397476442, 8.541106717981155, 2.1352766794952887, 242.9895257173486, 81.14051382082098, 76.8699604618304, 4.270553358990577, 161.84901189652766, 107.37863809848494, 54.47037379804273 ] } ], "layout": { "margin": { "b": 0, "l": 0, "r": 0, "t": 0 }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } } } } }, "metadata": {}, "output_type": "display_data" } ], "source": [ "make_dt_regressor_plot(\n", " y,\n", " weights,\n", " features_df,\n", " numerical_features,\n", " categorical_features,\n", " max_depth=3,\n", " type=\"sunburst\")" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "branchvalues": "total", "customdata": [ "", "block_size ≤ 3.29", "block_size ≤ 3.29
cpc_section_E ≤ 0.50", "block_size ≤ 3.29
cpc_section_E > 0.50", "block_size ≤ 3.29
cpc_section_E > 0.50
block_size ≤ 1.50", "block_size ≤ 3.29
cpc_section_E > 0.50
block_size > 1.50", "block_size > 3.29", "block_size > 3.29
prolificness ≤ 1.50", "block_size > 3.29
prolificness ≤ 1.50
country_DE ≤ 0.50", "block_size > 3.29
prolificness ≤ 1.50
country_DE > 0.50", "block_size > 3.29
prolificness > 1.50", "block_size > 3.29
prolificness > 1.50
block_size ≤ 87.50", "block_size > 3.29
prolificness > 1.50
block_size > 87.50" ], "hovertemplate": "Avg. Error: %{color:.2f}
Size: %{value}
Path:
%{customdata}", "ids": [ "node_0", "node_1", "node_2", "node_3", "node_4", "node_5", "node_6", "node_7", "node_8", "node_9", "node_10", "node_11", "node_12" ], "labels": [ "", "block_size ≤ 3.29", "cpc_section_E ≤ 0.50", "cpc_section_E > 0.50", "block_size ≤ 1.50", "block_size > 1.50", "block_size > 3.29", "prolificness ≤ 1.50", "country_DE ≤ 0.50", "country_DE > 0.50", "prolificness > 1.50", "block_size ≤ 87.50", "block_size > 87.50" ], "marker": { "colorbar": { "title": { "text": "Avg. Error" } }, "colors": [ 0.17123766985704647, 0.013513513513513506, 0, 0.2, 0, 1, 0.2738020444987036, 0.10526315789473684, 0.05555555555555556, 1, 0.3582964241528579, 0.2553012854275382, 0.5613329916655765 ], "colorscale": [ [ 0, "rgb(247,251,255)" ], [ 0.125, "rgb(222,235,247)" ], [ 0.25, "rgb(198,219,239)" ], [ 0.375, "rgb(158,202,225)" ], [ 0.5, "rgb(107,174,214)" ], [ 0.625, "rgb(66,146,198)" ], [ 0.75, "rgb(33,113,181)" ], [ 0.875, "rgb(8,81,156)" ], [ 1, "rgb(8,48,107)" ] ] }, "parents": [ "", "node_0", "node_1", "node_1", "node_3", "node_3", "node_0", "node_6", "node_7", "node_7", "node_6", "node_10", "node_10" ], "type": "treemap", "values": [ 401, 158.0104742826514, 147.33409088517493, 10.676383397476442, 8.541106717981155, 2.1352766794952887, 242.9895257173486, 81.14051382082098, 76.8699604618304, 4.270553358990577, 161.84901189652766, 107.37863809848494, 54.47037379804273 ] } ], "layout": { "margin": { "b": 0, "l": 0, "r": 0, "t": 0 }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ], "sequentialminus": [ [ 0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } } } } }, "metadata": {}, "output_type": "display_data" } ], "source": [ "make_dt_regressor_plot(\n", " y,\n", " weights,\n", " features_df,\n", " numerical_features,\n", " categorical_features,\n", " max_depth=3,\n", " type=\"treemap\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Additional Visualization Functions\n", "\n", "Some visualization functions are not shown here, including:\n", "\n", "- {py:func}`er_evaluation.plot_cluster_sizes_distribution` to visualize cluster size distributions.\n", "- {py:func}`er_evaluation.plot_entropy_curve` to display Hill number curves.\n", "- {py:func}`er_evaluation.add_ests_to_summaries` to add summary statistics estimators to the summary statistics visualization.\n", "- {py:func}`er_evaluation.plot_metrics` to display raw (non-representative) performance metrics." ] } ], "metadata": { "kernelspec": { "display_name": "er-evaluation", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.8" }, "orig_nbformat": 4, "vscode": { "interpreter": { "hash": "a7cfe58095853939316acebc405e10b3a49f46616572691f7cf90ff724f2bfcd" } } }, "nbformat": 4, "nbformat_minor": 2 }