{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Error Analysis"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Error analysis is the process of analyzing errors associated with a given set of predictions. You can investigate different types of errors, their frequencies, and their relationships with explanatory features of interest.\n",
"\n",
"To characterize errors, we define two types of error metrics:\n",
"\n",
"1. Cluster-level error metrics: Quantify errors associated with each cluster.\n",
"2. Record-level error metrics: Quantify errors associated with each record.\n",
"\n",
"We recommend using cluster-level error metrics, as they are easier to interpret and relate to cluster-level features. However, some advanced analyses require using record-level error metrics."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Cluster-level Error Metrics\n",
"\n",
"Cluster-level error metrics quantify errors associated with each ground truth cluster. We provide several metrics in the {py:module}`er_evaluation.error_analysis` module, including:\n",
"\n",
"- **Error Indicator**: This metric indicates whether there is an error associated to each true cluster. For the error indicator metric, an error means that there is no predicted cluster that matches the true cluster. In other words, an error indicator value of 1 means that the true cluster is not in the disambiguation. A value of 0 means that the true cluster was correctly found, i.e. it is part of the disambiguation.\n",
"- **Expected Extra Elements:** This metric represents the expected number of extraneous elements for each true cluster. In other words, it calculates the average number of erroneous links to a other records in a true cluster.\n",
"- **Expected Relative Extra Elements:** This metric represents the expected relative number of extraneous elements for each true cluster. It calculates the average relative number of erroneous links to a random record in a true cluster.\n",
"- **Expected Missing Elements:** This metric represents the expected number of missing elements for each true cluster. It calculates the average number of elements that are missing from the predicted clusters compared to the true clusters.\n",
"- **Expected Relative Missing Elements:** This metric represents the expected relative number of missing elements for each true cluster. It calculates the average relative number of elements that are missing from the predicted clusters compared to the true clusters.\n",
"\n",
"You can find more information about these metrics, including formal mathematical definitions, in the [er_evaluation.error_analysis](https://er-evaluation.readthedocs.io/en/latest/er_evaluation.error_analysis.html) module."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Example\n",
"\n",
"Here is an example based on PatentsView's disambiguation of patent inventor names. The {py:func}`er_evaluation.error_indicator` metric indicates whether or not the predicted disambiguation makes an error for the given true \"reference\" cluster."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"reference\n",
"9unk95ybl10788b3dxzyz0qlt 0\n",
"fl:a._ln:eversole-1 0\n",
"fl:ab_ln:patil-16 1\n",
"fl:ak_ln:ohno-16 0\n",
"fl:ak_ln:sawada-11 1\n",
" ..\n",
"on89lkbvct0i0fbi2jdngxwc1 0\n",
"t88yown1o8l8x6i2wo45xtn3z 0\n",
"uzgor2vfmuk5bytnhr71rgwni 0\n",
"ytt5secbbneclm84c5o8yy75u 0\n",
"zpj3f8n9vln5it7gx0y1v4bkr 0\n",
"Name: error_indicator, Length: 370, dtype: int64"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"import er_evaluation as ee\n",
"\n",
"predictions, reference = ee.load_pv_disambiguations()\n",
"prediction = predictions[pd.Timestamp('2017-08-08')]\n",
"\n",
"ee.error_indicator(prediction, reference)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Error Analysis with Decision Trees\n",
"\n",
"To identify combinations of features leading to performance disparities, we recommend doing error analysis using decision trees. First, define features associated with each cluster and choose an error metric to target. You can use any error metric from the [er_evaluation.error_analysis](https://er-evaluation.readthedocs.io/en/latest/er_evaluation.error_analysis.html) module. We recommend using thresholded 0-1 features for interpretability."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is an example with PatentsView data. First we define cluster-level features to consider."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"from statistics import mode\n",
"import er_evaluation as ee\n",
"\n",
"pv_data = ee.load_pv_data()\n",
"pv_data.set_index(\"mention_id\", inplace=True)\n",
"\n",
"def flatten_mode(x):\n",
" return mode(np.concatenate(x.apply(lambda x: np.unique(x)).values))\n",
"\n",
"features_df = (\n",
" pv_data.merge(pv_data[\"block\"].value_counts().rename(\"block_size\"), left_on=\"block\", right_index=True)\n",
" .assign(num_coauthors=pv_data[\"coinventor_sequence\"].apply(len))\n",
" .assign(\n",
" year_first=pv_data[\"filing_date\"].apply(lambda x: float(str(x).split(\"-\")[0]) if isinstance(x, str) else np.nan)\n",
" )\n",
" .assign(\n",
" year_last=pv_data[\"filing_date\"].apply(lambda x: float(str(x).split(\"-\")[0]) if isinstance(x, str) else np.nan)\n",
" )\n",
" .merge(reference.rename(\"reference\"), left_on=\"mention_id\", right_index=True)\n",
" .groupby(\"reference\")\n",
" .agg(\n",
" {\n",
" \"raw_inventor_name_first\": mode,\n",
" \"raw_inventor_name_last\": mode,\n",
" \"patent_id\": \"count\",\n",
" \"raw_country\": mode,\n",
" \"patent_type\": mode,\n",
" \"num_coauthors\": \"mean\",\n",
" \"block_size\": \"mean\",\n",
" \"cpc_section\": flatten_mode,\n",
" \"year_first\": min,\n",
" \"year_last\": max,\n",
" }\n",
" )\n",
" .rename(\n",
" columns={\n",
" \"raw_inventor_name_first\": \"name_first\",\n",
" \"raw_inventor_name_last\": \"name_last\",\n",
" \"patent_id\": \"prolificness\",\n",
" \"raw_country\": \"country\",\n",
" \"num_coauthors\": \"avg_coauthors\",\n",
" }\n",
" )\n",
")\n",
"\n",
"numerical_features = [\n",
" \"prolificness\",\n",
" \"avg_coauthors\",\n",
" \"block_size\",\n",
" \"year_first\",\n",
" \"year_last\",\n",
"]\n",
"categorical_features = [\"country\", \"patent_type\", \"cpc_section\"]\n",
"\n",
"pred = predictions[pd.Timestamp(\"2021-12-30\")]\n",
"y = ee.error_indicator(pred, reference)\n",
"weights = 1/ee.cluster_sizes(reference.dropna())\n",
"weights = len(y) * weights / weights.sum()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Afterwards, the function below fits and displays a decision tree modeling the chosen error metric as a function of provided features."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.plotly.v1+json": {
"config": {
"plotlyServerURL": "https://plot.ly"
},
"data": [
{
"branchvalues": "total",
"customdata": [
"",
"block_size ≤ 3.29",
"block_size ≤ 3.29
cpc_section_E ≤ 0.50",
"block_size ≤ 3.29
cpc_section_E > 0.50",
"block_size ≤ 3.29
cpc_section_E > 0.50
block_size ≤ 1.50",
"block_size ≤ 3.29
cpc_section_E > 0.50
block_size > 1.50",
"block_size > 3.29",
"block_size > 3.29
prolificness ≤ 1.50",
"block_size > 3.29
prolificness ≤ 1.50
country_DE ≤ 0.50",
"block_size > 3.29
prolificness ≤ 1.50
country_DE > 0.50",
"block_size > 3.29
prolificness > 1.50",
"block_size > 3.29
prolificness > 1.50
block_size ≤ 87.50",
"block_size > 3.29
prolificness > 1.50
block_size > 87.50"
],
"hovertemplate": "Avg. Error: %{color:.2f}
Size: %{value}
Path:
%{customdata}",
"ids": [
"node_0",
"node_1",
"node_2",
"node_3",
"node_4",
"node_5",
"node_6",
"node_7",
"node_8",
"node_9",
"node_10",
"node_11",
"node_12"
],
"labels": [
"",
"block_size ≤ 3.29",
"cpc_section_E ≤ 0.50",
"cpc_section_E > 0.50",
"block_size ≤ 1.50",
"block_size > 1.50",
"block_size > 3.29",
"prolificness ≤ 1.50",
"country_DE ≤ 0.50",
"country_DE > 0.50",
"prolificness > 1.50",
"block_size ≤ 87.50",
"block_size > 87.50"
],
"marker": {
"colorbar": {
"title": {
"text": "Avg. Error"
}
},
"colors": [
0.17123766985704647,
0.013513513513513506,
0,
0.2,
0,
1,
0.2738020444987036,
0.10526315789473684,
0.05555555555555556,
1,
0.3582964241528579,
0.2553012854275382,
0.5613329916655765
],
"colorscale": [
[
0,
"rgb(247,251,255)"
],
[
0.125,
"rgb(222,235,247)"
],
[
0.25,
"rgb(198,219,239)"
],
[
0.375,
"rgb(158,202,225)"
],
[
0.5,
"rgb(107,174,214)"
],
[
0.625,
"rgb(66,146,198)"
],
[
0.75,
"rgb(33,113,181)"
],
[
0.875,
"rgb(8,81,156)"
],
[
1,
"rgb(8,48,107)"
]
]
},
"parents": [
"",
"node_0",
"node_1",
"node_1",
"node_3",
"node_3",
"node_0",
"node_6",
"node_7",
"node_7",
"node_6",
"node_10",
"node_10"
],
"type": "sunburst",
"values": [
401,
158.0104742826514,
147.33409088517493,
10.676383397476442,
8.541106717981155,
2.1352766794952887,
242.9895257173486,
81.14051382082098,
76.8699604618304,
4.270553358990577,
161.84901189652766,
107.37863809848494,
54.47037379804273
]
}
],
"layout": {
"margin": {
"b": 0,
"l": 0,
"r": 0,
"t": 0
},
"template": {
"data": {
"bar": [
{
"error_x": {
"color": "#2a3f5f"
},
"error_y": {
"color": "#2a3f5f"
},
"marker": {
"line": {
"color": "#E5ECF6",
"width": 0.5
},
"pattern": {
"fillmode": "overlay",
"size": 10,
"solidity": 0.2
}
},
"type": "bar"
}
],
"barpolar": [
{
"marker": {
"line": {
"color": "#E5ECF6",
"width": 0.5
},
"pattern": {
"fillmode": "overlay",
"size": 10,
"solidity": 0.2
}
},
"type": "barpolar"
}
],
"carpet": [
{
"aaxis": {
"endlinecolor": "#2a3f5f",
"gridcolor": "white",
"linecolor": "white",
"minorgridcolor": "white",
"startlinecolor": "#2a3f5f"
},
"baxis": {
"endlinecolor": "#2a3f5f",
"gridcolor": "white",
"linecolor": "white",
"minorgridcolor": "white",
"startlinecolor": "#2a3f5f"
},
"type": "carpet"
}
],
"choropleth": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "choropleth"
}
],
"contour": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "contour"
}
],
"contourcarpet": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "contourcarpet"
}
],
"heatmap": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "heatmap"
}
],
"heatmapgl": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "heatmapgl"
}
],
"histogram": [
{
"marker": {
"pattern": {
"fillmode": "overlay",
"size": 10,
"solidity": 0.2
}
},
"type": "histogram"
}
],
"histogram2d": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "histogram2d"
}
],
"histogram2dcontour": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "histogram2dcontour"
}
],
"mesh3d": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "mesh3d"
}
],
"parcoords": [
{
"line": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "parcoords"
}
],
"pie": [
{
"automargin": true,
"type": "pie"
}
],
"scatter": [
{
"fillpattern": {
"fillmode": "overlay",
"size": 10,
"solidity": 0.2
},
"type": "scatter"
}
],
"scatter3d": [
{
"line": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatter3d"
}
],
"scattercarpet": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattercarpet"
}
],
"scattergeo": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattergeo"
}
],
"scattergl": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattergl"
}
],
"scattermapbox": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattermapbox"
}
],
"scatterpolar": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterpolar"
}
],
"scatterpolargl": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterpolargl"
}
],
"scatterternary": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterternary"
}
],
"surface": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "surface"
}
],
"table": [
{
"cells": {
"fill": {
"color": "#EBF0F8"
},
"line": {
"color": "white"
}
},
"header": {
"fill": {
"color": "#C8D4E3"
},
"line": {
"color": "white"
}
},
"type": "table"
}
]
},
"layout": {
"annotationdefaults": {
"arrowcolor": "#2a3f5f",
"arrowhead": 0,
"arrowwidth": 1
},
"autotypenumbers": "strict",
"coloraxis": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"colorscale": {
"diverging": [
[
0,
"#8e0152"
],
[
0.1,
"#c51b7d"
],
[
0.2,
"#de77ae"
],
[
0.3,
"#f1b6da"
],
[
0.4,
"#fde0ef"
],
[
0.5,
"#f7f7f7"
],
[
0.6,
"#e6f5d0"
],
[
0.7,
"#b8e186"
],
[
0.8,
"#7fbc41"
],
[
0.9,
"#4d9221"
],
[
1,
"#276419"
]
],
"sequential": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"sequentialminus": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
]
},
"colorway": [
"#636efa",
"#EF553B",
"#00cc96",
"#ab63fa",
"#FFA15A",
"#19d3f3",
"#FF6692",
"#B6E880",
"#FF97FF",
"#FECB52"
],
"font": {
"color": "#2a3f5f"
},
"geo": {
"bgcolor": "white",
"lakecolor": "white",
"landcolor": "#E5ECF6",
"showlakes": true,
"showland": true,
"subunitcolor": "white"
},
"hoverlabel": {
"align": "left"
},
"hovermode": "closest",
"mapbox": {
"style": "light"
},
"paper_bgcolor": "white",
"plot_bgcolor": "#E5ECF6",
"polar": {
"angularaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"bgcolor": "#E5ECF6",
"radialaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
}
},
"scene": {
"xaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
},
"yaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
},
"zaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
}
},
"shapedefaults": {
"line": {
"color": "#2a3f5f"
}
},
"ternary": {
"aaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"baxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"bgcolor": "#E5ECF6",
"caxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
}
},
"title": {
"x": 0.05
},
"xaxis": {
"automargin": true,
"gridcolor": "white",
"linecolor": "white",
"ticks": "",
"title": {
"standoff": 15
},
"zerolinecolor": "white",
"zerolinewidth": 2
},
"yaxis": {
"automargin": true,
"gridcolor": "white",
"linecolor": "white",
"ticks": "",
"title": {
"standoff": 15
},
"zerolinecolor": "white",
"zerolinewidth": 2
}
}
}
}
}
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"ee.make_dt_regressor_plot(\n",
" y,\n",
" weights,\n",
" features_df,\n",
" numerical_features,\n",
" categorical_features,\n",
" max_depth=3,\n",
" type=\"sunburst\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"You can see other visualization options in the {doc}`visualizations ` page."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Fairness Analysis\n",
"\n",
"The er_evaluation.plot_performance_disparities function helps you identify subgroups in the data with the largest performance disparity compared to the overall data performance. You can use any performance metric estimator provided by the package (see the {py:func}`er_evaluation.estimators` module).\n",
"\n",
"Here's an example using \"cpc_section\" (patent classification code section) as a feature to define subgroups:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.plotly.v1+json": {
"config": {
"plotlyServerURL": "https://plot.ly"
},
"data": [
{
"cliponaxis": false,
"hovertemplate": [
"%{y}: %{x} (group size: 3)",
"baseline: %{x} (group size: 401)"
],
"legendgroup": "Positive differences",
"line": {
"color": "limegreen",
"width": 8
},
"marker": {
"color": [
"white",
"#222222"
],
"line": {
"color": [
"limegreen",
"limegreen"
],
"width": [
2,
2
]
},
"size": 6,
"symbol": 0
},
"mode": "lines+text+markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.9926537417442056,
0.9371681204404455
],
"xaxis": "x",
"y": [
"D",
"D"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"error_x": {
"array": [
0.0039031333266229067,
0.010706518560063265
],
"color": "black",
"thickness": 0.75
},
"hoverinfo": "skip",
"legendgroup": "Positive differences",
"marker": {
"color": [
"white",
"#222222"
],
"size": 6,
"symbol": 0
},
"mode": "markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.9926537417442056,
0.9371681204404455
],
"xaxis": "x",
"y": [
"D",
"D"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"hovertemplate": [
"%{y}: %{x} (group size: 37)",
"baseline: %{x} (group size: 401)"
],
"legendgroup": "Positive differences",
"line": {
"color": "limegreen",
"width": 8
},
"marker": {
"color": [
"white",
"#222222"
],
"line": {
"color": [
"limegreen",
"limegreen"
],
"width": [
2,
2
]
},
"size": 6,
"symbol": 0
},
"mode": "lines+text+markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.9811196039835679,
0.9371681204404455
],
"xaxis": "x",
"y": [
"C",
"C"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"error_x": {
"array": [
0.013582384231800899,
0.010706518560063265
],
"color": "black",
"thickness": 0.75
},
"hoverinfo": "skip",
"legendgroup": "Positive differences",
"marker": {
"color": [
"white",
"#222222"
],
"size": 6,
"symbol": 0
},
"mode": "markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.9811196039835679,
0.9371681204404455
],
"xaxis": "x",
"y": [
"C",
"C"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"hovertemplate": [
"%{y}: %{x} (group size: 43)",
"baseline: %{x} (group size: 401)"
],
"legendgroup": "Positive differences",
"line": {
"color": "limegreen",
"width": 8
},
"marker": {
"color": [
"white",
"#222222"
],
"line": {
"color": [
"limegreen",
"limegreen"
],
"width": [
2,
2
]
},
"size": 6,
"symbol": 0
},
"mode": "lines+text+markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.961920666529861,
0.9371681204404455
],
"xaxis": "x",
"y": [
"B",
"B"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"error_x": {
"array": [
0.015485989945913534,
0.010706518560063265
],
"color": "black",
"thickness": 0.75
},
"hoverinfo": "skip",
"legendgroup": "Positive differences",
"marker": {
"color": [
"white",
"#222222"
],
"size": 6,
"symbol": 0
},
"mode": "markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.961920666529861,
0.9371681204404455
],
"xaxis": "x",
"y": [
"B",
"B"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"hovertemplate": [
"%{y}: %{x} (group size: 22)",
"baseline: %{x} (group size: 401)"
],
"legendgroup": "Positive differences",
"line": {
"color": "limegreen",
"width": 8
},
"marker": {
"color": [
"white",
"#222222"
],
"line": {
"color": [
"limegreen",
"limegreen"
],
"width": [
2,
2
]
},
"size": 6,
"symbol": 0
},
"mode": "lines+text+markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.9598632139437405,
0.9371681204404455
],
"xaxis": "x",
"y": [
"None",
"None"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"error_x": {
"array": [
0.019624562942953002,
0.010706518560063265
],
"color": "black",
"thickness": 0.75
},
"hoverinfo": "skip",
"legendgroup": "Positive differences",
"marker": {
"color": [
"white",
"#222222"
],
"size": 6,
"symbol": 0
},
"mode": "markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.9598632139437405,
0.9371681204404455
],
"xaxis": "x",
"y": [
"None",
"None"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"hovertemplate": [
"%{y}: %{x} (group size: 18)",
"baseline: %{x} (group size: 401)"
],
"legendgroup": "Positive differences",
"line": {
"color": "limegreen",
"width": 8
},
"marker": {
"color": [
"white",
"#222222"
],
"line": {
"color": [
"limegreen",
"limegreen"
],
"width": [
2,
2
]
},
"size": 6,
"symbol": 0
},
"mode": "lines+text+markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.9477516633300012,
0.9371681204404455
],
"xaxis": "x",
"y": [
"F",
"F"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"error_x": {
"array": [
0.025462716970043608,
0.010706518560063265
],
"color": "black",
"thickness": 0.75
},
"hoverinfo": "skip",
"legendgroup": "Positive differences",
"marker": {
"color": [
"white",
"#222222"
],
"size": 6,
"symbol": 0
},
"mode": "markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.9477516633300012,
0.9371681204404455
],
"xaxis": "x",
"y": [
"F",
"F"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"hovertemplate": [
"%{y}: %{x} (group size: 89)",
"baseline: %{x} (group size: 401)"
],
"legendgroup": "Positive differences",
"line": {
"color": "limegreen",
"width": 8
},
"marker": {
"color": [
"white",
"#222222"
],
"line": {
"color": [
"limegreen",
"limegreen"
],
"width": [
2,
2
]
},
"size": 6,
"symbol": 0
},
"mode": "lines+text+markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.9408635789822435,
0.9371681204404455
],
"xaxis": "x",
"y": [
"H",
"H"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"error_x": {
"array": [
0.01618974650471241,
0.010706518560063265
],
"color": "black",
"thickness": 0.75
},
"hoverinfo": "skip",
"legendgroup": "Positive differences",
"marker": {
"color": [
"white",
"#222222"
],
"size": 6,
"symbol": 0
},
"mode": "markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.9408635789822435,
0.9371681204404455
],
"xaxis": "x",
"y": [
"H",
"H"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"hovertemplate": [
"%{y}: %{x} (group size: 102)",
"baseline: %{x} (group size: 401)"
],
"legendgroup": "Negative differences",
"line": {
"color": "orangered",
"width": 8
},
"marker": {
"color": [
"white",
"#222222"
],
"line": {
"color": [
"orangered",
"orangered"
],
"width": [
2,
2
]
},
"size": 6,
"symbol": 0
},
"mode": "lines+text+markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.9335188813828104,
0.9371681204404455
],
"xaxis": "x",
"y": [
"G",
"G"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"error_x": {
"array": [
0.018208311512484726,
0.010706518560063265
],
"color": "black",
"thickness": 0.75
},
"hoverinfo": "skip",
"legendgroup": "Negative differences",
"marker": {
"color": [
"white",
"#222222"
],
"size": 6,
"symbol": 0
},
"mode": "markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.9335188813828104,
0.9371681204404455
],
"xaxis": "x",
"y": [
"G",
"G"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"hovertemplate": [
"%{y}: %{x} (group size: 50)",
"baseline: %{x} (group size: 401)"
],
"legendgroup": "Negative differences",
"line": {
"color": "orangered",
"width": 8
},
"marker": {
"color": [
"white",
"#222222"
],
"line": {
"color": [
"orangered",
"orangered"
],
"width": [
2,
2
]
},
"size": 6,
"symbol": 0
},
"mode": "lines+text+markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.914767116281536,
0.9371681204404455
],
"xaxis": "x",
"y": [
"A",
"A"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"error_x": {
"array": [
0.03713200137081081,
0.010706518560063265
],
"color": "black",
"thickness": 0.75
},
"hoverinfo": "skip",
"legendgroup": "Negative differences",
"marker": {
"color": [
"white",
"#222222"
],
"size": 6,
"symbol": 0
},
"mode": "markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.914767116281536,
0.9371681204404455
],
"xaxis": "x",
"y": [
"A",
"A"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"hovertemplate": [
"%{y}: %{x} (group size: 6)",
"baseline: %{x} (group size: 401)"
],
"legendgroup": "Negative differences",
"line": {
"color": "orangered",
"width": 8
},
"marker": {
"color": [
"white",
"#222222"
],
"line": {
"color": [
"orangered",
"orangered"
],
"width": [
2,
2
]
},
"size": 6,
"symbol": 0
},
"mode": "lines+text+markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.881404664723032,
0.9371681204404455
],
"xaxis": "x",
"y": [
"E",
"E"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"error_x": {
"array": [
0.07961177144390381,
0.010706518560063265
],
"color": "black",
"thickness": 0.75
},
"hoverinfo": "skip",
"legendgroup": "Negative differences",
"marker": {
"color": [
"white",
"#222222"
],
"size": 6,
"symbol": 0
},
"mode": "markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.881404664723032,
0.9371681204404455
],
"xaxis": "x",
"y": [
"E",
"E"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"hovertemplate": [
"%{y}: %{x} (group size: 31)",
"baseline: %{x} (group size: 401)"
],
"legendgroup": "Negative differences",
"line": {
"color": "orangered",
"width": 8
},
"marker": {
"color": [
"white",
"#222222"
],
"line": {
"color": [
"orangered",
"orangered"
],
"width": [
2,
2
]
},
"size": 6,
"symbol": 0
},
"mode": "lines+text+markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.8321934765194068,
0.9371681204404455
],
"xaxis": "x",
"y": [
"Y",
"Y"
],
"yaxis": "y"
},
{
"cliponaxis": false,
"error_x": {
"array": [
0.09739884432111112,
0.010706518560063265
],
"color": "black",
"thickness": 0.75
},
"hoverinfo": "skip",
"legendgroup": "Negative differences",
"marker": {
"color": [
"white",
"#222222"
],
"size": 6,
"symbol": 0
},
"mode": "markers",
"opacity": 1,
"showlegend": false,
"type": "scatter",
"x": [
0.8321934765194068,
0.9371681204404455
],
"xaxis": "x",
"y": [
"Y",
"Y"
],
"yaxis": "y"
},
{
"legendgroup": "Negative differences",
"legendgrouptitle": {
"text": "Negative differences"
},
"marker": {
"color": "white",
"line": {
"color": "orangered",
"width": 2
},
"size": 6,
"symbol": 0
},
"mode": "markers",
"name": "subgroup score",
"type": "scatter",
"x": [
null
],
"y": [
null
]
},
{
"legendgroup": "Negative differences",
"legendgrouptitle": {
"text": "Negative differences"
},
"marker": {
"color": "#222222",
"line": {
"color": "orangered",
"width": 2
},
"size": 6,
"symbol": 0
},
"mode": "markers",
"name": "baseline score",
"type": "scatter",
"x": [
null
],
"y": [
null
]
},
{
"legendgroup": "Positive differences",
"legendgrouptitle": {
"text": "Positive differences"
},
"marker": {
"color": "white",
"line": {
"color": "limegreen",
"width": 2
},
"size": 6,
"symbol": 0
},
"mode": "markers",
"name": "subgroup score",
"type": "scatter",
"x": [
null
],
"y": [
null
]
},
{
"legendgroup": "Positive differences",
"legendgrouptitle": {
"text": "Positive differences"
},
"marker": {
"color": "#222222",
"line": {
"color": "limegreen",
"width": 2
},
"size": 6,
"symbol": 0
},
"mode": "markers",
"name": "baseline score",
"type": "scatter",
"x": [
null
],
"y": [
null
]
}
],
"layout": {
"height": 400,
"template": {
"data": {
"bar": [
{
"error_x": {
"color": "#2a3f5f"
},
"error_y": {
"color": "#2a3f5f"
},
"marker": {
"line": {
"color": "#E5ECF6",
"width": 0.5
},
"pattern": {
"fillmode": "overlay",
"size": 10,
"solidity": 0.2
}
},
"type": "bar"
}
],
"barpolar": [
{
"marker": {
"line": {
"color": "#E5ECF6",
"width": 0.5
},
"pattern": {
"fillmode": "overlay",
"size": 10,
"solidity": 0.2
}
},
"type": "barpolar"
}
],
"carpet": [
{
"aaxis": {
"endlinecolor": "#2a3f5f",
"gridcolor": "white",
"linecolor": "white",
"minorgridcolor": "white",
"startlinecolor": "#2a3f5f"
},
"baxis": {
"endlinecolor": "#2a3f5f",
"gridcolor": "white",
"linecolor": "white",
"minorgridcolor": "white",
"startlinecolor": "#2a3f5f"
},
"type": "carpet"
}
],
"choropleth": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "choropleth"
}
],
"contour": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "contour"
}
],
"contourcarpet": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "contourcarpet"
}
],
"heatmap": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "heatmap"
}
],
"heatmapgl": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "heatmapgl"
}
],
"histogram": [
{
"marker": {
"pattern": {
"fillmode": "overlay",
"size": 10,
"solidity": 0.2
}
},
"type": "histogram"
}
],
"histogram2d": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "histogram2d"
}
],
"histogram2dcontour": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "histogram2dcontour"
}
],
"mesh3d": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"type": "mesh3d"
}
],
"parcoords": [
{
"line": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "parcoords"
}
],
"pie": [
{
"automargin": true,
"type": "pie"
}
],
"scatter": [
{
"fillpattern": {
"fillmode": "overlay",
"size": 10,
"solidity": 0.2
},
"type": "scatter"
}
],
"scatter3d": [
{
"line": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatter3d"
}
],
"scattercarpet": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattercarpet"
}
],
"scattergeo": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattergeo"
}
],
"scattergl": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattergl"
}
],
"scattermapbox": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scattermapbox"
}
],
"scatterpolar": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterpolar"
}
],
"scatterpolargl": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterpolargl"
}
],
"scatterternary": [
{
"marker": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"type": "scatterternary"
}
],
"surface": [
{
"colorbar": {
"outlinewidth": 0,
"ticks": ""
},
"colorscale": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"type": "surface"
}
],
"table": [
{
"cells": {
"fill": {
"color": "#EBF0F8"
},
"line": {
"color": "white"
}
},
"header": {
"fill": {
"color": "#C8D4E3"
},
"line": {
"color": "white"
}
},
"type": "table"
}
]
},
"layout": {
"annotationdefaults": {
"arrowcolor": "#2a3f5f",
"arrowhead": 0,
"arrowwidth": 1
},
"autotypenumbers": "strict",
"coloraxis": {
"colorbar": {
"outlinewidth": 0,
"ticks": ""
}
},
"colorscale": {
"diverging": [
[
0,
"#8e0152"
],
[
0.1,
"#c51b7d"
],
[
0.2,
"#de77ae"
],
[
0.3,
"#f1b6da"
],
[
0.4,
"#fde0ef"
],
[
0.5,
"#f7f7f7"
],
[
0.6,
"#e6f5d0"
],
[
0.7,
"#b8e186"
],
[
0.8,
"#7fbc41"
],
[
0.9,
"#4d9221"
],
[
1,
"#276419"
]
],
"sequential": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
],
"sequentialminus": [
[
0,
"#0d0887"
],
[
0.1111111111111111,
"#46039f"
],
[
0.2222222222222222,
"#7201a8"
],
[
0.3333333333333333,
"#9c179e"
],
[
0.4444444444444444,
"#bd3786"
],
[
0.5555555555555556,
"#d8576b"
],
[
0.6666666666666666,
"#ed7953"
],
[
0.7777777777777778,
"#fb9f3a"
],
[
0.8888888888888888,
"#fdca26"
],
[
1,
"#f0f921"
]
]
},
"colorway": [
"#636efa",
"#EF553B",
"#00cc96",
"#ab63fa",
"#FFA15A",
"#19d3f3",
"#FF6692",
"#B6E880",
"#FF97FF",
"#FECB52"
],
"font": {
"color": "#2a3f5f"
},
"geo": {
"bgcolor": "white",
"lakecolor": "white",
"landcolor": "#E5ECF6",
"showlakes": true,
"showland": true,
"subunitcolor": "white"
},
"hoverlabel": {
"align": "left"
},
"hovermode": "closest",
"mapbox": {
"style": "light"
},
"paper_bgcolor": "white",
"plot_bgcolor": "#E5ECF6",
"polar": {
"angularaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"bgcolor": "#E5ECF6",
"radialaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
}
},
"scene": {
"xaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
},
"yaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
},
"zaxis": {
"backgroundcolor": "#E5ECF6",
"gridcolor": "white",
"gridwidth": 2,
"linecolor": "white",
"showbackground": true,
"ticks": "",
"zerolinecolor": "white"
}
},
"shapedefaults": {
"line": {
"color": "#2a3f5f"
}
},
"ternary": {
"aaxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"baxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
},
"bgcolor": "#E5ECF6",
"caxis": {
"gridcolor": "white",
"linecolor": "white",
"ticks": ""
}
},
"title": {
"x": 0.05
},
"xaxis": {
"automargin": true,
"gridcolor": "white",
"linecolor": "white",
"ticks": "",
"title": {
"standoff": 15
},
"zerolinecolor": "white",
"zerolinewidth": 2
},
"yaxis": {
"automargin": true,
"gridcolor": "white",
"linecolor": "white",
"ticks": "",
"title": {
"standoff": 15
},
"zerolinecolor": "white",
"zerolinewidth": 2
}
}
},
"title": {
"text": "Largest performance differences
(Showing 10/10 cpc_section categories)"
},
"xaxis": {
"anchor": "y",
"domain": [
0,
1
],
"title": {
"text": "Pairwise F-score score"
}
},
"yaxis": {
"anchor": "x",
"domain": [
0,
1
],
"tickmode": "linear",
"title": {
"text": "cpc_section"
}
}
}
}
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"protected_feature = pv_data['cpc_section'].apply(lambda x: x[0] if isinstance(x, np.ndarray) and len(x) > 0 else 'None')\n",
"protected_feature = pd.concat([reference, protected_feature], join=\"inner\", axis=1).groupby(\"unique_id\").agg(\"first\")[\"cpc_section\"]\n",
"\n",
"ee.plot_performance_disparities(\n",
" prediction=predictions[pd.Timestamp('2021-12-30')],\n",
" reference=reference,\n",
" weights=\"cluster_size\",\n",
" protected_feature=protected_feature,\n",
")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "er-evaluation",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.8"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}