{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# Machine learning interpretability\n", "\n", "In modern day machine learning it is important to be able to explain how our models \"think\". A simple accuracy score isn't enough. This notebook explores the lesson on interpretability.\n", "\n", "Machine learning interpretability is an increasingly important topic in artificial intelligence. \n", "\n", "As machine learning models become more complex, understanding how they make predictions is becoming more difficult. This lack of transparency can lead to a lack of trust in the model. It can make it difficult to identify and correct errors. Interpretability is the ability to explain how a machine learning model arrived at a particular decision. It is essential to build trust and understanding in these powerful tools. \n", "\n", "This notebook will explore the importance of interpretability and provide practical examples of how it can be achieved." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## How To" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
longitudelatitudehousing_median_agetotal_roomstotal_bedroomspopulationhouseholdsmedian_incomemedian_house_valueocean_proximity
0-122.2337.8841.0880.0129.0322.0126.08.3252452600.0NEAR BAY
1-122.2237.8621.07099.01106.02401.01138.08.3014358500.0NEAR BAY
2-122.2437.8552.01467.0190.0496.0177.07.2574352100.0NEAR BAY
3-122.2537.8552.01274.0235.0558.0219.05.6431341300.0NEAR BAY
4-122.2537.8552.01627.0280.0565.0259.03.8462342200.0NEAR BAY
\n", "
" ], "text/plain": [ " longitude latitude housing_median_age total_rooms total_bedrooms \\\n", "0 -122.23 37.88 41.0 880.0 129.0 \n", "1 -122.22 37.86 21.0 7099.0 1106.0 \n", "2 -122.24 37.85 52.0 1467.0 190.0 \n", "3 -122.25 37.85 52.0 1274.0 235.0 \n", "4 -122.25 37.85 52.0 1627.0 280.0 \n", "\n", " population households median_income median_house_value ocean_proximity \n", "0 322.0 126.0 8.3252 452600.0 NEAR BAY \n", "1 2401.0 1138.0 8.3014 358500.0 NEAR BAY \n", "2 496.0 177.0 7.2574 352100.0 NEAR BAY \n", "3 558.0 219.0 5.6431 341300.0 NEAR BAY \n", "4 565.0 259.0 3.8462 342200.0 NEAR BAY " ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.model_selection import train_test_split\n", "import pandas as pd\n", "\n", "df = pd.read_csv(\"data/housing.csv\")\n", "df.head()" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "df = df.dropna()" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "x_train, x_, y_train, y_ = train_test_split(df.drop([\"longitude\",\"latitude\", \"ocean_proximity\", \"median_house_value\"], axis=1), \n", " df.median_house_value, test_size=.5, stratify=df.ocean_proximity)\n", "\n", "x_val, x_test, y_val, y_test = train_test_split(x_, y_, test_size=.5)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "from sklearn.ensemble import RandomForestRegressor" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "model = RandomForestRegressor()" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "RandomForestRegressor()" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.fit(x_train, y_train)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.6556994491877645" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.score(x_val, y_val)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Influence of Variables" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "C:\\tools\\Anaconda3\\envs\\skillshare\\lib\\site-packages\\sklearn\\utils\\deprecation.py:143: FutureWarning: The sklearn.metrics.scorer module is deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.metrics. Anything that cannot be imported from sklearn.metrics is now part of the private API.\n", " warnings.warn(message, FutureWarning)\n", "C:\\tools\\Anaconda3\\envs\\skillshare\\lib\\site-packages\\sklearn\\utils\\deprecation.py:143: FutureWarning: The sklearn.feature_selection.base module is deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.feature_selection. Anything that cannot be imported from sklearn.feature_selection is now part of the private API.\n", " warnings.warn(message, FutureWarning)\n" ] } ], "source": [ "import eli5" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "\n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
WeightFeature
\n", " 0.5668\n", " \n", " ± 0.0168\n", " \n", " \n", " x5\n", "
\n", " 0.1056\n", " \n", " ± 0.0121\n", " \n", " \n", " x3\n", "
\n", " 0.0957\n", " \n", " ± 0.0107\n", " \n", " \n", " x0\n", "
\n", " 0.0902\n", " \n", " ± 0.0134\n", " \n", " \n", " x2\n", "
\n", " 0.0736\n", " \n", " ± 0.0125\n", " \n", " \n", " x1\n", "
\n", " 0.0681\n", " \n", " ± 0.0131\n", " \n", " \n", " x4\n", "
\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", "\n" ], "text/plain": [ "Explanation(estimator='RandomForestRegressor()', description='\\nRandom forest feature importances; values are numbers 0 <= x <= 1;\\nall values sum to 1.\\n', error=None, method='feature importances', is_regression=True, targets=None, feature_importances=FeatureImportances(importances=[FeatureWeight(feature='x5', weight=0.5667877033603446, std=0.00841680749109398, value=None), FeatureWeight(feature='x3', weight=0.10564312361305908, std=0.006052261117665171, value=None), FeatureWeight(feature='x0', weight=0.09566374264038832, std=0.005333010779630694, value=None), FeatureWeight(feature='x2', weight=0.09023838276674524, std=0.006687473112443481, value=None), FeatureWeight(feature='x1', weight=0.07361192693024816, std=0.006256139007962976, value=None), FeatureWeight(feature='x4', weight=0.06805512068921464, std=0.006545124891306123, value=None)], remaining=0), decision_tree=None, highlight_spaces=None, transition_features=None, image=None)" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "eli5.explain_weights(model)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", "\n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "

\n", " \n", " \n", " y\n", " \n", "\n", "\n", " \n", " (score 268513.000)\n", "\n", "top features\n", "

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "
\n", " Contribution?\n", " Feature
\n", " +206727.112\n", " \n", " <BIAS>\n", "
\n", " +67946.952\n", " \n", " median_income\n", "
\n", " +17204.413\n", " \n", " total_bedrooms\n", "
\n", " +4310.367\n", " \n", " housing_median_age\n", "
\n", " -753.897\n", " \n", " total_rooms\n", "
\n", " -945.697\n", " \n", " households\n", "
\n", " -25976.250\n", " \n", " population\n", "
\n", "\n", " \n", " \n", "\n", " \n", "\n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", "\n" ], "text/plain": [ "Explanation(estimator='RandomForestRegressor()', description='\\nFeatures with largest coefficients.\\n\\nFeature weights are calculated by following decision paths in trees\\nof an ensemble (or a single tree for DecisionTreeClassifier).\\nEach node of the tree has an output score, and contribution of a feature\\non the decision path is how much the score changes from parent to child.\\nWeights of all features sum to the output score or proba of the estimator.\\n\\nCaveats:\\n1. Feature weights just show if the feature contributed positively or\\n negatively to the final score, and does not show how increasing or\\n decreasing the feature value will change the prediction.\\n2. In some cases, feature weight can be close to zero for an important feature.\\n For example, in a single tree that computes XOR function, the feature at the\\n top of the tree will have zero weight because expected scores for both\\n branches are equal, so decision at the top feature does not change the\\n expected score. For an ensemble predicting XOR functions it might not be\\n a problem, but it is not reliable if most trees happen to choose the same\\n feature at the top.\\n', error=None, method='decision path', is_regression=True, targets=[TargetExplanation(target='y', feature_weights=FeatureWeights(pos=[FeatureWeight(feature='', weight=206727.1120379797, std=None, value=1.0), FeatureWeight(feature='median_income', weight=67946.95215258111, std=None, value=5.4562), FeatureWeight(feature='total_bedrooms', weight=17204.41294790117, std=None, value=667.0), FeatureWeight(feature='housing_median_age', weight=4310.366776858451, std=None, value=28.0)], neg=[FeatureWeight(feature='population', weight=-25976.249508324312, std=None, value=2048.0), FeatureWeight(feature='households', weight=-945.6970320239207, std=None, value=685.0), FeatureWeight(feature='total_rooms', weight=-753.8973749721359, std=None, value=4485.0)], pos_remaining=0, neg_remaining=0), proba=None, score=268513.0, weighted_spans=None, heatmap=None)], feature_importances=None, decision_tree=None, highlight_spaces=None, transition_features=None, image=None)" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", "\n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "

\n", " \n", " \n", " y\n", " \n", "\n", "\n", " \n", " (score 147431.000)\n", "\n", "top features\n", "

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "
\n", " Contribution?\n", " Feature
\n", " +206727.112\n", " \n", " <BIAS>\n", "
\n", " +22868.678\n", " \n", " total_rooms\n", "
\n", " -3110.917\n", " \n", " households\n", "
\n", " -6803.699\n", " \n", " total_bedrooms\n", "
\n", " -9159.044\n", " \n", " housing_median_age\n", "
\n", " -20468.350\n", " \n", " population\n", "
\n", " -42622.781\n", " \n", " median_income\n", "
\n", "\n", " \n", " \n", "\n", " \n", "\n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", "\n" ], "text/plain": [ "Explanation(estimator='RandomForestRegressor()', description='\\nFeatures with largest coefficients.\\n\\nFeature weights are calculated by following decision paths in trees\\nof an ensemble (or a single tree for DecisionTreeClassifier).\\nEach node of the tree has an output score, and contribution of a feature\\non the decision path is how much the score changes from parent to child.\\nWeights of all features sum to the output score or proba of the estimator.\\n\\nCaveats:\\n1. Feature weights just show if the feature contributed positively or\\n negatively to the final score, and does not show how increasing or\\n decreasing the feature value will change the prediction.\\n2. In some cases, feature weight can be close to zero for an important feature.\\n For example, in a single tree that computes XOR function, the feature at the\\n top of the tree will have zero weight because expected scores for both\\n branches are equal, so decision at the top feature does not change the\\n expected score. For an ensemble predicting XOR functions it might not be\\n a problem, but it is not reliable if most trees happen to choose the same\\n feature at the top.\\n', error=None, method='decision path', is_regression=True, targets=[TargetExplanation(target='y', feature_weights=FeatureWeights(pos=[FeatureWeight(feature='', weight=206727.1120379797, std=None, value=1.0), FeatureWeight(feature='total_rooms', weight=22868.678498102803, std=None, value=1228.0)], neg=[FeatureWeight(feature='median_income', weight=-42622.78104499923, std=None, value=3.0225), FeatureWeight(feature='population', weight=-20468.349589479698, std=None, value=1603.0), FeatureWeight(feature='housing_median_age', weight=-9159.04427985055, std=None, value=30.0), FeatureWeight(feature='total_bedrooms', weight=-6803.698911011207, std=None, value=358.0), FeatureWeight(feature='households', weight=-3110.9167107417757, std=None, value=323.0)], pos_remaining=0, neg_remaining=0), proba=None, score=147431.0, weighted_spans=None, heatmap=None)], feature_importances=None, decision_tree=None, highlight_spaces=None, transition_features=None, image=None)" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", "\n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "

\n", " \n", " \n", " y\n", " \n", "\n", "\n", " \n", " (score 216857.000)\n", "\n", "top features\n", "

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "
\n", " Contribution?\n", " Feature
\n", " +206727.112\n", " \n", " <BIAS>\n", "
\n", " +17186.951\n", " \n", " households\n", "
\n", " +16984.921\n", " \n", " population\n", "
\n", " +15171.230\n", " \n", " total_rooms\n", "
\n", " +6615.827\n", " \n", " housing_median_age\n", "
\n", " +5175.545\n", " \n", " total_bedrooms\n", "
\n", " -51004.586\n", " \n", " median_income\n", "
\n", "\n", " \n", " \n", "\n", " \n", "\n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", "\n" ], "text/plain": [ "Explanation(estimator='RandomForestRegressor()', description='\\nFeatures with largest coefficients.\\n\\nFeature weights are calculated by following decision paths in trees\\nof an ensemble (or a single tree for DecisionTreeClassifier).\\nEach node of the tree has an output score, and contribution of a feature\\non the decision path is how much the score changes from parent to child.\\nWeights of all features sum to the output score or proba of the estimator.\\n\\nCaveats:\\n1. Feature weights just show if the feature contributed positively or\\n negatively to the final score, and does not show how increasing or\\n decreasing the feature value will change the prediction.\\n2. In some cases, feature weight can be close to zero for an important feature.\\n For example, in a single tree that computes XOR function, the feature at the\\n top of the tree will have zero weight because expected scores for both\\n branches are equal, so decision at the top feature does not change the\\n expected score. For an ensemble predicting XOR functions it might not be\\n a problem, but it is not reliable if most trees happen to choose the same\\n feature at the top.\\n', error=None, method='decision path', is_regression=True, targets=[TargetExplanation(target='y', feature_weights=FeatureWeights(pos=[FeatureWeight(feature='', weight=206727.1120379797, std=None, value=1.0), FeatureWeight(feature='households', weight=17186.951141769227, std=None, value=308.0), FeatureWeight(feature='population', weight=16984.921268894872, std=None, value=687.0), FeatureWeight(feature='total_rooms', weight=15171.229967841553, std=None, value=1125.0), FeatureWeight(feature='housing_median_age', weight=6615.827231125529, std=None, value=23.0), FeatureWeight(feature='total_bedrooms', weight=5175.544844203078, std=None, value=273.0)], neg=[FeatureWeight(feature='median_income', weight=-51004.58649181392, std=None, value=2.3182)], pos_remaining=0, neg_remaining=0), proba=None, score=216857.0, weighted_spans=None, heatmap=None)], feature_importances=None, decision_tree=None, highlight_spaces=None, transition_features=None, image=None)" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", "\n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "

\n", " \n", " \n", " y\n", " \n", "\n", "\n", " \n", " (score 217626.000)\n", "\n", "top features\n", "

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "
\n", " Contribution?\n", " Feature
\n", " +206727.112\n", " \n", " <BIAS>\n", "
\n", " +58160.159\n", " \n", " total_rooms\n", "
\n", " +32081.634\n", " \n", " households\n", "
\n", " +13152.518\n", " \n", " total_bedrooms\n", "
\n", " +8281.158\n", " \n", " housing_median_age\n", "
\n", " -7645.868\n", " \n", " population\n", "
\n", " -93130.714\n", " \n", " median_income\n", "
\n", "\n", " \n", " \n", "\n", " \n", "\n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", "\n" ], "text/plain": [ "Explanation(estimator='RandomForestRegressor()', description='\\nFeatures with largest coefficients.\\n\\nFeature weights are calculated by following decision paths in trees\\nof an ensemble (or a single tree for DecisionTreeClassifier).\\nEach node of the tree has an output score, and contribution of a feature\\non the decision path is how much the score changes from parent to child.\\nWeights of all features sum to the output score or proba of the estimator.\\n\\nCaveats:\\n1. Feature weights just show if the feature contributed positively or\\n negatively to the final score, and does not show how increasing or\\n decreasing the feature value will change the prediction.\\n2. In some cases, feature weight can be close to zero for an important feature.\\n For example, in a single tree that computes XOR function, the feature at the\\n top of the tree will have zero weight because expected scores for both\\n branches are equal, so decision at the top feature does not change the\\n expected score. For an ensemble predicting XOR functions it might not be\\n a problem, but it is not reliable if most trees happen to choose the same\\n feature at the top.\\n', error=None, method='decision path', is_regression=True, targets=[TargetExplanation(target='y', feature_weights=FeatureWeights(pos=[FeatureWeight(feature='', weight=206727.1120379797, std=None, value=1.0), FeatureWeight(feature='total_rooms', weight=58160.15902547827, std=None, value=1951.0), FeatureWeight(feature='households', weight=32081.634140011814, std=None, value=813.0), FeatureWeight(feature='total_bedrooms', weight=13152.51803503369, std=None, value=846.0), FeatureWeight(feature='housing_median_age', weight=8281.158449922435, std=None, value=42.0)], neg=[FeatureWeight(feature='median_income', weight=-93130.71359036738, std=None, value=1.5195), FeatureWeight(feature='population', weight=-7645.86809805855, std=None, value=2500.0)], pos_remaining=0, neg_remaining=0), proba=None, score=217626.0, weighted_spans=None, heatmap=None)], feature_importances=None, decision_tree=None, highlight_spaces=None, transition_features=None, image=None)" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", "\n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "

\n", " \n", " \n", " y\n", " \n", "\n", "\n", " \n", " (score 370269.130)\n", "\n", "top features\n", "

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "
\n", " Contribution?\n", " Feature
\n", " +206727.112\n", " \n", " <BIAS>\n", "
\n", " +91543.225\n", " \n", " population\n", "
\n", " +44576.228\n", " \n", " total_bedrooms\n", "
\n", " +15257.757\n", " \n", " households\n", "
\n", " +15210.430\n", " \n", " total_rooms\n", "
\n", " +13843.872\n", " \n", " housing_median_age\n", "
\n", " -16889.494\n", " \n", " median_income\n", "
\n", "\n", " \n", " \n", "\n", " \n", "\n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", "\n", "\n" ], "text/plain": [ "Explanation(estimator='RandomForestRegressor()', description='\\nFeatures with largest coefficients.\\n\\nFeature weights are calculated by following decision paths in trees\\nof an ensemble (or a single tree for DecisionTreeClassifier).\\nEach node of the tree has an output score, and contribution of a feature\\non the decision path is how much the score changes from parent to child.\\nWeights of all features sum to the output score or proba of the estimator.\\n\\nCaveats:\\n1. Feature weights just show if the feature contributed positively or\\n negatively to the final score, and does not show how increasing or\\n decreasing the feature value will change the prediction.\\n2. In some cases, feature weight can be close to zero for an important feature.\\n For example, in a single tree that computes XOR function, the feature at the\\n top of the tree will have zero weight because expected scores for both\\n branches are equal, so decision at the top feature does not change the\\n expected score. For an ensemble predicting XOR functions it might not be\\n a problem, but it is not reliable if most trees happen to choose the same\\n feature at the top.\\n', error=None, method='decision path', is_regression=True, targets=[TargetExplanation(target='y', feature_weights=FeatureWeights(pos=[FeatureWeight(feature='', weight=206727.1120379797, std=None, value=1.0), FeatureWeight(feature='population', weight=91543.22530016846, std=None, value=1026.0), FeatureWeight(feature='total_bedrooms', weight=44576.227773894316, std=None, value=687.0), FeatureWeight(feature='households', weight=15257.75688900795, std=None, value=592.0), FeatureWeight(feature='total_rooms', weight=15210.429938229026, std=None, value=2075.0), FeatureWeight(feature='housing_median_age', weight=13843.871962384443, std=None, value=30.0)], neg=[FeatureWeight(feature='median_income', weight=-16889.49390166382, std=None, value=3.1635)], pos_remaining=0, neg_remaining=0), proba=None, score=370269.13, weighted_spans=None, heatmap=None)], feature_importances=None, decision_tree=None, highlight_spaces=None, transition_features=None, image=None)" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "for x in range(5):\n", " display(eli5.explain_prediction(model, x_train.iloc[x, :]))" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "from sklearn.inspection import permutation_importance" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'importances_mean': array([0.30626287, 0.20438274, 0.49193051, 0.39923199, 0.23444423,\n", " 1.56607312]),\n", " 'importances_std': array([0.00300465, 0.00150659, 0.00796738, 0.00607609, 0.00408509,\n", " 0.01063025]),\n", " 'importances': array([[0.30934077, 0.30191579, 0.30701877, 0.30368526, 0.30935374],\n", " [0.20672229, 0.20405023, 0.20400822, 0.20210153, 0.20503143],\n", " [0.47992992, 0.49622957, 0.49475018, 0.50258828, 0.48615459],\n", " [0.40668523, 0.38972197, 0.39519039, 0.40363647, 0.40092587],\n", " [0.23050688, 0.2363061 , 0.24161848, 0.23186354, 0.23192618],\n", " [1.56854817, 1.5623001 , 1.54913014, 1.58194708, 1.56844011]])}" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "permutation_importance(model, x_train, y_train)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "from sklearn.inspection import plot_partial_dependence" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ0AAAEHCAYAAAB1IpuHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3gc5bX48e/ZVZfVJduyXOSKcW8YE0qMIdSACaGYhBJCICQQSnJvgJv8bkKAe0MKSUghIcClhJqE4gAONTYlgHvFxr3KRVaxetnd8/tjRmYtpNXK1hatz+d59tnZd9rZmd09OzPvvK+oKsYYY0w0eGIdgDHGmKOHJR1jjDFRY0nHGGNM1FjSMcYYEzWWdIwxxkRNUqwDiBeFhYVaWloa6zAMsGTJkv2qWtRTy7N9Gx9svyam7u5XSzqu0tJSFi9eHOswDCAi23pyebZv44Pt18TU3f1qp9eMMcZEjSUdY4wxUdNl0hGRfiLysIjMc1+PEZFrIh+aMcaYRBPOkc6jwGvAAPf1euCWSAVkjDEmcYWTdApV9TkgAKCqPsAf0aiMMcYkpHCSTr2IFAAKICIzgAMRjcoYY0xCCqfK9HeBucBwEXkfKAIuimhUxhhjElKXSUdVl4rI54FjAAE+UdXWiEdmjDEm4YRTe+0GoI+qrlHV1UAfEfl25EMzxhiTaMK5pnOtqla3vVDVKuDayIVkjDEmUYWTdDwiIm0vRMQLpEQuJGOMMYkqnIoErwHPicgfcWqwXQ/8M6JRGWOMSUjhJJ3bgG8C38KpSPA68FAkgzLGGJOYwqm9FgAecB/GGGPMYesy6YjIicCPgSHu9AKoqg6LbGjGGGMSTTin1x4GbgWWYM3fGGOMOQLhJJ0Dqjov4pEYY4xJeOEknX+JyM+B54HmtkJVXRqxqIwxxiSkcJLO8e7ztKAyBWb1fDjGGGMSWTi1106NRiDGGGMSn/UcaowxJmqs51BjjDFRYz2HGmOMiRrrOdQYY0zUWM+hxhhjosZ6DjXGGBM1nZ5eE5EL2x7A+ThJZxRwnlsWkoikichCEVkhImtE5E63PF9E3hCRDe5zXtA8d4jIRhH5RETODCqfKiKr3HH3t/XvIyKpIvKsW/6RiJQGzXOVu44NInJV9zeNMcaYnhbqSOc897kv8Dngbff1qcB8nBYKQmkGZqlqnYgkA++51a4vBN5S1Z+KyO3A7cBtIjIGmAOMxakp96aIjFJVP04L19cBHwKvAmcB84BrgCpVHSEic4B7gUtFJB/4Ec4NrQosEZG5bq+nxhhjYqTTIx1VvVpVr8b50R6jql9W1S/jJIUuqaPOfZnsPhSYDTzmlj8GXOAOzwaeUdVmVd0CbASmi0gxkK2qH6iqAo+3m6dtWX8DTnOPgs4E3lDVSjfRvIGTqIwxxsRQOLXXSlV1d9DrvTin2bokIl4RWQ7sw0kCHwH92pbnPvd1Jy8BdgTNvtMtK3GH25cfMo9blfsAUBBiWe3ju05EFovI4vLy8nDekuklbN8mJtuvvV84SWe+iLwmIl9zr428AvwrnIWrql9VJwEDcY5axoWYXDpaRIjyw50nOL4HVXWaqk4rKioKEZrpbWzfJibbr71fl0lHVW8E/gRMBCYBD6rqd7qzElWtxrkOdBaw1z1lhvu8z51sJzAoaLaBQJlbPrCD8kPmEZEkIAeoDLEsY4wxMRTOkQ6q+ryq3uo+XghnHhEpEpFcdzgdOB1Yh3PPT1ttsquAl9zhucAct0baUGAksNA9BVcrIjPc6zVXtpunbVkXAW+7131eA84QkTy3dtwZbpkxxpgYCqe76gtxaoX1xTlt1dZddXYXsxYDj4mIFye5PaeqL4vIB8BzbqOh24GLcRa4RkSeAz4GfMANbs01gG/htAGXjlNrra1TuYeBJ0RkI84Rzhx3WZUichewyJ3uJ6pa2dV7NcYYE1nhtEjwM+A8VV3bnQWr6kpgcgflFcBpncxzD3BPB+WLgc9cD1LVJtyk1cG4R4BHuhOzMcaYyArn9Nre7iYcY4wxpiPhHOksFpFngRc5tLvqrm4ONcYYYw4RTtLJBhpwLsa3UbpukcAYY4w5RDgNfl4djUCMMcYkvnC6qx4lIm+JyGr39QQR+WHkQzOJLhBQlm635vCMOZqEU5Hgz8AdQCscrJU2J5JBmaPD35fu5MI//JsPNlXEOhRjTJSEk3QyVHVhuzJfJIIxR4/aplbu/ecnTBmcy4xh+bEOxxgTJeFUJNgvIsP5tLvqi4DdoWcxJrTfvr2RivpmHvnaNNzukYwxR4Fwks4NwIPAaBHZBWwBvhrRqExCW7y1kkfe28LFUwcyYWBurMMxxkRROLXXNgOni0gm4FHV2siHZRJVWXUj1/9lKQPz0vnBOWNiHY4xJsrCaXutAKcXzpMAFZH3cNoys6u/plu2VdTzjccW09Tq5+lrjycnIznWIRljQmjxBahuaAGB3dVNvLdxP5MG5XLiiMLDXmY4p9eeAd4Bvuy+/irwLE6r0cZ0aUdlA69/vJf739oAwINXTGVkv6wYR2XM0aeirpltlQ00NPvZsK+W+mYfKUkeGlsCNPn8ZKZ4SU3yUtfsY+XOat7fVEGLL3DIMm48dUTEk06+qt4V9PpuEbmg06mNCfLGx3u57onFqMLEQbn8ds5kBhdkxDosYxJWbVMrH5fV8MneWjbsrWNbZQMHGlvZVdXA/rqWTudL8gi+gNPXpQgMyc/g8uOHMLQoE4DCzBQmD86jf07aEcUXTtL5l4jMAZ5zX1+E03uoMSH5/AH+d95ahhf14c9XTmNoYWasQzImYTS2+Nm8v465K8po9SkV9c2s2nWAzeX1B6fJSk1iaFEmOenJjB7dj5H9+jCsKJOMlCSGFmbSJzUJvyqZKUl4BJp9AVr9AdKSvSR7w+purdvCSTrfBL4L/AWn2rQXqBeR7xJevzrmKPX3pTvZXF7Pn66YagnHmCPkDyjbKxv4cHMFr63Zw783VtDiD5DsFVK8HnIzUjimfxYXTi5hzIBsxg7IoW9WarduSUhL9pKW7I3guwiv9pqdfDdhafUHWL+3ltW7DrB2dy3/WFHGpEG5nDGmX6xDM6bXKatuZNHWSnZWNfL80p1sr2yg1e+c/irJTeeKE4YwriSbE0cU0jfryE55RVM4tdcEp/LAUFW9S0QGAcUdtFJgjjJLtlXy1Ec7qKxvprrROY/c7F50TE/2ckz/LO6+YFxc3fy5bHsVdc0+8jNTGFronGYwJpZ8/gDLdlTz740VbN5fR32zn4r6ZpZtrz44zfFD8zl9TD+GFWYyeXAeI/v2iavvVXeE8437AxAAZgF3AXXA74HjIhiXiTMtvgAb99WxYH05K3ZUs6m8jg376shJT2ZIQQYZKV4unzGECQNzmDAwlyH5GXg88fel+NOCzfxzzR4AMlO8DMzLIC8zmWSvh2ZfgKlD8sjLSGbO9MFkp1mVbhM5G/bW8pcPt/H80l3UNvsQcY5gstKSyUpL4pbTR3Lm2P7kpCczIDc91uH2mHCSzvGqOkVElgGoapWIpEQ4LhMn3t+4nx++uJqtFfWoc2TP0MJMSgsy+Mrxg7n0uEG96mjhx+eP5eoTS6mob+G9jfvZX9tMVUMLja0+WnwBHpi/CYAH39nM1z5XyhfG9OeY/naG2RwZnz/Auj21LNpayZqyGj7aUsGOykaSvcJ5EwYw69i+fH5UEVlHwR+dcH4tWkXEy6dtrxXhHPmYBObzB7j/rQ389l8bGVaYyc2njWRwfgYnjiikX3bvOX/cXv+ctINVPs8ZX/yZ8T5/gI931/CjuWv4xevruf+tjVx9UinfOGkYRVmp0Q7X9FLVDS0s2lrF5vI6Vu46wPsb91Pd0ApAYR+n6vG1Jw/jnPHFFPY5uj5X4SSd+4EXgL4icg9OlWnrTyeBlVU3ctPTy1i8rYqLpg7kJ7PH9qqjmSOR5PUwYWAuL3z7RPbVNnHXy2t56N0tvLB0F7d+YRRjB2QzviSn155PN5HT4nP+sDyzcDt/W7Lz4D0vA3LSmDXaOZKZVppPSQKdKjsc4dRee1JElgCnAQJcoKprIx6ZiYntFQ3MefADapp8/PrSSVwwuSTWIcVM36w0fnvZZL49czi3PrucO55fBcCXJpdwybRBzBiWb8nHMHdFGU99tI2l26tp8QVISfJw2fTBnD9pAKP6ZllzT+10mnREJLiTk33A08HjVLUykoGZ6PEHlLLqRhZuqeSXr39CQ6ufZ785g7EDcmIdWlw4tjibeTefzPq9dby0fBd/XLCJF5btYuLAHKYOyefEEQWcdqxVCz/aHGhs5Sf/+Ji/L93J8KJMrpwxhAmDcjllZCG5GXbZuzOhjnSW4FzHEWAwUOUO5wLbgaERj85ETEOLj3+sKOPxD7bxyZ7ag6cCinPSePIbx1vCaUdEOKZ/Ft8/azTfmjmcF5eX8fRH23nyo2088v4Wzh1fzKh+WaQkeTh3fDGD8tPtKChB1Tf7ePi9Lfz5nc3Ut/i46bSR3HzaSLxxWFszHnWadFR1KICI/BGYq6qvuq/Pxhr77BWq6ltYvqOa9XtrqWlqpak1wO4DjWwur2f93loCCqP7Z3HtKcMYmJfOlMF5HNMvKy6rOseTrLRkrpgxhCtmDKHZ5+d/XlnLGx/v5ZVVTt+G9/5zHVmpSfzg3GO5ZNog2569WKs/wJb99Wwur2fB+n1s2V/P2t21HGhs5Ywx/bj59JH2B62bwrk6fJyqXt/2QlXnichdoWYwsfWvT/bxx/mbWLS1EvcAhiSPkJrkoV9OGkPyMzhjTD8+N6KQ44fadYkjkZrk5c7Z47hz9jiafX4q6lp4cfkuFnxSzu3Pr+KBBZuYXppPXmYKHhEG5aczosi5sW/ioBxSkyLb5Ig5PC2+AA+/t4WH3t1MRb3TSGZWWhLH9MvitGP7csWMIUwenBfjKHuncLur/iGftr12OdBlXzpuywWPA/1xqlg/qKq/ca8VPQuUAluBS1S1yp3nDuAawA/cpKqvueVTgUeBdOBV4GZVVRFJddcx1Y3pUlXd6s5zFZ/WsrtbVR8L4732Wv6AsnBLJY9/sJV5q/cwpCCDG08dwUkjizimn13MjIbUJC8DctP59swRfPOU4byyajd/W7KTBevLqWlqxR/Qg82YAORnpnDC8ALOGNPPajXFEX9A+dZflvDWun3MPKaI2ZMGMKQgkzHF2RFvl+xoEE7SuQynE7cXcJLOO25ZV3zA91R1qYhkAUtE5A3ga8BbqvpTEbkduB24TUTGAHOAscAA4E0RGaWqfuAB4DrgQ5ykcxYwDydBVanqCLcl7HuBS93E9iNgmhvzEhGZ25bcEsGBhlZe/3gPC7dUsnZPDdv2N1Db7CM92cv3vjCK6z4/zP5Fx5DXI5w/cQDnTxxwsEzVabBxW0UDDS0+5q3ew783VfDKSue0XEluOnmZyRT2SeXY4myGFmQyfWg+g/Iz7HpBhKkqa8pqeGrhduat2k1VQys/Pm8MXzvRLl33tHCqTFcCN3d3waq6G9jtDteKyFqgBJgNzHQnewyYD9zmlj+jqs3AFhHZCEwXka1Atqp+ACAijwMX4CSd2cCP3WX9Dfid21bcmcAbbTXs3GR3FkE18OLdtgrnPPL+umaaWv14PEJuegoHGltZtesALy7bRWOrn9yMZCYMzGXK4DyOK83ntGP7HjX31PQ2IsKQgkyGFDgtbp81rhh/QPm4rIZFWytZsr2K+mYfew408d6G/Qcrd3g9wqC8dMaV5DCsqA9FfVIYkJtOUVYqeRkp9ElNotkXoKHFR7LXQ/+ctIg1S59oFm2t5JWVu3nj473sqm4kJcnDOeP6c8qoIi6cMjDW4SWkqPw6iUgpMBn4COjnJiRUdbeI9HUnK8E5kmmz0y1rdYfbl7fNs8Ndlk9EDgAFweUdzBMc13U4R1AMHjz4sN9fT3t3QzlXPbLw4PWY9lKTPJw3cQBXnjCEcQNy7EJ1B+J137bn9QjjB+YwfmAOXw+qEOrzB9jmNmO/q6qRTeV1LN9RzSurdh9sjiiUlCQPqUke0pO9eD1CijvclqjyM1MYkJtGerLX6bFLldyMFDJTvahCU2sARfGI4BEnYTrrVVKTnGV6RBCBFn+AQMAp93ggyeMhJcnDkPwM8jJ7tupwT+7Xl5bv4pZnl5Pi9XDSiEJunDWCs8f1t+rOERbxpCMifYC/A7eoak2Ii9YdjdAQ5Yc7z6cFqg8CDwJMmzbtkPHffW452yoaUFVKCzOZMjiPL00uITM1sptMVbnvjfUU56Rz/2WTKOqTRkaqF59fqWpoITs9mf7ZaXa6pQuh9m1vkOT1MLyoD8OL+hxS3uoPUNXQwo7KBirrW6msb6ahxU9qkpf0FA/NrQH21TZT3+yj2RegqdWPP6C0+AM0tPjZV9vM9soGKupaqGv2RfQ9/OrSiXxpcs8eLfTUfn1m4XbueGEV00vzeeRrx0X8e20+FdEtLSLJOAnnSVV93i3eKyLF7lFOMc6Np+AcjQwKmn0gUOaWD+ygPHienSKSBOQAlW75zHbzzO9O7Mke55+hP6C8s34/zy/dxa/fXM91pwzjgkkl9I1Q+2PvbdzPsu3V3H3BOKYOyT9k3JF2E2t6v2Svh75ZaUfcf4qqUtfsO3hEA1BV30pjqx+AtGQPHhECqgQCEFBFBASh2ecksoA6y0lOcqZt9Qfw+dVNcn6OLY6//h1b/QH+8K9N/OrN9cw8pogHvjqV9BS79hlNoVok+C0dHB20UdWbQi3YvbbyMLBWVe8LGjUXuAr4qfv8UlD5UyJyH05FgpHAQlX1i0itiMzAOT13JfDbdsv6AKdNuLfdWm2vAf8jIm11Gs8A7ggVb3v3XjQh+L2ybEc1v3jtE/7n1XX877x1XDZ9MHfPHtdjp7ZUlaXbq/nfV9dRnJPGxdPsfLKJHBFxm9D/tKw3dQTWXT5/gPc27ucXr3/C6l01XDBpAD+7aCIpSXbtK9pCHeksPsJlnwhcAawSkeVu2X/hJJvnROQanJYNLgZQ1TUi8hzwMU7NtxvcmmsA3+LTKtPz3Ac4Se0Jt9JBJU7tN1S10r2XaJE73U+OpNkeEWHK4DyeunYGG/fV8fgHW3n8g20IHFYnZT5/wDlX7hH2HGjipeW7eGbRDrbsryc1ycMvLp5oNc+M6YYl2ypZsq2K3PQUCrNSUIV9tc3sqGxga0U9S7ZVsbemmYLMFP54+RTOGvfZFsZNdIRqkeCI7mtR1ffo+NoKOI2HdjTPPcA9HZQvBsZ1UN6Em7Q6GPcI8Ei48YZrRN8+/GT2OPqkJvGH+Zsor23m7gvGhX26be6KMn7wwipQKM5NY/3eOgCml+bz7ZnDOWtc/6OiTw1jetI76/fzm7c2fKY8ySMHW9uYPWkAp47ua3/oYiyc7qqLcKo0jwEO/rKq6qwIxhX3/vPMY8jNSOaXr6/ntF8u4NLjBnHJcYMOdiMbCCgf765hybYq9tc1s6+mmS3761m4tZIpg3MZMyCb7ZWNnD9xAOeML2ZYuwvGxpjw3ThrBNeeMuxgp3weEfIzUyjOSSPJqo/HlXAqEjyJ04LAucD1ONdQyiMZVG8gIlx3ynC+MKY/v3j9Ex7991Yeem8LRVmp5GekUHagkdomp3aQRyA/M5WBeencevoovn3qcLuPwpgelOz1kOz10Cc1iVIyYx2OCSGcpFOgqg+LyM2qugBYICILIh1YbzG0MJPff2UK+2qbePPjfSzcUkF9i5/pQ/OZOCiXE0cU0C8rze6lMcYYwuyu2n3eLSLn4lRXtqpV7fTNSuMrxw/mK8fH742IxhgTa+EknbtFJAf4Hk5V5Wzg1ohGZYwxJiGF0/bay+7gAeDUyIZjjDEmkYW6OfT7qvqzzm4S7ermUGOMMaa9UEc6a93nI71J1BhjjAFC3xz6D3ewQVX/GjxORDq8IdMYY4wJJZybRTpqs6xb7ZgZY4wxEPqaztnAOUCJiNwfNCobp200Y4wxpltCXdMpw7mecz6wJKi8FqsybYwx5jCEuqazQkRWA2ccaeOfxhhjDHRxTcftWqBARKz/VmOMMUcsnBYJtgHvi8hcoL6tsF3HbMYYY0yXwkk6Ze7DA2RFNhxjjDGJLJxmcO6MRiDGGGMSX7iduH0fGIt14maMMeYIhHNz6JPAOmAocCewFVgUwZiMMcYkqHCSToGqPgy0quoCVf06MCPCcRljjElA1ombMcaYqLFO3IwxxkRNqLbX0oDrgRFACfCwqlonbkdg5syZAMyfPz+scaGmP9xlHu76wonlcKaNB91531297k5ZOOMOZ7qu9Lb9c7g6e5+Hsw/DXVa44w5nunD0hn0b6prOY8A0YBVwNvDLqERkjDEmYYU6vTZGVccDiMjDwMLohGSMMSZRhTrSaatAgKpaVwbGGGOOWKgjnYkiUuMOC5DuvhZAVTU74tEZY4xJKKKqsY4hLohIOU7jpgCFwP4YhhPsaIxliKoW9dTC2u3bWIun/dmZSMWYqPs13vdppOPr1n61pNMBEVmsqtNiHQdYLImmN2zD3hBjPIn37RVv8YXTIoExxhjTIyzpGGOMiRpLOh17MNYBBLFYEktv2Ia9IcZ4Eu/bK67is2s6xhhjosaOdIwxxkSNJR1jjDFRY0mnHRE5S0Q+EZGNInJ7lNc9SET+JSJrRWSNiNzslueLyBsissF9zotSPF4RWSYiL8cyjngWYp/9WER2ichy93FO0Dx3uJ+vT0TkzKDyqSKyyh13v4hID8a51V32chFZ7JZ1uj9jEWO86qlt14PxPCIi+0RkdVBZ79mXqmoP9wF4gU3AMCAFWIHTBl201l8MTHGHs4D1wBjgZ8DtbvntwL1Riue7wFPAy+7rmMQRz48Q++zHwH90MP0Y93OVitMb7ybA645bCJyA0+rHPODsHoxzK1DYrqzD/RmrGOP10VPbrgfjOQWYAqzujfvSjnQONR3YqKqbVbUFeAaYHa2Vq+puVV3qDtcCa3G6lZiN0+o37vMFkY5FRAYC5wIPBRVHPY54F2KfdWY28IyqNqvqFmAjMF1EioFsVf1AnV+Ex4n89u1sf8ZTjPGqW9uuJ1esqu8AlUcSTyz3pSWdQ5UAO4Je7yT0D0jEiEgpMBn4COinqrvB+ZED+kYhhF8D3wcCQWWxiKPXaLfPAG4UkZXu6ZC20x2dfcZK3OH25T1FgddFZImIXOeWdbY/YxVjvOqJbRdpvWZfWtI5VEfnNKNep1xE+gB/B25R1Zqupo/A+r8I7FPVJdFed2/VwT57ABgOTAJ282l/VJ19xiL92TtRVafg9I11g4icEmLaWMUYr3pi28VK3O1LSzqH2gkMCno9ECiLZgAikozz4/Wkqj7vFu91D4dxn/dFOIwTgfNFZCvOKcZZIvKXGMTRK3S0z1R1r6r6VTUA/JlPT7F09hnb6Q63L+8RqlrmPu8DXnDj6Wx/xiTGeNVD2y7Ses2+tKRzqEXASBEZKiIpwBxgbrRW7tYeeRhYq6r3BY2aC1zlDl8FvBTJOFT1DlUdqKqlONvgbVW9PNpx9Aad7bO2HwDXl4C2mkZzgTkikioiQ4GRwEL3lEitiMxwl3klPbR9RSRTRLLahoEz3Hg6259RjzFe9dS2i0KovWdfRqO2Qm96AOfg1EDaBPwgyus+CecQdyWw3H2cAxQAbwEb3Of8KMY0k09rr8Usjnh9hNhnT+B09b4S54tfHDTPD9zP1ycE1RjC6R5+tTvud7gthvRAjMNwajCtANa0fa5D7c9oxxivj57cdj0Y09M4p2xbcY5YrulN+9KawTHGGBM1dnrNGGNM1FjSMcYYEzWWdIwxxkRNUqwDiBeFhYVaWloa6zAMsGTJkv3ajT7Xu2L7Nj7Yfu29dlQ2UN3YCkD/7DSKslIPjuvufrWk4yotLWXx4sWxDsMAIrKtJ5dn+zY+2H7tnZpa/Uy483W+PsVpsGD2pBJmDCs4OL67+9WSjjHGmE4t2lpJiy/AGWP6c+roI2/5qstrOiLST0QeFpF57usxInLNEa85TohIaXAT4cYYYz713ob9JHuF44fl98jywqlI8CjwGjDAfb0euKVH1m6MMSauLd5WxcSBuWSk9MyJsXCSTqGqPofb2rCq+gB/j6w9fnhjHYA5xLHidB4XtW4ljDGOHZUN3PLMMsqqG/EHlI/Lahg/MKfHlh9O6qoXkQLcFkhFZAZwoMciiA8jYx2AOcRa4HRgoYi8qar1sQ7ImKPByp3V/ObNDby1bh8fbq4kI9VLY6ufcQOim3S+i9N21HAReR8oAi7qsQjiwxZgRKyDMAeNAeYDacBgnCRkjImgbRX1nP+79wE4eWQhG/bWsbnc+b83riSKSUdVl4rI54FjcPpg+ERVW3ssgvjQHOsAzCE+VtVpsQ7CmKPJ0u1VAMwa3Zf7LplITnoyQ+94FYDhRZk9tp4uk46I3IDTT8ga93WeiFymqn/osSiM6YCITFbVZbGOw5hE5vMH8HqEFTsOkJHi5c9XTsPrcfp4e+PWU9ha0UCSt+carwnn9Nq1qvr7theqWiUi1wKWdEykjHGrsW8FvhjjWIxJSB9trmBNWQ1/XbKTwj4pVDe0Mm5AzsGEAzCyXxYj+2X16HrDSToeERF1+0AQES+Q0qNRxJCqbgXGTZs2zfp4iB92es2YCLvx6WWU1x56ZeHak4dGfL3hJJ3XgOdE5I84NdiuB/4Z0aiMMcZEzObyOsprmxmUn851Jw+joE8qG/bWcclxA7ue+QiFk3RuA74JfAunIsHrwEORDMoYY0zkzFu9B4DnvnkCxTnpTuH46Kw7nNprAeAB92GMMaaXe2/DfsYUZ3+acKIonNprJwI/Boa40wugqjossqEZY4zpSYu2VnLVIwtpaPFzzUmRv37TkXBOrz0M3AosIfGavzHGmKPGyyvKaGhxfsZPCOqeIJrCSToHVHVexCMxxhgTUTuqGg8OHze0Z1qN7q5wks6/ROTnwPME3bmvqksjFpUxxpget6bsAF+aXMLdF4wjMzU23amFs9bj3efg+yYUmNXz4RhjjImE/XXN7K1pZuyA7JglHAiv9tqp0QjEGGNM5CzbXg30bOOdh+Oo7znUGGOOBm+v20dmimpnFN4AACAASURBVJcpg/NiGof1HGqMMQlOVXl73V5OGVVESlLPNd55OKznUGOMSXCLt1Wxt6aZ047tF+tQwko6R0PPocYYk7AefX8r2WlJnDO+f6xDsZ5DjTEmUX2yp5adVQ38c80erjlpKBkpsau11sZ6DjXGmASgqtS3+PlkTw0f767l8uMHc95v36PFHyA/M4XrPz881iECIZKOiFzYyahRIoKqPh9qwSKSBrwDpLrr+Zuq/khE8oFngVKcTrouUdUqd547gGtwrhndpKqvueVTcSo0pAOvAjerqopIKvA4MBWoAC51+8dBRK4CfuiGc7eqPhYqXmOM6c1eWl7GLc8uP/h60sBcWvwBAO67ZCL5mfHRDVqoI53z3Oe+wOeAt93XpwLzcVooCKUZmKWqdSKSDLznVru+EHhLVX8qIrcDtwO3icgYYA4wFqem3JsiMkpV/TgtXF8HfIiTdM4C5uEkqCpVHSEic4B7gUvdxPYjnBtaFVgiInPbkpsxxiSavy3Zecjr3/9rIwCv3HQSYwfE9t6cYJ1WJFDVq1X1apwf7TGq+mVV/TJOUuiSOurcl8nuQ4HZQNtRx2PABe7wbOAZVW1W1S3ARmC6iBQD2ar6gdt76ePt5mlb1t+A00REgDOBN1S10k00b+AkKmOMSTiNLX4Wb6vkrLH9eeCrU8hKS+Kfa/aQmuThmB7ubvpIhVN7rVRVdwe93guMCmfhIuIVkeXAPpwk8BHQr2157nNfd/ISYEfQ7DvdshJ3uH35IfO4VbkPAAUhltU+vutEZLGILC4vLw/nLZlewvZtYrL9+ln+gHL78ytpag1w+YwhnD2+mNNGOz+rU4fkkeSN7X057YVTlWG+iLwGPI1zpDIH+Fc4C3dPjU0SkVzgBREZF2Jy6WgRIcoPd57g+B4EHgSYNm3aZ8ab3sv2bWKy/fpZ72/cz0vLy7j+88M5cYTTXcHPLprIFSeUMigv+p20daXLFKiqNwJ/AiYCk4AHVfU73VmJqlbjXAc6C9jrnjLDfd7nTrYTGBQ020CgzC0f2EH5IfOISBKQA1SGWJYxxiSUpdurEIEbTh2Oc3UBUpI8TB2SR9/stBhH91lhHXep6vOqeqv7eCGceUSkyD3CQUTSgdOBdTj3/FzlTnYV8JI7PBeYIyKpIjIUGAksdE/B1YrIDPd6zZXt5mlb1kXA2+51n9eAM0QkT0TygDPcMmOMSSjLtlczqm8WWWnJsQ4lLOF0V30hTq2wvjinrdq6q87uYtZi4DER8eIkt+dU9WUR+QB4zm00dDtwMc4C14jIc8DHgA+4wT09B/AtPq0yPc99gNOr6RMishHnCGeOu6xKEbkLWORO9xNVrezqvRpjTG8SCCjLtldx7oTiWIcStnCu6fwMOE9V13Znwaq6EpjcQXkFcFon89wD3NNB+WLgM9eDVLUJN2l1MO4R4JHuxGyMMb3JmrIaapp8TB0Sm15AD0c4p9f2djfhGGOMibxXVu0mySMHa6v1BuEc6SwWkWeBFzm0u+qubg41xhgTIarKK6vKOHFEIXlx0tpAOMJJOtlAA87F+DZK1y0SGGOMiZCyA03sqGzkGycNi3Uo3RJOg59XRyMQY4wx4Vu10+l+euKg3BhH0j3hdFc9SkTeEpHV7usJIvLDruYzxhjTc2qbWvnus8s55zfvMndFGSt3HiDJI4zuH1/N3HQlnIoEfwbuAFrhYK20OZEMyhhjzKHue2M9L60oo9Uf4Kanl/GH+ZsY0bcPacneWIfWLeEknQxVXdiuzBeJYIwxxnxWIKC8snI3px/bl1duOpkvTXaakpwxrCDGkXVfOBUJ9ovIcD7trvoiYHfoWYwxxvSUZTuq2FfbzDnji0lJ8vCrSyfxn2ceQ15G76m11iacpHMDTgN7o0VkF7AF+GpEo4ojqnqwPSNjjImFJz7YRnqyl1lB9+MMyI2/xjzDEU7ttc3A6SKSCXhUtTbyYcXeNY8uYmdVI9sq65kxrIDxJTlMH5rP54YX4vVYEjLGRMeGvbXMXVHGN04e1mvaVwslnLbXCnB64TwJUBF5D6cts4pIBxdLw4oyERHGD8zhtTV7WLC+HH0bBuWnc+f5Y5k1ul+sQzTGJCifP8D9b29kUF46v3lrA7kZKVx7cu+6H6cz4ZxeewZ4B/iy+/qrwLM4rUYnrB+cO+bg8M8vmkCzL8Cba/dy/1sb+Pqji7n+88O55fSRva7miDEm/v36zQ38zu1uuiAzhUevPo6irNQYR9Uzwkk6+ap6V9Dru0Xkgk6nTkAiQlqyly9OGMAXxvTjRy+t4Y8LNvHO+nKevnYGORm9/5DXGBMf1u6u4Q/zN3LhlBI+P6qIk0cWkd+LmrnpSjhVpv8lInNExOM+LgFeiXRg8So1yctPvzyBP185jQ37arnmsUU0tfq7ntEYY0JQVTbuq+O2v68kOz2Z//7iGGZPKkmohAPhJZ1vAk8BLTgNfj4DfFdEakWkJpLBxbMvjOnHry+dzJLtVVz+0Eds3FcX65CMMb3U2+v28vmfz+f0+xawYW8dP71wPLm9sDp0OMKpvda72liIonMnFOPXyfzX86v4wq8WMKEkh8umD+ZLU0pITbJrPcaYzpVVN3LvP9dxzvhivvfcCgbkpnHXBeOYNbovJb20OnQ4wqm9JjiVB4aq6l0iMggo7qCVgqPS+RMH8LnhBTyzcDuvrtrD7c+v4v63NvDsN09gUH5GrMMzpkepKpvK61m96wDltc0keYWm1gC1Ta3UNfuob/azZX8dja0B/vPMUVbLsxPr99by0LubeWl5GS8tLyM/M4X/u3p6QiebNuFUJPgDEABmAXcBdcDvgeMiGFevUtgnlRtnjeSGU0fw7ob93PDkUr7z9DIe+/p0ctKtkoHpHeqbfazedYBtlQ1sKq+jsq4FjwgV9c3sq22mrtnHgYZWKupbPjOv1yP0SU0iI8XLoPwMSjJTSU8O5+fl6LO9ooEzfvUOAJMH53Li8EKuPGEIfbPTYhxZdITzqTheVaeIyDIAVa0SkcQ82XiERIRTRhXx84sncONTy5j9u/d4+GvHMbyoT6xDM4a6Zh+b9tWxbk8N++taqGlqpbbJR22Tjz0HGlm2vRpfQAFI9goFmakoSl5GCkVZqQzKz6BPShKTBucyZXAexblp+P1KarKH9GSvtdwRpvnr9wHOPX8/v2giI/oeXb8P4SSdVhHx8mnba0U4Rz6mE2eNK+bZb6Zy3eNL+NLv3+f+yyYz85je052s6Z3qmn38dfEOXl65my3762ls8SMCaclefP4ANU2HttObkuShT2oS2WlJFPZJ5ZqThjJjWAFDCzMZmJdOkjecekamu95ZX87g/Aze+f6psQ4lJsJJOvcDLwB9ReQe4CLA+tPpwtQh+bx4w4l8/dFFfO3/FnHKqCJumDmc6UPz7R+hOWKqyt6aZj7aUsGashqWba9ixc4DtPgCjC/J4exx/clMTSIQUBpb/Xg9QnFOOkMKMhg3IIe+2al2Y3MUbauo566XP6asuon1e2u5bPrgWIcUM+HUXntSRJYApwECXKCqayMeWQIYlJ/BP75zEo/+eysPvrOZSx/8kAkDc7jqhFJmTxpg/yRNlwIBZV9tM2t317C9soFtFQ28u6GcbRUNtPidEw4pXg9jS7K5csYQzp1QzOTBeTGO2gTbur+ei/74Ac0+P5MG5TJnyCBu/cKoWIcVM50mHRHJD3q5D3g6eJyqVkYysESRluzl+s8P56oTSvn70p088t4WvvfXFTz03hZuPm0EZ40rjnWIJkZUlR2VjZTXNdHiU6oaWthV1cjm/fXsPtDIxn117DnQdPA6C0B6spfxA3OYdWxfirPTmFaazzH9s0i2PzBx67/nrqHVH+CFb3+OEX3tDpRQRzpLcK7jCDAYqHKHc4HtwNCIR5dA0lO8XD5jCF89fjDzVu/hp/PWcf1flnLlCUP43hnHWC23o4Q/oCzdXsVLy3fx1tp97D7Q9Jlp8jKSGZCbzuTBeQzKS6dfdhpjBmQzJD+DoqxUOz3bi2zcV8s768v5jzNGWcJxdZp0VHUogIj8EZirqq+6r88mwRv7jCQR4ZzxxZw5tj/3vLKWR97fwgvLdnH5jCGcN2EAo/tn4bGuExLCjsoGPtxcQV2zjwXry1m/p5b99S20+AKkJ3s5ZVQhN84aQUluOileD7kZKRTnpJGXYM2eHM0eencLKUmeo/oaTnvhVCQ4TlWvb3uhqvNE5K5QM5iueT3Cf583hgunlPDbtzfwpwWbeGD+JvpmpXLV50r56vGDE7YZjETk8wcoq25iU3kdH26pYHN5Pe9uKKep1bnuMqQggxnDCyjqk8rYkhxmje5Ln1S7jyVRbdlfz3/+dQWLt1Vx5QlDKOiTGC1E94Rwu6v+IfAXnNNtlwNd9qXjtlzwONAfp4r1g6r6G/da0bNAKbAVuERVq9x57gCuAfzATar6mls+FXgUSAdeBW5WVRWRVHcdU92YLlXVre48V/FpLbu7VfWxMN5r1I0ryeFPV0xjX00TC9aX88qq3fz8tU+47431fH5UEV+cUMyYAdmMKOpjFQ/iRCCgbK2oZ1d1I9UNrSzbXs3fl+7kQGMr4FzYH1yQweyJJXztxFLyMlLon3N03Ph3tFu2vQp/QPnVm+tZvK2KjBQv152SGP3g9JRwks5lOJ24vYCTdN5xy7riA76nqktFJAtYIiJvAF8D3lLVn4rI7cDtwG0iMgaYA4wFBgBvisgoVfUDDwDXAR/iJJ2zgHk4CapKVUeIyBzgXuBSN7H9CJjmxrxEROa2Jbd41Dc7jYunDeLiaYNYvesA/1hZxkvLynh7nXMjWUaKl6lD8phemk9mahL9c9KYMjjPfswiZHN5HSt3HmBXdSMFmSkkez3sqGpg4ZZKlu+opqHl05bFU7weTh1dxKzRfRmcn8nkwblWHfkodO8/1/HA/E0HX/+/L47hK9MHk55in4Vg4VSZrgRu7u6CVXU3sNsdrhWRtUAJMBuY6U72GDAfuM0tf0ZVm4EtIrIRmC4iW4FsVf0AQEQeBy7ASTqzgR+7y/ob8Du3rbgzgTfaati5ye4sgmrgxbNxJTmMK8nh+2eOZt2eGtbvrWXptmo+2lLBfW+uRz+tzMS0IXl89wujOGF4gV1gPgKBgLJuTy1vfLyXt9btZeXOAx1Od2xxNhdPHcjYATmUFmaSnZ5EaUGmJZmj3Dvry3lg/iYunTaIs8b3JxBQZh7T17q270BUTiqLSCkwGfgI6OcmJFR1t4i03apfgnMk02anW9bqDrcvb5tnh7ssn4gcAAqCyzuYp9fweoSxA3IYOyCHL00eCDh3nbf4AmyrqOfDzZX85cNtfOWhj+ifncaA3DRSk7wMLcokxeshJz2ZPqlJNLT42VvbRE56MkPyM8hMTcLrEZK9HrLSkkj2ekj2CunJXgYXZHSrhex9NU2s3VNLerKXzFQvqUleUpM8ZKR4yc1IwSMQUA55Dk6OgYAi7coi6YVlO9lR2YjXIyzbXs3a3TX4AgGqG1pp9gUQgdH9s/l/XxzDySMLGZSXwZ4ap4ZZv+xUMlLsOow5VEOLj/9+aTWlBRncOXus/QHpQsS/QSLSB/g7cIuq1oT4celohIYoP9x5gmO7Due0HYMH947aJX1SkyAV8jNTmDw4j6tPLOWl5bt4f2MFFfXN1Df7+efqPbT6AtS1+A4eFeVnplDT2HrIPR8dSfF6mDQ4l6EFmfRJS6KuyUeSm5B8AaXVHyAt2UuLL8Dm/XUs3VZNYzc6sUv2CrkZKfRJTcIjsKu6kZe/c3KPtz/V2b59d/1+nl+2C4DSggymDskjNclDbkYyw4r6cNbY/p+pPTa0MLNHYzOHLx6/s798fT1bKxp46trjLeGEIaJJR0SScRLOk6r6vFu8V0SK3aOcYpwbT8E5GhkUNPtAoMwtH9hBefA8O0UkCcgBKt3yme3mmd8+PlV9EHgQYNq0aaF/jeNUWrKXS48bzKXHffYL2OIL0Ozzk5rkJSXJg88fYE9NE02tfnwBpcUXoK7JR4s/gM+v1LmtDC/dXsVb6/ZR3+wjKy2JVn+AZl8Ar0dI8XpoavWjwKh+WZw5th+XHDcIf0Cpb/bR7AvQ4gtQ3+yjurGVgIJXhIAqHhGafH6qG1qoa/bj8weYeUxfUpN6voJEZ/v2vksncfeXxqEKmVZ7rNeJt+9sqz/AXxfvcLs4KYx1OL1CqBYJfksHRwdtVPWmUAt2r608DKxV1fuCRs0FrgJ+6j6/FFT+lIjch1ORYCSwUFX9bi+lM3BOz10J/Lbdsj7AaRPubbdW22vA/4hIW3sgZwB3hIo3EaUkeUgJ+kFP8noYmBe6j58LJve6s5DdZqfITE9ZvLWKmiYf54zvH+tQeo1Q377FR7jsE4ErgFUistwt+y+cZPOciFyD07LBxQCqukZEngM+xqn5doNbcw3gW3xaZXqe+wAnqT3hVjqoxKn9hqpWuvcSLXKn+4k122OM6SmBgLJgfTkPvrOZFK+Hk0cWxTqkXiNUiwRHdF+Lqr5Hx9dWwGk8tKN57gHu6aB8MTCug/Im3KTVwbhHgEfCjdcYYzpTWd9ysCsIgF+/uZ77395IRoqX750xyk7VdkM43VUX4VRpHgMcvClEVWdFMC5jjIkLrf4As345n+qGVn503hi+PHUgj/57K6cf24/fXjbZ7sPppnCu4D4JrMVp4PNOnFYEFoWawRhjEsUne2qpbnBam7jzHx8z5SdvUNPk44ZTh1vCOQzhHBMWqOrDInKzqi4AFojIgkgHZowx8aDtRuE3v3sK8z8pZ39dC6cd29f6LTpMYXVX7T7vFpFzcaorDwwxvTHGJIxVu6rJSU9meFEf656gB4STdO4WkRzgezhVlbOBWyMalTHGxIEWX4CPtlQyYWCONTPVQ8Jpe+1ld/AAcGpkwzHGmNjzB5S7Xv6Yl1fuZn9dMzefNjLWISWMUDeHfl9Vf9bZTaJd3RxqjDG9Uas/wG1/X8nzS3cxa3Rfzp84gNmTEv+m6WgJdaSz1n0+0ptEjTGm1/jfV9fx/NJdfO8Lo/iOHeH0uFA3h/7DHWxQ1b8GjxORDm/INMaY3kpVeXH5Lh799xa+evxgSzgREs59Oh21WXbUtWNmjElsD727hVufXcEx/bP5/lmjYx1Owgp1Teds4BygRETuDxqVjdM2mjHG9Hr765o59/53Ka9t5vRj+/HgFVPxWOdrERPqmk4ZzvWc84ElQeW1WJVpY0yC+OfqPeytaeakEYX874XjLeFEWKhrOitEZDVwxpE2/mmMMfGmxe0jat7q3QwrzOSJa6bbvThREPI+HbcvmwIRSVHVlmgFZYwxkaSqfOkP77P7QBOV9S18Z9YISzhREk6LBNuA90VkLlDfVtiuYzZjjOk1/r2pgjVlNQB885RhfHvmiBhHdPQIJ+mUuQ8PYA0PGWN6NVXlT+9sJj8zhX/fPou0ZGspOprCaQbnzmgEYowx0fCPlbt5Z305PzjnWEs4MRBuJ27fB8ZinbgZY3qR+mYfGSneg9dryqob+X8vrmbioFyuPrE0tsEdpcLtxG0d1ombMaYXqWlqZeyPXuP+tzYCTpfTV//fInz+AL+5dBJJ3nB+/kxPC2erF6jqw0Crqi5Q1a8DMyIclzHGHJG1bkWBX725Hp8/wA1PLmVLRT1/vnIapYWZMY7u6GWduBljEtK6PbUHhy984N+s3HmAX148kc+NKIxhVMY6cTPGJJyXV5bxzKIdeAROGF7Apn313Pvl8Xx5qv1fjrVQba+lAdcDI4AS4GFVtU7cjDFxrb7Zx41PLQNgfEkOT37DrgbEk1DXdB4DpgGrgLOBX0Ylojg2c+ZMZs6cGVZ5OGXdWV53xnd3unD05LLiTaj3Fs4+64n9avu05yxYX35weNPKjus8dXcf9VR5uOMPd9qenDdSQp1eG6Oq4wFE5GFgYXRCMsaYw/fKyt3kZSSTsuQpkhvKsasB8SXUkU5bBQJU1boyMMbEvecW7eCVVbu5ZNog0g9sJam1vuuZTFSFSjoTRaTGfdQCE9qGRaQmWgEaY0w4nl20ndueX8nJIwv5jzOPiXU4phOhujaw9iGMMb3CEx9u4/+9uJpTRhXx4BVTSbYbP+OWqGqsY4gLIlKO06J2JBUC+yO8jnhdf3fWPURVi3pqxVHat4cj1p+H7jrSeON1v8bzfojX2ILj6tZ+taQTRSKyWFWnHY3rj/V7j0e9bZv0tnjDFc/vK15jO5K47BjUGGNM1FjSMcYYEzWWdKLrwaN4/bF+7/Got22T3hZvuOL5fcVrbIcdl13TMcYYEzV2pGOMMSZqLOkYY4yJGks6R0hEHhGRfSKyOqgsX0TeEJEN7nNe0Lg7RGSjiHwiImcGlU8VkVXuuPulrX/d0OseJCL/EpG1IrJGRG6O1vpFJE1EForICnfdd0bzvfcGIrLVfV/LRWSxW9bt7RPhGGP2+Y2mzr4r7aaZKSIH3P21XET+O0qxfeZz0m68uNt0o4isFJEpUYrrmKBtsVycFmluaTdN97eZqtrjCB7AKcAUYHVQ2c+A293h24F73eExwAogFaf7702A1x23EDgBEGAecHYY6y4GprjDWcB6dx0RX787XR93OBn4CKdH2ai8997wwOnavbBdWbe3T6J+fqO8Lzr8rrSbZibwcjx8TtqNP8fdpuJ+xz6KQYxeYA/OjaBHtM3sSOcIqeo7QGW74tk4XUPgPl8QVP6Mqjar6hZgIzBdRIqBbFX9QJ09+XjQPKHWvVtVl7rDtcBanL6PIr5+ddS5L5Pdh0brvfdi3do+kQ4mlp/faArxXekNZgOPu9+5D4Fcd5tH02nAJlU94hYgLOlERj9V3Q3Ohx3o65aXADuCptvplpW4w+3LwyYipcBknCOOqKxfRLwishzYB7yhqlFbdy+hwOsiskRErnPLurt9YiGh92G770p7J7injOeJyNgohdTR5yRYPHw25gBPdzKuW9ssnO6qTc/p6Dy3higPb6EifYC/A7eoak2I0+k9un5V9QOTRCQXeEFExoUKsyfX3UucqKplItIXeENE1oWYtjdsh16/D9t/V9qNXopz+qhORM4BXgRGRiGsz3xO3CPQg2F3ME/Utq+IpADnA3d0MLrb28yOdCJjb9vhr/u8zy3fCQwKmm4gUOaWD+ygvEsikozzJXpSVZ+P9voBVLUamA+cFe11xzNVLXOf9wEv4Jwu6+72iYWE3IedfFcOUtWatlPGqvoqkCwihZGOq5PPSbBYfzbOBpaq6t72Iw5nm1nSiYy5wFXu8FXAS0Hlc0QkVUSG4vwjWOiewqgVkRlurZ8rg+bplDvtw8BaVb0vmusXkSL3CAcRSQdOB9ZF673HOxHJFJGstmHgDGA13dw+0Y36oITbhyG+K8HT9G+rdSci03F+HysiHFdnn5Ngc4Er3VpsM4ADbac/o+QyOjm1dljbLNq1IBLt4e6M3Tg9re4ErgEKgLeADe5zftD0P8Cp9fMJQTV8gGk4H7ZNwO9wW4voYt0n4RxmrwSWu49zorF+YAKwzF33auC/3fKovPd4fwDDcGp6rQDWAD843O2TqJ/fKO+Pzr4r1wPXu9Pc6O6rFcCHwOdi+DkJjkuA37vbdhUwLYrbLQMnieQElR3RNrNmcIwxxkSNnV4zxhgTNZZ0jDHGRI0lHWOMMVFjSccYY0zUWNIxxhgTNZZ0EoSIlEpQS8EhpvlK0OtpInJ/5KNLPOFs7witt67rqQ6Z/sci8h8dlMckfmMs6RxdSoGDSUdVF6vqTbELx5jeRUTmi8g0d/jVthuke2jZ14vIlT21vHhlSSdK3H+W60TkMXH6xPibiGSIyGkiskyc/jQeEZFUd/qtInKvOH3WLBSREW75oyJyUdByP/PP113XuyKy1H18zh31U+Bkcfq9uFWcvjBedufJF5EX3dg+FJEJbvmP3bjmi8hmEbEk9SmviPxZnP5ZXheRdBGZ5G6/lSLygrh90bT7sSoUka3u8Fh3/y535xnpll8eVP4nEfG2rVRE7hGngcUPRaSfWzZERN5yl/GWiAxuH6w4fd6sEJEPgBuCyjuMwYSmqueo0wRUTy3vj6r6eE8tL15Z0omuY4AHVXUCUAN8F3gUuFRVx+M0wPqtoOlrVHU6zh3ev+7GevYBX1DVKcClQNsptNuBd1V1kqr+qt08dwLL3Nj+C6d5+jajgTNx2oT6kThtWBmnGZjfq+pYoBr4Ms52u83djquAH3WxjOuB36jqJJy7+neKyLE4++1Et9wPfNWdPhP4UFUnAu8A17rlv8Np/n4C8CSf7vNg/wfcpKondBVDWO++lwj6w/eQiKwWkSdF5HQReV+cjuqmu83RPCIii9w/gbPdedNF5Bk3GT8LpActd6u47Yy5f9iWuH9Arguapq6jPwmdxHnwVKj7J6XtT+d6ETnZLfeKyC/cP6krReQ7bnmoP6//IyIfiMhiEZkiIq+JyCYRuT5o3f/pvveV4nbIGCmWdKJrh6q+7w7/BaePii2qut4tewynU602Twc9t/+hCCUZ+LOIrAL+itP5VldOAp4AUNW3gQIRyXHHvaJOHyr7cRJap1+co8wWVV3uDi8BhgO5qrrALWu/PzvyAfBfInIbTmu9jTifi6nAInG6jjgNp7kUgBbg5aB1lrrDJwBPucNP4OzPg9x9GRzbE13EkGhGAL/Bab5pNM5p5pOA/8D5k/UD4G1VPQ44Ffi5OG2hfQtocJP5PTj7pSNfV9WpOEn7JhEpcMs7+5MQjiT3T+ctfPrn5TqcDvQmt/3BEJE0Qv953eH+0XjXne4inM7gfgIgImfg/IGaDkwCpopIV5/bw2ZJJ7q62+aQdjDsw91vIiJASgfz3QrsBSbifAk6mqa9UM2nNweV+bEuMdq03y6hzu8f3G9AWluhqj6F02x8I/CaiMzC2RePuUekk/5/e2cTp/CYJgAAAzZJREFUElUUBeDvQCCtBHEXWVSUiwgCIYIgoXDbIkNaiNmqoJSgRVALCaIWURAVRhQRLQJ3IYT9gEXYz0aYEd3V7FokiVgRJJ4W5wxzG8enLuZFdj64zJv7zvWe531vzrnnPu5R1R2qOuBNfmll76qssai+16RGXZYOa41PqlpU1QVsr7CX/n8sYoa7AzjnRn4UG6MWzGl4BKCqBWzvtlr0iUh5/7GNVLb3X8pJWAnlnbDTdgeBQVWdd52+YhGULOf1iX8Wsayjc6r6BfgptibV4WUcS1XQSh1TOoTRyZcWESnPWI4CL4DN4us1QDfwKpHvSj7f+nGJird1CJvVVNMIfPYHrBtLNQswh6XqrcVrPIQjIu3AtC7ONxJkMwvMlEMh/DmeJSrjlq7JbQE+quoN7MdhF7bJZqdYfpXyetumZfoewxJtgY3jm/Skrz3Misi+RCZLh7VG6iAsJN8XMMMtwOHE0Leo6pTLZDqL/rwcBPb6jGacimOxUichS+e0XS3nYckEWlV/J73u8vfytV9Orn2bqt5bhZ6rIoxOvkwBPSJSAJqA60AvMOShsAVgMJFvEJH3QD82ewG4C+wXkQ/AHuB7jX5uez/vgO2JTAGY9/jymao2A0Cb63aFytb2werowUIzBSxUcdHrrwInRWQMSPONdAET7mG3Yusyk8AFLJtkAXgOLJeeuA/odflu7J6pphe45S8SpCG0RTqs+GrXDiPAaY8eICK7vT51xnZS2yA3AjOq+kNEWrHQVb14BpwQkXWuUxOWUiTLeV2OEeC4WII7RGRD2eGpC3ltkf2/F2x6PLEK+RLQ/Lf1jhLlXy/Vzx6+rpGew14QuIOFoCaAYT+/HniMOWwPsRllm58rYQ5EA/DUZYaw8Fy7y3xL+u0EHmToOQCc9ePRpJ9moOTH64BrwCSWTuCU1x/AZlhF4D7QkOrox8eAm0l/6bl+b1vEoipb6zUekdogJ8Tysg+ralZK51S+hN1003VUKwiCIFfC6ARBEAS5EW8hBUEQ5IiInAeOVFUPqeqlv6FP3sRMJwiCIMiNeHstCIIgyI0wOkEQBEFuhNEJgiAIciOMThAEQZAbvwE3uPVfcTc4vwAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plot_partial_dependence(model, x_train, x_train.columns)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Shap" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "import shap" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Setting feature_perturbation = \"tree_path_dependent\" because no background data was given.\n" ] } ], "source": [ "expl = shap.TreeExplainer(model)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Passing 10216 background samples may lead to slow runtimes. Consider using shap.sample(data, 100) to create a smaller background data set.\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "shap.TreeExplainer(model, data=x_train)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "shap_val = expl.shap_values(x_val)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "shap.initjs()" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "
\n", " Visualization omitted, Javascript library not loaded!
\n", " Have you run `initjs()` in this notebook? If this notebook was from another\n", " user you must also trust this notebook (File -> Trust notebook). If you are viewing\n", " this notebook on github the Javascript has been stripped for security. If you are using\n", " JupyterLab this error is because a JupyterLab extension has not yet been written.\n", "
\n", " " ], "text/plain": [ "" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "shap.force_plot(expl.expected_value, shap_val[0, :], x_val.iloc[0, :])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise\n", "\n", "Check out `shap` further and see which plots you can generate." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Additional Resources" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- [shap](https://github.com/slundberg/shap)\n", "- [Scikit Yellowbrick](https://www.scikit-yb.org/en/latest/)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" } }, "nbformat": 4, "nbformat_minor": 4 }