Machine learning fairness#

Machine Learning fairness is an important part of modern day data modeling. Here we explore an introduction to make models more fair and equitable.

Machine learning is a powerful tool that has revolutionized many industries by enabling computers to learn from data and make predictions or decisions.

However, as machine learning algorithms become increasingly ubiquitous in our daily lives, concerns about fairness and equity have emerged. Machine learning fairness refers to the idea that machine learning models should not perpetuate or exacerbate existing biases or discrimination. Fairness means that the model treats all individuals or groups fairly, regardless of race, gender, ethnicity, or other protected characteristics.

This notebook will provide an overview of the key concepts and challenges in machine learning fairness, as well as some techniques commonly used to address them.

How To#

from sklearn.model_selection import train_test_split
import pandas as pd

df = pd.read_csv("data/housing.csv").dropna()
df.head()
longitude latitude housing_median_age total_rooms total_bedrooms population households median_income median_house_value ocean_proximity
0 -122.23 37.88 41.0 880.0 129.0 322.0 126.0 8.3252 452600.0 NEAR BAY
1 -122.22 37.86 21.0 7099.0 1106.0 2401.0 1138.0 8.3014 358500.0 NEAR BAY
2 -122.24 37.85 52.0 1467.0 190.0 496.0 177.0 7.2574 352100.0 NEAR BAY
3 -122.25 37.85 52.0 1274.0 235.0 558.0 219.0 5.6431 341300.0 NEAR BAY
4 -122.25 37.85 52.0 1627.0 280.0 565.0 259.0 3.8462 342200.0 NEAR BAY
x_train, x_, y_train, y_ = train_test_split(df.drop(["longitude","latitude", "ocean_proximity", "median_house_value"], axis=1), 
                                                    df.median_house_value, test_size=.5, stratify=df.ocean_proximity)

x_val, x_test, y_val, y_test = train_test_split(x_, y_, test_size=.5)
from sklearn.ensemble import RandomForestRegressor
model = RandomForestRegressor().fit(x_train, y_train)
model.score(x_val, y_val)
0.6480775961527556
from sklearn.model_selection import cross_val_score
for cls in df.ocean_proximity.unique():
    print(cls)
    try:
        idx = df[df.ocean_proximity.isin([cls])].index

        idx_val = x_val.index.intersection(idx)
        print(model.score(x_val.loc[idx_val, :], y_val.loc[idx_val]))

        val = cross_val_score(model, x_val.loc[idx_val, :], y_val.loc[idx_val])
        print(val)
        print(val.mean(), " +- ", val.std(), "\n")
    except:
        print("Error in Validation")
    try:
        idx = df[df.ocean_proximity.isin([cls])].index

        idx_test = x_test.index.intersection(idx)
        print(model.score(x_test.loc[idx_test, :], y_test.loc[idx_test]))
        
        tst = cross_val_score(model,x_test.loc[idx_test, :], y_test.loc[idx_test])
        print(tst)
        print(tst.mean(), " +- ", tst.std(), "\n")
    except:
        print("Error in Test")
NEAR BAY
0.6544350853100038
[0.59873178 0.48977578 0.63153377 0.33855197 0.54630705]
0.5209800699841305  +-  0.10311413352044327 

0.5964387549003616
[0.48752953 0.57120308 0.52813922 0.50759256 0.59513534]
0.5379199479087139  +-  0.03983961584562086 

<1H OCEAN
0.6128366904750742
[0.66333023 0.60104977 0.58508043 0.62095972 0.69757825]
0.6335996793522752  +-  0.04135346569453807 

0.6083495123673812
[0.60005696 0.55730935 0.61306966 0.68535089 0.5584024 ]
0.6028378533940246  +-  0.04683626721859449 

INLAND
0.21061486650006345
[0.55358831 0.51266803 0.47884739 0.50378551 0.37720208]
0.48521826403548945  +-  0.0591283005216573 

0.1950366748400284
[0.55335927 0.50442003 0.57013208 0.51651743 0.33123904]
0.49513356908616124  +-  0.08534627276300144 

NEAR OCEAN
0.548119998691016
[0.65802852 0.544108   0.61553981 0.51796676 0.49629081]
0.5663867785808864  +-  0.060940162136261945 

0.6021430352150736
[0.63861788 0.65769445 0.57335902 0.59857214 0.55172915]
0.6039945277970832  +-  0.03945139615013755 

ISLAND
-84.90514148063541
Error in Validation
nan
Error in Test

Calculate Residuals#

from yellowbrick.regressor import residuals_plot, prediction_error
residuals_plot(model, x_train, y_train, x_test, y_test)
../_images/32e7beee6bb4ea33023d26d9977a52e4f51b514ae37fa432bea19db34277222e.png
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/yellowbrick/utils/wrapper.py:48, in Wrapper.__getattr__(self, attr)
     47 try:
---> 48     return getattr(self._wrapped, attr)
     49 except AttributeError as e:

AttributeError: 'RandomForestRegressor' object has no attribute 'line_color'

The above exception was the direct cause of the following exception:

YellowbrickAttributeError                 Traceback (most recent call last)
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/IPython/core/formatters.py:974, in MimeBundleFormatter.__call__(self, obj, include, exclude)
    971     method = get_real_method(obj, self.print_method)
    973     if method is not None:
--> 974         return method(include=include, exclude=exclude)
    975     return None
    976 else:

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/base.py:669, in BaseEstimator._repr_mimebundle_(self, **kwargs)
    667 def _repr_mimebundle_(self, **kwargs):
    668     """Mime bundle used by jupyter kernels to display estimator"""
--> 669     output = {"text/plain": repr(self)}
    670     if get_config()["display"] == "diagram":
    671         output["text/html"] = estimator_html_repr(self)

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/base.py:287, in BaseEstimator.__repr__(self, N_CHAR_MAX)
    279 # use ellipsis for sequences with a lot of elements
    280 pp = _EstimatorPrettyPrinter(
    281     compact=True,
    282     indent=1,
    283     indent_at_name=True,
    284     n_max_elements_to_show=N_MAX_ELEMENTS_TO_SHOW,
    285 )
--> 287 repr_ = pp.pformat(self)
    289 # Use bruteforce ellipsis when there are a lot of non-blank characters
    290 n_nonblank = len("".join(repr_.split()))

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/pprint.py:153, in PrettyPrinter.pformat(self, object)
    151 def pformat(self, object):
    152     sio = _StringIO()
--> 153     self._format(object, sio, 0, 0, {}, 0)
    154     return sio.getvalue()

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/pprint.py:170, in PrettyPrinter._format(self, object, stream, indent, allowance, context, level)
    168     self._readable = False
    169     return
--> 170 rep = self._repr(object, context, level)
    171 max_width = self._width - indent - allowance
    172 if len(rep) > max_width:

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/pprint.py:404, in PrettyPrinter._repr(self, object, context, level)
    403 def _repr(self, object, context, level):
--> 404     repr, readable, recursive = self.format(object, context.copy(),
    405                                             self._depth, level)
    406     if not readable:
    407         self._readable = False

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/utils/_pprint.py:189, in _EstimatorPrettyPrinter.format(self, object, context, maxlevels, level)
    188 def format(self, object, context, maxlevels, level):
--> 189     return _safe_repr(
    190         object, context, maxlevels, level, changed_only=self._changed_only
    191     )

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/utils/_pprint.py:440, in _safe_repr(object, context, maxlevels, level, changed_only)
    438 recursive = False
    439 if changed_only:
--> 440     params = _changed_params(object)
    441 else:
    442     params = object.get_params(deep=False)

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/utils/_pprint.py:93, in _changed_params(estimator)
     89 def _changed_params(estimator):
     90     """Return dict (param_name: value) of parameters that were given to
     91     estimator with non-default values."""
---> 93     params = estimator.get_params(deep=False)
     94     init_func = getattr(estimator.__init__, "deprecated_original", estimator.__init__)
     95     init_params = inspect.signature(init_func).parameters

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/yellowbrick/base.py:342, in ModelVisualizer.get_params(self, deep)
    334 def get_params(self, deep=True):
    335     """
    336     After v0.24 - scikit-learn is able to determine that ``self.estimator`` is
    337     nested and fetches its params using ``estimator__param``. This functionality is
   (...)
    340     the estimator params.
    341     """
--> 342     params = super(ModelVisualizer, self).get_params(deep=deep)
    343     for param in list(params.keys()):
    344         if param.startswith("estimator__"):

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/base.py:195, in BaseEstimator.get_params(self, deep)
    193 out = dict()
    194 for key in self._get_param_names():
--> 195     value = getattr(self, key)
    196     if deep and hasattr(value, "get_params") and not isinstance(value, type):
    197         deep_items = value.get_params().items()

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/yellowbrick/utils/wrapper.py:50, in Wrapper.__getattr__(self, attr)
     48     return getattr(self._wrapped, attr)
     49 except AttributeError as e:
---> 50     raise YellowbrickAttributeError(f"neither visualizer '{self.__class__.__name__}' nor wrapped estimator '{type(self._wrapped).__name__}' have attribute '{attr}'") from e

YellowbrickAttributeError: neither visualizer 'ResidualsPlot' nor wrapped estimator 'RandomForestRegressor' have attribute 'line_color'
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/yellowbrick/utils/wrapper.py:48, in Wrapper.__getattr__(self, attr)
     47 try:
---> 48     return getattr(self._wrapped, attr)
     49 except AttributeError as e:

AttributeError: 'RandomForestRegressor' object has no attribute 'line_color'

The above exception was the direct cause of the following exception:

YellowbrickAttributeError                 Traceback (most recent call last)
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/IPython/core/formatters.py:708, in PlainTextFormatter.__call__(self, obj)
    701 stream = StringIO()
    702 printer = pretty.RepresentationPrinter(stream, self.verbose,
    703     self.max_width, self.newline,
    704     max_seq_length=self.max_seq_length,
    705     singleton_pprinters=self.singleton_printers,
    706     type_pprinters=self.type_printers,
    707     deferred_pprinters=self.deferred_printers)
--> 708 printer.pretty(obj)
    709 printer.flush()
    710 return stream.getvalue()

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/IPython/lib/pretty.py:410, in RepresentationPrinter.pretty(self, obj)
    407                         return meth(obj, self, cycle)
    408                 if cls is not object \
    409                         and callable(cls.__dict__.get('__repr__')):
--> 410                     return _repr_pprint(obj, self, cycle)
    412     return _default_pprint(obj, self, cycle)
    413 finally:

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/IPython/lib/pretty.py:778, in _repr_pprint(obj, p, cycle)
    776 """A pprint that just redirects to the normal repr function."""
    777 # Find newlines and replace them with p.break_()
--> 778 output = repr(obj)
    779 lines = output.splitlines()
    780 with p.group():

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/base.py:287, in BaseEstimator.__repr__(self, N_CHAR_MAX)
    279 # use ellipsis for sequences with a lot of elements
    280 pp = _EstimatorPrettyPrinter(
    281     compact=True,
    282     indent=1,
    283     indent_at_name=True,
    284     n_max_elements_to_show=N_MAX_ELEMENTS_TO_SHOW,
    285 )
--> 287 repr_ = pp.pformat(self)
    289 # Use bruteforce ellipsis when there are a lot of non-blank characters
    290 n_nonblank = len("".join(repr_.split()))

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/pprint.py:153, in PrettyPrinter.pformat(self, object)
    151 def pformat(self, object):
    152     sio = _StringIO()
--> 153     self._format(object, sio, 0, 0, {}, 0)
    154     return sio.getvalue()

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/pprint.py:170, in PrettyPrinter._format(self, object, stream, indent, allowance, context, level)
    168     self._readable = False
    169     return
--> 170 rep = self._repr(object, context, level)
    171 max_width = self._width - indent - allowance
    172 if len(rep) > max_width:

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/pprint.py:404, in PrettyPrinter._repr(self, object, context, level)
    403 def _repr(self, object, context, level):
--> 404     repr, readable, recursive = self.format(object, context.copy(),
    405                                             self._depth, level)
    406     if not readable:
    407         self._readable = False

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/utils/_pprint.py:189, in _EstimatorPrettyPrinter.format(self, object, context, maxlevels, level)
    188 def format(self, object, context, maxlevels, level):
--> 189     return _safe_repr(
    190         object, context, maxlevels, level, changed_only=self._changed_only
    191     )

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/utils/_pprint.py:440, in _safe_repr(object, context, maxlevels, level, changed_only)
    438 recursive = False
    439 if changed_only:
--> 440     params = _changed_params(object)
    441 else:
    442     params = object.get_params(deep=False)

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/utils/_pprint.py:93, in _changed_params(estimator)
     89 def _changed_params(estimator):
     90     """Return dict (param_name: value) of parameters that were given to
     91     estimator with non-default values."""
---> 93     params = estimator.get_params(deep=False)
     94     init_func = getattr(estimator.__init__, "deprecated_original", estimator.__init__)
     95     init_params = inspect.signature(init_func).parameters

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/yellowbrick/base.py:342, in ModelVisualizer.get_params(self, deep)
    334 def get_params(self, deep=True):
    335     """
    336     After v0.24 - scikit-learn is able to determine that ``self.estimator`` is
    337     nested and fetches its params using ``estimator__param``. This functionality is
   (...)
    340     the estimator params.
    341     """
--> 342     params = super(ModelVisualizer, self).get_params(deep=deep)
    343     for param in list(params.keys()):
    344         if param.startswith("estimator__"):

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/base.py:195, in BaseEstimator.get_params(self, deep)
    193 out = dict()
    194 for key in self._get_param_names():
--> 195     value = getattr(self, key)
    196     if deep and hasattr(value, "get_params") and not isinstance(value, type):
    197         deep_items = value.get_params().items()

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/yellowbrick/utils/wrapper.py:50, in Wrapper.__getattr__(self, attr)
     48     return getattr(self._wrapped, attr)
     49 except AttributeError as e:
---> 50     raise YellowbrickAttributeError(f"neither visualizer '{self.__class__.__name__}' nor wrapped estimator '{type(self._wrapped).__name__}' have attribute '{attr}'") from e

YellowbrickAttributeError: neither visualizer 'ResidualsPlot' nor wrapped estimator 'RandomForestRegressor' have attribute 'line_color'
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/yellowbrick/utils/wrapper.py:48, in Wrapper.__getattr__(self, attr)
     47 try:
---> 48     return getattr(self._wrapped, attr)
     49 except AttributeError as e:

AttributeError: 'RandomForestRegressor' object has no attribute 'line_color'

The above exception was the direct cause of the following exception:

YellowbrickAttributeError                 Traceback (most recent call last)
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/IPython/core/formatters.py:344, in BaseFormatter.__call__(self, obj)
    342     method = get_real_method(obj, self.print_method)
    343     if method is not None:
--> 344         return method()
    345     return None
    346 else:

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/base.py:665, in BaseEstimator._repr_html_inner(self)
    660 def _repr_html_inner(self):
    661     """This function is returned by the @property `_repr_html_` to make
    662     `hasattr(estimator, "_repr_html_") return `True` or `False` depending
    663     on `get_config()["display"]`.
    664     """
--> 665     return estimator_html_repr(self)

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/utils/_estimator_html_repr.py:388, in estimator_html_repr(estimator)
    386 style_template = Template(_STYLE)
    387 style_with_id = style_template.substitute(id=container_id)
--> 388 estimator_str = str(estimator)
    390 # The fallback message is shown by default and loading the CSS sets
    391 # div.sk-text-repr-fallback to display: none to hide the fallback message.
    392 #
   (...)
    397 # The reverse logic applies to HTML repr div.sk-container.
    398 # div.sk-container is hidden by default and the loading the CSS displays it.
    399 fallback_msg = (
    400     "In a Jupyter environment, please rerun this cell to show the HTML"
    401     " representation or trust the notebook. <br />On GitHub, the"
    402     " HTML representation is unable to render, please try loading this page"
    403     " with nbviewer.org."
    404 )

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/base.py:287, in BaseEstimator.__repr__(self, N_CHAR_MAX)
    279 # use ellipsis for sequences with a lot of elements
    280 pp = _EstimatorPrettyPrinter(
    281     compact=True,
    282     indent=1,
    283     indent_at_name=True,
    284     n_max_elements_to_show=N_MAX_ELEMENTS_TO_SHOW,
    285 )
--> 287 repr_ = pp.pformat(self)
    289 # Use bruteforce ellipsis when there are a lot of non-blank characters
    290 n_nonblank = len("".join(repr_.split()))

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/pprint.py:153, in PrettyPrinter.pformat(self, object)
    151 def pformat(self, object):
    152     sio = _StringIO()
--> 153     self._format(object, sio, 0, 0, {}, 0)
    154     return sio.getvalue()

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/pprint.py:170, in PrettyPrinter._format(self, object, stream, indent, allowance, context, level)
    168     self._readable = False
    169     return
--> 170 rep = self._repr(object, context, level)
    171 max_width = self._width - indent - allowance
    172 if len(rep) > max_width:

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/pprint.py:404, in PrettyPrinter._repr(self, object, context, level)
    403 def _repr(self, object, context, level):
--> 404     repr, readable, recursive = self.format(object, context.copy(),
    405                                             self._depth, level)
    406     if not readable:
    407         self._readable = False

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/utils/_pprint.py:189, in _EstimatorPrettyPrinter.format(self, object, context, maxlevels, level)
    188 def format(self, object, context, maxlevels, level):
--> 189     return _safe_repr(
    190         object, context, maxlevels, level, changed_only=self._changed_only
    191     )

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/utils/_pprint.py:440, in _safe_repr(object, context, maxlevels, level, changed_only)
    438 recursive = False
    439 if changed_only:
--> 440     params = _changed_params(object)
    441 else:
    442     params = object.get_params(deep=False)

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/utils/_pprint.py:93, in _changed_params(estimator)
     89 def _changed_params(estimator):
     90     """Return dict (param_name: value) of parameters that were given to
     91     estimator with non-default values."""
---> 93     params = estimator.get_params(deep=False)
     94     init_func = getattr(estimator.__init__, "deprecated_original", estimator.__init__)
     95     init_params = inspect.signature(init_func).parameters

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/yellowbrick/base.py:342, in ModelVisualizer.get_params(self, deep)
    334 def get_params(self, deep=True):
    335     """
    336     After v0.24 - scikit-learn is able to determine that ``self.estimator`` is
    337     nested and fetches its params using ``estimator__param``. This functionality is
   (...)
    340     the estimator params.
    341     """
--> 342     params = super(ModelVisualizer, self).get_params(deep=deep)
    343     for param in list(params.keys()):
    344         if param.startswith("estimator__"):

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/base.py:195, in BaseEstimator.get_params(self, deep)
    193 out = dict()
    194 for key in self._get_param_names():
--> 195     value = getattr(self, key)
    196     if deep and hasattr(value, "get_params") and not isinstance(value, type):
    197         deep_items = value.get_params().items()

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/yellowbrick/utils/wrapper.py:50, in Wrapper.__getattr__(self, attr)
     48     return getattr(self._wrapped, attr)
     49 except AttributeError as e:
---> 50     raise YellowbrickAttributeError(f"neither visualizer '{self.__class__.__name__}' nor wrapped estimator '{type(self._wrapped).__name__}' have attribute '{attr}'") from e

YellowbrickAttributeError: neither visualizer 'ResidualsPlot' nor wrapped estimator 'RandomForestRegressor' have attribute 'line_color'
prediction_error(model, x_train, y_train, x_test, y_test)
../_images/88825963e874f2a28be260a11afbfa6b6d3f78d57ebd371538c3981d91d9d4cd.png
PredictionError(ax=<Axes: title={'center': 'Prediction Error for RandomForestRegressor'}, xlabel='$y$', ylabel='$\\hat{y}$'>,
                estimator=RandomForestRegressor())
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Confusion Matrix for Classifiers#

from sklearn.metrics import plot_confusion_matrix
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[10], line 1
----> 1 from sklearn.metrics import plot_confusion_matrix

ImportError: cannot import name 'plot_confusion_matrix' from 'sklearn.metrics' (/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/metrics/__init__.py)
from sklearn.ensemble import RandomForestClassifier

x_train, x_, y_train, y_ = train_test_split(df.drop(["longitude","latitude", "ocean_proximity"], axis=1), 
                                                    df.ocean_proximity, test_size=.5, stratify=df.ocean_proximity)

x_val, x_test, y_val, y_test = train_test_split(x_, y_, test_size=.5)

model = RandomForestClassifier().fit(x_train, y_train)
plot_confusion_matrix(model, x_test, y_test)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[12], line 1
----> 1 plot_confusion_matrix(model, x_test, y_test)

NameError: name 'plot_confusion_matrix' is not defined
plot_confusion_matrix(model, x_test, y_test, normalize="all")
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[13], line 1
----> 1 plot_confusion_matrix(model, x_test, y_test, normalize="all")

NameError: name 'plot_confusion_matrix' is not defined

Other Visualizations that are important#

from yellowbrick.classifier import confusion_matrix, classification_report, precision_recall_curve, roc_auc
confusion_matrix(model, x_train, y_train, x_test, y_test)
../_images/6694ee1cea9eb4e5158041b8e7c38edfcce272f15396a42f2b2c6b0d16f50a2d.png
ConfusionMatrix(ax=<Axes: title={'center': 'RandomForestClassifier Confusion Matrix'}, xlabel='Predicted Class', ylabel='True Class'>,
                cmap=<matplotlib.colors.ListedColormap object at 0x7f8ebbc326a0>,
                estimator=RandomForestClassifier())
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
classification_report(model, x_train, y_train, x_test, y_test)
../_images/74b6782a9da172ab461a5455815f72d712cfc8c416a0dad14f5f585e82c927d7.png
ClassificationReport(ax=<Axes: title={'center': 'RandomForestClassifier Classification Report'}>,
                     cmap=<matplotlib.colors.ListedColormap object at 0x7f8ebbba6550>,
                     estimator=RandomForestClassifier())
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
from sklearn.metrics import classification_report
print(classification_report(y_test, model.predict(x_test)))
              precision    recall  f1-score   support

   <1H OCEAN       0.65      0.86      0.74      2253
      INLAND       0.77      0.80      0.79      1599
      ISLAND       0.00      0.00      0.00         2
    NEAR BAY       0.52      0.26      0.34       609
  NEAR OCEAN       0.36      0.09      0.15       646

    accuracy                           0.67      5109
   macro avg       0.46      0.40      0.40      5109
weighted avg       0.63      0.67      0.63      5109
precision_recall_curve(model, x_train, y_train, x_test, y_test)
../_images/7bd3c5f7bc82433afb91f3a5d629ba0abecf2374fe5a5f6100a747ec4654c357.png
PrecisionRecallCurve(ax=<Axes: title={'center': 'Precision-Recall Curve for RandomForestClassifier'}, xlabel='Recall', ylabel='Precision'>,
                     estimator=OneVsRestClassifier(estimator=RandomForestClassifier()),
                     iso_f1_values={0.2, 0.4, 0.6, 0.8})
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
roc_auc(model, x_train, y_train, x_test, y_test)
../_images/9570e89324081ff11c16a494053b5c92de541ef2cdeda5b8c6e2478fa6646194.png
ROCAUC(ax=<Axes: title={'center': 'ROC Curves for RandomForestClassifier'}, xlabel='False Positive Rate', ylabel='True Positive Rate'>,
       estimator=RandomForestClassifier())
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Exercise#

Modify the code to generate dummy models for each class.

Additional Resources#