Finding and understanding relationships in data#

In today’s data-driven world, we are constantly bombarded with vast amounts of information. However, this raw data is often meaningless without the ability to identify relationships and patterns within it.

Finding and understanding relationships in data is crucial for making informed decisions, developing predictive models, and discovering new insights. Through the use of statistical techniques and machine learning algorithms, we can uncover hidden connections and dependencies between variables, enabling us to make accurate predictions and improve our understanding of complex systems.

Whether in business, science, or everyday life, the ability to analyze and interpret data is becoming increasingly important, and finding relationships within it is an essential skill for success.

How To#

import pandas as pd
df = pd.read_csv("data/housing.csv")
df.head()
longitude latitude housing_median_age total_rooms total_bedrooms population households median_income median_house_value ocean_proximity
0 -122.23 37.88 41.0 880.0 129.0 322.0 126.0 8.3252 452600.0 NEAR BAY
1 -122.22 37.86 21.0 7099.0 1106.0 2401.0 1138.0 8.3014 358500.0 NEAR BAY
2 -122.24 37.85 52.0 1467.0 190.0 496.0 177.0 7.2574 352100.0 NEAR BAY
3 -122.25 37.85 52.0 1274.0 235.0 558.0 219.0 5.6431 341300.0 NEAR BAY
4 -122.25 37.85 52.0 1627.0 280.0 565.0 259.0 3.8462 342200.0 NEAR BAY
df.corr()
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 1
----> 1 df.corr()

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/frame.py:10054, in DataFrame.corr(self, method, min_periods, numeric_only)
  10052 cols = data.columns
  10053 idx = cols.copy()
> 10054 mat = data.to_numpy(dtype=float, na_value=np.nan, copy=False)
  10056 if method == "pearson":
  10057     correl = libalgos.nancorr(mat, minp=min_periods)

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/frame.py:1838, in DataFrame.to_numpy(self, dtype, copy, na_value)
   1836 if dtype is not None:
   1837     dtype = np.dtype(dtype)
-> 1838 result = self._mgr.as_array(dtype=dtype, copy=copy, na_value=na_value)
   1839 if result.dtype is not dtype:
   1840     result = np.array(result, dtype=dtype, copy=False)

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/internals/managers.py:1732, in BlockManager.as_array(self, dtype, copy, na_value)
   1730         arr.flags.writeable = False
   1731 else:
-> 1732     arr = self._interleave(dtype=dtype, na_value=na_value)
   1733     # The underlying data was copied within _interleave, so no need
   1734     # to further copy if copy=True or setting na_value
   1736 if na_value is not lib.no_default:

File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/internals/managers.py:1794, in BlockManager._interleave(self, dtype, na_value)
   1792     else:
   1793         arr = blk.get_values(dtype)
-> 1794     result[rl.indexer] = arr
   1795     itemmask[rl.indexer] = 1
   1797 if not itemmask.all():

ValueError: could not convert string to float: 'NEAR BAY'
df.total_rooms.corr(df.households)
0.9184844926543082

More than linear correlation#

from discover_feature_relationships import discover
rel = discover.discover(df.sample(500))
beyond_corr = rel.pivot(index="target", columns="feature", values="score").fillna(1)
beyond_corr
feature households housing_median_age latitude longitude median_house_value median_income ocean_proximity population total_bedrooms total_rooms
target
households 1.000000 -0.079516 -0.271592 -0.364739 -0.586356 -0.422810 -0.010620 0.648563 0.961367 0.770929
housing_median_age -0.374121 1.000000 0.028650 -0.074090 -0.537152 -0.574565 0.108621 -0.386393 -0.335156 -0.426881
latitude -0.315587 -0.086278 1.000000 0.887432 -0.476145 -0.539417 0.324746 -0.457695 -0.527664 -0.397087
longitude -0.308965 -0.056548 0.897165 1.000000 -0.486003 -0.505310 0.253744 -0.536910 -0.451027 -0.370603
median_house_value -0.423181 -0.145256 0.051293 -0.037313 1.000000 0.334798 0.223184 -0.662422 -0.454812 -0.330681
median_income -0.459522 -0.225025 -0.190491 -0.183605 0.331014 1.000000 0.034568 -0.726395 -0.454748 -0.322960
ocean_proximity -0.398800 -0.092210 0.185398 0.335580 -0.454741 -0.506561 1.000000 -0.522781 -0.513087 -0.545224
population 0.698451 -0.060482 -0.361494 -0.259482 -0.596401 -0.521879 0.001988 1.000000 0.642862 0.587984
total_bedrooms 0.967683 -0.072555 -0.280985 -0.328171 -0.606435 -0.444023 -0.012289 0.616603 1.000000 0.765382
total_rooms 0.746730 -0.009843 -0.398969 -0.431537 -0.603426 -0.415243 -0.014568 0.606820 0.715104 1.000000
import seaborn as sns
sns.heatmap(beyond_corr, vmin=-1, vmax=1)
<Axes: xlabel='feature', ylabel='target'>
../_images/8a43ed946acecd6e5c0b842f10e1276f5ca47261ae48611ba3d315dace7e1dba.png

Exercise#

Additional Resources#