data-exploratory-analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(data-exploratory-analysis)=\n",
    "# Exploratory Data Analysis\n",
    "\n",
    "Exploratory data analysis (EDA) involves taking a first look at a dataset and summarising its salient characteristics using tables and graphics. It is (or should be) the stage before testing hypotheses and can be useful in informing hypotheses. In this chapter, we'll look at a few options for EDA using code.\n",
    "\n",
    "To show how to do EDA using code, we will need a dataset to explore. We'll use the Grinell housing dataset, which covers houses sold between 2005 and 2015 in Grinnell, Iowa. It was assembled by a local estate agent, Matt Karjalahti.\n",
    "\n",
    "First, let's import a couple of essential packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import warnings\n",
    "from rich import print"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51a55374",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "import matplotlib_inline.backend_inline\n",
    "\n",
    "# Plot settings\n",
    "plt.style.use(\n",
    "    \"https://github.com/aeturrell/coding-for-economists/raw/main/plot_style.txt\"\n",
    ")\n",
    "matplotlib_inline.backend_inline.set_matplotlib_formats(\"svg\")\n",
    "\n",
    "# Set max rows displayed for readability\n",
    "pd.set_option(\"display.max_rows\", 6)\n",
    "\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note}\n",
    "You may not have the **rich** package installed already. It provides pretty printing of text to screen (and much more besides). You can install it by running `pip install skimpy` or `conda install rich` on the command line.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preliminaries: extracting, typing, and initial cleaning\n",
    "\n",
    "Let's grab some data to do exploratory data analysis on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\n",
    "    \"https://vincentarelbundock.github.io/Rdatasets/csv/Stat2Data/GrinnellHouses.csv\",\n",
    "    index_col=0,\n",
    ")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some things we should do even before an exploratory data analysis; we should make sure we've cast the columns to the right types and converted any that need converting. Even a quick look at the data provided by `.head()` shows that the `Date` column is in a weird format. It also *seems* like the number of `Baths` column has some data issues because it is a floating point number rather than an integer. However, this a North American quirk: realtors refer to bathrooms with only a toilet and sink as a \"half bath\" (and sometimes those with a shower but no bathtub as a \"three-quarter bath,\" which also shows up in the data). So this is okay, and we mainly need concern ourselves with the Date column.\n",
    "\n",
    "Fortunately, there is guidance attached to the dataset as to what the Date column really means: it's just a counter of days that encodes the date of sale that begins with 1st Jan 2005=16436. The documents also tell us that the final year is 2015. We'd really like to have a more usable datetime column than this so let's sort this out first. Ideally, we'd like to *map* the `Date` column into a new `datetime` column that turns 16436 into 1st Jan 2005, 16437 into 2nd Jan 2005, and so on, until we have a mapping for every date between the first date in the dataset and the last. \n",
    "\n",
    "Our solution will be to create a function that maps the code for Date into a genuine datetime using the `pd.DateOffset` function and apply it to the `Date` column using `apply()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_date_code_to_datetime(date):\n",
    "    start_code = 16436\n",
    "    return pd.to_datetime(\"01-01-2005\") + pd.DateOffset(days=date - start_code)\n",
    "\n",
    "\n",
    "df[\"datetime\"] = df[\"Date\"].apply(convert_date_code_to_datetime)\n",
    "# Check the 'biggest' rows by datetime\n",
    "df.nlargest(5, columns=[\"datetime\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An extra column containing datetime has been added and, looking at the max rows, we can see that it does indeed run all the way to 2015 as expected from the documentation of the dataset.\n",
    "\n",
    "Okay, now we want to sort out the data type issues we saw earlier. But let's just check they're as bad as we think using `df.info()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**pandas** read most of them in sensibly, but not *all* of them, so let's adjust the ones we need to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.assign(\n",
    "    Address=df[\"Address\"].astype(\"string\"),\n",
    "    Bedrooms=df[\"Bedrooms\"].astype(\"category\"),\n",
    "    OrigPrice=df[\"OrigPrice\"].astype(float),\n",
    "    ListPrice=df[\"ListPrice\"].astype(float),\n",
    "    SalePrice=df[\"SalePrice\"].astype(float),\n",
    ")\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having to do some initial variable type cleaning is a *normal* and *unavoidable* part of data analysis, especially when reading in from a format like CSV (which does not preserve data type but has great interoperability across systems). It's important and what we've seen here is a typical pattern.\n",
    "\n",
    "There's just one more step before starting the EDA proper. If you haven't reached the Chapter on best practice, which talks about naming conventions, you won't know this yet but the usual naming convention for variables and columns in Python is so-called snake case (appropriately enough). An example of a word *not* in snake case would be 'ThisIsAColName'; actually that's in Camel Case. Rendered in snake case, it would be 'this_is_a_col_name', which is longer but slightly easier to read. So we should really change the column names to be in snake case. Fortunately, the [**skimpy**](https://aeturrell.github.io/skimpy/) package (drawing on functions from a package called **dataprep**) has us covered here; it provides methods to translate between pretty much every naming convention that exists. Let's change our columns to snake case, which is the default setting of `clean_columns()`.\n",
    "\n",
    "```{note}\n",
    "**skimpy** is likely to be a new package to you, and one that you haven't installed. You can install it by running `pip install skimpy` on the command line.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from skimpy import clean_columns\n",
    "\n",
    "df = clean_columns(df, case=\"snake\")\n",
    "print(df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can start the exploratory data analysis!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EDA using **pandas** built-in methods\n",
    "\n",
    "**pandas** has some great options for built-in EDA; in fact we've already seen one of them, `df.info()` which, as well as reporting datatypes and memory usage, also tells us how many observations in each column are 'truthy' rather than 'falsy', ie how many have non-null values.\n",
    "\n",
    "### Exploratory tables and descriptive statistics\n",
    "\n",
    "A small step beyond `df.info()` to get tables is to use `df.describe()` which, if you have mixed datatypes that include floats, will report some basic summary statistics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although helpful, that sure is hard to read! We can improve this by using the `round()` method too:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sum_table = df.describe().round(1)\n",
    "sum_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Published summary statistics tables often list one variable per row, and if your dataframe has many variables, `describe()` can quickly get too wide to read easily. You can transpose it using the `T` property (or the `transpose()` method):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sum_table = sum_table.T\n",
    "sum_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, although this is very basic, let's say you wanted to include it as a table of summary statistics in a paper. This is just a dataframe so you can export it just like you can *any* pandas dataframe. This includes options like `.to_excel()`. For inclusion in a paper or report, you're most likely to want to export it as one of a few common types, all of which can be achieved using the `to_formatname()` syntax (for example `to_string()` for plain text). Just to demonstrate these different output types, let's print them to screen (even though in practice we'd be writing them to a file). First, as text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(sum_table.to_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as Markdown:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(sum_table.to_markdown())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as HTML:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(sum_table.head(3).to_html())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or as latex:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(sum_table.style.to_latex(caption=\"Summary stats from EDA\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And remember, with all of these, you can write them to file using a variation on the following structure:\n",
    "\n",
    "```python\n",
    "\n",
    "open('sum_stats_table.tex', 'w').write(sum_table.style.to_latex(caption='Summary stats from EDA'))\n",
    "\n",
    "```\n",
    "\n",
    "Of course, the stats provided in this pre-built table are not very customised. So what do we do to get the table that we actually want? Well, the answer is to draw on the contents of the previous data chapters, particularly the introduction to data analysis. Groupbys, merges, aggregations: use all of them to produce the EDA table that you want.\n",
    "\n",
    "If you're exploring data, you might also want to be able to read everything clearly and see any deviations from what you'd expect quickly. **pandas** has some built-in functionality that styles dataframes to help you. These styles persist when you export the dataframe to, say, Excel, too.\n",
    "\n",
    "```{note}\n",
    "For some styling features, for example `style.format()`, you will need an up to date version of **pandas**. You can run `pip install -U pandas` on the command line to update a pip-installed **pandas** package, or `conda update pandas` for conda.\n",
    "```\n",
    "\n",
    "Here's an example that highlights some ways of styling dataframes, making use of several features such as: unstacking into a wider format (`unstack()`), changing the units (`lambda` function; note that `1e3` is shorthand for `1000` on computers), fill NaNs with unobtrusive strings (`.fillna('-')`), removing numbers after the decimal place (`.style.format(precision=0)`), and adding a caption (`.style.set_caption`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    df.groupby([\"year_sold\", \"bedrooms\"])\n",
    "    [\"sale_price\"]\n",
    "    .mean()\n",
    "    .unstack()\n",
    "    .apply(lambda x: x / 1e3)\n",
    "    .fillna(\"-\")\n",
    "    .style.format(precision=0)\n",
    "    .set_caption(\"Sale price (thousands)\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Another way to achieve the `groupby()`, `mean()`, and `unstack()` steps in one step is using `df.pivot_table(index=\"YearSold\", columns=\"Bedrooms\", values=\"SalePrice\", aggfunc=np.mean`.)\n",
    "\n",
    "Although a neater one than we've seen, this is still a drab table of numbers. The eye is not immediately drawn to it!\n",
    "\n",
    "To remedy that, let's take a look at another styling technique: the use of colour. Let's say we wanted to make a table that showed a cross-tabulation between year and number of bathrooms; that is the counts of objects appearing in both of these fields according to the categories, eg in 2015, there were *5* houses with 2 bathrooms. This is different to what we did before, which was the mean price based on groups of houses by number of bathrooms and year.\n",
    "\n",
    "To perform a cross-tabulation, we'll use the built-in `pd.crosstab()` but we'll ask that the values that appear in the table (counts) be lit up with a heatmap using `style.background_gradient()` too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.crosstab(df[\"bedrooms\"], df[\"year_sold\"]).style.background_gradient(cmap=\"plasma\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, `background_gradient()` highlights each number relative to the others in its column; you can highlight by row using `axis=1` or relative to all table values using `axis=0`. And of course `plasma` is just one of [many available colormaps](https://matplotlib.org/stable/tutorials/colors/colormaps.html)!\n",
    "\n",
    "```{admonition} Exercise\n",
    "Do a new cross-tabulation using a different colourmap.\n",
    "```\n",
    "\n",
    "Here are a couple of other styling tips for dataframes.\n",
    "\n",
    "First, use bars to show ordering:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    df.iloc[:10, -6:-1]\n",
    "    .style.format(precision=0)\n",
    "    .bar(subset=[\"cost_per_sq_ft\", \"sale_price\"], color=\"#d65f5f\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second, use `format()` to add informative suffixes and round numbers appropriately:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.groupby([\"bedrooms\"])[[\"splp_pct\"]].mean().style.format(\"{:.0f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Third, use `.hightlight_max()`, and similar commands, to show important entries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[:10, -6:-1].style.highlight_max().format(\"{:.0f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can find a full set of styling commands [here](https://pandas.pydata.org/pandas-docs/stable/user_guide/style.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploratory plotting\n",
    "\n",
    "**pandas** has some built-in plotting options to help you look at data quickly. These can be accessed via `.plot.*` or `.plot()`, depending on the context. Let's make a quick `.plot()` of the mean quarterly nominal Sale Price over time (with extra options passed via keyword arguments):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    df.set_index(\"datetime\")\n",
    "    .groupby(pd.Grouper(freq=\"3M\"))[\"sale_price\"]\n",
    "    .mean()\n",
    "    .apply(lambda x: x / 1e3)\n",
    "    .plot(\n",
    "        title=\"House sales in Grinnell, Iowa\",\n",
    "        xlabel=\"\",\n",
    "        ylabel=\"Mean nominal sale price (000s USD)\",\n",
    "    )\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, if you can get the data in the right shape, you can plot it. The same function works with multiple lines\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    df.set_index(\"datetime\")\n",
    "    .groupby(pd.Grouper(freq=\"3M\"))[[\"orig_price\", \"list_price\", \"sale_price\"]]\n",
    "    .mean()\n",
    "    .apply(lambda x: x / 1e3)\n",
    "    .plot(style=[\"-\", \":\", \"-.\"])\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see some of the other quick `.plot.*()` options.\n",
    "\n",
    "Here's a kernel density estimation (KDE) plot. Just to show what's possible, we'll use the `df.columnname` syntax, an alternative to `df['columnname']`, and set limits via keyword arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.square_feet.plot.kde(ylim=(0, None), xlim=(0, None));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A bar chart (use `barh()` for horizontal orientation; `rot` sets rotation of labels):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.value_counts(\"bedrooms\").sort_index().plot.bar(title=\"Counts\", rot=0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This chart shows why EDA is so important: if we are modelling residential housing, and we find that there are observations that don't have *any* bedrooms, perhaps they should be excluded from the downstream analysis.\n",
    "\n",
    "\n",
    "This next one, using `.plot.hist()`, provides another instructive example of how EDA can reveal interesting issues with datasets. First, there are many more houses built in 1900 than seem plausible and, checking the documentation for the dataset, it does say that when a construction date is not known, it is entered as 1900-which explains what's going on here. Second, some of the construction years are in the future! Is this plausible? It could be. Some houses are sold before the construction of an estate is completed; this could be the case here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"year_built\"].plot.hist(bins=30, title=\"Year of construction\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Boxplot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(df[[\"orig_price\", \"list_price\", \"sale_price\"]].plot.box());"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"class_ln_price\"] = pd.cut(np.log(df[\"orig_price\"]), bins=4, precision=0)\n",
    "\n",
    "(\n",
    "    df.set_index(\"datetime\")\n",
    "    .groupby([pd.Grouper(freq=\"A\"), \"class_ln_price\"])[\"class_ln_price\"]\n",
    "    .count()\n",
    "    .unstack()\n",
    "    .plot.area(ylabel=\"Count\")\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scatter plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.plot.scatter(x=\"square_feet\", y=\"sale_price\", alpha=0.7, ylim=(0, None));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hexbin:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.plot.hexbin(\n",
    "    y=\"list_price\",\n",
    "    x=\"splp_pct\",\n",
    "    gridsize=10,\n",
    "    cmap=\"inferno_r\",\n",
    "    C=\"cost_per_sq_ft\",\n",
    "    sharex=False,\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **skimpy** for summary statistics\n",
    "\n",
    "The **skimpy** package is a light weight tool that provides summary statistics about variables in data frames in the console (rather than in a big HTML report, which is what the other EDA packages in the rest of this chapter too). Sometimes `df.summary()` isn't enough but a HTML report is too much, and **skimpy** fills this gap. Let's see it in action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from skimpy import skim\n",
    "\n",
    "skim(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{admonition} Exercise\n",
    "Apply exploratory data analysis to the 'titanic' dataset, which can be loaded from [this link](https://github.com/datasciencedojo/datasets/raw/master/titanic.csv).\n",
    "\n",
    "First run **skimpy**'s `skim()` function. Then see what analysis you can do using **pandas** built-in exploratory methods.\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The **ydata-profiling** package\n",
    "\n",
    "```{warning}\n",
    "The live example in this section is not currently working due to **ydata-profiling**'s dependency on an older version of **numpy**. If you need to use something from this section, you probably still can: *coding for economists* has a huge number of dependencies, much more than a normal project, and this particular problem may not affect you. If you want to help, you can show your support for a resolution on [this GitHub issue](https://github.com/ydataai/ydata-profiling/issues/1456)—though please do bear in mind that most open source libraries are run by volunteers, and you should always be constructive in your interactions. The second is to contribute to the library yourself by creating a pull request that fixes the problem.\n",
    "```\n",
    "\n",
    "The EDA we did using the built-in **pandas** functions was a bit limited and user-input heavy. The [**ydata-profiling**](https://docs.profiling.ydata.ai/) library aims to automate the legwork of EDA for you. It generates 'profile' reports from a pandas DataFrame. For each column, many statistics are computed and then relayed in an interactive HTML report.\n",
    "\n",
    "Let's generate a report on our dataset using the `minimal=True` setting (the default settings produce a lot of computationally expensive extras):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from ydata_profiling import ProfileReport\n",
    "\n",
    "\n",
    "# profile = ProfileReport(\n",
    "#     df, minimal=True, title=\"Profiling Report: Grinnell House Sales\"\n",
    "# )\n",
    "# profile.to_notebook_iframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a full on report about everything in our dataset! We can see, for instance, that we have 17 numerical variables, 0 boolean, and 4 categorical (which includes string), and 1 datetime. The overview also tells us the number of observations, that we have 1.2% of values missing across all columns, and we have zero duplicates.\n",
    "\n",
    "The warnings page shows where **pandas-profiling** really shines. It flags *potential* issues with the data that should be taken into account in any subsequent analysis. For example, although not relevant here, the report will say if there are very unbalanced classes in a low cardinality categorical variable.\n",
    "\n",
    "Let's explore a couple of the warnings that have been raised. `Address` has high cardinality; but that's okay, we expect most addresses to be different because the turn-over of houses is not that frequent. We also see that there are a lot of missing values for `LotSize`.\n",
    "\n",
    "Now let's dig down into a specific variable, `SalePrice`. You can view more about it using the 'Toggle details' button. We get four extra pages of detail about this variable, in addition to the headline statistics: a comprehensive page of descriptive statistics, a page with a histogram of the variable, a page with a table of common values, and a page of extreme values.\n",
    "\n",
    "```{admonition} Exercise\n",
    "Run **ydata-profiling** on the 'titanic' dataset.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The **dataprep** package\n",
    "\n",
    "The [**dataprep**](https://dataprep.ai/) package offers very similar functionality to **ydata-profiling**; it produces an in-depth report on the input data. There isn't much difference between them in general apart from it seems like **dataprep** has slightly better support for string column types and is a little bit richer on visualisations (it has interactive plots that you can change the axes of). It's well worth checking out as an alternative to **ydata-profiling**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review\n",
    "\n",
    "That's it for this very brief tour of exploratory data analysis with code. You should know now how to do simple analysis of **pandas** dataframes using techniques from other chapters to create summary statistics, using styling of tables of summary statistics, using the quick visualisation options that are built-in to **pandas**, and using the **skimpy** package. You should also know where to go for more in-depth, heavy duty exploratory data analysis reports, namely **ydata-profiling** and **dataprep**."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.12 ('codeforecon')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "c4570b151692b3082981c89d172815ada9960dee4eb0bedb37dc10c95601d3bd"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}