Exploratory Data Analysis

Exploratory data analysis, or EDA, is the practice of looking at a dataset to learn its structure, gaps, and oddities before deciding what to compute from it. The NIST engineering statistics handbook describes it as an approach that uses mostly graphical techniques to gain insight, uncover structure, detect outliers and anomalies, and test assumptions, letting the data suggest what is there rather than imposing a model first.

What to look at first

Count rows and ask what one row represents; a support-ticket extract of twelve rows can describe eleven tickets if one was reopened. Check whether the assumed key is unique. Look for blanks and for zeros, and ask what each means: a blank reply time may be a ticket nobody answered, a zero may be an automatic acknowledgement. Inspect ranges and the time span; a value of 3,795 minutes can be a real weekend wait rather than an error. Count the groups you intend to compare, because a rate on five records moves by twenty percentage points when one record changes.

Plots such as histograms and time plots serve the same purpose for larger data, and their settings matter: a histogram’s bin width can hide or exaggerate a pattern, so choose it from the data’s precision and the question rather than a software default.

The output people forget to write down

EDA produces summary statistics, plots of distributions and relationships, flagged observations, and tentative hypotheses. The output most often left unwritten is the documented choice each finding leads to: keep the unanswered ticket and report it, replace the automatic reply with the human one, convert clock minutes to business minutes, treat a numeric category code as a label. Writing the reason beside each choice lets another person disagree with the choice instead of with an unexplained number.

A pattern seen during exploration is a hypothesis. The same data that suggested it cannot confirm it, and a difference found by trying many splits may be chance. Record exploratory choices made after seeing results, and test the hypothesis on evidence the exploration did not use.

Reference: NIST/SEMATECH e-Handbook: What is EDA?. Examples here are illustrative.