Data Science Foundations with Python & SQL
Describe variation before choosing a headline
Distinguish typical values, unusual observations and uncertainty about the future.
In this chapter
- Distinguish typical values, unusual observations and uncertainty about the future.
- Produce and check the chapter's practical artifact.
- Observed waits
- Distribution
- Resampling
- Qualified claim
Each transition should preserve enough information to explain and reproduce the result.
One number hides several stories
The mean adds observations and divides by their count. The median is the middle observation after sorting. Neither is universally the right summary. For queue waits of 2, 3, 3, 4 and 28 minutes, the mean is eight minutes while the median is three. Reporting only the median hides one very long wait; reporting only the mean can imply that eight minutes is typical for most visitors.
Start with the distribution: list or plot the values, report the sample size and explain the observation window. Ask whether an unusual value is an entry error or a genuine service failure. If the 28-minute wait is real, removing it because it makes the chart unattractive changes the question. Keep it and analyze the operating conditions that produced it.
Use a resampling thought experiment
The script repeatedly samples five values with replacement and calculates the mean. Its spread illustrates sensitivity to the small observed dataset. The resulting interval is an exploratory bootstrap percentile interval, not a guarantee about future waits. Independent, representative sampling is an assumption; five convenient observations rarely justify strong conclusions.
from statistics import mean, median
from random import Random
waits = [2, 3, 3, 4, 28]
rng = Random(41)
means = sorted(mean(rng.choices(waits, k=len(waits))) for _ in range(1000))
print("n:", len(waits), "mean:", mean(waits), "median:", median(waits))
print("exploratory interval:", means[25], means[974])Treat the sampling process as part of the analysis
A fixed random seed makes the classroom calculation reproducible; it does not make the sample representative. If volunteers recorded only quiet periods, more resampling repeats that bias. Sampling from multiple opening periods and recording missed observations may improve the evidence more than adding a sophisticated statistical technique.
For the staffing question, report waits by period together with arrival counts, staffing levels and known disruptions. Do not claim that high demand caused long waits merely because they occurred together. A service outage or a complex loan could explain the association. Plan a limited experiment with an explicit comparison period and measures of unintended effects.
Communicate the practical uncertainty: 'this small sample contains one serious delay, and we need coverage across more periods' is more useful than a precise-looking interval without a sampling description. Keep exploratory analysis separate from a pre-specified decision threshold.
Workbench
- Run the script and verify mean eight, median three.
- Compare summaries after adding one more 28-minute wait; explain why they change.
- Draft a sampling plan covering both busy and quiet periods.
Expected checks
- The original outlier remains visible.
- Your memo includes n=5 and avoids a causal claim.
Check your understanding
What does a fixed random seed provide?
- A representative sample
- Repeatable random draws
- A guaranteed forecast
Answer explanation
Repeatability and representativeness are separate properties.
Official tools & further reading
The curriculum
- Ask a question the data can answer — Free preview
Define a decision, an observation and a useful denominator before writing code.
- Make invalid records visible — Free preview
Parse a small dataset into validated records without hiding rejected observations.
- Turn an analysis into a reusable function — Sign-in access
Separate a calculation from input/output and check its boundaries.
- Join tables without multiplying the story — Sign-in access
Use keys, grouped queries and reconciliation checks to make SQL results trustworthy.
- Describe variation before choosing a headline — Free preview
Distinguish typical values, unusual observations and uncertainty about the future.
- Design a chart that survives questions — Sign-in access
Choose clear units, honest scales and a comparison that answers the decision.
- Build a pipeline that fails informatively — Sign-in access
Connect validation, aggregation and a deterministic output with explicit failure behavior.
- Deliver the Community Meter decision pack — Sign-in access
Combine code, checks and a careful recommendation in a complete local project.