Processing and analysing data
Once data is loaded, programs clean it, transform it, and summarise it. Libraries such as pandas make these steps fast and expressive, but careful thinking is still essential.
The data pipeline
Data work usually follows a pipeline: extract data from sources, transform it by cleaning, joining, and reshaping, and load it into its destination, such as a database, report, or model (ETL). Writing each step as a clear, repeatable script, rather than editing data by hand, makes results reproducible and errors traceable.
pandas basics
pandas is Python's leading data-analysis library. A DataFrame is a table with labelled columns: df = pd.read_csv("sales.csv"). df.head() shows the first rows, df.describe() summarises numeric columns, and df["region"] selects a column. Filtering uses conditions: df[df["amount"] > 100].
Cleaning data
Real data is messy. Common tasks: handle missing values (df.dropna() or df.fillna(0), depending on meaning), remove duplicates (df.drop_duplicates()), fix types (pd.to_datetime for dates), standardise text such as trimming spaces and fixing case, and check for impossible values like negative ages. Cleaning often takes most of an analysis's time.
Transforming and aggregating
New columns can be computed: df["total"] = df["price"] * df["qty"]. groupby summarises by category: df.groupby("region")["total"].sum() gives sales per region. merge joins tables on shared keys like SQL joins, and pivot_table reshapes data into summary grids.
Vectorised operations
pandas and its foundation, NumPy, perform operations on whole columns at once in fast compiled code, such as df["price"] * 1.2, instead of Python loops over rows. Vectorised code is typically many times faster and shorter. Looping over rows with iterrows is usually a sign that a faster approach exists.
Visualising
Charts reveal patterns that tables hide. Matplotlib is Python's foundational plotting library; seaborn and plotly build on it with statistical and interactive charts; df.plot() gives quick views. Choose charts to fit the question: lines for trends over time, bars for comparing categories, scatter plots for relationships, and histograms for distributions.
Pitfalls in analysis
Correlation does not imply causation: ice-cream sales and drownings both rise in summer. Averages hide outliers and skew, so look at medians and distributions too. Sampling bias distorts conclusions, and repeatedly slicing data until something looks significant produces false discoveries. Document assumptions and sanity-check results against expectations.
Worked example
Given sales.csv with columns date, region, price, and qty: df = pd.read_csv("sales.csv", parse_dates=["date"]); df = df.drop_duplicates(); df["total"] = df["price"] * df["qty"]; monthly = df.groupby([df["date"].dt.to_period("M"), "region"])["total"].sum(). If the North region shows a sudden zero month, check for missing data before concluding sales collapsed.
Key takeaways
Data pipelines extract, transform, and load (ETL) through repeatable scripts. pandas DataFrames load, inspect, select, and filter tabular data. Cleaning handles missing values, duplicates, types, text, and impossible values, and often takes the most time. New columns, groupby, merge, and pivot_table transform and summarise. Vectorised operations beat row loops. Matplotlib, seaborn, and plotly visualise; match chart types to questions. Beware correlation-causation confusion, misleading averages, bias, and fishing for significance.
Flashcard
Extract, transform, load
Check your understanding
Why is df["price"] * 1.2 preferred over looping through rows with iterrows to apply a price increase?