pandas missing data handling is something every real dataset eventually requires — spreadsheets have blank cells, APIs return null fields, sensors fail to report on certain days. This article covers pandas fillna dropna in depth, plus interpolate(), with a genuine decision framework for which strategy actually fits which missingness pattern, rather than just demonstrating all three and leaving you to guess.
Missing Data Is the Norm, Not the Exception
Missing data is inevitable
Missing data is an inevitable part of working with real-world datasets — spreadsheets have blank cells, APIs return null fields for optional data, sensors occasionally fail to report readings on certain days. Any pandas workflow built around the assumption that data will always be complete will eventually break on real data.
The sentinels pandas uses
Pandas uses specific sentinel values to represent different kinds of "missing," treated consistently by its detection and handling tools:
NaN(Not a Number) — for missing numerical data.None— for missing Python objects, like strings.NaT(Not a Time) — for missing datetime values.
Despite representing conceptually different "flavors" of missing, pandas' detection tools (covered next) treat all three consistently — you generally don't need to check for each one separately.
What this article covers
Detecting missing values, dropping them, filling them with fixed or computed values, interpolating for ordered/time-series data, and — the genuine focus of this article — guidance on which approach actually fits a given real-world situation.
Detecting Missing Values: isna() and notna()
df.isna()
import pandas as pd
import numpy as np
df = pd.DataFrame({
"name": ["Alex", "Sam", None, "Jordan"],
"age": [30, np.nan, 25, 35],
})
print(df.isna())
# name age
# 0 False False
# 1 False True
# 2 True False
# 3 False Falsedf.isna() returns a boolean DataFrame of the same shape as the original, marking True wherever a value is missing. .isnull() is a complete alias — both do exactly the same thing, and you'll see either used interchangeably across real-world code and documentation.
A practical detection workflow
print(df.isna().sum())
# name 1
# age 1
# dtype: int64
print(df.isna().sum().sum())
# 2Chaining .isna().sum() gives you a per-column count of missing values — genuinely the standard first step in any real workflow, before deciding on a strategy. Chaining a second .sum() collapses that down to a single overall total, useful for a quick, high-level sense of just how much missing data you're dealing with across the entire DataFrame.
notna(): the direct opposite
has_age = df[df["age"].notna()]
print(has_age)
# name age
# 0 Alex 30.0
# 2 None 25.0
# 3 Jordan 35.0.notna() is the direct opposite check, useful for explicitly filtering down to only the rows that do have a value present in a given column — genuinely handy when a specific downstream calculation absolutely requires that column to be non-missing.
Dropping Missing Data with dropna()
Basic usage
print(df.dropna())
# name age
# 0 Alex 30.0df.dropna() removes any row containing at least one missing value, by default — here, dropping rows 1 (missing age) and 2 (missing name), leaving only row 0, since it's the only row with no gaps at all.
print(df.dropna(axis=1))
# Empty DataFrame — both columns have at least one missing valueaxis=1 drops columns instead of rows — here, dropping both columns entirely, since each one has at least one missing value somewhere.
The how parameter for finer control
df.dropna(how="any") # default — drop if ANY value is missing
df.dropna(how="all") # only drop if EVERY value in that row/column is missinghow="all" is considerably more forgiving than the default — it only drops a row (or column) that's entirely empty, leaving partially-complete rows intact, even if they're missing some values.
The thresh parameter: a middle ground
df.dropna(thresh=1) # keep any row with at least 1 non-missing valuethresh sets a genuine middle ground between how="any" (aggressive) and how="all" (lenient): it requires a row (or column) to have at least the specified number of non-missing values to survive. This is genuinely useful when losing an entire row over a single missing field feels too aggressive, but keeping rows that are almost entirely empty feels too lenient — thresh lets you draw that line precisely, rather than being stuck choosing between the two extremes how offers.
Filling Missing Data with fillna() and interpolate()
fillna() with a fixed value or computed statistic
df["name"] = df["name"].fillna("Unknown")
df["age"] = df["age"].fillna(df["age"].mean())
print(df)
# name age
# 0 Alex 30.000000
# 1 Sam 30.000000
# 2 Unknown 25.000000
# 3 Jordan 35.000000fillna() replaces gaps with something meaningful: a fixed scalar ("Unknown"), or a computed statistic like the column's mean, median, or (via .mode()) most frequent value. Filling with the column mean is a genuinely common default for numerical data — it doesn't distort the column's overall average, though it's worth being aware it can understate the column's true variance if a large share of values were missing.
Filling different columns with different values
df = df.fillna({"name": "Unknown", "age": df["age"].mean()})Passing a dictionary to fillna() specifies a different fill value per column, all in a single call — usually the more sensible real-world approach, since a blanket fill value applied uniformly across an entire DataFrame rarely makes sense once different columns represent genuinely different kinds of data.
Forward fill and backward fill for ordered data
readings = pd.DataFrame({"temperature": [72, np.nan, np.nan, 75, 74]})
print(readings["temperature"].fillna(method="ffill"))
# 0 72.0
# 1 72.0
# 2 72.0
# 3 75.0
# 4 74.0
print(readings["temperature"].fillna(method="bfill"))
# 0 72.0
# 1 75.0
# 2 75.0
# 3 75.0
# 4 74.0method="ffill" (forward fill) carries the previous valid value forward into each subsequent gap — genuinely appropriate for time-ordered data where "the last known reading" is a reasonable stand-in for a missing one. method="bfill" (backward fill) does the reverse, pulling the next valid value backward.
readings["temperature"].fillna(method="ffill", limit=1)limit caps how many consecutive gaps get filled this way — genuinely important when a long stretch of missing data shouldn't all be filled with one increasingly stale value; capping the fill to just one or two consecutive gaps prevents a single known reading from being stretched across an unreasonably long gap.
interpolate(): a more statistically grounded option
readings = pd.DataFrame({"temperature": [72.0, np.nan, np.nan, 75.0, 74.0]})
print(readings["temperature"].interpolate())
# 0 72.0
# 1 73.0
# 2 74.0
# 3 75.0
# 4 74.0interpolate() fills gaps by estimating values between known points, defaulting to linear interpolation — here, smoothly stepping from 72 to 75 across the two-gap stretch, rather than simply repeating 72 (as forward fill would) or 75 (as backward fill would). For data with a genuinely curved, non-linear trend, interpolate() also supports method="polynomial", "quadratic", and "cubic", fitting a curve rather than a straight line between known points.
Choosing the Right Strategy for Your Data
A practical decision framework
This is the genuine core of this article, worth centering the close on:
If less than roughly 5% of rows are missing, and the gaps appear to be genuinely random (not concentrated in any particular pattern),
dropna()is generally safe — you're losing a small, presumably unbiased slice of your data, and the simplicity of dropping is hard to beat.If there's a meaningful default value, or a reasonable statistic to substitute,
fillna()is the better fit — a categorical field with an obvious "Unknown" fallback, or a numerical field where the column mean or median is a defensible stand-in for a missing value.If the data is ordered and numerical, with a genuine underlying trend — time series, sensor readings, anything where adjacent values are meaningfully related to each other —
interpolate()typically preserves the data's actual shape best, since it accounts for the surrounding values rather than substituting one fixed number everywhere.
A genuinely current best practice: avoid inplace=True
This deserves emphasis, and it's more than just a style preference: pandas 3.0 (released January 2026) made Copy-on-Write the default, permanent behavior — and under Copy-on-Write, calling an in-place method on a column selected out of a DataFrame (df["age"].fillna(0, inplace=True)) no longer reliably modifies the original DataFrame at all. The chained selection (df["age"]) behaves as its own copy under Copy-on-Write, so the in-place operation happens on that copy, not on df itself.
# Under pandas 3.0+, this does NOT modify df as you might expect
df["age"].fillna(0, inplace=True)
# The reliable, correct approach — plain assignment
df["age"] = df["age"].fillna(0)
# Or, operating on the whole DataFrame directly with a dict
df = df.fillna({"age": 0})Beyond this genuine correctness issue, assignment (df = df.fillna(0)) has always been the more readable choice anyway — it works naturally inside a chained sequence of transformations, and it makes each transformation step in a data-cleaning pipeline explicit and easy to follow, rather than relying on a method silently mutating something in the background. (One related detail worth knowing: as of pandas 3.0, in-place methods like fillna(), dropna(), and interpolate() now return the modified object itself — self — rather than None, when inplace=True is still used elsewhere on a genuinely standalone object; but given the chained-selection issue above, plain assignment remains the safer, more broadly reliable default regardless.)
A closing practical habit
Always re-run .isna().sum() after applying any missing-data strategy, to confirm the gaps were actually resolved as expected — genuinely important since it's entirely possible to apply fillna() to the wrong column, forget a column entirely when passing a dictionary, or (as covered above) fall victim to the Copy-on-Write inplace=True issue and end up with a DataFrame that looks handled, but genuinely isn't. Verifying the result directly, rather than assuming the operation worked as intended, is a small habit that catches a genuinely common class of silent, easy-to-miss mistake.