Null vs NaN in Polars: Avoid Silent Data-Cleaning Errors

Null and NaN describe different conditions. A null indicates a missing value; NaN is a floating-point value representing an undefined numerical result. In Polars, cleaning one does not automatically clean the other. Choose separate rules and test their effect before computing totals or filtering rows.

The difference becomes visible when a pipeline mixes CSV imports, calculations and joins. A blank source cell can become null, while a numerical operation can produce NaN later. A single generic 'remove bad values' step can hide those distinct causes.

Build a fixture that includes both

import polars as pl

sample = pl.DataFrame({
    "reading": [1.0, None, float("nan"), 0.0],
    "source": ["sensor", "missing", "calculation", "zero"]
})
print(sample.select(
    pl.col("reading").is_null().alias("missing"),
    pl.col("reading").is_nan().alias("not_a_number")
))

The zero is intentional. It gives the test a value that is valid but easy to damage with a broad cleanup rule. Inspect how the expressions behave for null rows in your current version instead of assuming every result is an ordinary true/false value.

The Polars missing-data guide explains these representations. Use that distinction to design your data policy; do not copy a blanket replacement from an unrelated dataset.

Preserve the reason a value is unavailable

A missing temperature reading might indicate a disconnected sensor. NaN after a calculation might indicate an invalid formula input. Replacing both with zero can make the dashboard look healthy while removing the evidence needed to fix the source problem.

Keep a reason field or error counter when the application needs that distinction. A numerical column alone may not carry enough context for later diagnosis. Decide whether invalid calculations should reject the batch, exclude a record or remain visible with a warning.

Use CSV Viewer to inspect an invented source sample. Do not assume that the literal string NaN, an empty field and a missing row are equivalent. CSV reader configuration and schema interpretation influence what enters the DataFrame.

Write replacement rules separately

Use the appropriate null and NaN operations for the intended policy. For example, an approved reporting rule may fill missing optional counts with zero while leaving invalid numerical results flagged for review.

clean = sample.with_columns(
    pl.col("reading").fill_null(0.0).alias("missing_filled")
)

This demonstrates null filling only. It does not claim that zero is the correct policy for sensor readings. Add NaN handling only after deciding what an undefined result means in your application.

Keep the original column when reviewing a new cleaning rule. Compare original and transformed values on a fixture before replacing the production field. That makes it possible to see which records were altered and why.

Check filtering and aggregation explicitly

Test the exact filter used by the application, including how it handles missing comparison results. Do not assume a comparison such as 'greater than zero' treats absent values the same way as an explicit missing-value check.

Calculate the expected aggregate manually for a small fixture. State which rows should contribute and which should be excluded. If the reported average changes after cleaning, determine whether the denominator changed, the numerator changed or both.

Include a group with every value missing and another with every value invalid. An apparently successful overall average can conceal those groups. A dashboard should have an intentional display for no usable observations rather than accidentally treating that state as zero activity.

Joins and casts can introduce new gaps

A join may create nulls when there is no matching record. That is a relationship problem, not necessarily a missing source measurement. Count unmatched keys and examine their expected coverage before filling the resulting fields.

A failed or permissive cast can also alter missing-value behavior. Test strings, whitespace, unusual numeric forms and out-of-range values according to your parser's supported options. Keep rejected rows in a controlled error report rather than silently dropping them.

Compare a small redacted JSON summary using JSON Diff: row count, null count, invalid-result count and unmatched-join count. Those numbers help explain a pipeline change without exposing every underlying record.

Keep cleaning observable

Record how many values each rule modifies. A sudden increase in replacements can indicate an upstream schema change or collection failure. Thresholds should come from the dataset's expected behavior, not from a generic percentage copied from another application.

Give each rule a version and a reason. When business policy changes, retain enough history to explain why this month's numbers differ from last month's. Cleaning logic is part of the meaning of the data, not merely preparation for displaying it.

Review export behavior as well as internal calculations. A downstream JSON or CSV consumer may represent missing and invalid numerical values differently. Create a tiny round-trip fixture and verify that its meaning survives the output format. If the format cannot express the distinction directly, include an explicit reason field rather than hoping a blank cell preserves it.

Clean deliberately rather than cosmetically

Separate absence from undefined arithmetic, preserve valid zeros and test every filter and aggregate on a small fixture. For reading large files efficiently after the policy is settled, see Polars Lazy CSV Scans.