Why Floating-Point Sums Change When You Reorder the Data

Reordering floating-point values can change their sum because intermediate results are rounded. The mathematical expression may be equivalent, but the machine performs a sequence of finite-precision operations. A parallel reduction, a new chunk size, or a different data order can therefore change a result without any missing rows.

This is particularly visible when large positive and negative values cancel while small contributions sit between them. The Python floating-point tutorial explains representation error and accurate summation options. The examples here use an explicit loop so they do not depend on changes to Python’s built-in sum implementation across versions.

Reproduce the order effect

Run this small diagnostic in an ordinary Python environment:

import math

def left_fold(values):
    total = 0.0
    for value in values:
        total += value
    return total

a = [1e16, 1.0, -1e16]
b = [1e16, -1e16, 1.0]
print(left_fold(a))  # 0.0
print(left_fold(b))  # 1.0
print(math.fsum(a))  # 1.0

In the first order, adding one to the large intermediate value cannot preserve that small increment in the usual binary floating-point representation. Subtracting the large value afterward does not recover the lost contribution. In the second order, the large terms cancel before the small value is added.

These comments describe the usual Python float environment. Record your interpreter and platform when using the example as a regression fixture. Never infer a universal error bound from three deliberately chosen values.

Check the input before blaming arithmetic

An order-dependent result is not proof that every discrepancy is numerical. Compare row counts, missing values, filters, and joins first. A duplicated join can change a total by much more than rounding, and no summation algorithm can restore the intended dataset.

Keep a reproducible input sample with the exact floating-point values that reached the reduction. If you save only rounded display strings, reloading them can change the problem. Include whether NaN, infinity, and signed zero are allowed and how they are treated.

Separate data validation from numeric evaluation. For a measurement pipeline, a physically impossible input should be flagged before aggregation rather than absorbed into a sum whose final magnitude looks plausible.

Choose the property you need

Accuracy and reproducibility are related but different. Accuracy asks how close the computed answer is to a suitable reference. Reproducibility asks whether a specified execution produces the same answer under the promised conditions.

A fixed left-to-right order can be reproducible while being inaccurate on cancellation-heavy data. A more accurate parallel algorithm can produce slightly different answers when its partitioning changes. Decide which combinations matter for the application instead of demanding an unexplained “exact sum.”

For an analytics chart, a documented tolerance may be enough. For a scientific comparison, use an independently computed higher-precision reference on representative cases. For an audit artifact, also preserve the reduction method and data ordering contract.

Compare alternative reductions

Pairwise summation combines smaller groups before combining the group totals. Compensated approaches track some lost low-order information. Python’s math.fsum is a practical option for many finite floating-point datasets. None of these choices should be described as a cure for invalid inputs or all possible numeric problems.

Benchmark accuracy and runtime separately. Include a mostly positive dataset, alternating signs, a large dynamic range, and values close to zero after cancellation. A test containing only similar positive values can hide the case that matters most to your workload.

Do not automatically sort every production dataset to improve one sum. Sorting costs time and memory and can destroy meaningful record ordering. If you use a deterministic ordering for a reduction, document that choice and test its performance at realistic scale.

Set tolerances from the domain

A useful comparison often combines an absolute tolerance near zero with a relative tolerance at larger magnitudes. Relative error alone becomes awkward when the correct total is zero or extremely small.

Suppose an invented sensor calculation has a reference total of 0.0002 units. A discrepancy of 0.0001 is small in absolute terms but enormous relative to that result. Whether it is acceptable depends on measurement uncertainty and the decision made from the total—not on the number of decimal places shown in a dashboard.

Write the tolerance beside the test and explain why it is appropriate. Avoid copying an arbitrary 1e-9 into every assertion. A tolerance that masks a missing record is not a numerical policy.

Make parallel changes reviewable

When changing a distributed reduction, record worker count, partition strategy, accumulator type, and library version. Compare both aggregate error and the worst affected groups. A global average can hide a small cohort whose values cancel heavily.

Retain the explicit-loop fixture above as a teaching test, but add real sanitized workloads before approving the change. A successful toy example demonstrates the mechanism; it does not establish the accuracy of your complete pipeline.

Conclusion

Floating-point addition is a sequence of rounded operations, so order matters. Verify the dataset, distinguish accuracy from reproducibility, and choose the reduction method and tolerances according to the decision your result supports.