Measurement resolution and rounding: avoid invented precision

Last updated: 5 Oct 20265 min read

tutorial

The number of printed digits is a display choice; the instrument resolution and collection process limit what those digits mean.

Read the instrument contract

A logger records temperature to the nearest 0.1 degree Celsius. A stored 8.2 does not mean the underlying quantity is known exactly to one decimal place. If nearest rounding is the only error source, a raw reading of 8.2 is consistent with a value near 8.15 through 8.25 under the tie rule. Calibration bias and sensor noise can widen that range. Keep resolution, unit, sensor model and time of measurement beside the value.

Round only at the final boundary

Converting Fahrenheit to Celsius, applying calibration and averaging six readings each introduce arithmetic steps. Rounding after every step creates extra error and may push a borderline shipment across a temperature limit. Preserve enough working precision, then round the final displayed value under one declared rule. Canonical units come before display formatting.

Distinguish representation from uncertainty

Binary floating point can display a decimal sum with unexpected trailing digits. Decimal arithmetic can make a monetary or recorded-decimal calculation reproducible, but it does not remove measurement uncertainty. A sensor that reports 8.2 with a 0.1-degree display resolution cannot become a 0.001-degree instrument merely because the database stores three decimal places. Derived uncertainty needs an explicit model of the inputs.

Apply threshold rules to raw and rounded values

Suppose an alert fires above 8.25 degrees Celsius. Rounding 8.249 to one decimal gives 8.2 and rounding 8.251 gives 8.3, but the classification should use the unrounded canonical reading if the sensor supports that precision. If the original device reported only one decimal, those three-digit values may be unjustified. Record whether the limit is strict or inclusive and what uncertainty band warrants review.

Make a boundary fixture

Test values exactly at, just below and just above the operating limit. Include positive and negative numbers because rounding ties can behave differently across languages and libraries. Save the rounding mode with the report version. When an upstream logger changes its resolution, treat that as a measurement-system revision and compare old and new readings before splicing the series.

Implementation

from decimal import Decimal, ROUND_HALF_EVEN

def display_temperature(canonical_celsius, places="0.1"):
    return Decimal(str(canonical_celsius)).quantize(
        Decimal(places), rounding=ROUND_HALF_EVEN
    )

assert display_temperature("8.249") == Decimal("8.2")
assert display_temperature("8.251") == Decimal("8.3")
assert display_temperature("8.25") == Decimal("8.2")

Performance and operating cost

Final display rounding is O(1) per fixed-precision reading. Keeping unrounded canonical values and metadata requires O(N) storage for N readings. Decimal operations cost more than float operations, but repeated premature rounding usually poses the larger analytical risk.

Common Mistakes

Do not equate extra stored decimals with better sensor accuracy.
Do not apply a threshold to a rounded display value unless the decision contract explicitly says to.
Do not omit the tie-breaking rounding mode from a reproducible report.