Quantity units and conversion contracts for mixed datasets

Last updated: 5 Oct 20265 min read

tutorial

A numeric column is not interpretable until its physical quantity, unit and conversion rule are known.

Name the quantity before the column

A cold-chain export has values labeled temperature, but one device sends Celsius and another Fahrenheit. A value of 46.4 is dangerous if treated as Celsius and ordinary if it means Fahrenheit. Store raw value, raw unit, canonical value, canonical unit, device ID and conversion version. The schema must reject unknown units instead of guessing from magnitude. Column profiling can reveal suspicious ranges, but it cannot establish the missing meaning by itself.

Separate scale conversion from unit conversion

Temperature conversion has an offset as well as a scale; multiplying Fahrenheit by five ninths alone is wrong. Length conversion between metres and kilometres uses a factor without an offset. A machine-learning standard score is a statistical transformation, not a physical unit conversion. Do not call both operations normalization in a pipeline log. Preserve the original measurement so a changed unit rule can be replayed and audited.

Avoid hidden aggregation across units

Summing 47 kilometres and 320 metres as if both values had the same unit produces a plausible-looking but invalid route length. Canonicalize first, then aggregate. For a rate such as kilometres per hour, ensure the numerator and denominator units match the reporting contract. A percent is also unitless but its underlying population and time window still matter. The metric contract supplies those dimensions.

Control conversion precision

Use a declared decimal representation for externally reported measurements when binary floating-point artifacts would complicate reconciliation. Keep enough intermediate digits for downstream calculations and round once at the published boundary. The converted display should not imply better sensor resolution than the raw reading. Resolution handling is a separate rule from arithmetic correctness.

Test reversible fixtures

Convert 46.4 degrees Fahrenheit to exactly 8 degrees Celsius under the stated formula, then convert that result back. Test both negative and positive values, since an offset error can hide on one fixture. Reject a bare 46.4 with no unit instead of assigning a default. Record the quantity type too, so pressure values cannot enter the temperature converter.

Implementation

from decimal import Decimal

def to_celsius(raw_value, raw_unit):
    measured = Decimal(str(raw_value))
    if raw_unit == "C":
        return measured
    if raw_unit == "F":
        return (measured - Decimal("32")) * Decimal("5") / Decimal("9")
    raise ValueError(f"unsupported temperature unit: {raw_unit}")

assert to_celsius("46.4", "F") == Decimal("8.0")
assert to_celsius("-40", "F") == Decimal("-40")

Performance and operating cost

Converting N rows costs O(N) time and O(1) extra memory if processed as a stream; retaining raw and canonical forms costs O(N) storage. Decimal arithmetic has a higher constant cost than binary float, so apply it where the reporting and reconciliation contract warrants it.

Common Mistakes

Do not guess a unit from the numeric range.
Do not treat a statistical z-score as a physical unit conversion.
Do not round each intermediate conversion before calculating a derived metric.