Input normalization turns raw material into a stable set of fields, units, timestamps, and identifiers. It is not the same as deleting messy rows: missing fields remain missing and conflicting values remain visible. A model can compare numbers only when the units and periods are explicit. Give the model a compact field dictionary and ask it to surface records that do not meet the contract. Deterministic parsing belongs in application code where possible; a prompt should interpret the normalized record, not substitute for basic validation.
Make this comfortable
Prompt inputs: normalize records before asking for conclusions
Decision in practice
A warehouse planner sends a spreadsheet export containing 63 stock movements. Some quantity fields are cartons, others are individual units, and two records reuse a shipment ID. The ingestion step labels units and preserves source row IDs. The prompt asks for the net movement by SKU only after a validator checks that each carton size is known. For the two reused shipment IDs it must report a conflict, not silently deduplicate them. An operator samples the arithmetic for three SKUs and checks that the reported unknown count matches the rejected rows.
Input record: row_id, shipment_id, sku, quantity, unit, carton_size.
Precheck: reject unknown unit or missing carton_size.
Prompt task: summarize valid movement; list rejected row IDs separately.
Pass: no unit conversion without a recorded factor.Performance and operating cost
Normalization adds an O(N) pass over N rows and a small storage copy, but prevents a costly conclusion from inconsistent data. Keep the raw export for audit and version the conversion rules. If a model is asked to infer carton size from nearby rows, one plausible answer can contaminate an entire inventory report. A failure report should count invalid rows and explain which field blocked calculation. Distinguish a duplicate business event from a duplicate identifier; those require different reconciliation.
Common Mistakes
- Do not silently convert unknown units.
- Do not drop conflicting rows without a count and reason.
- Do not assume a duplicated identifier proves a duplicated event.
Connected lessons
- Prompt Engineering
- Prompt engineering: define a task that can be checked
- Output contracts: parse a result and preserve an explicit unknown state
- Prompt intent: turn an open request into an acceptance check
- Few-shot prompting: select examples that cover decisions, not just easy cases
- Project: evaluate an incident handoff prompt
- Prompt design decisions
Continue with: Dataset intake prompts: define a column before analyzing it.
Continue with: Analytics prompts: reconcile identities, duplicates, and late arrivals.
Continue with: Invoice prompts: distinguish duplicate ingests from credits.
Continue with: Backfill prompts: map known states and quarantine unknowns.
prompt engineering
tutorial