Extraction validation: repair limits and conflicting values

Last updated: 6 Oct 20265 min read

tutorial

Separate missing, malformed, unsupported and conflicting field states, then repair only what the source can justify.

Make failure states explicit

An extractor can return a malformed object, a valid object with missing required fields, or two plausible values for one field. These are different failures. Preserve a state per field: found, missing, conflicting or unsupported. A value is unsupported when the source span does not substantiate the interpretation. Do not turn a missing approval into “approved” to satisfy a required schema. Schema-bound extraction gives each candidate a source span before repair begins.

Repair against the source, not the previous guess

A retry may fix a formatting error, but repeated model calls can also invent a plausible value. Limit retries and provide the original document plus the validator’s specific error. Reject a repair that changes evidence without a new valid span. If two versions of a window are present, the correct action may be human review rather than asking the model to choose. Keep each attempt, model version and validation result for audit. The review queue should show the competing phrases side by side.

Apply document policy before release

Required fields depend on the workflow. A search index may accept a partial record, while a scheduler must hold a change request without a verified service and window. Define these gates outside the model prompt and version them. Conflicting values can be resolved only by a trusted rule, such as a signed amendment superseding a draft; mere recency in a concatenated text block is not enough. Revision invalidation shows why a corrected field must update dependent outputs.

Measure repair harm

Track format-repair success separately from semantic correction. Count newly hallucinated fields, evidence-span changes, reviewer overrides and downstream release errors. Evaluate adversarial cases with duplicated labels, negated approval, obsolete sections and OCR noise. A high document completion rate can hide dangerous false positives. In the change-request project, a record with conflicting windows must be held even if every field is syntactically present.

Implementation

REQUIRED_FIELDS = {"service", "window_start", "approval"}

def extraction_release_state(field_states):
    unknown = set(field_states) - REQUIRED_FIELDS
    if unknown:
        return {"state": "review", "reason": "unknown-fields"}
    missing = REQUIRED_FIELDS - set(field_states)
    if missing:
        return {"state": "review", "reason": "missing-fields",
                "fields": sorted(missing)}
    if any(state == "conflicting" for state in field_states.values()):
        return {"state": "review", "reason": "conflicting-values"}
    if any(state != "found" for state in field_states.values()):
        return {"state": "review", "reason": "unsupported-value"}
    return {"state": "ready-for-evidence-review"}

states = {"service": "found", "window_start": "found",
          "approval": "conflicting"}
assert extraction_release_state(states)["reason"] == "conflicting-values"
assert extraction_release_state({**states, "approval": "found"})["state"] ==     "ready-for-evidence-review"

Performance and operating cost

With f field states, set creation and inspection take expected O(f) time and O(f) space. Retry inference cost depends on document length and model calls; cap attempts and preserve source spans. The ready state only advances to evidence review, because field-state labels alone cannot prove semantic correctness.

Common Mistakes

Retrying until the model invents a required field.
Calling a conflicting value merely missing.
Treating the latest paragraph as authoritative without document policy.
Using schema completeness as the only release metric.