Project: ship an auditable support-entity extractor

Last updated: 7 Oct 20265 min read

project

Build an extraction release that produces typed spans, survives tokenization changes and rejects uncertain identifiers before they enter a support workflow.

The operational contract

The support desk receives refund messages containing an order ID, purchase date or delivery postcode. The extractor returns original-text offsets, entity type, confidence and model bundle version. It never edits an order based only on a guessed span. Define a no-entity result and an abstain result separately: the former says the model found none, while the latter says the message needs review. Match the label policy to the corpus contract before collecting examples.

Build and split the corpus

Annotate character spans on the captured original text. Version guidelines for punctuation and copied email headers; adjudicate conflicting reviews and keep the adjudication reason. Group messages by conversation and customer, then split by capture time so repeated order IDs or copied text cannot leak across train and test. Freeze a small difficult holdout with code-switching, copied signatures and OCR-like substitutions. The split must be independent of tokenization. Grouped temporal validation gives the reusable split rule.

Train, decode and release

Derive BIO targets from the frozen span records, mask non-first subwords according to the alignment contract, then decode predictions against original offsets. Publish micro exact-span F1, per-type recall, false identifier rate and human-review volume. Block release if any required entity type falls below its reviewed threshold, even if aggregate F1 rises. Keep a sparse baseline for comparison; neural complexity should pay for itself with fewer costly errors. Exact-span evaluation supplies the scoring unit.

Operate the endpoint

Package tokenizer files, label order, model digest, calibration data, maximum input length and abstention threshold as one version. Reject unrecognized versions and strings whose spans cannot round-trip. Return only the minimum extracted fields needed by the desk. Maintain a restricted sample review queue for production mistakes, and never train on a corrected label until its annotation version and consent policy are recorded. Connect the endpoint to text serving and keep a rollback bundle.

Implementation

def validate_extraction(message, entities, allowed_types):
    validated = []
    for entity in entities:
        entity_type, start, end, extracted_text = entity
        if entity_type not in allowed_types:
            raise ValueError("unknown entity type")
        if not (0 <= start < end <= len(message)):
            raise ValueError("entity offset outside original message")
        if message[start:end] != extracted_text:
            raise ValueError("entity text does not round-trip")
        validated.append({"type": entity_type, "start": start,
                          "end": end, "text": extracted_text})
    return validated

message = "Refund order ZX-47, please."
assert validate_extraction(message, [("ORDER", 13, 18, "ZX-47")], {"ORDER"})[0]["start"] == 13

Performance and operating cost

Validation scans e extracted entities and compares their selected substrings, for O(e + total selected characters) time and O(e) output records. Model inference usually dominates latency; measure p95 by message length and language, not only overall average. Human review has a real queue cost: estimate it from abstention rate and arrival volume before choosing a threshold. Archive evaluation summaries without unrestricted raw ticket bodies.

Common Mistakes

Annotating normalized text while serving offsets into the original message.
Counting a missing entity and an abstention as the same response.
Splitting copied conversations across train and test.
Shipping a model without its exact tokenizer and label map.