Project: propose safe grammar edits for support replies

Last updated: 7 Oct 20265 min read

project

Build a draft editor that proposes minimal corrections, blocks changes to facts and leaves final approval with the support agent.

Define the input and output

The agent drafts a reply about order ZX-47 and a pending refund. The editor may correct agreement or punctuation but cannot change the order reference or promise a completion date. Output the source, proposed revision, edit spans, protected-field check and review status. Keep the original draft for rollback. The model never sends directly to a customer.

Assemble reviewed edits

Collect real drafts with agent consent and remove data not needed for the task. Annotate minimal acceptable edits, harmful rewrites, no-change cases and alternative valid corrections. Group drafts by ticket and customer before splitting. Create hard examples involving “may” versus “will,” a changed amount, quoted customer text and mixed-language fragments. The edit contract defines exactly what a proposal contains.

Compare and gate candidates

Start with a conservative rule baseline, then compare a model on the same frozen audit. Evaluate meaning preservation, agent acceptance, unnecessary edits and protected-token failures. Block a proposal that alters a protected fact; send large or uncertain edits for explicit review. The review policy separates safe proposals from doubtful ones. Show a visible diff so the agent does not have to reread the whole message.

Operate the release

Version tokenizer, edit model, field detector and policy together. Sample accepted and rejected proposals for quality review. Monitor agent time saved, meaning-change reports and false blocks. Retain raw drafts only under the support retention policy; keep aggregated metrics free of customer identifiers. A rollback restores the prior bundle and does not discard the agent’s original text.

Implementation

def support_edit_payload(original, proposed, protected_before, protected_after):
    if protected_before != protected_after:
        return {"state": "blocked", "reason": "protected-fields-changed"}
    if original == proposed:
        return {"state": "unchanged", "text": original}
    return {"state": "agent-review", "original": original,
            "proposed": proposed}

payload = support_edit_payload("Refund are pending for ZX-47.",
                               "Refund is pending for ZX-47.",
                               {"order": "ZX-47"}, {"order": "ZX-47"})
assert payload["state"] == "agent-review"

Performance and operating cost

The payload check is O(f + n + m) time for f protected fields and source/candidate lengths n and m; output copies both strings for agent review. Model inference and human approval dominate operating cost. Count time saved after full review, not just time to generate a candidate. A blocked draft still needs a usable path for the agent to continue manually.

Common Mistakes

Applying edits directly to a sent reply.
Comparing only the first occurrence of a repeated identifier.
Ignoring no-change drafts when reporting quality.
Discarding the original draft after model rewrite.