Abusive-language labels: context, target and quotation

Last updated: 7 Oct 20265 min read

tutorial

A text fragment can be harmful, quoted for reporting, reclaimed or misunderstood without conversation context. Store the judgment unit and target explicitly.

Define the moderation question

A support reply that quotes an abusive message to report it should not be assigned the same author intent as the original attack. Record the message under review, its speaker, preceding turns, quoted spans, target and label policy. A whole-thread label may be useful for case management, but the actionable judgment still needs a specific message and span. Discourse units separate quoted material from what the current speaker asserts.

Keep labels task-specific

Insult, targeted harassment, threat and identity-directed abuse are different decisions under a written policy. A profanity term alone cannot establish any of them. Annotators need enough conversation context to identify who is addressed, whether the phrase is a direct attack and whether the message is counterspeech or evidence in a complaint. Preserve unresolved labels where context is absent. Label-policy review prevents accidental collapse of distinct judgments into a noisy binary flag.

Protect privacy and dialect coverage

Moderation data can contain sensitive personal details. Restrict raw context, retain only what review requires and separate public explanations from internal evidence. Audit false positives for dialects, code-switching, reclaimed terms and community-specific usage without assuming that any language variety has one fixed meaning. Dialect slice review helps detect uneven errors, but small slices need careful interpretation and consent-aware data handling.

Measure the right failures

Evaluate false positive and false negative decisions by phenomenon, target, language and quotation status. Review boundary cases with multiple annotators and document reasons for disagreement. A model can score well on direct insults while missing contextual threats or silencing reports that quote abuse. The routing lesson keeps uncertain cases with trained reviewers; the project tests the full queue.

Implementation

def moderation_annotation(message_id, author_id, context_ids,
                          label, target_id, quoted_spans):
    allowed = {"targeted-abuse", "not-abusive", "unresolved"}
    if label not in allowed or not message_id or not author_id:
        raise ValueError("invalid annotation identity or label")
    if label == "targeted-abuse" and not target_id:
        raise ValueError("target is required for targeted abuse")
    return {"message_id": message_id, "author_id": author_id,
            "context_ids": tuple(context_ids), "label": label,
            "target_id": target_id, "quoted_spans": tuple(quoted_spans)}

report = moderation_annotation("reply-47", "reporter-82", ["message-91"],
                               "unresolved", None, [(12, 31)])
assert report["context_ids"] == ("message-91",)
assert report["quoted_spans"] == ((12, 31),)

Performance and operating cost

Copying c context IDs and q quote spans takes O(c + q) time and space. The example validates annotation shape, not whether the quote offsets are inside source text or whether the policy label is correct. Production annotation must check immutable text revisions, access controls and reviewer rationale.

Common Mistakes

Treating a reported quote as the reporter’s own attack.
Using one offensive word as a complete moderation policy.
Dropping speaker and target identity from annotation.
Assuming missing context means a safe negative label.