Topic discovery: corpus boundaries, model choice and human labels

Last updated: 6 Oct 20265 min read

tutorial

A topic model can organize thousands of tickets, but a coherent cluster is not automatically a product issue or a stable taxonomy.

Decide what a topic should represent

For support triage, the unit may be one customer problem rather than one reply. Strip quoted prior messages only when their removal is auditable; otherwise the same boilerplate dominates every topic. Keep case identity, product version, language, channel and capture time. Choose whether the purpose is exploration, routing or trend reporting. A model trained for exploration does not become a production classifier merely because its top terms look readable. Corpus identity defines the row boundary.

Compare representations

Count-based latent topics, sparse matrix factorization and embedding clusters answer different questions. Sparse methods expose characteristic terms and often make version comparison easier; embedding clusters can group paraphrases but may also join unrelated intents with similar wording. Fit vocabulary or encoder only within the training boundary. Try a no-model baseline of existing support tags before adding complexity. Separate rare security or billing cases from a broad “other” cluster so size does not hide importance.

Give topics human meaning

Top words are diagnostics, not labels. Sample representative and hard-edge tickets for each candidate topic and ask reviewers to name its shared issue, mark mixed clusters and flag sensitive terms. Store the model version, reviewed label, sample IDs and uncertainty. A topic number such as 7 is local to a fit and must not be used as a permanent business key. The drift review establishes how to compare later fits without pretending IDs are stable.

Test usefulness

Use held-out tickets to evaluate assignment stability, reviewer agreement, coverage of actionable themes and separation of unrelated cases. Coherence can help reject unreadable term lists, but it cannot establish product usefulness alone. Slice by language and channel; boilerplate or transliteration can become a “topic” of its own. The monitoring project makes these checks part of an editorial workflow, rather than publishing automated labels as facts.

Implementation

from collections import Counter

def representative_terms(documents, stop_terms, top_k=5):
    if top_k < 1:
        raise ValueError("top_k must be positive")
    term_counts = Counter()
    for document in documents:
        term_counts.update(term for term in document.lower().split()
                           if term not in stop_terms and len(term) > 2)
    return [term for term, _ in term_counts.most_common(top_k)]

tickets = ["gateway callback retry", "callback retry duplicate", "gateway timeout"]
assert representative_terms(tickets, {"the"}, 2) == ["gateway", "callback"]

Performance and operating cost

Counting terms is O(total tokens) time and O(v) space for vocabulary size v. Matrix factorization or clustering adds repeated passes over documents and may scale with vocabulary, embedding dimension and topic count. A larger topic count can improve apparent fit while producing tiny uninterpretable clusters. Budget reviewer time for representative and boundary examples; that cost decides whether discovery yields a usable taxonomy.

Common Mistakes

Treating a model-local topic number as a stable category.
Naming a topic from its five highest-weight words without reading cases.
Fitting vocabulary on the final time-split audit set.
Equating statistical coherence with operational usefulness.