Ranking and re-ranking: balance relevance with coverage and curriculum rules

Last updated: 7 Oct 20265 min read

tutorial

A ranker orders eligible candidates; a final pass enforces slate-level constraints that item scores alone cannot express.

Score comparable candidates

If several candidate sources produce their own scores, those values may be on incompatible scales. A common ranker can use context, item features and candidate-source identity, then calibrate against observed outcomes. Evaluate the declared learning metric rather than assuming a higher click probability is enough. Outcome choice controls the ranking objective.

Construct the slate

Five individually high-scoring lessons can all cover the same concept. Add a simple diversity rule, such as at most two items from one subject in a five-item slate, and record when it is relaxed. A prerequisite gate is harder than a diversity preference. Apply hard constraints before optimizing soft ones, and surface the reason when no valid slate exists.

Keep freshness and safety

Fresh material may need a discovery slot, but recency alone does not prove relevance. Remove withdrawn pages at request time and cap exposure of a new lesson until quality signals arrive. Candidate eligibility and serving-time validation are both needed if the index is stale.

Check tradeoffs

Make a fixture with four high-scoring lessons from one subject and two slightly lower-scoring lessons from another. A three-item slate with a two-per-subject cap must include at least one from the second subject. Report relevance cost and subject coverage alongside the final order. An apparently diverse slate that repeats the same prerequisite is not useful diversity.

Implementation

def capped_subject_slate(ranked_items, slate_size=3, per_subject_cap=2):
    if slate_size < 1 or per_subject_cap < 1:
        raise ValueError("slate limits must be positive")
    selected = []
    counts = {}
    for item in ranked_items:
        subject = item["subject"]
        if counts.get(subject, 0) >= per_subject_cap:
            continue
        selected.append(item)
        counts[subject] = counts.get(subject, 0) + 1
        if len(selected) == slate_size:
            break
    return selected

Performance and operating cost

This pass costs O(K) time and O(S) state for K ranked candidates and S encountered subjects. More complex slate optimization can be expensive; start with explicit hard gates and measurable simple constraints before introducing a learned re-ranker.

Common Mistakes

Do not compare uncalibrated scores from unrelated candidate sources.
Do not apply a soft diversity preference in place of a hard access rule.
Do not maximize slate variety while hiding relevance loss.