Ranking query groups and relevance labels

Last updated: 5 Oct 20265 min read

tutorial

A learning-to-rank record represents a candidate within a query group; relevance, eligibility and feature time must be defined at that group’s decision point.

Define one decision event

For a maintenance manual search, a query group is one technician request at one timestamp with its eligible candidate pages. The same page can appear under many requests, but it is not the evaluation unit. Record the request ID, query text version, candidate-set version, document revision and time each feature was available. A later repair outcome cannot become a feature for an earlier search. Prediction-time availability applies to ranking too.

Separate relevance from exposure

A page marked relevant by a reviewer answers the information need under a stated rubric. A clicked page only tells you that a result was displayed and selected under a particular layout. Unshown pages are not negative labels. If reviewers judge only displayed results, the label pool inherits the existing ranker’s blind spots. Exposure bias is the adjoining problem.

Keep group boundaries intact

Training and validation must not put candidates from the same request into different folds. For a future search deployment, split by request time and embargo cases whose judgments mature after the cutoff. If the same incident appears repeatedly, group those requests again by incident when leakage across repeated queries is plausible. Group and time validation gives the broader split rule.

Choose a meaningful label scale

A four-level rubric might mean 0 is unsafe or irrelevant, 1 offers background only, 2 identifies the repair step, and 3 gives the approved procedure with the correct equipment. Reviewers need examples of borderline grades and an adjudication path. Do not silently convert a click into grade 3; behavior and editorial relevance answer different questions.

Record the candidate denominator

A reranker cannot recover a relevant page absent from the retrieved set. Keep candidate recall and ranker quality as separate numbers, along with permission-filter failures and missing document revisions. Retrieval recall and reranking describes the upstream boundary; serving review measures it.

Implementation

requests = [
    {"request_id": "pump-47", "timestamp": 912, "document_id": "seal-a7", "grade": 3, "feature_ready": 903},
    {"request_id": "pump-47", "timestamp": 912, "document_id": "parts-b4", "grade": 1, "feature_ready": 909},
    {"request_id": "valve-62", "timestamp": 938, "document_id": "isolation-c9", "grade": 2, "feature_ready": 923},
]

def validate_ranking_records(records):
    by_request = {}
    for record in records:
        if record["grade"] not in (0, 1, 2, 3):
            raise ValueError("grade outside approved rubric")
        if record["feature_ready"] > record["timestamp"]:
            raise ValueError("future feature in search request")
        by_request.setdefault(record["request_id"], set())
        if record["document_id"] in by_request[record["request_id"]]:
            raise ValueError("duplicate candidate in request")
        by_request[record["request_id"]].add(record["document_id"])
    return {request_id: len(documents) for request_id, documents in by_request.items()}

assert validate_ranking_records(requests) == {"pump-47": 2, "valve-62": 1}

Performance and operating cost

Validation takes O(N) expected time and O(N) memory for N candidate records. Stored group IDs, document revisions and feature timestamps add ingestion cost, but they make leakage and denominator audits possible. Full query groups must remain together during training, evaluation and distributed processing.

Common Mistakes

Do not split rows at random while keeping candidates from one request on both sides.
Do not treat an unshown document as an observed negative.
Do not use a feature produced after the request timestamp.