NDCG at K with a declared query denominator

Last updated: 5 Oct 20265 min read

tutorial

NDCG at K compares discounted gain in the displayed top K against the best ordering available in each judged query group.

Define gain and discount first

For graded maintenance-page relevance, use gain 2 to the grade minus 1 and divide each position by log base 2 of rank plus one. A grade-3 page at position one contributes more than the same page at position five. This is a business-weighted metric choice, not a universal truth. Declare K, gain mapping and whether unjudged pages are excluded or treated separately before scoring.

Normalize within each request

Compute the ideal discounted gain from the same judged candidate set, then divide observed gain by that ideal. Requests with no positive grade have zero ideal gain; assign a documented policy, such as excluding them from NDCG and reporting their count separately. Silently assigning zero can turn a change in no-answer traffic into a ranking regression.

Aggregate without losing traffic shape

A macro mean gives each request equal weight; a traffic-weighted mean gives repeated request types more weight. Both can be useful, but they answer different questions. Show the number of eligible requests and slices for rare equipment, document age and permission groups. Group error audits explain why pooled gains can conceal unacceptable gaps.

Watch candidate-set changes

If one ranker is evaluated over a larger candidate pool, its ideal denominator may differ. Hold candidate retrieval fixed for a pure reranker comparison. When retrieval and ranking both change, show candidate recall and end-to-end metrics together. Retrieval versus answer evaluation separates related layers.

Pair offline quality with an operational outcome

A higher NDCG at 5 may still increase time to reach the approved procedure if snippets are poor or the first result is stale. Use a later controlled release to assess task completion, safety and latency. Metric denominators and guardrails provide the experiment contract.

Implementation

from math import log2

def ndcg_at_k(ordered_grades, cutoff):
    if cutoff < 1 or any(grade < 0 for grade in ordered_grades):
        raise ValueError("valid cutoff and nonnegative grades required")
    def discounted_gain(grades):
        return sum((2 ** grade - 1) / log2(position + 2)
                   for position, grade in enumerate(grades[:cutoff]))
    ideal = discounted_gain(sorted(ordered_grades, reverse=True))
    if ideal == 0:
        return None  # Track no-positive groups outside the mean.
    return discounted_gain(ordered_grades) / ideal

search_grades = {"pump-47": [1, 3, 0, 2], "valve-62": [2, 0, 1]}
scores = [ndcg_at_k(grades, 3) for grades in search_grades.values()]
macro_ndcg = sum(scores) / len(scores)
assert 0 < macro_ndcg < 1
assert ndcg_at_k([0, 0], 3) is None
assert ndcg_at_k([3, 2, 1], 3) == 1.0

Performance and operating cost

Sorting each group of M candidates for its ideal order takes O(M log M) time; scoring the observed top K takes O(K) time. Across requests, memory can be O(M) per streamed group. Store the group count, empty-ideal count and candidate-pool version with the aggregate so a later comparison is interpretable.

Common Mistakes

Do not flatten all candidates into one global ranking before calculating NDCG.
Do not hide no-positive requests by silently changing the denominator.
Do not compare rerankers on different candidate pools without measuring upstream recall.