Nonnegative PU risk objective and its assumptions

Last updated: 5 Oct 20265 min read

tutorial

A PU risk objective can combine labeled positives and an unlabeled mixture using an assumed class prior, but a negative component below zero signals empirical overfitting or assumption failure.

Decompose the binary risk

For an illustrative loss, the expected negative-class loss among unlabeled pumps includes both true negatives and hidden positives. Subtract the prior-weighted positive contribution to estimate negative risk. Add the prior-weighted positive loss for the full objective. This is an expectation identity under representative positive sampling and a correct prior; it is not a generic fix for selective tickets.

Keep the negative estimate bounded

With flexible models and few labeled positives, the empirical difference can become negative even though true risk cannot. A nonnegative variant bounds that term below by zero. The code computes this arithmetic from declared aggregate losses, not neural gradients. The bound prevents one failure mode but does not make labels or prior correct.

Compare against simple baselines

A ticket-versus-unlabeled classifier may be a useful ranking baseline, even if its raw probability targets confirmation rather than true failure. Compare a mature-label supervised subset, the simple baseline and the PU objective on the same independently adjudicated holdout. Holdout evaluation separates model choice from ticket availability.

Treat prior as a sensitivity parameter

A wrong positive prevalence changes the subtraction and the learned boundary. Train or evaluate a small grid of plausible priors and record how rankings and review volume move. Do not tune prior against the final test. Prior sensitivity defines the range.

Budget the batch mixture

If a minibatch contains only a few positive examples, its risk estimate can fluctuate sharply. Track the number of unique confirmed failures and avoid repeated near-duplicates from one asset. Identity-aware splits protect evaluation and prevent training batches from appearing more diverse than they are.

Implementation

def nonnegative_pu_risk(positive_loss_on_positives,
                            negative_loss_on_positives,
                            negative_loss_on_unlabeled,
                            positive_prior):
    if not 0 < positive_prior < 1:
        raise ValueError("class prior must be inside (0, 1)")
    positive_component = positive_prior * positive_loss_on_positives
    estimated_negative = (negative_loss_on_unlabeled
                          - positive_prior * negative_loss_on_positives)
    return positive_component + max(0.0, estimated_negative), estimated_negative

risk, raw_negative = nonnegative_pu_risk(0.38, 0.72, 0.19, 0.31)
assert round(raw_negative, 4) == -0.0332
assert round(risk, 4) == 0.1178
assert risk >= 0

Performance and operating cost

This scalar audit is O(1). Training requires repeated positive and unlabeled batches plus model forward/backward passes; cost depends on model size and batch construction. The nonnegative clamp stabilizes an empirical component but cannot identify a class prior or correct biased positive selection.

Common Mistakes

Do not apply the decomposition when confirmed positives are an unexamined selective subset.
Do not infer the true prior from the same objective without additional assumptions.
Do not report training risk as held-out failure recall.