Recommendation objective and interaction log: define what success means

Last updated: 5 Oct 20265 min read

tutorial

A recommender orders eligible items for a user or context according to a declared outcome and observation process.

Choose the outcome

AI Trove might recommend the next lesson after a reader finishes one. A click is easy to record but can reward misleading titles; a completed lesson or later quiz attempt may be more useful. State the decision point, eligible population, prediction window and primary outcome before selecting an algorithm. Metric denominators must count actual opportunities to see a recommendation.

Log the opportunity

An interaction record needs an anonymous or authorized learner key, item key, timestamp, placement, rank, policy version and whether the item was displayed. A missing click after an impression differs from an item that was never shown. Record the recommendation slate and eligibility snapshot so evaluation can reconstruct what the policy could have chosen. Exposure bias begins at this boundary.

Respect lifecycle rules

A completed lesson, unpublished draft or age-inappropriate resource should not be recommended merely because it has a high model score. Keep publication and progress state outside the training label. Define deletion and consent behavior for the interaction log, especially when a learner removes history. Aggregate reporting can use less identity detail than per-user serving.

Test a fixture

Give one learner two displayed lessons, one click and one completion. The click-through rate is one of two impressions; completion is one of two impressions or one of one clicks, depending on the predeclared metric. Neither denominator is interchangeable. Add an undelivered candidate and verify it does not become a negative example.

Implementation

def impression_metrics(records):
    shown = [record for record in records if record["displayed"]]
    if not shown:
        return {"ctr": None, "completion_per_impression": None}
    return {"ctr": sum(record["clicked"] for record in shown) / len(shown),
            "completion_per_impression": sum(record["completed"] for record in shown) / len(shown)}

Performance and operating cost

A scan over N logged decisions costs O(N) time and O(N) memory here because it retains shown rows; streaming sums use O(1) memory. The larger operating cost is logging a complete, privacy-bounded exposure record without losing rank and policy identity.

Common Mistakes

Do not treat an unshown item as a disliked item.
Do not optimize clicks without checking the learning outcome.
Do not serve unpublished or ineligible content because a score is high.