Weight extremes, effective sample size and trimming tradeoffs

Last updated: 5 Oct 20265 min read

tutorial

Unequal weights can reduce precision and make one response influential; trimming them changes the estimand unless totals are restored and assumptions are reviewed.

Read the weight distribution

After queue-level nonresponse adjustment, 48 standard respondents each carry weight about 16.67 and 32 partner respondents each carry 12.5. Their total represents 1,200 cases. Show minimum, median, maximum, selected quantiles, sum and the share of total weight held by the largest respondents. A single large weight may indicate a low-response cell, a rare design stratum or a coding error. Investigate its origin before applying an arbitrary cap.

Use a weights-only precision diagnostic

The Kish effective sample size for 80 respondents with these weights is (sum of weights) squared divided by sum of squared weights, about 78.5. It indicates how much unequal weights alone reduce a simple-random-sample-equivalent count. It ignores clustering, stratification, outcome association and nonresponse model error. Precision planning needs the full design when a confidence interval or target margin is required.

Understand the trimming bargain

If one respondent has weight 95 while many others carry five, capping 95 at 25 reduces concentration but also removes represented population mass. Raking or another calibration after trimming may restore known totals while moving other weights. Neither step guarantees that the outcome bias improved. Compare estimates and control margins before and after, and explain why the chosen cap is defensible.

Inspect estimates by subgroup

An overall effective size can hide a partner subgroup represented by only a few respondents. Report unweighted counts and effective size for each published domain. If one domain has no respondents, no amount of scaling creates evidence about its clarity answers. Positivity checks take priority over a tidy final total.

Use design-aware uncertainty

A naive standard error computed as if 80 equal-probability independent cases had been sampled misses stratification, finite-population sampling, nonresponse adjustment and calibration. Preserve the sample design and use a suitable survey variance method or documented replicate weights. The effective size is a diagnostic for weight dispersion, not a complete variance estimator. Calibration can alter the weight distribution again.

Implementation

def weight_diagnostics(final_weights):
    if not final_weights or any(weight <= 0 for weight in final_weights):
        raise ValueError("positive respondent weights required")
    total = sum(final_weights)
    square_total = sum(weight * weight for weight in final_weights)
    return {"respondents": len(final_weights),
            "weight_sum": total,
            "effective_size": total * total / square_total,
            "largest_share": max(final_weights) / total}

respondent_weights = [800 / 48] * 48 + [400 / 32] * 32
diagnostic = weight_diagnostics(respondent_weights)
assert diagnostic["respondents"] == 80
assert 78 < diagnostic["effective_size"] < 79

Performance and operating cost

Diagnostics over R respondents cost O(R) time and O(1) extra space. Sorting for quantiles adds O(R log R) time. A small effective size points to instability, but variance estimation still needs the original design and outcome structure.

Common Mistakes

Do not present a weights-only effective size as a complete survey standard error.
Do not cap an extreme weight without checking the population margins it represents.
Do not hide tiny subgroup respondent counts behind a healthy overall effective size.