Model cache warmup and memory planning for shared inference

Last updated: 6 Oct 20265 min read

tutorial

Plan resident models and cold-start behavior before a shared serving pool receives real traffic.

Budget bytes, not model counts

Two models of similar file size can have different resident memory after deserialization, runtime compilation or device transfer. Measure artifact download, load time, peak resident bytes and steady-state bytes for each digest on the actual worker type. Reserve room for features, request buffers and runtime overhead. A cache that holds the popular models during quiet tests may churn under burst. Admission control protects in-flight work; this page handles what happens before a model is ready.

Choose a warm set from demand

Keep frequently used, latency-sensitive models resident and prewarm them after deployment or autoscaling. A rarely used model can load on demand if the caller contract permits a delayed or retryable response. Measure the first-request penalty separately from warmed inference. An artifact digest change invalidates the warm set even if the model name stays the same. Do not silently serve a previous digest while the new one loads; promotion identity has to match the response record.

Avoid cache thrash

If total resident bytes exceed worker capacity, an unbounded set of sporadic models can repeatedly evict one another. Track loads, evictions, cold-start rate and p99 latency by model. Estimate memory headroom for the worst concurrent warm set, not the average number of loaded models. If the strict receipt model is repeatedly evicted by invoice variants, pin it or separate the pool. Latency budgets should include download and load time when those events can happen on a customer request.

Plan for scale-out and failure

A newly added worker begins cold, so autoscaling capacity is not immediately equivalent to ready capacity. Preload the approved digests, verify checksums and score a canary before shifting production traffic. Reserve enough ready workers to survive one failed instance without evicting every model. The applied capacity project compares cost and p99 response times for shared and dedicated pools under a burst and a forced worker restart. A capacity plan without startup measurements is incomplete.

Implementation

def warm_set_fits(models, worker_memory_mb, reserve_mb):
    if worker_memory_mb <= 0 or reserve_mb < 0:
        raise ValueError("invalid memory budget")
    if any(size < 0 for size in models.values()):
        raise ValueError("invalid model size")
    available = worker_memory_mb - reserve_mb
    required = sum(models.values())
    return {"fits": required <= available,
            "headroom_mb": available - required}

resident = {"receipt-risk": 820, "invoice-classifier": 460}
assert warm_set_fits(resident, 2048, 384) == {"fits": True, "headroom_mb": 384}
assert not warm_set_fits(resident, 1400, 384)["fits"]

Performance and operating cost

Summing m resident model sizes costs O(m) time and O(1) extra space. Storage, load and warmup cost scale with artifact bytes; model count alone says little about memory. The estimate uses measured resident sizes but omits fragmentation and temporary peak allocation, so validate it under concurrent requests and on scale-out workers.

Common Mistakes

Using artifact file size as resident memory without measurement.
Sending customer traffic to a new worker before warmup completes.
Ignoring a digest change because the alias name stayed the same.
Treating average cache hit rate as proof that critical models stay warm.