Client participation, dropouts and stale updates

Last updated: 5 Oct 20265 min read

tutorial

A federated round observes only clients that finish on time; selection and dropout can shift both the training population and its reported quality.

Track the full round funnel

For each depot, record eligibility, invitation, acceptance, local training start, completion, accepted update and reason for rejection. A round with four accepted updates from four invitees looks healthy; the same four from twenty eligible depots may not represent the network. Keep counts by site type and device class. Aggregation applies only after this funnel.

Reject stale updates deliberately

A client can finish after the server has moved to a new global version. Applying its delta to the new checkpoint changes the optimization equation. Choose a synchronous cutoff or a specifically designed asynchronous protocol with version-aware weights; do not silently append late updates. The code flags a stale base and an incomplete packet.

Distinguish randomness from systematic loss

If participation is approximately random, missing clients add variance. If low-connectivity or high-failure depots miss most rounds, the model can be biased toward easier sites. Compare participating and nonparticipating client metadata and pre-round performance, while respecting local data boundaries. Support overlap gives the broader selection warning.

Budget communication as a distribution

Log upload bytes, round duration, retry count and battery or compute limits. A median round time hides sites that consistently exceed the deadline. Shortening the timeout may speed training while systematically excluding them. Tune the cutoff with site coverage and target performance, not speed alone.

Audit exposure to personal data

Update packets and client metadata may leak information even if raw images remain local. Minimize identifiers and apply an explicit threat model before collecting site-level telemetry. Data minimization and aggregation risk set those boundaries.

Implementation

round_packets = [
    {"depot": "north-47", "invited": True, "finished": True, "base": "round-8", "examples": 180},
    {"depot": "west-62", "invited": True, "finished": True, "base": "round-7", "examples": 90},
    {"depot": "east-83", "invited": True, "finished": False, "base": "round-8", "examples": 30},
    {"depot": "south-94", "invited": False, "finished": False, "base": None, "examples": 66},
]

def round_funnel(packets, expected_base):
    invited = [packet for packet in packets if packet["invited"]]
    accepted = [packet for packet in invited if packet["finished"]
                and packet["base"] == expected_base and packet["examples"] > 0]
    stale = [packet["depot"] for packet in invited if packet["finished"]
             and packet["base"] != expected_base]
    return {"eligible": len(packets), "invited": len(invited),
            "accepted": len(accepted), "stale": stale}

assert round_funnel(round_packets, "round-8") == {
    "eligible": 4, "invited": 3, "accepted": 1, "stale": ["west-62"]
}

Performance and operating cost

The audit scans C client packets in O(C) time and stores O(C) accepted or rejected IDs in the worst case. Logging every step has network and retention costs. A shorter round can improve wall time but reduce site coverage; evaluate both before changing the timeout.

Common Mistakes

Do not count invited clients as successful participants.
Do not apply an old-checkpoint delta to the current model without a defined asynchronous method.
Do not describe missing clients as random without testing participation patterns.