DQN target networks, legal-action masks and value updates

Identify the two estimators

An online network estimates action values and receives gradient updates. A target network is a delayed copy used to form the next-state estimate. Without that delay, the value being fitted moves with every parameter update and can magnify errors. Copy all weights at a declared interval or use a declared soft-update coefficient. Save both model states and the target-update counter in a resumable checkpoint. Checkpoint identity matters when training continues after interruption.

Apply masks before taking a maximum

If an empty shelf cannot supply an item, its dispatch action must not compete for the maximum future value. Represent legal actions as a Boolean mask for each state and assert that each nonterminal state has at least one legal action. Masked values may use a sufficiently low finite sentinel, but choose it with the numeric dtype in mind. The code uses an explicit Boolean mask and a two-action synthetic batch. The same eligibility logic must operate during data collection and serving.

Compute a detached bootstrap target

For reward r and discount gamma, the target is r plus gamma times the best permitted next value unless the transition is terminal. The target path must not backpropagate through the target network. Double DQN chooses the next action with the online model and evaluates that action with the target model; this can reduce overestimation from taking a maximum over noisy estimates, but does not correct an invalid reward or observation. The snippet demonstrates the double-estimator choice.

Audit learning against a baseline

Report reward by whole episode, stockouts, late orders and action frequency on held-out simulated periods. Compare with the current dispatch rule under identical demand sequences. A low temporal-difference loss is not proof of an improved operating policy. Freeze the environment and random seeds for comparison, then vary demand and lead-time assumptions to see where the candidate fails. The applied project includes this counterfactual replay.

Bound operational exposure

A network may assign a high value to a rare action because the replay log barely contains it. Track state-action support and require a fallback or human review outside that support. Shadow scoring exposes policy disagreements without executing them. If online rollout is allowed, cap the affected inventory, add a hard legal-action filter and retain a rapid disable switch. An offline score alone cannot establish safety under a changed policy.

Implementation

import torch
from torch import nn
from torch.nn import functional as functional

torch.manual_seed(47)
online_values = nn.Linear(2, 2)
target_values = nn.Linear(2, 2)
target_values.load_state_dict(online_values.state_dict())
stock_before = torch.tensor([[8., 3.], [5., 0.]])
stock_after = torch.tensor([[7., 4.], [4., 0.]])
chosen_actions = torch.tensor([[1], [0]])
rewards = torch.tensor([2., -3.])
terminals = torch.tensor([False, True])
legal_next = torch.tensor([[True, True], [True, False]])
predicted = online_values(stock_before).gather(1, chosen_actions).squeeze(1)
with torch.no_grad():
    next_action = online_values(stock_after).masked_fill(~legal_next, -1e9).argmax(1, keepdim=True)
    bootstrap = target_values(stock_after).gather(1, next_action).squeeze(1)
    expected = rewards + 0.93 * (~terminals).float() * bootstrap
loss = functional.smooth_l1_loss(predicted, expected)
loss.backward()
assert torch.isfinite(loss) and target_values.weight.grad is None

Performance and operating cost

For B transitions, A actions and a value network requiring F operations per state, two network passes cost roughly O(BF + BA) and carry online activations proportional to the model width and B. A fixed target copy doubles parameter storage but avoids target-path gradient storage. Action masking adds O(BA) work. In a real simulator, generating sufficiently varied transitions may be far more expensive than this small update; measure episode throughput as well as accelerator time.

Common Mistakes

• Do not maximize over actions that are impossible at the next state.
• Do not allow target-network gradients into the optimizer step.
• Do not use training loss as a substitute for held-out episode outcomes.

Read next

• DQN transitions, replay sampling and reward contracts
• Project: audit a neural inventory dispatch policy before release
• Checkpoint recovery: save optimizer state and the run boundary
• AdamW decay groups and update-indexed schedules
• Temperature scaling and selective risk

Continue the workflow: PPO clipped policy ratios, critic loss and KL guards.