Video temporal convolutions and clip aggregation

Check the five axes

A common video tensor order is batch, channels, frames, height and width. Swapping channels with time can make a valid-looking convolution run on the wrong axis if dimensions happen to match. The code asserts the input and output shapes for a five-frame grayscale clip. Record whether values are RGB, grayscale or encoded motion and keep preprocessing identical at serving. The clip lesson defines the physical-time meaning of each frame slot.

Choose temporal support for the event

A width-three temporal kernel sees neighboring frames, while stacking layers broadens its receptive field. Early temporal pooling can discard a fault visible in only one or two frames. Global average pooling is easy but can dilute a brief jam; maximum pooling may react strongly to noise. Compare aggregation rules on reviewed short and sustained events with the same clip sampling. Keep spatial resolution high enough to see belt motion but measure its memory cost.

Distinguish offline and causal use

An offline clip classifier can inspect frames before and after an event midpoint. A live warning at time t cannot use frames after t. If deployment is live, construct training clips ending at t, then evaluate first-alert delay on replayed streams. A symmetric temporal convolution over a clip ending at t is still causal with respect to t, but a centered clip around t is not. Causal sequence reasoning applies despite the different model shape.

Control duplicate and background cues

Random crops from the same video can share camera texture, timestamps or compression artifacts. Split by recording and camera when needed, and use controlled background changes to see whether motion matters. Compare with a one-frame classifier and a simple frame-difference statistic. If the temporal model only wins on a split that shares source recordings, do not claim event generalization. Report error by event duration and the independent video count.

Measure the streaming envelope

The model may score overlapping clips every few frames. Count decoded frames, reused feature work, GPU memory and p95 end-to-end alert time. A long clip or dense stride can increase both coverage and compute. Package clip duration, sampling timestamps, frame resize, aggregation and score threshold. The project tests whether event recall improves at an acceptable false-alert rate and latency.

Implementation

import torch
from torch import nn

torch.manual_seed(47)
camera_clips = torch.rand(2, 1, 5, 8, 8)
temporal_features = nn.Conv3d(1, 4, kernel_size=(3, 3, 3),
                              padding=(1, 1, 1))(camera_clips)
assert temporal_features.shape == (2, 4, 5, 8, 8)
spatially_pooled = temporal_features.mean(dim=(-1, -2))
assert spatially_pooled.shape == (2, 4, 5)
clip_features = spatially_pooled.amax(dim=-1)
jam_head = nn.Linear(4, 2)
jam_logits = jam_head(clip_features)
assert jam_logits.shape == (2, 2)

Performance and operating cost

For B clips, C input channels, F filters, T frames, H-by-W pixels and a kernel of k_t by k_h by k_w, direct 3D convolution work is roughly O(BCFTHWk_tk_hk_w). Intermediate activations occupy O(BFTHW), often dominating the tiny classifier head. Temporal max pooling adds little compute but can amplify noisy peaks; average pooling may miss brief events. The example checks shape and aggregation only. Measure the full decode-and-score path at the target clip rate.

Common Mistakes

• Do not confuse channel and frame axes.
• Do not train on clips centered on future frames for a live-warning claim.
• Do not let global pooling hide short events without an event-duration audit.

Read next

• Video clip timestamps and recording-group evaluation
• Project: detect conveyor jams from timestamped video clips
• Convolution output geometry and receptive fields
• Causal dilated convolution and receptive field
• Inference contracts: preserve preprocessing and measure tail latency