Random forest feature subsampling and leaf support

Last updated: 5 Oct 20265 min read

tutorial

A random forest combines bootstrap-trained trees while restricting split candidates to a sampled feature subset; leaf size controls how narrowly each tree can fit.

Separate the two random choices

A shipment forest first samples training shipments for each tree. At each split it also considers only a sampled subset of input features. The second choice can stop every tree from choosing the same dominant backlog field and can reduce correlation between trees. It does not guarantee independent predictions. Feature sampling happens repeatedly at split nodes, not once for the entire forest. Bagging isolates the row-sampling part.

Keep leaves large enough to mean something

A leaf containing one training shipment memorizes its outcome. Setting a minimum leaf count forces splits to retain support on both sides, trading some training fit for stability. In rare-event classification, raw row count may hide that a leaf contains no positive cases. Inspect class counts and calibration by slice. The example searches one split for a sampled feature set and rejects splits below a minimum leaf size; it is an inspectable component, not a full forest implementation.

Score candidate splits on the right unit

The example uses weighted Gini impurity for a binary delayed-shipment label. It computes that impurity from training labels only. For a regression forest, the split criterion changes, commonly to a squared-error objective. Do not use a holdout label to pick thresholds. A holdout is for evaluating the fitted forest, and model selection keeps its final test untouched.

Avoid claiming causality from split frequency

A feature can be selected often because it has many possible thresholds, correlates with another field or is available only after a shipment was already delayed. Neither split count nor impurity reduction is an effect estimate. Check held-out permutation importance, feature availability, and whether the same conclusion survives alternative training windows. Correlated variables may substitute for one another.

Bound the operational cost

More trees reduce Monte Carlo variation but increase training, storage and inference roughly with tree count. Larger minimum leaves and shallower trees can lower latency. Measure the actual request budget, not only validation accuracy. Save feature schema and model version with the forest; an extra or reordered field at serving time can invalidate every split. The release review records the tradeoff.

Implementation

from random import Random

training_rows = [
    {"backlog": 9, "staff": 7, "delayed": 0},
    {"backlog": 12, "staff": 6, "delayed": 0},
    {"backlog": 16, "staff": 5, "delayed": 0},
    {"backlog": 22, "staff": 5, "delayed": 1},
    {"backlog": 26, "staff": 4, "delayed": 1},
    {"backlog": 33, "staff": 3, "delayed": 1},
]

def gini(rows):
    if not rows:
        raise ValueError("empty leaf")
    positive_rate = sum(row["delayed"] for row in rows) / len(rows)
    return 2 * positive_rate * (1 - positive_rate)

def best_supported_split(rows, candidate_features, minimum_leaf):
    candidates = []
    for feature in candidate_features:
        values = sorted({row[feature] for row in rows})
        for left_value, right_value in zip(values, values[1:]):
            threshold = (left_value + right_value) / 2
            left = [row for row in rows if row[feature] <= threshold]
            right = [row for row in rows if row[feature] > threshold]
            if min(len(left), len(right)) < minimum_leaf:
                continue
            score = (len(left) * gini(left) + len(right) * gini(right)) / len(rows)
            candidates.append((score, feature, threshold))
    return min(candidates) if candidates else None

candidate_features = Random(47).sample(["backlog", "staff"], k=1)
chosen_split = best_supported_split(training_rows, candidate_features, 2)
assert chosen_split is not None
assert chosen_split[0] < gini(training_rows)
assert chosen_split[1] in candidate_features

Performance and operating cost

At one node with N rows, F candidate features and up to N thresholds each, this direct implementation can cost O(FN²) time because it repeatedly scans rows. A production tree sorts or bins values to reduce split-search work. Forest memory and serving time grow with the number and depth of trees.

Common Mistakes

Do not sample features once per forest and describe it as per-node sampling.
Do not leave one-row leaves unconstrained on small or noisy training sets.
Do not read feature split counts as causal effects.