Feature Engineering: Your Best Feature Might Come from the Future

A delivery model suddenly becomes much more accurate after someone adds the latest carrier status. The new feature seems obvious: “out for delivery” is highly informative about whether a package will arrive today.

There is one problem. The product promises a delivery date at checkout.

The feature would be excellent for a different question—updating an estimate after dispatch. For this question, it arrives too late. Feature engineering begins with deciding what the model is allowed to know.

Write down the prediction moment

Consider a hypothetical model that estimates delivery duration when an order is placed. A candidate-feature review might look like this:

Candidate input	Available at checkout?	Decision
Destination region	Yes	Consider it
Chosen shipping service	Yes	Consider it
Weekend or weekday	Yes	Derive from the order timestamp
Assigned warehouse	Depends on the order flow	Verify before using
Actual dispatch time	No	Exclude
Latest carrier status from a later export	No	Exclude

This table is more useful than a long list of clever transformations. It connects the training columns to the system that will supply them in production.

Availability also applies to aggregates. “Average delivery time for this region” can be legitimate if calculated from outcomes known before the prediction. Calculating it from the entire dataset—including future deliveries—gives the model information it would not have had.

Better features are hypotheses

Suppose raw destination codes are hard for a simple model to use. We might derive a distance estimate, a regional category, or an indicator for areas served only on certain days.

Each feature expresses a hypothesis about the problem. Distance may relate to travel time. A service schedule may explain why the same distance produces different outcomes on different days.

These hypotheses can fail. Straight-line distance may be a poor substitute for the carrier’s network. A regional category may mask large differences within a region. Adding both raw and derived values can introduce redundancy without improving predictions.

Feature engineering is therefore an experiment, not a decoration contest. Start with a simple baseline. Add a coherent group of features. Evaluate on a split that resembles future use. Remove the group again to check whether the improvement depends on it. This removal test is often called an ablation.

Do not report an improvement merely because the training score rose. More opportunities to fit the existing data are not proof of better future estimates.

Some transformations must be fitted

A calendar feature can be calculated directly from a timestamp. A normalization scale or missing-value median is estimated from data. That distinction matters.

Fit learned preprocessing on the training portion, then apply the fitted transformation to validation and test data. If cross-validation is involved, fit it separately inside each training fold. Otherwise the supposedly held-out examples influence the representation used to evaluate them. Scikit-learn’s common-pitfalls guide shows why pipelines help enforce this boundary.

The production path needs the same definitions. “Last thirty days” must have an agreed timezone and cutoff. Missing values must mean the same thing online and in the training extract. A feature that takes an expensive query to compute may also be unsuitable for checkout, even if it is technically available.

Before accepting the model’s best feature, ask someone to produce its value for a new order using only information available at that instant. If they cannot, the promising score belongs to a different problem.