Data Cleaning for ML: The Rows You Delete Are a Decision

Suppose a delivery dataset contains these records:

Shipment	Duration	Unit	Status
A	45	minutes	Delivered
B	2	hours	Delivered
B	2	hours	Delivered
C	—	minutes	In transit
D	980	minutes	Delivered

This is an invented example. It looks like an easy cleaning exercise: convert units, remove the duplicate, fill the blank, and delete the outlier.

Only the first action is clearly justified by the table alone.

A duplicate according to whom?

The two B rows could be an accidental repeated export. They could also represent two parcels within the same shipment if the dataset has lost its parcel identifier.

Before dropping either row, establish the unit of observation. Is one row supposed to represent a shipment, a parcel, an attempt, or a status event? A uniqueness check is useful only after that definition exists.

If the source confirms a duplicated shipment record, removing the extra row is reasonable. Record the rule so the next export receives the same treatment. “Duplicates removed” is less useful than “one row per shipment ID, retaining the latest source revision.” The latter is a decision another person can examine.

A blank can describe the present

Shipment C has not arrived. Its duration is unknown because the event is unfinished, not because someone forgot to type a number.

Filling it with the average completed duration would invent an outcome. Dropping every unfinished delivery could bias a report toward quick deliveries, particularly if the export was taken shortly after orders were placed.

The appropriate handling depends on the question. A report on completed deliveries can exclude in-transit records while stating the population it covers. A study of delivery times may need a method that accounts for incomplete observation. An operational dashboard might show those shipments separately as still pending.

There is no universally correct replacement for a blank without knowing what the blank means.

An outlier might be the reason for the analysis

Shipment D took more than sixteen hours. Perhaps the source used seconds and labeled them minutes. Perhaps the package was actually delayed overnight.

The first possibility is a data error. The second may be the most important event in the table.

Deleting a legitimate delay because it is inconveniently far from the average can make a delivery service look more reliable than it is. Investigate extreme values against the source, then distinguish corrected errors from valid unusual observations. When uncertain, compare results with and without the questionable record and explain the sensitivity.

Keep the original and the reasoning

A useful cleaning process leaves three things behind: an unchanged source extract, reproducible transformations, and a short account of the decisions that required interpretation.

For this table, the account might say that hours were converted to minutes, B was deduplicated after checking the export, C remained pending, and D was retained after its timestamps were verified. Those are hypothetical findings, not facts we can infer from the table itself.

For predictive work, there is another boundary. If missing values are filled using a learned statistic such as the median, calculate it from the training data and reuse it on validation data. Otherwise, information from the evaluation set leaks into preparation. The scikit-learn guidance on common pitfalls explains this separation and the use of pipelines.

Cleaning makes data more suitable for a defined use. It does not make the underlying population fair, the measurements complete, or the business question sensible. A tidy table can still tell the wrong story—especially when nobody remembers which rows disappeared.