Regression Metrics: When MAE and RMSE Pick Different Models

Two models estimate delivery duration in minutes. In a tiny invented comparison, their absolute errors are:

Delivery	Model A	Model B
1	0 minutes	4 minutes
2	0 minutes	4 minutes
3	9 minutes	4 minutes

Model A is exactly right twice and substantially wrong once. Model B is moderately wrong every time. Which is better?

You cannot answer without deciding how to value the mistakes.

MAE prefers A; RMSE prefers B

Mean absolute error, MAE, averages the sizes of the errors. For A, it is (0 + 0 + 9) / 3 = 3 minutes. For B, it is 4 minutes. On this metric, A wins.

Root mean squared error, RMSE, squares the errors, averages them, then takes the square root. A scores approximately 5.20 minutes; B still scores 4 minutes. On this metric, B wins because squaring gives a large error disproportionate influence.

Here is the calculation in plain Python:

from math import sqrt

def metrics(errors):
    if not errors:
        raise ValueError("At least one error is required")
    mae = sum(abs(error) for error in errors) / len(errors)
    rmse = sqrt(sum(error ** 2 for error in errors) / len(errors))
    return round(mae, 2), round(rmse, 2)

print(metrics([0, 0, 9]))  # (3.0, 5.2)
print(metrics([4, 4, 4]))  # (4.0, 4.0)

Both metrics use the target’s units. Mean squared error, before taking the root, uses squared units and is therefore less direct to describe to someone thinking in minutes.

RMSE is not automatically superior because it reacts more strongly to large misses. If a large delay is particularly disruptive, that sensitivity may reflect something important. If extreme values are measurement errors, first investigate the data. Choosing MAE to conceal bad records or RMSE to appear rigorous solves neither problem.

The direction of an error can matter too

Absolute and squared errors treat equal-sized early and late estimates symmetrically. A delivery promised too early may produce a different cost from one promised too late.

If that distinction matters, report it directly: how often the promise was missed, by how much, and for which kinds of deliveries. A business-specific loss or a quantile prediction may be appropriate, but it needs a stated reason. MAE and RMSE remain useful summaries; they are not a complete account of the service.

R-squared compares against a reference

The usual R-squared calculation compares squared residuals with the variation around the observed target mean in the evaluation set. A perfect prediction scores 1. A score of 0 matches the squared-error performance of predicting that evaluation-set mean for every example. Scores can be negative when the predictions are worse than that reference.

Constant targets require special handling because the usual denominator is zero. The scikit-learn model-evaluation documentation describes these cases.

R-squared does not give the size of an error in minutes. It can also differ across datasets with different target variability, even when absolute errors look similar. Comparing it across unrelated problems is often misleading.

Finally, adding predictors cannot worsen the optimally fitted training sum of squared errors in ordinary, unregularized least squares with nested predictors and an intercept. That narrow property is sometimes misreported as “more features improve R-squared.” It says nothing about performance on new deliveries.

Choose the metric before repeatedly tuning against the results. Otherwise “the better model” can become whichever model looks best under the metric selected afterward.