Choose a Loss Before a More Complex Model

In this chapter

• Compute mean absolute and mean squared error.
• Explain why outliers influence losses differently.
• Separate a training objective from held-out usefulness.

Establish an honest starting point

A regression model outputs a number, but the loss determines what kinds of mistakes matter during fitting and evaluation. For equipment turnaround, a two-hour miss may be inconvenient while a twelve-hour miss can undermine planning. Mean absolute error averages the magnitude of each miss and retains the target's units. Mean squared error squares the misses before averaging, giving large errors more influence and producing squared units. Neither metric automatically encodes the coordinator's actual decision cost.

Begin with a constant prediction learned only from training labels. The training mean minimizes squared error among constant predictions; a training median minimizes absolute error. These are strong diagnostic baselines because their behavior is transparent. If an elaborate model cannot improve on the appropriate baseline in the later validation period, examine data quality and feature availability before adding complexity. A lower training loss alone is not evidence of a better deployed service.

Work through an outlier by hand

The four synthetic turnaround values include one long case. The mean prediction moves toward it more than the median does. The snippet compares both predictions on the same training values only to illustrate the objective. It prints the absolute and squared errors; do not report these numbers as held-out performance.

from statistics import mean, median

hours = [2.0, 3.0, 4.0, 15.0]

def mae(actual, prediction):
    return mean(abs(value - prediction) for value in actual)

def mse(actual, prediction):
    return mean((value - prediction) ** 2 for value in actual)

mean_guess = mean(hours)
median_guess = median(hours)
assert mean_guess == 6.0 and median_guess == 3.5
assert mae(hours, median_guess) == 3.5
assert mae(hours, mean_guess) == 4.5
assert mse(hours, mean_guess) < mse(hours, median_guess)
for name, guess in (("mean baseline", mean_guess), ("median baseline", median_guess)):
    print({"model": name, "prediction_hours": guess, "mae_hours": mae(hours, guess), "mse_hours_squared": mse(hours, guess)})

Inspect residuals, not just one score

The failure mode is choosing a model from a single aggregate metric while ignoring where it fails. A model could improve average error yet consistently underestimate long repairs. Record residuals as prediction minus actual, group them by equipment family and time period, and inspect both large underestimates and overestimates. Use groups that describe the equipment process rather than personal characteristics of staff. Include counts so a group containing two cases does not appear as reliable as one containing hundreds.

As a documented extension, compare the baseline with a scikit-learn regression pipeline fitted within the same split rules. Evaluate on original hour units even if training uses a transformed target. Check whether back-transformation changes the meaning of the estimated quantity. A useful model card states the chosen loss, baseline, sample sizes and decision limitations; it does not claim the metric is a universal measure of intelligence or business value.

Build a baseline comparison sheet

• Create a separate later-period list of synthetic hours and evaluate both frozen training predictions on it.
• Add a case with a much longer turnaround and observe its effect on both losses.
• Write down whether underestimation and overestimation have equal planning costs for the intended review workflow.

Expected checks

• Validation labels never recompute the constant predictions.
• MAE is reported in hours and MSE in squared hours.
• The report includes large residual examples and the constant baseline.

Check your understanding

Why can a single long turnaround pull the mean baseline away from the typical short case?

• Squared-error optimization gives a large deviation substantial influence.
• The mean automatically removes outliers.
• The median always predicts the same value as the mean.

Official tools & further reading

• scikit-learn regression evaluation reference