Overfitting vs. Underfitting: Reading Training and Validation Scores

Three decision trees predict whether a support ticket needs escalation. One is allowed only a single split. Another has moderate depth. The third can keep splitting until it closely fits the training examples.

You would expect their training scores to differ. The interesting question is what happens on tickets they have not seen.

The following is a schematic comparison, not an experiment or a benchmark:

Pattern	Training performance	Validation performance	First question
A	Poor	Poor	Can the representation and model express the relationship?
B	Good	Similar, with some gap	Does this hold on a realistic, untouched test?
C	Excellent	Much worse	Is the model fitting sample-specific details?

Pattern C suggests overfitting. Pattern A suggests underfitting or another limitation of the setup. Neither diagnosis should be made from a score alone.

In short, underfitting is a failure to capture enough of the useful relationship, often visible as weak training and validation performance. Overfitting is a failure to generalize from the training examples, often visible as strong training performance and much weaker validation performance. These score patterns are clues, not diagnoses by themselves.

Overfitting is a relationship, not a model name

A deep tree can memorize combinations that happen to occur in a small training set. Those combinations may not recur in new tickets. Its flexibility has captured details that do not generalize.

But “deep tree” and “overfitted” are not synonyms. Whether capacity becomes a problem depends on the data, task, training procedure, and evaluation. A small model can overfit a tiny or repeatedly consulted dataset too.

Regularization, limiting tree depth, or stopping training earlier can help control this behavior. More representative data can help. None guarantees success if the labels are inconsistent or the evaluation split is misleading.

A bad score can have more than one cause

If both training and validation performance are poor, increasing capacity is one hypothesis. The inputs may also lack the information needed to predict escalation. Perhaps the decisive fact appears only in a phone conversation that the dataset never records.

If training performance is strong but performance on later tickets falls, inspect changes in the environment as well. A new support policy or a newly launched product can shift the task. A large gap between old training data and new traffic is not proof that reducing model complexity will solve the problem.

Check data leakage too. A suspiciously good validation score may mean the model received information unavailable at prediction time. That is a different failure from ordinary overfitting, even if both lead to disappointment in production.

Do not turn the validation set into training data by habit

Validation guides choices: features, model settings, thresholds. After enough rounds of choosing whatever scores best on the same examples, the whole development process can adapt to that set.

Cross-validation provides repeated train/validation splits and can make estimates less dependent on a single split. It is an evaluation method, not a cure that prevents overfitting. Learned preprocessing must still be fitted inside each training fold. See the scikit-learn guidance on leakage.

Keep a final test for assessing the selected approach, and make its construction match the intended use. For future tickets, time matters. For unseen customers, customer overlap matters.

A useful next experiment changes one plausible cause. If a smaller tree improves validation performance while losing some training accuracy, that supports a capacity-related diagnosis. If every tree fails on a newly introduced ticket category, investigate the category and training coverage.

The aim is not to make two scores identical. It is to understand what the gap says about the next prediction the system will make.