Supervised Learning with Labeled Data: Who Decides the Correct Answer?

Two people read the same support ticket: “I have been charged twice, and nobody has replied for three days.” One labels it billing. The other labels it urgent.

Neither has necessarily made a mistake. They may be answering different questions.

If those labels enter a training dataset as competing categories, the resulting model inherits a confused task. More examples will not resolve the ambiguity unless the labeling process changes.

A target is a decision about the world

Supervised learning uses examples that pair inputs with target answers. For a ticket-routing model, the input could be the message and account information available at submission. The target might be the team that should receive it.

“Urgent” and “billing” belong to different dimensions: priority and subject. We could train separate predictors for them, or define a different output structure. The important work happens before choosing an algorithm.

A useful labeling guide would specify how to handle a ticket mentioning several problems, whether previous replies are allowed as context, and what to do when the correct destination is unclear. A small round of independent labeling can expose disagreements before they become thousands of inconsistent examples.

Agreement does not prove the labels are objectively correct. It does show whether people can apply the definition consistently enough to train and evaluate a system.

Classification and regression differ in the answer

Routing a ticket to a team is a classification problem: predict a category. Estimating how many hours resolution will take is a regression problem: predict a numerical quantity.

Both are supervised when we train with corresponding target outcomes. The distinction is about the target, not whether the input happens to contain numbers or text.

For resolution time, even an apparently objective label needs a definition. Does the clock stop at the first response, the agent’s resolution click, or the customer’s confirmation? Are weekends counted? Historical timestamps can provide precise measurements of a poorly chosen quantity.

Hold out the right examples

A sensible experiment separates training examples from evaluation examples. But a random row split is not always sufficient.

If the same long-running customer dispute produces ten related tickets, placing some in training and others in testing may overstate performance on a new customer. Keeping related tickets together can make the test more demanding and more relevant. If the intended use is next month’s traffic, a time-based split can reveal changes a random split obscures.

Inputs must also match the prediction moment. A ticket’s final resolution note may explain exactly which team handled it, but it is unavailable when the ticket first arrives. Including it would leak the answer. The same principle appears in features built from future information.

Read the disagreements

After training, inspect cases where prediction and label differ. Some will expose model weaknesses. Others will reveal ambiguous tickets, outdated routing rules, or incorrect labels.

This is not an excuse to rewrite every inconvenient test answer until the model looks good. Keep an untouched evaluation process, document genuine corrections, and resist adjusting definitions after seeing results just to improve a score.

The point is that a labeled dataset is a record of human and organizational decisions. It may encode past constraints as readily as timeless truth.

For the double-charge ticket, a useful system might recommend the billing team, flag urgency separately, and allow a person to override both. That design follows from understanding the job. The learning algorithm comes afterward.