Thresholds and calibration - Tacet

The API returns calibrated probabilities, so a confidence of 0.9 is right about nine times in ten on data like ours. There’s nothing to configure.

The act-or-defer pattern

route = result.answers["route"]

if route.confidence >= 0.90:
    send_to_team(route.top)   # confident: act now
else:
    ask_a_human(ticket)       # unsure: hand it off

Choosing a threshold

1

Collect a few hundred real examples with known answers.

2

Send them through the API and record each confidence and whether the top answer was right.

3

For each candidate threshold, count how many you’d automate and how accurate those would be.

4

Pick the lowest threshold that meets your accuracy target.

Recheck your threshold when your inputs change a lot, for example a new product line or a new language.