Define the action vocabulary
Start with a small set of mutually exclusive actions:Send the request
cURL
Map the result
Readoutput.choice and map it to an internal action:
Add a fallback
Choose a policy before shipping:- Send low-confidence decisions to human review.
- Add a
needs_reviewchoice if it is a real product outcome. - Retry only for transport failures or temporary
503responses, not for a valid low-confidence result. - Keep the original input, decision ID, model, and request ID in your audit record when your privacy policy permits it.
Evaluate before production
Build a test set from representative support requests and label the expected action. Track:- Accuracy by choice.
- Low-confidence rate.
- Confusion between similar choices.
- The percentage routed to human review.
- Changes after a model or catalog revision.