Nebo · tutorial 08 · keep customers

Staff a support desk that knows when to stop

Easy questions answered from your own documentation, hard ones classified and routed, and nothing guessed.

TimeAbout an hour
Rolessupport-triage, knowledge-retrieval
Triggerwatch

The failure mode that ruins AI support is confident invention. A support employee that answers 90% correctly and fabricates the other 10% is worse than one that answers 60% and escalates the rest, because you cannot tell which is which until a customer acts on a wrong answer.

This recipe is built entirely around making it stop.

Steps

1

Give it a corpus before you give it a mailbox

Install knowledge-retrieval and point it at your help docs. An employee with no corpus has nothing to be faithful to, so it will improvise from general knowledge about products like yours.

Reads and writes are gated differently

Searching the knowledge base is ungated; adding to it or changing it is gated. That asymmetry is deliberate — you want retrieval to be frictionless and corpus changes to be deliberate, so nobody's mistaken answer quietly becomes next week's source of truth.

2

Make citation mandatory

Answer only from our documentation. Every factual claim in a reply must cite the document it came from. If the answer is not in the documentation, say "I don't have that documented" and escalate. Do not infer, do not reason from how similar products work, and never state a future release date.

Requiring a citation is what makes "I don't know" the easy path. Without it, the model has no cost for guessing.

3

Classify before answering

Have it set a type and urgency on every ticket, whether or not it answers. Even when it escalates, the classification saves your team the triage.

For each ticket: classify the type, set urgency, and identify the right owner. Then decide whether you can answer from documentation. Draft a reply only if you can.
4

Watch the queue rather than polling it

"trigger": { "type": "watch", "plugin": "...", "event": "message" }

A watch trigger streams changes, so a ticket at 2am is picked up at 2am rather than at the next poll. Use a schedule only if your helpdesk has no streaming integration.

5

Add quality review after a fortnight

Install live-chat-qa and have it sample conversations against a rubric. You are looking for answers that were confidently wrong, which is the category you cannot detect by reading customer satisfaction scores.

6

Set the gate

Stays human

Refunds and credits, always. Any promise about a future release date, always — which is best enforced as a persona refusal rather than a gate, since the damage is in the sentence rather than in an operation. Start with replies at approval for the first fortnight, then loosen once you have read enough of them.

Verify

  • Ask it five questions your docs genuinely do not answer, and confirm it escalates all five
  • Ask a question whose answer changed recently and confirm it uses the current doc
  • Confirm every reply cites something real
  • Send an angry ticket and confirm it stays professional and routes rather than placating

When it goes wrong

It answers confidently from nowhere

Citation is not being enforced. Make it structural: no citation, no send. If it cannot name a document, the reply is not valid.

It escalates everything

Your corpus is too thin or badly chunked. Fix the documentation; the behaviour is correct.

It cites the right doc but the wrong section

Chunking. Long pages need headings so retrieval lands on the relevant part.