When you’re developing a chatbot, the first time it understands you and responds correctly is like magic! And when it answers you perfectly dozens of times in a row, there’s a tendency to think that you’re ready to launch. But don’t! Launching a chatbot is not like launching a website.
On a website project, a small team can traverse the navigation, click all the links, try a few different browsers and be reasonably sure that it’s ready for launch.
On a chatbot project, it’s different. A small team can’t possibly anticipate all the questions a person could potentially ask the bot, or all the different ways a person could express themselves.
To ensure success with your chatbot, you need a slow-growing pilot. It should look something like this:
Initially, a small team of stakeholders and subject-matter experts establishes an initial body of questions and answers. Then they roll out the chatbot to a friendly pilot group of 10 or 20 people, and they watch the transcripts coming in from the chatbot conversations. They’ll see dozens of queries that weren’t anticipated or that the bot is misunderstanding. They’ll teach the bot to properly handle those queries, and then grow the pilot to another 10 to 20 people. They’ll watch again as the transcripts come in, teach the bot more, and keep on growing and growing.
As the pilot grows, the percentage of successful responses climbs higher and higher. Once the success rate is in the 90–95% range, the company is ready to launch the chatbot to the entire audience. But they need to reach that point by growing the pilot gradually. Training the chatbot will continue after launch, of course; but it’s the pilot period that gets the chatbot to a reasonable level of performance.