Implement a multi-channel long running Natural Interfaces framework with Swift and Apache OpenWhisk
Speak to a machine has always been one of the huge and passionate examples of how technology could innovate daily lives and as a cycling event now a lot of interest is again focused on things like personal assistant and natural interfaces.
Of course speech to text and speech recognition techniques are the basis for implementing this new kind of natural services but there is definitely more needed for easily implementing something that could be really usable for end users. This is specially true nowadays where speech and NLP technologies are becoming a commodity service available from multiple vendors and even capable to run on powerful mobile devices.
For understanding users intents, entities and strictly managing a dialog conversation there are several specialized solutions available on the market including IBM’s which include a very powerful runtime and web authoring tool with Watson Conversation that could help on managing a natural conversation flow between an automated service, the Bot, and the end user.
Anyway to really become useful a Bot first of all must be able to map user intents to specific actions such as a Restful API to call. Moreover for many real use cases it should support the capability to extend a classic quick request/response flow supporting a “long running conversation” scenario to deliver the right information to the user at the right time.
Long running conversation is basically the capability to extend the context of a conversation between a user and a Bot across different interactions, eventually even across different conversation channels, on a fully event based model. It basically provides a capability to implement a backend Bot logic on a totally asynchronous communication flow enabling Bots to reply and push to the user as soon as it has some interesting information and even eventually allowing the Bot re-initialize a new conversation directly from the backend.
First Things First: Multi-Channel Support
As said personal assistance and voice first solutions are becoming very popular nowadays and even in the context of human to human communications messenger like services such as Facebook Messenger are becoming one of the much used way to access the Internet. One billion people use just the Facebook Messenger every month and of course this is just one the available communication channels.
Most of this messenger channels are extending their capability nowadays allowing third parties to implement Chat Bot services able to interact with the user in a more natural and easy way than traditional Apps.
Corporations are struggling to deliver the right information to employees at the right time and are now watching at this Chat Bot opportunity as the best way to interact with their customers on the communication channels they already use.
Even on the App side there are lot of new opportunities available now with latest release of mobile operating system such as iOS 10 that allows to extend the traditional single task approach of mobile App enabling developers to develop application extensions that could easily offer their services in the context of infrastructure services such as Siri, Maps, iMessage etc.
Not to mention a huge trend in App User Experience in order to directly add Natural Interface capability within the App to simplify the interaction with the user and very important to allow the user use the service totally hand free.
The capability to support multiple channels, or messaging platforms, is therefor a mandatory requirement for a modern Bot and in general Natural Interfaces backend infrastructure that want to expose modern Internet services using this new modern way to interact with the user.
In this post I will guid you first through an open multi-channel bot framework I develop to easily allows the implementation of this basic functionalities and then in the second part we will talk about how to extend and use this framework with a powerful Event-Driven serverless infrastructure such as OpenWhisk to connect several IBM Watson powered services to different communication channels.
Kitura and Swift on Cloud/Server
Finally start talking about implementation and technicalities let me first introduce the Cloud, or in general Server, infrastructure platform I chose for implementing this multi-channel bot framework.
I chose to use the open source Swift language originally created by Apple for quickly and safely build iOS and Mac App but now available on Linux and powerful server containers infrastructure. In particular I used the open source IBM Kitura framework and implement on top of it a Web/Restful server infrastructure.
The reason why I chose Swift for server programming other than Node.js, Go or the many other options available are personally more than its amazing performance because I believe it is a much better modern language that support both traditional object oriented, imperative style, programming but also other useful pattern as functional and in general declarative programming style.
KituraBot, available open source at https://github.com/JacopoMangiavacchi/KituraBot, is basically a multi-channel Bot framework develop in Swift on top of the Kitura framework and it let developers very easily implement basic bot logic in a easy declarative way supporting a totally open and extensible architecture.
Indeed KituraBot is designed with ease of use but also with the capability to easily plugin particular components for implementing for example specific communication channels or message persistence and caching store.
KituraBot Swift Package in particular implements KituraBot structure and classes and define the KituraBotProtocol for implementing KituraBot channel specific webhooks and APIs. As said KituraBot architecture is designed to support multi-channel and it allows to plugin several channels on the same Kitura app implementing in a unique and central way the same Bot logic for different channels.
KituraBotFacebookMessenger) is a KituraBot plugin available for supporting Chat Bot on the Facebook Messenger channel.
KituraBotMobileAPIWithBluemixPush (https://github.com/JacopoMangiavacchi/ KituraBotMobileAPIWithBluemixPush) is a KituraBot plugin available for supporting a Mobile App Natural Interfaces channel. It basically allows a mobile app on iOS to send message through an exposed Restful API and basically receive response message from a Push Notification channel.
KituraBotClient (https://github.com/JacopoMangiavacchi/KituraBotClient) is a simple iOS application written in Swift that use Apple SFSpeechRecognizer and AVSpeechSynthesizer frameworks and implement a chat like UX to let the user interact with a Bot implemented server side on the KituraBot framework. It also support the SiriKit framework to let the user interact with the KituraBot framework directly from Siri.
Conversation Context Management
Conversation context management is a very critical part for implementing a bot service and in particular different channels has different way, or even in many case no way at all to deal with this. Generally speaking backend cognitive conversation service such as IBM Watson Conversation are totally stateless and let the client manage a user conversation context that should followup any inbound and outbound conversation flow between the Bot and the channel. A particular complexity here is that channels like for example Facebook Messenger do not provide any way to let this context flow to the real user and there fore the frontend that implement the Facebook Messenger webhook need to deal with context persistence and caching.
KituraBot manage context in a simple common but extendible way allowing the developer to implement it’s own context persistence object. The KituraBot package define the KituraBotMessageStoreProtocol for allowing developers to implement their own storage engine for dealing with conversation context persistence and caching. A very simple in process KituraBotMessageStore implementation is provided in the GIT repository above but an implementation based on a more scalable caching engine such as Redis.
The KituraBotMobileAPIWithBluemixPush instead let the conversation context flow end to end between the Bot logic and client App allowing the App to eventually extend the context providing
implicit context information such as location or in general any context that could be easily implicitly acquired by the mobile App from infrastructurere services such as Healthkit.
The diagram below illustrate the most relevant components and interfaces used by the KituraBot framework.
Sample Facebook Messenger Echo Bot
The code snippet below illustrate how easily is to build a simple synchronous Bot logic in Swift with the KituraBot framework following basically just three basic step:
In the first step it create a KituraMessageStore plugin. In the second step it initialize the KituraBot engine passing the main Kitura Router instance, the KituraMessageStore plugin and the Swift closure method that implement the synchronous Bot logic.
The KituraBot init accept also other two parameter for configuring some Restful API that it expose on the passed Kitura router to allow implementation of more complex asynchronous Bot logic. I will better talk about this later in this post.
The last set is about plugging in to the KituraBot instance some specific channel interfaces. In this simple code snippet a Facebook Messenger channel is added using the available KituraBotFacebookMessenger class.
The KituraBotFacebookMessenger init method receive specific Facebook Messenger parameter needed for configuring the Facebook webhook. Please see the standard Facebook documentation for configuring Facebook Messenger webhook and to get these parameters for your specific Facebook Messenger bot channel at https://developers.facebook.com/docs/ messenger-platform/implementation#subscribe_app_pages
KituraBot bot logic implemented in the second step on the KituraBot init closure is of course generic and used eventually even across different channel plugin added to the KituraBot.
The code snippet below for example illustrate as the third step could be extended to add more than one channel at the same KituraBot instance.
Long Running and Strongly Decoupling
The sample code provided above was just a super simple and totally unusable Bot that reply immediately to user request from different channels echoing the same request message. The code of course was provided just to introduce few basic concept about the more powerful KituraBot framework.
Indeed KituraBot support both a traditional synchronous model as well as a full asynchronous model. In the traditional synchronous model, as seen in the sample above, the Bot respond immediately in the context of the caller webhook HTTP request.
In the async model the webhook could call an event driven system on a backend such as IBM OpenWhisk to strongly decoupling the Bot implementation logic and implement a “Long Running Conversation” model.
Basically the asynchronous support allows the OpenWhisk backend logic to take all the necessary time to process a user request and respond to the user with the right channel at the right time. Very importantly it also allows the backend Bot logic to initialize a conversation directly from the backend side for example informing the user about some events.
In order to support this asynchronous support for long running conversation the KituraBot framework implement a dual interfaces. The first interface was already covered while talking about the simple synchronous scenario and it basically consist of the multi channel webhooks and in general APIs used in the communication between the KituraBot frontend and the different channels. As described in the diagram below the second interface is a specific Restful API that it exposed by the KituraBot frontend to allow the OpenWhisk backend Bot logic to send back, or push back, message from the Bot to the user across one of the configured channels.
OpenWhisk KituraBot front end sample code
In order to support this more powerful asynchronous model and strongly decoupling the Bot logic to actions managed by OpenWhisk the simple KituraBot frontend code provide above could be easily extended with few lines of code.
The code snippet below illustrate how easy it is to invoke OpenWhisk actions triggering a Triggers from any request coming from any channels plugged in to the KituraBot framework.
The JSON payload passed to the OpenWhisk trigger could be of course totally customized but basically the following are the information that must be passed from the KituraBot frontend to the Bot logic implemented in OpenWhisk:
- messageText: the original message coming from the user through one of the available channel
- context: a JSON object representing the conversation context
- userId: unique identifier of the user as passed from the caller channel
- channelName: the name provided for the channel receiving the message as defined in the bot.addChannel() call in the KituraBot frontend
- botUrl and securityToken: defined in the next code snippet are basically the path to the Send Rest API exposed by KituraBot for sending back, or pushing, a message back to the user
The following code snippet illustrates finally how to enable on the KituraBot frontend the synchronous model and basically expose on the KituraBot the Send Rest API. In particular the exposeAsyncPush method of the KituraBot class must be called to setup the path for this Rest API and a security token to verify any single call to this API. This security token and the path are basically the values for the botURL and securityToken fields of the JSON payload described just above.
The exposeAsyncPush also allow the KituraBot frontend to implement some specialization code when processing a request coming from OpenWhisk through this Sent Rest API. Basically the options here are to proceed with the same message and channel name coming from the OpenWhisk Bot logic or eventually allow the frontend to specialize the delivery of a modified message eventually trough another channel.
OpenWhisk Triggers and Actions Sequence
On the OpenWhisk side in order to connect all the dots and implement this time some real conversation logic there are few configuration steps to do and finally still few lines of code to write in order to invoke some Watson services for managing the dialog conversation with the user and also to send back the response message from Watson to the KituraBot multichannel frontend.
The first thing to do is to configure an OpenWhisk Trigger with the same name used above in the KituraBot frontend fireOpenWhiskTrigger method and then configure an OpenWhisk Rule that associate the trigger to some OpenWhisk actions.
OpenWhisk Sequence is an incredible feature that could be used here to split the entire Bot logic in different microservices, implemented as OpenWhisk actions, and let the OpenWhisk chain the calls between the different actions.
In the BoTson sample code available at https://github.com/JacopoMangiavacchi/BoTson there
are already available the source code of a couple of OpenWhisk actions to chain together in a OpenWhisk Sequence and connect to the OpenWhisk trigger in order to implement a full multichannel Kitura, OpenWhisk, Watson conversation demo.
- The ProcessNewMessageNode action basically forward the user request coming from the KituraBot frontend to the Watson Conversation service to understand the users intents, entities and to manage the dialog conversation. It receive back from Watson Conversation service a JSON payload that it basically pass as output of the action to the OpenWhisk even/driven engine
- The SendBackResponseToKituraBot action, connected to the previous action as said through an OpenWhisk Sequence, transform the response and send back the message text and the Watson Conversation context to the KituraBot frontend using the Send API described above
The following diagram illustrate how this conversation logic implementation between Watson and the KituraBot frontend orchestrated by OpenWhisk.
OpenWhisk Actions sample code
Watson Conversation Dialog management
Watson Conversation Service, available on IBM Bluemix cloud platform, integrate together in a single service NLC (Natural Language Classifier) capability and a Dialog management runtime.
If you are new to Watson Conversation please review this document to have a full understanding of their great capability for managing dialog conversation and understand user Intents and Entities: https://www.ibm.com/blogs/watson/2016/10/app-developers-guide- creating-first-watson-bot/
Here I just want to summarize that Watson Conversation Service provide also a very powerful web based authoring tool that allow to manage a rule graph for all the steps of a conversation allowing to easily configure an extensible JSON context payload to add to any single message exchange.
The diagram below illustrates just an example of a Dialog flow focusing on the capability to customize the message and the JSON context payload.
Extend the Bot logic
OpenWhisk is an amazing powerful tool for extending the capability of Watson Conversation and “react” to context information.
Indeed the simple OpenWhisk actions sequence chaining described before could be extended adding in a super easy and scalable way new specific actions to “understand” the returned JSON context payload and connect it to specific API calls or in general to integrate with SOR.
The illustration below describe a particular use case for a scenario quite popular nowadays in the News and Broadcast market. Many corporations in this market are struggling to deliver the right information to users at the right time and are considering emerging areas like Cognitive Computing where the dissemination of stories is vital to grow the market.
The diagram below illustrate a specific example of how according to this architecture OpenWhisk could be used for accessing different Watson and Bluemix services and easily implement such a multi channel News based Bot.
Basically the previous simple OpenWhisk Actions sequence could be extended in a totally transparent way for the KituraBot frontend adding a new action:
• The RetrieveDocument action could be added in a total transparent way for the previous ProcessNewMessage action and it will filter the response message and its relative context JSON payload coming from Watson Conversation before these data are sent to the SendBackResponseToKituraBot action. It will basically check the presence of particular tag in the JSON payload and, if present, it will search for the right document using for example the Watson Retrieve and Rank service. A link to the document will be transparently added to the response context and therefor passed to the SendBackResponseToKituraBot action for returning back the complete new response to the user.
Moreover the original ProcessNewMessage action could be easily extended, or yet again a new specific action could be added to the sequence, to store on a custom no-sql database such as Cloudant the specific request coming from the user building on the fly a sort of User Profile database about the most desired interesting news for any single user.
A periodic trigger, basically a cron, could then be configured on OpenWhisk to execute at determinate time a new chain of actions that will start with a new specific atcion:
• The RetrieveNewMessagePerUser action will retrieve for users that for example are not connecting to the Bot since a while some new document recently added that could be interesting for them according to the User Profile database on Cloudant This action therefor will select data from Cloudant and simulating a message user request from the client will connect to the previously described sequence of OpenWhisk actions to retrieve the document from the Watson Retrieve and Rank service and push a message to the user on the right channel.
Source: Chatbots Magazine