A case study on examining the purpose of entity fulfillment in conversational AI chatbots: when and why it is applied in the workflows for conversation designers, why the bot needs it to keep moving forward in the run-time conversation, and the appropriate interaction model in our UI. Let's deep dive into the foundational research that informed the UX and implementation of the entity fulfillment feature into a user-friendly and simplistic workflow in the [24]7.ai Conversations SAAS platform.
Preface
In this article, I will not be focusing on Entities. Entities play a huge role in info collection and confirmation, but this article will focus more on how we unified and reconciled the user interactions for designing entity fulfillment in our conversation design tool. You can learn more about Entities in my article: Advancing the AI Chatbot In Real-Time with Entities. This article focuses on definition, purpose, user objectives, and deep entity understanding. So there was preconceived knowledge of Entities prior to this next requirement/project. Our objective was to expand on and form-fit entities with the addition of entity fulfillment for the voice modality/channel within our new interaction paradigm.
Entity fulfillment definition
The collection of user information needed by the bot to proceed and advance a task or return results.
ROIs and KPIs
Current voice deployments that utilize entity fulfillment pre-conversation builder tool = 32 deployments amounting to $3.6M in revenue generation monthly, $51M annually. Vivid speech accounts for 2 of those 32 deployments.
Current digital deployments that utilize entity fulfillment pre-conversation builder tool = 5 deployments amounting to $92,525 in revenue generation monthly, $1.2M annually.
Tool adoption and client migration due to lack of feature parity and functional support = 0% adoption and migration.
Previous 'Time To Design' entity fulfillment conversations using cards and conversationally = 20 hours
Previous 'Time To Build' entity fulfillment conversations using cards and conversationally = 315 hours
Bot run-time scenario
In the chat example below, a user has told the AI chatbot that they would like to purchase a jacket. The chatbot then asks the user to provide product details (entity requirements) so that it can return the appropriate product results based on the criteria. The user responds with their desired jacket specifications (entity inputs). The bot would need to process each criteria (entity) individually and map it to the correct entities in the system. The bot would then confirm the entities received with a confirmation message or product results (entity outputs).
Collecting entities in a conversational manner
The bot asks the user a question one at a time to collect each entity. This can be the desired choice of collecting entities in both digital (typing) and voice (speaking), but most suitable for voice.
Collecting entities with a form card
The bot sends the user a form card to collect all entities at once. This is most optimal for the digital channel since it supports visual interfaces and typing. Some voice channels can't support visual interfaces (like pure telephone IVR's), so cards are completely out of the question. But most mobile phones can send a link to the user via text message and enable them to fill out a form in their native mobile phone's OS web browser application.
There are cases when entity fulfillment will need to be handled both conversationally and with a card in a single modal/channel.
In the digital experience, consider the case when a card is sent to a user and they ignore it and just start sending info as messages. The bot would need to pivot and accept this info without requiring the use of the card. In the voice channel, it could be that the bot is having trouble interpreting the entities vocally, so it will send the user a card to type the info instead.
Disambiguating and confirming the entities.
Once entity fulfillment has been completed in run-time, the bot needs to confirm the entities are correct with the user :
Disambiguate and confirm after each entity is given
Disambiguate and confirm after all entities are given
Design-time scenario
A conversation designer would need to design entity fulfillment experiences for both digital and voice channels concurrently in design-time. Each channel would require different and/or similar methods that are specific to the modal or channel. In a single journey, both channels can potentially have both ways to handle entity collection. A designer would need to define the entities needed per channel and method, construct the bot responses for each entity conversationally, and apply a card with the designated entity form collection fields, all while still designing a single conversational flow between the different channels and modals in a bot application.
Problem statements
Conversation designers currently don’t have a way to design the various methods and conversations to collect information from the user for both digital and speech modals in the Conversation Builder tool.
Conversation designers rely on developers to integrate entities into their bot application, requiring dependencies and extending the time to develop and design.
Conversation designers still create separate conversation design flows for each modal and channels since the journeys might be different based on the nuances of each.
User stories
As a conversation designer, I want to design a single conversation flow that remains the same between digital and voice; same amount of nodes and the same sequence of nodes, so that I can deploy to multiple channels and modals with a consistent experience.
As a conversation designer, I want to design entity fulfillment in a conversational manner only for both modals in a single conversation, so that I can collect info from the user allowing the bot to advance.
As a conversation designer, I want to design entity fulfillment using a card only for both modals in a single conversation, so that I can collect info from the user allowing the bot to advance.
As a conversation designer, I want to design entity fulfillment in a conversational manner for one modal and use a card for the other modal in a single conversation, so that I can collect info from the user allowing the bot to advance.
As a conversation designer, I want to design entity fulfillment in a conversational manner and use a card for both modals in a single conversation, so that I can collect info from the user allowing the bot to advance.
Mission
To ensure real-time bot advancement using entity fulfillment methods for both digital and voice modalities, integrate the design-time functionalities to support the design of these methods in the conversation design platform.
Design goals
Enable conversation designers to independently and quickly declare, manage, and use entity collection methods in their conversational design flows based on the modality or channel.
Enable conversation designers to see where entities are sequenced in their conversational design flow as separate nodes, since this would represent a step for the user in run-time.
Enable conversation designers to use a form card and/or handle entity fulfillment conversationally in each modal or channel.
Current design-time user journey
The designer collaborates with a client to establish the business rules in which will determine the entity requirements needed by the journey to solve a customer intent.
A list of entities are constructed and declared on the designer's conversational design flow. At this point, the designer is modal-agnostic or channel-agnostic, and just needs to list out the entities that may be applicable to all modals and channels.
The designer will then create the content for how to collect the entity in run-time. At this point, the designer will determine what best methods to use based on the content to collect the entity.
If the designer feels the content to collect the entity is best served by individual messages, they will determine the conversational method as the means to collect the entity.
If the designer feels the content to collect the entity is best served by using a form for typing, they will determine the card method as the means to collect the entity.
The designer will then construct the required bot responses: disambiguation, confirmation, no match, and no input for each entity fulfillment one at a time.
The designer will design additional nodes in their flow for holistic entity validation by using API nodes, subsequent confirmations by using regular bot responses, or variable sub-journeys based on the user's status/state/conditions by using logic nodes.
Once the entity fulfillment flow is complete, the designer will want to preview the bot run-time experience for each modal and channel.
Interaction paradigm explorations
Initial Concept
An Entity Fulfillment tab appears in every bot response node by default. When a user adds a card or chooses an entity to create conversational messages, and entity icon appears on the node in the flow diagram (displayed in the below screenshot next to the card icon in the 'Package_Track' node). In this UI model, a user would first choose the channel (digital or voice), then select the entity fulfillment method (card or conversation), and lastly, add the content (form fields or bot responses).
User selects to collect entities conversationally by constructing individual bot responses for each entity. This is most applicable for the Voice modality, but there are edge cases when an end-user ignores the card in run-time.
When a user clicks to add a bot response for each entity in the entity fulfillment table, a modal pop up appears with the bot responses edit UI.
User feedback
Denoting an entity is applied in the conversation flow appears indiscoverable and not enough emphasis and affordance/feedback making the entities feel buried inside of nodes. Users preferred to denote an entity has been applied as a node on the flow, representing the actuality of the run-time experience, since they are steps the user has to fulfill. This also helped identify there is entity fulfillment at a first glance and easier to parse.
The construct of an entity requires multiple layers of modal dialogs, making it feel convoluted and buried. Users preferred to expose entities on the flow, reducing layers of editing and feels more organized and manageable.
Entities should belong in the designated and initiated from the bot response node it is required and not separately. This is the natural mental model of the conversation designer. The current bot response should lead into entity collection or confirmation display.
Optimized exploration 1
A logic node can be applied to the flow diagram to divide separate channel journeys. The 'Package_Track' node leads into multiple children nodes that are specific to each modal or channel. This helped to display how journeys represented entity fulfillment on the flow diagram.
User feedback
This would require the user's time to individually construct responses and flow for each, making it more time-consuming and laborious.
Didn't satisfy the requirement to display entities at a glance and easily on the flow diagram.
There was no easy way to cohesively lead all journeys back into a single API validation node and close out the journey. Users had to build individual validation and wrap ups for each channel or modality.
Optimized exploration 2
Entities are its own node type displayed below with the purple borders, encapsulated by an entity frame. This interaction model was inherited from previous explorations the previous year and had been resurfaced.
User feedback
Users liked how easy it was to identify entities in the flow. They thought it accurately represented the real-time bot behavior and it visually helped to flow out their conversation.
Entities shouldn't be applied to the flow as separate nodes, they need to originate from the previous bot response node that is introduced and being collected confirmed, or displayed.
Grouping the individual entity nodes into a single Entity Fulfillment node enabled users to lead into a single API validation node and close out the journey with a single wrap up for all channels and modals.
Optimized exploration 3
An entity node is applied to the conversational flow diagram, and encapsulates all entities within its node. A user would still apply it, but it reduces the visual noise by only displaying a single node. A user would have to click on the node to view all entities individually.
User feedback
Users like how simple it looked but it still lacked displaying the array of entities in the flow diagram. Entities are still buried and this didn't truly represent the run-time experience.
What we learned
Individual entities must be displayed on the conversational flow diagram and denote whether it's a card or conversation.
Workflows for entities need to be simple and easy to organize and manage. The UI needed to alleviate multiple pop-ups and layers.
Entities should be initiated and originate from a conversation bot response node and not originate on its own, since it's being introduce and collected within a bot response.
Entity fulfillment journeys should be shared across modals and channels, helping the designer to design efficiently.
All entities should lead into a single API validation node and also share a single close out or wrap up, so users don't have to create these multiple times for each entity node.
Final solution
Entities are denoted in the original conversation bot response node signified below as an icon next to the card icon. The user declares the entities, method, and channels within that original node, then an 'Entity Fulfillment' node appears on the flow automatically displaying the entity method and each entity being used. Below a card has been chosen to collect entities.
In the screenshot below, the designer has chosen to handle entity collection through multiple bot responses (or conversationally). Comparing the screens above and below, you can get a sense of how we solved the representation of entity fulfillment while retaining the same sequence between channels with the same conversational flow.
In the Entity Fulfillment tab, a user can add a card. Our technical limitations had prevented us from enabling the user to create a shared list of entities between channels and methods. A card had to be produced in another internal tool where it also required the user to declare and create entities. In the screenshot below, a card has been applied with entities already. Since we didn't support cards for Voice in our first milestone, a card only represented the digital channel.
The Conversationally tab would allow the user to map entities to be used, which then required the user to construct individual bot responses for each entity. Displayed below, you can see the individual nodes in the Entity Fulfillment node on the conversational flow diagram.
When the user clicked on each bot response for an entity, the properties panel appears, displaying the UI to edit the bot response and all other responses for the entity capturing scenarios.
Summary
After reviewing the final designs with users, receiving consultations from executive stakeholders, and also receiving technical feedback from our development team, the UI got the thumbs up and sign off and proved to be a viable solution that solved all of our user problems. Users easily understood the mental model and interaction paradigm as it helped recognize entities on the flow and represented the run-time experience accurately. Users can build single entity fulfillment experiences consistently and quickly across channels and modals. The experience was intuitive and best represented our user's mental model.
About the author
Sean Lazo is a Principal UX Designer at [24]7.ai who leads the inception, assembly, and design of [24]7.ai Conversations, an industry-leading omni-channel AI chatbot SAAS platform. His passions are DesignOps, detailed design, research, and human relations.
Commentaires