A leading American personal care consumer products company focused on innovation, worked with Valorem Reply to implement a chatbot based on Microsoft technology on their web application to improve the customer experience. Microsoft Bot Framework powered by Azure Cognitive Services was used to build a virtual assistant, to more effectively connect customers, information, products, and services in a conversational experience. In this case study, we explore how Valorem Reply’s testing team utilized Microsoft Power Automate to implement a low code test automation solution to quickly evaluate the core components such as Content Moderator, QnA, and LUIS of the conversational AI platform to improve the accuracy of bot responses and increase the speed to market by identifying bugs quickly and reducing the testing turnaround time.

CHALLENGE

• Complexity involved in the testing of an AI driven bot framework.
• Strategy required for continuous testing based on the adaptive learning capability of an intelligent bot.
• Effort required to maintain enormous amount of test data with uncertain user conversation trajectory .
• Tiresome, redundant process to validate each response of the chatbot manually
• Enormous regression test effort required to validate the response of the bot after each code change

SOLUTION

• Microsoft Power Automate flow to validate the Azure Content Moderator service by passing custom user inputs to validate the response and confirm whether the bot has the capability to identify and filter PII, profane/obscene terms.
• Automation script to connect with the Azure QnA service to evaluate the responses based on the outputs of two different knowledge bases.
• Build Sites in SharePoint Online and Teams, migrate content.
• Custom LUIS connected automation process to send more than 1700 utterances to evaluate the accuracy of response based on the top scoring intent.

RESULT

• Low code automation solution which can be maintained by quality analysts without the support of a quality engineer.
• Reusable automation scripts which can be customized and implemented across multiple environments with minimal effort.
• ~50% functional and ~88% regression test effort reduction with massive test coverage improvement.
• Ability to run around 2000 tests within 20-30 minutes, which can be expanded further .
• Data driven model with the provision to expand data set without modifying the automation flows.
• Quality Analysts can focus more on exploratory tests such as non-deterministic and erroneous scenarios to uncover concealed bugs.