Identify objections in customer conversations using Amazon Comprehend to enhance customer experience without ML expertise

According to a PWC report, 32% of retail customers disengage after a negative experience and 73% of customers say that customer experience influences their purchase decisions. In the global retail industry, pre- and post-sale support are important aspects of customer service. Numerous methods, including email, live chat, bots, and phone calls, are used to provide customer support. As conversational AI has improved in recent years, many companies have adopted cutting-edge technologies such as AI-based chatbots and AI-based agent support to improve customer service while increasing productivity and reducing costs .

Amazon Comprehend is a fully managed and continuously trained natural language processing (NLP) service that can extract information about the content of a document or text. In this post, we explore how the AWS Pro360 customer used Amazon Comprehend’s custom classification API, which allows you to easily build custom text classification models using enterprise-specific tags without requiring you to learn machine learning ( ML), to improve customer experience and reduce operating costs.

Pro360: Accurately spot customer objections to chatbots

Pro360 is a marketplace that aims to connect industry-specific talent specialists with potential clients, enabling them to find new opportunities and expand their professional network. It allows customers to communicate directly with experts and negotiate a customized price for their services based on their individual requirements. Pro360 charges clients when successful matches occur between specialists and clients.

Pro360 had to deal with a problem related to unreliable charges that led to consumer complaints and reduced trust with the brand. The problem was that it was difficult to understand the client’s objective during convoluted conversations full of multiple objectives, polite denials, and indirect communication. These conversations led to erroneous charges that reduced customer satisfaction. For example, a customer may start a conversation and immediately stop, or end up politely declining by saying “I’m busy” or “Let me cut it.” Also, due to cultural differences, some customers may not be used to expressing their intentions clearly, especially when they want to say “no”. This made it even more difficult.

To solve this problem, Pro360 initially added options and choices for the customer, such as “I’d like more information” or “No, I have other options.” Instead of writing their own question or query, the customer simply chooses the options provided. However, the problem was still not solved because customers preferred to speak clearly and in their own natural language while interacting with the system. Pro360 identified that the problem was a result of rule-based systems and that moving to an NLP-based solution would result in better understanding of customer intent and better customer satisfaction.

Custom classification is a feature of Amazon Comprehend, which allows you to develop your own classifiers using small datasets. Pro360 used this feature to create a model with 99.2% accuracy by training on 800 data points and testing on 300 data points. They followed a three-step approach to build and iterate the model to achieve the desired accuracy level of 82% to 99.3%. First, Pro360 defined two classes, reject and non-reject, that they wanted to use for classification. Second, they removed irrelevant emojis and symbols like now ~ i ... and identified negative emojis to improve model accuracy. Finally, they defined three additional content classifications to improve the misidentification rate, including conversations, ambiguous response, and rejection with a reason, to further iterate the model.

In this post, we share how Pro360 used Amazon Comprehend to track consumer objections during discussions and used a human-in-the-loop (HITL) mechanism to incorporate customer feedback into model improvement and accuracy, demonstrating ease of use and efficiency. from Amazon Understand.

“At first, I thought that implementing AI would be expensive. However, the discovery of Amazon Comprehend allows us to efficiently and economically take an NLP model from concept to implementation in just 1.5 months. We are grateful for the support provided by the AWS account team, solution architecture team, and ML experts from the SSO and service team.”

– LC Lee, founder and CEO of Pro360.

Solution overview

The diagram below illustrates the solution architecture covering real-time inference, feedback workflow, and human review workflow, and how these components contribute to the Amazon Comprehend training workflow .

In the following sections, we walk you through each step of the workflow.

Real-time text classification

To use Amazon Comprehend’s real-time custom classification, you must deploy an API as an entry point and call an Amazon Comprehend model to perform real-time text classification. The steps are as follows:

1. The client side calls Amazon API Gateway as an entry point to provide a client message as input.
2. API Gateway passes the request to AWS Lambda and calls the Amazon DynamoDB and Amazon Comprehend API in steps 3 and 4.
3. Lambda checks the current version of the Amazon Comprehend endpoint that stores data in DynamoDB and calls an Amazon Comprehend endpoint for real-time inference.
4. Lambda, with a built-in rule, checks the score to determine whether it is below the threshold or not. It then stores this data in DynamoDB and waits for human approval to confirm the evaluation result.

Feedback workflow

When the endpoint returns the classification result to the client side, the application asks the end user for a hint to get their feedback and stores the data in the database for the next round (the workflow of training). The steps for the feedback workflow are as follows:

1. The client side sends the user feedback by calling API Gateway.
2. API Gateway bypasses the request to Lambda. Lambda checks the format and stores it in DynamoDB.
3. Feedback from Lambda users is stored in DynamoDB and will be used for the next training process.

Human review workflow

The human review process helps us clarify data with a confidence score below the threshold. This data is valuable for improving the Amazon Comprehend model and is added to the next recycling iteration. We used Elastic Load Balancing as an entry point to carry out this process because the Pro360 system is built on Amazon Elastic Complute Cloud (Amazon EC2). The steps for this workflow are as follows:

1. We use an existing API in Elastic Load Balancer as an entry point.
2. We use Amazon EC2 as a compute resource to create a front-end dashboard so that the reviewer can flag input data with lower confidence scores.
3. After the reviewer identifies the objection from the input data, we store the result in a DynamoDB table.

Amazon Comprehend Training Workflow

To start training the Amazon Comprehend model, we need to prepare the training data. The following steps show you how to train the model:

1. We use AWS Glue to perform extract, transform, and load (ETL) tasks and combine data from two different DynamoDB tables and store it in Amazon Simple Storage Service (Amazon S3).
2. When the Amazon S3 training data is ready, we can fire up AWS Step Functions as an orchestration tool to run the training job and pass the S3 path to the Step Functions state machine.
3. We invoke a Lambda function to validate that the training data path exists, then fire an Amazon Comprehend training task.
4. After the training job starts, we use another Lambda function to check the status of the training job. If the training job is complete, we get the model metric and save it to DynamoDB for further evaluation.
5. We test the performance of the current model with a Lambda model selection function. If the current version performs better than the original, we deploy it to the Amazon Comprehend endpoint.
6. Next, we invoke another Lambda function to check the status of the endpoint. The function updates information to DynamoDB for real-time text classification when the endpoint is ready.

Summary and next steps

In this post, we showed how Amazon Comprehend enables Pro360 to build an AI-based application without ML experts, which is able to increase the accuracy of customer objection detection. Pro360 was able to build a custom NLP model in just 1.5 months and is now able to identify 90% of polite customer rejections and detect customer intent with 99.2% overall accuracy. This solution not only improves the customer experience, increasing retention rate growth by 28.5%, but also improves financial results, decreasing the cost of operation by 8% and reducing the workload of customer service agents.

However, identifying customer objections is only the first step in improving the customer experience. As we continue to iterate on the customer experience and accelerate revenue growth, the next step is to identify the reasons for customer objections, such as lack of interest, time issues, or the influence of others, and generate the right answer to increase sales conversion. fee

To use Amazon Comprehend to build custom text classification models, you can access the service through the AWS Management Console. For more information about using Amazon Comprehend, see the Amazon Comprehend Developer Resources.

About the Authors

Ray Wang is a solution architect at AWS. With 8 years of experience in the IT industry, Ray is dedicated to building modern cloud solutions, especially in NoSQL, big data and machine learning. As an avid lover, he passed all 12 AWS certifications to make his technical field not only deep but broad. He loves to read and watch sci-fi movies in his spare time.

Josie Cheng is an HKT AI/ML Go-To-Market on AWS. His current focus is on retail and CPG business transformation using data and ML to drive massive business growth. Prior to joining AWS, Josie worked for Amazon Retail and other Internet companies in China and the US as a growth product manager.

Shanna Chang is a solution architect at AWS. It focuses on observability in modern architectures and cloud-native monitoring solutions. Before joining AWS, she was a software engineer. In his spare time, he enjoys hiking and watching movies.

Wrick Talukdar is a senior architect on the Amazon Comprehend Service team. Work with AWS customers to help them adopt machine learning at scale. Outside of work, he enjoys reading and photography.