Announcing the updated Microsoft OneDrive connector (V2) for Amazon Kendra

Amazon Kendra is an intelligent search service powered by machine learning (ML), enabling organizations to provide relevant information to customers and employees, when they need it.

Amazon Kendra uses ML algorithms to enable users to use natural language queries to search for information scattered across a company’s various data sources, including commonly used document storage systems such as Microsoft OneDrive.

OneDrive is an online cloud storage service that lets you host your content and automatically sync it across multiple devices. Amazon Kendra can index document formats such as Microsoft OneNote, HTML, PDF, Microsoft Word, Microsoft PowerPoint, Microsoft Excel, Rich Text, JSON, XML, CSV, XSLT, and plain text.

We’re excited to announce that we’ve updated the OneDrive for Amazon Kendra plugin to add even more capabilities. For example, we’ve added support for searching OneNote documents. Additionally, you can now choose to use identity information or ACLs to make your searches more granular.

The plugin helps index documents and their access control information to limit search results to only documents that the user can access. To display search results based on user access rights and using only user information, the plugin provides an identity tracker to load key information such as user and group maps to a main warehouse.

In this post, we demonstrate how to set up multiple data sources in Amazon Kendra to provide a central place to search your document repository.

Solution overview

For our solution, we demonstrate how to index a OneDrive repository or folder using the Amazon Kendra plugin for OneDrive. The solution consists of the following steps:

1. Create and configure an application in the Microsoft Azure Portal and obtain authentication credentials.
2. Create a OneDrive data source using the Amazon Kendra console.
3. Index the data in the OneDrive repository.
4. Run a sample query to get the information.
5. Filter the query by users or groups.

Prerequisites

To test the Amazon Kendra Plugin for OneDrive, you need the following:

Configure an Azure application and assign connection permissions

Before setting up the OneDrive data source, we need some details about the OneDrive repository. Complete the following steps:

1. Sign in to Azure.
   
2. After logging in with your account credentials, choose Application logsthen choose New registration.
   
3. Give your app a suitable name and register it.
4. Collect the information about customer ID, tenant ID and other application details.
   
5. To get a client secret, choose Add a certificate or secret under Customer Credentials.
6. choose New customer secret and provide appropriate description and expiration.
   
7. Note the client-id, tenant-id, and secret-id values. We use them to authenticate the OAuth2 application.
8. Navigate to Appchoose API permissions in the navigation panel and choose Add a permission.
   
9. choose Microsoft Graph.
   
10. Under Application permissionscome in Dossier in the search bar and in Files, select Files.Read.All.
11. choose Add permissions
    
12. Similarly, add the following permissions to the file Microsoft Graph option for the app you created:
    1. Group.Read.All
    2. Notes.Read.All

Upon completion, the API permissions will look like the screenshot below.

Set up the Amazon Kendra plugin for OneDrive

To configure the Amazon Kendra plugin, follow these steps:

1. In the Amazon Kendra console, choose Create Index.
   
2. For index name, enter a name for the index (for example my-onedrive-index).
3. Enter an optional description.
4. choose Create a new role.
5. For Role Nameenter an IAM role name.
6. Configure optional encryption options and tags
7. choose next
   
8. In the Configure user access control section, select Yes under Access control configurations
9. For Token write, choose JSON in the drop-down menu.
10. Leave the remaining values ​​as defaults.
11. choose next

Before moving on to the next configuration step, we need to provide Amazon Kendra with a role that has the necessary permissions to connect to the site. These include permission to obtain and decrypt the AWS Secrets Manager secret that contains the application ID and secret key needed to connect to the OneDrive site.

12. Open another tab for your AWS account, and in the IAM console, navigate to the role you created earlier (for example AmazonKendra-us-west-2-onedrive).
13. choose Add permissions i Create a policy online.
14. For servicechoose Kendra.
15. For Shareschoose write and specify BatchPutDocument.
16. For Resourceschoose All resources.
    
17. choose Review Policy.
18. For First nameenter a name (eg BatchPutPolicy).
    
19. choose Create a policy.
20. Add this policy to the role you created.
21. Also, attach the SecretsManagerReadWrite AWS managed the policy for the role
    
22. Return to the Amazon Kendra tab.
23. Select Developer Edition and choose To create.
    

This creates and propagates the IAM role and then creates the Amazon Kendra index, which can take up to 30 minutes.

25. Return to the Amazon Kendra console, choose Data sources in the navigation panel and choose Add a data source.
    
26. Under OneDrive Plugin V2.0choose Add a plugin.
    
27. For Name of the data sourceenter a name (eg my-onedrive).
28. Enter an optional description.
29. choose next.
    
30. For OneDrive tenant IDenter the tenant ID you collected earlier.
31. For Configure VPC and security groupleave the default value (No VPC).
32. keep Identity tracker is on selected This imports identity information into the index.
33. For the IAM role, choose Create a new role.
34. Enter a role name, such as AmazonKendra-us-west-2-onedrivethen choose next.
    
35. In the authentication section, choose Create and add a secret.
36. Create a secret with clientId i clientSecret as keys
37. Add their respective values ​​with the information you gathered earlier.
38. choose next.
39. In the Configure sync settings section, add the OneDrive users whose documents you want to index.
    
40. Select the synchronization mode for the index. For this post, we select Synchronization of new, modified or deleted content.
    
41. Choose the indexing frequency as Run on demandthen choose next

Field maps allow you to establish the searchability and relevance of fields. For example, the lastUpdatedAt The field can sort or increase the ranking of documents based on recent update.

42. Keep all default values ​​in the file Establish field maps section and choose next.
    
43. On the review page, choose Add a data source
    
    
44. choose Sync now

Synchronization may take up to 30 minutes to complete.

Try the solution

Now that you’ve indexed your OneDrive content, you can test it by checking the index.

1. Go to your index in the Amazon Kendra console and choose Search for indexed content in the navigation pane.
   
2. Enter a search term and press come in.
   

Note that without a token, ACLs prevent a search result from being returned.

3. expand Test query with an access token and choose Apply token.
   
4. Enter the appropriate token with a user who has permissions to read the file and choose To apply.
   
5. Search again for the information present in OneDrive.
   

You can verify that Amazon Kendra presents classified results as expected.

Congratulations, you’ve configured Amazon Kendra to index and search documents in OneDrive and control access to them using ACLs.

conclusion

With the Microsoft OneDrive V2 Plugin for Amazon Kendra, organizations can access commonly used business document stores, securely using intelligent search powered by Amazon Kendra. You can enhance the search experience by integrating the data source with the Custom Document Enrichment (CDE) capability in Amazon Kendra to perform additional attribute mapping logic and even custom content transformation during ingestion

About the authors

Pravinchandra Varma is a Senior Customer Delivery Architect with the AWS Professional Services team and is passionate about machine learning applications and AI services.

Supratim Barat is a software developer engineer with the AWS Kendra Yellowbadge Team and is a blockchain and cybersecurity enthusiast