Leverage Cognitive AI Services with Sitefinity

Cognitive capabilities enable a wealth of options for Web Content Manager (WCM) users, and can be applied in many directions beyond analyzing large amounts of data like in Sitefinity DEC. An effective and intelligent content creation experience includes much more than just an optimized UI, even a personalized one. With the help of democratized AI services we can now boost the work of authors and marketers so they can more easily publish best in class content.

Digital assets like images are essential for many content creators, and AI can be used to automatically analyze an image and quickly generate metadata like tags and alternative text, as well as check for inappropriate content. Cognitive services can also directly help with content analysis and content generation, providing sentiment, hashtags, categories and autogenerated summaries. Together, these services can boost the quality of the content and save significant amounts of time. Below I will demonstrate how this can be achieved in Sitefinity.

TL;DR

If you are not a tech person, feel free to skip ahead to the “Image Processing” and “Text Analysis” sections.

Sitefinity Cognitive Sample Projects

In GitHub you will find three folders containing the source code – the Sitefinity Cognitive Module project, a blank Sitefinity project referencing the module to help you package it as an add-on, and the developer SDK for the new admin app. Read below for more details about what each of them is doing.

Cognitive Module

The Cognitive Module handles the image processing and exposes web services which analyze the content items.

Under the hood the Cognitive Module hooks onto the upload events, gets the image and sends it over to Microsoft AI service (Azure Computer Vision API), which provides the metadata. In addition, in the CognitiveImageProcessor implementation, the image is checked for adult content first. If you go deeper you will find that barebones HTTP calls are made to the AI endpoints.

In Web->Services->Controllers you will find the exposed services analyzing the content items. To analyze the text, we bring the power of Natural Language Processing, in our case Aylien .NET SDK.

Admin App Extension SDK

The Admin App sample shows how to plug a button into the new UI and analyze the text by calling the webservices from the Cognitive Module. In this article you can learn how to extend the Sitefinity Admin App. Get the generated extension bundle and place it in the AdminApp folder of your Sitefinity website.

Sitefinity Web App

The Sitefinity Web App is used to ease the development of this functionality by giving you the ability to distribute the cognitive capabilities with an add-on. Build the solution and then pack an add-on for distribution. In this project there is already a .nuspec file in “SitefinityCognitiveServices\SitefinityWebApp\App_Data\Sitefinity\Export\ CognitiveServices”. The first step is to generate an add-on from it. If you open the .nuspec file you will notice that it references two other packages  - "Telerik.Sitefinity.Core” and "Aylien.TextApi" - and the binary file of the Cognitive Module. Also, we added a reference to the generated extension file to distribute it with the add-on. Keep in mind that you can safely remove it and play with just the generated file from the Admin App Extension project.

When the add-on (nuget) is installed on a Sitefinity website, the Aylien nuget will reference an older version of Newtonsoft (8.x), and once the package is installed you will need to update the Newtonsoft reference back to 10.0.3.

Configuration

Install the add-on through the package console to your website and go retrieve the API keys and endpoints from Microsoft Cognitive Services and Aylien. The last step is to open Administration->Settings->Advanced->CognitiveServices and enter them there. Now, you are ready to generate best in class content with the help of AI.

Distribution

For your convenience we distribute a NuGet in the release tab, so you can use it right away into your Sitefinity website, without downloading and setting up all of the above project.

Image Processing

Once the add-on is installed the CognitiveServices module will send every image to the Microsoft Vision service, check for inappropriate content and generate metadata for you. Here is how a sample image upload works.

The metadata in the green boxes is generated automatically by the AI. Keep in mind that the service that you are using should be trained in the vertical of your business to be more accurate.

Text Analysis

Once the new Admin App extension is plugged into your website and the CognitiveService add-on is installed, a green button will appear on the right side of the editor of your content item.

Clicking “Analyze Text” generates all the metadata above it automatically, instantly giving you intelligent suggestions.

Conclusion

Leveraging all of those cognitive services will definitely ease the work of your content creators and marketers. Sitefinity is easily extensible, so you can choose any AI service you like to use in this way. The above examples could be modified to do things like add the categories from the suggestion list with click of a button.

To see all this in a demo you can watch our Sitefinity 11.0 webinar. Furthermore, I would be more than happy to learn how you leverage the powerful capabilities of cognitive services in your own solutions, so please share your feedback in the comments below and enjoy the new Sitefinity Cognitive Services release.