Microsoft Chat Bot


Introduction

Recent trends show that chatbots are essential everywhere, whether it’s an entertainment, medical, or knowledge-based website. Since this is an emerging technology, developers all around the world will be wanting to know more about this, specifically, what is happening in the background of chat bot applications. In this article we are developing the simple chatbot application using ASP.Net Core 3.1 with Core Bot Template.

Note : We have published the detail description of Chatbot conversational AI as a free E-book last week ( March 19 / 2021 ). We have added the link of the e book in the reference section please read before dive into this article.

Bot Framework

The Bot Framework have a capability to build a highly standard and rich enterprise conversational AI experience to the customers or end users. Also Bot Framework is a highly branded virtual assistant development framework or chatbot development framework by Microsoft. So we can build a chat bot with all kind of AI feature like Cognitive Services, QnA Maker, Luis, etc. There are many confusion on which situation we should use all this AI features in a chatbot development. So going forward we will understand in which situation this feature are more helpful for the chat bot conversational flow.

As of now by default Visual Studio do not contain any Bot Framework in template section. The SDK is available for Bot Framework template integration in Visual Studio, Please click here to download and install latest SDK for Bot framework in Visual Studio.

Azure Bot Service

First of all don’t be confused on Bot Framework and Azure Bot Service, As we discussed earlier Bot Framework SDK includes lot of open source SDK for building conversational AI.

Azure Bot Service is a platform provided by Microsoft to create and manage the chatbot application in Azure cloud environment using a single service or an endpoint call. There are many channels integrations available in Azure Bot Service for the interactive communication in chatbot application. The available channels are MS Teams, Slack, Messengers, etc.

Conversation Work flow

The below diagram is one of the example of chat bot conversational flow which means we can consider any type of flow as per our business requirement and this is one of the POC of conversational flow. The most of the Microsoft bot will use the same pattern for chatbot communication . Now we can dive into each segments that we have used in the following diagram.

Azure Subscription

A valid Azure subscription key is required for the creation of any of the services in the Azure portal. We can create an Azure subscription in multiple ways, either a free or paid service.

Please click here to create free Azure subscription -> Azure Free Subscription

Active Directory

There are many world-wide users communicating with the chatbot application, sometimes this will create a security attack and it create major problems in all the applications not only in chatbot. The AD or Active Directory is an enterprise identity service provides single sign-on ( SSO ) and multi-factor authentication to help protect your users from 99.9 percent of cybersecurity attacks. Also this is more helpful for managing an user in an organization with the help of azure environment.

Note: The security is more concern about all customers in world-wide and also expecting a highly availability so this we can achieve in azure environment without any single doubt.

App Registration

Within Azure Active Directory we can find out App registration, This will be more helpful to assign a large number of permission into the registered application inside the AD. I think everyone are familiar about Graph API, using this API we can give two levels of permissions ( Delegated & Application Level ) in a registered application in azure app registration pane. When you are developing a chatbot for internal organization or private communication inside the organization then those kind of scenarios we can handle through Graph API user level permission. Apart from that we can prevent the unauthorized user access and implement a role based ( Graph API can provide a complete user details in a AD) communication & permission in a chatbot application.

Note : As we have discussed earlier we need to create an Azure Bot Service in Azure environment , This will automatically register chatbot application in an App registration section included Bot channel registration. Please check our previous article for Web App Bot Creation in Azure.

Core Bot Template

We are using VS 2019 and Installed Bot Framework SDK V4 in our system environment. As you can see this chatbot application developing top of .Net Core 3.1 using C# with all AI features. By default it will provide LUIS ( Language Understanding Intelligent Service) code in the Core Bot template.

Visual Studio 2019 -> Bot Template

Create Web App Bot

Once after creation of Core Bot Template then we can host the application in azure environment with the help of Azure Bot Service. Please check our previous article for Web App Bot Creation in Azure.

After the creation of Web App Bot we can host our chatbot application in Azure environment using Visual Studio publish option. The successful deployment will give you the following result in “Test in Web Chat”. Just click on “Test in Web Chat” menu pane to open web chat channel for the chatbot testing.

Channel Configuration

Channel is nothing but it’s like a bridge to communicate between user and the bot application !!

As per the definition of Microsoft Docs :

“The Bot Service provides protocols and API’s through which you can connect your Bot to the services where your customers are. A channel represents the service you wish to reach your users on, such as Skype, Microsoft Teams or Facebook.”

These are the current available channels in Azure Bot Service.

Channel Configuration

We have already learned about how to configure slack channel in Azure Bot Service Slack Channel Configuration. Now we can learn the most popular “MS Team” channel with in simple steps.

First click on the Channels menu pain in our Bot service app and click on the “MS Team” icon.

Then select the respective radio button based on the business requirement and click on save.

Within a second MS Team channel configured in our Azure Bot Service with “Running” status.

Click on the MS Team icon and open the application in either browser or in MS Team application in your system. If the bot is not replaying with the welcome message then just try to enter one message.We are doing this for just testing the bot application in ms team channel.

Note : The best way to host the ms team channel application in an organization level is you need to create a bot app in app studio editor part of ms team application.

OUTPUT

Middleware

In Asp.Net Core, Middleware is software that is assembled into an app pipeline to handle requests and responses. So here we are registering all the chatbot related services like Storage, IBot, User State, Conversation State and other services as well.

The below example is state that “IBot” is inherited to “DialogAndWelcomeBot” and again this internally inherited into “ActivityHandler” and many more , we need to resolve this dependency using the middleware registration. The Ibot is the initial triggering point of chatbot application.

Example :

services.AddTransient<IBot, DialogAndWelcomeBot<MainDialog>>();

Storage

Managing the state is the vital role of communication in all kind of application not only chatbot, So how we can store the state in chatbot application, It’s pretty simple because Bot framework is provided by default package libraries for storing the state in Azure or InMemory.The Inmemory storage is basically using for testing purpose so when you are working on production then better to go with highly recommended Azure Cosmos Db. Why it’s highly recommended because Azure Cosmos Db have millisecond response times, and automatic and instant scalability, guarantee speed at any scale. Also highly available with enterprise-grade security. These are more important when you are deploying the chatbot applications for world wide usage. We can also use Azure Table Storage for the state management.

In Memory :

 // Create the storage we'll be using for User and Conversation state. (Memory is great for testing purposes.)
 services.AddSingleton<IStorage, MemoryStorage>();

Cosmos Db :

services.AddSingleton<IStorage>(c => new CosmosDbPartitionedStorage(new CosmosDbPartitionedStorageOptions()
            {
                AuthKey = Configuration["CosmosAuthenticationKey"],
                ContainerId = Configuration["ConfigCollectionName"],
                DatabaseId = Configuration["CosmosDbName"],
                CosmosDbEndpoint = Configuration["CosmosEndpoint"],

            }));

QnA Maker

QnA Maker is a prebuilt AI by Microsoft, It provide a Knowledge Base with Question and Answer Pairs. In chatbot application we can consider QnA maker as a database or a Knowledgebase to answer all kind of questions that we have already added in the QnA Maker Knowledgebase. These are predefined questions & answers adding by users themselves.

For Eg, In an organization if you have multiple department and you want to assure that the selected department ( UI menu selection for asking respective question) answers are properly getting as response through chatbot. These scenario we can achieve in many ways, One is we need to maintain multiple Knowledgebase in QnA Maker for each departments. Achieve this scenario we required LUIS for intent ( like category ) identification or department identification. Here intent is considered as department and inside the intent we can add utterances or questions for the department identification. Based on the intent we can deviate the respective Knowledgebase call in Qna Maker. another way to achieve this scenario is Luis dispatcher so please refer the E-book in reference section.

Language Understanding Intelligent Service ( LUIS )

LUIS is a prebuilt AI by Microsoft, Why we are using LUIS in a chatbot application that we have discussed in QnA maker section. Luis will identify the department or intent if organization have multiple department scenario in an application, this will give a clear picture at the end for which knowledgebase need to be connect inside the Qna maker. Here intent we can consider as department and inside the intent we can add utterances or questions for the department identification. So please refer the E-book in reference section for more detail about LUIS.

QnA Maker With Empty Response

If Qna Maker didn’t answer the question asked by user in that we need to handle in a tricky way either we need to display generic messages like

“Sorry, We are unable to answer your question and could you please rephrase it”

Another way is connecting to Bing Search API, Google Search API, etc for un answered question from qna Maker but this is not a good practice for domain bot application for an organization.

Live Agent Connect / HandOff:

Sometimes user want to connect a human agent or live agent, These are our findings and implementation details.

  1. As per the github documentation from Microsoft, I understood that there is an experimental package available for handoff in C# code ( Microsoft Handoff library ). As of now this library is not recommended for production use and please check this experimental docs reference for more info and below comments in the github account :

The samples in this folder should not be used in a production environment. They are not supported and the team is not implying a given approach used in these samples will be integrated into a future version of the Bot Framework SDK. Instead, we want to provide a way to engage on topics that can help guide our roadmap for future work.

  1. This is the best open source code for live agent ( handoff library ) that I found from github. The author of the code is an approved microsoft open source contributor and he has developed an agent hub routing + multiple channel connectivity in the bot. This will handle multiple user request to an agent with the help of agent hub. The agent can open a communication channel for user connectivity. You will get a better understanding once you go through the documents.
  2. This is the one I have implemented reference, This will be a simple redirection to an agent from an user with his comments or issues through microsoft team channel group. I have implemented one way communication but at least you can inform to the group ( created for agent team ) for user issues with his account details in the channel in ms team. So any of the agent in the team can personally contact him through ms team account and resolve the issues.

Reference

Summary

From this article we have learned the important background process and implementation of chatbot application.

My First E-Book


My first book ( E-Book ) !!.

The Implementation & handling of multiple Microsoft chatbot modules is based on my work experience of rigorous 8 years on the Microsoft Stack included chatbots. During my IT days, I am passionately involved and dedicated my good time on this subject matter and it happened to be my first book ( E-Book ) in C# Corner .

How a person feel gratitude when there is so many people responsible for greatfulness. Respected Mr.Praveen, Mr.Mahesh and C# Corner Team for laying out a wonderful platform for the technical contributions, I am indebted to my most respected mentor Mr.Tadit dash for his immense contributions.

Unquestionably there is a chronicle behind every successful achievements, Two years back I was planning to write a book related to automation in C# Corner community but that plan was dropped due to the suggestion from dear friend Priyaranjan, as my functional area is based on latest technologies and must not shift to automation, his suggestions was very valid. The reason I planned to write a book in latest technologies, but it got delayed due to covid pandemic and obviously some of the office workloads.

Also many of the famous publishers reached out me to write books in upcoming technologies and I discussed several topics with most admired Mr.Gaurav but at the end I had a different thought in my mind, Nine-Tenths of wisdom is being wise in time so was need to publish the book completely free for the user and I remember seeking knowledge is like opening doors and I know doors everywhere.

Thank you all of my friends & family who has supported my ups and downs in my life.

E-Book : THE IMPLEMENTATION & HANDLING OF MULTIPLE MICROSOFT CHATBOT MODULES

My 5th C-SharpCorner MVP Award


Finally I have received my 5th consecutive C-SharpCorner mvp award 🏆( 2015 to 2020 ). Always my sincere gratitude to taditdash who has introduced me to this wonderful community in 2014 and Thank you praveen sir , dinesh ir, Atul Gupta sir , mahesh sir for your greatest support. Thank you all my readers and supporters 🙏 Stay Safe and Stay Home 🏠. #mvp #csharpcorner #award

MVP Award 2020

Get started with Blazor Apps


Introduction

We have developed many Single Page Application using Angular,  Aurelia, React, etc. Finally the microsoft has introduced Blazor Apps and It is a programming framework to build client side rich web application with dotnet using c#. What ? Is it possible to create a Single Page Application using C# without using JavaScript ? Yes we can create a rich interactive User Interface ( UIs ) using C# instead of JavaScript and also the client and server code logic has written in the app using .Net. The Innovations are the major concerns of every era, So as a dotnet developer we can develop a Single Page Application ( SPA ) using Microsoft C#.

Blazor app have five different editions and two are now available in the Visual Studio and three editions are in a planing stage.

  • Blazor WebAssembly
  • Blazor Server
  • Blazor Progressive Web Apps (PWAs)
  • Blazor Hybrid
  • Blazor Native

The following are the currently available Blazor apps.

Blazor WebAssembly

Blazor WebAssembly is a single-page app framework for building interactive client-side web apps with .NET ( dotnet ). Blazor is running client-side in the browser on WebAssembly and the client side code has been written in C# instead of using javascript. So the .Net code is running inside the browser with the help of Webassembly(abbreviated wasm). Yes , It will work all the modern web browsers and including mobile browsers. There’s no .NET server-side dependency but the app is fully functional  after downloading the .Net runtime to the client. This will create a client-side dependency and Serverless deployment scenarios are possible because ASP.NET Core web server doesn’t required to host the app.

As per the Microsoft information “Blazor WebAssembly” is still in preview and expect to live by may 2020. So it is clear that Blazor WebAssembly is not ready for the production use and currently in a development stage.  If you’re looking for a production solution, then you can choose “Blazor Server” and it is recommended by Microsoft.

Blazor Server

Blazor Server provides support for hosting Razor components on the server in an ASP.NET Core app that means Blazor can run your client logic on the server.  The UI updates, event handling, and JavaScript calls are handled over with the help of “SignalR” ( a real-time messaging framework. ) connection. The download size is smaller than Blazor Server Comparing to Blazor WebAssembly app because those are handling in server side and the app loads much faster. Serverless deployment is not possible because an ASP.NET Core server is required to serve the app.

As daniel mentioned in the document “We have expect to have official previews of support for Blazor PWAs and Blazor Hybrid apps using Electron in the .NET 5 time frame (Nov 2020). There isn’t a road map for Blazor Native support yet. The following are the Blazor App planning editions”.

Blazor PWAs ( Progressive Web Apps )

Blazor PWAs ( Progressive Web Apps  ) are web apps that support the latest web standards to provide a more native-like experience. It will work offline and online scenarios and support push notifications, OS integrations, etc.

Blazor Hybrid

Native .Net readers to Electron and Blazor Hybrid apps don’t run on WebAssembly but instead uses a native .NET runtime like .NET Core or Xamarin. It will work offline or online scenarios.

Blazor Native

Same programming model but rendering non-HTML UI.

Note :  The application is tested in .NET Core 3.1 SDK & Visual Studio 2019 16.4 version and all the steps depend on the respective versions.

Prerequisites

  1. Install Visual Studio 2019 16.4 or later with the ASP.NET and web development workload.
  2. Install the .NET Core 3.1 SDK.

1.  Create a new project in Visual Studio 2019 ( version 16.4 ) and It lists the available project templates. So we have to  choose “Blazor App” template for development.

Create a new project

Create a new project

2. Configure new project name, solution name and location in our system.

Configure new project

3.  Based on the selected Blazor template ( Step 1 ) It will display the two available Blazor app in Visual Studio. So we have selected “Blazor Server App” from the list.

Output

We can run the application and see the first output of our Blazor Server App.

Reference

Summary

From this article we have learned the basics of microsoft Blazor Apps With Visual Studio 2019. I hope this article is useful for all the Blazor ASP.NET Core beginners.

C# Corner One Million Readers Club


Thank you all my readers !!! Thanks for the gift Atul Gupta , Praveen Kumar and C# corner team. #MVP #CsharpCorner

Getting Started With Angular Routing Using Angular CLI – Part Three


Introduction

In this article we are going to learn the basic routing concepts in Angular 6 using Angular CLI( Command Line Interface ). So before reading this article, you must read the following articles for more knowledge.

Routing in Application

In every application we do a minimal number of route concepts. For Eg While clicking the menu page it will redirect into child page or different page,etc. How can we do this in Angular ? Yes, There are few basic stuff we need to configure before the routing implementation in Angular.

Creating a Routing Module

By using command line parameter "--routing" we’re able to create a routing module in the Angular application. So by using any of the following command we can create routing module in Angular with the default routing setup. We have given “app-routing” as our routing module name.

ng generate module app-routing --flat --module=app
ng g m app-routing --flat --module=app

Note :

--flat puts the file in src/app instead of its own folder.
--module=app tells the CLI to register it in the imports array of the AppModule

The following code is generated after the above command execution.

import { NgModule } from '@angular/core';
import { CommonModule } from '@angular/common';

@NgModule({
  imports: [
    CommonModule
  ],
  declarations: []
})
export class AppRoutingModule { }

Import Router in the Application

Angular Router is an optional service and it is not a part of "@angular/core" and Router Modules are the part of "@angular/router" library package.  So we need to import “RouterModule” and “Routes” from "@angular/router" library package.

import { RouterModule , Routes } from '@angular/router';

Add Routes

A very simple explanation in Angular docs is “Routes” tell the router which view to display when a user clicks a link or pastes a URL into the browser address bar. So the scenario is whether a click or paste a URL into the browser !!

const routes: Routes = [{}];

We have created a empty routes in our routing module, Now we need to add redirect page, default page , 404 page , etc. For this just type “a-path” inside the “{}” and It will display the possible routing options in the routing module.

Now we have added path and component name in the Routes.

const routes: Routes = [{ path: 'customer', component: CustomerComponent }];

We already know “app component” is our default launching  page in Angular application. So we need to setup a routing configuration in the app component.

RouterModule.forRoot()

We first initialize and import the router setup and it start listening the browser location changes.Whatever routes we have mentioned earlier will be injected into the forRoot().

@NgModule({
  exports: [RouterModule] ,
  imports: [
    CommonModule, [RouterModule.forRoot(routes)]
  ],
  declarations: []
})

RouterModule.forChild()

forChild() is used for the submodules and lazy loaded submodules as the following way.

@NgModule({
  exports: [RouterModule] ,
  imports: [
    CommonModule, [RouterModule.forChild(routes)]
  ],
  declarations: []
})

Router outlet

RouterOutlet  acts as a placeholder and it looks like a component. So when you place this outlet in an app component then it will be dynamically filled based on the current router state.

<router-outlet></router-outlet>

Navigation

The navigation we have added in the same app component page and while clicking on the “/customer” string content in “routerLink” then  It will redirect into the respective page. We can add more functionality in the anchor tag like active,binding array of links,etc

The Router Output

 

Page Not Found – 404 Error !!

If user is trying to access different page in the application and that is not part of the routing configuration then we need to display an Error message page called as Page Not Found. Any of the following command will create a “Page Not Found” page with using inline template.

ng generate component PageNotFound --inline-template
ng g c PageNotFound -t

We have modified the PageNotFound typescript file.

If you want to add bootstrap style in the application then import the following reference link in the style.css.

@import url('https://unpkg.com/bootstrap@3.3.7/dist/css/bootstrap.min.css');

We have updated the router URL as “PageNotFoundComponent” with path mentioned as “**” in the last route is a wildcard. if the requested URL doesn’t match any paths for routes defined in the following configuration. This will help us to displaying a “404 – Not Found” page or redirecting to another route.

const routes: Routes = [{ path: '', component: EmployeeComponent },
{ path: 'customer', component: CustomerComponent },
{ path: '**', component: PageNotFoundComponent }];

Output

Download :

Reference

Summary

From this article we have learned the basic routing concepts of  Angular CLI. I hope this article is useful for all the Angular CLI beginners.

Getting started with Angular 6 using Angular CLI – Part 1


Introduction

Today we are going to learn most popular Single Page Application called as Angular Js using Angular CLI ( Command Line Interface ). We know by using Angular we can build modern applications such as web, mobile, or desktop in the real world. Previously we learnt about another famous single page application called as Aurelia.

Prerequisites

  • Download and Install : Node.Js ( Angular requires Node.js version 8.x or 10.x. )
  • Download and Install : VS Code ( A best opensource editor for developing Angular Application,etc ).
  • Download and Install : Git ( Not mandatory but use for a best practice ).

Angular CLI Installation

We can install Angular Command Line Interface after the installation of “node js” in our environment.  This will help you to create projects, generate application and library code, and perform such variety of ongoing development tasks such as testing, bundling, and deployment in our application. So this will reduce the time of development activities because of command line Interface register and generate files automatically at the beginning stage.  So that is why we need a version control for newly changed item in the application and it will identify  in which files what types of changes are reflected. This is more helpful for the beginners in   Angular CLI application development. Now we can install the Angular CLI globally by using the following command in our application.

npm install -g @angular/cli

Create a Project

Now we are going to create a simple project repository in our environment. The repository I have created in Github account and it is been cloned  in my environment(machine). The best practice of Angular CLI or any other application is configuring a version control in our VS Code otherwise we blindly create and check-in the code in project repository. No issues we can continue without configuring a version control in our application.

Go to our repository or  the place where we are planning to create a project setup , open the command prompt and run the following command.

ng new angular-app

Eg : “ng new [ProjectName]” our project name is a “angular-app”

Angular CLI created “angular-app” in our repository.

Install Angular Essential Extension

Click on Visual Studio Code Extension menu in left side bar then search “Angular Essential”  (john papa) in the search box. Once we install Angular Essential,  it will install all other supporting packages in the application automatically. Also It will generate a different icons for all the folders,ts files,style,json,etc.

Angular Build

We are generated a dummy application in our repository. The next step will be to build our application. For that open a “Command Terminal” in Visual Studio Code.

  1. Click on Visual Studio “Terminal” menu from the top of the menu list.
  2. “Terminal” menu displays a list of options ,  just click “New Terminal (  Cntrl + Shift + ~ )”.

There is one more short key for opening “Terminal” in VS Code ( “Cntrl + ~ ” ) . We can see the following terminal displayed on the right side bottom of  VS Code.

Now we need to build our application for that we need to open root directory of the application. May be when you open the Terminal in the VS code it will display the repository path of our application. So just change to application path by the following way.

The build artifacts will be stored in the dist/ directory folder in the application. Now run the Angular CLI build Command.

ng build

If you are getting the following error then that means we need to install “@angular-devkit/build-angular” dev dependency. This package is newly introduced in Angular 6.0.

The following command will help us to create devkit dependency in our application.

npm install --save-dev @angular-devkit/build-angular

If you are facing this same issue after the installation of dev kit then you need to uninstall and install the angular cli.

App Component

Components are the basic UI building block of an Angular app. So here we can see there is a “app” component generated under the “src -> app” folder in “angular-app” application. If you are using Angular CLI then it will auto generate all the files that are relevant for your basic application. For example in the following screen shot app folder contain such auto generated files are css,spec,ts and module.

Angular Serve

Now build is succeeded, Yes our application is ready to serve. Please run any of the following command ( one is alias or a short command ) to open our application in a browser.

ng serve --open
ng s -o

or If we don’t want to open application in a browser then just run the following command and navigate to "http://localhost:4200/".

ng serve

Bundling the application

So we can bundle our application using any of the following command and flag “–prod” bundle for production.

ng build --prod
ng serve --prod

for more options click here.

Changing the default port number

Every application should navigate to "http://localhost:4200/" as default. So If you are really thinking to open an application in a different port address then it is possible. Just run any of the following command.

ng s --port 3000 --open
ng s --port 3000 -o

Output :

As I mentioned earlier the default port we have changed to “3000” instead of “4200”.

Download :

Reference

Summary

From this article we have learned the basic configuration of Angular 6 using Angular CLI and few basic CLI commands. I hope this article is useful for all the Angular CLI beginners.

Cognitive Services – Optical Character Recognition (OCR) from an image using Computer Vision API And C#


Introduction

In our previous article we learned how to Analyze an Image Using Computer Vision API With ASP.Net Core & C#. In this article we are going to learn how to extract printed text also known as optical character recognition (OCR) from an image using one of the important Cognitive Services API called as Computer Vision API. So we need a valid subscription key for accessing this feature in an image.

Optical Character Recognition (OCR)

Optical Character Recognition (OCR) detects text in an image and extracts the recognized characters into a machine-usable character stream.

Prerequisites

  1. Subscription key ( Azure Portal ).
  2. Visual Studio 2015 or 2017

Subscription Key Free Trail

If you don’t have Microsoft Azure Subscription and want to test the Computer Vision API because it requires a valid Subscription key for processing the image information. Don’t worry !! Microsoft gives a 7 day trial Subscription Key ( Click here ). We can use that Subscription key for testing purposes. If you sign up using the Computer Vision free trial, then your subscription keys are valid for the westcentral region (https://westcentralus.api.cognitive.microsoft.com ).

Requirements

These are the major requirements mentioned in the Microsoft docs.

  1. Supported input methods: Raw image binary in the form of an application/octet stream or image URL.
  2. Supported image formats: JPEG, PNG, GIF, BMP.
  3. Image file size: Less than 4 MB.
  4. Image dimension: Greater than 50 x 50 pixels.

Computer Vision API

First, we need to log into the Azure Portal with our Azure credentials. Then we need to create an Azure Computer Vision Subscription Key in the Azure portal.

Click on “Create a resource” on the left side menu and it will open an “Azure Marketplace”. There, we can see the list of services. Click “AI + Machine Learning” then click on the “Computer Vision”.

Provision a Computer Vision Subscription Key

After clicking the “Computer Vision”, It will open another section. There, we need to provide the basic information about Computer Vision API.

Name : Name of the Computer Vision API ( Eg. OCRApp ).

Subscription : We can select our Azure subscription for Computer Vision API creation.

Location : We can select our location of resource group. The best thing is we can choose a location closest to our customer.

Pricing tier : Select an appropriate pricing tier for our requirement.

Resource group : We can create a new resource group or choose from an existing one.

Now click on the “OCRApp” in dashboard page and it will redirect to the details page of OCRApp ( “Overview” ). Here, we can see the Manage Key ( Subscription key details ) & Endpoint details. Click on the Show access keys links and it will redirect to another page.

We can use any of the subscription keys or regenerate the given key for getting image information using Computer Vision API.

 

Endpoint

As we mentioned above the location is the same for all the free trial Subscription Keys. In Azure we can choose available locations while creating a Computer Vision API. We have used the following endpoint in our code.

https://westus.api.cognitive.microsoft.com/vision/v1.0/ocr

View Model

The following model will contain the API image response information.

using System.Collections.Generic;

namespace OCRApp.Models
{
    public class Word
    {
        public string boundingBox { get; set; }
        public string text { get; set; }
    }

    public class Line
    {
        public string boundingBox { get; set; }
        public List<Word> words { get; set; }
    }

    public class Region
    {
        public string boundingBox { get; set; }
        public List<Line> lines { get; set; }
    }

    public class ImageInfoViewModel
    {
        public string language { get; set; }
        public string orientation { get; set; }
        public int textAngle { get; set; }
        public List<Region> regions { get; set; }
    }
}

Request URL

We can add additional parameters or request parameters ( optional ) in our API “endPoint” and it will provide more information for the given image.

https://[location].api.cognitive.microsoft.com/vision/v1.0/ocr[?language][&detectOrientation ]

Request parameters

These are the following optional parameters available in computer vision API.

  1. language
  2. detectOrientation

language

The service will detect 26 languages of the text in the image and It will contain “unk” as the default value. That means the service will auto detect the language of the text in the image.

The following are the supported language mention in the Microsoft API documentation.

  1. unk (AutoDetect)
  2. en (English)
  3. zh-Hans (ChineseSimplified)
  4. zh-Hant (ChineseTraditional)
  5. cs (Czech)
  6. da (Danish)
  7. nl (Dutch)
  8. fi (Finnish)
  9. fr (French)
  10. de (German)
  11. el (Greek)
  12. hu (Hungarian)
  13. it (Italian)
  14. ja (Japanese)
  15. ko (Korean)
  16. nb (Norwegian)
  17. pl (Polish)
  18. pt (Portuguese,
  19. ru (Russian)
  20. es (Spanish)
  21. sv (Swedish)
  22. tr (Turkish)
  23. ar (Arabic)
  24. ro (Romanian)
  25. sr-Cyrl (SerbianCyrillic)
  26. sr-Latn (SerbianLatin)
  27. sk (Slovak)

detectOrientation

This will detect the text orientation in the image, for this feature we need to add detectOrientation=true in the service url  or Request url as we discussed earlier.

Vision API Service

The following code will process and generate image information using Computer Vision API and its response is mapped into the “ImageInfoViewModel”. We can add the valid Computer Vision API Subscription Key into the following code.

using Newtonsoft.Json;
using OCRApp.Models;
using System;
using System.Collections.Generic;
using System.IO;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;

namespace OCRApp.Business_Layer
{
    public class VisionApiService
    {
        // Replace <Subscription Key> with your valid subscription key.
        const string subscriptionKey = "<Subscription Key>";

        // You must use the same region in your REST call as you used to
        // get your subscription keys. The paid subscription keys you will get
        // it from microsoft azure portal.
        // Free trial subscription keys are generated in the westcentralus region.
        // If you use a free trial subscription key, you shouldn't need to change
        // this region.
        const string endPoint =
            "https://westus.api.cognitive.microsoft.com/vision/v1.0/ocr";

        /// 
<summary>
        /// Gets the text visible in the specified image file by using
        /// the Computer Vision REST API.
        /// </summary>

        public async Task<string> MakeOCRRequest()
        {
            string imageFilePath = @"C:\Users\rajeesh.raveendran\Desktop\bill.jpg";
            var errors = new List<string>();
            string extractedResult = "";
            ImageInfoViewModel responeData = new ImageInfoViewModel();

            try
            {
                HttpClient client = new HttpClient();

                // Request headers.
                client.DefaultRequestHeaders.Add(
                    "Ocp-Apim-Subscription-Key", subscriptionKey);

                // Request parameters.
                string requestParameters = "language=unk&detectOrientation=true";

                // Assemble the URI for the REST API Call.
                string uri = endPoint + "?" + requestParameters;

                HttpResponseMessage response;

                // Request body. Posts a locally stored JPEG image.
                byte[] byteData = GetImageAsByteArray(imageFilePath);

                using (ByteArrayContent content = new ByteArrayContent(byteData))
                {
                    // This example uses content type "application/octet-stream".
                    // The other content types you can use are "application/json"
                    // and "multipart/form-data".
                    content.Headers.ContentType =
                        new MediaTypeHeaderValue("application/octet-stream");

                    // Make the REST API call.
                    response = await client.PostAsync(uri, content);
                }

                // Get the JSON response.
                string result = await response.Content.ReadAsStringAsync();

                //If it is success it will execute further process.
                if (response.IsSuccessStatusCode)
                {
                    // The JSON response mapped into respective view model.
                    responeData = JsonConvert.DeserializeObject<ImageInfoViewModel>(result,
                        new JsonSerializerSettings
                        {
                            NullValueHandling = NullValueHandling.Include,
                            Error = delegate (object sender, Newtonsoft.Json.Serialization.ErrorEventArgs earg)
                            {
                                errors.Add(earg.ErrorContext.Member.ToString());
                                earg.ErrorContext.Handled = true;
                            }
                        }
                    );

                    var linesCount = responeData.regions[0].lines.Count;
                    for (int i = 0; i < linesCount; i++)
                    {
                        var wordsCount = responeData.regions[0].lines[i].words.Count;
                        for (int j = 0; j < wordsCount; j++)
                        {
                            //Appending all the lines content into one.
                            extractedResult += responeData.regions[0].lines[i].words[j].text + " ";
                        }
                        extractedResult += Environment.NewLine;
                    }

                }
            }
            catch (Exception e)
            {
                Console.WriteLine("\n" + e.Message);
            }
            return extractedResult;
        }

        /// 
<summary>
        /// Returns the contents of the specified file as a byte array.
        /// </summary>

        /// <param name="imageFilePath">The image file to read.</param>
        /// <returns>The byte array of the image data.</returns>
        static byte[] GetImageAsByteArray(string imageFilePath)
        {
            using (FileStream fileStream =
                new FileStream(imageFilePath, FileMode.Open, FileAccess.Read))
            {
                BinaryReader binaryReader = new BinaryReader(fileStream);
                return binaryReader.ReadBytes((int)fileStream.Length);
            }
        }
    }

}

API Response – Based on the given Image

The successful json response.

{
  "language": "en",
  "orientation": "Up",
  "textAngle": 0,
  "regions": [
    {
      "boundingBox": "306,69,292,206",
      "lines": [
        {
          "boundingBox": "306,69,292,24",
          "words": [
            {
              "boundingBox": "306,69,17,19",
              "text": "\"I"
            },
            {
              "boundingBox": "332,69,45,19",
              "text": "Will"
            },
            {
              "boundingBox": "385,69,88,24",
              "text": "Always"
            },
            {
              "boundingBox": "482,69,94,19",
              "text": "Choose"
            },
            {
              "boundingBox": "585,74,13,14",
              "text": "a"
            }
          ]
        },
        {
          "boundingBox": "329,100,246,24",
          "words": [
            {
              "boundingBox": "329,100,56,24",
              "text": "Lazy"
            },
            {
              "boundingBox": "394,100,85,19",
              "text": "Person"
            },
            {
              "boundingBox": "488,100,24,19",
              "text": "to"
            },
            {
              "boundingBox": "521,100,32,19",
              "text": "Do"
            },
            {
              "boundingBox": "562,105,13,14",
              "text": "a"
            }
          ]
        },
        {
          "boundingBox": "310,131,284,19",
          "words": [
            {
              "boundingBox": "310,131,95,19",
              "text": "Difficult"
            },
            {
              "boundingBox": "412,131,182,19",
              "text": "Job....Because"
            }
          ]
        },
        {
          "boundingBox": "326,162,252,24",
          "words": [
            {
              "boundingBox": "326,162,31,19",
              "text": "He"
            },
            {
              "boundingBox": "365,162,44,19",
              "text": "Will"
            },
            {
              "boundingBox": "420,162,52,19",
              "text": "Find"
            },
            {
              "boundingBox": "481,167,28,14",
              "text": "an"
            },
            {
              "boundingBox": "520,162,58,24",
              "text": "Easy"
            }
          ]
        },
        {
          "boundingBox": "366,193,170,24",
          "words": [
            {
              "boundingBox": "366,193,52,24",
              "text": "way"
            },
            {
              "boundingBox": "426,193,24,19",
              "text": "to"
            },
            {
              "boundingBox": "459,193,33,19",
              "text": "Do"
            },
            {
              "boundingBox": "501,193,35,19",
              "text": "It!\""
            }
          ]
        },
        {
          "boundingBox": "462,256,117,19",
          "words": [
            {
              "boundingBox": "462,256,37,19",
              "text": "Bill"
            },
            {
              "boundingBox": "509,256,70,19",
              "text": "Gates"
            }
          ]
        }
      ]
    }
  ]
}

Download

Output

Optical Character Recognition (OCR) from an image using Computer Vision API.

Reference

See Also

You can download other ASP.NET Core source codes from MSDN Code, using the link, mentioned below.

Summary

From this article we have learned Optical Character Recognition (OCR) from an image using One of the important Cognitive Services API ( Computer Vision API ). I hope this article is useful for all Azure Cognitive Services API beginners.

 

C# Corner MVP 2018 !! 3 years of glory !! Hatrick 🏆


Hatrick !! Yes three in a row , It is a great privilege to my professional career as well a great motivation through out my upcoming years. I believe that one day each award I receive will mark my identity in the industry.I would like to extend my gratitude to my family and friends..Tadit , Sayed , Priyan , Ronen, C# Corner Team,etc are my greatest inspiration to write articles in technical community.

Thank you for all support 🙏

Three years of glory : 2015-16 , 2016-17 , 2017-18

Thank you Stratis for your awesome Tshirt & Power bank.

Fabulous Tshirts…!!

Cognitive Services : Analyze an Image Using Computer Vision API With ASP.Net Core & C#


Introduction

One of the important Cognitive Services API is Computer Vision API and it helps to access the advanced algorithms for processing images and returning valuable information. For example By uploading an image or specifying an image URL, Microsoft Computer Vision algorithms can analyze visual content in different ways based on inputs and user choices. So we will get various information about the given image. We need a valid subscription key for accessing this feature.

Prerequisites

  1. Subscription key ( Azure Portal ).
  2. Visual Studio 2015 or 2017

Subscription Key Free Trail

If you don’t have Microsoft Azure Subscription and want to test the Computer Vision API because it requires a valid Subscription key for processing the image information. Don’t worry !! Microsoft gives a 7 day’s trail Subscription Key ( Click here ) . We can use that Subscription key for testing purpose. If you sign up using the Computer Vision free trial, Then your subscription keys are valid for the westcentral region ( https://westcentralus.api.cognitive.microsoft.com )

Requirements

These are the major requirements mention in the Microsoft docs.

  1. Supported input methods: Raw image binary in the form of an application/octet stream or image URL.
  2. Supported image formats: JPEG, PNG, GIF, BMP.
  3. Image file size: Less than 4 MB.
  4. Image dimension: Greater than 50 x 50 pixels.

Computer Vision API

First, we need to log into the Azure Portal with our Azure credentials. Then we need to create an Azure Computer Vision Subscription Key in the Azure portal.

Click on “Create a resource” on the left side menu and it will open an “Azure Marketplace”. There, we can see the list of services. Click “AI + Machine Learning” then click on the “Computer Vision”.

Provision a Computer Vision Subscription Key

After clicking the “Computer Vision”, it will open another section. There, we need to provide the basic information about Computer Vision API.

Name : Name of the Computer Vision API.

Subscription : We can select our Azure subscription for Computer Vision API creation.

Location : We can select our location of resource group. The best thing is we can choose a location closest to our customer.

Pricing tier : Select an appropriate pricing tier for our requirement.

Resource group : We can create a new resource group or choose from an existing one.

Now click on the MenothVision in dashboard page and it will redirect to the details page of MenothVision ( “Overview” ). Here, we can see the Manage Key ( Subscription key details ) & Endpoint details. Click on the Show access keys links and it will redirect to another page.

We can use any of the Subscription key or Regenerate the given key for getting image information using Computer Vision API.

Endpoint

As we mentioned above the location is same for all the free trail Subscription Key. In Azure we can choose available locations while creating a Computer Vision API. The following Endpoint we have used in our code.

https://westus.api.cognitive.microsoft.com/vision/v1.0/analyze

View Model

The following model will contain the API image response information.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

namespace VisionApiDemo.Models
{
public class Detail
{
public List<object> celebrities { get; set; }
}

public class Category
{
public string name { get; set; }
public double score { get; set; }
public Detail detail { get; set; }
}

public class Caption
{
public string text { get; set; }
public double confidence { get; set; }
}

public class Description
{
public List<string> tags { get; set; }
public List<Caption> captions { get; set; }
}

public class Color
{
public string dominantColorForeground { get; set; }
public string dominantColorBackground { get; set; }
public List<string> dominantColors { get; set; }
public string accentColor { get; set; }
public bool isBwImg { get; set; }
}

public class Metadata
{
public int height { get; set; }
public int width { get; set; }
public string format { get; set; }
}

public class ImageInfoViewModel
{
public List<Category> categories { get; set; }
public Description description { get; set; }
public Color color { get; set; }
public string requestId { get; set; }
public Metadata metadata { get; set; }
}
}

Request URL

We can add additional parameters or request parameters ( optional ) in our API “endPoint” and it will provide more information for the given image.

https://%5Blocation%5D.api.cognitive.microsoft.com/vision/v1.0/analyze%5B?visualFeatures%5D%5B&details%5D%5B&language%5D

Request parameters

Currently we can use 3 optional parameters.

  1. visualFeatures
  2. details
  3. language

visualFeatures

The name itself clearly mentions it returns Visual Features of the given image. If we add multiple values in a visualFeatures parameters then put a comma for each value. The following are the visualFeatures parameters in API.

  • Categories
  • Tags
  • Description
  • Faces
  • ImageType
  • Color
  • Adult

details

This parameter will return domain specific information whether it is Celebrities or Landmarks.

Celebrities : If the detected image is of a celebrity it identify the same.

Landmarks : If the detected image is of a landmark it identify the same.

language

The service will return recognition results in specified language. Default language is english.

Supported languages.

  • en – English, Default.
  • zh – Simplified Chinese

Vision API Service

The following code will process and generate image information using Computer Vision API and its response is mapped into the “ImageInfoViewModel”. We can add the valid Computer Vision API Subscription Key into the following code.

using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.IO;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;
using VisionApiDemo.Models;

namespace VisionApiDemo.Business_Layer
{
public class VisionApiService
{
const string subscriptionKey = "<Enter your subscriptionKey>";
const string endPoint =
"https://westus.api.cognitive.microsoft.com/vision/v1.0/analyze";

public async Task<ImageInfoViewModel> MakeAnalysisRequest()
{
string imageFilePath = @"C:\Users\Rajeesh.raveendran\Desktop\Rajeesh.jpg";
var errors = new List<string>();
ImageInfoViewModel responeData = new ImageInfoViewModel();
try
{
HttpClient client = new HttpClient();

// Request headers.
client.DefaultRequestHeaders.Add(
"Ocp-Apim-Subscription-Key", subscriptionKey);

// Request parameters. A third optional parameter is "details".
string requestParameters =
"visualFeatures=Categories,Description,Color";

// Assemble the URI for the REST API Call.
string uri = endPoint + "?" + requestParameters;

HttpResponseMessage response;

// Request body. Posts a locally stored JPEG image.
byte[] byteData = GetImageAsByteArray(imageFilePath);

using (ByteArrayContent content = new ByteArrayContent(byteData))
{
// This example uses content type "application/octet-stream".
// The other content types you can use are "application/json"
// and "multipart/form-data".
content.Headers.ContentType =
new MediaTypeHeaderValue("application/octet-stream");

// Make the REST API call.
response = await client.PostAsync(uri, content);
}

// Get the JSON response.
var result = await response.Content.ReadAsStringAsync();

if (response.IsSuccessStatusCode)
{

responeData = JsonConvert.DeserializeObject<ImageInfoViewModel>(result,
new JsonSerializerSettings
{
NullValueHandling = NullValueHandling.Include,
Error = delegate (object sender, Newtonsoft.Json.Serialization.ErrorEventArgs earg)
{
errors.Add(earg.ErrorContext.Member.ToString());
earg.ErrorContext.Handled = true;
}
}
);
}

}
catch (Exception e)
{
Console.WriteLine("\n" + e.Message);
}

return responeData;
}

static byte[] GetImageAsByteArray(string imageFilePath)
{
using (FileStream fileStream =
new FileStream(imageFilePath, FileMode.Open, FileAccess.Read))
{
BinaryReader binaryReader = new BinaryReader(fileStream);
return binaryReader.ReadBytes((int)fileStream.Length);
}
}
}
}

API Response – Based on the given Image

The successful json response.

{
"categories": [
{
"name": "people_group",
"score": 0.6171875,
"detail": {
"celebrities": []
}
},
{
"name": "people_many",
"score": 0.359375,
"detail": {
"celebrities": []
}
}
],
"description": {
"tags": [
"person",
"sitting",
"indoor",
"posing",
"group",
"people",
"man",
"photo",
"woman",
"child",
"front",
"young",
"table",
"cake",
"large",
"holding",
"standing",
"bench",
"room",
"blue"
],
"captions": [
{
"text": "a group of people sitting posing for the camera",
"confidence": 0.9833507086594954
}
]
},
"color": {
"dominantColorForeground": "White",
"dominantColorBackground": "White",
"dominantColors": [
"White",
"Black",
"Red"
],
"accentColor": "AD1E3E",
"isBwImg": false
},
"requestId": "89f21ccf-cb65-4107-8620-b920a03e5f03",
"metadata": {
"height": 346,
"width": 530,
"format": "Jpeg"
}
}

Download

Output

Image information captured using Computer Vision API.For demo purpose, I have taken only a few data even though you can get more information about the image.

Reference

See Also

You can download other ASP.NET Core source codes from MSDN Code, using the link, mentioned below.

Summary

From this article we have learned how to implement One of the important Cognitive Services API ( Computer Vision API ). I hope this article is useful for all Azure Cognitive Services API beginners.

%d bloggers like this: