Data Analysis and graph by PySpark

Disclaimer :-  This blog is nothing related to Covid-19 information / guidelines. This is using data provided by ecdc (European Centre for Disease Prevention and Control) for data analysis and data ploting in graphs in python.

First create blob storage in Azure to use , here in Azure we will save CSV files which receive from ecdc. Second register for Data bricks community edition  from here.

Once you have Data Bricks community edition, create one notebook for python

Now open the python file. First command is to mount the Azure blog Storage where we can save the CSV File. Above path /mnt/covid19/ is mount for using at data bricks

Now download the Covid19 Data from ecdc site via get request and save it with today's date if this file already exists then use the same else create a new one.

Now read the CSV File and do the grouping and filtering

Above will display the graph like below

Interface vs Abstract Class in C#

Feature	Interface	Abstract Class
Definition	An interface cannot provide any code, just the signature.	An abstract class can provide complete, default code and/or just the details that have to be overridden.
Access Modifiers	Everything Public	Can have public, private, internal for any method  , function.
Adding functionality (Versioning)	If we add a new method to an Interface then we have to track down all the implementations of the interface and define implementation for the new method.	If we add a new method to an abstract class then we have the option of providing default implementation and therefore all the existing code might work properly.
When to use	If various implementations only share method signatures then it is better to use Interfaces.	If various implementations are of the same kind and use common behavior or status then abstract class is better to use.
Multiple inheritance	A class may inherit several interfaces.	A class may inherit only one abstract class.

Service Fabric Deployment via Power shell

Here is the power-shell to deploy Service fabric application. 
Just change Service Instances and Application Instances Variable as highlighted below.

clear $sfURL = 'locahost:19080' $AppPkgPathInImageStore = 'ProcessENTSFApp' $sfApplicationBaseName = 'fabric:/rsoni.ENT.Process.SFApp'         $sfApplicationName='' $sfApplicationTypeName ='rsoni.ENT.Process.SFAppType' $sfApplicationTypeVersion='1.0.0' $sfAppServiceTypeName = 'rsoni.ENT.Process.Service.ProcessENTType' $AppFolderPath = 'C:\GitMapping\ENTSolution\rsoni.ENT.Process.SFApp\pkg\Debug' $TotalServiceInstances=5 $TotalApplicationInstances=4 $connectArgs = @{  ConnectionEndpoint = $sfURL;  WindowsCredential = $True;  } #Connect-ServiceFabricCluster @connectArgs Connect-ServiceFabricCluster  #-ConnectionEndpoint $sfURL $OnlyRemove=1 $sfAppType=Get-ServiceFabricApplicationType -ApplicationTypeName  $sfApplicationTypeName  -ApplicationTypeVersion $sfApplicationTypeVersion if ($sfAppType) {    if ($OnlyRemove -eq 1)     {         $sfApps=  Get-ServiceFabricApplication -ApplicationTypeName $sfApplicationTypeName           foreach ($sfAppobj in $sfApps)         {              $sfApplicationName =$sfAppobj.ApplicationName              $sfApp = Get-ServiceFabricApplication -ApplicationName $sfApplicationName              if($sfApp)                 {                         for ($i=1 ;$i -le $TotalServiceInstances ; $i++)                         {                             $sName =$sfApplicationName +'/'+ $sfAppServiceTypeName+'_'+$i                                 $sfAppService0=Get-ServiceFabricService -ServiceName $sName    -ApplicationName $sfApplicationName                             if ($sfAppService0)                             {                                 Write-Output "Removing  Service for $sName "                                 Remove-ServiceFabricService  -ServiceName $sName -ForceRemove -Force                             }                         }                     #Get-ServiceFabricServiceType                                 Remove-ServiceFabricApplication -ApplicationName $sfApplicationName -ForceRemove -Force                 }         }         Unregister-ServiceFabricApplicationType $sfApplicationTypeName $sfApplicationTypeVersion -Force         Remove-ServiceFabricApplicationPackage -ApplicationPackagePathInImageStore $AppPkgPathInImageStore     } } else {     if ($OnlyRemove -eq 0)     {         #Register Application Type         #Copy-ServiceFabricApplicationPackage -ApplicationPackagePath $AppFolderPath -CompressPackage -SkipCopy         Copy-ServiceFabricApplicationPackage -ApplicationPackagePath $AppFolderPath -ApplicationPackagePathInImageStore   $AppPkgPathInImageStore -TimeoutSec 1800         Register-ServiceFabricApplicationType -ApplicationPathInImageStore $AppPkgPathInImageStore         for ($ap=1 ;$ap -le $TotalApplicationInstances ; $ap++)         {             $sfApplicationName =$sfApplicationBaseName+'_'+$ap             $sfApp = Get-ServiceFabricApplication -ApplicationName $sfApplicationName                    Write-Output "Creating New Application"             New-ServiceFabricApplication -ApplicationName $sfApplicationName -ApplicationTypeName $sfApplicationTypeName -ApplicationTypeVersion $sfApplicationTypeVersion             for ($i=1 ;$i -le $TotalServiceInstances ; $i++)                 {                     $sName =$sfApplicationName +'/'+ $sfAppServiceTypeName+'_'+$i                     $sfAppService0=Get-ServiceFabricService -ServiceName $sName    -ApplicationName $sfApplicationName                     if (!$sfAppService0)                     {                         Write-Output "Creating Service for $sName "                         #New-ServiceFabricService -Stateless -ServiceName $sName  -ApplicationName $sfApplicationName  -ServiceTypeName $sfAppServiceTypeName -InstanceCount 1 -PartitionSchemeSingleton -ServicePackageActivationMode ExclusiveProcess                         New-ServiceFabricService -Stateful -ServiceName $sName  -ApplicationName $sfApplicationName  -ServiceTypeName $sfAppServiceTypeName -PartitionSchemeSingleton -ServicePackageActivationMode ExclusiveProcess -TargetReplicaSetSize 1 -MinReplicaSetSize 1                     }                 }           }      } } exit

Copy Data from Azure SQL to Azure Data Lake via Azure Data Factory

First create app Registration and give access to Azure Data Lake

Now create a new app registration

Provide details for App registration

Copy the Application ID and then Generate a key

App Id - 4f0fcf16-e148-411e-8b91-92c600d83c9e

Key

Once Key and AppId is created go back to Azure Data lake to provide permission

Go to Access and select the App Registration we created

Then Select permission for Read, Write and Execute

Now this new App Registration will have access for Azure Data Lake.

Now in Azure Data Factory  first go to Copy Data Wizard.

Now Select Source, which is SQL Server existing Connection

Create a new Azure Data Lake linked service for Destination

Provide the details of Azure Data Lake Store

Also Provide the Service Principal key to Access

Click on Test Connection - "Connection Successful" means connected.

Select Folder for Data Lake Store

Go Next  with default options

Summery before executing the pipeline.

Now you will see if Copy pipeline execution is Completed.

you can also see in Monitor tab for the Pipeline runs successfully.

Now go back to Data Lake Store  and click Data Explorer, you will see a new file created there for ADFDemo

Once you open this ADF Demo File you will see the comma separated values.

This is how we can copy data from SQL Server to Azure Data lake via Azure Data factory.