Friday, February 10, 2017

Microsoft Azure StorSimple : Hybrid Cloud Storage - Part 2

This is the part 2 in my blog series about StorSimple. In the first part, we introduced StorSimple, the different models and the pricing. You can find the blog here. In this blog we will look into the key features of StorSimple device in detail.

Automatic Tiering

StorSimple uses three layers of storage - SSD, HDD and Cloud storage. The read & write of fresh data will always happen in the SSD tier. When data gets aged and is  accessed less, it is tiered to HDD layer. The cold data, ie the least accessed data will be tiered to the Azure Cloud storage tier. With this architecture, organizations need not worry about local storage capacity management and planning since cloud storage is integrated with the solution and archival data is automatically tiered to it.

Now let us take a look into the automated tiering workflow. When data is written first to the device it goes to the SSD tier. Inline deduplication and compression will be active but the archival procedure doesn't kick in until much later. So the data continues to be written in this tier until first a defined low threshold limit and  later a high threshold is reached. At this point the system starts identifying the non-working set of data, ie the oldest data and this  data is spilled over to the next tier ie HDD. Lower and higher threshold is always kept empty because we want to keep some buffer space available if the user wants to restore some archival data. at a later point. The threshold limit  is  94% for 8000 series,  after which data is migrated from SSD to HDD and from HDD to AzureStorage . These processes are transparent to users and applications, and there is no impact on how they access the data.



In the case of  StorSimple Virtual array, we do not have a concept of SSD and HDD tiers. Hence the data is tiered directly to Cloud from the local storage ie the virtual hard disk .It is done based on a data heat map, which tracks usage of data , its age and relationship with other data. The active data or hot data is stored locally and cold /inactive data is tiered to cloud storage

Deduplication, Compression and Encryption

Dedeplication is enabled by default in StorSimple devises and there are no special licenses associated with it. When data comes into the StorSimple device, it  is written as 64 Kb blocks. For every block, hashkeys are built and a metadata map is created. SSD tier consists of raw storage and this metadata map. Deduplication happens in the SSD layer, thereby ensuring performance. When data comes in, it is matched with the metadata map. If  block exists it discarded and the pointers are updated. Same is the case with data being read. This helps in optimal utilization of  local capacity and makes operations like data migrations time efficient.

As mentioned earlier , data is spilled over to HDD layer once the high watermark is reached in SSD tier. It is in HDD tier that lossless compression of the data sets are done. The type of compression used is deflate compression. That means, data residing  in the HDD tier is fully deduped and compressed . However the users can continue to access the data in the device without any noticeable difference as the entire process is transparent


When data is tiered out from StorSimple array local storage to Cloud it is encrypted using AES-256 encryption. Customer holds the encrypted data. Data is converted to iscsi blocks , deduped ,compressed and then encrypted before sending to Azure. The data is sent to Azure over HTTPS. Data residing in Azure is further protected by mechanisms RBAC, login password, auditing, Access keys etc.  These are the different layers of security for your data in StorSimple.

 
To summarize the process , deduplication happens in SSD tier and when SSD reaches capacity the data is compressed and moved to HDD tier. When the data is ready to be tiered to cloud it is encrypted and send to cloud storage over HTTPS. However in case of virtual array there is a small difference wherein the data that resides in the local storage is not deduplicated and compressed .The deduplication, compression and encryption happens before data is tiered  to Cloud storage .

Local Snapshots and Cloud Snapshots
 
Snapshots refer to the inbuilt backup mechanism of StorSimple devices. There are two types of backups -  local snapshots and cloud snapshots

Local snapshots are point in time copies of data in StorSimple local storage. They are usually scheduled on daily and weekly basis with shorter retention periods. They are useful for restoring any recently deleted data. Local Snapshots uses Copy reference On Write(CROW) method. It makes use of volume metadata references for creating storage efficient snapshots and is stored locally in the devices

Cloud snapshots are point in time copies of data in Azure Cloud storage. Cloud snapshots are typically scheduled with longer retention periods, like weeks and months and are useful in DR scenarios. In case of cloud snapshots, entire data and the metadata is copied over to cloud when the snapshot is taken for the first time. All subsequent snapshots are incremental , ie only the changed data and metadata is copied over to cloud thereby optimizing the cloud storage  usage
 
StorSimple physical array supports both local and cloud snapshots. However Virtual array supports only Cloud snapshots

In the next part of this blog series we will look into the different management tools and some important StorSimple terminologies /concepts
 

Picture courtesy:http://www.slideshare.net/MicrosoftSuomi/microsoft-azure-storsimple-paul-chiola
 

Thursday, February 9, 2017

Microsoft Azure StorSimple : Hybrid cloud Storage - Part 1

Exponential data growth is one of the biggest challenges faced by organizations today. Traditional storage solutions are giving way to more robust cloud based storage systems. Azure StrorSimple is Microsoft's offering in the Hybrid cloud storage space that is capable of catering to all key storage requirements like primary storage, data archival, tape replacement, intelligent tiering, offsite storage etc. It has automatic storage tiering built in , and can tier all your less used and archival data to Azure cloud storage without any operational overhead. This cloud integrated storage mechanism was developed by a company called StorSimple which was acquired my Microsoft. Now this solution is offered under the umbrella of Azure Hybrid Cloud storage solution

 
Some of the key features of this storage solution are: 
 
  • Seamless integration of cloud storage with local storage
  • Automated tiering mechanism
  • Combination of SAS and SSD drives for local storage
  • Deduplication and compression of data
  • Thin provisioning
  • Local snapshots and Cloud snapshots for backup
  • Certified support from VMware
  • Inbuilt resiliency for hardware device with dual controller, hot swappable
  • Non-disruptive software upgrade
  • Integrated DR capabilities for recovery from cloud storage
  • Deterministic thin restores to download only the working data set

Now let us take a look at the  different types of StorSimple devices available
 
StorSimple is available as  physical rack mountable storage device, a virtual appliance and a cloud  appliance
 

StorSimple Physical devices - 8100 & 8600 models:
 
There are mainly two models - 8100 series and 8600 series. 8100 is a 2U device with a total usable local capacity of 15 TB and a maximum capacity of 200 TB including cloud storage. 8600 is a 4U device with an Extended Bunch of Device(EBOD) enclosure and related components. The total usable local capacity is 40 TB and a maximum capacity is 500 TB including cloud storage. The storage enclosures includes a combination of SSD and HDD drives for local storage. There are 12 disk drive slots per enclosure, with 8600 devices having additional 12 slots in the EBOD enclosure. The drive slots support SAS disk drives, which can be combination of SSD and HDD. The devices had resiliency built in , with share processors and storage along with mirrored controllers in active passive mode

StorSimple Cloud appliance - 8010 & 8020 models :

StorSimple Cloud appliance, as the name indicates, runs in Azure as a Virtual machine. In the Microsoft StorSimple documentation, you may find this being referred to as StorSimple Virtual device. It comes in two models - 8010 and 8020. 8010 devices can support a maximum capacity of 30 TB and 8020 devices upto 64 TB. 8020 is the latest model and has the capability to support premium storage for high performance work loads. You can connect volumes exposed by the StorSimple  cloud appliance to your Virtual machines in Azure using iscsi protocol.Another important target use case for cloud appliance is Disaster Recovery(DR). You can failover from your physical StorSimple device to a cloud appliance in the event of a disaster and bring up your machines in the cloud.

Imp: StorSimple Cloud appliance is always used in conjunction with a  StorSimple Physical appliance. One of the prerequistes of using a StorSimple Cloud appliance is that you should have a physical device registered with the StorSimple manager service running from the Azure portal.


 StorSimple Virtual array - 1200 model :

StorSimple virtual array is an ova format of the StorSimple Physical device that can be deployed both in VMware and Hyper-V hypervisors.It can provide a maximum local storage capacity of 6.4 TB and total capacity including cloud storage upto 64 TB. This solution targets enterprises who want to go for a cost-effective , but hybrid cloud based storage solution. It supports volumes being exposed as iscu targets along with SMB/CIFS based file shares. You can use it the same way that you would use a physical StorSimple device in terms of management and confirmation. There are certain limitations though when compared to the Physical device. For eg: it supports lesser storage capacity including cloud storage , ie 64 TB, when compared to 200 TB or 500 TB supported by   8100 and 8600 devices respectively. Also being a virtual device the performance has a dependency on the underlying Infra and Virtualization platform. However it supports all major use cases like cloud based storage, backup and DR. It can also be registered and managed from the Azure Portal like the Physical device

StorSimple Pricing

StorSimple Physical array and Cloud appliance requires you to have an Enterprise Agreement . Once you have registered the Physical device you can create the cloud appliance as well in Azure . This Cloud appliance is nothing but a Virtual machines of type Standard_A3 in case of 8010 model and Standard_DS3 if you go for 8020 model. Deploying these appliances will incur additional VM compute and Storage charges like in the case of a normal  Azure VM.

StorSimple Virtual Array on the other hand, is now available on a pay as you go model also. You can find the latest announcement regarding the same here :
You can download the OVA from the Azure portal , deploy it on you on-prem infra, and register it with the management service in Azure portal. The Virtual array uses a pay as you go model, where as you will be charged on a per day basis. In addition to that you will have to pay for Azure blob storage and data egress charges. For latest pricing of the virtual array, please refer to the following link : https://azure.microsoft.com/en-in/pricing/details/storsimple/
 

Stay tuned for the next part of this blog series, where we will do a deep dive into the  salient features of StorSimple that makes it stand apart from the crowd!!
 
Update : You can read part 2 of the blog series here

Saturday, November 26, 2016

Windows Server 2016 - Introducing Nano Server

This is the first blog post on the series that I am planning to write on Windows Server 2016 and its exciting new features. Lets start with the cool Nano Server!!

Nano server comes with the smallest OS foot print possible, which significantly reduces the management overhead and is keenly focused on cloud based deployment model. It is quite different from the existing OS flavors of Windows Server that we are familiar with. To start with Nano Server is headless, ie it doesn't provide any local logon capabilities. You can only manage it remotely using tools like powershell remoting ,wmi, winRM etc. Even the version of PowerShell that is shipped with NanoServer is a stripped down core edition. That means not all features will be available in this version of NanoServer. It is built on a reduced footprint version of .Net core, that means you may not be able to run all C# commands on PowerShell Core.Also it supports only 64 bit applications. You cannot promote a Nano Server as Active directory, not can you apply group policies to it. Certain tools like SCCM and SCDPM are not supported .

 The advantage of NanoServer is its focus on Just Enough OS. This is built with cloud based deployments in mind, ie you get smaller image sizes, attack surfaces and faster boot times. The lower foortprint model is ideal for several scenarios like VM hosting, scale out file servers, DNS server,Web server etc.You need to install additional packages to enabled the roles and features since they will not be enabled by default. That means precisely you install only what you need to to run your applications in a NanoServer. Nothing more and nothing less. This eliminates a lot of administrative overhead in terms of patch management, service management etc.


Let us take a look at how you can quickly deploy a Nano server, with IIS role installed in it. I am creating a VHD disk that will be used to create a Nano Server VM in Hyper-V

First of all you need to navigate to the contents of Windows Server 2016 ISO.


PS C:\windows\system32> cd C:\iso\NanoServer\
PS C:\iso\NanoServer> cd .\NanoServerImageGenerator\

Import the NanoServerImageGenerator PowerShell Module

PS C:\iso\NanoServer\NanoServerImageGenerator> Import-Module .\NanoServerImageGenerator.psd1

Now Create the new VHD file using the New-NanoServerImage command
PS C:\iso\NanoServer\NanoServerImageGenerator> New-NanoServerImage -Edition Standard -DeploymentType Guest -MediaPath c:\ISO -BasePath c:\Base -TargetPath c:\NanoServerVM\NanoServerVM1.vhd -ComputerName nanoiis
cmdlet New-NanoServerImage at command pipeline position 1
Supply values for the following parameters:
AdministratorPassword: *********

You will be asked for administrator password of the VM when prompted. "Media path" is the location of ISO. "Basepath" is the location to which NanoServer WIM and packages will be copied to . The vhd/vhdx will be copied over to the "TargetPath" specified

Next step is adding the packages required for IIS using DISM

Navigate to the base folder and run the following commands

--------------------------------------------------------------------------------------------------------------------------
PS C:\iso\NanoServer\NanoServerImageGenerator> cd c:\base
PS C:\base> mkdir mountdir
PS C:\base> dism.exe /Mount-Image /ImageFile:c:\NanoServerVM\NanoServerVM1.vhd /Index:1 /MountDir:.\mountdir
Deployment Image Servicing and Management tool
Version: 10.0.14393.0
Mounting
[==========================100.0%==========================]
The operation completed successfully.
PS C:\base> dism.exe /Add-Package /PackagePath:.\packages\Microsoft-NanoServer-IIS-Package.cab /Image:.\mountdir
Deployment Image Servicing and Management tool
Version: 10.0.14393.0
Image Version: 10.0.14393.0
Processing 1 of 1 - Adding package Microsoft-NanoServer-IIS-Package~31bf3856ad364e35~amd64~~10.0.14393.0
[==========================100.0%==========================]
The operation completed successfully.
PS C:\base> dism.exe /Add-Package /PackagePath:.\packages\en-us\Microsoft-NanoServer-IIS-Package_en-us.cab /Image:.\mountdir
Deployment Image Servicing and Management tool
Version: 10.0.14393.0
Image Version: 10.0.14393.0
Processing 1 of 1 - Adding package Microsoft-NanoServer-IIS-Package~31bf3856ad364e35~amd64~en-US~10.0.14393.0
[==========================100.0%==========================]
The operation completed successfully.
PS C:\base> dism.exe /Unmount-Image /MountDir:.\MountDir /Commit
Deployment Image Servicing and Management tool
Version: 10.0.14393.0
Unmounting
[==========================100.0%==========================]
The operation completed successfully.
PS C:\base> dism.exe /Unmount-Image /MountDir:.\MountDir /Commitdism.exe /Unmount-Image /MountDir:.\MountDir /Commit
--------------------------------------------------------------------------------------------------------------------------


You can use the commands in the above section if you have already created a VHD and want to modify it. You can also add the IIS related packages during VHD creation while running the New-NanoServerImage command by using the package parameter "-Package Microsoft-NanoServer-IIS-Package"




 

Wednesday, November 2, 2016

Azure Automation: Using PowerShell workflow

In my previous blog, I discussed about getting started with Azure automation and how to do basic tasks like starting/stopping VMs at a given time using Graphical runbooks. You can find  the blog post here : http://thetechnologychronicle.blogspot.in/2016/05/azure-automationusing-graphical-runbooks.html

In this blog, we will look into a little bit more complex  way of creating runbooks, ie using PowerShell workflows. You can also write simple PowerShell code and create runbooks using that. However, if you have to parameterize your inputs and change them each time you run, then you will have to use the PowerShell workflow based format. For creating a new PowerShell workflow based runbook, go to your  Azure Automation account-> Runbooks->Add a runbook. You can select the quick create option and runbook type as PowerShell workflow




Sample Usecase:

The usecase that I am going to discuss here is automated provisioning of VMs with the number of Data disks that you define. You can also specify the size of the data disks to be provisioned. We didn't have any runboook readily available in the gallery to do this task. Hence an Azure PowerShell script to create a new Azure VM in ARM portal was tweaked to achieve this: https://msdn.microsoft.com/en-us/library/mt603754.aspx

The tweaks include:
  • Converted the PowerShell script to workflow to allow input parameters
  • Minor changes to use existing storage and network
  • Commands to add datadisk
  • Had to introduce InlineScript in the workflow so that the PowerShell commands are executed independently. If this is not done, it will throw errors due to issues in data conversion
  • Introduced basic For loop to add datadisks based on provisioning requirements

Runbook:

The runbook that I created is given below for reference. Feel free to give it a spin !!

=============================================================
workflow dynamicDDwithparamter
{   
        param (
         
      
        # If you do not enter anything, the default values will be taken
        # VM name, availability set and NIC card name
        [parameter(Mandatory=$true)]
        [String]$VMName,
        [parameter(Mandatory=$true)]
        [String]$ComputerName,
        [parameter(Mandatory=$true)]
        [String]$AvailabilitySetname,
        [parameter(Mandatory=$true)]
        [String]$InterfaceName,
        ## Compute - Name of VM to be created,Vm size, datadiskname
        [parameter(Mandatory=$true)]
        [String]$UserName,
        [parameter(Mandatory=$true)]
        [String]$Password,
        ## Storage - Name of existing storage
        [parameter(Mandatory=$true)]
        [String]$StorageName = "testsql295p",
        ## Global - Uses an existing resourcegroup
        [parameter(Mandatory=$true)]
        [String]$ResourceGroupName = "autotest",
        [parameter(Mandatory=$true)]
        [String]$Location = "WestEurope",
        ## Network - Name of existing network. This should match the network settings of others VMs in the target availability set
        [parameter(Mandatory=$true)]
        [String]$Subnet1Name = "Subnet1",
        [parameter(Mandatory=$true)]
        [String]$VNetName = "VNet10",
        [parameter(Mandatory=$true)]
        [Int]$Disknumber ,
        [parameter(Mandatory=$true)]
        [Int]$DisksizeinGB ,
        ## Compute - Vm size
        [parameter(Mandatory=$true)]
        [String]$VMSize = "Standard_A2"

       
    )
    InlineScript{
        $VMName =$Using:VMName
        $StorageName = $Using:StorageName
        $ResourceGroupName = $Using:ResourceGroupName
        $Location = $Using:Location
        $InterfaceName = $Using:InterfaceName
        $Subnet1Name = $Using:Subnet1Name
        $VNetName = $Using:VNetName
        $ComputerName = $Using:ComputerName
        $VMSize = $Using:VMSize
        $AvailabilitySetname = $Using:AvailabilitySetname
        $UserName = $Using:UserName
        $Password = $Using:Password
        $Disknumber = $Using:Disknumber
        $DisksizeinGB =$Using:DisksizeinGB


$connectionName = "AzureRunAsConnection"
    # Get the connection "AzureRunAsConnection "
    $servicePrincipalConnection=Get-AutomationConnection -Name $connectionName     
    "Logging in to Azure..."
    Add-AzureRmAccount `
        -ServicePrincipal `
        -TenantId $servicePrincipalConnection.TenantId `
        -ApplicationId $servicePrincipalConnection.ApplicationId `
        -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint


#$StorageType = "Standard_GRS"


#$VNetAddressPrefix = "10.0.0.0/16"
#$VNetSubnetAddressPrefix = "10.0.0.0/24"
$OSDiskName = $VMName + "OSDisk"
$dataDiskName = $VMName + "DataDisk"
# Resource Group
#New-AzureRmResourceGroup -Name $ResourceGroupName -Location $Location
# Storage - Creates Storage account
"Get storage account..."
#$StorageAccount = New-AzureRmStorageAccount -ResourceGroupName $ResourceGroupName -Name $StorageName -Type $StorageType -Location $Location
$StorageAccount = Get-AzureRmStorageAccount -ResourceGroupName $ResourceGroupName -AccountName $StorageName
"Collected storage account details ..."
# Network - Creates Public IP,Vnet and NIC card
"configure NIC..."
$PIp = New-AzureRmPublicIpAddress -Name $InterfaceName -ResourceGroupName $ResourceGroupName -Location $Location -AllocationMethod Dynamic -Force
#$SubnetConfig = New-AzureRmVirtualNetworkSubnetConfig -Name $Subnet1Name -AddressPrefix $VNetSubnetAddressPrefix
#$VNet = New-AzureRmVirtualNetwork -Name $VNetName -ResourceGroupName $ResourceGroupName -Location $Location -AddressPrefix $VNetAddressPrefix -Subnet $SubnetConfig
$vnet = Get-AzureRmVirtualNetwork -Name $VNetName -ResourceGroupName $ResourceGroupName
$subnetconfig = Get-AzureRmVirtualNetworkSubnetConfig -VirtualNetwork $vnet
#$subnetconfig = Get-AzureRmVirtualNetworkSubnetConfig -VirtualNetwork $Using:vnet
$Interface = New-AzureRmNetworkInterface -Name $InterfaceName -ResourceGroupName $ResourceGroupName -Location $Location -SubnetId $VNet.Subnets[0].Id -PublicIpAddressId $PIp.Id -Force
#$Interface = New-AzureRmNetworkInterface -Name $Using:InterfaceName -ResourceGroupName $Using:ResourceGroupName -Location $Using:Location -SubnetId $Using:VNet.Subnets[0].Id -PublicIpAddressId $Using:PIp.Id -Force
"configured NIC..."
# Compute
## Setup local VM object
"creating VM object properties..."
$secpasswd = ConvertTo-SecureString $Password -AsPlainText -Force
$mycreds = New-Object System.Management.Automation.PSCredential ($UserName, $secpasswd)
$AvailabilitySet = Get-AzureRmAvailabilitySet -ResourceGroupName $resourcegroupName -Name $AvailabilitySetname
$VirtualMachine = New-AzureRmVMConfig -VMName $VMName -VMSize $VMSize -availabilitysetID $AvailabilitySet.id
$VirtualMachine = Set-AzureRmVMOperatingSystem -VM $VirtualMachine -Windows -ComputerName $ComputerName -Credential $mycreds -ProvisionVMAgent -EnableAutoUpdate
$VirtualMachine = Set-AzureRmVMSourceImage -VM $VirtualMachine -PublisherName MicrosoftWindowsServer -Offer WindowsServer -Skus 2012-R2-Datacenter -Version "latest"
$VirtualMachine = Add-AzureRmVMNetworkInterface -VM $VirtualMachine -Id $Interface.Id
$OSDiskUri = $StorageAccount.PrimaryEndpoints.Blob.ToString() + "vhds/" + $OSDiskName + ".vhd"
#$DataDiskVhdUri01 = $StorageAccount.PrimaryEndpoints.Blob.ToString() + "vhds/" + $dataDiskName + ".vhd"
$VirtualMachine = Set-AzureRmVMOSDisk -VM $VirtualMachine -Name $OSDiskName -VhdUri $OSDiskUri -CreateOption FromImage
For ($i=1; $i -le $Disknumber; $i++) {
    $dataDiskName = $dataDiskName + $i  
   $DataDiskVhdUri01 = $StorageAccount.PrimaryEndpoints.Blob.ToString() + "vhds/" + $dataDiskName + ".vhd"
   $VirtualMachine = Add-AzureRmVMDataDisk -VM $VirtualMachine -Name $dataDiskName -Caching 'ReadOnly' -DiskSizeInGB $DisksizeinGB -Lun $i -VhdUri $DataDiskVhdUri01 -CreateOption Empty
   $dataDiskName = $VMName + "DataDisk"
    }
     
#$VirtualMachine
#$VirtualMachine = Add-AzureRmVMDataDisk -VM $VirtualMachine -Name $dataDiskName -Caching 'ReadOnly' -DiskSizeInGB 10 -Lun 0 -VhdUri $DataDiskVhdUri01 -CreateOption Empty
"created VM object properties..."
## Create the VM in Azure
"creating Virtual machine..."
New-AzureRmVM -ResourceGroupName $ResourceGroupName -Location $Location -VM $VirtualMachine
"created Virtual machine..."
    }
}
 

Thursday, October 13, 2016

Security in the cloud - Disk encryption in Azure


Security in the cloud is a priority for every organization planning to adopt public cloud for mission critical applications. In Azure, these security concerns are addressed at different layers starting from the platform layer up to the VM OS layer. This picture shows an overview of the different layers of security in Azure
 
 

Any traffic directed to your applications hosted in Azure will first hit the platform's native DDOS protection mechanism. If a DOS attack is happening a specific IP is targeted, the DDOS protection mechanism will blackhole the traffic and the endpoint will be brought down. Thereby the surrounding resources will be protected. If you have resiliency built in, you can bring up another endpoint and ensure that your service is available

At the next layer you have endpoints, ie traffic will be received only at the designated endpoints in case of classic model, or as defined in NSGs in case of ARM model. The VMs can be placed in different virtual networks which are isolated from each other by default. Inside Vnet you can use NSGs and UDRs to manage traffic. If you want to notch up the security one level higher , you can introduce  Virtual Network Appliances in your network.For eg: Firewalls, IDS/IPS devices etc that are available in the Azure marketplace.
In this blog , I am going to talk about security at the VM layer, ie encryption of your data at rest using disk encryption. This is achieved using Bitlocker encryption of Windows VMs and DMcrypt in Linux VMs. It leverages Azure KeyVault services to store your encryption keys which is again an additional level of security. The process is quite straight forward and encryption can be done for new as well as existing VMs. In case of Windows IaaS VMs, we can encrypt both the OS and Data disk, while in case of Linux IaaS VMs the data disk can be encrypted.You will have to use the ARM deployment model to leverage this feature, since it does not work with the classic model. There are ARM templates readily available in GitHub that can readily help you with the encryption of existing and New VMs.
 Let us take  a look at the steps to be done to enable OS and Data disk encryption of IaaS windows VMs in Azure
Prerequisites:
1) Create an Azure KeyVault
This can be now done from the ARM portal itself. KeyVault is available in the Azure Portal in preview mode. You can create a new keyVault  by providing basic information like KeyVault Name , Resource group Name,Location etc. In addition to that , by default your user account will have access in the "Access Policies". You can edit the "Advanced Access Policy" and enable all the three options given there
 
 
Alternately, you can also use the ARM template available at this location to create a KeyVault:
This template will create a KeyVault for you with all the three advanced access policies including the Volume encryption policy that we need for Disk encryption.
 
2) Create an application in Azure AD with permission to access KeyVault:
This is a very important step since you will be using this application id and key during VM encryption
1. Select the organization's active directory from the classic portal and select the application tab



2.:Click on add from the bottom menu to add a new application. Select the first option, ie add an application my organization is developing



3. Provide name of the application in the next step


3. Add the Sign-on URL and App URI. You can enter any value here in URI format, it need not have to be an existing application. Only requirement is that it should be unique for an organization


4.Now click on configure, and copy the client Id of the application



5.Next we need the application key. This can be generated from the portal from under the keys session. Select duration as 1 year from the drop down. Once you save the configuration, a key will be displayed which can be copied over



6. Next step is to provide this application access to keyvault. It can be done from an Azure PowerShell window using the following command
Set-AzureRmKeyVaultAccessPolicy -VaultName $keyVaultName -ServicePrincipalName $aadClientID -PermissionsToKeys 'WrapKey' -PermissionsToSecrets 'Set' -ResourceGroupName $rgname
 

You have to set the variables $keyVaultName,$aadClientID,$rgname  to have values of your keyvault name, client id of the application that we got at Step 4 above and Resource group name

Now you have the client id and the key that you will need during the ARM template execution. Now lets proceed with the VM disk encryption

Create a new encrypted VM using ARM template and KeyVault

Deploy the following template available in GitHub to encrypt a new VM : https://github.com/Azure/azure-quickstart-templates/tree/master/201-encrypt-create-new-vm-gallery-image

Click on the "deploy to Azure" option in the page to deploy the ARM template directly in Azure

Provide the mandatory parameter values like VM name, admin username, password, Storage account name, Virtual network, subnet,keyvault name and keyvault resource group along with the client ID and Client Secret  of the Azure AD application that we created earlier. Additionally there will be two options for Key Encryption key and URL.It is not mandatory and we are not using that in this example. You can then agree to the terms and conditions , click on purchase and the deployment of encrypted VM will start


Encrypt an existing VM using ARM template and keyvault

Deploy the following template available in GitHub to encrypt an existing VM : : https://github.com/Azure/azure-quickstart-templates/tree/master/201-encrypt-running-windows-vm

 
Deploy the template to Azure and provide details of the VM that you want to encrypt.'Volume type' can be OS,Data or All(default value) depending on which disk you want to encrypt


Check encryption status

Now that you have enabled the encryption you might want to verify the status. There are few ways to check this.

Easiest way is to check from the Azure portal.Navigate to the Disks information of the VM from the portal. It will show the OS disk status as Encrypted.



The data disk encryption status is currently not reflected in the portal. However, you can check the data disk encryption status by using Azure PowerShell command Get-AzureRmVmDiskEncryptionStatus and providing resource group name and Vmname as parameters

Get-AzureRmVmDiskEncryptionStatus  -ResourceGroupName $rgname -VMName $Vmname



You will see OSVolumeEncrypted and DataVolumesEncrypted status as True if encryption is enabled

You can also check the status of disk encryption from within the VM using ‘manage-bde’ command and providing the drive letter as parameter. Sample output given below



Also, it can been seen from the GUI of the server , the drives will have a lock signal associated

 
Now you know how to enable disk encryption for protection of data at rest in Azure!!!!

Disk encryption is only one of the security measures available in Azure. There are many other options like NSGs,Virtual appliances, UDRs etc that we discussed in the introduction. Also, there is a new service called Azure Security Center that takes cares of your Azure environment security requirements holistically . Keep watching this space for more information on it!!!



https://azure.microsoft.com/en-in/documentation/articles/azure-security-disk-encryption/

Friday, September 16, 2016

#MyAzureLabs: DRaaS using Azure: Test your DR strategy

This is Part 2 of my blog post on DRaaS using Azure. You can view first part of the blog here

In first part, we discussed how to protect your on-prem physical servers using Azure Site Recovery services. Having a DR strategy and enabling protection is not always enough. You should ensure that your DR strategy will work as expected when a disaster strikes. In case of usual DR solutions, it is not always possible to test the DR strategy without downtimes. However, Azure Site Recovery provides you with an option to test your DR strategy and keep it well oiled and battle ready!! Test failover to the rescue..  Lets see how we can do a test failover of on-prem physical services to Azure.

Select the vault where your replicated data resides. Select the settings, and choose the replicated items.



Select the option "Test failover"



Select the settings of the test failover. The failover direction will be automatically selected, ie from on-prem to Azure. You can select the recovery point and the virtual network. Recovery point can be the latest point in time, latest processed point in time, latest application consistent point in time or even a custom restore point. The virtual network should not be same as what you will be using in your production, ie the one that you specified in the target configuration while enabling the replication



Click ok and the test failover will begin!!

Now, if you go and check the job status you can see a number of things happening. The test failover will create a test environment, spin up your VM and at one point wait for your input to complete testing



The VM created by the test failover will be listed in the portal now. You will have to do some additional configurations though to connect to the VM and test your applications

- Go to the settings of the VM, select the NIC card and associate a public address to it
- Now associate a NSG to the NIC card and create the rules that you require, for eg: RDP access, access to a web site that you might be running in the VM etc

After that the connect button of the VM will be enabled and you can connect to it and test your applications.

All done now and if you are happy with the DR test, you can clear the failover environment with a single click. Go back to the test failover job, click on  "complete failover" . You will have to provide a reason, and then click ok.



That's and the test environment will be magically cleaned up by ASR !! Now that is a very easy and clean way of testing your DR strategy without disturbing any of your existing environments..

Monday, August 15, 2016

#MyAzureLabs : Azure Point to site VPN configuration for existing Vnet

Azure Point-to-Site enables VPN connectivity from client machines to Azure Vnet. This is especially useful for mobile users,  who could be travelling and is not connected to your office network. There is a very good documentation available on how to configure Point 2 site VPN for a new Vnet, both for classic and new portal . It  can be found here : https://azure.microsoft.com/en-in/documentation/articles/vpn-gateway-point-to-site-create/

What if  you already have a Vnet in Azure with resources connected to it ? In this blog, I will elaborate on how to enable Point-to-Site VPN for an existing Vnet . It is documented based on the testing done in new portal. The Vnet was already existing, and a VPN gateway was created from the new portal using the graphical interface and connected to the Vnet. For the remaining steps, PowerShell was used.

1. Create VPN gateway . Go to new portal->Virtual network gateway and create new. You will have to select the Vnet for which you want to create the gateway, provide a gateway subnet IP range , select a public Ip address , select the Gateway type as VPN and VPN type as route based. Please note that for enabling Point-to-Site VPN connectivity , the gateway type should be route based.

2. Create an IP pool , from which IP will be allocated to VPN clients. This should be done using Azure PowerShell. First login to your Azure account

Login-AzureRmAccount

Create VPN Client Address pool

Set-AzureRmVirtualNetworkGatewayVpnClientConfig -VirtualNetworkGateway $Gw -VpnClientAddressPool "192.168.100.0/24"

3. Create a self signed certificate for your client machine using the following steps mentioned in the following link

https://azure.microsoft.com/en-in/documentation/articles/vpn-gateway-certificates-point-to-site/

4. Once the client certificate is created installed , next steps is to export the root certificate in .cer format. You can do that from the certificate management mmc. Ensure that you select the option "Base-64 encoded X.509 (.cer)" in the certificate export wizard






5.You need to copy over this .cer file to the machine from which the Azure PowerShell commands will be executed. Run the following command from Azure PowerShell .

$Gw = Get-AzureRmVirtualNetworkGateway -Name "p2sgwtst" -ResourceGroupName "p2stest"

 Here p2sgwtst is name if my Virtual Network Gateway and p2stest is Resource Group Name

6. Get information about your root certificate

$Text = Get-Content -Path "\<path of root certificate>\Root.cer"

7. Add the root certificate to the VPN gateway

$rootCert = Add-AzureRmVpnClientRootCertificate -VpnClientRootCertificateName "RootCertificateNp.cer" -PublicCertData ($CertificateText | out-string) -VirtualNetworkGatewayName $gw.Name -ResourceGroupName "p2stest"


 8. Download the Azure VPN package

Get-AzureRmVpnClientPackage -ResourceGroupName "p2stest" -VirtualNetworkGatewayName $gw.Name -ProcessorArchitecture Amd64

You will get a download link for the exe as output. Copy it over to the client machine and install using administrative privileges.  If it is all installed correctly, you will see the network in the list of VPNs in your machine











 

Sunday, July 3, 2016

DRaaS using Azure: How to protect your on-prem physical machines.... #MyAzureLabs

BC/DR is a key consideration for all organizations big or small. Thanks to Azure, we now have an affordable and easy to implement BC/DR solution . Azure site recovery service(ASR) can be used for a multitude of disaster recovery scenarios, with an economic pay-as-you-go costing model. The DR scenarios catered to by ASR currently are:

DR site in Azure
- Physical machines to Azure
-VMware environment to Azure
- Hyper-v(with or without VMM) to Azure

DR site in a secondary DC, and orchestration by ASR
-VMM site to site
-VMware/Physical to VMware
-VMM to VMM(SAN replication)

This week in my Azure labs, I tried out the first scenario, ie DR from On-Prem Physical machines to Azure. This blog is all about my little experiment and some tips and tricks that I learned during the same.

The following link , which explains the procedure for protecting Physical/VMware environment is a good starting point: https://azure.microsoft.com/en-in/documentation/articles/site-recovery-vmware-to-azure/

I will use this article as reference point, which is very detailed and well written. I will be going into further more details on few of the areas mentioned in the link .Based on my experience,  I think it will be helpful for someone trying to set up a Physical server to Azure replication for the first time.

You should ensure that the prerequisites for physical server protection mentioned in the link are taken care of. You need to run the Site Recovery Unified SetUp for installing the configuration and process server. Refer to "Step 2: Set up the source environment" in the link above for details on initial set up of the vault, setting up configuration server, registering it in Azure etc. These steps are  pretty straight forward. Detailed explanation of the configuration server setup on-prem  is also mentioned in  Step 2 of the article

Lets assume that you done the initial vault creation , configuration server setup , created target environment in Azure(Resource group, storage, network etc) and have also created the replication policies to be used. All these come under "Step 1: Prepare your infrastructure" in your site recovery vault. These steps are again clearly explained in the official documentation :  https://azure.microsoft.com/en-in/documentation/articles/site-recovery-vmware-to-azure/

Now lets see what needs to be done at the physical server end to enable the protection

Steps to be done on Physical server:

1)Set up the registry key entry






2)Enable the following in Allow an app or feature through Firewall.
    > File and print sharing
    >Windows Management Instrumentation






3)Add an account that has admin privilege in the target physical machine in the cspsconfigtool.  It can be found in the following location in the configuration server






















Click on Add account

4)In my case, the physical machine was not added to domain. Hence I added a local admin user. The friendly name can be anything, it is just for identifying that account in Azure portal.






















5)Now you can install the mobility agent on the physical server. The installer can again be found in the configuration server at the following location. You need to select the installer based on the operating system type. In my case I selected the Windows installer










Select option to install Mobility service
















Enter Configuration server IP and Passphrase











Specify install location. That is all that is required. You can go to the next step and wait for the installation to be complete





Steps to be done in Azure portal:

Now that the mobility agent is installed, you can refresh the configuration server in the Azure management portal

Go to <recovery services vault> -> Settings->Site recovery infrastructure->Servers
select the configuration server and click refresh server












Click ok on the message and wait for the refresh to be completed.

Once the refresh is completed, ideally the new physical server will be reflected in the connected agents list













Now you can go ahead and enable replication for your physical server. In the Management portal, go to <Recovery services vault>->Settings->Site Recovery->Enable replication





















Enter the source. This will be your configuration server .Machine type will be Physical machines and Process server in this installation is same as configuration server
















Configure the target environment in Azure.


You need to select the target physical server at the next step. Click on the + sign
















Enter details of your on prem physical server, ie server name , IP and the OS type









Click ok and wait for the server to be added
Once the server is added, it will be listed in the blade. Select the server and click ok

















In the next step configure properties. If the agent is installed correctly and is detected by the portal, you will be able to select the disks that you want to backup . ie , disks other than the OS disks
From the account dropbox you can select the account that you had created earlier in the cspsconfigtool.(Refer step no: 3)








In the configure replication settings page, select the replication policy that you had created earlier



















Now all the steps are done, and you can click "Enable replication" to protect your on-prem physical server
























You  can click on notifications to see progress of the task. You can also go to <site recovery vault>->Jobs->"Site Recovery Jobs"-> and select the "Enable protection" job to see the status



















If you see all green ticks, your machine protection is enabled . You can see the status of replication from  site recovery vault>->Replicated Items. Once the replication is completed, the status will be shown as protected











Now that the Physical server is replicated and protected, we might want to test if everything will work as expected during a Disaster. Right? That is where the Test failover feature will help. I will cover that in my next blog post. Keep watching this space for more!!


Update: The second part of the blog is published, which covers the test failover process. You can find it here

This article is also published in my MSDN : blog : https://blogs.msdn.microsoft.com/shijimolak/2017/01/24/how-to-protect-your-on-prem-physical-machines/