Calculating Standard Deviation using PowerShell

I have been reading a book on data science recently, and as a result I have had to revisit my high school maths!
One interesting thing I found is that there is more than one way to calculate the end result, and that the standard taught method may not be the most accurate!
The most accurate (according to the experts) is a method created by B.P.Welford, which is detailed in-depth in Donald Knuth’s ‘Art of computer programming’.

My brain thought “I wonder if anyone has implemented this in PowerShell”?

I found it in many difference languages, but not PowerShell, so I thought I would take someone else’s hard work and translate!

The site I choose is https://blog.logentries.com/2016/10/overview-of-online-algorithm-using-standard-deviation-example/#Welford

My version does not yet work as a class where you may ‘pop’ on another value dynamically, it assumes instead that you have the figures, and you would like to work out the Standard Deviation using either a Sample or a Population variance.

The code is as follows:


function Get-StandardDeviation {
    [CmdletBinding()]
    Param (
    # Array of double values
    [Parameter(Mandatory=$true, ValueFromPipelineByPropertyName=$true,ValueFromPipeline=$true)]
    [ValidateNotNullOrEmpty()]
    [double[]]$Values
    )
    Begin {
        $count=0.0
        $mean=0.0
        $sum=0.0
    }#begin

    Process {
        foreach ($value in $values) {
            ++$count
            $delta = $mean + (($value - $mean) / $count)
            $sum += ($value - $mean) * ($value - $delta)
            $mean = $delta
        }#foreach
    } # process

    End {
        $VariancePopulation = $sum/($count)
        $VarianceSample = $sum/($count-1)
        $obj=[PSCustomObject]@{
            "VariancePopulation" = $VariancePopulation
            "VarianceSample" = $VarianceSample
            "STDEVPopulation" = [Math]::Sqrt($VariancePopulation)
            "STDEVSample" = [Math]::Sqrt($VarianceSample)
            "Mean" = $mean
            "Count" = $count
        }#obj
        Write-Output $obj
    } #end

}#function

To test this, create an array of doubles, and then use either method shown below to get the results.

$data = (50.0, 45.0, 55.0, 58.0, 43.0, 49.0, 50.0)</code>

Get-StandardDeviation -Values $data
$data | Get-StandardDeviation

Advertisements

Error: The Microsoft Online Services Module is not configured properly

What I was trying to achieve

I needed to use the MSOnline PowerShell Module to restore a deleted user from the Azure Active Directory (AAD) Recycle Bin. The replacement Module AzureAD does not (to my knowledge) have this functionality.

The Error

When trying to connect to AAD using the Connect-MsolService command, I received the following error:

Connect-MsolService : The Microsoft Online Services Module is not configured properly. Please uninstall and then reinstall the module.

I took the error messages advice, but to no avail.

Solution

The solution is in the registry…

Create a .reg file with the following content:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSOnlinePowerShell]
"Version"="1.0.0"
"InstallPath"="c:\\Program Files\\WindowsPowerShell\\Modules\\MSOnline\\1.0\\"
"InstallLanguage"="en-us"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSOnlinePowerShell\Path]
"WebServiceUrl"="https://provisioningapi.microsoftonline.com/provisioningwebservice.svc"
"FederationProviderIdentifier"="microsoftonline.com"

Import the reg file, then retry the command.

Thank you to user ‘Froggy’ who commented the solution at the following URL: https://stackoverflow.com/questions/36672088/how-do-i-correctly-install-the-powershell-msonline-module-in-windows-8-1-enterpr

Richard

 

Connecting Virtual Networks in different Azure Subscriptions to an ExpressRoute circuit in Resource Manager

Scenario

I have an ExpressRoute circuit configured with a connection to a Version 2 (Resource Manager) Azure Virtual Network. I also have an additional Virtual Network in a different Subscription which I need to connect to the same circuit.

The Issue

The instructions on connecting another Virtual Network in a different Subscription are a little confusing. It’s also worth noting that some of the parameters are different now with the latest version of the PowerShell Azure cmdlets. The original instructions may be found at this URL: https://azure.microsoft.com/en-us/documentation/articles/expressroute-howto-linkvnet-arm/#connect-a-virtual-network-in-a-different-azure-subscription-to-an-expressroute-circuit

Continue reading

Automatically Naming Resources in an Azure Resource Management Template

Why?

Resource templates are a great concept, but are fraught with danger. Badly named resources and naming inconsistency across resources in different subscriptions can make it difficult to determine purpose.

If a resource such as a Virtual Network or a Storage Account is badly named in a resource template file, you may find that you have to rely on the icon pictures in the Azure portal to indicate what type of resources you are looking at.

What if you need to see at a glance:

  • What region the resource belongs to i.e. Australia East or Australia Southeast?
  • What environment the resource belongs to i.e. Production or Testing?
  • What type of resource it is?

The problem comes from trying to keep the names meaningful and consistent. How do you ensure that your naming standards are adhered to?

Scenario

In my case I would like to name resources as follows:

<Resource Prefix>_<Environment>_<Location>

For instance:

"Vnet_Prod_auSoutheast"
 Continue reading 

Azure Resource Groups -Preventing Accidental Deletion with Resource Locks

Question

Did you know that when you delete an Azure Resource Group, it deletes all the resources in that group?

Scenario

You have built a Resource Group in Azure that contains your infrastructure resources including:

  • Virtual Network
  • Subnets
  • Network Security Groups (NSG)
  • Storage account to hold diagnostic logging for the NSGs

The subnets may host your IaaS Virtual Machines, maybe define your DMZ and your reverse proxy. So questions around risk need to be asked including:

  • How easy is it to delete a Resource Group?
  • Who can delete a Resource Group?
  • What can be done to protect a Resource Group?

Continue reading

Windows 10 (64-bit) upgrade error: Something happened…

Spoiler: It’s down to Windows Driver signature verification.

Something Happened

I’ve seen this error message twice:

  • Trying to upgrade Windows 8.1 64-bit to the last Windows 10 Insider Build
  • Trying to upgrade the Insider Build to the full release of Windows 10 64-bit

The Setupact.log file (Found under C:\$Windows.~BT\Sources\Panther), showed errors when attempting to mount C:\$Windows.~BT\Sources\SafeOS\WINRE.WIM.

Attempting to mount this WIM file manually using DISM, presented another error:

Windows cannot verify the digital signature for this file. A recent hardware or software change might have installed a file that is signed incorrectly or damaged, or that might be malicious software from an unknown source.

As a test I disabled ‘Driver Signature Verification’ using bcdedit from an elevated command prompt:

bcdedit /set testsigning on

After rebooting I tested mounting the WINRE.WIM file again – Success!

And the Windows 10 upgrade? – Worked perfectly.

Don’t forget to run: “bcdedit /set testsigning off” after the upgrade has completed.