Mr. Praline: Look, matey, I know a dead parrot when I see one, and I’m looking at one right now.
Owner: No no he’s not dead, he’s, he’s restin’!
My lab died!
It had been running quite happily for several weeks, then disaster struck…
Well to be precise (and a lot less dramatic), my Microsoft System Center Virtual Machine Manager (SCVMM) lost the ability to control any of my Hyper-V clusters.
I originally built this lab to prove a concept for a customer around a single instance of SCVMM, Azure Site Recovery (ASR) and stretched subnets across two datacentres. You’ll be able to read the results of this Proof of Concept (PoC) in another blog post.
The primary error was:
An internal error has occurred trying to contact the ‘hyperv03.mydomain.corp’ server:
WinRM: URL: [http://hyperv03.mydomain.corp:5985], Verb: [INVOKE], Method: [GetVersion], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/AgentManagement]
The request is not supported (0x80070032)
Followed by recommendations to check that Windows Remote Management (WinRM) was running (it was) and that the SCVMM agent was installed on the Hyper-V host (it was).
I went through the usual troubleshooting steps for WinRM:
- Test-WSMan – No errors
- Enable-PSRemoting – All good
- Enable-WSManCredSSP – No problems there
- Check local policy for ‘Allow Delegating Fresh Credentials’ – All set correctly
Then by chance, I searched using DuckDuckGo (privacy focused search engine) for “CredSSP the request is not supported” and found the following article:
Microsoft released an update for CredSSP in March 2018 (CVE-2018-0886) which patches a known vulnerability that allow remote code execution (CredSSP encryption Oracle remediation). This fix was updated in May (last month).
The simplest solution is to patch all servers immediately, but as we all know, patching takes time, and in a production environment with mandated maintenance windows, it takes planning.
A short-term workaround is available. Set the Group Policy value for “Computer Configuration/Administrative Templates/System/Credentials Delegation/Encrypted Oracle Remediation” to ‘Vulnerable’.
Note: Make sure that you understand the impact of setting this value which is detailed here:
Now that all my servers are patched, SCVMM is happily talking to my Hyper-V clusters.
I was lucky – this only impacted a lab. Imagine if this was your production environment?
While it’s great that Microsoft are providing regular fixes for issues and bugs, it is a timely reminder that installing patches is not without some risk.
Ironically as my practice manager proofread this blog post, he realised that it would fix his issue with accessing his Virtual Machine in Azure!
Mr. Praline: Now that’s what I call a dead parrot.
Owner: No, no…..No, ‘e’s stunned!
Mr. Praline: STUNNED?!?