Yesterday I decided it was time to rebuild my VCSA. Being a brand new one, I thought it would be a good opportunity to replace the default untrusted SSL certificates with genuine ones issued from my in-house Microsoft Certificate Authority.

Replacing the machine SSL certificate went smoothly, however when it came to configuring the VMCA as a subordinate of my Issuing CA, all hell broke loose. Fortunately I’d learned from previous mistakes and had a snapshot at the ready to rollback to.

One thing that seemed odd in the certificates was a signature algorithm of “RSASSA-PSS”:

After speaking to Féidhlim O’Leary at VMware and another helpful chap on Twitter, it would appear this isn’t supported in vSphere 6. Whilst the vSphere 5.5 documentation specifically states this in http://pubs.vmware.com/vsphere-55/index.jsp?topic=%2Fcom.vmware.vsphere.security.doc%2FGUID-DC65ECA2-E6DC-4AD4-8542-AF1F4CEACC90.html:

The algorithm RSASSA-PSS with OID 1.2.840.113549.1.1.10 is not supported

…6.0 doesn’t mention it.

Unfortunately, the solution was rather painful.

When I build my Certificate Authorities, I do so on Server Core and use a custom CApolicy.inf. To make sure this doesn’t happen again you need to set the following:

AlternateSignatureAlgorithm=0

For me that meant the worst news – I have to reissue my root and issuing CA certs. Not an easy job 😦

Thankfully I’d only issued certificates to my VMware View environment at this point, so not a lot was affected. Had it been a full distributed vRealize Automation deployment then that would have been a different story…