Certificates play a vital role in securing communications between your vCenter Server and its connected components, such as ESXi hosts, vSphere clients, and other management tools.

However, certificate management can often be a daunting task, especially when it comes to renewal.

In this post, we will see step-by-step the process of renewing our vCenter Server certificate, ensuring the environment remains secure and operational without unnecessary downtime.

We can either generate here self-signed certificates by using the in vSphere built-in VMware Certificate Authority (VMCA) since vSphere 6.0, or we can use certificates issued by a trusted internal CA or public CA.


About how to replace the ESXi certificate with a custom certificate I will show in my following post.

How to replace the ESXi Certificate with a Custom Certificate


Table Of Contents
  1. VMware vCenter Server Management – Login failed with administrator@vSphere.local
  2. Regenerate vSphere 6.x, 7.x, and 8.0 Certificates using self-signed VMCA
  3. Regenerate vSphere 6.x, 7.x, and 8.0 Certificates using our internal CA (AD CS)
  4. Refresh the Security Token Service (STS) Root Certificate
  5. Links


VMware vCenter Server Management – Login failed with administrator@vSphere.local

Recently I run into the following error at a customers vSphere environment when trying to login to the vSphere client.

Exception in invoking authentication handler [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1007)


As we can already see by the error message, here the SSL certificate of the vCenter Server Appliance (vCSA) is expired.

Below I will show how we can renew resp. regenerate this certificate by using the CLI.

First we will see how to renew the certificate by using a self-signed certificate and later how to use a certificate from an internal CA (in my case Active Directory Certificate Services AD CS).

Regenerate vSphere 6.x, 7.x, and 8.0 Certificates using self-signed VMCA

With the vSphere 6.0 release the VMware Certificate Authority (VMCA) was introduced which acts as an internal Certificate Authority, automatically generating and managing certificates for vCenter Server, ESXi hosts, and other services.

Before we regenerate the vCenter Server certificate, we should first ensure that the STS certificate is valid.

If STS certificate is expired or corrupted, certificate regeneration will fail due to the service dependencies like vmware-stsd and vmware-vapi-endpoint failing to start without a valid token.


Checking Expiration of STS Certificate on vCenter Servers

Therefore we can use a python script which we first need to upload to the VCSA.

If you receive an error when connecting through SFTP protocol to the VCSA via CLI or WinSCP like below.

Received too large (1433299822 B) SFTP packet. Max supported packet size is 1024000 B


When using WinSCP.


Run the following command from the VCSA CLI.

When using the SFTP protocol, part of the operation occurs on the target system. The default Appliance Shell cannot be the remote partner of SFTP.

By default, VCSA is set to Appliance Shell, but SFTP requires Bash shell.

To resolve the issue, enable the Bash shell on the appliance. In the Bash shell, change the default shell to Bash.

# chsh -s /bin/bash root


To revert the changes after we uploaded the python script by using WinSCP, use the below command.

# chsh -s /bin/appliancesh root


Source: https://knowledge.broadcom.com/external/article?articleNumber=326317


Download the attached checksts.py script attached to this article. Upload the script to the VCSA. 

For example: /tmp

Connect to the vCenter by using the SFTP protocol
PS> sftp root@vcenter.matrixpost-lab.net

change to the /tmp folder on vCenter
sftp> cd /tmp

checking on which path we are rigth now on our sftp client
sftp> lpwd

changing the local path
sftp> lcd C:\Users\MarcusRath\Downloads\script

list files and folders in our local path
sftp> lls

uploading the script to vCenter and its actual path we are right now, /tmp
sftp> put checksts.py


Once the script has been successfully uploaded to VCSA, change the directory to /tmp.

Run python checksts.py.

# python checksts.py

Renew VCSA Certificate

We can now renew our vCenter (VCSA) certificate by using the built-in certificate manager of vSphere.

Note: Take a snapshot or a backup of the vCenter before proceeding.


For vCenter Server 6.x/7.x/8.x Appliance: 

Connect to the bash shell of vCenter and execute the following command.

root@vCenter [ ~ ]# /usr/lib/vmware-vmca/bin/certificate-manager


Select Option 4 (Regenerate a new VMCA Root Certificate and replace all certificates)

Note: Selecting Option 8 (Reset all Certificates), both options perform the same functionality. (The difference is that option 8 does not perform automatic Rollback of the certificates). Option 8 is needed when both the Machine SSL certificate and Solution users certificates are expired.


Enter these values as prompted by the VMCA.


Stopping services, regenerating and replacing the certificates and finally starting the services will take a few minutes. Especially here at 85% when starting the services again.


Type Yes (Y) above to the confirmation request to proceed.

Note: This step automatically restarts the vCenter Server services. Additionally, the Name, Hostname, and VMCA values should match the Primary Network Identifier (PNID) of the node where the certificates are being replaced. The PNID should always match the Hostname. In order to obtain the PNID please run these commands: 


We can check the PNID (Primary Network ID) by using the following commands. Finally the PNID is nothing else than the FQDN of the VCSA.

# hostname -f
# /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost



Finally this is the certificate we regenerated for the vCenter Server.

Regenerate vSphere 6.x, 7.x, and 8.0 Certificates using our internal CA (AD CS)

As already mentioned, we can of course also use custom certificates issued by a trusted public CA or a trusted internal CA like the Active Directory Certificate Services (AD CS) and shown below.

Take a snapshot of the vCenter before performing this activity. If the vCenter is in linked mode, take offline snapshots of all the linked vCenter together.


Create the Certificate Signing Request (CSR)

Connect to the bash shell of vCenter and execute the following command.

root@vCenter [ ~ ]# /usr/lib/vmware-vmca/bin/certificate-manager


First select 1. to replace the vCenter (machine SSL certificate) and then also 1. to create the CSR.


Enter the appropriate values for the certificate and finally the path where to store the CSR, I will use the /tmp folder.


We will now exit above the certificate manager.

The CSR is generated successfully and placed to /tmp/vmca_issued_csr.csr. Further also the corresponding private key for the certificate is created and placed here.


I will connect to the vCenter by using the SFTP protocol to download the CSR in order to sign it by using my internal CA (AD CS).

Connect to the vCenter by using the SFTP protocol
sftp root@vcenter.matrixpost-lab.net

changing the local path
sftp> lcd C:\Users\MarcusRath\Downloads\ca

download the CSR
sftp> get vmca_issued_csr.csr



Request a new Certificate on the internal CA by using the CSR

I will now use this CSR to create the certificate (public key) and signed by my internal CA. I will select to request a certificate below using the certificate enrollment web service.

https://<your ca>/certsrv


About how to setup you own private PKI by using the Active Directory Certificate Services (AD CS), you will find in my following post.

Set up a 2-tier PKI in Active Directory Certificate Services (AD CS) – Part 1



Select advanced certificate request.

Here we have to paste the content of our CSR into and select for the certificate template an existing web server template. Finally click on Submit.


We need to download the certificate in Base 64 encoded.


We also need to download our CA certificate (root certificate) which we need later on the vCenter Server. In my case I need to download the certificate chain because I will use here a standalone offline root CA and an intermediate CA which will finally issue the certificates.


The CA certificate chain I will download in DER format and later convert to PEM to upload and import to vCenter by using VMware PowerCLI.


To use the CA certifiate chain file (DER fomat) in vCenter, I will first convert it to PEM as mentioned by using OpenSSL.

PS> openssl pkcs7 -print_certs -inform DER -in root-chain.p7b -out root-chain-matrixpost.pem


The output of this *.pem file should look like this, we can see that finally both certificates, the root CA and intermediate CA of the internal CA are included.

Adding Root Certificate or Certificate Chain to vCenter Server and ESXi

As mentioned we can use VMware’s PowerCLI to add a root certificate or certificate chain to the trusted root store of  vCenter Server and to the certificate stores of the connected ESXi hosts.

We can install PowerCLI by entering the following command in Windows PowerShell.

PS> Install-Module -Name VMware.PowerCLI -Scope CurrentUser


Connect VMware PowerCLI to vCenter.

PS> Connect-VIServer -Server vCenter.matrixpost-lab.net


List the trusted root certificates on your vCenter Server system and the connected ESXi hosts.

PS> Get-VITrustedCertificate


Create a variable with the root certificate or certificate chain you want to add.

PS> $trustedCertChain = Get-Content "C:\Users\MarcusRath\Downloads\ca\root_chain_matrixpost.cer" -Raw


Add the root certificate or certificate chain to  vCenter Server and to the connected ESXi hosts.

PS> Add-VITrustedCertificate -PemCertificateOrChain $trustedCertChain


Verify that your new root certificate or certificate chain has been added to the trusted certificate stores.

PS> Get-VITrustedCertificate

Replace vCenter Server Certificate (Machine SSL Certificate)

Before replacing the certificate we first need to upload our new custom machine certificate for vCenter we signed previously by our internal CA.


Further we also need to upload the signing certificate for this customer certificate, in my case this is the intermediate root CA.


Then we need to run the certificate manager again to replace the Machine SSL certificate with our newly created custom certificate. As previously select 1 to replace the Machine SSL certificate.

root@vCenter [ ~ ]# /usr/lib/vmware-vmca/bin/certificate-manager


This time we need to select 2. for the next step to import our created custom certificate and signed by our internal CA.


Next we need to provide the custom certificate (machine SSL cert), the private key file and finally the signing certificate which in my case is the Machine SSL certificate.


As usual finally starting the vCenter services again will take some minutes.


Finally the vCenter Server machine SSL certificate is replaced by the new custom certificate issued by my internal CA, therefore also trusted by all domain member machines.


We can now check if the certificate is really successfully replaced by using the vSphere client and as usual by checking the SSL connection in the web browser shown below.

Checking the certificate using the vSphere client.



By checking the SSL connection in the web browser.

Refresh the Security Token Service (STS) Root Certificate

vCenter Single Sign On provides a Security Token Service (STS). The Security Token Service is a Web service that issues, validates, and renews security tokens. User can manually refresh the existing Security Token Service certificate when it expires or changes.

When user use vCenter Single Sign On with vSphere, consider the following types of certificates.

  •  SSL certificates, which are used to establish a secure connection with the Single Sign On server. These certificates are not used to validate tokens or authenticate solutions, and they are not the same SSL certificates that vCenter Server uses.
  • STS certificates, which are used for Single Sign On token validation.


STS certificates expire or change periodically and user must update or refresh them. In some environments, system administrator might implement automatic updates of the certificate. Otherwise, user can update the certificate manually using the Single Sign On administration tool. (in vSphere web client)

Source: https://support.hpe.com/hpesc/public/docDisplay?docId=sf000051034en_us&docLocale=en_US

Links

Regenerate vSphere 6.x, 7.x, and 8.0 certificates using self-signed VMCA
https://knowledge.broadcom.com/external/article?legacyId=2112283

Using vSphere Certificate Manager to Replace SSL Certificates
https://knowledge.broadcom.com/external/article/318946/how-to-use-vsphere-certificate-manager-t.html

Checking Expiration of STS Certificate on vCenter Servers
https://knowledge.broadcom.com/external/article/318968

Connecting to vCenter Server Virtual Appliance using WinSCP fails with the error: Received too large (1433299822 B) SFTP packet. Max supported packet size is 1024000 B
https://knowledge.broadcom.com/external/article?articleNumber=326317

Changing your vCenter Server’s FQDN
https://blogs.vmware.com/vsphere/2019/08/changing-your-vcenter-servers-fqdn.html

VMware Certificate Authority overview and using VMCA Root Certificates in a browser
https://blogs.vmware.com/vsphere/2015/03/vmware-certificate-authority-overview-using-vmca-root-certificates-browser.html

Add a Root Certificate to vCenter Server and ESXi
https://techdocs.broadcom.com/us/en/vmware-cis/vcf/power-cli/latest/powercli/managing-vsphere-with-powercl/managing-certificates/add-a-root-certificate-to-vcenter-server-and-esxi.html