On-Prem Cloud Connector Stopped

[jking316]

We have an on-prem install of UEM and today, for some reason, the cloud connector service won't start. In event viewer on the cloud connector server I see, "Module:AirWatch.CloudConnector.CloudConnectorService.TasksFailed. Message: All listener threads have terminated; killing application." No changes were made so I'm not sure what's going on. I've reinstalled the cloud connector by downloading the installer from the console but that hasn't helped. Has anyone seen this?

[Akito Ogushi]

Hi, 

Is there any error logs in the ACC log file?
The ACC log is located at the following path:
C:\VMware\AirWatch\Logs\CloudConnector\CloudConnector.log

If there are no error logs, you can further investigate by setting the log level to "Verbose" using the steps mentioned in following document:
https://docs.omnissa.com/ja-JP/bundle/TroubleshootingandLoggingGuide/page/IntegratedServicesLogging.html#vmware_airwatch_cloud_connector_acc
>Change ACC log level

After changing the log level, reproduce the error and check the error log again.

[Daniel Langley]

Hi Jking316,

Are there any certs (443) possibly that have expired (on that ACC or any in UEM)?  Sometimes when the ACC won't start, it's due to a certificate issue of some sort.  Take a look at the ACC logs, like Akito recommended, should have some good info in there.  Let us know how it goes.

[jking316]

20 hours ago, Daniel Langley said:

Hi Jking316,

Are there any certs (443) possibly that have expired (on that ACC or any in UEM)?  Sometimes when the ACC won't start, it's due to a certificate issue of some sort.  Take a look at the ACC logs, like Akito recommended, should have some good info in there.  Let us know how it goes.

Yes, an auto-renewing cert didn't auto-renew. I got the new cert but I'm not able to download the Secure Channel Certificate Installer from the console so I can't seem to update the new cert on the servers. I have a local account with admin access so I'm not sure why I can't download it, I just get a locked door screen.

[Daniel Langley]

3 hours ago, jking316 said:

Yes, an auto-renewing cert didn't auto-renew. I got the new cert but I'm not able to download the Secure Channel Certificate Installer from the console so I can't seem to update the new cert on the servers. I have a local account with admin access so I'm not sure why I can't download it, I just get a locked door screen.

I don't think you need the Secure Channel Cert for the ACC, it's for servers that host the AWCM service.  If the 443 cert expired on the ACC, I think you just need to update it in IIS for port 443.  Which cert are you trying to update?

[jking316]

18 minutes ago, Daniel Langley said:

I don't think you need the Secure Channel Cert for the ACC, it's for servers that host the AWCM service.  If the 443 cert expired on the ACC, I think you just need to update it in IIS for port 443.  Which cert are you trying to update?

It was the ACC cert that expired. I updated it to the new cert in IIS but when checking https://server:2001/awcm it shows it's still using the old cert.

[Daniel Langley]

6 minutes ago, jking316 said:

It was the ACC cert that expired. I updated it to the new cert in IIS but when checking https://server:2001/awcm it shows it's still using the old cert.

OK, Let's take a step back, are you trying to update the self-signed 443 server certificate on ACC (AirWatch Cloud Connector), or the AWCM (AirWatch Cloud Manager) certificate?

[jking316]

22 minutes ago, Daniel Langley said:

OK, Let's take a step back, are you trying to update the self-signed 443 server certificate on ACC (AirWatch Cloud Connector), or the AWCM (AirWatch Cloud Manager) certificate?

It's the AWCM cert. They installed AWCM on the ACC when they set this up.

[Daniel Langley]

1 minute ago, jking316 said:

It's the AWCM cert. They installed AWCM on the ACC when they set this up.

OK got it, I thought we were troubleshooting ACC.  Take a look at this -> https://docs.omnissa.com/bundle/AirWatchCloudMessaging/page/RenewSSLCertificateforAWCM.html

[jking316]

1 hour ago, Daniel Langley said:

OK got it, I thought we were troubleshooting ACC.  Take a look at this -> https://docs.omnissa.com/bundle/AirWatchCloudMessaging/page/RenewSSLCertificateforAWCM.html

I tried that but just get errors and it seems as though it wants to install the full suite of software initially. I'm also not sure why a full reinstall of the system is necessary to renew a certificate.

[Daniel Langley]

42 minutes ago, jking316 said:

I tried that but just get errors and it seems as though it wants to install the full suite of software initially. I'm also not sure why a full reinstall of the system is necessary to renew a certificate.

Yeah so you have to 'X' out the AWCM service (this feature will not be available | which uninstalls it as you've seen), then run the installer again and reinstall AWCM back, and upload the cert during the install wizard. You should ONLY be uninstalling/reinstalling the AWCM component, not anything else (not the Device Services Not Device management, etc).  It looks like possibly you're reinstalling the Console Node in the screen shot saying it can't connect to the signing service?  Looks like the AWCM can't reach the SQL server or the SQL server URL is wrong that you were using in the first error.

Edited June 21 by Daniel Langley

[jking316]

23 hours ago, Daniel Langley said:

Yeah so you have to 'X' out the AWCM service (this feature will not be available | which uninstalls it as you've seen), then run the installer again and reinstall AWCM back, and upload the cert during the install wizard. You should ONLY be uninstalling/reinstalling the AWCM component, not anything else (not the Device Services Not Device management, etc).  It looks like possibly you're reinstalling the Console Node in the screen shot saying it can't connect to the signing service?  Looks like the AWCM can't reach the SQL server or the SQL server URL is wrong that you were using in the first error.

I've tried this several times now and it seems like it's trying to install the device services server services while removing the AWCM. I don't get the option to Add/Remove AirWatch features when starting the installer, I get a license agreement, a multi-server configuration setup with options to export/import installer configuration, and then the airwatch features installs. I guess I can go through and install all those services while removing AWCM and then run it again and remove all the device server services and install the AWCM. 

[Glyn Dobson]

AWCM is a java application and uses its own keystore for its certificates. You should be able to use the keytool command to update the certificate. The keystore is called awcm.keystore  and is located in C:\AirWatch\AirWatch [version]\AWCM\config

To update the certificate

1. Make a backup of the keystore (awcm.keystore file)
2. List the keystore contents. If using the self signed cert, the password is password: 

   keytool -list -keystore acm. keystore

3. Delete the current awcmcert certificate:

   keytool -delete -alias "awemcert" -keystore awcm.keystore

4. Import the pfx file containing the full chain and private key. The source keystore password is the password set when creating/exporting the pfx:

   keytool -importkeystore -srckeystore myserver.pfx -destkeystore acm. keystore -deststoretype jks

5. Change the alias to match the original certificate alias:

   keytool -changealias -alias "559811f1-4b62-42d5-995b-ec4eea8542fb" -destalias awcmcert -keystore acm.keystore

6. List the certificates again for visual confirmation of the updated certificate:

   keytool -list -keystore awcm. keystore

    

7. Restart the AirWatch Cloud Messaging Service from Windows Services

Note: The password for the keystore is stored in an encrypted format in the file awcm.properties. This is the password that the system will use to open the keystore. If the password is changed, AWCM will fail to start.

New certificate is now being used and matches the cert in the screenshot above:

[jking316]

1 hour ago, Glyn Dobson said:

AWCM is a java application and uses its own keystore for its certificates. You should be able to use the keytool command to update the certificate. The keystore is called awcm.keystore  and is located in C:\AirWatch\AirWatch [version]\AWCM\config

To update the certificate

1. Make a backup of the keystore (awcm.keystore file)
2. List the keystore contents. If using the self signed cert, the password is password: 

   keytool -list -keystore acm. keystore

3. Delete the current awcmcert certificate:

   keytool -delete -alias "awemcert" -keystore awcm.keystore

4. Import the pfx file containing the full chain and private key. The source keystore password is the password set when creating/exporting the pfx:

   keytool -importkeystore -srckeystore myserver.pfx -destkeystore acm. keystore -deststoretype jks

5. Change the alias to match the original certificate alias:

   keytool -changealias -alias "559811f1-4b62-42d5-995b-ec4eea8542fb" -destalias awcmcert -keystore acm.keystore

6. List the certificates again for visual confirmation of the updated certificate:

   keytool -list -keystore awcm. keystore

    

7. Restart the AirWatch Cloud Messaging Service from Windows Services

Note: The password for the keystore is stored in an encrypted format in the file awcm.properties. This is the password that the system will use to open the keystore. If the password is changed, AWCM will fail to start.

New certificate is now being used and matches the cert in the screenshot above:

Unfortunately the password for the keystore isn't documented anywhere so I have no clue what it was set to originally.

[Glyn Dobson]

17 hours ago, jking316 said:

Unfortunately the password for the keystore isn't documented anywhere so I have no clue what it was set to originally.

Unfortunately, you will need to re-install. There is no way to re-encrypt the password outside of the installer. See this KB for the steps:

https://kb.omnissa.com/s/article/2960970

[jking316]

33 minutes ago, Glyn Dobson said:

Unfortunately, you will need to re-install. There is no way to re-encrypt the password outside of the installer. See this KB for the steps:

https://kb.omnissa.com/s/article/2960970

Okay, that solved it. Had to reinstall AWCM on the device services server as well. Guess it needs to be on both the DS and AWCM servers. Thanks so much for the help.

[Glyn Dobson]

52 minutes ago, jking316 said:

Okay, that solved it. Had to reinstall AWCM on the device services server as well. Guess it needs to be on both the DS and AWCM servers. Thanks so much for the help.

You shouldn't need AWCM on both servers, only one instance is needed. The AWCM instance used by the system is shown under site URL's in the console (example below is from a cloud UEM tenant but the screen is the same). If cases where AWCM is not on its own sever, it is typically installed on DS:

[jking316]

13 minutes ago, Glyn Dobson said:

You shouldn't need AWCM on both servers, only one instance is needed. The AWCM instance used by the system is shown under site URL's in the console (example below is from a cloud UEM tenant but the screen is the same). If cases where AWCM is not on its own sever, it is typically installed on DS:

It must have been on the DS server originally then because nothing was working until I removed and reinstalled AWCM there. Not sure how or why it ended up installed on the ACC server. Luckily we are in the process of moving off this on-prem installation to SaaS.

[Glyn Dobson]

12 minutes ago, jking316 said:

It must have been on the DS server originally then because nothing was working until I removed and reinstalled AWCM there. Not sure how or why it ended up installed on the ACC server. Luckily we are in the process of moving off this on-prem installation to SaaS.

Glad you managed to get it resolved. SaaS definitely has many benefits over hosting your own deployment and you'll be able to take advantage of the newer features that have not made their way into On-Prem.