Jump to content

Go to solution Solved by Glyn Dobson,

Recommended Posts

We have an on-prem install of UEM and today, for some reason, the cloud connector service won't start. In event viewer on the cloud connector server I see, "Module:AirWatch.CloudConnector.CloudConnectorService.TasksFailed. Message: All listener threads have terminated; killing application." No changes were made so I'm not sure what's going on. I've reinstalled the cloud connector by downloading the installer from the console but that hasn't helped. Has anyone seen this?

event viewer.png

Service Stopping.png

Link to comment
Share on other sites

Hi, 

Is there any error logs in the ACC log file?
The ACC log is located at the following path:
C:\VMware\AirWatch\Logs\CloudConnector\CloudConnector.log

If there are no error logs, you can further investigate by setting the log level to "Verbose" using the steps mentioned in following document:
https://docs.omnissa.com/ja-JP/bundle/TroubleshootingandLoggingGuide/page/IntegratedServicesLogging.html#vmware_airwatch_cloud_connector_acc
>Change ACC log level

After changing the log level, reproduce the error and check the error log again.

  • Like 1
Link to comment
Share on other sites

  • Employee

Hi Jking316,

Are there any certs (443) possibly that have expired (on that ACC or any in UEM)?  Sometimes when the ACC won't start, it's due to a certificate issue of some sort.  Take a look at the ACC logs, like Akito recommended, should have some good info in there.  Let us know how it goes.

Link to comment
Share on other sites

20 hours ago, Daniel Langley said:

Hi Jking316,

Are there any certs (443) possibly that have expired (on that ACC or any in UEM)?  Sometimes when the ACC won't start, it's due to a certificate issue of some sort.  Take a look at the ACC logs, like Akito recommended, should have some good info in there.  Let us know how it goes.

Yes, an auto-renewing cert didn't auto-renew. I got the new cert but I'm not able to download the Secure Channel Certificate Installer from the console so I can't seem to update the new cert on the servers. I have a local account with admin access so I'm not sure why I can't download it, I just get a locked door screen.

Link to comment
Share on other sites

  • Employee
3 hours ago, jking316 said:

Yes, an auto-renewing cert didn't auto-renew. I got the new cert but I'm not able to download the Secure Channel Certificate Installer from the console so I can't seem to update the new cert on the servers. I have a local account with admin access so I'm not sure why I can't download it, I just get a locked door screen.

I don't think you need the Secure Channel Cert for the ACC, it's for servers that host the AWCM service.  If the 443 cert expired on the ACC, I think you just need to update it in IIS for port 443.  Which cert are you trying to update?

Link to comment
Share on other sites

18 minutes ago, Daniel Langley said:

I don't think you need the Secure Channel Cert for the ACC, it's for servers that host the AWCM service.  If the 443 cert expired on the ACC, I think you just need to update it in IIS for port 443.  Which cert are you trying to update?

It was the ACC cert that expired. I updated it to the new cert in IIS but when checking https://server:2001/awcm it shows it's still using the old cert.

Link to comment
Share on other sites

  • Employee
6 minutes ago, jking316 said:

It was the ACC cert that expired. I updated it to the new cert in IIS but when checking https://server:2001/awcm it shows it's still using the old cert.

OK, Let's take a step back, are you trying to update the self-signed 443 server certificate on ACC (AirWatch Cloud Connector), or the AWCM (AirWatch Cloud Manager) certificate?

Link to comment
Share on other sites

22 minutes ago, Daniel Langley said:

OK, Let's take a step back, are you trying to update the self-signed 443 server certificate on ACC (AirWatch Cloud Connector), or the AWCM (AirWatch Cloud Manager) certificate?

It's the AWCM cert. They installed AWCM on the ACC when they set this up.

Link to comment
Share on other sites

1 hour ago, Daniel Langley said:

OK got it, I thought we were troubleshooting ACC.  Take a look at this -> https://docs.omnissa.com/bundle/AirWatchCloudMessaging/page/RenewSSLCertificateforAWCM.html

I tried that but just get errors and it seems as though it wants to install the full suite of software initially. I'm also not sure why a full reinstall of the system is necessary to renew a certificate.

failure 1.png

failure 2.png

Link to comment
Share on other sites

  • Employee
Posted (edited)
42 minutes ago, jking316 said:

I tried that but just get errors and it seems as though it wants to install the full suite of software initially. I'm also not sure why a full reinstall of the system is necessary to renew a certificate.

failure 1.png

failure 2.png

Yeah so you have to 'X' out the AWCM service (this feature will not be available | which uninstalls it as you've seen), then run the installer again and reinstall AWCM back, and upload the cert during the install wizard. You should ONLY be uninstalling/reinstalling the AWCM component, not anything else (not the Device Services Not Device management, etc).  It looks like possibly you're reinstalling the Console Node in the screen shot saying it can't connect to the signing service?  Looks like the AWCM can't reach the SQL server or the SQL server URL is wrong that you were using in the first error.

Edited by Daniel Langley
Link to comment
Share on other sites

23 hours ago, Daniel Langley said:

Yeah so you have to 'X' out the AWCM service (this feature will not be available | which uninstalls it as you've seen), then run the installer again and reinstall AWCM back, and upload the cert during the install wizard. You should ONLY be uninstalling/reinstalling the AWCM component, not anything else (not the Device Services Not Device management, etc).  It looks like possibly you're reinstalling the Console Node in the screen shot saying it can't connect to the signing service?  Looks like the AWCM can't reach the SQL server or the SQL server URL is wrong that you were using in the first error.

I've tried this several times now and it seems like it's trying to install the device services server services while removing the AWCM. I don't get the option to Add/Remove AirWatch features when starting the installer, I get a license agreement, a multi-server configuration setup with options to export/import installer configuration, and then the airwatch features installs. I guess I can go through and install all those services while removing AWCM and then run it again and remove all the device server services and install the AWCM. 

failure 3.png

Link to comment
Share on other sites

  • Employee

AWCM is a java application and uses its own keystore for its certificates. You should be able to use the keytool command to update the certificate. The keystore is called awcm.keystore  and is located in C:\AirWatch\AirWatch [version]\AWCM\config

To update the certificate

  1. Make a backup of the keystore (awcm.keystore file)
  2. List the keystore contents. If using the self signed cert, the password is password: 
    keytool -list -keystore acm. keystore
  3. Delete the current awcmcert certificate:
    keytool -delete -alias "awemcert" -keystore awcm.keystore
  4. Import the pfx file containing the full chain and private key. The source keystore password is the password set when creating/exporting the pfx:
    keytool -importkeystore -srckeystore myserver.pfx -destkeystore acm. keystore -deststoretype jks
  5. Change the alias to match the original certificate alias:
    keytool -changealias -alias "559811f1-4b62-42d5-995b-ec4eea8542fb" -destalias awcmcert -keystore acm.keystore
  6. List the certificates again for visual confirmation of the updated certificate:
    keytool -list -keystore awcm. keystore

     

  7. Restart the AirWatch Cloud Messaging Service from Windows Services

Note: The password for the keystore is stored in an encrypted format in the file awcm.properties. This is the password that the system will use to open the keystore. If the password is changed, AWCM will fail to start.

image.thumb.png.aaacbfffe53776ac72af36eecc3b3930.png

New certificate is now being used and matches the cert in the screenshot above:

image.png.51b7c9e037afc83d71ee4277148ad20a.png

Link to comment
Share on other sites

1 hour ago, Glyn Dobson said:

AWCM is a java application and uses its own keystore for its certificates. You should be able to use the keytool command to update the certificate. The keystore is called awcm.keystore  and is located in C:\AirWatch\AirWatch [version]\AWCM\config

To update the certificate

  1. Make a backup of the keystore (awcm.keystore file)
  2. List the keystore contents. If using the self signed cert, the password is password: 
    keytool -list -keystore acm. keystore
  3. Delete the current awcmcert certificate:
    keytool -delete -alias "awemcert" -keystore awcm.keystore
  4. Import the pfx file containing the full chain and private key. The source keystore password is the password set when creating/exporting the pfx:
    keytool -importkeystore -srckeystore myserver.pfx -destkeystore acm. keystore -deststoretype jks
  5. Change the alias to match the original certificate alias:
    keytool -changealias -alias "559811f1-4b62-42d5-995b-ec4eea8542fb" -destalias awcmcert -keystore acm.keystore
  6. List the certificates again for visual confirmation of the updated certificate:
    keytool -list -keystore awcm. keystore

     

  7. Restart the AirWatch Cloud Messaging Service from Windows Services

Note: The password for the keystore is stored in an encrypted format in the file awcm.properties. This is the password that the system will use to open the keystore. If the password is changed, AWCM will fail to start.

image.thumb.png.aaacbfffe53776ac72af36eecc3b3930.png

New certificate is now being used and matches the cert in the screenshot above:

image.png.51b7c9e037afc83d71ee4277148ad20a.png

Unfortunately the password for the keystore isn't documented anywhere so I have no clue what it was set to originally.

Link to comment
Share on other sites

  • Employee
  • Solution
17 hours ago, jking316 said:

Unfortunately the password for the keystore isn't documented anywhere so I have no clue what it was set to originally.

Unfortunately, you will need to re-install. There is no way to re-encrypt the password outside of the installer. See this KB for the steps:

https://kb.omnissa.com/s/article/2960970

Link to comment
Share on other sites

33 minutes ago, Glyn Dobson said:

Unfortunately, you will need to re-install. There is no way to re-encrypt the password outside of the installer. See this KB for the steps:

https://kb.omnissa.com/s/article/2960970

Okay, that solved it. Had to reinstall AWCM on the device services server as well. Guess it needs to be on both the DS and AWCM servers. Thanks so much for the help.

Link to comment
Share on other sites

  • Employee
52 minutes ago, jking316 said:

Okay, that solved it. Had to reinstall AWCM on the device services server as well. Guess it needs to be on both the DS and AWCM servers. Thanks so much for the help.

You shouldn't need AWCM on both servers, only one instance is needed. The AWCM instance used by the system is shown under site URL's in the console (example below is from a cloud UEM tenant but the screen is the same). If cases where AWCM is not on its own sever, it is typically installed on DS:

image.thumb.png.a32935e433db871dc2944302c964024c.png

Link to comment
Share on other sites

13 minutes ago, Glyn Dobson said:

You shouldn't need AWCM on both servers, only one instance is needed. The AWCM instance used by the system is shown under site URL's in the console (example below is from a cloud UEM tenant but the screen is the same). If cases where AWCM is not on its own sever, it is typically installed on DS:

image.thumb.png.a32935e433db871dc2944302c964024c.png

It must have been on the DS server originally then because nothing was working until I removed and reinstalled AWCM there. Not sure how or why it ended up installed on the ACC server. Luckily we are in the process of moving off this on-prem installation to SaaS.

Link to comment
Share on other sites

  • Employee
12 minutes ago, jking316 said:

It must have been on the DS server originally then because nothing was working until I removed and reinstalled AWCM there. Not sure how or why it ended up installed on the ACC server. Luckily we are in the process of moving off this on-prem installation to SaaS.

Glad you managed to get it resolved. SaaS definitely has many benefits over hosting your own deployment and you'll be able to take advantage of the newer features that have not made their way into On-Prem.

  • Like 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...