The last implementation of a SCOM 2016 environment included 2 Gateway servers and integration to Operations Management Suite (OMS) in Azure. I used the final release of SCOM 2016 to deploy the servers and when all the infrastructure objects were in place I updated it with the recent Update Rollup (UR1). In the implementation phase of the project the OMS integration was completed successfully and the servers started to send their data to Azure. A few weeks later Microsoft released UR2 and because our policy is to hand over the environment to the customer with fully patched and running the latest version, I decided to install UR2. It was out on the market for more than a Month, so I thought if anything was wrong, it would have already been surfaced.
The UR2 installation went without any issue, however after a few Hours I realized that one of the Gateway Servers health is in critical state. I checked the Health Explorer and saw the following error
It says that the monitor is unable to find the Advisor Connector on this machine. I checked the state changes of the Monitor and it looks it went to Critical just after installing UR2 on the Gateway servers (UR1 package had no update for Gateway Servers).
I started my little investigation to find out what could be the issue here. I started with looking for articles on the internet about similar issues, however the only thing I could find is that the reason could be that the server is on wrong (different than other servers in the environment) UR level. I knew it is not the problem, because the issue is started when I installed the latest UR on the server.
In the next step I was looking for information about the monitor and try to find where it gets the information about the Advisor Connector. Again, not much on the internet I could find. Then I turned to my old and trusty friend to analyze the monitor. It is called MPViewer.exe. Everybody who is in a deep relationship with SCOM since 2007 know this little application. It proved itself a very useful friend over the years. So, I checked which Management Pack contains this monitor
I loaded it into the MPViewer and checked the configuration of the monitor. I remembered that this Management Pack has an updated version in the UR2 pack, so I used that file
The monitor does not show any useful information, so I had to look further. In the Classes folder there is a class called AdvisorConnector Class which means this object is discovered by this management pack. IT was a good news, because then it has to contain the discovery for this class as well, so I turned my attention to the Discoveries folder there it is. It is called Advisor Connector Discovery
As you can see the discovery checks a registry value to see what is the version of the Advisor Connector. So, I decided to compare the entry on the 2 Gateway Servers. I went to the specified registry key on both servers and voila. There goes the issue. On the second Gateway Server the HKLM\Software\Microsoft\System Center Operations Manager registry key was basically empty. On values or keys in it other than the usual Default value. After that I checked the Gateway folder in the installation path and saw that on the Gateway server with issue there are at least 30 items less in the folder than on the other server. This two things caused me a feeling that something went wrong on this server.
Unfortunately, I could not identify the reason why the Advisor Connector and some other things were missing from this server. And because I could not identify exactly what other things were missing from this server, I decided to re-install the Gateway feature on this server. To do this, there are several task needed to be done:
- Make sure no agent is reporting primarily to this Gateway Server
- Uninstall any updates related to the Gateway installation using Programs and Features (Installed Updates) on the server
- Uninstall System Center Operations Manager on this servers using Programs and Features
- Delete all the folders/files left at the installation folder
- Install the Gateway feature again
- Install the latest UR on the server
After I went through all of these steps and the server was up and running again, the Advisor Connector information was completely available and the monitor stayed Healthy. After that, I compared the installation folders and registry keys. Those also looked identical on both Gateway Servers.