What started off in hopes of finding some great, unknown/undocumented Networker bug turned into an annoying support experience, but even better for you the reader a funny story (funny if it doesn’t happen to you!). In trying to reverse engineer an environment, much of which was not or was poorly documented, I needed to determine how our backups were configured so off I went to review the EMC Networker 8.0 configuration, only problem was I couldn’t log in. Now I
knew was pretty certain it was running correctly as I could see the backups from Networker getting larger in Data Domain, I would get email alerts etc. The only, seemingly, relevant post in the support.emc.com site suggested the web server was not installed/running correctly but GST started, verified the service was installed properly by running chkconfig –list gst and I wasn’t seeing any errors in the daemon.raw log about the web server not being found; so I continued my search. Even the ECN was lacking in posts/questions on Networker (I suspect because I am not an early fan of it, probably not to many people using it). I also tried to stopping/starting Networker by running /etc/init.d/networker stop and start – again no errors. After my initial search turned up empty, the next step was to open a support ticket – after all I would have to wait a while anyways so kick that process into gear.
Once my ticket was created I went back to basics; name resolution was working on the Networker servers, The resolv.conf and and interface configuration files all appeared correct, I could ping itself via name and IP, I could ping it from the Windows machine I was running the console from. I also attempted to access the console via IP to further eliminate name resolution as the problem. I verified that iptables was not blocking access and then proceeded to smash my head into my keyboard. I rebooted the Networker VM (3 times). Once on the phone with support, they had me check for an additional service not listed in the KB – the nsrd service. You can check this in a RHEL/CentOS environment by running chkconfig –list nsrd – it found nothing. The tech suggested that there was some problem and it needed to be re-installed, even though running ps -ef | grep nsrd showed a process running (I have never used Networker so wasn’t about to argue with the support engineer). At this point he told me to re-install the NMC and left me me for some fun. I decided it was about time to backup the config, the following files according to the Networker 8 documentation are what you should care about:
“bAutoWidth”: true ,
“sLengthMenu”: “Display _MENU_ records per page”,
“sZeroRecords”: “Nothing found – sorry”,
“sInfo”: “Showing _START_ to _END_ of _TOTAL_ records”,
“sInfoEmpty”: “Showing 0 to 0 of 0 records”,
“sSearch”: “Search: “,
“sInfoFiltered”: “(filtered from _MAX_ total records)”
Additionally, there are some operating system files which are suggested to be backed up (personally I backed up these files to a separate directory because I am CDO like that)
cp /etc/rpc /etc/rpc.orig
cp /etc/ld.so.conf /etc/ld.so.conf.orig
Okay, now I have a backup and we just happened to have a consultant in the office the next day to help with a Data Domain problem we were having, he was originally involved in the design of the backup systems so he had a quick peak. After a week plus of the support engineer not calling me back the consultant ran a simple command that shed light on the whole thing: nsrwatch and low and behold…
Registration Alert event: NetWorker evaluation mode has expired.
Regstration Alert event: Server is disabled: Install base enabler
They never finished setting up Networker – at least I only lost about 2 hours of my life chasing the support engineer down for help and its a DR site so no new data is going in.
- Check to make sure the previous person actually installed and registered it properly!
- Verify there are no errors in the daemon.raw log after service startup
- Verify DNS resolution and proper IP connectivity
- Check that the networker, gst and nsrd services are running, if either are missing you may need to reinstall
- If they are running, restart both by typing the command: serivce gst stop, followed by service gst start. Repeat for nsrd and networker.
- Call support and hope you don’t get the engineer I had!