jump to navigation

HP NIC nonsense August 17, 2009

Posted by jamesisaac in Uncategorized.
trackback

A frustrating hour spent tonight on something that should have taken five minutes… and the issue isn’t even solved, yet.

I have an HP DL580 G2 with the standard motherboard Broadcom gigabit NICs, NC7782 in HP terminology. The NICs are teamed for fault-tolerance. In order to set up our backup-to-disk connection, I planned on adding a virtual NIC by using the VLAN feature of the HP network configuration utility and converting my “untagged” physical network ports to “tagged” VLAN ports, with one virtual NIC retaining the same ip information as the teamed adapter previously had, and the second virtual NIC acquiring an address in the backup network scope. Sounds pretty simple, right? Just go into HP network config, add the VLAN interfaces, and address them, and you’re done.

Now, from previous experience, I know that adding the VLAN configuration should create a brand-new virtual NIC with no ip addressing, as well as disabling the current NICs, so this isn’t a job to be done remotely (unless you have the RIB all confgured. Actually, come to think of it, I could have done this through our KVM-over-IP, but it turns out that it was a good thing that I didn’t.) So, a trip to the data center was in order.

Five minutes after arrival, I have the VLAN 802.1q information entered and hit “apply” in the HP network configuration utility. And wait.

And wait.

10 minutes later, the HP network configuration utility is still on “configuring adapters, please wait… this may take a few seconds per adapter” or some such verbiage. Clearly something is not going well.

The Network control panel shows that my physical NICs are now marked as “disabled” and what used to be the “Teamed” virtual adapter has turned into one of the VLAN-assigned adapters, but is also disabled. A fourth NIC has arrived – the second VLAN-assigned adapter. It is enabled but unplugged, which is odd because nothing has changed with the cabling. I’m guessing that it is unplugged because the physical NICs are disabled.

I use Task Manager to kill the HP network configuration utility. I can’t enable the network adapters. They show as disabled, and when I right-click and choose “enable”, Windows tells me that they are enabling, but they stay disabled.

Reboot.

Seven minutes later, I’m looking at the “Preparing network connections.” Crap. This stays stuck for about three minutes. I was about ready to power off and boot into safe mode. Log in, find the adapters – they’re still disabled. Not good. Device Manager shows them as NC7782 with little red X’s through them. Why? What happened?

The HP network configuration utility shows NO adapters – probably because, again, they’re disabled. But the HP network configuration utility was the tool that disabled them~! So now I’m stuck.

I log into a different machine, go to http://itrc.hp.com, and search for drivers. I download the latest Broadcom driver and the HP ncu onto a USB thumb drive that I happen to have in the data center kit box (note: always have a box of spare bits and pieces tucked away in your rack). I load the drivers and the HP ncu, and reboot.

And wait.

10 minutes later, I’m again faced with “Preparing network connections…” and after logging in, I have disabled network adapters – they’re using the latest versions of the drivers, but still disabled. Crap! Now what? Do I have to regedit something to enable these drivers? I look around and find a post that suggests uninstalling the drivers and rebooting, then going through a whole routine of flashing BIOS, firmware, and reloading drivers. I removed one NIC, uninstalled the driver from the other one, then – on a whim – rescanned the hardware in device manager. What’s this? New NIC detected?

Huge sigh of relief. Windows, all by itself, sees the NICs as new devices and loads the HP driver and enables the network cards. I start up the HP ncu (actually, I seriously considered just leaving it all alone and not dealing with ncu anymore), team the adapters, cross my fingers as the pNICs are disabled and the new Teamed interface is created, then everything is enabled again. I set the ip address and we’re back in business.

I’m still at a loss to explain why the server disabled the NICs and could not re-enable them. I’ve used VLANs through the HP ncu before, and it is usually a straight-ahead configuration. My guess is that this is an old server, probably not up to par with regards to both Windows patches and HP driver updates, and something in the OS / driver stack conflicted with the changes I was attempting to make, which left it in a disabled state.

Lessons learned:

  • nothing is ever as easy as you think it should be.
  • Don’t do maintenance on a customer-facing server on Sunday night.
  • Purchase a new NIC for the backup network instead of trying to “virtualize” the existing NICs by adding VLAN tags.
Advertisements

Comments»

No comments yet — be the first.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: