Migrating to VSphere 6

The ‘big’ migration

I don’t have a big VMware installation. A pair of host servers with shared iSCSI storage solves our needs. It is pretty affordable, compared to big EMC SANs, or the newer complex setups from Nutanix, or their non-VMware compatible competitors.
My little environment runs Essentials Plus, which gives me a bit of access to some HA features, but not some of the more advanced stuff found in something amazing like Enterprise, or Enterprise Plus. Much as I would like to check out Operations Management, it doesn’t seem to be included in my license (even though it’s listed on my downloads page).

I got some new hardware recently. A pair of dual-cpu 10-core servers with 256gb memory, and 10 1gbit ports (each), and a TrueNAS Z20-HA w/ 10gbe. It was to replace my aging HP solution, that has seen better days. Everything was going well, complete with the usual oddities that VMware brings, such as doing round-robin iSCSI connections with the 1gbit connections on the host servers (no LAGG connections here). Synthetic benchmarks were showing me performance 4+ times faster than the existing system. They’re synthetic, so they need to be doubted, but it’s nice to have a repeatable benchmark showing some sort of major improvement.

The (invalid) VMs

Anyway, onto the meat of the problem: A bunch of our Virtual Machines were created back in VMware 4. Then we migrated to 5.1, and the environment sort of sat there. Partially, because I was simply too afraid to push the entire load to one of the host servers and watch the thing cringe while doing the upgrade on the other.

Now with the new servers, I’ve taken to installing VMware vSphere 6 on them. Migrating the VMs from one system to the other was no problem, until I encountered one that had been around for a very long time. I went and registered the VM in the new servers, and then scratched my head when it said “vmguest (invalid).” Invalid? What?

I went back to the old servers, re-registered the VM. Works fine. Back the new servers, re-try the registration. Invalid. Crap.

The internet has a few theories, most of which are related to 5.x installs of vSphere. The most promising included deleting a .vmxf file, but didn’t work. I was close to giving up when it occurred to me that the VM has been around for a long time. So, what if I try to clone it first? Registered the cloned vm in the new servers - works fine. So, what’s the difference?

So I downloaded the .vmx file of the original VM, as well as one from the new VM, and started comparing them. My old VM’s had some lines about hbr_filter as well as some old scsi0:0.filter lines. I remembered (being a PHDVirtual customer) that there was an issue with my PHDVirtual backups and those scsi0:0.filter lines (though at this time, that problem eludes me). The new VMs didn’t have any of these lines.

hbr_filter.configGen = ""
scsi0:0.filters = ""
hbr_filter.rpo = ""
hbr_filter.destination = ""
hbr_filter.port = "0"
hbr_filter.gid = ""
hbr_filter.protocol = ""
hbr_filter.quiesce = "TRUE"
hbr_filter.opp = "TRUE"
hbr_filter.pause = "TRUE"

I did the odd idea: I deleted the lines.

Register the modified .vmx and …. It looked normal. Powered it on. Answered the question (moved or copied). Everything looks fine.

I didn’t go through the work to find out if it’s the .filters lines from the scsi0:0 lines, or if it’s the hbr_filter.* lines, but at this point, it just doesn’t seem to matter. The VM works fine, gets upgraded fine, etc. I couldn’t be happier with how easy this is to finish.

Migrating storage (with Essentials Plus)

I sort of cheated a little bit on my migration. I have a valid license, and licensed my vCenter install. When I went to license the ESXi installations, I realized (again) that I’d lose access to storage-vmotion. One of the cool things about storage-vmotion is that it would let me shutdown the VM on the old hardware, register & boot the VM on the new host servers, then vmotion the storage (vmdks, etc) to the new TrueNAS unit, while the machine is busy doing what it normally does. My downtime per system (and active time spent babysitting it) drops to just a few minutes, instead of having to do the copy from storage unit to storage unit by hand. This works amazingly well, and hopefully VMware will forgive me for having usage for 2 weeks, before I decommission the old 5.x installation.

Something I didn’t try

Since my Essentials Plus installation actually has 3 server licenses, I thought about an idea, but I wasn’t willing to potentially risk my current environment’s HA setup to try it.

In some theories, I could add the servers in the current installation to the new installation (though, it might screw with licensing). Then I could vmotion the machine to the new hardware, without ever turning it off. Potentially even including a storage vmotion. But I wasn’t willing to risk VMware’s HA doing something funky with the servers while I try that, plus, there just isn’t quite enough resources in the existing systems to do them one at a time anyway. It was a nice thought, though.

So this brings my vSphere 6 migration experience to an end. New hardware makes for a substantially better experience, and abusing the trial licensing for a week gives me an easy-ish migration. Still wish the new UI wasn’t Flash-based, but it’s definitely improved since the 5.x version. All in all, happy.