The last few months I participated in a couple of NSX-T migrations. Although we ran into different issues during the migrations, we were able to finish all of the migrations successfully. Today I want to share some of the lessons I learned during these migration. In Dutch we have a phrase “(Een open deur intrappen/Kicking down an open door)” which means something like “stating the obvious”. Some of the things I’m going to describe might seem like open doors but because some things are easy to overlook, I’m going to point them out anyway.
Choose your migration strategy wisely and plan accordingly
There are a few different methods to migrate from V to T. They are all very well explained in the official VMware documentation and the other blogs around the world so I’m not gonna deep-dive into that. Be sure to pick the scenario which suits your needs for the best.
In every scenario their are (some) steps described that you must perform before you start the migration and of course some steps you must take afterwards. And don’t forget the pre-requisites that need te be in place.
I Would advise you to create a checklist of every action you need to perform, so you can keep track of your progress during migration. Be sure to include configuration values of BGP/OSPF timers etc, so you have all the information you need at hand.
Don’t forget to test Before (and after)
Most of the time we only plan testing at the end of a migration or upgrade, but its way more useful to do a test before ánd after migration. For example; Which reply do you get from the load balancer in V. It’s easy to compare this with the result after migration so you know what to expect. This also applies to applications running on the platform. So engage your application owners/engineers to perform some tests to determine a baselines which can be validated after migration. This saves you from trying to troubleshoot things after migrating that were broken in the first place.
We also used iPERF to determine the bandwidth between different environments and redo these test after the migration to see if there was any difference. I will make another blogpost about iPERF in the near future ;).
Work in shifts
One of my colleagues did this before on another migration and I must say it was a real eye opener for me. On some migration we exceeded the expected working time by more than 12 hrs. Although we do have our secret superpower(coffee) there is a limit to your productivity. So if you have the colleagues and the hours on the project I strongly suggest to team up.
We did a kickoff with all the engineers involved at the start of the migration and as soon as we were up to migrating hosts only one engineer took over and the other waited for their turn. After a predetermined number of hours, he/she hands over the mouse to the next engineer
Eat / Sleep / Migrate / Repeat
This may sound obvious but from my own experience I know how temping it can be to get everything to work and to finish the migration as soon as possible. In this rush it’s easy to forget about yourself. When everything is online and functional and there is no direct impact it can be much more productive to take a step back and get some rest. Get something to eat, take a short walk outside or, even take a quick nap. It might not give you direct insights in the next steps, but it can help you clear your mind and give you some new energy to focus on the problem at hand.
If you have a problem, If no one else can help…
Although the full tagline from the A-team series may not be completely accurate here (You will have no problem finding GSS), the first part surely applies. I often find myself desperate lost in log files looking for errors/warning/clues to fix the issues. Although this isn’t a bad idea I would keep in mind that GSS has far more resources to dive in the logfiles and looking for specifiek strings to get a clue of what wen’t/goes wrong.
After you created the case upload the logfiles as soon as you can. Select only components you want to include. Too much nodes for collection costs more time and if you have an Severity 1, you can save some time by only collecting the useful logs.
Another quick tip. By default the log-age will export all data that is available. During upgrade/ migration however, only the current data is relevant, so size down the log collection to speed it up.
Be sure you know the environment and the customer, so you can evaluate impact of changes GSS wants to make.
If you have the right support contract you are entitled to log a pro-active support case. You can use these cases to get VMware engaged before you start the migration. There are a couple of options within the pro-active support case situation. From my opinion the 2 most important below;
- Technical or procedural query / Upgrade Plan Review Request – This can be used for VMware Support to review plans for any maintenance or upgrade. This can be opened any time at least 10 business days prior to the planned work, and is closed once the review is completed.
- Upgrade Awareness / Maintenance Awareness – This SR type can be used for any other planned maintenance (VMware or 3rd-party product) for VMware to be aware of any key details and provide reactive support if/when needed during that maintenance window, and is closed at the end of the window.
There are some guidelines to get pro-active support because you need to give VMware some time to arrange everything so keep that in mind.
More information in this KB; https://kb.vmware.com/s/article/2146880
As you may well know, VMware uses Zoom for their support calls. To be one step ahead in the case you need them, I suggest to download it to your management server and test the connectivity. You don’t want to be running to your firewall admin in the middle of a migration to get the right ports open. Be sure to test it from a system which you can use even when NSX-V/T is down.
Enjoy the proces
It sometimes may seem a rocky road you are riding during a V to T migration, but remember you are doing a major change. The way V and T work are somehow comparable, but also very different. So to assume it’s just a simple migration is not fair. Keep in mind you are doing something very complex and that it’s okay to hit a few snags. At the end of the day you will get it to work, so have fun during the process and accept that you’ll stumble. Just get back up and continue. “Keep your eyes on the prize”: a fully migrated environment from NSX-V to T.