When you move a ESXi system disk to other machine, you need to recreate the vmknic0 in order to reassign the MAC address.
If you don’t, the vmknic0 inherits the MAC address from other machine. This means, if you ever reintroduce the other machine on the same network, MAC address is duped, and you will have a partial but somewhat catastorophic network problem as you cannot reach to the ESXi server.
I just went through this. I put myself in a tight spot, and therefore this entry must be made for my future self.
Background
I have been using a pfSense for nearly 15 years since switching over from dd-wrt. First, I was using a small machine running a Linux, and run VirtuaiBox to host a pfSense. This lasted for years until the internet speed reached 10Mbps. To upgrade, I switched to use VMware/ESXi for better performace. Now, looking into the possibility of gigabit, I decided to upgrade the ESXi host.
Both machines got ESXi normally, and I moved the pfSense config rather than copying VM instance. One day, I shut both ESXi hosts, and the new ESXi’s pfSense became the firewall. All is good.
Then, since the new box has a lot of storage and CPU power, I decided to put a new FreeBSD server. Up to this point, no problems.
I started taxing the new box, then, the new ESXi host malfunctioned. Most likely cause is overheatng but that is not relevant here. (So far, it appears that NVME slot is not happy.) With it, I decided to bring back the old ESXi host machine, moved the system SSD, and all Is good.
TL;DR – ESXi-A – replaced by ESXi-B – went back to ESXi-A with the disk of B.
Crisis
Problem started
with the reintroduction of ESXi-B, I could not access ESXi-A, because the Disk-B is still using the MAC address of ESXi-B. IOW vmknic0 of both ESXi hosts are using the same MAC.
I don’t have to use the ESXi-B but naturally, I have it, and want to use it after the hardware issue resolved.
Compounding the problem
After identifying the MAC addr dupe, I googled and found this. At this point, I was not aware that I can use the shell session of ESXi on the physical host. So, I was SSH-ing into it.
# esxcfg-vmknic -d -p Management\ Network
Then, OBVIOUSLY the networe connection is lost.
Panic sets in
I should have studied VMware a little better. Anyhow, I reboot the box, with a monitor and keyboard. Goes into ESXi’s 19th century user interface. At this point, ESXi web interface is lisening on http://0.0.0.0 and no go.
I cannot recreate the management network as the pfSense VM auto-starts and blocks from resetting the network config.
Shoot.
Obi-wan (YouTube), you are my only hope
Another round of googling lands on a random YouTube, and “Hit F1 go get to the shell”. Today, first time, I learned that I can use the shell of ESXi on the physical host.
Hit alt-F1 (like any Linux terminal session), and the console awaits the username/password.
I was finally in clear. Not completly but I know how now.
On the way to recovery
Now, I can do the 2nd part of creating the new vmknic0.
esxcfg-vmknic -a -i 192.168.1.2 -n 255.255.255.0 Management\ Network
which picked up the MAC addr from ESXi-A.
Also, I lost the ESXi’s virtual switch using vmknic0 so that also needed to re-assign.
Lessons Learned
- vmknic0 MAC address Is sticky and you cannot move the disk from one ESXi host to the other without recreating it
- To do so, you enable the shell, and you have access to the CLI on the physical host
- Before doing so, it is your firewall running host, print out all the commands you need beforehand.
- Have a hot-spot capable phone just in case so you can google in comfort.