ESXi’s VMFSL size: systemMediaSize=min to the boot option of installer

I bought a fanless Intel N5100 + 4 x Intel 225l ethernet ports on AliExpress. ESXi 7.0u3 does not have the driver for this, and you have to create your own ISO instailler with the community driver added, but that’s another story some other time.
Anyhow, I put on small memory (8GB) and small-ish NVME. My original intension was to install pfSense and be done with it. I however decided to install ESXi, and got surprized to see 120GB of disk space is occupied by VMFSL.
Apparently, that’s the ESXi’s system. I sure don’t need that much space for ESXi system. After a few installation attempts later, finally found the answer how to make this partition smaller.
The credit goes to here.
TL;DR
1. ESXi always takes up 120GB. I did not pay attention to this before. All other home ESXi machine’s system disk is gigantic to notice. Sigh.
2. At the very begging of ESXi installer, press “Shift-O” to bring up the boot options.
3.append systemMediaSize=min to it.
4. Hit return and continue the rest of installation.

The valid option values are “min”, “small”, “default”, “max”. I think for my needs, “min” is the only value I care.

Moving ESXi disk to other machine – postmortem

When you move a ESXi system disk to other machine, you need to recreate the vmknic0 in order to reassign the MAC address.
If you don’t, the vmknic0 inherits the MAC address from other machine. This means, if you ever reintroduce the other machine on the same network, MAC address is duped, and you will have a partial but somewhat catastorophic network problem as you cannot reach to the ESXi server.
I just went through this. I put myself in a tight spot, and therefore this entry must be made for my future self.

Background

I have been using a pfSense for nearly 15 years since switching over from dd-wrt. First, I was using a small machine running a Linux, and run VirtuaiBox to host a pfSense. This lasted for years until the internet speed reached 10Mbps. To upgrade, I switched to use VMware/ESXi for better performace. Now, looking into the possibility of gigabit, I decided to upgrade the ESXi host.
Both machines got ESXi normally, and I moved the pfSense config rather than copying VM instance. One day, I shut both ESXi hosts, and the new ESXi’s pfSense became the firewall. All is good.
Then, since the new box has a lot of storage and CPU power, I decided to put a new FreeBSD server. Up to this point, no problems.
I started taxing the new box, then, the new ESXi host malfunctioned. Most likely cause is overheatng but that is not relevant here. (So far, it appears that NVME slot is not happy.) With it, I decided to bring back the old ESXi host machine, moved the system SSD, and all Is good.
TL;DR – ESXi-A – replaced by ESXi-B – went back to ESXi-A with the disk of B.

Crisis

Problem started

with the reintroduction of ESXi-B, I could not access ESXi-A, because the Disk-B is still using the MAC address of ESXi-B. IOW vmknic0 of both ESXi hosts are using the same MAC.
I don’t have to use the ESXi-B but naturally, I have it, and want to use it after the hardware issue resolved.

Compounding the problem

After identifying the MAC addr dupe, I googled and found this. At this point, I was not aware that I can use the shell session of ESXi on the physical host. So, I was SSH-ing into it.

# esxcfg-vmknic -d -p Management\ Network

Then, OBVIOUSLY the networe connection is lost.

Panic sets in

I should have studied VMware a little better. Anyhow, I reboot the box, with a monitor and keyboard. Goes into ESXi’s 19th century user interface. At this point, ESXi web interface is lisening on http://0.0.0.0 and no go.
I cannot recreate the management network as the pfSense VM auto-starts and blocks from resetting the network config.
Shoot.

Obi-wan (YouTube), you are my only hope

Another round of googling lands on a random YouTube, and “Hit F1 go get to the shell”. Today, first time, I learned that I can use the shell of ESXi on the physical host.
Hit alt-F1 (like any Linux terminal session), and the console awaits the username/password.
I was finally in clear. Not completly but I know how now.

On the way to recovery

Now, I can do the 2nd part of creating the new vmknic0.

esxcfg-vmknic -a -i 192.168.1.2 -n 255.255.255.0 Management\ Network

which picked up the MAC addr from ESXi-A.
Also, I lost the ESXi’s virtual switch using vmknic0 so that also needed to re-assign.

Lessons Learned

  1. vmknic0 MAC address Is sticky and you cannot move the disk from one ESXi host to the other without recreating it
  2. To do so, you enable the shell, and you have access to the CLI on the physical host
  3. Before doing so, it is your firewall running host, print out all the commands you need beforehand.
  4. Have a hot-spot capable phone just in case so you can google in comfort.