November 10, 2019November 15, 2019 Dave

Configuring ESXi after Boot from SAN

Cisco UCS, General IT, VMware Tips

You know what sucks? iSCSI Boot from SAN with Cisco UCS.

Well- not really.

What sucks is doing Boot from SAN on UCS for a small environment.

(Hi, Cisco, I’m only kidding, I like BfS).

It sucks in a small environment because

You don’t set it up that often
When you do, you’ve forgotten how and have to re-learn it

Seriously- just pay for the Cisco SD cards unless you have hundreds or thousands of hosts.

I’m not going to show you Boot from SAN in UCS today, because it’s actually covered pretty well elsewhere. I will, however, introduce you to another problem with Boot from SAN in vSphere- that is- configuring vSphere Networking for fully separate paths after your iSCSI Boot from SAN is built.

Why do you care?

You care/this will help you if:

you’re booting your ESXi hosts from SAN instead of local storage
you’re using something like Cisco UCS- centralized server management
you have completely separate paths to storage to minimize points of failure

My Setup

Cisco UCS with B200 M4/M5 blades
Cisco UCS Chassies and various interconnects
Cisco 6428 Fabric Interconnects
Cisco 5Ks redundant switches pair
NetApp FAS 2552 (dual controllers)

Install ESXi

Once you can Boot from SAN (BfS from now on, because I’m lazy), you should be able to install ESXi on your neat little boot LUN. Do that. Do all of your hosts while you’re at it.

Get this host into vCenter

You need vCenter for some of these options. Install the VCSA and pull in your hosts.

Configure ESXi Management interfaces

At this point, you should be able to get into ESXi by the KVM and configure the basics:

IP and DNS config
Probably turn off IPv6, because if you’re using IPv4 you can’t mix them in vSphere
Select the redundant Management Interface

Configure Additional ESXi Networking

Now, we know that by default you’re going to end up with

a Management VMKernel Port on vSwitch 0
an iSCSI Boot Port Group (shockingly) labeled iScsiBootPG on your iScsiBootvSwitch
a Standard Port Group called VM Network- also on vSwitch 0

We are using completely separate networking paths for our iSCSI, so we are going to use 2 separate vSwitches and Port Groups: iSCSI-A and iSCSI-B. These paths go all the way back to our storage so we can eliminate single points of failure. We have 4 paths on our NetApp.

iSCSI Redundancy

Let’s go ahead and give the iSCSI in our setup some redundancy. We will build out a new vSwitch, add 2 new VMKernel ports.

Steps:

A caution here- if you do not set this up correctly, you can lose connection to your iSCSI- including for your ESXi host configuration- which can cause you to have to reboot the host and do some steps over. Follow these steps carefully.

Build our your iSCSI-B

Your iSCSI-A is already made, it’s just named iScsiBootPG. We will straighten all that out in a bit, but first we need to give you redundancy so that, if you jack something up, you don’t completely break you host then force it to do a network rollback (ask me how I know).

Create yourself a new Standard vSwitch and a VMKernel port for your iSCSI-B. We’ll leave the default name for it as vSwitch1 (it’s really hard to change anyway, just don’t). Both iSCSI-A and B are going to run on it.

In vCenter: Your Host > Configure > Virtual Switches > Add Networking
Select Connection Type: VMKernel Network Adapter radio button.
Select New Standard Switch. I put an MTU of 9000 for Jumbo Frames.
Create a Standard Switch: this is where you add a physical adapter. You will select one side of your iSCSI that goes back to your storage. It should be on a different VLAN than your existing iScsiBootPG. For me, it was vmnic3.
Port properties: Fill in your Network Label. Mine is iSCSI-B.
IPv4 settings: Self-explanatory. Assuming your iSCSI is not routed (it should not be), you don’t need a default gateway and should not override the defaults.
Ready to complete: Review it, and if it looks good, go ahead and hit Finish.

Add additional iSCSI targets

Whoops! When your host starts, it only connects to 1 iSCSI target- the one you need to boot from. But it only connects via one side of your iSCSI- so you need to add the other 3 so you can utilize all of your paths to storage.

This will be completed under Configure > Storage Adapters > your iSCSI adapter (vmhba(N)) > Dynamic Discovery. (You’ll notice your single iSCSI target is present under the Static Discovery tab). You will actually add all 4 to the Dynamic Discovery tab- even the one you have under Static Discovery already. Do the following for all 4:

Hit Add
Add the IP address of your target
Hit Ok

Once all 4 are present, rescan the adapter.

NICE! 4 paths per device. That’s what we’re looking for. This will give you the redundancy you need when

one of your pair of switches fails or is restarted
your storage fails over from one controller to another
you have a logical (misconfiguration) or physical problem (loss of hardware) in your UCS or server configuration

Move off the default iScsiBootPG and iScsiBootvSwitch

At this point, you now have fully-redundant iSCSI paths to storage. This is great! You still have some cleanup to do, however. Your default iScsiBootvSwitch and iScsiBootPG are still out there, and you want to move to your single vSwitch1 for simplicity.

Since you’ve already confirmed redundancy, you should be ok to proceed- just go slow and pay attention to what you’re doing. A good side note- when you are migrating Port Groups around, you start that migration on the switch you want it to go to.

NOTE: You are going to get alarms during this process, because you are effectively removing 1/2 of your paths back to storage. DON’T do this with VMs running on it.

First, let’s migrate the iScsiBootPG Port Group to vSwitch1 under Configure > Networking > Virtual Switches > Standard Switch: vSwitch1

Hit the ellipse (three dots)
Migrate VMKernel Adapter
In the new window: Select VMkernel adapter > pick vmk1 (iScsiBootPG) and hit Next
Set Network label (iSCSI-A in our case)
No VLAN for me (maybe not so for you) then hit Next
Ready to complete – review and hit Finish when ready

Next, let’s move the physical adapter (vmnic) from the old iScsiBootvSwitch to vSwitch1:

Back up on the iScsiBootvSwitch, hit Manage Physical Adapters
Select your vmnic (mine is vmnic2) and hit the red X to remove it from Assigned Adapters. Hit OK.
You’ll get a warning saying there are no physical adapters on this switch. You expect this because there’s also no Port Groups. Hit OK.
On vSwitch1, hit Manage Physical Adapters
Select that vmnic (mine is vmnic2) and hit OK.
Hit OK again.

A good side note- when you are migrating Port Groups around, you start that migration on the switch you want it to go to.

Last, we will set vmnic2 to only work on iSCSI-A and vmnic3 to only work on iSCSI-B. Failover will not work correctly if these paths are separated.

On iSCSI-A, click the ellipse and then Edit Settings
Click Teaming and failover
You will notice that both vmnic2 and vmnic3 are selected as active. Let’s fix that.
Click the Override checkbox
Select vmnic3 and click the blue downarrow until vmnic3 moves from Active Adapters to Unused Adapters. It’s going to stay on the switch but we don’t want iSCSI-A to use it because is the wrong VLAN.
Hit OK.

Now- if you click iSCSI-A, it should only go to vmnic2. Do the same process for iSCSI-B, but only use vmnic3.

Clean up iScsiBootvSwitch

At this point, that iSCSIBootvSwitch should be empty- no PortGroups and no Adapters.

Go ahead and remove it:

click the ellipse and hit Remove
Confirm by hitting YES

Confirm you’re all good

Go back and rescan your storage to confirm that you still have all the paths you need under Configure > Storage > Storage Adapters > click your iSCSI Software Adapter (maybe vmhba64) and hit Rescan Storage.

Look at the Paths tab below. If you still have your 4 paths per LUN, you are done!

Now you have properly configured ESXi for iSCSI Boot from SAN with redundant paths.