Category Archives: VMware

Use PowerCLI to Automate Disaster Recovery Failover On Nutanix

Using VMware SRM on Nutanix has a few challenges.  SRM expects replication to happen at a datastore level.  By default Nutanix protection domains replicate at a VM level.  It is possible to set up Nutanix replication at a datastore level, but you lose granularity of being able to take VM specific snapshots.  SRM is also dependent on vCenter and SSO.  We were having a few issues that caused us to migrate from the Windows version of vCenter to the vCenter Server Appliance, and in doing so broke SRM so it had to be set up again.  Well, instead of setting it up again, I figured we would get more flexibility if I could do the same thing with PowerCLI.  Unfortunately, Nutanix’s Powershell CMDLET Migrate-NTNXProtectionDomain was published before actually implementing the failover part of the command, so after the script runs you still need to perform the additional step of logging into PRISM and clicking migrate. The script checks to see if the VMs are Windows or Linux. If they are Linux, the script expects a file to be staged called failover, that copies a staged network interface configuration file.

Change Nutanix CVM RAM with PowerCLI

*Update – story behind the script*
Finally I have a few minutes to write the story behind this script.

One of our VMware View environments was experiencing performance problems. The CPUs on our VMs would constantly spike to 100% after they were powered on. Our admins relayed back to engineering that they were having density issues. We reached out to Nutanix who recommended that we increase the cache size to be able to absorb more IOPS. To increase the cache size on Nutanix you simply need to power off the controller virtual machine (CVM) on a host, increase RAM, and power it back on. While is a non disruptive process if you power the CVMs on and off one at a time, it becomes a very disruptive process if someone makes a mistake and powers off more than one CVM at a time. It is also very time intensive because you must check that the CVM services are completely back up before you perform the procedure on the next CVM. With 120 hosts in our environment, and averaging 10 minutes per manual CVM procedure, it looked like it was going to take about 20 hours to perform this task. For us this means 3-4 days in maintenance windows!

I figured there has to be a way to automate this and eliminate the human component so we could perform this maintenance task all in one maintenance window. Well a couple hours of fiddling with powerCLI and trying to figure out which service is the last CVM service to power on, and running the script in our test environment to work out the bugs and we were ready to run it in production. In our environment the average run time per CVM was about 5 minutes, but the best part is that it really saves hours of admin time. An admin only needs to babysit the script while it is running instead of needing to perform an intensive manual process. This shows the huge benefit of Software Defined Storage. Imagine trying to update cache on a traditional SAN without any downtime… isn’t going to happen.

It later turned out that the issue in our environment was a classic VMware View admin mistake of installing updates and then shutting down immediately and recomposing the pool. The updates needed to finish installing after reboot, so they finished installing on all of the linked clones when they powered on. Combined with refresh on logoff which occurs multiple times per day and it was a sure way to test max performance of our equipment!

VMware View Guy Admits that Citrix XenDesktop is Just As Good

So I’ll admit it, I knew nothing about Citrix.  Well I mean other than all the FUD VMware was spewing about how much “fun” I would have if I ever implemented it for a customer.  Citrix actually showed up in the office about 4 years ago to try to explain what was going on but all I remember is that they showed me something called Dazzle and I thought, “how the hell am I supposed to explain to my customers what a Dazzle is supposed to do?” and then went back to installing VMware View.

Really, I was just too busy running around deploying View to get a couple hours to deploy XenDesktop and do my own fact checking.  And really, that is all it takes, is a couple hours. 

One of my vendors insisted that I was missing out.  They introduced me to the Federal team over at Citrix, who got me into Citrix Synergy and introduced me to Bob Mensah, Systems Engineer for Citrix.  Bob is an amazing font of Citrix knowledge!  Bob was able to walk me through the installation of XenDesktop in my lab in a couple hours while I was literally sitting at Honda waiting for my wife’s van to be serviced.

If you’ve been doing View for any significant period of time it’s not that hard to pick up.  Yeah, all the services have different names, but they have the same functionality.  Here’s a chart to help you figure it out:

Horizon View Citrix XenDesktop
vCenter vCenter (but could also be XenCenter or SCVMM)
View Connection Server StoreFront
View Composer Machine Creation Services
View Administrator Citrix Studio
Horizon Workspace StoreFront
Install license key on host Licensing Server
Need 3rd party load balancer Netscaler included
ThinApp (packaged executables) XenApp (Streamed Applications)
Blast (run ThinApps, XenApps, or RDSApps) StoreFront / XenApp

Bob Mensah even pointed me toward these guides that helped me set up CAC authentication in my lab:
Citrix – Create a JITC test CAC environment for XenDesktop/XenApp
Microsoft Technet – Step by Step Guide – Single Tier PKI Hierarchy Deployment

The Citrix administrative tools are Windows only, which could be seen as a draw back, but really the vSphere Web Client and View Administrator client are written in Flash and are slow, so I think Citrix actually has better functioning tools here.

Using Citrix Receiver to connect to a Windows desktop feels a lot like using the View Client.  The one thing that I did notice using my CAC was that I had to use my PIN two times.  Once to authenticate to StoreFront and then another to authenticate to the Windows VM.  With View I only have to put in my PIN once to authenticate to the View Connection Server and that gets passed to the VM.  Citrix told me that this is to overcome a security issue with having the PIN cached on the connection broker, but really I have never had an IA person tell me that was an issue with View so I am curious to understand where that requirement came from.

One thing that the Citrix Receiver has going for it is that it works with the new Tactivo iPad CAC Reader from Precise Biometrics.  CAC Authentication for iPad is nothing new, but previously it could only be accomplished on a per app basis with specialized apps designed to interact with some kind of Bluetooth CAC reader or dongle.  Neither were very convenient.  The Bluetooth reader meant that you needed to carry around an extra peripheral, charge it, and hope nothing interrupted your bluetooth connection.  The dongle… was just cumbersome and silly.  The Tactivo is a sleek integrated case, shown below in the iPad mini model with a magnetic smart cover (not included).  It connects via the lightning adapter and has a micro USB port that supports charging only.  See my photos of the unit below.  The VMware View client does not support this unit yet and I’m suspecting that it will actually fuel a lot of interest in Citrix until they do.

photo 3 photo 2 photo 4

Using XenApp you can now wrap CAC authentication around any application and present it on the iPad, including presenting entire Windows desktops complete with paired bluetooth keyboard and mouse (explained below)!

photo 6         

The other innovative thing about the Citrix Receiver client for iPad is that they have cleverly overcome the iOS inability to pair with a bluetooth mouse!  You can use another iOS device with the Citrix Receiver client installed on it as a touchpad!  The only silly part about this was that I had to set up the storefront connection on the extra device before I could pair it.  I am assuming that it either communicates between the iDevices through wireless or bluetooth, so I think that having to set up the client before you can use it as a touchpad is unnecessary.  However it works really well.  While the screen is a little small on the iPad mini, I was able to open applications and even play a movie just like I could with the Windows client.  My opinion is that it would definitely be a better experience with a full size iPad.

The only other issue I had when I was using the Citrix Receiver client is that there are a lot of extra options in the settings (shown in the picture below) that weren’t intuitive.  Here is the documentation for the client, but if you look through it you will see that the settings in the picture below are not documented.  If you look at the documentation for the View Client for iOS you see that every little feature in the client has a blurb explaining what it does.


In all, my initial impression of Citrix XenDesktop is that it has just as much functionality as VMware View.  I just wish that some things had more effort put into documentation rather than getting the functionality ready to ship.

Nutanix and VMware vSphere Host Profiles

Host profiles seem like a great idea… Make sure that all of your hosts are configured consistently and enforce compliance. However, when it comes to actually applying a host profile the caveat is that you need to put the host in maintenance mode to apply it. This means that you have to vMotion any running VMs to another host and then enter maintenance mode… A process that could take quite a while depending on the number of VMs you have running.

On Nutanix there is the pesky issue that there is one VM that you can not vMotion to another host… the CVM! The CVM (Controller Virtual Machine) is the storage controller that lives on the host. The physical disks are presented to the VM through VMDirectPath. Since Virtual Machines that are tied to physical devices on the host can not be vMotioned the host will fail to enter maintenance mode. It is possible to shut down a CVM on one node, then put that host into maintenance mode, apply the host profile, exit maintenance mode, power on the CVM, then SSH into the CVM to make sure it is back into the storage cluster before you rinse and repeat for all of your hosts. However, that is a very manual process! It would be bearable to perform on one block (four Nutanix hosts), but if you have hundreds of hosts it will take weeks and a small army of dedicated sys admins to complete the task.

It’s too bad that VMware couldn’t have host profiles distinguish between minor and major changes when dealing with applying host profiles. For example adding a port group would be a minor change, not requiring entering into maintenance mode, while attaching a vSwitch to a vNIC would be a major change requiring maintenance mode because of its potential to disrupt traffic for all of the VMs on that host.

Do we really need host profiles? Nutanix is trying to market the idea that infrastructure should be web-scale. I don’t really like the term web-scale because I think it implies that you’re trying to build some kind of internet service, but that’s beside the point… What they are trying to say is that it should be easy to massively scale infrastructure. This includes having to manually configure a bunch of settings. Putting all of the hosts in your environment into maintenance mode just to apply some settings definitely isn’t scalable. There is no reason to do it!

Every change that a host profile makes can be accomplished through PowerCLI without putting your host into maintenance mode. My recommendation for Nutanix hosts is to use PowerCLI to make any changes to your hosts that you want to be consistent throughout your environment, and then maintain your PowerCLI script and apply it to new hosts that you add to your environment.

You could also make a script that checks the settings on the hosts to monitor for compliance, for example to make sure that no one has added a vLAN to just one host. If you are using vCloud in your environment VMware includes VCM (vCenter Configuration Manager) which accomplishes the same task, with the added component of generating automated compliance reports.

Of course I’m implying that your hosts are running VMware, Nutanix also supports running Hyper-V and KVM where it’s almost inherently implied that you are going to need scripts to maintain consistency in the environment.

Nutanix CVM Autopathing Test

I have a Nutanix cluster that needs to be upgraded from 3.1.2 to (or if it is out by the time I get around to upgrading it). That got me to thinking about the upgrade process. When you perform a Nutanix Operating System (NOS) upgrade, it performs what Nutanix calls a “rolling upgrade”. This in effect only performs the upgrade on one CVM at a time. While the CVM is being upgraded, the storage on that node is directed to another CVM.

My first thought was, “How does that actually work”? Thanks to Zach Vaughn @z_n_v, Nutanix SE Extraordinaire, my eyes were opened.  When the cluster detects that a CVM is down, it SSHs to the Hypervisor (I’m referring to ESXi) and adds a route to the external IP of another CVM in the cluster. The cluster performs this check every 30 seconds, so it is possible that your VM will be without storage for 30 seconds. How disasterous could this be? (I’m told that as of NOS version this will be much faster than 30 seconds). The following video shows what happens.

This test was performed on a Nutanix 1350 block running NOS The desktop is running on Node C. I start encoding a video using handbrake which is writing to the user’s desktop on the local disk. When I shut down the CVM on Node C the desktop appears to hang for 20 seconds. However, it is possible that the PCoIP server process stops responding for those 20 seconds, as when the desktop resumes you can see that it has still received pings from the hypervisor.

I ran this test from a different machine and the View Client seemed to stay connected. The difference being that it was an iMac connected via ethernet and I recorded the video on my Macbook Pro connected via wireless. The desktop continued to receive pings, but the handbrake process stopped while the disk was unavailable for about 20 seconds and then resumed when the route to the CVM was changed on the hypervisor. If I can get that to work again I’ll try to post another video.

Export Nutanix Configuration to CSV through Powershell and REST API

What do you do when you have over 100 Nutanix nodes scattered across multiple datacenters and need to audit the configurations, or record the configurations for documentation?

Write a powershell script that queries the REST API of course!

In this instance I needed a known starting point.  I didn’t have all of the IP addresses of the CVMs, hosts, etc in a format that I could query.  What I did have was all of the hosts in vCenter along with all of their CVMs.  So this script starts by connecting to all of the vCenters in the Datacenters and getting a list of all of the CVMs and their IP addresses.  It then runs REST API queries against the CVM IPs.

Here’s what the output looks like when opened in Excel (and scrubbed of proprietary information):


Any blocks that are not configured yet, or are not running a version of NOS that has the REST API, or do not have network connectivity will return System.Collections.Hashtable values as you can see below.


Upgrade Nutanix 1350 block to ESXi 5.5

Nutanix recommends that you upgrade to vSphere 5.5 using the VMware Update manager instead of directly mounting the ISO.

Another way to upgrade instead of installing Update Manager is to just download the offline bundle and run the command:

esxcli software vib update –d “FILEPATH to OFFLINE BUNDLE”

Here are the steps that I used to upgrade my nodes from ESXi 5.0.0 to ESXi 5.5.

  1. Download ESXi 5.5 bundle from VMware.
  2. Upload the bundle to the root of my Nutanix datastore


  3. SSH to the CVM.  From the CVM we can execute a script that will run on all of the hosts:

    for i in hostips; do echo $i && ssh root@$i "esxcli software vib install -d /FILEPATH TO OFFLINE BUNDLE"; done

    *I missed that the hostips is encapsulated with backticks and not ‘’ single quotes so I just logged onto each host and ran “esxcli software vib install –d /FILEPATH TO OFFLINE BUNDLE”


  4. Shutdown the CVM.  We are able to shut down one CVM at a time without disrupting the state of the cluster.   Then reboot the host.


  5. Rut-roh!  My host didn’t come back into vCenter.  When I try to force it to reconnect it tells me that some virtual machines powered back on without following the cluster EVC rules.  Upgrading to ESXi 5.5 must have reset the EVC setting on that host.


    To remedy it I shut down the CVM, force the host to reconnect, then power the CVM back on.  On the next node I just put the host into maintenance mode before I reboot.

Copy files between ESXi hosts using SCP

Need a quick way to move files on one datastore to the datastore of another host that is not within the same vCenter?

In a Nutanix environment SSH is enabled on the hosts so we can use SCP to do this.  I needed to move an ISO repository from the production cluster to the TEST / DEV cluster.  Log into the source host as root, change directory to the datastore folder  (/vmfs/volumes/DATASTORE/FOLDER) and then run the following command:

scp –r * root@DESTINATION:/vmfs/volumes/DATASTORE/FOLDER

# The destination FOLDER must already exist on the destination DATASTORE.

Use PowerCLI to get an inventory of VMs

Say your boss asks you to plan for expansion and you need to get an inventory of the VMs in your current environment with their resource consumption.  What’s the VMware answer for this?  Oh buy VCOPs… wait that’s right I don’t have 100k that I can drop right now.  Oh there’s a free way to export this to excel?  How would I do that?!  PowerCLI!

To get an inventory of VMs from your ESXi hosts using PowerCLI:

Get-VM | Export-Csv –path “c:\users\josh\desktop\myVMs.csv” –NoTypeInformation

This will export a CSV file with the following fields:



If you don’t need all of those fields you can select the ones you need with the following syntax:

Get-VM | select Name, Guest, MemoryGB, ProvisionedSpaceGB, UsedSpaceGB | Esxport-CSV –path c:\users\josh\desktop\myVMs.csv –NoTypeInformation

If you don’t type –NoTypeInformation then you will get the following at the beginning of your CSV: #TYPE Selected.Vmware.VimAutomation.ViCore.Impl.V1.Inventory.VirtualMachineImpl