All posts by Josh Sinclair

Export an ISO from the Image Service on Nutanix AHV

Nutanix AHV has an image service built into PRISM that lets you upload ISO files and connect them to VMs.  Currently there is no export for the image service built into PRISM. 


However, it is not hard to export this ISO if you need to.

Step 1: Find the vmdisk_uuid for the Image

Log into a CVM and go into acli.

List the images using: image.list

Get the details for the image using image.get [Image Name]. 
Example: image.get Windows10

Note the vmdisk_uuid.



Step 2: Use a SCP tool to copy the vmDisk

Use a SCP tool like WinSCP to log into the CVM.  You need to log in with a PRISM user/pass to port 2222.  In this example I use the default login “admin”.



The .acropolis folder is hidden so use the open folder button to browse to the .acropolis/vmdisk folder.



Right click on the UUID and select download.

Screen Shot 2017-02-16 at 10.30.00 AM


Type in a name for the iso.  In this example I name it Windows10.iso



Export a VM from Nutanix AHV to VMware ESXi

In this example I export a CentOS 7 template from AHV to ESXi.  There are a couple of ways to accomplish this task, depending on whether you need a thin provisioned file or a thick provisioned file.

Thin Provisioned

Step 1: Find UUID of the vDisk.

Connect to a CVM, enter aCLI and run the command vm.get [vm name]

Copy the vmdisk_uuid.  (Notice the size of the VM under the STORAGE column in PRISM, that should be the size of the exported file… assuming the VM only has 1 vdisk)




Step 2: Export the vDisk

vDisks of AHV VMs are located in a hidden folder on the container named .acropolis.  We use the qemu-img command to export the vDisk.  The vdisk is exported in a thin format and should match the size of the VM in PRISM.  If the disk is large then the command might take longer to complete than the timeout value of the SSH session.  In order to not have the conversion corrupted by the SSH session timing out either use keep alives or run the task in the background by using a ‘&’ at the end of the command.  In this example I will run the task in the background.

Make sure the VM is powered off, then run the following command:

qemu-img convert –O vmdk nfs://[container]/.acropolis/vmdisk/[UUID] nfs://[container]/[vmdisk].vmdk &

qemu-img convert -O vmdk nfs:// nfs:// &

We can check that the task is still running using the command PS –A | grep qemu.  When the command returns nothing we know it has completed.



Step 3: Copy the vDisk

Once the export completes, you can now whitelist a Windows 2012 R2 server and simply browse to the container and copy the vDisk.  Alternatively you can also use a SCP tool  by connecting with admin@[host]:2222.




Step 4.  Create a new Virtual Machine and Upload the VMDK to ESXi.

Here I create a new VM with no virtual disk, because I am going to upload the VMDK to the VM’s folder.

Use a SCP tool to connect to ESXi and upload the VMDK to the VM’s folder.



Step 5: Use vmkfstools to create the vmdk disk descriptor.

ESXi expects vmdks to have a disk descriptor file that points to the raw vmdk file.  We can use vmkfstools to create that using the following command:

vmkfstools –i [sourceVMDK] [destinationVMDK] –d thin
vmkfstools –i CentOS_7.vmdk CentOS7.vmdk –d thin
Once the disk descriptor is created you can delete the original file.


Step 6: Attach the VMDK to the VM and power it on




Thick Provisioned

Exporting a thick provisioned disk is similar to the process above, except we don’t need to use qemu.  We can just SCP the disk from the .acropolis directory.

Step 1: Find UUID of the vDisk.

Connect to a CVM, enter aCLI and run the command vm.get [vm name]

Copy the vmdisk_uuid.  (Notice the size of the VM under the STORAGE column in PRISM, that should be the size of the exported file… assuming the VM only has 1 vdisk)




Step 2: SCP the vdisk from the .acropolis/vmdisk directory.

Use a SCP tool to connect to the Nutanix CVM.  If you use WINSCP you will have to use the Open Directory button (CTRL+O) to open the hidden .acropolis/vmdisk directory.



Copy the vDisk that matches the UUID from vm.get. Then copy the file to ESXi.


Step 3.  Create a new Virtual Machine and Upload the VMDK to ESXi.

Here I create a new VM with no virtual disk, because I am going to upload the VMDK to the VM’s folder.

Use a SCP tool to connect to ESXi and upload the VMDK to the VM’s folder.



Step 4: Use vmkfstools to create a VMDK descriptor file, then replace the flat file with the exported vDisk.

Following the process in VMware KB 1002511 we recreate a vDisk descriptor file.  The file has to be created with the exact same size as the exported vDisk.  Use ls –l to check the size of the exported vDisk.  Then use the following command to create the vDisk descriptor file:
vmkfstools –c [vDisk size] [destination file] –d thin

vmkfstools -c 42949672960 CentOS_7.vmdk -d thin

Once the file is created then replace the –flat.vmdk file with the exported vDisk.
mv fea6b382-43ec-4236-b521-edac7ac923cb CentOS_7-flat.vmdk



Step 5: Attach the VMDK to the VM and power it on



Export a VM from AHV raw format to VMware VMDK

In this example I’m going to export my Windows 2012 R2 template from AHV to ESXi.

Step 1: Find UUID of the vDisk.

Connect to a CVM, enter aCLI and run the command vm.get [vm name]

Copy the vmdisk_uuid.  (Notice the size of the VM under the STORAGE column in PRISM, that should be the size of the exported file… assuming the VM only has 1 vdisk)




Step 2: Export the vDisk

vDisks of AHV VMs are located in a hidden folder on the container named .acropolis.  We use the qemu-img command to export the vDisk.  The vdisk is exported in a thin format and should match the size of the VM in PRISM.

Make sure the VM is powered off, then run the following command:

qemu-img convert –O vmdk nfs://[container]/.acropolis/vmdisk/[UUID] nfs://[container]/[vmdisk].vmdk

qemu-img convert -O vmdk nfs:// nfs://


Step 3: Copy the vDisk

Once the export completes, you can now whitelist a Windows 2012 R2 server and simply browse to the container and copy the vDisk and import it into ESXi.  Alternatively you could just mount the container as NFS on the ESXi host and storage vMotion it to another datastore.



Step 4.  Create a VM with the new vdisk and power it on.  Remember to install VMware Tools.


Export a VM on AHV

Step 1: Find UUID of the vDisk.

Connect to a CVM, enter aCLI and run the command vm.get [vm name]

Copy the vmdisk_uuid.



Step 2: Export the vDisk

vDisks of AHV VMs are located in a hidden folder on the container named .acropolis.  We use the qemu-img command to export the vDisk.  One cool thing is that the vDisk is exported in a thin format, so even if it is provisioned as a 100GB drive, it will only export the actual size used.

Make sure the VM is powered off, then run the following command:

qemu-img convert –c nfs://[container]/.acropolis/vmdisk/[UUID] –O qcow2 nfs://[container]/[vmdisk].qcow2

qemu-img convert -c nfs:// -O qcow2 nfs://

Step 3: Copy the vDisk

Once the export completes, you can now whitelist a Windows 2012 R2 server and simply browse to the container and copy the vDisk.


Use Cluster Host_Upgrade to Automatically install VMware Patches on Nutanix

Many Nutanix customers running VMware today use VMware Update Manager (VUM) to apply patches.  Unfortunately VUM does not understand the process of shutting down one CVM at a time and rolling through the cluster.  With this process you can use leverage the back end of the One Click Hypervisor Upgrade feature, sit back, and grab a sandwich while your patch is applied.

Step 1: Nutanix Hypervisor Upgrade Requirements

Disable hypervisor lockdown mode on the hypervisor host

Enable vSphere HA and the following settings:

  • Enable Host Monitoring
  • Disable Admission Control > Select: Do not reserve failover capacity. Allow VM power on operations that violate availability constraints

Enable vSphere DRS and these settings:

  • Power Management > Off

vCenter Server Settings > Advanced Settings

  • config.migrate.test.CompatibleNetworks.VMOnVirtualIntranet value of false


Step 2: Download Patch from myVMware


Click on the Build Number KB and copy the md5sum of the patch.



Step 3: Upload Patch to a CVM

In this example I put the patch in /home/nutanix/tmp



Step 4: Run the cluster host_upgrade command

On the CVM that you uploaded the patch to run:

cluster –md5sum=[md5sum of patch] –bundle=[file path to patch] host_upgrade




Step 5: Sandwich run!

The Tesla of the Datacenter

A few months ago I was in the market for a new car.  I have three children so safety features were an important thing to consider.  I also live in California so fuel economy was an important factor as well.

There were lots of different models to choose from: sedan, fullsize, SUV as well as a variety of manufacturers … which one met my requirements?

The engineer in me built an internal capabilities matrix:

Wheels Model A Model B
Seat Capacity 5 8
MPG 30 18
Brakes Drum Disc
Entertainment System No Yes
Price $ $$$

No one really buys a car based on a capabilities matrix.  Also, I would argue that while price is an important factor people don’t buy cars primarily based on price.  People buy cars based on overall value, utility, and the quality of the product.

If I was trying to satisfy my basic requirement of mobility I could buy any vehicle with 4 wheels.  However, since I wanted to transport my kids I considered a vehicle with a built in entertainment system.  The entertainment system had nothing to do with the mobility requirement, but has everything to do with the quality of the mobility.  The entertainment system makes transporting kids easier and makes the ride a much better experience for everyone.

I could have bought a basic vehicle and added an aftermarket entertainment system.  However I would then need to install the system, run wires, mount some kind of awkward interface, etc.  To me, the integrated entertainment system provides value because it comes ready at time of purchase and with one button I can get the kids quiet and enjoy the drive.  It also saves me the time and cost of integrating the aftermarket solution.

Tesla of the Datacenter

Tesla is without a doubt revolutionizing the automobile industry.  Tesla’s competitors are behemoths.  They are saying that Tesla is just another me-too alternative vehicle that has no future, while internally they are desperately trying to figure out how to develop products that match the capabilities of the Tesla.  Nutanix is trying to revolutionize the enterprise datacenter the same way that Tesla is trying to revolutionize the automobile.

Neither Tesla nor Nutanix are able to compete with the bare bones economy product of their competitors.  It’s expensive to hire rock star engineers to develop a product, amazing marketing people to advertise, veteran support engineers… it would be impossible to compete with an automated factory churning out economy products.

In order to compete with the existing behemoths in their marketplace Tesla and Nutanix must accomplish both of these things:

1. Create a product that is unique and provides value in their market
2. Provide insane customer service and support

Unique in the market
Nutanix found its foothold in the VDI market by providing storage that delivered the needed performance, granular scalability and was bound only by the physical constraints of the datacenter.  The early VDI users soon realized that it was so much more efficient than SAN that they started to run server workloads on Nutanix.  The Nutanix operating system then evolved.  Now in addition to the initial storage product it includes integrated management, global management, file management, VM snapshots, backups, analytics, security, REST API, hardware monitoring, alerting, Disaster Recovery, one click firmware and software upgrades, application mobility, cloud migration, capacity planning, and multiple hypervisor support.  This is what makes Nutanix unique and valuable… This isn’t a suite of products or virtual appliances marketed as a single product, all of the features are actually all integrated in the same software!  Nutanix has true feature integration, not just marketing integration.

As a Nutanix customer this integration provided real value to me since I didn’t have to spend my time trying to get the multiple components of a software suite to try to talk together.  It’s the same reason you buy the car with all the features already integrated, that value is that you drive it off the lot and have everything work at the touch of a button.

Insane customer service and support
Nutanix invited their first customer on stage at their .NEXT to kick off the event.  Nutanix really sees their customers as partners… more than just a pile of dollar bills that is up for grabs every time they come into the dealership.  The customers are really what drives the product development.  More than one of the features that I listed above was a request that I had as a customer.

I would love to say that Nutanix is all unicorns and rainbows and makes you toast with orange marmalade, but the reality is that it is a software product, and software will always need maintenance.  As a customer I was impressed by the agility of Nutanix to quickly come up with a workaround and be respond to eliminate bugs and release fixes within a week or two.

Sometimes people forget that support is part of the product too, but it shows up as a line item on the PO!  Nutanix believes that you should actually get great service with that line item, instead of just forking over money so you can download the next software update.

Nutanix also has very experienced support engineers that are not only able to diagnose software issues, but all of the other vendor’s hypervisor platforms that we support as well.

Nutanix has follow the sun support.  As a customer I was impressed when I called at midnight… the guys in Australia that answered was just as capable and knowledgeable as his US counterparts to help me resolve the issue I was having.

Why do I have to pay for it?
I understand why a college kid would want to drive a Tesla, but the reality is that is probably not in their budget, nor will be in their budget for a few more years.  Tesla can’t afford to sell their current product to college kids, they need to pay their engineers, marketing, support, etc.

Nutanix is an enterprise product, targeting mission critical enterprise workloads that need the automation, security and all the other integrated features of Nutanix… way too many features to use simply as a storage platform.  Would you buy all the upgraded features for a vehicle and then never use them?

I understand why a small business would want all of the features that Nutanix provides, but we can’t play in that space and afford to pay our engineers either.  And in reality, a small business’s primary IT infrastructure need is storage.  There are other virtual san storage platforms that would be a great fit these small workloads.

With hard work, however, and the support of our customers, one day we will reach critical mass, scale up production and that college kid can buy a economy version Tesla and the SMB can get small business sized Nutanix. It is only a matter of time.

Nutanix – The difference is SERVICE!

It’s still sinking in that I spent the past week at new hire training for one of the hottest and fastest growing tech startups. I’m joining their federal sales team as a systems engineer.  I am humbled by the opportunity to work with so many great people.

Part of the training comprised meeting all of the senior leadership of the company, each giving a 30-60 minute overview of their role at the company and how to interface with them.  They definitely have the vision and drive to be disruptive to the tech industry.  And passion. It’s not often that you become part of something where everyone you work with has the same passion for the product.

At the end of sales training I went down to a empty conference room on the first floor of Nutanix HQ.  As I sat there getting caught up on email that had filled my new mailbox and messaging my wife about the day I was surprised by the CEO of Nutanix, Dheeraj Pandey.  He popped in and asked me what I had thought of the training.  I told him that it was like drinking kool-aid from a firehose.  I realized by his facial expression that what I had said probably didn’t come out right.  What I would have said if I could get a do over would be something like that coming from a service delivery role I was a little overloaded with information on sales-y type stuff, but felt challenged to practice and hone my messaging so I could share my passion for the product and company… yeah that sounded sales-y.

Obviously I need some practice.   Why would I want to challenge myself by going from service delivery to pre sales?  Because I want to share my passion for the technology and be a part of building the future.

As a kid I was fascinated by tech.  I was always collecting computer parts and getting old hand me down 486s to connect to the internet.  In high school I ran my own web/gaming server in my bedroom.  I was passionate about learning technology.  More than anything else I wanted to work at the local ISP.  I thought if I could get a job there I would be able to get my hands on the latest technology and be part of building the future.  I tried to apply for a job there every year.  Finally when I was 18 I was able to convince them that I was passionate about the technology they had and they relented / gave me a chance and hired me as a help desk tech.

The helpdesk techs were also given the task of taking all of the new sales calls.  We were often asked the question why should we buy your local service for $X when I can get the same service for less somewhere else?  I would have rather been asked how to configure Trumpet Winsock to connect using SLIP on Windows 3.11.

After a couple weeks of fumbling for answers I asked my boss for help.  He asked me a couple of questions.

  1. When you call a company, do you like to wait on hold before you are able to speak to someone?
  2. When your car needs maintenance, do you take it somewhere far away for the work to be performed?
  3. Do you like being able to know the name of the mechanic that performs the maintenance on your car?

My boss explained to me that the difference between us and our competitors was SERVICE!  Our customers preferred our service to our competitors even though we weren’t the least expensive service in town.  It was our responsibility to make sure that we delivered that premium level of service or our customers would leave.  No matter what we should answer the phone before the call queued.  No matter what we should solve our customer’s issue before they got off the phone.  We would even let our customers bring their computer into the shop and we would fix their issue for free if we could.

As a Nutanix customer I experienced the same level of customer service and commitment.  Just one example of this is when HR hired 1000 additional people and told us they were starting in a week.  We panicked.  I’m sure Nutanix panicked when we told them we needed to double our order and needed it before Monday.  Somehow they pulled through and our hardware arrived in less time then I could have even submitted a PO with other vendors.  We met our crazy deadline.  Nutanix even sent someone on-site to help install it… at no additional charge!  And that is the difference between other infrastructure vendors and Nutanix… the difference is SERVICE!

New hire training confirmed to me, like my first boss so many years ago, the message that customer service is THE MOST IMPORTANT THING extends from the senior leadership at Nutanix. They get it.

That day when I added all of the additional hardware for 1000 unexpected VDI users I recognized I had just seen something that I had never seen before.  I powered on the hardware, and a few minutes later I had 1000 additional desktops, without having to configure any LUNs, switches, or cable anything other than power and ethernet.  I had seen the future of infrastructure.  It was the power of the software defined datacenter.  It was webscale.  It was the same feeling that I had when I saw vMotion for the first time and thought holy shit! This changes everything!  This is amazing! I need to learn all that I can about this technology!  This is the future and I need to be a part of building the future!

Two years later Nutanix is giving me a chance to be part of building the future.  I will try my hardest to keep up with these amazing people and continue to share my passion and enthusiasm with all of you.

Use PowerCLI to Automate Disaster Recovery Failover On Nutanix

Using VMware SRM on Nutanix has a few challenges.  SRM expects replication to happen at a datastore level.  By default Nutanix protection domains replicate at a VM level.  It is possible to set up Nutanix replication at a datastore level, but you lose granularity of being able to take VM specific snapshots.  SRM is also dependent on vCenter and SSO.  We were having a few issues that caused us to migrate from the Windows version of vCenter to the vCenter Server Appliance, and in doing so broke SRM so it had to be set up again.  Well, instead of setting it up again, I figured we would get more flexibility if I could do the same thing with PowerCLI.  Unfortunately, Nutanix’s Powershell CMDLET Migrate-NTNXProtectionDomain was published before actually implementing the failover part of the command, so after the script runs you still need to perform the additional step of logging into PRISM and clicking migrate. The script checks to see if the VMs are Windows or Linux. If they are Linux, the script expects a file to be staged called failover, that copies a staged network interface configuration file.

Change Nutanix CVM RAM with PowerCLI

*Update – story behind the script*
Finally I have a few minutes to write the story behind this script.

One of our VMware View environments was experiencing performance problems. The CPUs on our VMs would constantly spike to 100% after they were powered on. Our admins relayed back to engineering that they were having density issues. We reached out to Nutanix who recommended that we increase the cache size to be able to absorb more IOPS. To increase the cache size on Nutanix you simply need to power off the controller virtual machine (CVM) on a host, increase RAM, and power it back on. While is a non disruptive process if you power the CVMs on and off one at a time, it becomes a very disruptive process if someone makes a mistake and powers off more than one CVM at a time. It is also very time intensive because you must check that the CVM services are completely back up before you perform the procedure on the next CVM. With 120 hosts in our environment, and averaging 10 minutes per manual CVM procedure, it looked like it was going to take about 20 hours to perform this task. For us this means 3-4 days in maintenance windows!

I figured there has to be a way to automate this and eliminate the human component so we could perform this maintenance task all in one maintenance window. Well a couple hours of fiddling with powerCLI and trying to figure out which service is the last CVM service to power on, and running the script in our test environment to work out the bugs and we were ready to run it in production. In our environment the average run time per CVM was about 5 minutes, but the best part is that it really saves hours of admin time. An admin only needs to babysit the script while it is running instead of needing to perform an intensive manual process. This shows the huge benefit of Software Defined Storage. Imagine trying to update cache on a traditional SAN without any downtime… isn’t going to happen.

It later turned out that the issue in our environment was a classic VMware View admin mistake of installing updates and then shutting down immediately and recomposing the pool. The updates needed to finish installing after reboot, so they finished installing on all of the linked clones when they powered on. Combined with refresh on logoff which occurs multiple times per day and it was a sure way to test max performance of our equipment!

VMware View Guy Admits that Citrix XenDesktop is Just As Good

So I’ll admit it, I knew nothing about Citrix.  Well I mean other than all the FUD VMware was spewing about how much “fun” I would have if I ever implemented it for a customer.  Citrix actually showed up in the office about 4 years ago to try to explain what was going on but all I remember is that they showed me something called Dazzle and I thought, “how the hell am I supposed to explain to my customers what a Dazzle is supposed to do?” and then went back to installing VMware View.

Really, I was just too busy running around deploying View to get a couple hours to deploy XenDesktop and do my own fact checking.  And really, that is all it takes, is a couple hours. 

One of my vendors insisted that I was missing out.  They introduced me to the Federal team over at Citrix, who got me into Citrix Synergy and introduced me to Bob Mensah, Systems Engineer for Citrix.  Bob is an amazing font of Citrix knowledge!  Bob was able to walk me through the installation of XenDesktop in my lab in a couple hours while I was literally sitting at Honda waiting for my wife’s van to be serviced.

If you’ve been doing View for any significant period of time it’s not that hard to pick up.  Yeah, all the services have different names, but they have the same functionality.  Here’s a chart to help you figure it out:

Horizon View Citrix XenDesktop
vCenter vCenter (but could also be XenCenter or SCVMM)
View Connection Server StoreFront
View Composer Machine Creation Services
View Administrator Citrix Studio
Horizon Workspace StoreFront
Install license key on host Licensing Server
Need 3rd party load balancer Netscaler included
ThinApp (packaged executables) XenApp (Streamed Applications)
Blast (run ThinApps, XenApps, or RDSApps) StoreFront / XenApp

Bob Mensah even pointed me toward these guides that helped me set up CAC authentication in my lab:
Citrix – Create a JITC test CAC environment for XenDesktop/XenApp
Microsoft Technet – Step by Step Guide – Single Tier PKI Hierarchy Deployment

The Citrix administrative tools are Windows only, which could be seen as a draw back, but really the vSphere Web Client and View Administrator client are written in Flash and are slow, so I think Citrix actually has better functioning tools here.

Using Citrix Receiver to connect to a Windows desktop feels a lot like using the View Client.  The one thing that I did notice using my CAC was that I had to use my PIN two times.  Once to authenticate to StoreFront and then another to authenticate to the Windows VM.  With View I only have to put in my PIN once to authenticate to the View Connection Server and that gets passed to the VM.  Citrix told me that this is to overcome a security issue with having the PIN cached on the connection broker, but really I have never had an IA person tell me that was an issue with View so I am curious to understand where that requirement came from.

One thing that the Citrix Receiver has going for it is that it works with the new Tactivo iPad CAC Reader from Precise Biometrics.  CAC Authentication for iPad is nothing new, but previously it could only be accomplished on a per app basis with specialized apps designed to interact with some kind of Bluetooth CAC reader or dongle.  Neither were very convenient.  The Bluetooth reader meant that you needed to carry around an extra peripheral, charge it, and hope nothing interrupted your bluetooth connection.  The dongle… was just cumbersome and silly.  The Tactivo is a sleek integrated case, shown below in the iPad mini model with a magnetic smart cover (not included).  It connects via the lightning adapter and has a micro USB port that supports charging only.  See my photos of the unit below.  The VMware View client does not support this unit yet and I’m suspecting that it will actually fuel a lot of interest in Citrix until they do.

photo 3 photo 2 photo 4

Using XenApp you can now wrap CAC authentication around any application and present it on the iPad, including presenting entire Windows desktops complete with paired bluetooth keyboard and mouse (explained below)!

photo 6         

The other innovative thing about the Citrix Receiver client for iPad is that they have cleverly overcome the iOS inability to pair with a bluetooth mouse!  You can use another iOS device with the Citrix Receiver client installed on it as a touchpad!  The only silly part about this was that I had to set up the storefront connection on the extra device before I could pair it.  I am assuming that it either communicates between the iDevices through wireless or bluetooth, so I think that having to set up the client before you can use it as a touchpad is unnecessary.  However it works really well.  While the screen is a little small on the iPad mini, I was able to open applications and even play a movie just like I could with the Windows client.  My opinion is that it would definitely be a better experience with a full size iPad.

The only other issue I had when I was using the Citrix Receiver client is that there are a lot of extra options in the settings (shown in the picture below) that weren’t intuitive.  Here is the documentation for the client, but if you look through it you will see that the settings in the picture below are not documented.  If you look at the documentation for the View Client for iOS you see that every little feature in the client has a blurb explaining what it does.


In all, my initial impression of Citrix XenDesktop is that it has just as much functionality as VMware View.  I just wish that some things had more effort put into documentation rather than getting the functionality ready to ship.