PowerCLI | Josh Sinclair

Using VMware SRM on Nutanix has a few challenges. SRM expects replication to happen at a datastore level. By default Nutanix protection domains replicate at a VM level. It is possible to set up Nutanix replication at a datastore level, but you lose granularity of being able to take VM specific snapshots. SRM is also dependent on vCenter and SSO. We were having a few issues that caused us to migrate from the Windows version of vCenter to the vCenter Server Appliance, and in doing so broke SRM so it had to be set up again. Well, instead of setting it up again, I figured we would get more flexibility if I could do the same thing with PowerCLI. Unfortunately, Nutanix’s Powershell CMDLET Migrate-NTNXProtectionDomain was published before actually implementing the failover part of the command, so after the script runs you still need to perform the additional step of logging into PRISM and clicking migrate. The script checks to see if the VMs are Windows or Linux. If they are Linux, the script expects a file to be staged called failover, that copies a staged network interface configuration file.

# THIS SCRIPT REQUIRES THE NUTANIX POWERSHELL MODULES AND POWERSHELL VERSION 4.0

# Interface of Windows VMs needs to be named "Ethernet"

# VMs must be powered on

 

# 1.  Get list of VMs and IPs from a VMware Folder

# 2.  Check that all VMs are in the Protection Domain

# 3.  Change the IPs of the VMs and shut them down

# 4.  Change the static DNS entries

 

 

### DECLARE CONSTANTS ###

 

#Source Domain Controller

$dnsServer = "dc01.test.com"

 

#Establish PSSession to target Domain Controller

$s = new-PSSession -Computer $dnsServer

Import-PSSession -Session $s -Module DNSServer

 

# Need to load PowerCLI and Nutanix modules

Add-PSSnapin NutanixCmdletsPSSnapin

 

#Make sure that any previous Nutanix or vCenter sessions are disconnected

Disconnect-NTNXCluster *

Disconnect-VIServer * -Confirm:$false

 

#Name of target VMware folder

$folder = "DRTest"

 

#Name of the Protection Domain

$protectionDomain = "Test"

 

#Destination Network - 10.XXX.XXX.0

$destinationNet = "10.0.1.0"

$destinationNet = $destinationNet.Split(".")

 

#Destination DNS 01

$destinationDNS01 = "10.0.1.11"

$destinationDNS02 = "10.0.1.12"

 

#Nutanix Cluster IP

$nutanixClusterIP = "10.0.0.51"

 

#Source vCenter

$vCenter = "vc01.test.com"

 

### BEGIN SCRIPT ###

 

#Clear Powershell Console

clear

 

#Connect to vCenter

Connect-VIServer $vcenter

 

#Connect to the Nutanix Cluster

Connect-NutanixCluster -Server $nutanixClusterIP -UserName admin -Password admin -AcceptInvalidSSLCerts | Out-Null

 

#Get the names of the VMs in the target Folder

#Initialize an array to hold the names of the VMs in the folder

$folderVMs=@()

 

#Add the names of the VMs in the folder to the array

Foreach ($vm in (get-vm -Location $folder)) { $folderVMs = $folderVMs + $vm.Name }

 

#Initialize an array to hold the names of the VMs in the protection domain

$protectedVMs=@()

 

#Add the names of the VMs in the protection domain to the array

Foreach ($vm in (Get-NTNXVM | Where {$_.protectionDomainName -eq $protectionDomain} | Select vmName)) { $protectedVMs = $protectedVMs + $vm.vmName }

 

#compare the arrays of VMs to check if there are any missing VMs and add them to $missingVMs

$missingVMs = Compare-Object $folderVMs $protectedVMs

 

#Initialize an array to contain VMs that are in the folder but not in the protection domain

$pdMissing = @()

 

#Initialize an array to contain VMs that are in the protection domain but not in the folder

$folderMissing = @()

 

#Initialize an array to contain the DNS Updates

$dnsupdates = @()

 

#Iterate through $missingVMs and notify the user which VMs are missing

if ($missingVMs.Count -ne 0) {

foreach ($vm in $missingVMs)  {

if ($vm.SideIndicator -eq "<=")  {

$pdMissing = $pdMissing + $vm

}

if ($vm.SideIndicator -eq "=>")  {

$folderMissing = $folderMissing + $vm

}

}

echo "These VMs are in the $folder VMware Folder but not in the $protectionDomain protection domain."

foreach ($pdvm in $pdMissing)  { echo $pdvm.InputObject }

echo `n

echo "These VMs are in the $protectionDomain protection domain but not in the $folder VMware Folder."

foreach ($foldervm in $folderMissing)  { echo $foldervm.InputObject }

echo `n

#If there are missing VMs then end the script

break

}

 

#Pop up a dialog box to get the linux username and password and test that the linux credential actually works with a linux VM to prevent an epic fail

#if the credential fails then prompt for the password again

$crTest = "echo test"

do {$linuxCredential = Get-Credential -Username "root" -Message "Type the ROOT username and password for Linux machines"}

Until (Invoke-VMScript -VM (get-vm -Location $folder | where {$_.Guest.OSFullName.Contains("Linux")} | Sort)[0] -ScriptText $crTest -GuestCredential $linuxCredential)

 

 

#Get the IP Address of the VM, convert it into an array, and replace the source network with the destination network

Foreach ($vm in (get-vm -Location $folder | Sort))  {

$oldip = $vm.Guest.IPAddress[0]

$octets = $vm.Guest.IPAddress[0].Split(".")

#Create the new IP address from the destination network

$octets[0] = $destinationNet[0]

$octets[1] = $destinationNet[1]

$octets[2] = $destinationNet[2]

$newip = [string]::Join(".",$octets)

#Create the DNS Update

$dnsupdates += @{"OldIP" = $oldip; "NewIP" = $newip}

#Change the last octet to .1 for the gateway IP

$octets[3] = "1"

$vmgateway = [string]::Join(".",$octets)

 

#Check if the Guest OS is Windows and update the IP Address

if ($vm.Guest.OSFullName.Contains("Windows")) {

$chNet = "netsh interface ip delete dnsservers Ethernet all; netsh interface ip add dnsservers Ethernet $destinationDNS01; netsh interface ip add dnsservers Ethernet $destinationDNS02 index=2; netsh interface ip set address name=Ethernet static $newip 255.255.255.0 $vmgateway"

Invoke-VMScript -VM $vm -ScriptText $chNet

}

#Check if the Guest OS is Linux and update the IP Address

if ($vm.Guest.OSFullName.Contains("Linux")) {

$chNet = "/root/failover"

Invoke-VMScript -VM $vm -ScriptText $chNet -GuestCredential $linuxCredential

}

}

 

#Shutdown the VMs

Foreach ($vm in (get-vm -Location $folder))  {

Shutdown-VMGuest -VM $vm -Confirm:$false

}

 

#Update the static DNS A record for each VM

foreach ($dnsupdate in $dnsupdates)  {

$oldIP = [string]$dnsupdate.OldIP

$newIP = [string]$dnsupdate.NewIP

echo "Changing DNS Record $oldIP to $newIP"

$oldDNS = Get-DnsServerResourceRecord -zonename test.com -RRType A | where {$_.RecordData.IPv4Address -eq $oldip}

$newDNS = $oldDNS.Clone()

$newDNS.RecordData.IPv4Address = [ipaddress]$newip

Set-DnsServerResourceRecord -ZoneName "test.com" -OldInputObject $oldDNS -NewInputObject $newDNS

}

Remove-PSSession $s

 

Disconnect-NTNXCluster *

Disconnect-VIServer * -Confirm:$false

 

#Wait for the VMs to shut down

Sleep 60

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

# THIS SCRIPT REQUIRES THE NUTANIX POWERSHELL MODULES AND POWERSHELL VERSION 4.0

# Interface of Windows VMs needs to be named "Ethernet"

# VMs must be powered on

# 1. Get list of VMs and IPs from a VMware Folder

# 2. Check that all VMs are in the Protection Domain

# 3. Change the IPs of the VMs and shut them down

# 4. Change the static DNS entries

### DECLARE CONSTANTS ###

#Source Domain Controller

$dnsServer = "dc01.test.com"

#Establish PSSession to target Domain Controller

$s = new-PSSession -Computer $dnsServer

Import-PSSession -Session $s -Module DNSServer

# Need to load PowerCLI and Nutanix modules

Add-PSSnapin NutanixCmdletsPSSnapin

#Make sure that any previous Nutanix or vCenter sessions are disconnected

Disconnect-NTNXCluster *

Disconnect-VIServer * -Confirm:$false

#Name of target VMware folder

$folder = "DRTest"

#Name of the Protection Domain

$protectionDomain = "Test"

#Destination Network - 10.XXX.XXX.0

$destinationNet = "10.0.1.0"

$destinationNet = $destinationNet.Split(".")

#Destination DNS 01

$destinationDNS01 = "10.0.1.11"

$destinationDNS02 = "10.0.1.12"

#Nutanix Cluster IP

$nutanixClusterIP = "10.0.0.51"

#Source vCenter

$vCenter = "vc01.test.com"

### BEGIN SCRIPT ###

#Clear Powershell Console

clear

#Connect to vCenter

Connect-VIServer $vcenter

#Connect to the Nutanix Cluster

Connect-NutanixCluster -Server $nutanixClusterIP -UserName admin -Password admin -AcceptInvalidSSLCerts | Out-Null

#Get the names of the VMs in the target Folder

#Initialize an array to hold the names of the VMs in the folder

$folderVMs=@()

#Add the names of the VMs in the folder to the array

Foreach ($vm in (get-vm -Location $folder)) { $folderVMs = $folderVMs + $vm.Name }

#Initialize an array to hold the names of the VMs in the protection domain

$protectedVMs=@()

#Add the names of the VMs in the protection domain to the array

Foreach ($vm in (Get-NTNXVM | Where {$_.protectionDomainName -eq $protectionDomain} | Select vmName)) { $protectedVMs = $protectedVMs + $vm.vmName }

#compare the arrays of VMs to check if there are any missing VMs and add them to $missingVMs

$missingVMs = Compare-Object $folderVMs $protectedVMs

#Initialize an array to contain VMs that are in the folder but not in the protection domain

$pdMissing = @()

#Initialize an array to contain VMs that are in the protection domain but not in the folder

$folderMissing = @()

#Initialize an array to contain the DNS Updates

$dnsupdates = @()

#Iterate through $missingVMs and notify the user which VMs are missing

if ($missingVMs.Count -ne 0) {

foreach ($vm in $missingVMs) {

if ($vm.SideIndicator -eq "<=") {

$pdMissing = $pdMissing + $vm

}

if ($vm.SideIndicator -eq "=>") {

$folderMissing = $folderMissing + $vm

}

echo "These VMs are in the $folder VMware Folder but not in the $protectionDomain protection domain."

foreach ($pdvm in $pdMissing) { echo $pdvm.InputObject }

echo `n

echo "These VMs are in the $protectionDomain protection domain but not in the $folder VMware Folder."

foreach ($foldervm in $folderMissing) { echo $foldervm.InputObject }

echo `n

#If there are missing VMs then end the script

break

}

#Pop up a dialog box to get the linux username and password and test that the linux credential actually works with a linux VM to prevent an epic fail

#if the credential fails then prompt for the password again

$crTest = "echo test"

do {$linuxCredential = Get-Credential -Username "root" -Message "Type the ROOT username and password for Linux machines"}

Until (Invoke-VMScript -VM (get-vm -Location $folder | where {$_.Guest.OSFullName.Contains("Linux")} | Sort)[0] -ScriptText $crTest -GuestCredential $linuxCredential)

#Get the IP Address of the VM, convert it into an array, and replace the source network with the destination network

Foreach ($vm in (get-vm -Location $folder | Sort)) {

$oldip = $vm.Guest.IPAddress[0]

$octets = $vm.Guest.IPAddress[0].Split(".")

#Create the new IP address from the destination network

$octets[0] = $destinationNet[0]

$octets[1] = $destinationNet[1]

$octets[2] = $destinationNet[2]

$newip = [string]::Join(".",$octets)

#Create the DNS Update

$dnsupdates += @{"OldIP" = $oldip; "NewIP" = $newip}

#Change the last octet to .1 for the gateway IP

$octets[3] = "1"

$vmgateway = [string]::Join(".",$octets)

#Check if the Guest OS is Windows and update the IP Address

if ($vm.Guest.OSFullName.Contains("Windows")) {

$chNet = "netsh interface ip delete dnsservers Ethernet all; netsh interface ip add dnsservers Ethernet $destinationDNS01; netsh interface ip add dnsservers Ethernet $destinationDNS02 index=2; netsh interface ip set address name=Ethernet static $newip 255.255.255.0 $vmgateway"

Invoke-VMScript -VM $vm -ScriptText $chNet

}

#Check if the Guest OS is Linux and update the IP Address

if ($vm.Guest.OSFullName.Contains("Linux")) {

$chNet = "/root/failover"

Invoke-VMScript -VM $vm -ScriptText $chNet -GuestCredential $linuxCredential

}

#Shutdown the VMs

Foreach ($vm in (get-vm -Location $folder)) {

Shutdown-VMGuest -VM $vm -Confirm:$false

}

#Update the static DNS A record for each VM

foreach ($dnsupdate in $dnsupdates) {

$oldIP = [string]$dnsupdate.OldIP

$newIP = [string]$dnsupdate.NewIP

echo "Changing DNS Record $oldIP to $newIP"

$oldDNS = Get-DnsServerResourceRecord -zonename test.com -RRType A | where {$_.RecordData.IPv4Address -eq $oldip}

$newDNS = $oldDNS.Clone()

$newDNS.RecordData.IPv4Address = [ipaddress]$newip

Set-DnsServerResourceRecord -ZoneName "test.com" -OldInputObject $oldDNS -NewInputObject $newDNS

}

Remove-PSSession $s

Disconnect-NTNXCluster *

Disconnect-VIServer * -Confirm:$false

#Wait for the VMs to shut down

Sleep 60

*Update – story behind the script*
Finally I have a few minutes to write the story behind this script.

One of our VMware View environments was experiencing performance problems. The CPUs on our VMs would constantly spike to 100% after they were powered on. Our admins relayed back to engineering that they were having density issues. We reached out to Nutanix who recommended that we increase the cache size to be able to absorb more IOPS. To increase the cache size on Nutanix you simply need to power off the controller virtual machine (CVM) on a host, increase RAM, and power it back on. While is a non disruptive process if you power the CVMs on and off one at a time, it becomes a very disruptive process if someone makes a mistake and powers off more than one CVM at a time. It is also very time intensive because you must check that the CVM services are completely back up before you perform the procedure on the next CVM. With 120 hosts in our environment, and averaging 10 minutes per manual CVM procedure, it looked like it was going to take about 20 hours to perform this task. For us this means 3-4 days in maintenance windows!

I figured there has to be a way to automate this and eliminate the human component so we could perform this maintenance task all in one maintenance window. Well a couple hours of fiddling with powerCLI and trying to figure out which service is the last CVM service to power on, and running the script in our test environment to work out the bugs and we were ready to run it in production. In our environment the average run time per CVM was about 5 minutes, but the best part is that it really saves hours of admin time. An admin only needs to babysit the script while it is running instead of needing to perform an intensive manual process. This shows the huge benefit of Software Defined Storage. Imagine trying to update cache on a traditional SAN without any downtime… isn’t going to happen.

It later turned out that the issue in our environment was a classic VMware View admin mistake of installing updates and then shutting down immediately and recomposing the pool. The updates needed to finish installing after reboot, so they finished installing on all of the linked clones when they powered on. Combined with refresh on logoff which occurs multiple times per day and it was a sure way to test max performance of our equipment!

#Nutanix CVM RAM Add Script
#This script leverages The SSH.NET powershell module from http://www.powershelladmin.com/wiki/SSH_from_PowerShell_using_the_SSH.NET_library
#Download it at: http://www.powershelladmin.com/w/images/a/a5/SSH-SessionsPSv3.zip

Import-Module SSH-Sessions

#This script records the time to collect metrics on how long it takes to perform the memory upgrade.
$scriptStart = (Get-Date)

#Connect to the vCenter Servers
connect-viserver vc01.test.com

#Find all of the Nutanix CVMs that have less than 24GB RAM
$vms = Get-VM -name NTNX* | Where MemoryGB -lt 24

#Sort the CVMs by IP address (just to watch the CVMs be done in order)
$vms = $vms | Sort-Object guest.IPAddress[0]

#Loop though the CVMs and upgrade them one at a time
foreach ($vm in $vms) {

   #Use the IP address of the CVM to connect to it with SSH
   $CVM = $vm.guest.IPAddress[0]

   #Using the default user/pass
   New-SshSession -ComputerName $CVM -Username 'nutanix' -Password 'nutanix/4u'

   #Check to make sure that the CVMs in the cluster are all up.  If a CVM is not UP it will be in DOWN state.
   $result = Invoke-SshCommand -ComputerName $CVM -Command '/home/nutanix/cluster/bin/cluster status | grep Down'
   Remove-SshSession -RemoveAll

   #Perform memory upgrade if there are no CVMs DOWN and SSH connection was successful   
   If ($result -NotLike '*Down*' -and $result -notlike "*No SSH session found*")  {
      write-host "Shutting down $CVM"
      Shutdown-VMGuest $vm -Confirm:$false

      #Wait a period of time to make sure the CVM is shutdown before changing settings
      sleep 60

      #Set CVM memory
      write-host "Setting $CVM Memory"
      Set-VM $vm -MemoryGB 24 -Confirm:$false

      #Power-on CVM
      write-host "Starting $CVM"
      Start-VM $vm -Confirm:$false
    
      #Wait for CVM to start before checking that it is UP
      sleep 60
      write-host "Checking $CVM state"

      #Check that the services are started on the CVM before performing the upgrade on the next CVM.  From what I could tell by watching the services start, alert_manager is the last service to start.

      Do {
         New-SshSession -ComputerName $CVM -Username 'nutanix' -Password 'nutanix/4u'
         $result = Invoke-SshCommand -ComputerName $CVM -Command "/home/nutanix/cluster/bin/genesis status | grep alert_manager"
         Remove-SshSession -RemoveAll

         #If the services are not started yet they will have a status of []
         If ($result -contains 'alert_manager: []') {write-host "$CVM Down"}
         #Wait before attempting to make another SSH connection
         sleep 5
      }  
      #If the service is started there will be port numbers in the brackets.  Check to see if the brackets are empty.  Make sure to escape the [] characters with `.
      Until ($result -notlike '*`[`]*' -and $result -notlike "*No SSH session found*")  
      write-host "$CVM Up"
      $scriptEnd = (Get-Date)

      #Calculate the timespans
      $totalTime = New-Timespan -Start $scriptStart -End $scriptEnd

      #Write a time summary
      write-host "Total Script Time"
      write-host $totalTime
   }
   Else { 
   #If a CVM is down at the beginning of the script, just end the script.
   write-host "Cluster not ready."
   Disconnect-VIServer -Confirm:$false   
   Break
   }
}
Disconnect-VIServer -Confirm:$false

#Nutanix CVM RAM Add Script

#This script leverages The SSH.NET powershell module from http://www.powershelladmin.com/wiki/SSH_from_PowerShell_using_the_SSH.NET_library

#Download it at: http://www.powershelladmin.com/w/images/a/a5/SSH-SessionsPSv3.zip

Import-Module SSH-Sessions

#This script records the time to collect metrics on how long it takes to perform the memory upgrade.

$scriptStart = (Get-Date)

#Connect to the vCenter Servers

connect-viserver vc01.test.com

#Find all of the Nutanix CVMs that have less than 24GB RAM

$vms = Get-VM -name NTNX* | Where MemoryGB -lt 24

#Sort the CVMs by IP address (just to watch the CVMs be done in order)

$vms = $vms | Sort-Object guest.IPAddress[0]

#Loop though the CVMs and upgrade them one at a time

foreach ($vm in $vms) {

#Use the IP address of the CVM to connect to it with SSH

$CVM = $vm.guest.IPAddress[0]

#Using the default user/pass

New-SshSession -ComputerName $CVM -Username 'nutanix' -Password 'nutanix/4u'

#Check to make sure that the CVMs in the cluster are all up. If a CVM is not UP it will be in DOWN state.

$result = Invoke-SshCommand -ComputerName $CVM -Command '/home/nutanix/cluster/bin/cluster status | grep Down'

Remove-SshSession -RemoveAll

#Perform memory upgrade if there are no CVMs DOWN and SSH connection was successful

If ($result -NotLike '*Down*' -and $result -notlike "*No SSH session found*") {

write-host "Shutting down $CVM"

Shutdown-VMGuest $vm -Confirm:$false

#Wait a period of time to make sure the CVM is shutdown before changing settings

sleep 60

#Set CVM memory

write-host "Setting $CVM Memory"

Set-VM $vm -MemoryGB 24 -Confirm:$false

#Power-on CVM

write-host "Starting $CVM"

Start-VM $vm -Confirm:$false

#Wait for CVM to start before checking that it is UP

sleep 60

write-host "Checking $CVM state"

#Check that the services are started on the CVM before performing the upgrade on the next CVM. From what I could tell by watching the services start, alert_manager is the last service to start.

Do {

New-SshSession -ComputerName $CVM -Username 'nutanix' -Password 'nutanix/4u'

$result = Invoke-SshCommand -ComputerName $CVM -Command "/home/nutanix/cluster/bin/genesis status | grep alert_manager"

Remove-SshSession -RemoveAll

#If the services are not started yet they will have a status of []

If ($result -contains 'alert_manager: []') {write-host "$CVM Down"}

#Wait before attempting to make another SSH connection

sleep 5

}

#If the service is started there will be port numbers in the brackets. Check to see if the brackets are empty. Make sure to escape the [] characters with `.

Until ($result -notlike '*`[`]*' -and $result -notlike "*No SSH session found*")

write-host "$CVM Up"

$scriptEnd = (Get-Date)

#Calculate the timespans

$totalTime = New-Timespan -Start $scriptStart -End $scriptEnd

#Write a time summary

write-host "Total Script Time"

write-host $totalTime

}

Else {

#If a CVM is down at the beginning of the script, just end the script.

write-host "Cluster not ready."

Disconnect-VIServer -Confirm:$false

Break

}

Disconnect-VIServer -Confirm:$false

Josh Sinclair

Category Archives: PowerCLI

Use PowerCLI to Automate Disaster Recovery Failover On Nutanix

Change Nutanix CVM RAM with PowerCLI

Technologist