Compress thin-provisioned VMDKs even further

Here in the lab I’m running out of resources to run all my VMs concurrently on the one host.  Rather than add another physical host at this site (or switch off some VMs), and with long-distance vMotion now available in vSphere 6.0, I decided to place another at my secondary site in the UK.

Before the move to vSphere 6.0 (I’m currently studying for my VCAP so want to stay on 5.5 for the time being), I decided it would be best to migrate my 14-host Exchange 2010 environment over the secondary site, and setup SRM in the process.  However before I could do that I needed to setup vSphere Replication to get the actual VMs over.

With the VPN to the secondary site not being the quickest, it was important to compress my already thin-provisioned VMs down as much as possible.

It’s an old one, but always worth doing in this type of scenario.

On each VM, I downloaded Mark Russinovich’s SDelete and ran it on each of the Windows drives:

sdelete -z

This balloons each VMDK up to its maximum size, but don’t worry about that for now.  When it finished, I powered down each VM and switched to the ESXi host console (using SSH) and ran:

vmkfstools -K /vmfs/volumes/path-to-VM/VM.vmdk

The important thing here to remember is to target the VMs normal VMDK, not the one ending in “-flat.vmdk”.

After a while it finished and freed up a lot of space, ready to be replicated using vSphere Replication 5.8!


Each VM still took two days though 😦

Recovering damaged VMFS partitions

Last year a client suffered a power outage at one of their major sites.  Unfortunately the Powerchute installation I had configured on a vSphere vMA didn’t work as expected, and all hosts, storage and networking equipment died when the UPS ran out of juice.  This however, was only the beginning of what was to become a very long day.

When power was restored, a member of the Operations team brought all the kit back on-line, but unfortunately the SAN controllers came up before the expansion arrays… therefore marking them as dead.  When the ESXi hosts came back up, they were missing quite a few LUNs, and got visibly upset.

Despite all the storage controllers and arrays being online, ESXi refused to recognise any of the partitions and only offered the ever-helpful option of adding new storage (and therefore destroying what was already there).  That’s when Ops decided to escalate the issue.

After calling VMware support, the prognosis was not good… the data was lost and we had to restore from backup.  Knowing the client would not be happy with this, I decided to step-in and take over.

I didn’t believe the data was lost, but merely needed a little nurturing to bring it back to life.  First, the storage had to be brought back in the right order:

  1. Shutting down all ESXi servers
  2. Disconnecting controller B fibre
  3. Disconnecting controller A fibre
  4. Shutting down enclosure 2
  5. Shutting down enclosure 1
  6. Shutting down controller B
  7. Shutting down controller A

If the SAN controllers can’t see all the arrays, then the hosts have no chance of seeing the LUNs.

Then at two minute intervals:

  1. Reconnecting controller A and B fibre
  2. Powering up enclosure 2
  3. Powering up enclosure 1
  4. Starting controller A
  5. Starting controller B
  6. Powering on ESXi host 1

Rescanning the LUNs still gave me nothing, so I SSH’d onto the host and listed the disks it could see using:

esxcli storage core path list | grep naa

This gave me two devices:


Then I listed the partition table (on each disk) using:

partedUtil getptbl /vmfs/devices/disks/naa.60080e50002ea8b600000318524cd170

This showed as empty (ie. no entry starting with “1” or the word “vmfs” anywhere) – not good.  I then checked for the beginning and end blocks on each disk (thank you VMware for the following command):

offset="128 2048"; for dev in `esxcfg-scsidevs -l | grep "Console Device:" | awk {'print $3'}`; do disk=$dev; echo $disk; partedUtil getptbl $disk; { for i in `echo $offset`; do echo "Checking offset found at $i:"; hexdump -n4 -s $((0x100000+(512*$i))) $disk; hexdump -n4 -s $((0x1300000+(512*$i))) $disk; hexdump -C -n 128 -s $((0x130001d + (512*$i))) $disk; done; } | grep -B 1 -A 5 d00d; echo "---------------------"; done

I then listed the usable sectors:

partedUtil getUsableSectors /vmfs/devices/disks/naa.60080e50002ea8b600000318524cd170

That came back with two numbers.  I needed the large one (4684282559).  I then rebuilt the partition table using:

partedUtil setptbl /vmfs/devices/disks/naa.60080e50002ea8b600000318524cd170 gpt "1 2048 4684282559 AA31E02A400F11DB9590000C2911D1B8 0”

So to recap,

naa.60080e50002ea8b600000318524cd170 = disk device
2048 = starting sector
4684282559 = end sector
AA31E02A400F11DB9590000C2911D1B8 = GUID code for VMFS

I then mounted the partition:

vmkfstools –V

After repeating the above command for the second disk, ESXi host 1 then saw both partitions.  I could then bring up all the VMs and the remaining ESXi hosts.

Bullet…. dodged.