Update (16 April 2014): see bottom of blog
I recently blogged about Windows Server 2012 R2 Update. As usual any update and certainly an update as large as this, has some risks. Therefore we usually advise to postpone Windows Updates, Update Rollups an Hotfixes and leave a couple of weeks before deploying updates in production. Always test in a lab if you can and if you can’t, keep an eye on the forums and the blogs from MVP’s specializing in the related technology.
Testing Host Level Hyper-V Backup
This weekend I came across a tweet from Richard Skinner who reported an issue related to Veeam Backup & Replication and Hyper-V Backup after applying the Windows Server 2012 R2 (Spring) Update. Meanwhile, Veeam had already confirmed the problem and was working frantically into the weekend to fix this nasty problem.
I decided to report a support case with Veeam as well, even though I’m only running it in a Windows Azure Pack lab. I found that the problem was easily reproducible, but only if VSS was enabled in the backup job.
The Problem: Using Hyper-V Checkpoints
When this selection is made as an alternative to Veeam’s default changed block tracking (CBT), the backup fails because it cannot deal with the file path of the checkpointed VHDX files. When you zoom in on the directory of a VM that is being protected, as soon as VSS kicks in, a checkpoint is made of the active disks. This causes the writes to be redirected from the VHDX to a corresponding AVHDX file which makes it possible for the backup software to take a clean and ‘frozen’ copy of the virtual hard disk. When the backup is ready, the written data to the AVHDX file is merged back into the VHDX file. Only briefly you’ll see n AutoRecovery.avhdx file created which is deleted when it is ready with the merge operation.
Important: Microsoft started to coin the term checkpoint in VMM. After Hyper-V had used the term Snapshot for a long time, this changed with Windows Server 2012 R2. We can now better distinguish between VSS snapshots and Hyper-V checkpoints. Backup software now uses Hyper-V checkpoints just as in Hyper-V Replica.
If you want to read more about the changed method of Hyper-V backup in Windows Server 2012 R2, please take a look at fellow MVP Aidan Finn’s post:
All VMMs failed to backup and were impacted by WMI incorrectly returning a proper path for the AutoRecovery.avdhx file.
Looking at the Veeam log files, I noticed that a \ was missing from the AVHDX file path:
Failed to open file [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy14\VM\HANS-VM01Windows Server 2012 R2 Datacenter_disk_1-AutoRecovery.avhdx] in readonly mode.
[06.04.2014 15:01:25] <15> Error Client error: The system cannot find the file specified.\nFailed to open file [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy12\VM\HANS-VM01Windows Server 2012 R2 Datacenter_disk_1-AutoRecovery.avhdx] in readonly mode.\n (System.Exception)
Just before the general release of KB 2919355, Veeam released an update for the code that talks to the VSS Hyper-V Integration components. In fact that response was very prompt and successful. The fix was to rename the Veeam Hyper-V Integration files and replace them by the updated files provided by Veeam. All Hyper-V hosts and Off-Host Backup Proxies should be updated.
Don’t forget to restart the Hyper-V Virtual Machine Management service.
Mind you, these performance numbers are not representative of a state-of the-art production environment. For this test I was using an HP BL460c Gen1 blade which is officially unsupported for Hyper-V.
So thanks Veeam for a fast resolution!