There is a lot to say about System Center VMM 2012 SP1. Some say System Center VMM 2012 is a must for any serious Private Cloud, others say some of its features should have been incorporated into the Windows Server OS. Early adopter of Windows Server 2012 were forced to design their environment without System Center VMM and are facing some challenges when to want to move their management functionalities to System Center VMM 2012 SP1. I did a couple of sessions on networking in System Center VMM 2012 SP1. Most Fabric Administrators find this part of VMM a bit daunting, to put it mildly.
No matter what you think of System Center 2012 SP1, if you decide to use System Center VMM 2012 SP1 to manage your Private, Hybrid or Hosted Cloud (and you should) you will find that with some features will create dependencies on System Center VMM 2012 SP1. A solid basis for your management environment requires redundancy at all levels.
In this blog I will walk you through the step to create a High Available (HA) System Center 2012 SP1 environment. The design exists of a Hyper-V cluster for physical redundancy, a guest based SQL Server 2012 AlwaysOn Availability Group for database redundancy, a guest based System Center VMM 2012 SP1 failover cluster for VMM redundancy and a Scale Out File Server (SOFS) for Library redundancy.
The quorum settings for the VMM cluster and the SQL cluster are based on Node and File Majority. In a single location design the preferred additional cluster vote is a quorum disk. The advantage of a Node and File Majority is that you can divide the management environment over two geographical locations without the need for synchronous replication of a SAN based quorum disk. You could use a Windows Azure Virtual Machine to provide the file share for the SQL and the VMM cluster node and file majority.
A Scale Out File Server (SOFS) requires a shared disk for a high available share. Each location will have its own SOFS. This blog describes to steps to configure the SQL AlwaysOn Availability Group and the System Center VMM Failover Cluster.
If you need guidance on how to set up a Scale Out File Server check this great blog by Jose Barreto.
Before you start make sure you have the following prepared.
- Active Directory domain
- Windows Server 2012 host cluster dedicated for management
- Scale Out File Server with the following shares
- Two virtual machines, domain joined and named SQL1 and SQL2
- SQL1 and SQL2 have an additional VHDX for storing the databases
- Two virtual machines, domain joined and named VMM1 and VMM2 with a minimum of 4GB Memory
- It is recommended (but not required) that you have two Hyper-V switches in each physical host connected to NIC teams. Add both Hyper-V switches to the virtual machines SQL1, SQL2, VMM1 and VMM2. In the virtual machine configure one NIC for management (with a default gateway) and one NIC for Cluster traffic (without a default gateway). It is also possible to add two virtual network adapters to a virtual machine, attached to the same Hyper-V switch, that connects to one NIC team.
Required user accounts
You can designate separate service accounts for each SQL Server 2012 service and you will create additional Run As accounts in System Center VMM 2012 SP1. The following domain user accounts are required as a minimum:
- SQL Server service account (domainSVCSQL)
- VMM service account (domainSVCVMM)
SQL Server 2012 AlwaysOn Availability Groups
Logon to SQL1 as a domain administrator and install the following features
- .Net Framework 3.5
- Failover Clustering
After a default Windows Server 2012 installation the .NET Framework 3.5 installation files are not location on your hard drive. When you add the feature, Server Manager allows you to specify an additional path where the files are located.
After repeating these steps on SQL2 open the Failover Cluster console from the Server Manager. Select to validate a configuration and add server SQL1 and SQL2 to the test. Specify to run all test and complete the Wizard.
The Failover cluster wizard performs several tests to check if SQL1 and SQL2 can be clustered. Since there is no shared storage, some tests will generate a warning. In you have not created a second Hyper-V switch for Cluster traffic, additional warnings will be generated. When the wizard completes without errors, you can create the cluster using the validated nodes.
Specify the name and IP address of the cluster. Please take note that this name is not the Client Access Point.
When the cluster is successfully configured, logon to SQL1 with as a domain administrator.
SQL Server 2012
Create a firewall rule on SQL1 and SQL2 that allows inbound TCP port 1433 and TCP port 5022. These ports are required for the SQL Server AlwaysOn Availability Group.
Start the installation of SQL Server 2012. Select a new SQL Server stand-alone installation. Include the SQL Server updates, select SQL Server Feature Installation and specify what features you want to install. For this blog we will choose the minimum. Database Engine Services and the Management Tools. Select to use the default instance.
In the server configuration tab of the installation wizard provide the domain service account and password we created earlier.
System Center VMM 2012 SP1 is compatible with the default SQL Server 2012 collation Latin_General_CI_AS.
The Database Engine Configuration tab of the installation wizard allows you to select an Authentication Mode. For System Center VMM 2012 SP1 HA you can use Windows authentication mode. SQL Server 2012 SP1 AlwaysOn Availability Groups replicates databases that are stored on local disks on the SQL Servers. Shared storage is not required. Specify a location for the databases on the Data Directories Tab.
Complete the installation and repeat the SQL Server 2012 installation steps on SQL2.
Logon to SQL1 and open SQL configuration and services, right click SQL Server service and select properties.
Open the AlwaysOn High Availability tab and enable AlwaysOn Availability Groups.
Restart the SQL Server Service and repeat these steps on SQL2.
We need to enable the Availability Group before we can use the Availability Group Listener. This procedure requires a database. We will manually delete this temporary database when System Center VMM 2012 SP1 has created its database in the Availability Group. Each database that will be added to the availability group must meet two prerequisites.
- Database recovery model must be set to Full
- A backup of the database is completed
Logon to SQL1 as a domain administrator. Open SQL Server Management Studio. Create a database. The name is not really relevant. I used Enable AlwaysOn so the role of the database is clear.
Open the properties of the SQL server, select the Logging tab and verify that the recovery model is set to Full.
Apply the settings and perform a backup of the database to local disk.
In the SQL Server Management Studio right click Availability Groups and start the New Availability Group Wizard. Specify a group name for the availability group. This group name is only used as a label in SQL Server. The select databases tab display all the databases on SQL1 and if these databases meet the prerequisites to be added to the availability group. Select the database you just created.
The specify replicas screen shows four tabs. In the replica tab select Add Replica and connect to SQL2 with Windows Authentication. Enable the checkmarks on Automatic Failover and Synchronous Commit on both Replicas.
Open the Listener tab and select Create an availability group listener. Specify a Listener DNS name, a port and IP address. We will use the Listener DNS name later when configuring System Center 2012 SP1.
On the Select Data Synchronization screen select full data synchronization and specify a shared network location for initial synchronization. The share will be used to create a backup of the database by SQL1 and a restore of the backup on SQL2.
With the prerequisites in place the Validation screen should pass successfully. When the wizard runs you will probably end up with an error on the Create Availability Group Listener step.
Clicking on the Error hyperlink will display SQL Server error: 19471
The WSFC duster could not bring the Network Name resource with DNS name ‘SQL’ online. The DNS name may have been taken or have a conflict with existing name services, or the WSFC duster service may not be running or may be inaccessible. Use a different DNS name to resolve name conflicts, or check the WSFC duster log for more information. The attempt to create the network name and IP address for the listener failed. The WSFC service may not be running or may be inaccessible in its current state, or the values provided for the network name and IP address may be incorrect, check the state of the WSFC duster and validate the network name and IP address with the network administrator.
This error message can be somewhat misleading. The real cause for this issue is that the SQL cluster virtual network name account does not have permissions on its OU in Active Directory to create computer objects. To resolve the issue open the Active Directory Users and Computers console right click the Organizational Unit that contains the SQL Server computer accounts and the SQL Server cluster virtual network name account and select properties.
Open the security tab and click Advanced. Select Add and add the Create Computer objects permissions for the SQLCluster account. Apply these settings to this object and all descendant objects.
The previous New Availability Group wizard created a backup on the file share and restored a database copy on SQL2. We need to delete the backup file on the file share and the restored database on SQL2 before running the New Availability Group wizard again on SQL1. The New Availability Group wizard should complete successfully now.
System Center Virtual Machine Manager 2012 SP1 high availability – the first node
With a high available SQL Server in place we are ready to install System Center 2012 SP1. High Availability in System Center VMM 2012 SP1 also uses Windows Server 2012 failover clustering. Install the Failover Clustering feature on VMM1 and VMM2. Logon to VMM1 as a domain administrator and open the Failover Cluster Console from the Server Manager. Select to validate a configuration and add server VMM1 and VMM2 to the test. Specify to run all test and complete the wizard.
Start the Create Cluster wizard after the tests complete without errors. You need to specify a name and IP address for the Cluster (the Client Access Point will be created later by the System Center 2012 SP1 installation wizard).
System Center VMM 2012 SP1 requires the Deployment Tools and the Windows Preinstallation Environment features from the Windows Assessment and Deployment Kit (ADK). You can download the Windows Assessment and Deployment Kit here (http://www.microsoft.com/en-us/download/details.aspx?id=30652). Select the required features in the ADK installation wizard.
Before we start the System Center VMM 2012 SP1 setup add the VMM service account (hypervnuSVCVMM) to the local administrators group on server VMM1 and VMM2.
Logon to server VMM1 as a domain administrator and run the System Center 2012 VMM SP1 setup. The prerequisite check should complete without warnings or errors. In the Database configuration screen specify the SQL Server 2012 AlwaysOn Availability Group DNS listener we created earlier (in this example SQL). The wizard will communicate with the SQL server to verify the SQL Server environment. If you have not changed the default port in SQL Server it is not necessary to enter the port number in the wizard.
The cluster configuration screen allows you to specify the System Center VMM 2012 SP1 cluster name and IP address (in this example VMM). These settings are used when clients are accessing the environment.
VMM encrypts some data in the VMM database. In a high available System Center VMM 2012 SP1 environment multiple servers need to access this encrypted data. The setup wizard allows you to specify a container in Active Directory to store the encryption keys. This container should be created before continuing with the setup wizard.
Logon to a domain controller, open adsiedit.msc and connect to the default naming context. Right click the domain object and select New > Object.
Select the container class and specify VMMDKM as value. Right click the CN=VMMDKM container you just created and select properties. Select the security tab and give the VMM Service account created earlier (in this example hypervnuSVCVMM) Full control on this object and all descendant objects.
When the object is created and the permissions are applied we can continue with the System Center 2012 SP1 setup wizard. In the Configure service account and distributed key management screen specify the VMM service account and password we created earlier. In the Distributed key Management section specify the common name for the VMMDKM object we just created.
Please take note of the misleading dot at the end of the example. You could mistake the example to enter the location is the following format.
This dot at the end will result in the following error.
The error is not related to permissions. Entering the common name without the ending dot will result in a successful validation by the setup wizard.
The Port configuration screen allows you to change the ports used for various VMM features. It is not supported to host a Virtual Machine Library on a System Center VMM 2012 SP1 node in a high available environment. The setup wizard will prevent you from doing so by greying out the Library configuration screen.
When the setup wizard completes you will probably end up with the following errors.
- Starting the clustered VMM service VMM failed. Ensure that the user has permission, the VMM service is installed properly, and cluster resources can be brought online.
- A service connection point (SCP) could not be registered in Active Directory Domain Services (AD DS) for the VMM management server. Run “C:Program FilesMicrosoft System Center 2012Virtual Machine ManagersetupConfigureSCPTool.exe –install VMM.hyper-v.nu HYPERVNUVMM$” in a command windows and then check AD DS. If a SCP is not registered, VMM consoles on other computers will not be able to connect to this VMM management server and deploying a Hyper-V host to a bare-metal computer will not work.
Starting the clustered VMM service failed
A high available System Center VMM 2012 SP1 installation is configured as a generic cluster service. When you open the failover cluster manager on VMM1 you will notice that the VMM clustered service is in the status failed.
The Failover cluster virtual network name account does not have the required permissions on its Organization Unit in Active Directory. To resolve this issue logon to a domain controller and open an Active Directory Users and Computers console. Right click the Organizational Unit where the vmmcluster.hyper-v.nu failover cluster virtual network name account is located and select properties. Give create computer objects permissions to the vmmcluster.hyper-v.nu failover cluster virtual network name account on this object and all descendant objects.
After applying the settings open the failover cluster manager on VMM1 again. You should now be able to start the VMM clustered service. When the clustered service is started an additional failover cluster virtual network name account (in our example VMM) is created in the same Organization Unit.
A service connection point (SCP) could not be registered in Active Directory
The service connection point for System Center VMM 2012 is a child object to the failover cluster virtual network name account that was created in the previous step. Since this parent object could not be created during the installation, the creation of the child object also failed. Before we can resolve this issue the VMM clustered service must be successfully started (which ensures the required object in Active Directory is present).
Open a command prompt on VMM1 as domain administrator and run the following commands.
Cd C:Program FilesMicrosoft System Center 2012Virtual Machine Managersetup
ConfigureSCPTool.exe -Install <Client Access Point FQDN> <domain><Client Access Point NetBIOS Name>$
In this example
ConfigureSCPTool.exe -Install VMM.hyper-v.nu HYPERVNUVMM$
This command creates a service connection point object as a child object to the Client Access Point in Active Directory. You can verify the creation of the object by opening adsiedit.msc on a domain controller. The SCP has a common name of MSVMM.
If you skip this step and install the second VMM server in the cluster the setup wizard will result with the same error but a different command to resolve the issue.
Cd C:Program FilesMicrosoft System Center 2012Virtual Machine Managersetup
ConfigureSCPTool.exe -AddNode <Client Access Point FQDN> <domain><Client Access Point NetBIOS Name>$
In this example
ConfigureSCPTool.exe -AddNode VMM.hyper-v.nu HYPERVNUVMM$
Please take not that running the AddNode command will not create the SCP.
Preventing the issues
The issues can be prevented by applying the create computer objects permissions to the vmmcluster failover cluster virtual network name account before running System Center VMM 2012 SP1 setup.
Enable high availability for the System Center VMM 2012 SP1 database
Before adding the second node to the VMM cluster it is a good idea to test the availability of the database in the SQL Server AlwaysOn Availability Group. Login to SQL1 and open the SQL Server Management Studio. To prepare the VMM database for the availability group the logging model must be adjusted to Full. Open the properties of the VirtualManagerDB database and select the options tab. Change the Recovery model from Simple to Full.
Next an initial backup of the database must be performed before we can add it to the availability group. After the initial backup is completed right click the Availability Databases entry of the Availability Group (in this example SQL) and select Add Database.
The Add Database to Availability Group wizard will display all available databases and if they meet the requirements to be added to the Availability Group. Select the VirtualManagerDB database. Select a full data synchronization and complete the wizard.
The VirtualMachineDB database is now part of the SQL Server AlwaysOn Availability Group and can be moved to SQL2 by failing over the availability group. Open the Failover Cluster Manager on SQL1, right click the SQL clustered service, select move and select node SQL2.
The failover will complete successfully. But the Failover Cluster manager on VMM1 will display the VMM clustered service in a failed status.
SQL Server Availability Groups synchronizes databases between the cluster nodes. Other settings are not synchronized. When you compare the Logins between SQL1 and SQL2 you will notice that SQL2 does not have a login for the VMM service account (in this example hypervnuSVCVMM).
Right click logins in SQL Server management studio connected to SQL2 and select add. Add the VMM service account (in this example hypervnuSVCVMM). The database already contains the settings for this domain account so other settings do not need to be changed.
After adding the account the VMM clustered service can be started successfully. The SQL Server Availability group is now configured correctly for System Center VMM 2012 SP1.
System Center Virtual Machine Manager 2012 SP1 high availability – the second node
With the VMM database in an operational SQL Server AlwaysOn availability Group we are ready to add the second node to the System Center VMM 2012 SP1 cluster. Before running the setup wizard verify the prerequisites on VMM2 are completed.
- Installed the Deployment Tools and the Windows Preinstallation Environment features from the Windows Assessment and Deployment Kit (ADK)
- Added the VMM service account (hypervnuSVCVMM) to the local administrators group
After verifying the prerequisites start the System Center VMM 2012 SP1 setup wizard. The setup wizard will detect that VMM2 is part of a Failover Cluster and prompt you to add this server as a node.
The Database configuration screen will be grayed out. In the Configure service account and distributed key management screen specify the password for the VMM service account. With all prerequisites in place the setup wizard should complete successfully.
Open a Failover Cluster manager console on VMM2 and move the VMM clustered service from VMM1 to VMM2. With all servers in place you now own yourself a high svailable System Center 2012 SP1 environment, with high availability at the host level, the database level and the VMM level.
I got a small slap on the wrist from Allan Hirt (twitter @SQLHA) on using the proper terminology for SQL Server AlwaysOn Availability Groups. Reading his post AlwaysOn is the New Active/Passive and Active/Active made me think of people that reply to your email but misspell your name. Come on, it’s right there at the bottom of my initial mail. Allan, I apologize ;-) I rectified Always On to AlwaysOn.