Instead of using U-Move, if you attempt to move or copy Active Directory using a disk image utility (for example VMware, Symantec Ghost, or Acronis True Image), you may encounter errors with replication due to “USN rollback”.
When USN rollback occurs the following message may appear in the Event Log: “The Active Directory database has been restored using an unsupported restoration procedure. Active Directory will be unable to log on users while this condition persists.” (NTDS General, Event ID 2103)
What is USN Rollback?
A domain controller tracks objects in AD based on their Update Sequence Numbers (USN). Every object in AD has a USN. As objects are modified, the USN increases monotonically, like an odometer on a car. The latest USN on each DC is called the “high water mark”. During replication each DC compares its USN high water mark with the USN high water mark of its neighbors.
USN rollback happens when an older copy of Active Directory is restored but the computer fails to notify the other domain controllers that it was rolled back to an out-of-date copy of AD (and therefore that its high water mark has rolled back).
When you use U-Move to restore AD it notifies the other DCs that it has been rolled back. The other DCs respond by “playing back” all changes made to AD since then, bringing the restored computer up to date.
However, if you use a disk imaging utility (for example, if you restore an old disk image created with Symantec Ghost or Acronis True Image), the computer will be unaware that it has been rolled back. If the restored disk is older than the most recent actual disk that successfully replicated with the other domain controllers, any more recent changes made to AD on other domain controllers will not be “played back” to the out-of-date DC. This is because the restored DC is unaware that it has been rolled back.
USN Rollback With VMware or Hyper-V
USN rollback can happen if you use VMware or Hyper-V to roll back a virtual DC to a prior snapshot without simultaneously rolling back all the other virtual DCs. .
When running a DC inside a virtual machine (VM), USN Rollback is generally not an issue if the guest OS is running Windows Server 2012 or later and the VM host is running at least VMware version 9 or Hyper-V version 3. This is because VMware and Hyper-V will change the VM Generation ID to allow the restored VM to notify the other DCs that it was rolled back.
Consequences of USN Rollback
When another DC detects a replication request with a rolled-back USN, it instructs the rolled-back DC to initiate the following “quarantine” procedure:
- Pause the NETLOGON service in order to prevent the processing of any further user logon requests or user password change requests
- Disable any further replication
- Generate Event ID 2103 in the Directory Service event log
- Generate Event ID 2095 in the Directory Service event log: “During an Active Directory replication request, the local domain controller (DC) identified a remote DC which has received replication data from the local DC using already-acknowledged USN tracking numbers. Because the remote DC believes it is has a more up-to-date Active Directory database than the local DC, the remote DC will not apply future changes to its copy of the Active Directory database or replicate them to its direct and transitive replication partners that originate from this local DC. The most probable cause of this situation is the improper restore of Active Directory on the local domain controller. User Actions: If this situation occurred because of an improper or unintended restore, forcibly demote the DC.”
How to Avoid USN Rollback
To prevent USN rollback always use an Active Directory-aware backup utility such as U-Move to restore or move Active Directory. U-Move can restore AD from any disk image including a VM snapshot. U-Move will contact the other DCs and arrange to play back all changes to bring the restored DC up to date.
How to Fix USN Rollback
If the DC has been quarantined due to USN rollback, use one of the following procedures to recover the DC:
- Restore Active Directory from a System State backup that was taken before Event ID 2095 was generated. Note that the System State can only be restored on the same VM or computer from where the backup was taken.
- Use U-Move to replace the bad AD database with a good copy. The good copy can come from from any supported source such as a VM snapshot, a Volume Shadow Copy (Previous Version) snapshot, an unbootable hard disk, an NTBACKUP file, or a Windows Server Backup. U-Move will arrange to play back all changes to bring the good copy up to date. (If the Windows Server Backup was written to DVD or BD disks use a utility such as U-Recover to read the backup image from the disks.)
- Run DCPROMO or Server Manager to demote the domain controller, then re-promote it again. This requires that you have a second good DC that is serving the domain. You might need to erase the metadata for the demoted DC before promoting it again. (See the technical articles below).
- Last-ditch method: Use this method only as a last resort.
Demote all other DCs from the domain, leaving the
bad DC as the only one remaining. Reset the bad DC's rollback status, seize the FSMO roles, and finally re-promote the other DCs back again.
Warning: Do not attempt this method if you have other domains in your Active Directory forest.
- Run U-Move to create a backup snapshot of AD on every DC. This gives you a safety checkpoint so you can undo your work if something goes wrong.
- Disconnect the network cable from the bad DC. Note: Never disable the network interface controller (NIC) in Device Manager. Doing so will prevent loopback connectivity to AD within the domain controller itself. Instead disconnect the network cable.
- On the bad DC run Regedit and locate the registry value Dsa Not Writable under HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NTDS\Parameters. If Dsa Not Writable has the value 4 (0x00000004) it indicates the DC is in USN rollback state. If the value is not 4 (or is missing), then stop; the DC is not in USN rollback state. You need to investigate other reasons why replication is not working. See KB2023007.
- If the value is 4, right-click and delete the registry value.
- Add the registry value Database restored from backup with the REG_DWORD value 1 (0x00000001).
- USN rollback state will have previously disabled all of the replication links on the bad DC. Turn replication back on:
repadmin /options MyServer -DISABLE_OUTPUT_REPL
repadmin /options MyServer -DISABLE_INBOUND_REPL
- Restart the computer. Note that during the restart there might be a delay of up to 30 minutes until the logon prompt appears. This is because the DC is attempting to replicate with the other DCs to check if it is up to date. After roughly 30 minutes it will time out and you can log on. Check the Event Log under Directory Services to verify that the DC is no longer in USN rollback state.
- While the restored DC is still disconnected from the network, give the DC a Global Catalog if it does not already have one, then seize all of the domain-level FSMO roles.
- On the restored DC do a forced removal of the other DCs in order to to delete the metadata of the other DCs from Active Directory (as if they were all dead). Make sure the restored DC is still disconnected from the network.
- Connect the restored DC back to the network.
- Re-promote the other DCs that had you had just demoted. They will fetch a clean copy of AD from the restored DC.
- Redistribute the FSMO roles to their proper locations and clean up.
For more information
For more information about USN rollback see the Microsoft articles:
- “How to detect and recover from a USN rollback in Windows Server 2003, Windows Server 2008, and Windows Server 2008 R2” (KB875495).
- “Running Domain Controllers in Hyper-V: Operational Considerations for Virtual Domain Controllers” (https://technet.microsoft.com/en-us/library/virtual_active_directory_domain_controller_virtualization_hyperv.aspx).
The above articles discuss using an “Active Directory-aware backup utility” versus other methods. U-Move is an “Active Directory-aware backup utility”.
|U-Move for Active Directory|