Hey y'all, Mark here providing an AskPFEPlat late night PSA. We are seeing some reports that some Active Directory Domains times reverting to November 2000. So what just happened? I thought the Mayans said this type of stuff didn’t happen until December.

UPDATE: If you are unsure of these actions contact CTS, they will help you through it. We will be providing an in-depth post around this sometime in the future. Don't wait for this future post to solve your issue today however, call CTS. 

Check where your root PDC is getting its time settings from. In your root domain run the command w32tm /monitor. You can also run the command w32tm /monitor /domain:domain for other domains.

See where mine says ContosoDC01.Contoso.com****PDC*** and then NTP saying it’s getting time from itself. I’m willing to bet you are getting it from “usno.navy.mil”. BTW my lab is wrong it should be getting from an external time source but that’s a different post for a different day. Let’s get you fixed up.

Step 1.) Run W32tm /stripchart /computer:NTPServerAddress to make sure you are getting the correct time from your new source.

Step 2.) Set the correct time settings to a known good source. I would check to make sure you are following  http://support.microsoft.com/kb/816042 but we are really focusing on step 4 in that KB to set a new source.

Specify the time sources. To do this, follow these steps:

1. Locate and then click the following registry subkey:

   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters

2. In the pane on the right, right-click NtpServer, and then click Modify.
3. In Edit Value, type Peers in the Value data box, and then click OK.

    

   Note Peers is a placeholder for a space-delimited list of peers from which your computer obtains time stamps. Each DNS name that is listed must be unique. You must append ,0x1 to the end of each DNS name. If you do not append ,0x1 to the end of each DNS name, the changes that you make in step 5 will not take effect.

3.) Run w32tm /config /update

4.) Shake your fist at those Mayans, you are all done.


Ok so is everything replicating? Great you are done for now! If not read on.

I’m guessing you might see some Event ID 2042.

IMPORTANT: Ensure that the “Strict Replication Consistency” registry value(HKLM\System\CurrentControlSet\Services\NTDS\Parameters\Strict Replication Consistency ) has been set to the default for Server 2003 and greater value, which is 0x1 before you do.“Strict Replication Consistency” will ensure that no lingering objects will be replicated out if they really exist after we force the replication to happen.

Note: This forces the replication even if the AD replication has failed with the partner for more than the tombstone lifetime. In most cases, if the server is really having the replication failure and is not caused by this unexpected time jump issue, you would see new replication errors caused by lingering objects as long as “Strict Replication Consistency” is 0x1; then no lingering objects would be really replicated out.

We will now want to follow  http://technet.microsoft.com/en-us/library/cc757610(WS.10).aspx

On the downstream domain controller where it reported the replication error code 8614 (ERROR_DS_REPL_LIFETIME_EXCEEDED), setup the “Allow Replication With Divergent and Corrupt Partner” registry value.

1. Click Start, click Run, type regedit, and then click OK.

2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters

3. In the details pane, create or edit the registry entry as follows:

   If the registry entry exists in the details pane, modify the entry as follows:

   1. In the details pane, right-click Allow Replication With Divergent and Corrupt Partner, and then click Modify.
   2. In the Value data box, type 1, and then click OK.

   If the registry entry does not exist, create the entry as follows:

   1. Right-click Parameters, click New, and then click DWORD Value.
   2. Type the name Allow Replication With Divergent and Corrupt Partner, and then press ENTER.
   3. Double-click the entry. In the Value data box, type 1, and then click OK.

No restart is needed. Force replication in AD Sites and Services between the destination source and destination servers.

Remember to change the “Allow Replication With Divergent and Corrupt Partner” value back to 0x0 after the issue has been sorted out.

If you are STILL having replication issues they could have been happening BEFORE all this happened. Just call into Premier support and we’ll get you figured out.

How Did This Happen?

Well, you should really be syncing from a reliable INTERNAL time source and not one on the internet for various reasons. Also you most likely do not have the regkeys set for MaxPosPhaseCorrection and MaxNegPhaseCorrection, follow this KB http://support.microsoft.com/kb/884776.

This type of thing is also caught in our famous Active Directory Risk Assessment (ADRAP). This leads me to have a discussion about how Windows Time works while doing one. Everyone says there is no way this can happen, and yet here we are. Never had one? Contact your TAM and let them know you need one right away and this blog told you so. Not a Premier customer? Contact us here and we’ll get you in touch with the right folks. 

Thanks to Chulin Xu and Hakimuddin Wadlawala who had a very busy night

Mark “Party Like Its 1999” Morowczynski

Hey y’all, Mark back again with some more detail around what to when the system time rollback to November 19^th, caused Active Directory replication and other time-sensitive operations to fail in your environment. This post contains guidance by a small army of Microsoft PFEs, support professionals and developers. If you have any questions about the recommendations in this post, feel free to give CTS a call and they can guide you through the recovery. Recovering from a time rollback is a complex situation so read each step carefully and don’t skip ahead or you’ll make the problem worse. Also this post is going to be a long one and will probably break the record for additional links so you’ll want to get comfortable.

Here is what this post is going to cover.

How Did This Happen?

What Are The Symptoms?

Mitigation

1.) Correct Time

2.) Check For Replication Errors

3.) Additional Mitigation

Ongoing Tasks

How Did This Happen?

On November 19th, 2012, time servers at USNO.NAVY.MIL incorrectly provided time samples listing CY 2000 as the current year between the hours of 21:07 UTC and 21:59 UTC (16:07-16:59 EST). Get more info here.

Forests most impacted by this time rollback shared two traits:

1. The forest root PDC or master time servers in the forest lacked time jump protection discussed in in KB 884776 (probably because they were running the W2K3 OS)

2. The forest contained Windows Server 2003 DCs (more on this below)

Windows added support for time jump protection starting with the Server 2003 (and XP member workstations) in the form of two registry values: MaxPosPhaseCorrection and MaxNegPhaseCorrection (we’ll refer to both these keys going forward as max*phasecorrection). By default, the max*phasecorrection settings are not populated on Windows Server 2003 DCs. As a result, such DCs adjust the system time after receiving forward or back-dated time samples. Windows Sever 2008 and later DCs set the max*phasecorrection settings to 48 hours and ignore time samples that vary by more than 48 hours from locally configured time.

Time jump protection is not defined on Windows member workstations or servers until enabled by an administrator for the following reasons. Microsoft Commercial Support has observed massive time jumps (from days to multiple decades in the past and future) in customer forests for the last 10 years. Multiple root causes exist but up until now have never been caused by a highly accurate time servers giving out inaccurate time. While the max*phasecorrection settings offer a degree of protection when the time service is running, it offers no protection when inaccurate time is adopted during a reboot or while the time service is not running. Furthermore, the use of max*phasecorrection can prevent client and server computers from adjusting back to accurate time. While smaller max*phasecorrection values make Windows time clients less susceptible to adopting bad time, they also make it hard for such clients to self-correct if good time varies by more than max*phasecorrection seconds in the past or future. For example, setting max*phasecorrection to say 1 hour would prevent time client from self-correcting from a time zone or AM | PM misconfiguration. Given the ratio of domain controllers to member servers and workstations, Microsoft elected not to configure time jump protection on such computers. More information on time jump protection can be found in KB 884776.

Additional information about how time works in an AD forest can be found in this document. The cliff notes version follows: The root PDC gets time from a reliable time source which could be a highly accurate GPS clock, reference time servers on the internet or one or more references from both of those groups. Time flows hierarchically from root to child then grandchild domains. Some other things to take note of: when configuring a reliable time source on the root PDC or manual time servers. The best practice is to source from stratum 2 or stratum 3 level reference times’ sources. If you are configuring multiple time sources, all time sources should have the SAME stratum level AND the same stratum level as the previously configured external time source

What Are The Symptoms?

If system time moved back to year 2000 on November 19^th between the hours of 21:07 UTC and 21:59 UTC (16:07-16:59 EST), it’s a pretty safe bet you were affected by the time rollback as USNO.NAVY.MIL.

It’s also possible that your domain moved from current time back to November 19^th 2000 then back to current time.

Assuming your event logs have not wrapped, one clue that your DCs experienced a time rollback is to look for calendar year 2000 events bracketed by year 2012 events.

Event Source:        NTDS Replication
Event Category:     Replication
Event ID:  2042
Date:                    11/21/2012
User:                     NT AUTHORITY\ANONYMOUS LOGONComputer:             ContosoDC

Description:

It has been too long since this machine last replicated with the named source machine. The time between replications with this source has exceeded the tombstone lifetime. Replication has been stopped with this source.

The reason that replication is not allowed to continue is that the two machine's views of deleted objects may now be different. The source machine may still have copies of objects that have been deleted (and garbage collected) on this machine. If they were allowed to replicate, the source machine might return objects which have already been deleted.

Time of last successful replication:

2000-11-19 14:09:12

Other side effects of a time rollback.

Active Directory replication fails with Event 2042 reporting “It has been too long since this machine last replicated” and replication status 8614: “The Active Directory cannot replicate with this server because the time since the last replication with this server has exceeded the tombstone lifetime.”

Operations and Applications, requiring Kerberos authentication, including Active Directory replication fail and even accessing a file server fail with one of two errors:

Error 5: access is denied

Or

AD Replication Status -2146893022: target principal name is incorrect

Other date-dependent operations and applications may also fail including those based on lease intervals, caching or object lifetime (think DHCP, DNS, object lifecycles, date-driven password changes on computer accounts, trust relationships).

Mitigation

If you’ve made it this far you are probably effected by the time jump. I can’t stress, that following the recovery steps in ORDER is key. Taking shortcuts can actually make things worse so stay on the path. If you are unsure about any of the recovery steps, contact CTS and we can help you through this. Don’t be a hero. Here is a bird’s eye view of what we are about to take on.

1.) Correct Time

a. Don’t immediately reboot

b. Configure each forest root PDC with reliable time sources

c. Monitor time on DCs and critical application servers

d. Add time jump protection to servers with good time

e. Re-monitor time on DCs and critical application servers

f. Correct Servers with inaccurate time

2.) Check for Replication Errors

a. Fix DCs with replication Event 2042/ Replication Error 8614

i. Confirm that strict replication is enabled

ii. Check for lingering objects and remove if present

iii. Set “allow replication with divergent and corrupt replication partner to 1”

iv. Trigger replication or wait for scheduled replication to occur

v. Troubleshooting Error 2146893022: target principal name is incorrect or 5: access is denied

b. Confirm FRS Is Working

3.) Additional Mitigation

1a.) Don’t immediately reboot.

I know this goes against all your instinct when something is wrong but don’t just reboot. AD Replication, Kerberos and possibly secure channels on trusts and computer accounts could be impacted by the time jump. A reboot may trigger the reading of invalid time on OS shutdown or on subsequent OS startup, especially on virtualized guest computers. Don’t try to fix replication and authentication until the system time is corrected. We’ll get to the other issues later.

1b.) Configure each forest root PDC with reliable time sources

Windows computers use the NTP protocol to source time. NTP time assigns stratum levels to define how a close a given computer is to the reference time source. Stratum 2 level server’s source time from government and military stratum 1 computers which source time from stratum 0 atomic clocks and GPS satellites.

Domain-joined Windows clients and servers by default use NT5DS hierarchy for example a stratum 3 forest root PDC or manually configured Windows master time servers source time from an external stratum 2 time server. Time then time flows hierarchically down to domain controllers and clients in subordinate domains. Once again here are some guidelines when configuring external time servers.

1.) Verify that new and existing external time servers have a stratum level no lower than 2 and no higher than 3

2.) When adding a new external time server, make sure it has the same or a lower level as the previously configured external time source. For example if the old time server was stratum 3 and new is stratum 4 for clients will not accept this time change until the time service is restarted. To say this another way, every time service on member servers would need to be restarted or they will start to drift.

3.) A stratum level of 0 for example represents an uninitialized time server and is invalid. Do not use stratum 0 advertising time sources.

To identify stratum level for a reference time server, run the following command

w32tm /stripchart /packetinfo /computer:<DNS name or host name of time source>

In the output there should be a value for stratum. In this case you’ll want to pick a nearby stratum 2 server. A list of stratum 2 servers can be found here

To configure the root PDC to have a reliable time source. We do that with the following commands.

w32tm /config /update /manualpeerlist:DNS Address /reliable:YES /syncfromflags:MANUAL

This will do the following:

- Sets w32time to manually sync from the NTP server you provide

- Sets the “Reliable Time Source” flag for this machine in NETLOGON.

- Prevents w32time from discovering any machines in the domain as a time source.

We’ll then want to resync the time by running this command:

w32tm /resync /rediscover /nowait

- Updates the configuration and forces it to be immediately applied.

A little side note about the forest root PDC. The machine configured as the reliable time source for the forest probably should NOT be a virtual machine for two reasons:

1. The built-in time synchronization between the guest OS and host needs to be turned off so that the configured time source is actually used. Unless this is done, the machine will still get its time from the host regardless of the time source configured.

2. Virtual machines maintain “stable” time by constantly getting time updates from the host. Even with (1) being done, the virtual machine will likely be “less stable” (meaning that more time drift will be seen by clients syncing from it).

Let’s verify our previous commands worked as expected. To that we are going to run the following command.

w32tm /query /configuration /verbose

We will check 3 things in the output.

-The AnnounceFlags value should be >= 8

-The Type (under ‘NtpClient’ in [TimeProviders]) should be NTP

-The NtpServer (under ‘NtpClient’ in [TimeProviders]) should be the time provider you provided.

We can further confirm this by requesting that NLTEST locate a DC with the GTIMESERV flag, DCLOCATOR should find the DC in question. If it isn’t, the NETLOGON changes might not have propagated, so try targeting the DC directly with /SERVER:domainController

Nltest /dsgetdc:domain name /GTIMESERV [/FORCE]

As we can see here we have the TIMESERV flag so we are good to go. For more information around this topic, read these 2 links.

http://blogs.msdn.com/b/w32time/archive/2008/04/02/configuring-a-standalone-time-server.aspx
http://blogs.msdn.com/b/w32time/archive/2008/05/29/to-be-reliable-or-not-to-be-reliable.aspx

1c.) Monitor time on DCs and critical application servers

Now that we got the root PDC getting the correct time we need to go figure out what other members have bad time. There are a few ways to do this. To grab time of all your DCs fellow PFE Tom Moser wrote a script to help with this. You’ll want to start at the root domain focusing on the DCs, the virtualized hosts then application servers in priority order.

The script is located at the bottom of the post.

Syntax .\Get-TimeInfo.ps1 will write the csv output to the working directory as DCTimes.csv

1d.) Add time jump protection to servers with good time

We will now want to set the servers that already have good time with the MaxPosPhaseCorrection and MaxNegPhaseCorrection registry settings prevent Windows computers from adopting time when time servers send time samples with forward or back dated dates. Once again we’ll want to follow KB 884776

1e.) Re-monitor time on DCs and critical application servers

Using the same strategy in step 1C you’ll want to re-monitor the time in your environment to find out what DCs and critical application server’s time is incorrect. Also to note some of the servers time may have gone from bad to good now that the root PDC is giving out proper time.

1f.) Correct Servers with inaccurate time

On servers that still have the bad time you’ll want to do the following.

-Stop the time service (net stop W32time or Services Pane)

-Reset the time of the server by using the net time command to point it a good time server

-net time \\goodtimeserver /set

-This will then asking you to confirm that you want to set the time of the local computer to match the time of the \\goodtimeserver your provided. Hit Y

-Verify the system time is now good.

-Once again set the MaxPosPhaseCorrection and MaxNegPhaseCorrection registry settings

-Start the time service (net start W32time or Services Pane)

2) Check for Replication Errors

Maybe today is your lucky day and you don’t have any replication errors. We have two ways to quickly check. The first option is ADReplStatus, a recently released replication status reporting tool available for download from Microsoft.com. Keep this in your tool box for future use.

1.) Download ADReplStatus then install and run the tool. After the tool completes the replication status phase, click the Errors Only button in the toolbar. Click on column headers or drag column headers to the filter bar to provide the view that helps you focus on domain controllers, partitions and replication errors of interest.

Here is what the replication status might look like in your environment if you've encountered this issue:

If you prefer REPADMIN, here are the steps:

1. From a DC, run the following command to generate a forest-wide replication status report:

repadmin /showrepl * /csv >showrepl.csv

2. Open the file in Excel and filter on the last replication status result column (column K) to identify DCs with the replication failures (replication status 8614 is commonly associated with this issue).

Not so lucky huh. That’s ok read on we’ll get you going.

2a) Fix DCs with replication Event 2042/ Replication Error 8614

We are now going to tackle each DC individually. Follow all the steps provided until it is replicating properly then move on to the next DC in the list starting right back here at the top.

To prevent the spread of lingering objects, the operating system halts if a destination DC hasn’t inbound replicated over a given connection in tombstone lifetime # of days (default 60 or 180 days). There are 2 scenarios that can trigger this behavior: (1.) a destination Dc really did fail inbound for TSL # of days or (2.) replication engine got has the “appearance” of having failed for TSL # of days due to a time jump.

2ai) Confirm that strict replication is enabled

Remember how I warned earlier in this post about not skipping steps. This is what I’m talking about. You can make things much worse later if you don’t do this step. Strict mode replication prevents lingering objects from being replicated or reanimated on destination DCs that have used garbage collection to create, delete, and permanently purge intentionally deleted objects. You will want to enable this on the all DCs in the forest if it’s not enabled already.

Note: This command has to be run with elevated command prompt with Enterprise Admin credentials

repadmin /regkey * +strict > strict.txt

The command enables strict replication on all DCs in the forest by modifying the following registry path, setting and value: 

Registry path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters
Registry setting (DWORD Value -not case sensitive): Strict Replication Consistency
Value: 1

2aii) Check for lingering objects and remove if present

There has been a lot written on lingering objects so we won’t get into too much here. Use the free tool Repldiag created by fellow PFE Ken Brumfield and check out this post by PFE Glenn LeCheminant http://blogs.technet.com/b/glennl/archive/2007/07/26/clean-that-active-directory-forest-of-lingering-objects.aspx to get that all cleaned up.

Additional ReplDiag Resources

Cleaning lingering objects across the forest with ReplDiag.exe [Part 2 of 4]

Why does ReplDiag.exe error out with the message that the topology isn’t stable? [Part 3 of 4]

Can I clean one partition at a time with ReplDiag, and other tips [Part 4 of 4]

Note: DCs that ran in strict replication consistency prior to the time jump likely have few to no lingering objects to remove. Those that ran in loose replication consistency prior to the jump likely contained lingering objects prior to the November 19^th time jump. The enabling of strict replication is generally a requirement to stop the spread of lingering objects. Failure to enable strict replication during lingering object cleanup typically means such DCs will inbound replicate the just removed objects from another DC. At the same time, the enabling of strict replication may block needed Active Directory changes located in the replication queue (AD Replication Status 8606/8333 and Event ID 1988). Evaluate whether loose replication needs to be configured so that replication can occur to run the business with the notion of scheduling a more exhaustive cleanup when time permits.

2aiii) Set “allow replication with divergent and corrupt replication partner to 1”

The next step is we will want to disable the time-based replication quarantine via repadmin or by using regedt32:

You must run the following command from a repadmin.exe version included in the RSAT tools (Windows Server 2008 or later) or from a server that had the AD DS role installed. You’ll also want to run this from an admin-elevated command prompt.

Set the value on a single DC (destination DC in replication report) at first and then expand scope of command as needed.

DO NOT SET THIS KEY UNTIL YOU CONFIRM that strict replication was enabled on destination DCs logging replication error status 8614/Directory Services Event 2042. The only time you should relax the “it has been too long” replication quarantine is if destination DCs are configured with strict replication or you have first tested for and removed lingering objects if present. If you don’t want to enable strict replication consistency, check for and remove lingering objects before relaxing the “allow replication with divergent or corrupt replication partner” setting

Repadmin /regkey DestinationDCName +allowDivergent

If you don’t have that available here is how you can do it via registry.

Registry path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters
Registry setting (DWORD Value -not case sensitive):

Allow Replication With Divergent and Corrupt Partner
Value data:

1

A value of 1 is used to allow replication to occur even though replication hasn't completed in tombstone lifetime number of days over a given replication connection. It is important to put this protection back in place after the environment has been recovered by setting the value back to 0 when we are done (0 = disallow, 1 = allow). We have a reminder to do this down the road.

For more info

Troubleshooting AD Replication error 8614: "The Active Directory cannot replicate with this server because the time since the last replication with this server has exceeded the tombstone lifetime"

http://support.microsoft.com/kb/2020053

AD Replication Error 8614 (event ID 2042)

http://technet.microsoft.com/en-us/library/cc757610(v=ws.10).aspx

2aiv) Trigger replication or wait for scheduled replication to occur

You can now force replication to occur or wait for it to follow its normal schedule. If everything is working as expect without any errors you are done. If that’s the case make sure you remove the setting to allow replication with corrupt and divergent partners. Once again this should be run from RSAT tools (Windows Server 2008 or later)

Repadmin /regkey DestinationDCName -allowDivergent

2av) Troubleshooting Error 2146893022: target principal name is incorrect or 5: access is denied

Common errors seen around this are Error 2146893022: target principal name is incorrect or 5: access is denied.

The easiest way to resolve this is to disable the Kerberos Key Distribution Service (KDC) and simply reboot the DC. Don’t worry its ok now, the time is fixed remember. Recheck replication. If it’s working as expected now make sure you remove the setting to allow replication with corrupt and divergent partners. Once again this should be run from RSAT tools (Windows Server 2008 or later)

Repadmin /regkey DestinationDCName -allowDivergent

If it’s still not working follow these detailed steps below.

Error 2146893022: target principal name is incorrect

This can have multiple root causes but we commonly encounter this replication status in this scenario because the DC has invalid Kerberos tickets.

Each DC impacted by this issue (source DC in AD Replication report) will need new tickets issued by a KDC other than itself.

1. Stop the Kerberos Key Distribution Center service. Make sure you do not stop this service on ALL DCs in a given domain and you shouldn’t be if you are following the directions and troubleshooting 1 DC at a time.

*The remaining KDCs must be reachable across the network.

2. Purge local system tickets

3. Run the following command from an elevated command prompt:

Klist -li 0x3e7 purge

4. Test replication

5. If replication fails with the same error then a reboot may be necessary as we may have failed to flush tickets in the right context.

An interesting thing that may also be happening if this doesn’t work is that we are getting a ticket from a DC that is still broken. If that is the case stop the KDC service on all bad DCs leaving at least one per domain online.

For more info feel free to check out

Troubleshooting AD Replication error -2146893022: The target principal name is incorrect.

http://support.microsoft.com/kb/2090913

Once again if this resolves your problem make sure to re-enable this key. This should be run from RSAT tools (Windows Server 2008 or later)

Repadmin /regkey DestinationDCName -allowDivergent

If you encounter replication status 5 "Access is Denied" for domain controllers in between domains

Temporarily add the Replicator Allow SPN Fallback registry value. To do this, follow these steps.
Note Perform steps 1 through 6 on this same domain controller.

1. Click Start, click Run, type regedit, and then click OK.
2. Locate and then click the following registry subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters

1. On the Edit menu, point to New, and then click DWORD Value.
2. Type Replicator Allow SPN Fallback, and then press ENTER.
3. Double-click Replicator Allow SPN Fallback in the right-pane, type 1 in the Value data box, and then click OK.

6. Restart the domain controller.

After this has been solved don’t forget to delete or change this value back to 0 and restart the domain controller in order to reverse this setting after recovery operations are complete.

Find out more info at

Troubleshooting AD Replication error 5: Access is denied

http://support.microsoft.com/kb/2002013

Orphaned child domain controller information may not be replicated to other Windows 2000 Server-based domain controllers

http://support.microsoft.com/kb/887430

Once again if this resolves your problem make sure to re-enable this key. This should be run from RSAT tools (Windows Server 2008 or later)

Repadmin /regkey DestinationDCName -allowDivergent

2b) Confirm FRS Is Working

The File Replication Service will be negatively impacted by time jumps as well. It is quite possible that changes to FRS replicated content is not happening after returning to the correct time settings. This can be especially crucial for SYSVOL content. There is a backgrounder on this from the Microsoft Knowledge Base:

289668 Advancing time on production computers and the effect on Active Directory and FRS

http://support.microsoft.com/kb/289668/EN-US

The impact on FRS depends on the duration the environment was using the incorrect time and what changes have been happening during that time. If you are encountering problems with FRS, we recommend you contact Microsoft Support Services to investigate the problems and determine a resolution.

In the worst case you need to restart FRS for SYSVOL in the domain. If you are in a large scale environment you will want to contact CTS for support with this. The steps are in this KB article:

289668 Advancing time on production computers and the effect on Active Directory and FRS

http://support.microsoft.com/kb/289668/EN-US

Note You do not need to follow the steps to rebuild the file system objects such as directories and junctions.

3) Additional Mitigation

Now that we got everything under control in Active Directory we’ll want to go ahead and want to set this time jump protection on the rest of the servers in the environment. You may configure max*phasecorrection directly in the registry by following KB 884776

Ongoing Tasks

Continue to run REPAMDIN or ADREPLSTATUS to detect these and other AD Replication failures. Resolve replication errors prioritized by failure duration and criticality within the replication topology.

I hope this extremely long blog post has been helpful recover from this issue. Don’t forget if you have any questions contact CTS to help get this resolved.

-Mark Morowczynski, Justin Turner, A. Conner

First, let me thank everyone who participated in our PFE Live Chat on Monday 26 November.  It was a great opportunity to talk about technology and it’s impact in real-world cases, for real-world customers.  For those of you who missed the opportunity, let me re-cap some of the topics:

Windows Server 2012 Storage Spaces

It turns out that a number of customers are looking at use cases for Windows Server 2012 Storage Spaces.  If you have no idea what Storage Spaces are, take a look at this blog from Martin Lucas.  It does a great job of describing the feature.  Watch for future blogs on interesting use cases.

SMB 3.0

We discussed SMB 3.0 and other file sharing goodness in Windows Server 2012.  There’s a lot of questions on what SMB 3.0 can and cannot do. There were also some questions on leveraging SMB 3.0 with existing file sharing technologies like DFS. 

Windows Server 2012 NIC Teaming

We spent some time covering the NIC Teaming capabilities in Windows Server 2012.  There was some discussion on how/when customers could leverage these features.  Another good idea for a blog topic.

Hyper-V

One of the reasons we love Windows 8 is it’s capability to do Hyper-V.  This lead to an interesting conversation on the capabilities of Windows 8 Hyper-V versus Windows Server 2012 Hyper-V including the awesome new features of Windows Server 2012 Hyper-V, including Live Migrations and Hyper-V replicas.

Central Certificate Store (CCS)

CCS falls into the category of one of those features in Windows Server 2012 that many of us never knew existed.  CCS allows certificates to be stored on a file share, so servers/services can dynamically update their certificates, on demand.

How about Some More Info?

Hopefully, we provided some answers, however, we definitely raised some interesting questions.  Rather than provide you with a long list of hyper links for each of these topics (you do know how to search, don’t you?),  we thought we could be of more value by providing you future blogs on some of these topics, so keep following us for more information.  Remember, you can allows contact us with more questions by:

1. Clicking Contact Us
2. Commenting on Our Blogs
3. Attending a Future Live Chat

Hey y’all, Mark here with another quick real world WDRAP (Windows Desktop Risk Assessment Program) post. This one also involves a time issue which seems to be a theme around here as of late. Also I've already had some customers contact their TAM to get a PFE on site based on these posts which is great that means people are actually reading them. Who knew! Make sure you get yours soon don’t delay! (Queue the car salesman music) Kidding but seriously you really probably should.

We took our initial trace and we were greeted by this monster.

Hi!

What in the world is going on here? As we know from other posts that this is not normal to have a gap like that. Explorer should start right after Winlogon finishes. We have a nice 23 to 25 second about gap here. Did the explorer process get into a DeLorean, do 88 MPH, travel 24 seconds into the future to meet us at this point? Where did this time just disappear to? Well the customer was not buying my DeLorean story. Besides you can’t just go buy plutonium anywhere! So we had to dig in to this.

First we use our clone selection to highlight that section of time. We then use the “Zoom To Selection” to update all our graphs to that time range only. The Services pane was quiet so the next logical spot to check to see if any processes are running for this amount of time. We go to the Processes Pane, right click, select view, to select our entire area and then right click select process summary table. We can tell that we are zoomed in the below picture as the time axis is much more granular than normal.

Now we are back to our familiar view we used in the past but we are only looking at processes that are running during this time frame. They could have started earlier in the boot process or started sometime during this time span. Now don’t forget we can drag all those columns around up top to get the data to what is interesting to you. We will want to drag the Duration column all the way to the left and sort by that. For this example we are looking to see if anything fits our duration of 23 to 25 seconds sure enough we do.

Well we went from not having any clue to what was going on to a list of 4 suspects in a few clicks. That’s pretty good. The suspects are a command prompt, Conhost, slstart and something called wKiX32.exe. This is heavy Doc. Well conhost is normal so it’s probably not that but it could be. I like to check the more obvious stuff first. We can also see that wKiX32.exe parent process is 3,628. This is also the Process ID for one of our suspects, SLstart.exe. Getting warmer I bet. That turns out to be Script Logic. We disabled that service for testing purposes and low and behold our gap was gone! Now the customer had some concrete evidence what was causing the delay and can start to remediate with that vendor in their environment. Until next time….

Mark “Dances At The Enchantment Under The Sea” Morowczynski

Time really does fly when you are having fun. Today marks the day when we went up live last year. We set off with a modest goal to share our field and real world experience with you, to perhaps one day talk about an issue that’s relevant to your job, to spark some light of interest for an existing Windows technology or talk about something that’s totally new to all of us. We hope we did some of that. In process, to date we have posted 66 blog articles that have received around 300 comments, we have been visited by 173 countries, with total of over 220,000 unique visitors. But the numbers don’t quantify our progress towards achieving our goal, we hope to continue to share some of our experience as well as rely on your comments and feedback to make this blog a valuable resource for you. So again, a BIG thanks to YOU for reading this blog and a HUGE gratitude for our team members for taking the time to write. We have great things on the way for you, so stick around.

Happy Birthday to Ask Premier Field Engineering Blog !!

Until next time, Rick “keeping it short” Sheikh.

By Roger Osborne

Like many of you, when I test out new Windows features, I often turn to TechNet documentation, to be certain I’m following Microsoft guidelines and recommendations along the way. A lot of work and effort goes into making this information available, and, more often than not, the details provide all the necessary pieces to get the feature working. That wasn’t the case, however, when I decided to configure the Hyper-V Replica connection broker following this post: http://technet.microsoft.com/en-us/library/jj134153.aspx#BKMK_1_4.

After carefully following the step-by-step instructions, the wizard appeared to successfully create the connection broker and close, only then to have the role status show it was in a failed state.

After scratching my head for a minute and mumbling a few choice words at the computer screen, I decided to put on my troubleshooting hat.

Since the Replica Broker role was being created in the Failover Cluster Manager, I felt pretty confident the cluster, itself, needed specific permissions to successfully configure the role in AD; however, what I didn’t know was which permissions were needed, or what took place when the wizard was able to successfully create the role, for that matter!

The other clue was found on the summary page of the Replica Broker Wizard (which I decided to finally read after my 3^rd or 4^th unsuccessful go at it!!). As you can see from the screenshot below, it displays the full OU path where my Hyper-V hosts’ computer objects were located.

In order to feel confident I would be successful on my next attempt, I opened Active Directory Users and Computers (ADUC) on my Domain Controller and drilled down to my Hyper-V OU. Next, I right-clicked and selected Delegate Control. I then added my cluster computer object and gave it full control of the OU, just to be certain it could do anything it needed. (This is when having a test lab comes in handy, as you definitely wouldn’t want to do this in production!)

With the delegation now in place I went through the wizard again. Lo and behold, the Replica Broker was created successfully and came online without issue! Next, I went back to my Hyper-V OU and discovered the Replica Broker Wizard creates a new computer object, which is used to by the role to keep the broker service running and able to move from node-to-node, just like a VM.

So, we know delegating full control of the Hyper-V OU to the cluster computer object works, but that’s obviously not the least privilege approach. The base requirement, however, is simple: a computer object needs to be created in the Hyper-V OU to facilitate the Replica Broker role.

Now I can hear you saying, “Roger, what’s the best method to get this role working?” I’m so glad you asked!! J There are two ways! Although I would personally lean more towards option one, there may be some who prefer the second option, so I’ve included both. Only you can decide which works best in your environment!

OPTION #1:

The first option is to pre-stage a computer object in the Hyper-V OU that is the name of the Replica Broker role you wish to create (e.g. ReplicaBroker1). Once the object is created, you will then need to go to the Security tab, add the cluster computer object; finally, give it full control.

Here you can see the pre-staged computer object I created in my Hyper-V OU:

After creating the object, called REPLICABROKER1, I right-clicked on it then went to the Security tab. Next, I added the Cluster computer object and gave it full control. (In this screenshot, you’ll notice my cluster is named HV1-HV2-2012CLU.) Click Apply then OK.

Once that is complete, you can successfully create the Replica broker in the Failover Cluster Manager!

OPTION #2:

The second option is to delegate control of your Hyper-V host OU, giving the cluster computer object the necessary rights to create computer objects within the OU.

After opening ADUC (Active Directory Users and Computers), locate and right-click the Hyper-V OU and select Delegate Control. Click Next on the Welcome screen, then click Add under Selected users and groups.

When presented with the following screen, select Object Types.

Click to add Computers and press OK.

Next, add the cluster computer object.

Now we need to select the Create a custom task to delegate, then press Next.

Select Only the following objects in the folder, check Computer objects, then check Create selected objects in this folder. Press Next.

Finally, under Permissions, check Write and click Next.

On the summary screen click Finish.

Once that is complete, you can successfully create the Replica broker in the Failover Cluster Manager!

In closing, I hope you enjoyed this post and found it informative!

Roger Osborne

Remember when Windows Server 2008 R2 was released, and one of the exciting new features was Managed Service Accounts?  Managed Service Accounts (MSAs) held so much promise – automatic password management and automatic SPN registration.  Remember all of those service you have in the domain, that are over-privileged, and whose passwords haven’t changed in the past 5 years?  You dreamed of replacing them with MSAs, and then you read the fine-print.  MSA’s are not supported for applications like Exchange or SQL.  MSA’s cannot be shared across multiple hosts.  MSA’s cannot even be used to run a scheduled task.  After we totally harshed your mellow, you didn’t even bother to check with your 3rd party vendors to see if their applications could use MSAs.

Enter Windows Server 2012 Group Managed Service Accounts

Windows Server 2012 has come to the rescue with the Group Managed Service Account (gMSA).  Think of Group Managed Service Accounts as a usable version of the Managed Service Account.  With gMSAs, Windows Server 2012 has addressed most of the limitations of MSAs.  Specifically:

• A single gMSA can be used on multiple hosts.
• A gMSA can be used for scheduled tasks.
• A gMSA can be used for IIS Application Pools, SQL 2012 and potentially other applications - check with the vendor :)

Now is the time to learn about Group Managed Service Account, and test their potential use in your environment.

What is a gMSA?

When you extend your schema for Windows Server 2012, a new object class is added for gMSAs – msDSGroupManagedServiceAccount. 

It’s derived from the computer class, with five new/additional attributes:

• msDS-GroupMSAMembership – Governs which computers (groups of computers) are allowed to retrieve the password and make use of the gMSA.
• msDS-ManagedPassword – a binary blob containing (among other things) the current password, previous password and password change interval.
• msDS-ManagedPasswordInterval – configured at account creation (can’t be changed later), determines how often (number of days) the password must be changed.
• msDS-ManagedPasswordID – key identifier used by Key Distribution Service (KDS) to generate current password.
• msDS-ManagedPasswordPreviousID – key identifier from previous password.

Password Behavior

Unlike the previous MSAs, the password for gMSAs are generated and maintained by the Key Distribution Service (KDS) on Windows Server 2012 DCs.  This allows multiple hosts to use the gMSA.  Member servers that wish to use the gMSA, simply query the DC for the current password.  Usage of the gMSA is restricted to only those computers specified in the security descriptor, msDS-GroupMSAMembership.

As the password for the gMSA is needed, for example when a host using the gMSA retrieves it, the DC will determine if a password change is necessary.  If so, it uses a pre-determined algorithm to compute the password (120 characters).  This algorithm depends upon a root key ID that is shared across all Windows Server 2012 KDS instances (pre-generated by an administrator), the current time (translated to an epoch) and the SID of the gMSA.  Thus, any Windows Server 2012 KDS can generate the password, and all KDS instances use the same algorithm and will generate the same password.

Since the password (or, more precisely, the password hash) for the gMSA will be stored in Active Directory, down-level DCs will still be able to handle authentication requests – for example, to respond to a Kerberos TGS-REQ for a service ticket.

Group Managed Service Accounts Requirements

• At least one Windows Server 2012 Domain Controller
• A Windows Server 2012 or Windows 8 machine with the ActiveDirectory PowerShell module, to create/manage the gMSA.
• A Windows Server 2012 or Windows 8 domain member to run/use the gMSA.

Using Group Managed Service Accounts

Like most new features in Windows Server 2012, creating/configuring gMSAs are easy.  In essence, there are three steps:

1. Create the KDS Root Key (only has to be done once per domain).

2. Create and Configure the gMSA

3. Configure the gMSA on the host(s)

Let me demonstrate with an example.

Using a gMSA for a Scheduled Task

I’ve got customers that run scheduled tasks on domain controllers.  For example, they like to run a batch file nightly to perform a custom job.  Unfortunately, some of these custom jobs require that the account that runs the process be an Administrator, or Domain Admin.  Since they schedule tasks across multiple domain controllers, using the same account, they rarely (if ever) change the password.  As soon as they upgrade their DCs to Server 2012, I’ll be nagging them to transition to a gMSA.  Here’s how:

1. Create the KDS Root Key (only once per domain).  This is used by the KDS service on DCs (along with other information) to generate passwords.

From a Windows Server 2012 Domain Controller (or Windows Server 2012/Windows 8 host with the ActiveDirectory PowerShell module) run:

Add-KDSRootKey –EffectiveImmediately

Then go home and continue tomorrow .  Seriously, in-spite of what you might think –EffectiveImmediately means wait up to 10 hours.  This is a safety measure to make sure all DCs have replicated and are able to respond to gMSA requests.  There is a trick to bypass this safety measure, but should only be used in a lab.

2. Create and configure the gMSA

First, identify or create a security group and add the computer objects of the hosts that will be allowed to use the gMSA.  While you could grant individual computer objects the ability to use the gMSA, creating a security group to hold these computer objects will give you more administrative flexibility going forward.  The only downside to using a group, is that computers/hosts will need to be re-booted after being added/removed from the group to reflect membership changes.  In this example, the hosts are domain controllers and there is already a group for Domain Controllers (which contains all the DCs in the domain) that I can leverage:

Next, you must use PowerShell (with the Server 2012 AD cmdlets) to create the gMSA.  During the creation you must specify a name (SamAccountName) and dnsname.  You’ll also want to specify the group allowed to use the gMSA (see above) and potentially SPNs for the account:

New-ADServiceAccount -name <ServiceAccountName> -DNSHostName <fqdn> -PrincipalsAllowedToRetrieveManagedPassword <group> -ServicePrincipalNames <SPN1,SPN2,…>

In my case, I’ll only specify the Name, DNSHostName, and PrincipalsAllowedToRetreiveManagedPassword:

If you get a “key does not exist” error you forgot to do Step 1, or you were too impatient.

If everything works as expected, you’ll notice a new gMSA object in your domain’s Managed Service Accounts OU:

3. Configure the gMSA on the host

First, you’ll want to install and test the gMSA on the host.  While this isn’t always necessary, it’s safe practice.  You’ll run the following PowerShell cmdlets on the host which will be using the gMSA:

Install-AdServiceAccount <gMSA>

Test-AdServiceAccount <gMSA>

You’ll notice the Test-ADServiceAccount, returns “True”.  If it returns “False”, it should include a verbose error message.

Next, if you are going to use the gMSA for a Service, an IIS Application Pool, or SQL 2012, you would simply plug it in the Logon/Credentials UI.  The trick is to append a $ after the account name, and leave the password blank:

If necessary, Windows will grant the account the “Log On As a Service” right, and once the service is started, the password will be automatically retrieved from a Windows Server 2012 DC.

4. Using the gMSA for a Scheduled Task

Since my use case was a scheduled task, I’ll show you some of the interesting nuances around gMSA and scheduled tasks.  The fundamental problem is that you can’t use the Task Scheduler UI.  So we’ll use PowerShell cmdlets, instead. (You could also use schtasks.exe with an XML config file, but I’ll let you figure that one out yourself).

To use the PowerShell cmdlets, you need to define an Action (what), a Trigger(when) and a Principal(under which identity):

$action = New-ScheduledTaskAction  "c:\scripts\backup.cmd"
$trigger = New-ScheduledTaskTrigger -At 23:00 -Daily
$principal = New-ScheduledTaskPrincipal -UserID child\myAdminAccount$ -LogonType Password

After the –LogonType switch you type the word Password, and not an actual password.  This tells the scheduled task to retrieve the actual password for the gMSA from a domain controller.

Now you plug these three variables into a Register-ScheduledTask cmdlet

Register-ScheduledTask myAdminTask –Action $action –Trigger $trigger –Principal $principal

Be aware:  If you are using the gMSA to run scheduled batch jobs/scripts, you will have to grant the gMSA the ability to “Log on as a batch job” on the machine:

You may also need to grant the gMSA membership in a local group (like Administrators, or Backup Operators) so it has the necessary rights to accomplish the task.

Open the Task Scheduler and you should see your scheduled task listed.  You should be able to manually execute the task (to test it).  Otherwise, you cannot use the gui to edit the task.  Changes have to be made using PowerShell cmdlets.

I hope you enjoyed the blog, and now have another reason to start deploying Windows Server 2012.  Until next time

Doug Symalla

Thanks for great feedback and for reading my prior AskPFEPlat post regarding Storage Spaces in Windows Server 2012.  Based on feedback and questions following that post, it seemed like a great time to show how to import a storage pool on another server.   Storage Spaces in Windows Server 2012 writes the configuration about the storage pool onto the disks themselves.  Therefore, if disaster strikes and the server hardware requires replacement or a complete re-install – there is a relatively simple procedure involved to mount and access a previously created storage pool...perhaps on another server.  Notice I said server.  The implementation of Storage Spaces on Windows 8 doesn’t offer the same feature set that Windows Server 2012 does, so therefore you can only import a storage pool on the same OS version for which it was created.

Step By Step Example

1.     Make the disks available to the server

The first step, which may have multiple sub-steps, is to physically attach and successfully make available all drives from the pre-existing storage pool.  To be successful in this endeavor, you would need appropriate drivers, cables, firmware, etc. to make this work.   Obviously if these are SAS drives and the destination server doesn’t have a SAS interface, this will be somewhat of a difficult task without the proper interface.  Once you’ve completed this task, from Server Manager you should see something similar to the following (please click on any of the included screen captures for a clearer, more detailed image):

2.     Make the storage pool read-write.

As you can see from the prior screenshot, the storage pool (as well as the virtual disk) has a warning exclamation mark symbol to indicate a problem.  By default, Windows Server 2012 will detect the foreign volume and mark it read-only.  The indicator appears because of this condition.  To proceed, right-click the storage pool and choose Set Read-Write Access…

Make it read-write.

Choose the server where the storage pool may be mounted read-write.  If this is a clustered storage pool, multiple nodes may appear in the Choose a Server for Read-Write Access dialog.

3.     Attach the virtual disk.

The activities in prior steps make the storage pool available.  However, within the storage pool will be any virtual disks previously created using pool resources.  Just like a VHD, these must be attached in order to be accessible.  Therefore, the yellow exclamation warning sign indicates that the virtual disk is not currently attached.  To proceed, right-click on each virtual disk and choose Attach Virtual Disk.

If all is successful, each Virtual Disk chosen will be attached.

4.     Online the resulting logical disk(s).

As you may recall, Windows Server 2008 Enterprise and Datacenter editions, by default do not automatically mount every disk device seen (providing that the default SAN policy has not been changed.)   Windows Server 2012 is no different.  To use any of the virtual disks that belong to the imported storage pool, you must online them first.

In Server Manager under Volumes\Disks, select the disk, right-click, and choose online.

Notice at this point that the volume now appears and remains accessible.

Concluding Thoughts

Hopefully the above notes and screenshots illustrate how easy it is to import a storage pool when you have a need to do so, or as a proof of concept when learning about Storage Spaces.   The example above was created using Hyper-V Virtual Machines, and using iSCSI targets (for example only, not production) made available to a second newly installed VM after the first VM was conveniently destroyed.  You could also use individual VHD files as components for the storage pool members.

When I first heard of ReFS my immediate first question was whether ReFS as a file system was a direct replacement for NTFS. It didn’t take long before my customers began to ask me the same question. In learning about this new file system in Windows Server 2012, it became apparent fairly quickly that ReFS, while a new file system, is built differently. ReFS stands for Resilient File System. NTFS has its place, and so does ReFS. While ReFS may appear to have some similarity to NTFS, it does not contain all the underlying NTFS features and scales efficiently to handle data sets far larger than NTFS.

The Resilient File System

Although ReFS inherited some of the NTFS code base initially, it is a different file system with different uses in mind. In fact, disk tools that work with the NTFS Master File Table (MFT) won’t be able to work with ReFS because ReFS has its own mechanism for keeping up with file metadata. ReFS is ideal for storing large amounts of data and can be leveraged for file shares. Applications that run locally on the server and rely on specific NTFS features may not work with ReFS. However, many may have no issues due to ReFS compatibility with many of the Win32 storage APIs. For example, Windows Deployment Services (WDS) explicitly requires NTFS because it relies on specific features in order to implement the RemoteInstall folder structure used for storing images. These are features that a conventional file server or data repository does not require.

CHKDSK isn’t applicable to ReFS. Yes…I did just indicate that there’s no need to run CHKDSK on a ReFS partition. Are you feeling that the tool you’ve wanted to avoid for so long is now something you might want to hold onto…just in case? A love-hate relationship perhaps? The counselor is in. It’s okay to have those feelings if you have them. The truth is that in terms of ReFS partitions, ReFS doesn’t need CHKDSK because repair functionality is built-into the file system. Repair, if needed, occurs on-the-fly. Yep…there’s no need for extra tools to go fix corruption like with other file systems. And for what it’s worth, Windows Server 2012 contains improvements for CHKDSK.

ReFS can use checksums to detect if data has changed since last written and is able to detect and recover from corruption quickly. In fact, when data is written to disk, it is written to a new location on disk rather than over the top of existing data. Once successfully written, the file system can free the space used by the old data stream. ReFS is able to recover from corruption within the file system rapidly without limiting availability of the volume. Further, ReFS may be used with clusters, Hyper-V, file shares, data archival, and many other uses.

Additional integrity data streams may be enabled if you have additional needs for data protection. By default, ReFS uses conventional streams which behave identically to NTFS data streams. However, don’t think that because this is so that file system metadata is conventional also. File system metadata is protected against corruption. If you want additional protection of data streams, you may enable Integrity Streams. When configured to do so, checksums are used against written data and updates are done using copy-on-write. You may enable Integrity Streams on particular folders, volumes, or even granularly on a per-file basis. When coupled with redundancy through Storage Spaces, it is default behavior for Integrity Streams to be enabled for the entire volume. Storage Spaces and ReFS complement each other. When coupled with a mirrored Storage Space, duplicate copies of data will automatically be leveraged by ReFS. With this configuration, if corruption were to be encountered, ReFS can immediately leverage redundant data within Storage Spaces to expediently address the issue. One other example of how Storage Spaces compliments ReFS would be how ReFS periodically scrubs file system data to look for differences in an event to combat bit flips that can occur over time due to data stored over a long period of time in the same location.

As a file system, ReFS is not only good for resiliency, it is great for maintaining extremely large amounts of data. With data integrity and recovery features built-into the file system, there is no need to wait for CHKDSK to run to fix corruption. What if you needed to store 500 billion gigabytes in one place?. But why stop there? I recently read that ReFS can handle up to 1 Yottabyte (YB). That wasn’t in my geek vocabulary until now and I’m not sure I’m going to remember that name next week. To give you an idea of how large a number that is, 1GB can be represented as 10⁹. 1YB is represented by 10²⁴. That is like 1 quadrillion GB. That seriously messes with my head. Good luck purchasing that much available hard drive space for your home lab in the basement. Imagine trying to power all that storage without shorting out the power for the whole neighborhood. Can you imagine how long even a modern efficient version of CHKDSK might run on a volume that size? ReFS answers the needs of data integrity and efficiency and is mainly intended for very large volumes on file servers. It is a file system that goes beyond capability of NTFS.

NTFS Improvements

NTFS remains the file system of choice for the operating system boot volume as well as any other general needs for data storage. CHKDSK remains the tool of choice for dealing with NTFS file system issues should they happen. CHKDSK in Windows Server 2012 contains improvements…so CHKDSK continues to evolve and get better. The average NTFS deployment currently is around 500GB. In many cases administrators were hesitant to go beyond that due to the time potentially required to run CHKDSK against a volume that size. The time required to run CHKDSK has not been predictable due to file system structure complexity differences from one volume to another as well as a variety of other factors. The average number mentioned above has increased over time because the efficiency of CHDKSK has evolved with each new Windows Server release for quite some time now. As the following TechNet reference indicates, you can safely deploy multi-terabyte volumes based on the improvements in CHKDSK and the existing capabilities of the NTFS file system. Windows Server 2012 builds upon the self-healing capabilities of Windows Server 2008 R2 NTFS. NTFS fixes the corruption it can on-the-fly and what can’t be addressed immediately can already be calculated in how it needs to be fixed to that when you choose to mitigate the issue, the time required is truly minimal compared to the CHKDSK of years past.

http://technet.microsoft.com/en-us/library/hh831536.aspx

Comparing ReFS and NTFS Features

Although ReFS is a different file system, there are similar features between ReFS and NTFS. The easiest way to compare those is to look at the feature list side by side. Consider the following feature sets:

How did I pull this information? There are charts available on the net…and people that know me can testify that I like charts. Yet, it is easy to obtain the available features of each file system yourself from the live file systems that you already use. If you don’t have a volume available that uses ReFS, it is easy to create a VHD file, attach it, and format as ReFS to use for your own testing. Once you have NTFS and ReFS file systems at your fingertips, use FSUTIL to gather the file system supported features. Then it’s a matter of comparing each list of features. FSUTIL syntax is very simple:

FSUTIL fsinfo volumeinfodriveletter

This simple command lists the capabilities provided by the file system in use on the supplied drive letter. Simply run this command against each type of file system and compare.

Here are a few screen shots of using Disk Management to create a VHD, initialize as GPT, format as ReFS, and using FSUTIL against the newly created ReFS volume.

For this example I used dynamic expansion. If I were really going to use this VHD for data I would not thin provision like in this example. I only want to inventory
ReFS features for this example.

Right-click on the new disk. Choose Initialize. Choose GPT. Right-click on the disk and format as ReFS.

Notice that key capabilities missing from ReFS as compared to NTFS are EFS encryption, quotas, and compression. BitLocker may be used for encrypting these volumes while EFS is not an option. Thus, when using ReFS, encryption doesn’t need to be part of the file system. BitLocker satisfies the need for encrypting data on the volume as it encrypts the contents of the entire disk. Quotas may be managed outside the file system rather than through the file system like NTFS. Not supporting Hard Links is a key reason why you wouldn’t use ReFS for a system disk; files in the system32 folder are really hard linked back into the WinSxS folder structure. You might think that ReFS would have data deduplication built-into the file system. The fact that it doesn’t may not prohibit other components or third-party solutions from interfacing with ReFS through the API set provided.

Command Line Directives for ReFS

For those that are script fanatics (you know who you are), I’ve written this section just for you. We all know that the need for scripting language happens all the time. Perhaps you’re just creating re-creatable virtual environments for one reason or another…and now you need to include the creation of some ReFS volumes. Below are some valuable commands including conventional and PowerShell examples.

Formatting a ReFS Volume

While this can be done from the UI quite easily by choosing ReFS as the file system in the drop down from the Format dialog, this is also easily done from the command line. Full format example below:

Command-Line

Format /fs:ReFS J:

PowerShell Equivalent

Format-Volume –DriveLetter J –FileSystem ReFS -Full

In fact, typical command line syntax and optional parameters apply. Therefore, if you want this to be a quick format, just append /q to the above command line. However, you have the option to enable Integrity Streams for the volume. Note the following command:

Command-Line

Format /fs:ReFS /q /i:enable J:

PowerShell Equivalent

Format-Volume –DriveLetter J –FileSystem ReFS –SetIntegrityStreams $true

The preceding command enables Integrity Streams on drive J: and performs a quick format. The /i option offers you the ability to enable|disable this feature for the volume. If you enable this option, all files created on the volume will be configured with integrity. You may turn this off for individual files or folders using the Integrity command. However, know that if a file is non-empty and has data streams created with Integrity Streams, you cannot disable the feature for that file. You cannot change the integrity status for a file once the file contains integrity data streams. You could copy the file to another partition, delete the original, and then copy it back without Integrity Streams.

Additional Thoughts

During the beta for Windows Server 2012, there were blog posts on the net that mentioned a tool called INTEGRITY.EXE. In the released version of Windows Server 2012 this tool does not exist. This is not a bug or mistake. Development provided PowerShell cmdlets to address configuration options for ReFS instead of providing another utility to keep up with that had narrow focus to begin with. I plan to construct a future post on storage cmdlets that will include how to adjust ReFS behavior using PowerShell. Also, just because a file system is self-healing doesn’t mean that backups should be forgotten. If a large chunk of blue ice falls from the sky and crushes your massive ReFS file system in one loud thunk…you’re more than likely going to need to restore the data. While that's my conservative side speaking, unexpected events do happen. It never hurts to be prepared for the unexpected or the unlikely in a datacenter.

Hey y’all, Mark here with a quick post that all you PowerShell slackers should check out. Ashley McGlone aka GoateePFE (I've seen his goatee in real life and it is magnificent)  has written up a nice little doc to help you finally let go of all your old commands for Active Directory and how to do the exact same thing or better in PowerShell. These old commands are starting to go away. You can already promote a DC using nothing but PowerShell as Greg Jaworski showed you here. So print out this cube sheet and get to learning! You've probably dropped a New Year's Resolution already so add this instead, we wont tell. 

http://blogs.technet.com/b/ashleymcglone/archive/2013/01/02/free-download-cmd-to-powershell-guide-for-ad.aspx

Mark "flowing locks PFE" Morowczynski

"The Cloud" is here to stay.

Businesses continue asking questions to their IT departments and wondering out loud:

"Are we 'Cloud-ready' and if not, when will we be?"

"What apps can we port to The Cloud right now? Next 6-12 months?"

"How will we use The Cloud to reduce costs or improve service levels – or both?"

Like all the technology we work with, The Cloud is always evolving and improving. Windows Server 2012 is Microsoft's first cloud-focused Operating System and has many features to integrate with, leverage and blur the line between Cloud-based resources, local resources and those in between.

As an IT Pro, I wasn't sure what to make of The Cloud at first. To be honest, I was more than a bit worried about the future of the IT Pro career if everyone was going to move IT to The Cloud.

Not being one to just 'wait and see,' I decided to find out for myself and I began to research The Cloud.

Lately, I've been learning more about Microsoft's Windows Azure Cloud platform. One of the more interesting factoids I learned is that Windows Azure uses more server compute capacity than was used on the planet in 1999.

Service Overview

Windows Azure has numerous services such as SQL, Active Directory, a rich development platform, VPN connectivity, etc.

The most exciting Azure feature to me – and the focus of this post – is the Windows Azure Virtual Machines (VMs) service.

This is something you can use today – this afternoon.

With Windows Azure and the addition of the Virtual Machines offering, there are tools, features and functions that enable you as an IT Pro, to have a readily-available lab without shelling out a lot of (or any) money for enterprise-class software and hardware. Of course there are other ways this functionality can be used but this post covers the idea of a simple lab.

Some of the great features I've discovered in working with Azure VMs:

• Quickly create VMs from 'canned' OS images
• Upload your own VM images
• Customize a VM image and deploy it like a template
• Attach additional virtual disks for data
• Azure has PowerShell providers
• There are a lot more…

• Download the Windows Azure poster for a good one-pager overview of the various facets of the Azure platform:
  • http://www.microsoft.com/en-us/download/details.aspx?id=35473

Let's get started … I urge you to follow along

Step 1: Sign up for a free trial…

• You are required to enter credit card information in order to verify your identity
• You are NOT charged unless you explicitly enable/change your subscription to a fee-based option
• Curious about the costs beyond the free trial? Use thePricing Calculator

    VM Size reference table (courtesy of Keith Mayer – a link to his excellent blog is at the bottom of this post):

Step 2: Sign-in to http://www.windowsazure.com/en-us/ then connect to the "Portal"

• Be sure to explore some of the resources from the main Portal page

Step 3: Let's create a new Virtual Machine

Click VIRTUAL MACHINES and then click the circled arrow. A web frame opens and you can perform a 'Quick Create' VM:

• Enter a name (will be the hostname of the OS)
• Choose an image type (OS type and version)
• Pick the size of the server in terms of proc and memory

• Enter a password for the Administrator account
  • Select an Azure datacenter location
    • Here's a cool article about the Azure datacenter technology - http://www.neowin.net/news/main/09/11/02/inside-windows-azures-data-center-one-of-worlds-largest )

Click 'CREATE VIRTUAL MACHINE' and you're off and running. Your new VM bakes for about 10 minutes.

Here is a screen-shot displaying the details of the VM I just created:

A few comments about the above screen-shot:

• The graph displays performance metrics over time (CPU, disk, memory, network in/out, etc)
• I did a bit of image-editing magic here to show what the graph would look like over time (the VM had just been created so there wasn't any data in the "real" graph)
• The 'quick glance' area shows key information for your VM
  • STATUS - the VM will be stopped or running
    • You can start a VM via the START arrow/button in the bottom row of buttons
  • HOST NAME – your VM name
  • INTERNAL IP ADDRESS – the private IP for your VM
• The VM(s) continues to count against your free trial 'capacity' even if powered off
  • Delete the VM(s), disk(s) and other associated resources if you want to avoid any hit on your trial capacity level
• To connect to the VM, click 'CONNECT' to open an RDP file/connection to it

Here we are logged into the VM:

Yes folks, it is REALLY just that easy.

The VM you created will be isolated for the moment and that's ok if you just want to check something out in a single-instance of the OS.

You can also change the size of the VM via the CONFIGURE option:

Step 4: The Gallery Option

What if we want to build out an Active Directory with multiple DCs and member servers all networked together to really 'build out' a lab?

That's when we choose the option "FROM GALLERY" which opens a different set of forms with more options:

Again, you choose from a variety of OS instances/patch levels

• NOTE – the Microsoft products are activated/licensed – BONUS!!

Give the VM OS a hostname, give the Administrator account a password and select the VM "size"

IMPORTANT NOTES FOR THE SCREEN BELOW:

• This is where you get the option to connect to other VMs you've created
• Select the lower "CONNECT TO AN EXISTING VIRTUAL MACHINE" if you want to enable network communications between other VMs
  • The option here is to connect the new VM to an existing Cloud Service
  • In my screen-shots, the name of the Cloud Service is the same as my first VM
    • HILDE-VM01.CLOUDAPP.NET

If you've setup an Availability Set, you can add this VM to that Set, if desired

• Research Availability Sets – they're out of scope for this post
• http://www.windowsazure.com/en-us/manage/windows/common-tasks/manage-vm-availability/

Click the checkmark to begin provisioning the VM

The bottom of the Portal UI has real-time feedback and status messages:

Now we need to chat a bit more about what Azure calls a "Cloud Service."

This is basically a unit of service in Azure. If we create ANYTHING in Azure, there has to be somewhere to host it – and this is called a Cloud Service. For this demo, I conceptualize the Cloud Service as "my VM Host" – it's where my VMs live.

• In this case, the Cloud Service was automatically created when I created my first VM
• It is given the same name as my first VM
• The Cloud Service can be seen in the UI

   

• At this point, we have two VMs created in the same Cloud Service

Since the two VMs are in the same Cloud Service, they can "see" each other.

The next step is to enable the VMs to communicate. This is similar to setting up to any other servers (physical or virtual) to communicate with each other:

• Configure IP networking on the VMs
  • Azure assigns the VMs an IP address which lasts until the VM is deleted
    • Don't set the VMs with static IPs, even if you use the IP assigned by Azure
    • I read that setting the VMs statically with their dynamic IP can cause communication issues with the VMs 
  • Define the DNS server entry on the NICs on both VMs to point to the first VM's IP address
    • The first VM will be DCPROMO'd first, establishing the AD forest
• Next, I DCPROMO'd the first VM and created the AD forest
• I joined the second VM to the new domain I created, then I DCPROMO'd that second VM
• There are screen shots from each system below
  • One WS 2012 VM and the AD Administrative Center
  • One WS 2008 R2 VM and the Server Manager UI

   

A Cloud Inside of a Cloud

Here's a high level diagram of how the Azure environment, my Cloud Service, the two VMs in that Cloud Service and remote access to the VMs all interact.

That, my friends, is how you can create a two-DC domain, for free, which is accessible from nearly anywhere.

• "Anywhere" includes your in-laws house over the holidays as well as via your shiny new Microsoft Surface device you got for Christmas J

Hopefully, this demo will get you thinking about possibilities with Windows Azure and The Cloud.

Check out these links for some more good info:

http://www.windowsazure.com/en-us/manage/windows/fundamentals/

http://blogs.msdn.com/b/windowsazure/

So to all you IT Pros out there …. Go for a walk in the Windows Azure Cloud.

Your Personalized Go-Do List (just for you)

• Research how you can connect your on-prem infrastructure to Azure via VPN tunnels/end-points.
• Research how you can tie in your on-prem AD to a private Azure AD instance.
• Be the Hero who avoids the cost to her company for expensive lab equipment.
• Be the first one on your team to get certified on WS 2012 and use Azure VMs to help you study.
• Be the decorated engineer who reduces his company's costs and time to recover from outages by utilizing DR VMs hosted in an Azure Cloud Service connected via VPN to his corporate intranet.
• Check out Keith Mayer's excellent blog which covers Azure and many other great topics for IT Pros - http://blogs.technet.com/b/keithmayer/
  • Special thanks to Keith for the VM sizing table details listed in Step 1 of this post

Until next time…take care!

Hilde

I hope you enjoy the blogs we provide at AskPFEPlat.  We love sharing our passion for Microsoft Technologies.  But, believe it or not, we do have day jobs.  As Premier Field Engineers we spend most of our time with Microsoft Premier Support customers, helping them learn, test, deploy and troubleshoot the technology.  It’s this hands-on experiences with customers that helps drive the content that we deliver in our blogs.

One of the dimensions of a Microsoft Platforms PFE includes delivering Risk Assessments for Microsoft Platforms-related technologies, such as Active Directory (ADRAP), Failover Clustering (CSRAP) or the Windows Desktop (WDRAP). 

Fellow platforms PFE, Yong Rhee, has a great blog that describes an evolution in our traditional Risk Assessments – RAP as a Service (RaaS).  The blog does a great job of explaining RaaS and includes details for requesting more information.  If you do engage PFE services, remember to tell them that AskPFEPlat sent you.  We might even be able to arrange to have one of our blogging superstars deliver for you.  And if they’re not available, maybe we can send Mark.

Doug "Cheap Shot" Symalla

Hey y’all, Mark here again. Another year has come to an end, the Mayans were wrong and we started to look back on what our most popular post of the last year were. For you long time readers (AD MVP Mike Kline, our very first comment) this will be a trip down memory lane. For any of our new readers, you can now get a quick “Year 2012” mix tape, remember those! Those were sort of like playlist for your walkmen which used these things called cassettes ….never mind. Like a good mix make sure you pass this on to a friend.

10.) Want Remote PowerShell Management from your browser? See how PowerShell Web Access in Windows Server 2012 may help…

9.) Slow Boot Slow Logon (SBSL), A Tool Called XPerf and Links You Need To Read

8.) MCM: Core Active Directory Internals

7.) How to Implement the Central Store for Group Policy Admin Templates, Completely (Hint: Remove Those .ADM files!)

6.) HYPER-V 2008 R2 SP1 Best Practices (In Easy Checklist Form)

5.) How to become a Premier Field Engineer (PFE)

4.) Virtual Domain Controller Cloning in Windows Server 2012

3.) Windows Server 2012 Storage Spaces: Is it for you? Could be…

2.) Introducing the first Windows Server 2012 Domain Controller (Part 1 of 2)

1.) Did Your Active Directory Domain Time Just Jump To The Year 2000? / Fixing When Your Domain Traveled Back In Time, the Great System Time Rollback to the Year 2000

Our 5^th most popular blog was actually our a reader topic request so if you have a question you’d love to get to bottom of you probably are not the only one. Let us know either in the comments, the Contact button at the top of the page or tweet about AskPFEPlat. We will do our best to get back to you.

Is your favorite post not listed above, let us know in the comments and why you loved it.

Mark “use a pencil to rewind it if you need to” Morowczynski

Hi All! DougG here to share some insight on password policies – enjoy.

We were all excited when Windows 2008 Domain Functional level introduced FGPP (Fine Grained Password Policies). After several years in the field I have not seen abuse of this feature. In-fact, I am pleased to share that those using the FGPP are taking the conservative approach. By that, I mean I am not finding 20 or 30 FGPPs in domains. Rather, only 1 to 3 FGPP have been typically sufficient for most customers using FGPP. This means you are still using the Default Domain Policy to manage passwords for users that do not have a FGPP applied to them.

It is the Domain Password Policy that is the more complex of the two and the reason for this post.

WARNING!

First and for most, if you try to demo this or follow along on a domain make sure you are in a lab environment. What I am about to show you will impact the users and if you are in a production environment this will cause an RPE or CLM (Resume’ Producing Event or Career Limiting Move) – take your pick.

If you know these three facts you will be able to understand why things work the way they do.

1. Only policies applied at the DOMAIN level will apply a password policy to domain users. This can be the Default Domain Policy DDP or a policy that you have added that has a higher precedence (lower number than the DDP). Or it can be a combination of policies at the domain level.

2. Any policies applied to OUs, INCLUDING the domain controller OU, which has password policies will not be applied.

3. The password policy is written at the domain head by the PDCe (remember the stance that we don’t support sub OUs for domain controllers?)

Where do we get tripped up? RSOP for one. RSOP results will show you the policy including any OU that has a password policy setting. But only policies at the Domain level apply to the domain users. So when you look at an RSOP result you scratch your head, because what the user is experiencing is not what is on the RSOP report.

For example: If you wanted users in a specific OU to have shorter password length requirement and applied a password policy on their OU you will see your new settings in an RSOP report for those users. However, the real answer can be seen with

Net Accounts /Domain

Create a simple lab:

I have a 2008 R2 single domain forest with two domain controllers. Good to have two so you can prove one of the points about the PDCe.

I have one Windows 7 client – I have placed this machine account in the W7User OU to facilitate demonstration. It is not recommended to combine users and machines in the same OU.

I have User 1 in the same OU, W7Users, with a policy defining password policy settings.

Problem number 1:

Trying to apply a password policy at an OU.

Hopefully you know this won’t work, but going to show it anyway.

Logged in as user1 on my LAB domain, NWT.COM, using net accounts /domain you see I am getting the policy with the default settings that come from the domain:

However, my administrator tried to give me a much easier password policy. He created a policy on my W7Users OU where both my user account and machine account reside with the new settings. You can see from the RSOP result on my Win7-32 client that User 1 has logged on at least once.

Look closely and it appears that I am getting some settings from the Default Domain Policy and some from MyPasswordPolicy. Using net accounts /domain you can see what the password policy is for the user:

No matter how many time you reboot or run GPupdate /force, the OU policy will not affect the domain user account.

Where this setting does take affect is on the local machine for local users.  To see that, drop the “/domain” from the net accounts and you will see the settings applied to the local machine.

Net accounts

You can tell which “net accounts” you ran by the last line.  If it says “Primary” or “Backup” then you are getting that from the domain via the  PDCe or another domain controller and the command was run with the /domain switch.   If it says “Workstation” it is the local workstation policy and the command was ran without the /domain switch.

Just to continue to prove a point, we can just leave this garbage policy on the OU. It isn’t doing anything other than making the client read a policy that doesn’t do anything to him – but it is consuming time reading the policy and retrieving information from SYSVOL.

Problem Number 2

Blocking inheritance on OUs with the domain controller OU or on sub OUs within the domain controller OU.

Many environments have sub OUs under the domain controller OU to facilitate applying updates via WSUS or other tool. To help with that, some have created an OU structure so the patches are rolled in stages. So I created a North and South OU for my environment. I have also blocked inheritance on the North OU (this is where my PDCe is now) because the Security Team is patching the DCs tonight and I don’t want the North DCs to get the patch until next week.

The PDCe is the path to get the policy applied to the domain. So you can probably guess what is going to happen or rather not happen. Not to mention the fact that the Default Domain Controller Policy is also blocked now. What a mess. You are in your lab, right?

The next day the Security team requests to change the password policy to require 10 character passwords. No problem, you get the change request and open the default domain policy and modify it as requested:

Ticket closed.

Couple of hours later the Security Team calls you and informs you that users are still able to have 7 character passwords. You look at the policy settings and send the screenshot above showing the setting and tell the security team to be patient it is probably just a replication timing thing, you know how that goes.

You return to work tomorrow with an email from the Security Team reporting the password policy still is not in affect and has an output from user 1’s “net accounts /domain” command (smart guys).

It is then you remembered that you had temporarily blocked inheritance on the North OU where your PDCe is located. Patching went well the other night, so you get a change request to allow the North DCs to be patched as well and unblock inheritance. Once that was complete you ran “GPupdate /force” on the PDCe. It would have applied in 5 minutes anyway, but we were in a hurry.

You call security and let them know the issue is resolved and they reply with a thank you and a screen shot of the net accounts /domain. Clients will have to restart, run gpupdate /force, or wait for background refresh to update the policy.

Problem Number 3

You have a work around for last month’s password problem when you had blocked inheritance on OUs with the domain controller OU or on sub OUs with the domain controller OU.

You again blocked inheritance on the North OU, but to work around the issue you also linked the DDP to the North OU as well, just in-case there is a change request.

Cool! We’re covered. Of course the 10 character password we set last month is causing some problems so the Security Team requested a change to return the password back to 7 characters. No problem you get the ticket to change the setting back to 7 characters and since DDP is linked to the domain and the North OU, the PDCe will pick it up. Right? Well let’s see.

Since you learned from last month you run “net accounts /domain” on the PDCe and get the following results:

It didn’t work.

Again you have to unblock inheritance on the North OU to allow the PDCe to read the domain level policy to apply the password policy to the domain.

This has been a rough couple of months any you have decided to change your patching process and go the supported configuration and move all DCs back to the domain controller OU and delete all the sub OU with in the domain controller OU.

Problem Number 4

Multiple policies at the domain level with password policy settings.

The password policy applied to the domain does not have to be in the default domain policy. However, it is best practice to keep it there.

On the off-chance that someone has added policies at the domain level with password settings, they may not apply. By default a new policy added will be lower on the priority list and will be overridden by the default domain policy if there is a conflict in a setting. You would have to consciously change the precedence of the new policy to supersede the DDP.

I have shown you everything so far, so let’s do it one more time.

New policy on the domain head with password settings. I set a couple of items so we can see what happens. Changes include:

Maximum password age: 90 Days

Minimum password length: 10

I think you have got the trend down from this blog post, so let’s jump straight to net accounts /domain. Note that the new password age and length requirements did not apply.

Surprised? Nothing from our new policy came through. If you use GPMC and click on the domain name (nwt.com in my case) you can see the order of precedence:

But you probably also noticed there are arrows to change the precedence, so let’s do that.  Depending on which one is highlighted, hit the up or down arrow to move “New Password Setting” to the top (or at least a lower number than the DDP).

Gpupdate /force

Net accounts /domain again. And now you can see the changes take affect with a (0 day password with a 10 character length requirement.

What is interesting here, is the password policy is FINALLY acting like other GPOs. If there is a conflict in settings, the higher precedence wins. If the setting is only defined in one of the policies, it will be applied regardless of precedence.

Some notes to bring FGPP back into the discussion

1. FGPP take precedence if assigned to a security principle (groups or users typically)

a. You cannot assign FGPP to OUs.

2. If no FGPP is applied to the user logging in, they follow the password policy applied to the domain.

What should you get out of this post? Hopefully a better understanding of Domain Password Policies. To wrap this post up I have copied the three things you need to know to understand how passwords are applied:

If you know these three facts you will be able to understand why things work the way they do.

1. Only policies applied at the DOMAIN level will apply a password policy to domain users. This can be the Default Domain Policy DDP or a policy that you have added that has a higher precedence (lower number than the DDP). Or it can be a combination of policies at the domain level.

2. Any policies applied to OUs, INCLUDING the domain controller OU, which has password policies will not be applied.

3. The password policy is written at the domain head by the PDCe (remember the stance that we don’t support sub OUs for domain controllers?)

Now wasn't that fun?

One common and potentially time consuming administrative task I hear my customers discuss is maintenance of SSL certificates on large Web farms (initial set up, renewals, etc.). These Web farms can be for Outlook Web Access, Ecommerce, intranet pages, etc. One of the new features we added in IIS8 in Windows Server 2012 is Central Certificate Store (CCS). The gist of this feature is to allow IIS8 to go get certificates for SSL sites it hosts on demand from a central location instead of its local certificate store.

What is CCS?

I originally planned on breaking down how it works, etc. however a colleague named Kaushal Kumar Panday has already done a fantastic job covering CCS in the blog post found here:

http://blogs.msdn.com/b/kaushal/archive/2012/10/11/central-certificate-store-ccs-with-iis-8-windows-server-2012.aspx

So instead of repeating all this info, I thought it would be cool to peek into a couple areas of the feature to get to know it a little better, as well as provide guidance for a robust implementation. So if you want to geek out on CCS, read on!

Let's Geek Out on CCS!

One thing I thought would be neat to see in a packet capture via Network Monitor is the interaction between the Web server and the file server that hosts the certificates for CCS. Here is my setup which will be used for the duration of this post.

1. Server2012-2 (10.10.14.152) - Web server hosting an SSL site "centcertdemo" configured to use CCS.
2. Server2012-3 (10.10.14.153) – File server hosting the share with the certificate. Note this server does not have to be a Windows Server 2012 server but there is a security enhancement you will gain by using 2012. I'll cover this a bit later in the "Security" portion of this article.
3. Win8ent (10.10.14.154) – Windows 8 client accessing the SSL site.

The first thing I did was an IISReset on the Web server just to get the trace as clean as possible. It's also the #1 thing I use to troubleshoot IIS issues but that's just because I am a newb when it comes to IIS, but I digress J. Let's see how this works:

1. Win8ent makes a TCP connection (3-way handshake) and initial SSL hello to Server2012-2 (Web server). We can see that here in the trace.
   
   
   
   
   In that Hello should be the Server Name as part of the ClientHelloExtension for the site the client is requesting. We can see that here in the Hello packet.
   
   
   
2. Next, Server2012-2 (Web server) should go look on the share hosted on Server2012-3 (File server) for the certificate file name matching the same of the site/servername the client is requesting. This is occurring via SMB2 as we see in the trace snip below. The key point here is that the file name must match the site the client is requesting.
   
   
   
   
3. Now here's something interesting, Server2012-2 (Web server) doesn't ACK back the SSL Hello until after it has the certificate, then finishes the SSL Hello.
   
   

So perhaps one thing to learn here is to use at least Windows Server 2008 so you can reap the benefits of the updated TCP/IP stack as well as SMB2.

High Availability

Another thing to plan for is failure conditions. It would be most unfortunate if the file server that hosts the central certificate store was unavailable when an SSL request came in and the cert was not cached. At first I wondered if we kept a local copy so I decided to test that theory out.

To set up the scenario where the file share hosting the certs for CCS dies, I stopped the server service on Server2012-3 (File server) then did an IISReset on Server2012-2 (Web server)

1. Same as before, Win8ent makes a TCP connection and initial SSL hello.
   
   
   
   
2. Now Server2012-2 calmly calls over to Server2012-3 to get the cert but notice packet 77, this Reset is expected since the Server service is stopped on Server2012-3.
   
   
   
   
3. Server2012-2 freaks out and frantically starts to reattempt to set up SMB2 and pull the cert, but his/her attempts are futile.
   
   
   
   
4. Now here's the interesting part, Server2012-2 finishes the SSL Hello, but check out what cert is used.
   
   
   
   
   
   That cert is bound to another 443 SSL Web site on the server (Powershell Web Access) and is not retrieved via the CCS functionality. The client also got a nasty cert error since the names don't match.
   
   

    

So as you can see it's critical to make sure your certs are hosted on a highly available file share. Failover clustering would be a nice choice J.

Security

I also want to point out a couple things from a security perspective pertaining to this screenshot. To navigate to this screen, in the IIS Manager, click the Server name in the Connections pane on the left, then double-click on the Centralized Certificates icon under Management in the center pane. As a reminder, the centralized Certificates icon will only show up if you have installed the feature via Server Manager.

1. Use least privilege practices for the account you configure IIS to use to access the share for CCS. Lock down your share and NTFS permissions as well as remove the account from Domain Users and set workstation logon restrictions (don't forget to add the computer objects for the servers hosting the share!). Also restrict others from accessing the share since it will have full certificates stored there.
   1. At this time I am not aware of the ability to use Group Managed Service Accounts for this feature. I will update the blog if I find out that you can.
2. Bitlocker protect the volume that hosts the share for the certs. We introduced the ability to Bitlocker protect Cluster Shared Volumes in Windows Server 2012. The volume hosting the share holding your SSL certs might be a good candidate perhaps? J
3. Also note in the config dialog how the Private Key password is optional. This assumes that you did not use a password to protect the private key when you exported the certificate to place on the share. I can't think of a reason as to why you would NOT want to use a private key password. So set a private key password and configure this option.

Powershell Integration

Finally, there are Powershell commands which can be used to configure CCS. This is a good way to further automate Web server build processes that host SSL sites.

That's all for now on CCS. I hope you enjoyed this blog and please post in the comments if you have any questions, comments or concerns.

Jake Mowrer

Happy last Friday of January to all that read our Ask PFE Platforms blog!  This afternoon’s post is based on a submission to the Ask PFE Platforms Mail Bag.  The chosen question was submitted using the Contact tile just to the right of this article when viewed from our TechNet blog.

Question

Is it necessary to restart a server as soon as possible after installing updates or is it safe to suppress the restart for a day, week, month, or longer?

Thoughts regarding the question

Within Windows, there are many features and capabilities.  Restart suppression when applying updates is just such a feature.  The ability to suppress a system restart following the application of an update or other system configuration change exists so that additional changes may be performed prior to restart.  While the option exists to suppress a restart, keeping the restart suppressed for extended time may not be the best option and may carry more risk.   This reminds me of a prior blog post of mine where I discussed the effect of using 512 byte clusters on a system disk, the potential side effects with VSS and system performance, and the additional administrative overhead involved to keep things functioning at a healthy level.  Point being, there are configuration options like restart suppression.  However, just because it appears you could suppress an update related restart for extended periods of time, doing so is likely not a best practice and may have side effects.  

First, let’s revisit updates.   Some updates or security hotfixes require a system restart after application.  Many times accompanying bulletins or KB articles indicate the requirement of a system restart following the application of the update.  Those are the cases where the need to restart is known.  For example, if the update replaces NTOSKRNL.EXE, that definitely requires a restart.  There are other situations where a restart may be required because the update process needs access to the files already in use by the system that may be blocked for unexpected reasons (Anti-virus filter driver has it blocked, etc...)   Starting with Windows Server 2008, many times DLLs in use by running processes may often be updated without need to restart, and sometimes without the need to restart applications or services.   On occasion, files that need to be updated may be locked and need to be replaced prior to shut down or as part of system startup…and thus, the system must be restarted to apply the update.  Also, it is important to note that updates may be to other aspects of the operating system than just binaries.  Some updates may add or remove registry settings, change registry permissions, or even apply permissions to changed files.  In some cases, replaced files are not protected until after a restart takes place.   Some components load settings at boot time and are cached, therefore a restart must be performed in order for those components to have updated settings.   After restart, advanced installers for components may perform a variety of tasks to complete the installation of components.  Pending.xml gets populated with files and registry values needed to install updates.   For more information about pending.xml or how service packs are handled by Windows, the following link is a great reference:

http://blogs.technet.com/b/joscon/archive/2011/03/11/why-you-don-t-want-to-edit-your-pending-xml-to-resolve-0xc0000034-issues.aspx

If you suppress a restart after applying an update, then the update is not completely in place.   If this update were a security update, the system would remain vulnerable until the restart and completion of the update after boot.  How long should the restart be suppressed?  How long should a system remain vulnerable?   Further, group policy can influence application of updates and handling of restarts.   For more information on Windows Update and group policy, consult this post on the Microsoft Update Product Team Blog:

http://blogs.technet.com/b/mu/archive/2008/10/02/windows-update-and-automatic-reboots.aspx

In theory, you could apply updates, suppress the reboot, and wait an infinite period of time.  However, the longer the restart suppression, the higher the chance that unpredictable things could happen…especially if we’re talking about multiple updates.  While suppressing a restart, there is no guarantee that there won’t be side effects based on what changes were applied and those that remain pending.  At least, I’m not aware of any. I certainly have observed system hangs or other odd behavior with systems of my customers when they’ve suppressed restarts.  However, I’ve seen these issues less with Windows Server 2008 and later as compared to Windows Server 2003 or older versions.  The moral of the story here is to restart as soon as you can after updates or changes that require it.  My own conservative approach is to apply updates during a window of opportunity for restart and not to suppress them for extended periods.

Windows Server 2012 Pending Restart Notifications

Windows Server 2012 does indicate pending restarts in Server Manager.   When I first saw this my thought was that it was annoying.  However, if you think about this notification in the context of a datacenter where an administrator updated something, suppressed the restart, and then left for the day and otherwise forgot to complete the job and you’re now troubleshooting strange server behavior a month later…it is nice that a notification like this could clue you in that maybe a simple restart is all the system needs to complete pending changes.  I can think of at least one all day troubleshooting call within the last year that this type of notification would have prevented on an older system where an admin updated things a month before and never restarted the system.

I hope this post addresses concerns and helps shed some light on why restarts after updates are good to provide sooner than later and perhaps why restarts need to happen with some updates.  Until next time!

Martin

Hi this is Brandon Wilson and today I will be providing you with a quick reference for troubleshooting Netlogon error codes.

I say “quick reference” very loosely here, because this is one of those sticky subjects that can easily expand into many more areas and become a very long discussion. So, I’m going to do my best to focus on just the codes and possible solutions for the error codes that are more common to see. Anyone who has dealt with some of these issues knows that some of these outlined areas can lead to huge revenue loss for a business, or at the very least an annoying authentication prompt J

I welcome you to leave comments and questions, or suggestions for articles that you as the reader would be interested in seeing us do on AskPFE Platforms (http://blogs.technet.com/b/askpfeplat/contact.aspx).

Netlogon logging overview:

Where do I enable the Netlogon logging?

How do I enable verbose Netlogon logging?

Differences between logging level verbosity:

Netlogon.log Maximum File Size:

Let’s dig into the errors!

0xC000005ESTATUS_NO_LOGON_SERVERS

0xC0000022 (or 0x00000005 (0x5))STATUS_ACCESS_DENIED

0xC0000064STATUS_NO_SUCH_USER

0xC000018ASTATUS_NO_TRUST_LSA_SECRET

0xC000006DSTATUS_LOGON_FAILURE

0xC000009ASTATUS_INSUFFICIENT_RESOURCES

0xC0020050 (Decimal -1073610672)RPC_NT_CALL_CANCELLED

0xC0000017STATUS_NO_MEMORY

0xC000006ESTATUS_ACCOUNT_RESTRICTION

0xC000006CSTATUS_PASSWORD_RESTRICTION

0xC0000070STATUS_INVALID_WORKSTATION

0xC000006ASTATUS_WRONG_PASSWORD

0xC0000193STATUS_ACCOUNT_EXPIRED

0xC0000192STATUS_NETLOGON_NOT_STARTED

0xC0000071STATUS_PASSWORD_EXPIRED

0xC000006FSTATUS_INVALID_LOGON_HOURS

0xC0000234STATUS_ACCOUNT_LOCKED_OUT

0xC0000072STATUS_ACCOUNT_DISABLED

0xC00000DC (Decimal -1073741604)STATUS_INVALID_SERVER_STATE

0xC0000224STATUS_PASSWORD_MUST_CHANGE

Netlogon logging overview:

I suppose I can’t assume that everyone in the world knows this, so let’s kick off the topic by covering where to enable Netlogon logging and the ways to enable it. How to enable Netlogon logging is also outlined at http://support.microsoft.com/kb/109626.

Where do I enable the Netlogon logging?

If you are having NTLM authentication or PAC validation issues, be prepared to enable verbose Netlogon logging across the entire authentication chain. Let use a common example, a web server servicing authentication:

1. Web server services users from the local domain only:

a. Enable verbose Netlogon logging on the application server

b. Enable verbose Netlogon logging on the domain controllers in the same logical Active Directory site (or any covering the site via Auto Site Coverage).

i. You can also enable logging on the web server first to identify the domain controller being contacted (or that contact is attempted with) when an issue occurs. From there, you can enable the logging on the domain controller you identify. A word of warning on using this method: if the issue is not persistent or is intermittent, you may lose your chance to gather all the necessary data the first time around.

2. Web server services users from another domain in the same forest:

a. Enable verbose Netlogon logging on the application server

b. Enable verbose Netlogon logging on the domain controllers from the web server’s domain that are in the same logical site

c. Enable verbose Netlogon logging on the domain controllers in the same logical site in the forest root (if the web server’s local domain is not the forest root)

d. Enable verbose Netlogon logging on the domain controllers in the same logical site in the target domain (if the target domain for authentication is a different child domain of the forest root)

NOTE: As mentioned before, you can also enable the logging selectively based on the DC discovery calls within the Netlogon log to identify the next level in the authentication chain.

3. Web server services user from another domain in a different forest:

a. Enable verbose Netlogon logging on the application server

b. Enable verbose Netlogon logging on the domain controllers from the web server’s domain that are in the same logical site

c. Enable verbose Netlogon logging on the domain controllers in the same logical site in the forest root (if the web server’s local domain is not the forest root)

d. Enable verbose Netlogon logging on the domain controllers in the same logical site in the forest root of the target domain. If the same logical site name does not exist in the target forest, you will need to identify the domain controller that is being contacted. This can be done with a network trace while the issue is occurring, or via the Netlogon logs.

e. Enable verbose Netlogon logging on the domain controllers in the same logical site in the target domain. If the same logical site name does not exist in the target forest, you will need to identify the domain controller that is being contacted. This can be done with a network trace while the issue is occurring, or via the Netlogon logs.

NOTE: As mentioned before, you can also enable the logging selectively based on the DC discovery calls within the Netlogon log to identify the next level in the authentication chain. In the case of cross forest authentication, it may actually be necessary initially to identify the domain controller we are talking to.

Remember, there are calls in the Netlogon log that represent the establishment of the secure channel with the other domain and will denote the domain controllers we are talking to (these are found in [SESSION] lines).

How do I enable verbose Netlogon logging?

1. From the command line:

a. To enable Netlogon logging, run the following command (w/o quotes): “nltest /DBFlag:0x2080FFFF”

b. To disable Netlogon logging, run the following command (w/o quotes): “nltest /DBFlag:0x0”

2. From the “Microsoft Fix it” button:

a. Browse to http://support.microsoft.com/kb/109626

b. Scroll down to the “Fix it for me” section

c. Click the appropriate “Microsoft Fix it” button to enable or disable Netlogon logging

IMPORTANT NOTE: The Netlogon.log and Netlogon.bak files that are generated can be found in %systemroot%\Debug.

Differences between logging level verbosity:

When DBFlag is set to 0x0, it is common to have a 1kb file. This may of course not be the case if Netlogon logging has been enabled at any level in the past. On your Domain Controllers, you may see entries stating NO_CLIENT_SITE that can be useful to track and control straying clients.

When the value is set to the maximum verbosity (0x2080FFFF), you will see every single action taken by the Netlogon service. This does have a bit of overhead in terms of disk and memory utilization, hence I do not recommend you keep the maximum verbosity enabled at all times unless you have frequent issues relying on Netlogon logs to troubleshoot.

The example below was taken with maximum verbosity and a restart of the Netlogon service was performed. In this snip you can see a session setup failing to a trusted domain with a no logon servers available error (we will cover that error in another post). As you can see though, a LOT of data is reported.

For reference, a full reference to the debug flags and what they give you can be found at the bottom of http://support.microsoft.com/kb/109626.

Netlogon.log Maximum File Size:

If your issue is intermittent, or spans longer intervals, you may wish to increase the maximum log file size for the Netlogon.log and Netlogon.bak file to help ensure pertinent data is not overwritten.

The default size of the Netlogon.log (and Netlogon.bak) is 200,000 bytes (~20MB), and this can be increased up to 4GB (4294967295 bytes) at its maximum (I do not recommend you actually ever make it this large). If you are increasing the maximum log file size, double and triple check your disk space to ensure you are setting a reasonable size and will not stress your disk space.

You can also increase the maximum log file size by setting the MaximumLogFileSize DWORD value in the registry in HKLM\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters. The value data should contain the maximum log file size in bytes (decimal).

Alternatively, you can use group policy and configure the Maximum Log File Size setting under Computer Configuration\Administrative Templates\System\Net Logon.

Let’s dig into the errors!

Now that my longer than expected due diligence is done, let’s start discussing some of the errors and warnings you may encounter within the Netlogon log….

This error tends to be the number one cause of some VERY expensive outages at companies that occur. It is commonly reported in errors as: “There are currently no logon servers available to service the logon request”. Due to its potential impact, I will go a bit more into depth on this error…and hopefully NOT bore you to sleep in the process.

When and IF you have a MCA (MaxConcurrentAPI) issue, this is likely what you will see littering your Netlogon logs, and potentially your event logs as well. A MaxConcurrentAPI (MCA) issue occurs when the threads within lsass.exe that handle NTLM authentication (as well as Kerberos PAC validation) begin to time out. This takes 45 seconds, as measured on a thread by thread basis (just consider the thread an authentication attempt to make it easier). Once this timeout occurs, you throw the 0xC000005E error, and logon fails. MCA issues will stand out from the crowd of data in the Netlogon log with a [CRITICAL] line stating “can’t allocate client API slot” (see example below). If you see this, you have a MCA issue, and you might as well skip straight to the common causes now J

MCA example (this is not the only indicator, but is a dead giveaway):

6/3 14:17:43 [CRITICAL] FakeDomain: NlAllocateClientApi timed out: 0 258

6/3 14:17:43 [CRITICAL] FakeDomain: NlpUserValidateHigher: Can't allocate Client API slot.

If you don’t see this message, continue reading….

This error does not however always indicate a MCA issue is occurring! There are also some common causes, such as trusts that exist with decommissioned domains (or trusts with domains that cannot be contacted for another reason). Since I went with this example, I may as well tell you that it will result in the same error code in the logs, along with 5719 events in your System event log from Netlogon indicating the trusted domain could not be contacted.

Another example would be a client with an invalid DNS configuration. This DNS configuration can in turn cause the client to not be able to find the domain and domain controller, which would also leave us with a “no logon servers available” error. NOTE: “Client” in this context can be a workstation, member server, or even another domain controller.

The list below covers some common causes for the notorious “no logon servers are available…” error message, and in some cases, suggestions for implementing a fix:

1. DNS forwarders (if crossing domain/forest boundaries) – maybe somebody forgot to update the IP when it was changed on a target domain/forest DNS server

a. Correct any “catch all” forwarders (Windows 2000) to point to the target forest’s DNS servers in the sending domain’s DNS configuration (also validate and correct the other end) -OR-

b. Correct any conditional forwarders (Windows 2003 and above) for the target forest in the sending domain’s DNS configuration (also validate and correct the other end)

REFERENCES:

How to configure DNS for internet access in Windows 2000: http://support.microsoft.com/kb/300202 (see the section labeled “To configure forwarders”)

Assign a conditional forwarder for a domain name (Windows 2008/2008 R2): http://technet.microsoft.com/en-us/library/cc794735(v=WS.10).aspx

Configure forwarders for a DNS server: http://technet.microsoft.com/en-us/library/cc755608(v=WS.10).aspx

Conditional Forwarding in Windows Server 2003: http://support.microsoft.com/kb/304491 (provides a good explanation of forwarding in general)

2. DNS records (A, AAAA, SRV) for domain controllers in the target domain may be missing or incorrect

a. Validate DNS records exist for the target domain controllers (A and SRV)

b. Restart the Netlogon service on the target domain’s domain controllers and allow up to 15 minutes for the DNS records to occur

c. If necessary, attempt to manually create the DNS records (NOTE: This should not be considered a permanent solution)

i. If you had to manually recreate the DNS records, you still need to troubleshoot why you failed to dynamically register the applicable records.

3. 1B/1C WINS records for domain controllers in the target domain may be missing or incorrect (see: http://support.microsoft.com/kb/139410)

a. Registration of WINS records may be failing

b. WINS replication may be broken

c. Incorrect static WINS records may exist

d. You may have conflicting entries in your LMHOSTS (or HOSTS) file

4. You may have invalid entries in your HOSTS and/or LMHOSTS files for the domain name, domain controller name, 1B record, or 1C record

a. Correct or remove the conflicting entry in your HOSTS or LMHOSTS file

5. You may be experiencing network timeouts due to faulty or misconfigured network hardware (ex: black hole router or MTU size set too small)

a. Follow KB314825 (http://support.microsoft.com/kb/314825) to determine if a black hole router is a potential culprit (this will also help you determine the MTU size)

b. Spanning tree protocol may be enabled at the hardware level (switch, router, etc) with PortFast disabled

i. Enable PortFast

c. VPN tunnel, router, or switch may be having hardware issues

6. SYSVOL and Netlogon may not be shared on the domain controller a connection attempt was made to (see: http://support.microsoft.com/kb/257338)

7. Network issues

a. Capture network traffic:

i. Check for excessive packet fragmentation

ii. Check for dropped packets

b. Disable SNP features per http://support.microsoft.com/kb/948496 in the registry and at the NIC driver level

8. You may be logging onto a RODC that does not have connectivity to a writeable DC (see: http://support.microsoft.com/kb/949048)

9. Logical site names in Active Directory may not match in the source and target forests (applicable only when DNS is used for cross domain name resolution)

a. If the site names are the same, then domain controller covering the site:

i. May be down/restarting

ii. No domain controllers are covering the site automatically (if no domain controller is assigned to the logical site)

iii. DNS SRV records may be missing

Perhaps I should clarify a bit on #9 above…

When DNS is used to perform a domain controller lookup in a target domain/forest, it will query for the logical site name of the requesting machine. If that site does not exist in the target forest with the exact name, spelling, etc, then Windows will default to performing a basic LDAP record lookup against the domain. This in turn can lead us to stray to locations for authentication by a domain controller that we may not want to stray to (for instance, a domain controller across a very slow link). If however you DO have a site name that is the exact same in the target forest, then LDAP SRV record lookups will occur against those domain controllers (or any registered for auto site coverage).

In addition to the seeing this error code in the Netlogon log, you may also see this error code logged in Netlogon error events within the System event log (commonly a 5722). Since the 0x5 error is so common, I also included a few additional possibilities outside of the scope of Netlogon.

Common causes:

1. You are attempting to join a machine who’s name already exists in Active Directory

2. Secure channel may be broken

a. Reset secure channel

i. nltest /sc_reset:<domainname>

OR

b. Rejoin domain

3. Trust password may be mismatched

a. Reset trust password

i. nltest /sc_change_pwd:<domainname>

4. Incorrect credentials may have been used (0x5)

a. Enter the appropriate credentials

5. NTLM blocking may be enabled

NOTE: NTLM blocking is only available in Windows 7, Windows Server 2008 R2, Windows 8, and Windows Server 2012

a. Disable NTLM blocking immediately and perform NTLM auditing

i. For Windows 7, Windows Server 2008 R2, Windows 8, and Windows Server 2012 you can use the NTLM Auditing abilities built into the operating system (see: http://blogs.technet.com/b/askds/archive/2009/10/08/ntlm-blocking-and-you-application-analysis-and-auditing-methodologies-in-windows-7.aspx)

ii. Once you have COMPLETELY eliminated NTLM authentication, you may re-enable NTLM blocking

Some other potential causes:

6. LM compatibility level mismatch

a. LMCompatibilityLevel must be at a level where authentication can be negotiated between the source and target (whether that is LM, NTLM, or NTLMv2). For example, a setting of 0 on the client and 5 on a domain controller or target server will result in an inability to negotiate a valid authentication mechanism.

b. This must be reviewed on the source/sender and the target/receiver.

i. You can find the current setting by looking in the registry at HKLM\SYSTEM\CurrentControlSet\Control\Lsa. The value is named LMCompatibilityLevel (if by chance you are still REALLY old school and are running Win9x, the value is named LMCompatibility).

ii. Valid values are 0 – 5

c. Reference table of the settings:

d. The settings, if they are incompatible, can be configured in two ways:

i. Using group policy (recommended) –

NOTE: For this example, I will assume we are using a domain level policy. The same method applies for policies at the Domain Controllers OU level, or any other.

1. Open the policy for editing using GPMC, AGPM, or Active Directory Users and Computers (whichever method you use typically)

2. Expand Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options

3. Double click the “Network security: LAN Manager authentication level” setting and change it to the desired value

4. Allow time for replication (or force replication) if necessary

5. DON’T FORGET TO UPDATE YOUR POLICY! (gpupdate /force)

ii. In the registry (this may be overwritten by group policy settings) -

1. HKLM\SYSTEM\CurrentControlSet\Control\Lsa

2. Double click the LMCompatibilityLevel registry value

3. Set the value to the desired setting (as described in the above reference table)

7. User rights assignment configuration (allow access from the network) (0x5)

a. User rights assignments are located in Computer Configuration\Windows Settings\Security Settings\Local Policies\User Rights Assignment

i. Alteration of either of the following user rights can result in an access denied:

1. Access this computer from the network

a. The default setting for Access this computer from the network is:

i. Workstations and servers:

1. Administrators

2. Backup Operators

3. Power Users

4. Users

5. Everyone

ii. Domain controllers:

1. Administrators

2. Authenticated Users

3. Everyone

2. Deny access to this computer from the network

a. The setting for Deny access to this computer from the network overrides the Access this computer from the network user right.

i. OOB, nothing is defined.

1. A typical security measure is to add the Guest account and/or the Guests group

8. Incompatible SMB signing options between the source and target machine

a. If a SMB connection is being made, SMB signing options must be compatible or it may result in an access denied error.

i. If you require SMB signing on the target, yet have it disabled on the source, then connectivity will be affected (and vice versa).

b. You can validate SMB signing options in the registry at:

i. HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters

1. EnableSecuritySignature – this value defines whether SMB signing can be used and corresponds to the group policy setting “Microsoft network client: Digitally sign communications (if server agrees)”

2. RequireSecuritySignature – this value defines whether SMB signing is required and corresponds to the group policy setting “Microsoft network client: Digitally sign communications (always)”

ii. HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters

1. EnableSecuritySignature – this value defines whether SMB signing can be used and corresponds to the group policy setting “Microsoft network server: Digitally sign communications (if client agrees)”

2. RequireSecuritySignature – this value defines whether SMB signing is required and corresponds to the group policy setting “Microsoft network server: Digitally sign communications (always)”

c. If you need to make a correction to the settings, there are two methods:

i. Using group policy (recommended) –

1. Open the policy for editing using GPMC, AGPM, or Active Directory Users and Computers (whichever method you use typically)

2. Expand Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options

3. Double click the “Microsoft network client: Digitally sign communications (if server agrees)” setting and change it to the desired value

4. Double click the “Microsoft network client: Digitally sign communications (always)” setting and change it to the desired value

5. Double click the “Microsoft network server: Digitally sign communications (if client agrees)” setting and change it to the desired value

6. Double click the “Microsoft network server: Digitally sign communications (always)” setting and change it to the desired value

7. Allow time for replication (or force replication) if necessary

8. DON’T FORGET TO UPDATE YOUR POLICY! (gpupdate /force)

ii. Using the registry (may be overwritten by group policy settings) –

1. Browse to HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters

2. Double click the EnableSecuritySignature registry value and set the value to the desired setting (0 = disabled; 1=enabled)

3. Double click the RequireSecuritySignature registry value and set the value to the desired setting (0 = disabled; 1=enabled)

4. Browse to HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters

5. Double click the EnableSecuritySignature registry value and set the value to the desired setting (0 = disabled; 1=enabled)

6. Double click the RequireSecuritySignature registry value and set the value to the desired setting (0 = disabled; 1=enabled)

9. Secure channel may be in the process of resetting (client reset its secure channel) when an authentication is attempted

a. This should clear itself up, providing the secure channel reset succeeds successfully

10. Active Directory replication to/from the target domain controller

a. There may have been recent changes that have not replicated across domain controllers (a recent secure channel reset, a group policy change, etc.)

11. You may not be applying group policy properly

a. Examine the Event Viewer to analyze group policy application

i. If necessary, enable diagnostic logging for group policy (gpsvc.log; userenv.log; winlogon.log)

12. References:

a. Nltest syntax reference: http://technet.microsoft.com/en-us/library/cc731935(v=WS.10).aspx

The most common cause of this is pretty straightforward:

1. Incorrect username was used

a. Verify the username you are using is typed correctly

Some other possibilities include:

2. Active Directory replication to/from the target domain controller may not be complete (ex: new user creation)

a. This may indicate problems with Active Directory Replication, FRS, or DFSR

3. Domain controller may be in the process of shutting down or restarting when the connection is made (see: http://support.microsoft.com/default.aspx?scid=kb;EN-US;973667)

4. If running Windows 2008 SP2, you may be experiencing the problem described in http://support.microsoft.com/default.aspx?scid=kb;EN-US;982801

5. Target domain controller resource load (high lsass.exe utilization, high memory consumption, paged for example)

a. Use Performance Monitor, Resource Monitor, or Xperf to analyze the performance of the system and identify the problem

i. Examples:

1. Paged pool memory exhaustion

2. Physical memory exhaustion

3. High CPU

The most common causes are:

1. Secure channel corruption with the host or target domain’s domain controllers

a. Reset the secure channel (nltest /sc_reset:<domainname>

2. The computer object has been deleted from Active Directory

a. Rejoin the domain

i. You may need to reinstall applications as a result of rejoining the domain

Some of the other potential causes are:

3. Blocked ports on a firewall

a. Ensure all required ports for domain functionality are enabled per http://support.microsoft.com/kb/832017 or http://support.microsoft.com/kb/179442

4. Active Directory Replication may not be complete (if the computer has been recently joined to the domain)

Some of the potential causes for this

1. An invalid username and/or password was used

a. Verify you are using the correct username or password

2. LM Compatibility mismatch between the source and target

b. LMCompatibilityLevel must be at a level where authentication can be negotiated between the source and target (whether that is LM, NTLM, or NTLMv2). For example, a setting of 0 on the client and 5 on a domain controller or target server will result in an inability to negotiate a valid authentication mechanism.

c. This must be reviewed on the source/sender and the target/receiver.

i. You can find the current setting by looking in the registry at HKLM\SYSTEM\CurrentControlSet\Control\Lsa. The value is named LMCompatibilityLevel (if by chance you are still REALLY old school and are running Win9x, the value is named LMCompatibility).

ii. Valid values are 0 – 5

d. Reference table of the settings:

e. The settings, if they are incompatible, can be configured in two ways:

i. Using group policy (recommended) –

NOTE: For this example, I will assume we are using a domain level policy. The same method applies for policies at the Domain Controllers OU level, or any other.

1. Open the policy for editing using GPMC, AGPM, or Active Directory Users and Computers (whichever method you use typically)

2. Expand Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options

3. Double click the “Network security: LAN Manager authentication level” setting and change it to the desired value

4. Allow time for replication (or force replication) if necessary

5. DON’T FORGET TO UPDATE YOUR POLICY! (gpupdate /force)

ii. In the registry (this may be overwritten by group policy settings) -

1. HKLM\SYSTEM\CurrentControlSet\Control\Lsa

2. Double click the LMCompatibilityLevel registry value

3. Set the value to the desired setting (as described in the above reference table)

3. Time difference between the source and target is greater than 30 minutes (NTLMv2 only)

a. Ensure the source and target are within a 30 minute time skew

i. Synchronize time if necessary: w32tm /resync

4. Secure channel may be broken

a. Reset the secure channel (nltest /sc_reset:<domainname>

Some potential causes for this error are:

1. Available physical memory exhaustion

2. Paged pool or non-paged pool memory exhaustion

3. Free System PTE (Page Table Entries) exhaustion

To troubleshoot this issue, use Performance Monitor, Resource Monitor, Xperf, or other performance diagnostics tool.

This issue can be difficult to track down. Some of the potential causes are:

1. Verify Scalable Networking Pack (SNP) features are disabled in the registry as well as in the driver/hardware settings

a. The driver level settings must be disabled via the configuration for the NIC itself (NOTE: Support for this action comes from the hardware vendor)

i. Access the NIC properties

ii. Click the Configure button next to the network adapter name

iii. Click the Advanced tab

1. Identify the setting for the chimney offload (typically called “Large Send Offload” or “TCP Offload Engine”) and set it to disabled

2. Identify the setting for Receive Side Scaling and set it to disabled

3. Identify the setting for TCPA or NetDMA and set it to disabled

b. This can be disabled in the registry at HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

i. EnableTCPChimney – this value enables and disables the TCP Chimney Offload feature (0 = disabled; 1 = enabled)

ii. EnableRSS – this value enables and disables Receive Side Scaling (0 = disabled; 1 = enabled)

iii. EnableTCPA – this value enables and disables NetDMA (TCPA) functions (0 = disabled; 1 = enabled)

2. RPC ephemeral port closures or limitations

a. Validate RPC related ports are open using portquery or another similar tool

3. 3^rd party antivirus or endpoint protection product may be blocking ephemeral communications

a. Try uninstalling the antivirus or endpoint protection product to see if it changes the behavior

4. Network issue causing packet timeouts/drops (bad switch port, router, etc)

5. RPC bind time negotiation failure

a. Disable bind time negotiation per http://support.microsoft.com/kb/899148

6. No network path exists to the target domain controller or machine

7. Verify RPC interface restrictions are not in place, and if they are, that they are compatible (see: http://technet.microsoft.com/en-us/library/cc781010(WS.10).aspx)

8. LMCompatibilityLevel may be incompatible between the source and target

a. LMCompatibilityLevel must be at a level where authentication can be negotiated between the source and target (whether that is LM, NTLM, or NTLMv2). For example, a setting of 0 on the client and 5 on a domain controller or target server will result in an inability to negotiate a valid authentication mechanism.

b. This must be reviewed on the source/sender and the target/receiver.

iii. You can find the current setting by looking in the registry at HKLM\SYSTEM\CurrentControlSet\Control\Lsa. The value is named LMCompatibilityLevel (if by chance you are still REALLY old school and are running Win9x, the value is named LMCompatibility).

iv. Valid values are 0 – 5

c. Reference table of the settings:

d. The settings, if they are incompatible, can be configured in two ways:

v. Using group policy (recommended) –

NOTE: For this example, I will assume we are using a domain level policy. The same method applies for policies at the Domain Controllers OU level, or any other.

1. Open the policy for editing using GPMC, AGPM, or Active Directory Users and Computers (whichever method you use typically)

2. Expand Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options

3. Double click the “Network security: LAN Manager authentication level” setting and change it to the desired value

4. Allow time for replication (or force replication) if necessary

5. DON’T FORGET TO UPDATE YOUR POLICY! (gpupdate /force)

vi. In the registry (this may be overwritten by group policy settings) -

1. HKLM\SYSTEM\CurrentControlSet\Control\Lsa

2. Double click the LMCompatibilityLevel registry value

3. Set the value to the desired setting (as described in the above reference table)

9. SMB signing settings may be incompatible

a. If a SMB connection is being made, SMB signing options must be compatible

i. If you require SMB signing on the target, yet have it disabled on the source, then connectivity will be affected (and vice versa).

b. You can validate SMB signing options in the registry at:

i. HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters

1. EnableSecuritySignature – this value defines whether SMB signing can be used and corresponds to the group policy setting “Microsoft network client: Digitally sign communications (if server agrees)”

2. RequireSecuritySignature – this value defines whether SMB signing is required and corresponds to the group policy setting “Microsoft network client: Digitally sign communications (always)”

ii. HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters

1. EnableSecuritySignature – this value defines whether SMB signing can be used and corresponds to the group policy setting “Microsoft network server: Digitally sign communications (if client agrees)”

2. RequireSecuritySignature – this value defines whether SMB signing is required and corresponds to the group policy setting “Microsoft network server: Digitally sign communications (always)”

c. If you need to make a correction to the settings, there are two methods:

i. Using group policy (recommended) –

1. Open the policy for editing using GPMC, AGPM, or Active Directory Users and Computers (whichever method you use typically)

2. Expand Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options

3. Double click the “Microsoft network client: Digitally sign communications (if server agrees)” setting and change it to the desired value

4. Double click the “Microsoft network client: Digitally sign communications (always)” setting and change it to the desired value

5. Double click the “Microsoft network server: Digitally sign communications (if client agrees)” setting and change it to the desired value

6. Double click the “Microsoft network server: Digitally sign communications (always)” setting and change it to the desired value

7. Allow time for replication (or force replication) if necessary

8. DON’T FORGET TO UPDATE YOUR POLICY! (gpupdate /force)

ii. Using the registry (may be overwritten by group policy settings) –

1. Browse to HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters

2. Double click the EnableSecuritySignature registry value and set the value to the desired setting (0 = disabled; 1=enabled)

3. Double click the RequireSecuritySignature registry value and set the value to the desired setting (0 = disabled; 1=enabled)

4. Browse to HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters

5. Double click the EnableSecuritySignature registry value and set the value to the desired setting (0 = disabled; 1=enabled)

6. Double click the RequireSecuritySignature registry value and set the value to the desired setting (0 = disabled; 1=enabled)

10. NIC drivers may need to be updated

This error code is typically reported in errors as “Not enough storage is available to process the command”. This has the side effect, based on the friendly error description, of sending people down the wrong path in troubleshooting. Don’t fear, this error description does not mean you are low on disk space J It means you probably have memory or port contention issues, hooray!

Some potential causes of this error are:

1. Domain controller, client, or target server may have exhausted virtual memory/page file or physical memory

a. Check your available memory – if it’s extremely low then you may need to consider adding more RAM and identifying the offending process

NOTE: For busy x64 domain controllers, if you have a large ntds.dit (Active Directory database), but have less RAM than the size of the dit file, you may experience performance degradation and high lsass utilization (memory and processor) that could lead to this error. If you have busy x86 domain controllers that are running and peak memory utilization, it may be time to consider upgrading to a 64 bit version of Windows to prevent excessive trimming. Remember, lsass caches search results it returns which adds to its memory utilization! For 64 bit versions of Windows, I recommend Windows Server 2008, Windows Server 2008 R2, or better yet, Windows Server 2012 J

b. Check your page file usage with Performance Monitor – if it’s extremely high at most times (90%+) consider increasing the page file size or adding more RAM

c. Look for handle leaks with Performance Monitor, Resource Monitor, or Task Manager

2. User ports may be exhausted (see: http://support.microsoft.com/kb/196271)

Some of the more common, but self-explanatory errors are listed below:

Many posts that can be found on the new features of directory services on Windows 2012 both from Microsoft and from others. However, you need to get your hands on the features to learn them.

Like other PFE that post on this blog I get to talk to you when I am on site. One item that I find we all can improve on is having a personal lab to learn. While you may have access to controlled labs that mimic your production environment, you will be limited on what you can introduce to a lab supporting your production environment. Very few engineers I meet have a personal lab setup because they don’t have time or who knows why. I am sure the reason varies.

With availability of Hyper-V on Windows 8 you have no excuses. You will need licenses for your guests of course (check with your company or look at a personal TechNet license).  So, for the purposes of learning Windows Server 2012 and what is new with Directory Services we can build a lab to meet most of our needs.

So in this short post, I am going to share what my lab looks like. This lab is also the lab we built for our Windows Server 2012 Directory Services training classes (I also have a lab for 2008R2).

1) Forest 1 – Windows 2008 R2 Functional Level

a. Windows 2008 R2 Core domain controller (1)

b. Windows Server 2012 member servers - full (2)

c. Windows 2012 core member server (1)

d. Windows 7 Client (1)

e. Windows 8 Client (1)

2) Forest 2 – Windows 2012 Forest Functional Level

a. Windows 2012 Core domain controller (1)

b. Windows 2008 R2 Member Server – full (1)

       i. Useful for a GUI for those light on PowerShell

My lab machine is my laptop with 16GB RAM, 4 core Intel I7 processor, and an SSD drive for speed. I can run all machines in the lap and they are very responsive to any lab that I have done thus far. My previous machine had 8 GB of RAM with a SATA drive and I ran most any lab I needed for 2008 R2 on that machine as well.

The lab above gives us 8 virtual machines (about 50-70GB disk space) that provide a multi-forest lab to test out features in Windows 2012. What am I testing? Simple. Go to any post listing the new features for directory services on Windows Server 2012, and I have probably done it in this lab.

Here is a link with new features in many of the 2012 products:
http://technet.microsoft.com/en-us/library/hh831769.aspx

Need a deep dive on the new features for Directory Services?
http://technet.microsoft.com/en-us/library/hh831484.aspx

Your first lab assignment is to build the lab. You can do this one of two ways.

1) Build the entire lab setup above

or

2) Just build Forest 1 – you can add forest 2 later when you need it for a lab the traverses a forest trust of some kind.

So building this lab doesn’t really count as a lab does it? I will answer that for you: No it doesn’t. Your first real lab will be upgrading forest 1 to a 2012 forest, demoting the 2008 R2 domain controller, and raising the functional level to 2012.

Hint: Before you start upgrading your lab, export your VMs so you can return to the initial state and do it again.

Need help building the lab? Some searches with Bing will give you plenty of help. Here are a few

A Global Enterprise … in your basement?
http://blogs.technet.com/b/askpfeplat/archive/2012/06/04/a-global-enterprise-in-your-basement-lab.aspx

Introducing the first Windows Server 2012 Domain Controller (Part 1 of 2)http://blogs.technet.com/b/askpfeplat/archive/2012/09/03/introducing-the-first-windows-server-2012-domain-controller.aspx

Windows Servers 2012 Test Lab Guides – this has guides for many different services
http://social.technet.microsoft.com/wiki/contents/articles/7807.windows-server-2012-test-lab-guides.aspx

Don’t know what to with the lab once it is built? Look at some previous posts on AskPFEplat for some lab assignments.

So, next time you see me and I ask if you have a lab for yourself, hopefully you will say yes and can show it to me. There are no excuses, so get motivated and build a personal lab you can practice and learn with.

Your friendly PFE – Doug Gabbard

KMS Activation

I still receive TONS of questions on KMS even though it has been around for quite some time now. It’s fairly easy to ramp up on. I can generally bring an admin up to speed in under an hour, if they don’t want to read the documentation (located on TechNet: http://technet.microsoft.com/library/ff793409.aspx ).

It’s not very complicated. It’s easy to setup. Just very different from Windows Server 2003. So a bit of a fresher on KMS. Don’t worry. I won’t bore you with too many details as there’s a ton of good information out there on volume activation using KMS.

So what is KMS?

KMS is a service that activates volume license versions of Windows Vista and later as well as Office 2010 and later. Since I’m not an “Office” person, I’ll focus on the Windows side of things. But if you’re curious about Office 2013, look here: http://technet.microsoft.com/en-us/library/ee624357.aspx

In order to activate client operating systems, it requires a count of 25 or server operating systems require a count of 5. These can be any combination of client or server operating systems. By count, we mean that this number of unique KMS clients had to have contacted the KMS host prior to the KMS host activating all KMS clients. Activation lasts for 180 days and attempts to renew with the KMS host every 7 days by default.

To setup the KMS host, we use the command line interface slmgr.vbs to install the KMS host key. The KMS host can be cohosted on a VM or physical server of standalone by itself. You can have one or many. If DDNS is enabled, the KMS host automatically creates an SRV record in DNS so that KMS clients can locate a KMS host and activate against it. Here’s a demo that shows how to do this: http://technet.microsoft.com/en-us/windows/ff716620.aspx?ITPID=flpbook

Easy enough.

Very little has changed for Windows 8 and Windows Server 2012. However, we added a GUI. Prior to Windows 8 and Windows Server 2012.  For those of you that have KMS hosts setup to support earlier versions of the operating system, you can still use these to activate Windows 8 and Windows Server 2008 R2 as long as the KMS host is running on a Windows 2008 R2 or later operating system. It does require installing an update mentioned in the following article:

http://support.microsoft.com/kb/2757817

Afterwards, you then need to install the Windows Server 2012 volume license key and activate it.  This key will activate Windows Server 2012, Windows 8, and client and server operating systems all the way down to Windows Vista and Windows Server 2008.

Now for the new stuff.

Active Directory-Based Activation

With Windows 8 and Windows Server 2012, we also introduced something better.

It is called Active Directory-Based Activation.

It only works with Windows 8, Windows Server 2012, and later and it is forest wide. So for Windows 7/2008 R2 and earlier, you’ll still need to maintain those old KMS hosts.

You do not need to have your forest and functional levels at 2012, but you must have updated the schema to support these operating systems using ADPREP. ADPREP is still located on the Windows media if you plan on running it from one of the existing DCs in the environment.

1)      If you haven’t already done so, run ADPREP from the Windows Server 2012 media to update the schema to support Active Directory-Based Activation.

Note: Make sure you be cognizant and cautious as with any schema update.

2)      On a Windows Server 2012 machine, install the Volume Activation Services Role

3)      After the role has installed, from Server Manager, select Tools, and then select Volume Activation Tools.

4)      In the wizard, Active Directory-Based Activation

5)      Enter your KMS host volume license key for Windows Server 2012. You’ll forgive me for not showing my right? :-)

You can optionally choose to enter a display name for the AD object you will be creating.

By default, the name is by default the Activation Object is named Windows® Operating System, Volume_KMS_Channel. I chose to entire in a unique object name for my demo.

6)      Complete the wizard, but make sure to read the dialog. There’s a trick one at the end.

Click close on the Activation Succeeded window instead of Next. Last thing you want to do is delete the AD object you just created (although it does have a safety precaution of requiring you to check the box).

7)     The volume license key must be activated before the domain and clients can be activated. You can do this from the GUI or from the old slmgr.vbs command line.

From here on, all volume licensed versions of Windows 8 and Windows Server 2012 will be activated as soon as they join the domain.

Once you’re activated, if you run slmgr.vbs –dlv, you’ll see the following:

The Application Event log will show the activation event:

Using ADSI, you can view the AD object.

Multiple activations can be listed here. If you have both client and server SKUs, you'll have two activation objects. As long as the server object is available, the client can be safely deleted as the server object will activate both clients and servers.

These objects can be manually deleted using ADSI, but the preferred method is to use Volume Activation Tools.

To do so, go back into the same wizard and select the radio button to Skip to Configuration.

Simply check the Delete checkbox and click on Commit.

­Other Details

Activations still last for 180 days.  When a re-activation event, the client will query AD for the Activation Object. 

Since AD-Based Activation uses AD, we use LDAP instead of the RPC 1688 tcp port used with KMS.

In the event that the Active Directory object is unreachable, clients will attempt to use the next available activation method which is the KMS activation method. This means if the AD object is unreachable, the client will go check DNS for an SRV record for a KMS host.

If you unjoin a client from the domain, activation will fail on the next license evaluation. This typically occurs when a system is rebooted or the Software Protection Service is restarted. Side note: Don’t disable this service. I’ve seen too many instances of that. It leads to wonky behavior.

Enjoy!

Charity “AD Activation Makes Activation Even Easier” Shelbourne

Some have noticed that my posts over the last few months have a focus on Windows Server 2012 storage features.  (In case you're new to the Ask PFE Platforms blog, I'll include links to those at the end of this post.)  On a typical day I'm not obsessed with storage.  However, I just couldn’t help but blog about something new I saw and used over the holidays: Windows Azure Online Backup.  This great new online backup tool and service is a cool example of how easy it is to leverage cloud storage with Windows Server 2012.  Cloud based storage isn't just for your phone, your tablet, or your desktop...it is useful for servers, too.  As of this writing, there is a free six month preview of Windows Azure Online Backup available that provides you with 300GB of available secure online storage space for backups.  This little gem came in quite handy when I needed to do a quick backup of data on my main server in my basement lab prior to a significant reconfiguration.

Windows Azure Online Backup

With this service you can perform secure file and folder backups of Windows Server 2012.  You do need a broadband connection, to register for the trial, and to download and install the agent.  The allocated space may be used for more than one server.  You can even establish a backup schedule.   Coupled with Data Protection Manager in System Center 2012, you can backup datacenter servers to the cloud.  Using this method it’s not just for files and folders, it’s also for virtual machine backups as well.   DPM local backups remain quicker and faster, but Azure Online Backup is attractive as offsite secure storage.  And right now during the trial, free is cheap.

This tool was incredibly easy to configure, uses block based incremental backups for efficiency, compresses what it backs up and uploads.  The data uploaded is encrypted…and get this: all without saturating the network.   How long is the data retained?  That’s up to you because it is user configurable.

How could you try this out?

The answer to that is not only simple, I’ll walk you through it with the steps below.

1. Use the following link to create an Azure online account ID if you don’t already have one.

https://activedirectory.windowsazure.com/Signup/MainSignUp.aspx?OfferId=6061479A-02FA-46f0-9DA0-244298C3CD7B&ali=1

2. Once signed into the Windows Azure Online Backup portal, you may download the agent or even watch a video tutorial.  The account you create using your company name will be something like yourcompany.onmicrosoft.com.   Behind the scenes, so to speak, is a mini-active directory structure for your account and any others you may wish to create.   Notice on the overview page of the portal in the lower left there is a link for a Windows Azure Active Directory portal.  That is where you can setup additional accounts affiliated with your backup portal for other users that may be performing or accessing these backups.  There you will find an additional option to sync with an external Active Directory.  Be certain you fully understand what that entails.  My guess is that you won’t need that at all for the purposes of doing a quick backup or evaluating this backup service.  You can also examine the health of the online backup solution by viewing worldwide backup services by region in case you would like to know about any current outages, stability history, or upcoming maintenance.

Figure 1: The Windows Azure Online Backup Portal

Figure 2: Service Health

3. Once you install the backup agent, it should prompt you to install KB 2779569 to obtain updates to the backup agent.

4. When you’ve installed the update and run the backup agent, you will be prompted for proxy information and optional credentials if needed for accessing the Internet.

Figure 3: Register Server Wizard

5. This next step is very important.  You must specify a passphrase for encryption to protect the confidentiality of the backup.   You may save a local copy of this passphrase…which I encourage you to do.  Why?  Because you can’t call Microsoft later to get access to your data if you forget or lose this passphrase. You will need this passphrase later in order to restore data.

Figure 4: Encryption Settings

6. Once you’ve provided the passphrase successfully, you must enter the credentials for the Windows Azure Online ID you created earlier.  This will be something like username@yourcompany.onmicrosoft.com.

Figure 5: Account Credentials

7. To perform a backup, you must first walk through the scheduling wizard to choose what will be backed up.  This agent backs up data and is not a direct replacement for system state or Bare Metal Restore (BMR) type backups.   I suppose one could backup a volume that contained some of these backup files though.  :)

Figure 6: Backup Scheduling Wizard

Figure 7: Item Selection

Figure 8: Backup Time Schedule

Figure 9: Setting Retention Policy

Figure 10: Review Your Choices!

8. At this point, you can make changes to the backup schedule, initiate a backup now, or wait for the backup to start based on the schedule chosen.  To delete a backup schedule, use the option to change backup schedule and you’ll find the option there to delete.  The example below is of a running backup.  You can double click on a completed backup or a backup in progress to see details.  Notice that 17GB of data compressed down to approximately 6GB.

Figure 11: Completed Backup in Online Backup Agent Tool

Figure 12: Backup Status

9. Login to the online portal and see how much space was used.

Figure 13a: Backup Status In Online Portal

After installing the Windows Azure Online Backup Agent, you may also notice that the next time you launch Windows Server Backup in Windows Server 2012, it now contains an option for online backups.

Figure 13b: Online Backup Option in Windows Server Backup

Concluding Thoughts

Windows Azure Online Backup leverages VSS, so after taking the snapshot of the filesystem, there was no interruption of applications required for the backup to complete.  Further, during my testing there was no adverse impact to my internet connection due to the ability to adjust how much bandwidth the agent consumed.   Once the snapshot was taken, for the agent it was just a matter of encrypting the data in to a data bundle and uploading to the secure space within the Windows Azure cloud.   Just remember, you’ll need the created passphrase in order to restore the data.  Protect the passphrase just like the key to your car, boat, or anything else you don’t want others to have.  Incidentally, for those of you that love using PowerShell, the Windows Azure Online Backup Agent comes with cmdlets.   Check out all of these below that say MSOnlineBackup.  Quite a few!

Figure 14: PowerShell Cmdlet List for Windows Azure Online Backup Agent

For More Information About Windows Azure Online Backup

Windows Azure Online Backup

http://technet.microsoft.com/library/hh831419.aspx

Command Line Installation Instructions

http://technet.microsoft.com/en-us/library/hh831761.aspx#BKMK_installagent

Martin's Prior Posts on Windows Server 2012 Storage Features

Resilient File System (ReFS)

http://blogs.technet.com/b/askpfeplat/archive/2013/01/02/windows-server-2012-does-refs-replace-ntfs-when-should-i-use-it.aspx

How to Import a Storage Pool

http://blogs.technet.com/b/askpfeplat/archive/2012/12/24/windows-server-2012-how-to-import-a-storage-pool-on-another-server.aspx

Windows Server 2012 Built-in iSCSI Target

http://blogs.technet.com/b/askpfeplat/archive/2012/10/22/windows-server-2012-iscsi-target-is-it-the-same-as-the-downloadable-target-for-windows-server-2008-r2.aspx

Storage Spaces

http://blogs.technet.com/b/askpfeplat/archive/2012/10/10/windows-server-2012-storage-spaces-is-it-for-you-could-be.aspx