Stephen Mathews here to talk to you about generating Group Policy Object reports quickly and efficiently. I had a customer ask me if there was an easy way to find all the RunOnce GUIDs in their GPOs. The customer was concerned that they would generate duplicate GUIDs in their deployment of Group Policy Preferences. Yes, there is a chance this could happen… after about 70k years. For a fascinating look into GUID randomness, please check out
GUID guide, part three.

However, the customer’s fears were not unwarranted, because there is still a very real threat of duplicate GUIDs while copying GPOs and GPPs. This problem is documented here Ein Mal und nie wieder – GP Preferences “Apply once” Option. Since I can’t read German, I’m going to give my explanation. The GUIDs are generated automatically when the setting “Apply once and do not reapply” is used inside GPP. All the RunOnce GUIDs are stored in their corresponding Client Side Extension XML files which are nested inside their respective GPO in the SYSVOL\POLICIES share. After Group Policy executes on the target machine the GUID is populated under the “<HIVE>\Software\Microsoft\Group Policy\Client\RunOnce” registry key. A client machine will only have the RunOnce GUIDs from the GPOs that it applies – it will not have them for out-of-scope or filtered policies. So you’d have to parse all the XML files in the SYSVOL\POLICES share – unfortunately the individual GPP XMLs don’t have a parent GPO reference inside them. This means you’ll also have to parse the file path to pull out the GPO GUID and then use a Group Policy or Active Directory call to turn it into a human readable name.

Luckily the PowerShell Group Policy module does the hard work for you. By using the aptly name Get-GPOReport cmdlet, you can output an entire GPO report that includes all of its CSE XMLs into one either XML or HTML file. The cmdlet by itself will output the report directly into the shell in string format, which you can capture inside a variable or you can cast it directly to an object. For this project I chose to leave them as strings since I’ll be parsing them and I can cast the string to an XML object as needed.

$GPOReportStr = Get-GPOReport -Name <NAME> -ReportType Xml
[xml]$GPOReportXML = Get-GPOReport -Name <NAME> -ReportType Xml

You can also generate the report directly from a GPO object using the GenerateReport method.

$GPOReport = (Get-GPO -Name <NAME>).GenerateReport(“XML”)

Here’s the simplest way to collect all the reports.

$GPOReports = Get-GPOReport -All -ReportType Xml

Here’s two other options for serially collecting the reports, first with the cmdlet and second with the method. The method will be faster because the object is already created from Get-GPO, the cmdlet will create another object which requires additional processing.

$GPOReports = @()
Get-GPO -All |
Foreach-Object {
$GPOReports += (Get-GPOReport -Guid $_.Id -ReportType Xml)
}
$GPOReports = @()
Get-GPO -All |
Foreach-Object {
 $GPOReports += $_.GenerateReport(“XML”)
}

Due to the sheer number of GPOs, processing these serially can take an unacceptable amount of time. We can counteract this time requirement by using a multithreading technique, such as Start-Job, Workflows, and Runspaces. This Parallel processing with PowerShell article compares multithreading techniques as well. I’ll show each one of these, starting with Start-Job.

***WARNING – This exhausted my system’s memory and required a restart. See screenshot.***

$GPOReports = @()
foreach ($GPO in (Get-GPO -All)) {
Start-Job -Name $GPO.DisplayName -ScriptBlock {
Param ($GPO)
Get-GPOReport -Guid $GPO.Id -ReportType Xml
} -Argumentlist $GPO
}
$GPOReports = Get-Job | Wait-Job | Receive-Job

The limitation with Start-Job is the overhead required for the job to be performed, which increases the individual processing time. This becomes a net gain by running the jobs concurrently, although you can only realize the time savings if the job finishes! This did not occur in my tests; I consistently ran out of memory before they completed. While you can counteract the memory exhaustion with throttling, it wouldn’t be a fair comparison to the Workflows and Runspaces, which I did not throttle either.

The next technique uses PowerShell Workflows.

$GPOReports = @()
$GPOs = Get-GPO -All
workflow GPOReports {
param ($GPOs)
foreach -parallel ($GPO in $GPOs) {
 Get-GPOReport -Guid $GPO.Id -ReportType Xml
}
}
$GPOReports = [string[]](GPOReports -GPOs $GPOs) 

While the Workflows completed successfully and were much faster than the serial processing techniques. I just had to compare it to PowerShell Runspaces. This was my first foray into the subject, so please forgive my oversight. Boe Prox has written about them extensively, please check out his 4-part Hey, Scripting Guy! blog series called Beginning Use of PowerShell Runspaces.

$GPOs = Get-GPO -All
$GPOReports = @()
$Invokes = @()
$Instances = @()
$RunspacePool = [runspacefactory]::CreateRunspacePool(1,$GPOs.Count)
$RunspacePool.Open()
foreach ($GPO in $GPOs) {
$ScriptBlock = {
 Param ($GPO)
   Try {Get-GPOReport -Guid $GPO.Id -ReportType Xml -ErrorAction Stop}
 Catch {$_}
}
$Instance = [powershell]::Create()
 $Instance.RunspacePool = $RunspacePool
$Instance.AddScript($ScriptBlock).AddArgument($GPO) | Out-Null
 $Invoke = $Instance.BeginInvoke()
 $Invokes += $Invoke
 $Instances += $Instance
}
do {
$Invokes |
 Where-Object {$_} |
 ForEach-Object {
 $i = $Invokes.IndexOf($_)
 If ($Invokes[$i].IsCompleted) {
 $GPOReports += $Instances[$i].EndInvoke($Invokes[$i])
           $Instances[$i].Dispose()
 $Instances[$i] = $null
 $Invokes[$i] = $null
}
}
} until (($Invokes.IsCompleted -notcontains $false) -and ($Invokes.IsCompleted.Count -le 0))
$RunspacePool.Close()

Here’s my results, after retrieving over 500 GPO reports using the above methods; you can see Runspaces were the clear winner.

Now that we have all the reports stored in a variable, I can answer my customer’s question. I will create a hashtable that has the GPO name with the value set to all of its RunOnce GUIDs. I find these by parsing the report string for “FilterRunOnce” and then do some string manipulation using the Split() method.

$RunOnceIds = @{}
$GPOReports |
Where-Object {$_ -match “filterrunonce”} |
 ForEach-Object {
 $Name = ([xml]$_).GPO.Name
 [string[]]$IDs=$_.Split(“`n”) |
 Select-String “filterrunonce” |
 ForEach-Object {
              $_.ToString().Split(‘{}’) | Select-Object -Skip 1 -First 1
}
$RunOnceIds.Add($Name,$Ids)
}
$Values = [string[]]$RunOnceIds.Values.Split() | Sort-Object
$Values
$Values |
Group-Object |
Where-Object {$_.Count -gt 1} |
 Select-Object -Property @{Name=“Duplicate”;Expression={$_.Name}}

As you can see, there are indeed duplicate GUIDs. Now to find the offending GPOs.

$RunOnceIds.GetEnumerator() | Where-Object {$_.Value.Contains(<GUID>)}

Fortunately, you can regenerate a new GUID by reapplying the “Apply once and do not reapply” setting. Another note about RunOnce, it populates the registry even if the preference fails to execute. For example, if a GPP sets a registry key value from 0 to 1, but the registry security permissions denied the change, the RunOnce entry is still tattooed in the registry and the GPO won’t attempt to change it again. So you can reset the GUID to reapply your RunOnce settings again without recreating them. And finally since you have all your RunOnce GUIDs (from $Values variable above), you can actually cleanup the entries inside your client machines’ Group Policy RunOnce keys if you’re so inclined.

I hope you enjoyed our foray into the different methods of collecting GPO reports and using the different multithreading techniques inside PowerShell. I’m now a Runspace believer and can’t wait to use them again. Until next time, thanks for reading!

Disclaimer: The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

Hello, Paul Bergson back again with a discussion on the upcoming changes to our monthly patch releases to align down-level supported operating systems, updating practices to coincide with the Windows 10 Service Model. This includes Windows 7/8/8.1 and Windows Server 2008 R2/2012/2012R2.

“From October 2016 onwards, Windows will release a single Monthly Rollup that addresses both security issues and reliability issues in a single update. The Monthly Rollup will be published to Windows Update (WU), WSUS, SCCM, and the Microsoft Update Catalog. Each month’s rollup will supersede the previous month’s rollup, so there will always be only one update required for your Windows PCs to get current.”
https://blogs.technet.microsoft.com/windowsitpro/2016/08/15/further-simplifying-servicing-model-for-windows-7-and-windows-8-1/

Beginning in October 2016 onwards,
don’t expect to see individual KB’s but instead expect to see the following in the monthly patch release cycle:

1. Security-Only Update
   
   1. Collects all of the security patches for that month into a single update
      
2. Monthly Rollup 
   1. Security updates from previous bullet point
      
   2. All updates, rollups, patches, and security updates for that month
      
   3. Proactive addition of previously shipped patches (within 6-8 months the monthly rollup will include all patches back to the last baseline and will be fully cumulative)
3. .Net Framework Security-Only Update
   
   1. Contains only security updates
      
4. .Net Framework Rollup *1
   
   1. .Net Framework Security Updates from Previous Bullet Point
      
   2. Reliability updates
      

This change brings up a key question:

“With the new Windows as a Service: Service Model, can we back out a single patch (KB) if it causes issues since they are all rolled up?”

The short answer is “No”, you can’t control which KB’s can be applied, so the complete roll up would need to be backed out. But the answer is more complex than a simple no.

The point of rollups is to correct the fragmentation caused by systems containing a mix of individual updates. It will not be possible to uninstall specific KB’s of a rollup. If there is a problem the partner will need to open up a case and provide business justification to drive the discussion with Microsoft.

“Historically, we have released individual patches for these platforms, which allowed you to be selective with the updates you deployed. This resulted in fragmentation where different PCs could have a different set of updates installed leading to multiple potential problems;”
*2

Typical customer patch fragmented operating system

Microsoft pre-release patch test environment (Fully patched)

“Each month’s rollup will supersede the previous month’s rollup, so there will always be only one update required for your Windows PCs to get current”
*2

A few details as to why the decision was made by Microsoft to follow the new servicing model.

“Traditionally, new versions of Windows have been released every few years. The deployment of those new versions within an organization would then become a project, either by leveraging a “wipe and load” process to deploy the new operating system version to existing computers, or by migrating to the new operating system version as part of the hardware replacement cycle. Either way, a significant amount of time and effort was required to complete these tasks.

With Windows 10, a new model is being adopted. This new model, referred to as “Windows as a service,” requires organizations to rethink how they deploy and upgrade Windows. It is no longer a project that happens every few years, it is a continual process.” *3

“Instead of new features being added only in new releases that happen every few years, the goal of Windows as a service is to continually provide new capabilities. New features are provided or updated two to three times per year, while maintaining a high level of hardware and application compatibility.

This new model uses simpler deployment methods, reducing the overall amount of effort required for Windows servicing. By combining these simpler methods (such as in-place upgrade) with new techniques to deploy upgrades in phases to existing devices, the effort that used to be performed as part of a traditional deployment project is spread across a broad period of time.” *4

Windows 10 servicing model terms.

Windows 10 release types and cadences

Although Microsoft releases flight builds to Windows Insiders, Microsoft will publish two types of Windows 10 releases broadly to the public on an ongoing basis:

• Feature updates that install the latest new features, experiences, and capabilities on devices that are already running Windows 10. Because feature updates contain an entire copy of Windows, they are also what customers use to install Windows 10 on existing devices running Windows 7 or Windows 8.1, and on new devices where no operating system is installed.
  
• Quality updates that focus on the installation of security patches and other important updates. Microsoft expects to publish an average of two to three new feature updates per year, and to publish quality updates as needed for any feature updates that are still in support. Microsoft will continue publishing servicing updates on Update Tuesday (sometimes referred to as Patch Tuesday). Additionally, Microsoft may publish additional quality updates for Windows 10 outside the Update Tuesday process when required to address customer needs.
  

The cumulative nature of all Windows 10 releases It is important to note that, in order to improve release quality and simplify deployments, all new releases that Microsoft publishes for Windows 10 will be cumulative. This means new feature upgrades and servicing updates will contain the payloads of all previous releases (in an optimized form to reduce storage and networking requirements), and installing the release on a device will bring it completely up to date. Also, unlike earlier versions of Windows, you cannot install a subset of the contents of a Windows 10 quality update. For example, if a quality update contains patches for three security vulnerabilities and one reliability issue, deploying the update will result in the installation of all four patches. *5

Note 1: There will continue to be a ratings rollup. Just like how we rate cumulative updates in Win10, if there is a new security patch in the rollup and the security patch is critical, then the entire rollup is marked as critical.  Since both the security-only update and the monthly rollup will contain the same new security patches each month they will both also have the same security ratings each month.  The customer can choose whichever one they prefer to deploy to stay compliant.

Note 2: For slow bandwidth sites, WSUS can be configured for Express installation files, but requires additional bandwidth on the WSUS side, but minimizes downloads on the client side.  See Express Install Files section in this link: https://technet.microsoft.com/en-us/library/cc708456(v=ws.10)

1. https://blogs.msdn.microsoft.com/dotnet/2016/08/15/introducing-the-net-framework-monthly-rollup/
   
2. https://blogs.technet.microsoft.com/windowsitpro/2016/08/15/further-simplifying-servicing-model-for-windows-7-and-windows-8-1/
   
3. https://technet.microsoft.com/en-us/security/gg309155
   
4. https://technet.microsoft.com/en-us/itpro/windows/plan/windows-10-servicing-options
   
5. https://technet.microsoft.com/en-us/itpro/windows/manage/introduction-to-windows-10-servicing
   

Hopefully this information has helped and we encourage you to read all the linked documentation and share it with your staff as you prepare for the upcoming changes.

UPDATE (9/26/2016 @ 10:45a ET): Thank you all for your participation in this AskPFEPlat Ask Me Anything Q&A blog session! We are ending active questioning at this time. Rest assured, if you have asked us questions and have not yet gotten a response, we are still working on it. Additionally, we would like to request feedback on whether you, our readers, would like to see more posts like this, and maybe even have some live sessions with members of the AskPFEPlat blog team in the future (free of charge)! Please let us know in the comments or via email! Thank you for your continued reading!

Hi everyone! Brandon Wilson here with you today with the entire AskPFEPlat team, not to talk with you about a specific subject, but to open the floor to questions for our outstanding audience across the globe. We write and write, and we try our best to answer comments and questions as we go along, but there are always those stragglers and those questions you’ve wanted to ask, but just haven’t yet…

Well good news, because today’s posting is for you! It’s for you to ask us your questions that you have related to the topics covered on our blog (Azure, Active Directory, core OS, performance, networking, clustering, DNS, system center, etc.). So whether that’s “Hey what is this adminSDholder thing I heard about?”, “We have a server that keeps hanging, what should I look at?”, “Why should I consider moving to Azure?”, or anything beyond or in between, ask away!

Please feel free to ask your questions in the comments so others may share any knowledge gained, or if you’d like, you can send an email to us to ask your question.

Thank you for participating, and we are all looking forward to your questions!

-The AskPFEPlat team!

Happy Monday folks and what a great Monday it is. Microsoft Ignite is taking place this week. I’m super excited for all the announcements that will take place this week and the great things ahead. For those of you that are unable to attend in person, check out the live streaming online at https://ignite.microsoft.com/

I’m currently watching Scott Guthrie announce Windows Defender Application Guard and Ann Johnson demo this new security feature.

As a bonus, we also released the Windows Server 2016 evaluation edition today. Download it here: https://www.microsoft.com/en-us/evalcenter/evaluate-windows-server-2016

Enjoy!

Charity “Windows Server 2016 is here!!” Shelbourne

Hello, Paul Bergson back again. I have been on the road a bit more than normal doing security training/POC deliveries (POP-SLAM *1) for our customers related to Pass-the-Hash and credential protection. I have noticed an alarming trend in how credential protection is thought to resolve a customer’s credentials from being compromised. Enterprises that are investing in vaulting software, and not ensuring the users of this vault have workstations that are isolated from internet and e-mail, are being lulled into a false sense of security!

Credential randomization and vaulting software has begun to expand; this is a great step as enterprises move to protect their assets from exposure but accessing the vault, from an insecure workstation, bypasses the protective steps taken to secure these credentials. “Securing privileged access is a critical first step to establishing security assurances for business assets in a modern organization.” *2

By randomizing passwords, the task for an administrator to use these credentials requires them to open the vault and check them out.  As soon as an insecure workstation connects to the vault any of the credentials retrieved can no longer have their integrity assured.  Making matters worse, I have seen administrators want to reduce their trips to the vault by foreseeing possible future activity and copying ALL their privileged accounts to their desktop at the start of their day and pasting them in the clear to an open application such as Notepad.  Capturing pasted credentials to the clipboard or an application is trivial on a compromised workstation.

If an enterprise allows administrators to use their workstation for both unprivileged activities that have public e-mail & internet browsing available as well as remote administration they have NOT increased their credential protection.  All the labor and expense that has been committed to vault and protect credentials has been wasted.

Looking at the example at the end of this document you can see that an engineer without a protected/isolated workstation, that is saving their password locally, can easily have their secrets harvested.  Even if a user is safe and brings up a browser and only reads their password (never placing on the clipboard or into a text based app) the result is the same, the password can be harvested.

An engineer’s workstation should be isolated and protected from any potential malware threats.  Microsoft has a published a document that guides our customers on how to configure their engineer’s workstation.  The guidance is called Privileged Access Workstation (PAW *3).  Customers can use this guidance without any further assistance from Microsoft, to secure their workstations.

A Microsoft PAW implementation won’t require any additional hardware, as long as the current hardware can run a virtualization stack such as Windows 10. So, there should be no new net expense just a requirement to rebuild the user’s/Administrator’s workstation.  If the current workstation is using Win10, it should be fully licensed for the Win10 guests of a PAW implementation, at no additional cost.

“Any user of a Licensed Device, or any device used by a Licensed User; may remotely access up to four Instances of the Software Running in Virtual OSEs or one Instance of the Software Running in one Physical OSE on (a) device(s) dedicated to Customer’s use.”  *4

Why a dedicated workstation?

“The current threat environment for organizations is rife with sophisticated phishing and other internet attacks that create continuous risk of security compromise for internet exposed accounts and workstations.

This threat environment requires an organization to adopt an “assume breach” security posture when designing protections for high value assets like administrative accounts and sensitive business assets. These high value assets need to be protected against both direct internet threats as well as attacks mounted from other workstations, servers, and devices in the environment.”  *5

As a part of protecting credentials within a vault, “Credential Tiering” should also be deployed.  Credential Tiering is a configuration where credentials are only allowed to be used within a predefined Tier.  Tiering will compliment network isolation when the isolation isn’t effective by restricting what administrators can control and where they can log on.

“The Tier model is composed of three levels and only includes administrative accounts, not standard user accounts”.  *6

· Tier 0 – Manage the identity store and a small number of systems that are in effective control of it

o DC’s, PKI, Radius, etc…

· Tier 1 – Manage enterprise servers, services, and applications

· Tier 2 – Manage enterprise desktops, laptops, printers, and other user devices

PAW workstations should only be allowed to extract credentials and manage assets of a single Tier.  This protects against Tier escalation via what an account can manage and control.

Attack scenario example below:

It is trivial to retrieve the password from memory using a debugger, once a host has been compromised.

1. What the heck is a POP-SLAM?

a. https://channel9.msdn.com/Blogs/Taste-of-Premier/Proactively-Secure-your-IT-Environment-from-Credential-Theft-with-POP-SLAM

2. https://technet.microsoft.com/windows-server-docs/security/securing-privileged-access/securing-privileged-access

3. Plat blog on PAW

a. https://blogs.technet.microsoft.com/askpfeplat/2016/03/14/secure-administrative-workstations/

4. http://download.microsoft.com/download/9/8/D/98D6A56C-4D79-40F4-8462-DA3ECBA2DC2C/Licensing_Windows_Desktop_OS_for_Virtual_Machines.pdf

5. https://technet.microsoft.com/en-us/windows-server-docs/security/securing-privileged-access/privileged-access-workstations

6. https://technet.microsoft.com/windows-server-docs/security/securing-privileged-access/securing-privileged-access-reference-material#ADATM_BM

Hopefully this has sparked some thought and gotten you to understand that simply purchasing a vault product (Or using our free LAPS tool) isn’t enough to protect your secured credentials. I would suggest folks that aren’t following this guidance to form a plan to protect any workstations that have access to credentials.

This blog post brought to you by eighteen year veteran Microsoft Premier Field Engineer David Morgan.

Goal of this Post

Today I’d like to impart to you how you can collect a few of what Microsoft thinks are important best practices around Hyper-V. (Note: You can use these same methods for all the other Best Practice features in the later Windows Server Products.) This post will contain information only specific to Hyper-V. There is a much more detailed and in-depth TechNet article here about the BPAs: Run Best Practices Analyzer Scans and Manage Scan Results. (Note: I also support Failover Cluster and would have included that here, but, the Failover Cluster feature does not have a set of BPAs like the ones this article speaks to (BPAs are only for Roles). The Failover Cluster feature includes a much more robust set of tests and BP recommendations produced by the Failover Cluster Validation tool. The Cluster Validation tool also includes some tests for Hyper-V Clustering.)

Also, I want to make sure everyone is aware of the Microsoft RAP as a Service offering which in a sense are a larger Best Practice definition engagement between Microsoft Premier Field Engineers and customers where deep-dive Best Practice analysis is performed. The main Services page where all of the RAPs are defined is here and the details for Failover Cluster & Hyper-V RAPaaS are here:

• Failover Cluster (521 BPA tests performed)
  
  • Datasheet
    
  • Prerequisites
    
• Hyper-V (1018 BPA tests performed)
  
  • Datasheet
    
  • Prerequisites
    

If you are interested in requesting a RAP as a Service for eligible Microsoft products, please see your Technical Account manager (TAM).

Best Practice Analyzers (BPAs)

BPAs first came on the scene with the release of Windows Server 2008 R2. In this post I’m only going to be demonstrating Windows Server 2012 and 2012 R2 BPAs and how to generate a Best Practice list using PowerShell. If you wish to investigate the original BPA version released with 2008 R2 please start here. And, yes, the BPAs exist in the 2016 editions of Windows Server.

BPA GUI outputs are found on the Dashboard page of the Server Manager interface and will show BPA results for any role; for any server listed in the Local Server or All Servers groups in the left Navigation column. These GUI reported BPA notifications are useful but they don’t list all the possible tests the BPA can perform tests for.

There are no master lists of BPA tests for Microsoft Windows Server roles. Also note that not all roles have BPAs. It’s also important to remember that over time as BPA related binaries are updated over the lifetimes of the products, new tests will be added and some older ones may be retired; so repeating the processes below at set intervals is probably wise so as to maintain the latest and greatest BP lists. Finally, these methods will not work unless the BPA model components are installed for a role; a role has to be installed for the BPAs to provide test descriptions and results. Were one to want a reference list of BPAs one could certainly take a temporary server, say a virtual machine, and install all the roles of interest and then use the detail below in a loop to output all the desired BPA lists.

So, let’s generate a Hyper-V BPA list.

Here are the basic steps:

• Install the Role                Install-WindowsFeature
  
• Get the BPA model IDs for the Role    Get-BPAModel
  
• Run the BPA                    Invoke-BPAModel
  
• Review the results                Get-BPAResults
  

Here are the steps in detail:

• Install the role you wish to find BPA test details for:
  
  • For Windows Server Use:
    
    • GUI Server Manager Add Roles and Features or
      
    • PowerShell’s Install-WindowsFeature <role> cmdlet
      
• At this point it would also be a good idea to:
  
  • Install any role Updates and
    
  • Execute Update-Help to make sure you have the latest PowerShell related files
    
• To be able to identify the correct BPAs you’ll need the BPA model ID:
  
  • Use PowerShell’s Get-BPAModel -ModelID command
    

And here are the PowerShell commands to do the work:

(Note: One can copy/paste from #region thru #endregion into the Powershell ISE window, press F5, and execute all the commands below at once to produce the output file.)

#region

# First we need to know the Model ID for Hyper-V, (it’s “Microsoft/Windows/Hyper-V”)

# Let’s get it and store it in a variable.
$HyperV = (Get-BpaModel *hyper-v*).id

# Next we need to run the BPA scan.
Invoke-BpaModel $HyperV

# Once the scan is finished we get the results and export it to a CSV list of BPAs.

# The last two lines just remove some text and delete a temp file.
Get-BpaResult -ModelId $HyperV | select-object Severity, Title | export-csv “c:\Hyper-V-BPA-1.csv”
Get-Content “c:\Hyper-V-BPA-1.csv” | Select-Object  -skip 1 | out-file “c:\Hyper-V-BPA.csv”
Remove-Item “c:\Hyper-V-BPA-1.csv”

#endregion

An example of the Hyper-V-BPA.csv output: (The full output here was 66 individual test and Best Practice recommendations.)

Information    Domain membership is recommended for servers running Hyper-V

Information    Enable all integration services in virtual machines

Information    Compression is recommended for replication traffic

Warning    The WFP virtual switch extension should be enabled if required by third party extensions

Error        Use RAM that provides error correction

Conclusion:

I’ve always like to tell my customers that their server is exactly that, “their server”, and given that, they are welcome to configure it any way they wish. But one should remember that some ways, although they might be made to work, induce risk into an equation, whereas following best practices found over time by numerous product engineers and other customer experiences (not to dismiss the knowledge of those who designed a product to function a certain way) is generally; a “best practice” J.

So take this opportunity to use the BPA lists for educational purposes, compiling IT policies for your department, or any other productive way you can think of. They cost nothing extra, are free and included in the product; and most of all they can save some serious cash on occasion by preventing configuration mistakes.

Till next time……………

This blog post brought to you by eighteen year veteran Microsoft Premier Field Engineer David Morgan.

Goal of this Post

One of the best troubleshooting tools around for Windows Failover Cluster since Windows Server 2008 is the Cluster Validation Wizard. The Validation Wizard can make certain your new failover cluster deployment goes right the first time, but it’s also a great tool to use to find problems in an existing production cluster. Did you know having a current Validation Wizard report is a part of the Microsoft Support statement for failover clustering? It is; and a current report can shave off a lot of questions & troubleshooting work the support engineer will do when they assist you.

What I want to talk about today is one part of the Validation Report and that is storage testing. For the Validation Wizard to test your clustered storage it must have storage to test and more importantly, storage that is in an offline state. For example, if you have a SQL cluster the odds are you will have to take your SQL Service offline to test any associated storage hardware. The average cluster generally has no free storage to test.

Well, most failover clusters do have a witness disk and here lies a quick method of getting storage testing performed. The witness disk since Server 2008 has been different from what clustering had in its quorum model in Server versions 2003 and older. In those legacy clusters the cluster could not survive the loss of the quorum disk but a 2008 or newer cluster can lose it and keep right only providing high availability. Now, the method I’m going to provide instructions for shortly won’t provide the ‘best’ storage test results but it will provide you a quick, safe method to get ‘some’, that may be better than taking your cluster out of production or having to have a storage team carve out and assign a piece of storage for you. The following method requires just a few clicks of the mouse. To mitigate the slight risk involved here one could just ‘change’ the quorum model to a File Share Witness instead until after the storage validation tests are complete and then change it back to a disk witness. I’ll put a link in the directions below to FSW configuration at the time you would need to configure it.

A great TechNet article one should read prior to performing this procedure:

• Understanding Cluster Validation Tests: Storage
  
• https://technet.microsoft.com/en-us/library/cc771259(v=ws.11).aspx
  

Instruction Detail
(at wizard screens not shown in this document just click next):

STEP 1:    Note the three red squared items we will be working with in the main Cluster Administrator window.

STEP 2:

• Right click the Cluster Name
  
• Choose More Actions
  
• Choose Configure Cluster Quorum Settings
  

STEP 3:

• Choose Advanced Quorum Configuration
  

STEP 4:

• Choose Do Not Configure A Quorum Witness
  
  • Note: If you wish to use the risk mitigation option of an FSW choose Configure A File Share Witness instead and follow the instructions in that path.
    

STEP 5:

• Note the warning and choose Next
  

STEP 6:

• Note the No Witness Configured result and choose Finish.
  

STEP 7:

• In Cluster Administrator select Disks in the navigation pane and verify the Disk Witness disk has been returned to Available Storage.
  

STEP 8:

• Start the Validate a Cluster option in the main Cluster Administrator window
  
• In the Validate a Configuration Wizard choose Run Only The Tests I Select
  

STEP 9:

• Deselect all but the Storage option
  

  
  

  STEP 10:
  

  • Select the disk you just returned to Available Storage
    

  
  

  STEP 11:    Allow the Wizard to take the disk offline and run the tests.
  

  
  

  STEP 12:    View the report and verify the tests completed
  

STEP 13:    Right click the Cluster Name:

• Choose More Actions
  
• Choose Configure Cluster Quorum Settings
  
• Choose Select the Quorum Witness
  

STEP 14:    Choose Configure a Disk Witness

STEP 15:    Select the disk that was the previous Disk Witness

STEP 16:    Verify the Disk Witness has been configured successfully when the wizard finishes

STEP 17:    Verify the Disk Witness has been configured successfully in the Cluster Core Resources pane in Cluster Administrator

STEP 18:    Verify the Disk Witness has been configured successfully via the Storage/Disks option in the navigation pane

STEP 19:    Review the storage test results and troubleshoot as necessary.

To summarize:

• We just removed and replaced the Witness Disk in a functioning cluster with no down time, reboots, failovers, restarts of the cluster service, etc. We’ve given ourselves the opportunity to test the entire storage stack, OS file system through the HBAs and all the way down to a specific physical disk, with the testing capabilities of the Failover Cluster Validation Wizard.

Hi everyone.  Graeme Bray here with a quick article around KMS and Server 2016.  KMS and Server 2016 you say?  Shouldn’t I be using Active Directory Based Activation?  Yes, you should, but in case you are not, let’s go over the pre-requisites to activate Windows Server 2016 via KMS.

First, lets review the pre-requisite updates that must be installed.

If your KMS host is running Windows Server 2012, you need two updates:

* https://support.microsoft.com/kb/3058168

o Direct Download: https://www.microsoft.com/en-us/download/details.aspx?id=47649

* https://support.microsoft.com/kb/3172615

o Direct Download: https://www.microsoft.com/en-us/download/details.aspx?id=53316

If your KMS host is running Windows Server 2012 R2, you need two updates:

* https://support.microsoft.com/kb/3058168

o Direct Download: https://www.microsoft.com/en-us/download/details.aspx?id=47622

* https://support.microsoft.com/kb/3172614

o Direct Download: https://www.microsoft.com/en-us/download/details.aspx?id=53333

If your KMS host is running Windows Server 2008 R2:

* There is no update to allow Windows Server 2008 R2 to activate Windows Server 2016.  Windows Server 2008 R2 is in extended support.  During this phase, we only release security updates and do not release updates that add additional functionality.

Lets review what these updates add:

* The first update allows the activation of Windows 10 from Windows 8, 8.1, and Windows Server 2012 R2 based systems.

* The second is an update rollup that allows KMS to activate Windows 10 1607 long-term servicing branch (LTSB) systems and Windows Server 2016 clients.

KMS License Key

After that, all you need to do is add your KMS License Key from the Volume License site.

But wait, how do I find my KMS License Key?!  Have no fear, there is a KB article (and detailed steps) for you!

Retrieve KMS License Key from the VLSC for Windows Server 2016.

To resolve this problem, follow these steps:

1. Log on to the Volume Licensing Service Center (VLSC).

2. Click License.

3. Click Relationship Summary.

4. Click License ID of your current Active License.

5. After the page loads, click Product Keys.

6. In the list of keys, locate Windows Srv 2016 DataCtr/Std KMS

7. Install this key on the KMS host.
https://support.microsoft.com/kb/3086418

Client Licensing

After your KMS host is activated, use the Client Setup Keys to activate your shiny new Windows Server 2016 hosts.

https://technet.microsoft.com/library/jj612867.aspx

What is a client setup key you ask?  A CSVLK is the key that is installed, by default, on your Volume License media that you pulled down day 1.  If you are using volume license media, no key change is required.

If you are converting an install from Retail, MSDN, etc, you will want to use the client setup key to convert to a Volume License key to allow activation.

Active Directory Based Activation

Graeme, this sure seems like a lot of work to get KMS working for Windows Server 2016.  Is there a better way to do this?

The recommendation at this point is to leave your existing KMS system alone.  Whether it is running on Windows Server 2008 R2, Windows Server 2012, or Windows Server 2012 R2, continue to service the machine via security and quality updates.  Allow your KMS system to activate down-level operating systems and Office installs (Windows 7, Windows Server 2008/2008 R2, and Office 2010).  Utilize Active Directory Based Activation (ADBA) for all new clients (Windows 8, 8.1, Windows Server 2012, 2012 R2, 2016, Windows 10, Office 2013, and Office 2016).

Active Directory Based Activation provides several key benefits:

1. Activation is near instantaneous when a system is brought online.  As soon as the system talks to Active Directory, the system is activated.

2. One less server to maintain and update.  Once all downlevel (2008 R2 & prior) systems are migrated, you can remove your KMS host.

3. AD-Based activation is forest wide.  KMS hosts require additional configuration to support multiple domains.

Have more questions?

Q: Where can I get even more details around KMS and AD Based Activation?

A: Refer to these other posts by one of my colleagues, Charity Shelbourne.

https://blogs.technet.microsoft.com/askpfeplat/2013/02/04/active-directory-based-activation-vs-key-management-services/ https://blogs.technet.microsoft.com/askpfeplat/2015/11/09/kms-migration-from-2008-r2-to-windows-server-2012-r2-and-kms-activation-known-issues/

Q: Can Microsoft help me with this process?

A: Absolutely!  Reach out to your TAM and you can engage Premier Field Engineering for assistance.

Thanks for reading!

Graeme “Keeping KMS easy” Bray

Hello, my name is Michael C. Bazarewsky, and I’m a Public Sector Secure Infrastructure PFE, spending most of my time working on Azure-related engagements.

Today, I wanted to present two quick notes about some issues I saw while helping a customer perform a parallel installation of Azure AD Connect, to upgrade from an older DirSync installation. These are not necessarily issues you’ll see – in fact, they are very unusual, in my experience. They were both issues I had never seen, and I’ve helped customers with this many, many times.

Before beginning, I should reference the official documentation, located here. The documentation explains how to do the export and import process, so I’m not going to repeat that here. Instead, I’m just going to discuss the two specific issues that happened in this case.

The first issue occurred during the export. The export appeared to take a very long time – much longer than normal – before finally failing, indicating that the FIM Synchronization Service failed to restart, with error code %%-2145188858. Sure enough, when we looked at the Services Control Panel, the FIM Synchronization Service entry showed in was stuck in a “Starting” state. I know from experience this sometimes is a SQL Server connection issue. In this case, the customer was running a local SQL Server Express instance (as is very common.) So, I asked the customer to restart the SQL Server instance in the Services Control Panel. This caused the FIM Synchronization Service to stop with a failure. However, it restarted successfully when attempted to in the Services Control Panel. Emboldened by this, I then had the customer attempt the export again. This time, the export was successful. I’m still not completely sure what was happening there – something strange in the local SQL Server Express state, I suspect – but as we were doing a parallel install, and thus this server was not long for this world anyway – the customer was willing to accept “cosmic rays” as a tongue-in-cheek explanation and move on.

The second issue was even more subtle. The configuration process on import appeared to move steadily, until the final verification and configuration was being performed. When the verification of the Azure Active Directory connection was performed, we received the following error:

Unexpected exception thrown. Action: PingProvisioningServiceEndPoint, Exception: An error occurred. Error Code: 6. Error Description: Your credentials are not authorized to access Windows Azure Active Directory. Please check your Administrator credentials and try again. Tracking ID: [guid]

What was very frustrating about this error is that when you are doing a parallel installation, you are forced to use the existing Azure Active Directory account from the working installation. Further, the configuration wizard verifies the password is accurate on the page you enter it. This means that we knew the username and password had to be correct! Some quick Bing’ing suggested that this could be an issue with incorrect local credentials. However, we knew that not to be the case here. We also knew there was not a proxy in the on-premises environment to interfere with communications.

On a hunch, I asked the customer to sign in to the Office 365 Portal at https://portal.office.com as a Global Admin, to verify the directory synchronization user was in fact a Global Admin also. This is easy enough to check in the user list in the (new) Office 365 Admin Portal:

1. Select Users.
2. Select Active users.
3. Locate the Directory Sync user in the list, and click it.
4. In the Roles section, confirm the user is a Global administrator.

That told us the account was a valid account, with valid permissions. So, what could possibly be the problem?

On another hunch, I asked the customer to sign in to the Office 365 Portal as the Directory Sync user. When we did, that’s when something interesting happened. As some of you reading this know, the Office 365 Portal is very aggressive about ensuring all Global Admin users have a fallback phone number and e-mail address, in the case of misplaced credentials. It turns out, if your synchronization account is old enough, predating these aggressive requirements, then it may be in a state where although your current synchronization is working, the new synchronization cannot be configured, because the account won’t be allowed in! This was clear when the customer signed-in, as requested, and encountered the “dreaded” “update your admin contact info” redirect:

The customer entered appropriate information and verified it with the codes sent to each destination. Once they did that, they hit Retry in the Azure AD Connect wizard, and they were successfully able to move forward.

I hope you find this helpful if you have either of these situations occur.

— Michael C. Bazarewsky, PubSec Secure Infrastructure PFE

Unexpected TCP resets, intermittent “Network Path Not Found”, and SMB dialects being downgraded. These errors point to something very odd and potentially very bad, happening on the network. If you are like many AD administrators, at the first sign of network impropriety, you likely engage the network team and request the network issues be addressed. While on the call with the networking team, you may hear a very unexpected resolution to the issue. The team may request that you add their non-Windows network devices to your domain as an RODC (Read Only Domain Controller) or grant a service account permission to replicate with your domain controllers.

Before you hang up on the networking team or stop reading this blog, hear me out.  WAN optimizers and other caching technology’s such as Branch Cache can offer substantial improvement in WAN performance.  This is especially true in parts of the globe where high speed WAN connections are either cost prohibitive or non-existent.   For many companies, the benefits of these devices far outweigh the support impact and potential security implications.

This is Premier Field Engineer Greg Campbell, here to remove a little of the mystery about how Riverbed’s Steelhead appliances alter what we see on the wire, how best to reduce or prevent support issues and the significant security considerations when integrating them with the Active Directory.  Let’s start with a little background on the devices and how they operate.   Then we can address the security questions followed by some troubleshooting tips. Here’s what we are going to cover.

• Riverbed Primer – What you need to know when looking at network captures
• Security Implications – What you need to know before integrating the Steelheads with Active Directory
• Troubleshooting info – Common issues you may encounter with Steelheads in the environment.

While the Steelheads can optimize HTTPS traffic (with access to the web site’s private key) and Mapi traffic, the focus of this blog will be on optimizing SMB traffic.

Before we go any further please review our recently published support boundaries on this topic.

https://support.microsoft.com/en-us/kb/3192506

1 – Riverbed Primer

Riverbed Technology Inc is the manufacture of WAN optimization including the Steelhead branded products. Steelhead comes in many forms including appliance devices, a soft client for Windows and Mac as well as a cloud based solution.

The Steelheads can perform three levels of traffic optimization:

• TCP optimization – This includes optimizations at the TCP layer, including optimizing TCP acknowledgment traffic, and TCP window size.
• Scalable data reference (data deduplication) – Instead of sending entire data sets, only changes and references to previously sent data are sent over the WAN. This is most effective for with small changes to large files.
• Application latency optimization – For applications such as file sharing (SMB), Steelhead appliances optimize the protocol by reducing protocol overhead and increasing data throughput.

The last one, application latency, can provide the most impactful performance gains. However, to accomplish application latency optimization, the SMB traffic either needs to be unsigned or the Steelheads will need to inject themselves into the signed client-server communication. I will address that massive can of worms I just cracked, later. For now, just remember that there are up to 3 levels of optimization that can be performed.

Three TCP Connections

Don’t confuse this with the traditional 3-way handshake. Its 3 separate TCP sessions each with their own 3-way handshake. I guess you could say it’s a 9-way handshake. Here are the 3 legs of the journey (see figure 1).

1. Client to Steelhead (LAN) – The first session is between the client and the local Steelhead appliance operating in the client role. This traffic is not optimized and is a regular TCP session.
2. Steelhead to Steelhead (WAN) The second session is TLS encrypted between the two Steelhead appliances. This TLS protected TCP session is a proxied connection that corresponds to the two outer TCP sessions to the client and the server. This traffic is invisible to the client and server and is where the bulk of the optimization takes place.
3. Steelhead to Server (LAN) -The 3^rd session is between the Steelhead operating in the server role. This traffic is not optimized and is a regular TCP session.

Figure 1 – The 3 separate TCP sessions

When looking at simultaneous network (client/server) network captures, there will be completely different TCP sequence numbers, packet ID numbers and possibly different packet sizes. This is because they are different TCP sessions. Though they are different TCP sessions, the Steelhead appliances use a feature called IP and port transparency. This means the IP and port on the client and the server are not altered by the optimization. This can be helpful when attempting to align the client side and server side conversions in the two network captures.

While Steelhead appliances do not have configured roles, to help explain traffic flow and AD requirements, the Steelhead nearest the client is the in the “client role”, or C-SH. The Steelhead nearest the server, is in the “server role” or S-SH. These roles would be reversed if the traffic were reversed. For example, a server in the data center accessing the client for system management etc.

Steelhead Bypass Options

There are times when traffic cannot or should not be optimized by the Steelheads. During troubleshooting, it can be helpful to try bypassing the Steelheads to determine if they are involved in the issue. Because there are different ways to bypass the Steelheads, it’s helpful for the AD admin to know which method of bypass was used. Especially if the bypass method didn’t address the issue. Most of the bypass methods do not completely bypass the Steelheads nor disable all levels of optimization. If the issue persists after bypassing has been enabled, it may be necessary to use a different bypass method.

Steelhead’s IP Blacklist

The Steelhead appliances will attempt to optimize all SMB traffic that is not already excluded. If the traffic cannot be application latency optimized, the IP addresses are put in dynamic exclusion list called a “blacklist” for 20 minutes. The next time a connection is attempted, the Steelhead will allow the traffic to bypass latency optimization. If a single IP address appears on the blacklist several times, it will be put onto a long-term blacklist, so the first failure is avoided. The long-term blacklists will persist until the Steelhead is rebooted or its manually cleared by an administrator.

In the case of signed SMB traffic that cannot be application latency optimized, first attempt to connect will likely fail. In the client side capture, you may see an unexpected TCP Reset from the server. At the next attempt to connect, the Steelheads only perform the first two levels of optimization and the connection typically succeeds. The blacklist is short lived (20 minutes). If the client application silently retries, the user may not see any issue. However, some users may report that sometimes it works and sometimes it doesn’t. Or they may have noticed that first attempt fails but the second works. Keep this in mind when troubleshooting intermittent network issues.

In-Path Bypass rules and Peering rules

If the Steelhead appliances are not AD integrated, the best option is to have the Steelhead administrators exclude the traffic from optimization by adding an In-Path, pass-through, rule on the Steelheads. Because domain controllers always sign SMB traffic, In-Path, Pass-Through rules are recommended for Domain Controllers. On the Steelheads operating in the client role, create an in-path rule to pass-through all traffic to the domain controllers. For more information, see:

https://support.riverbed.com/bin/support/static/bkmpofug7p1q70mbrco0humch1/html/ocgj3m4oc178q0cigtfufl0m68/sh_ex_4.1_ug_html/index.html#page/sh_ex_4.1_ug_htm/setupServiceInpathRules.html

Optionally, a peering rule can be applied on the Server side Steelhead. Work with your Riverbed support professional to determine which method is recommended for your environment.

Riverbed Interceptor and Bypass Options

For troubleshooting, it may be necessary to completely bypass the Steelhead appliances for testing. If the environment is using the load distribution device “Riverbed Interceptor”, the by-pass rule can be added either on the Inceptor or on the Steelheads. The interceptor balances optimization traffic across the Steelhead appliances within the site. The interceptor can be configured to bypass all the Steelhead appliances, sending traffic un-optimized directly to the router. If the bypass rule is set on the Steelheads instead of the interceptor, the traffic may still be TCP and SDR (Scalable data reference) optimized. When deciding which bypass method to use, note the following:

1. In-path, bypass rules for the Steelheads. When configured on the Steelhead, only application latency optimization is disabled. TCP optimization and scalable data reference optimization are still active. This still provides a measure of optimization. When troubleshooting, this method along may still introduce issues.
2. In-path, bypass rules for Interceptors. Configured on the Riverbed Interceptors, when these rules are enabled, the traffic will completely bypass the Steelhead appliances. No optimization is performed. Setting the bypass rule on the Interceptor may be required in some troubleshooting scenarios.

Figure 2 – Bypass rules configured at the interceptor

2 – Security Implications

In this section, we review the security impact of integrating Steelheads with Active Directory. This integration is required to fully optimize signed SMB traffic.

Why SMB traffic Should Be Signed

Consider the scenario where a client’s traffic is intercepted and relayed to another host. An adversary is acting as a man-in-the-middle (MitM) and the connection information may be used connect to another resource without your knowledge. This why it’s important that the client have some way to verify the identity of the host its connecting to. For SMB, the solution to this problem goes back as far as Windows NT 4.0 SP3 and Windows 98. SMB signing is the method used to cryptographically sign the SMB traffic. This is accomplished by using a session key that is derived from the authentication phase, during SMB session setup. The file server uses its long-term key, aka, computer account password, to complete this phase and prove its identity.

When traffic is “application optimized”, the Steelhead appliances operate as an authorized man-in-the-middle. They are intercepting, repackaging and then unpacking traffic to and from the real server. To sign this traffic, the Steelhead operating in the server role needs access to the session key. Without AD integration, the Steelhead appliances do not have access to the server’s long-term keys and cannot obtain the session key. The Steelhead is unable to sign the SMB Session setup packets and cannot prove it is the server the client had intended to communicate with. SMB signing is doing its job and preventing the man-in-the-middle. For the Steelheads, this protection means the traffic cannot be application latency optimized.

Signing with SMB3 and Windows 10 (Secure Negotiate)

SMB3 added another layer of protection called secure negotiate. During the client / server SMB negotiation, the client and server agree on the highest supported SMB dialect. The server response contains the negotiated SMB dialect and with field in the packet containing the dialect value signed by the server. This step ensures that the SMB dialect cannot be downgraded to an older, weaker dialect by a man-in-the-middle. Compared to traditional SMB signing, far less of the SMB packet is signed. However, this still causes an issue for the Steelheads that are not able to sign the field containing the negotiated dialect.

In Windows 8, it was possible to disable Secure Negotiate. While unadvisable, many admins opted to restore WAN performance at the expense of SMB security. In Windows 10 the capability to downgrade security by disabling Secure Negotiate, is no longer available. The leaves only two options for Windows 10:

• Integrate the Steelhead appliance (server role in the datacenter) with AD.
• Bypass the Steelhead appliances for application optimization. This significantly reduces the effectiveness of the Steelhead appliances in some scenarios.

NOTE: When a Windows 10 client (SMB 3.1.1) is communicating with a down-level server, the client will use Secure Negotiate. However, when a Windows 10 client is communicating with a server running SMB 3.1.1, it will use the more advanced variant, Pre-Authentication Integrity. Pre-Authentication Integrity also uses signed packets and has the same Steelhead requirements as Secure Negotiate. For more information, see https://blogs.msdn.microsoft.com/openspecification/2015/08/11/smb-3-1-1-pre-authentication-integrity-in-windows-10/

RIOS versions 9.1.3 and below, currently do not support Pre-Authentication Integrity. Contact your Riverbed Support professional for options if this scenario applies your environment

For an in-depth discussion of the Secure Negotiate feature, see: https://blogs.msdn.microsoft.com/openspecification/2012/06/28/smb3-secure-dialect-negotiation/

Steelhead options for Optimizing Signed SMB Traffic

Now we know that Steelhead needs the server’s long term key (computer account password), to sign the response and subsequently fully “application optimize” the traffic. The required permissions vary by authentication protocol (NTLM or Kerberos) and RIOS version. For this discussion, we will focus on RIOS version 7 and above.

AD Integration – Kerberos

The Steelheads use one of two methods to obtain the session keys needed to sign optimized SMB traffic. To optimize SMB traffic authenticated using NTLM, the Steelheads use a computer account with RODC privileges. More on this in the next section. To optimize traffic authenticated using Kerberos, the Steelheads do not need an RODC account but instead use a highly privileged service account along with a workstation computer account. While this mode requires far more privileges than the RODC approach, it is required for optimizing Kerberos authentication sessions. Additionally, this method does not suffer from the RODC account related issues discussed in the troubleshooting section.

As of RIOS version 7.0 and above, the “End-To-End Kerberos Mode” or eeKRB is recommended by Riverbed. This mode replaces the deprecated “Constrained Delegation Mode” in older version of RIOS. The account requirements for Steelhead operating in the server role are:

1. A Service Account with the following permission on root of each domain partition containing servers to optimize.

   “Replicate Directory Changes”

   “Replicate Directory Changes All”.

2. The Steelheads need to be to be joined to at least one domain in the forest as a workstation. Not as an RODC.

The “Replicate Directory Changes All” permission grants the Steelhead service account access replicate directory objects including domain secrets. Domain secrets include the attributes where account password hashes are stored. This includes the password hashes for the file servers whose traffic is to be optimized, as well as hashes for users, administrators, trusts and domain controllers.

The other critical piece of the “Replicate Directory Changes All” permission is that it bypasses the RODC’s password replication policy. There is no way to constrain access only to the computer accounts to be optimized.

• Extended rights https://technet.microsoft.com/en-us/library/cc772673(v=WS.10).aspx
• Domain Secrets https://technet.microsoft.com/sv-se/library/cc754218(v=ws.10).aspx

In most environments, all SMB traffic should already be using Kerberos for authentication. In these environments, it is not necessary to configure the Steelheads with an RODC account. This is important as it will prevent many of the issues with the RODC account approach. The security impact of this configuration is considerable and needs careful planning and risk mitigation. More on this below.

AD Integration – NTLM

To support NTLM authentication, the Steelhead operating on the server role will need to be joined as a workstation and the UserAccountControl set to 83955712 (Dec), or 0x5011000 (Hex). This maps to the following capabilities:

• PARTIAL_SECRETS_ACCOUNT
• TRUSTED_TO_AUTH_FOR_DELEGATION
• WORKSTATION_TRUST_ACCOUNT
• DONT_EXPIRE_PASSWORD

The UserAccountControl settings grant access to the partial secrets. These are the same permissions that are used by RODCs to replicate the domain secrets for accounts included in the RODC Password Replication Policy for the account. The flag TRUSTED_TO_AUTH_FOR_DELEGATION allows the account to perform protocol transition. KB3192506 describes the risk:

“The account can impersonate any user in the Active Directory except those who are marked as “sensitive and not allowed for delegation.” Because of Kerberos Protocol transition, the account can do this even without having the password for impersonation-allowed users.”

UserAccountControl value of 0x5011000 will identify the account as an RODC. This cause issues for tools that query for DCs based on the UserAccountControl value. See the section below for more details Troubleshooting issues with UserAccountControl set to 0x5011000.

Security Implications of AD Integration – What is exposed?

The decision to integrate Steelhead should be the outcome of a collaboration between a company’s security team, the company’s networking team and the company’s Active Directory team. To inform this discussion, consider the value of the data being shared with the Steelhead appliances along with the resulting “effective control” of that data. Microsoft’s current guidance regarding protection of data and securing access can be found in the Securing Privileged Access Reference. The reference defines a concept called the Clean Source Principle.

“The clean source principle requires all security dependencies to be as trustworthy as the object being secured.

“Any subject in control of an object is a security dependency of that object. If an adversary can control anything in effective control of a target object, they can control that target object. Because of this, you must ensure that the assurances for all security dependencies are at or above the desired security level of the object itself. “

Source: https://technet.microsoft.com/windows-server-docs/security/securing-privileged-access/securing-privileged-access-reference-material#a-name-csp-bm-a-clean-source-principle

In this case, the data is the secrets for all domains in the forest. Because a domain controller only holds secrets for its own domain, the service account has higher privileges than any single DC. The service account can authenticate as any user in the forest to any resource within the forest or any trusting forest where permissions are granted. This means the service account has either direct or indirect control of all data in the forest as well as any trusting forests (where permissions are granted). To put this in perspective, the service account can act as any account (user, computer, DC) to access any resources in the forest including all servers containing high value IP, email, financial reports, etc. Since the Riverbed administrators have control of the service account, these admins now have indirect control of the entire forest.

Below are some guidelines for securing privilege access. While this applies to domain controllers and domain administrators, these practices and requirements may be extended to any highly privileged device or account. When reviewing the requirements below, consider these questions:

Can the Steelhead environment, including appliances, management workstations and all accounts with direct or indirect access be secured to the same level that the DCs and domain administrators?

If so, what is the financial impact of both initial and ongoing operational changes needed to secure the Steelhead environment?

Finally, do the optimization benefits outweigh the increased operation cost and potential increase in risk to the forest?

In order to secure any accounts or appliance that have this level of privilege the Securing Privileged Access Reference provides a good starting point.

Tier 0 administrator – manage the identity store and a small number of systems that are in effective control of it, and:

• Can only log on interactively or access assets trusted at the Tier 0 level
  
• Separate administrative accounts
  
• No browsing the public Internet with admin accounts or from admin workstations
• No accessing email with admin accounts or from admin workstations
  
• Store service and application account passwords in a secure location
  
  • physical safe,
    
  • Ensure that all access to the password is logged, tracked, and monitored by a disinterested party, such as a manager who is not trained to perform IT administration.
    
• Enforce smartcard multi-factor authentication (MFA) for all admin accounts, no administrative account can use a password for authentication.
  
• A script should be implemented to automatically and periodically reset the random password hash value by disabling and immediately re-enabling the attribute “Smart Card Required for Interactive Logon.”
  

Additional guidance comes from “Securing Domain Controllers Against Attack“

https://technet.microsoft.com/en-us/windows-server-docs/identity/ad-ds/plan/security-best-practices/securing-domain-controllers-against-attack

Highlights include

• In datacenters, physical domain controllers should be installed in dedicated secure racks or cages that are separate from the general server population.
  
• OS Updates and Patch Deployments – To ensure known security vulnerable are not exploited all appliances should be kept up to date in accordance with the manufacturer’s guidance. This may include running the current operating system and patch level.
• Remote access such RPD or SSH highly restricted to only permitted from secured admin workstations.

The above list is not an endorsement of Steelhead security, but serves as a starting point to plan the security of extending Tier 0 beyond the DCs and domain admins. Additional, mitigations should include timely patching, complex password that is regularly changed and mitigations provided by Riverbed.

3 – Troubleshooting Network Issues when Steelheads are involved

This section will cover common issues and troubleshooting guidance. Troubleshooting with Steelheads generally follows this flow:

• Are Steelheads involved in the conversation? Even if Steelheads are present in the environment, they may not be causing the networking issue. They may not be deployed everywhere in the environment or Steelheads can be configured to bypass some traffic. So even if they are in use and in the network path they may not be negatively affecting the network traffic.
• Are the Steelheads the cause of the network issue? Network issues have existed long before Steelheads arrived. Careful diagnosis is required. The Steelheads may be optimizing traffic but not causing the network issue. When in doubt, have the Steelhead administrators bypass the traffic for the affected machines for testing.
• Test by bypassing the Steelheads. When the Steelheads are determined to be the cause of networking issues, they can be bypassed in one of several ways. See the bypass section for more details.

Detecting Riverbed Probe Info in captures

When troubleshooting connectivity issues, if you suspect that a Steelhead appliance is involved, there are several ways to detect Steelhead appliances. Most of them require a simultaneous capture on the client and the server. With these two methods, the capture is only performed on one side.

Riverbed Probe Info

The capture must be from the server side and it must include the SYN packet from the client. Locate the SYN packet in the capture and check for option 76. This packet will also show which Steelhead the client accessed. This can be helpful when engaging the networking team. Wireshark parsers show this as a Riverbed Probe Query. Use the display filter ‘tcp.options.rvbd.probe’

Internet Protocol Version 4, Src: <source IP Adderss>, Dst: <Destination IP address>

Transmission Control Protocol, Src Port: <source port>, Dst Port: <destination port>, Seq: 0, Len: 0

Riverbed Probe, Riverbed Probe, No-Operation (NOP), End of Option List (EOL)

No-Operation (NOP)

Riverbed Probe: Probe Query, CSH IP: <Steelhead IP Address>

Kind: Riverbed Probe (76)

Reserved: 0x01

CSH IP: <Steelhead IP Address>

Riverbed Probe: Probe Query Info

Probe Flags: 0x05

For a complete list of Riverbed display filters, see https://www.wireshark.org/docs/dfref/t/tcp.html

Search for filters starting with “tcp.options.rvbd”. Or enter “tcp.options.rvbd” in the Wireshark filter field and the list of available filters will be displayed.

Detecting Steelheads using the TTL / Hoplimit values in the IP header

The IP Header each packet contains a TTL value for Ipv4. For IPv6 this is called Hoplimit. The default value for Windows systems is 128. The value is decremented by 1 each time the packet traverses a router. When the packets originate from non-Windows systems, the value is often considerably lower. For Steelheads, this value starts at 64 and is typically 60 to 63 after the packet traverses a router or two. While a value of 64 below does not always mean the packet traversed a Steelhead Appliance. But of the packet was sent by a Windows system, it’s clear the TTL was modified in-route and the Steelhead Appliance may be source of that change.

IPv4 TTL Example

Ipv4: Src = XXXXX Dest = XXXXX 3, Next Protocol = TCP, Packet ID = 62835, Total IP Length = 1400

  + Versions: IPv4, Internet Protocol; Header Length = 20

  + DifferentiatedServicesField: DSCP: 0, ECN: 0

    TotalLength: 1400 (0x578)

    Identification: 62835 (0xF573)

  + FragmentFlags: 16384 (0x4000)

    TimeToLive: 64 (0x40)

    NextProtocol: TCP, 6(0x6)

    Checksum: 22330 (0x573A)

    SourceAddress: XXXXX

    DestinationAddress: XXXXX

IPv6 Hoplimit Example

+ Ethernet: Etype = IPv6,DestinationAddress:[XXXXX],SourceAddress:[XXXXX]

– Ipv6: Next Protocol = TCP, Payload Length = 272

+ Versions: IPv6, Internet Protocol, DSCP 0

PayloadLength: 272 (0x110)

NextProtocol: TCP, 6(0x6)

HopLimit: 64 (0x40)

SourceAddress: XXXXX

DestinationAddress: XXXXX

Detecting Steelheads with Simultaneous network captures

In many cases, simultaneous captures will be required to verify where the failure occurred in the conversation. In the capture, look for the following to determine if the traffic is traversing Steelhead appliances. Align the captures by IP address and port number. Then align the start of the conversation using common packet such as “SMB Negotiate” that is present in both captures. If the packets traverse a Steelhead, you will likely see:

• The SMB Session ID does not match.
• The packet size and sequence numbers for the same traffic do not match between the two captures.
• Additionally, you may see a TCP reset at the client that was never sent by the server. This can happen if the conversation is signed and the Steelheads are not AD Integrated. See Steelhead’s IP Blacklist earlier in this blog for more info.

SMB Negotiates down to 2.02

Steelheads have a mode called “Basic Dialect” which is often used to disable client leasing and force the use of oplocks. In this mode, the Steelheads intercept and modify available dialects supported by the server. While this behavior may not cause any issues for SMB 2.02 supported capabilities, it does mean that SMB3 capabilities will be disabled. Here is list capabilities that will be disabled in this mode from https://support.microsoft.com/en-us/kb/2709568 and https://blogs.technet.microsoft.com/josebda/2015/05/05/whats-new-in-smb-3-1-1-in-the-windows-server-2016-technical-preview-2/

• SMB Transparent Failover
• SMB Scale Out
• SMB Multichannel
• SMB Direct
• SMB Encryption including support for AES-128-GCM
• VSS for SMB file shares
• SMB Directory Leasing
• SMB PowerShell
• Cluster Dialect Fencing
• Pre-Authentication Integrity

In this example of network capture, a Server 2012 R2 client is connecting to a Server 2012 R2 Server. The traffic is signed and not latency optimized. This is the packet that left the client. Notice the SMB dialect offered by the client is 202 through 302.

SMB2 (Server Message Block Protocol version 2)

SMB2 Header

Negotiate Protocol Request (0x00)

StructureSize: 0x0024

Dialect count: 4

Security mode: 0x01, Signing enabled

Reserved: 0000

Capabilities: 0x0000007f, DFS, LEASING, LARGE MTU, MULTI CHANNEL, PERSISTENT HANDLES, DIRECTORY LEASING, ENCRYPTION

Client Guid: 5c050930-680d-11e6-80d0-9457a55aefad

NegotiateContextOffset: 0x0000

NegotiateContextCount: 0

Reserved: 0000

Dialect: 0x0202

Dialect: 0x0210

Dialect: 0x0300

Dialect: 0x0302

Here is same packet when it arrived at the server. Notice that the Steelhead has removed all available dialects except 202.

Altered packet that arrives to the Server

SMB2 (Server Message Block Protocol version 2)

SMB2 Header

Negotiate Protocol Request (0x00)

StructureSize: 0x0024

Dialect count: 4

Security mode: 0x01, Signing enabled

Reserved: 0000

Capabilities: 0x0000007f, DFS, LEASING, LARGE MTU, MULTI CHANNEL, PERSISTENT HANDLES, DIRECTORY LEASING, ENCRYPTION

Client Guid: 5c050930-680d-11e6-80d0-9457a55aefad

NegotiateContextOffset: 0x0000

NegotiateContextCount: 0

Reserved: 0000

Dialect: 0x0202

Dialect: 0x0000

Dialect: 0x0000

Dialect: 0x0000

Troubleshooting

This section covers common troubleshooting scenarios where Steelheads are involved in the conversation.

Intermittent Connectivity – Knock twice to enter

While intermittent connectivity can be a transient network issue, the effects of Steelheads have a very specific pattern. The first attempt to connect fails with a “Network Path Not found” or other generic network failure. A network capture on the client will show either an expected TCP reset from the server, or an unexpected Ack, Fin terminating the connection. A capture taken at the server will show that these packets were never sent from the server.

The second (or sometimes the third attempt) to connect is successful. The connection works for over 20 minutes and then may fail again. Retrying the connection twice more establishes the connection once again. This pattern occurs when the SMB session or negotiated dialect field is signed and the Steelheads are not AD Integrated. The Steelhead is attempting to perform all 3 levels of optimization and is not able to perform the last one, Application Latency, due the signing requirements. The Steelhead then puts the clients and servers’ s IP addresses on a temporary bypass list with a lifetime of 20 minutes. The existing session must be torn down and new session established with bypass in place. The TCP reset is what triggers the tear down. The next time the client retries the connection, the Steelheads will not attempt application latency optimization and the operation succeeds.

DC Promo Failure

During promotion of a 2012 R2 Domain controller using a 2012 R2 helper DC (both using SMB 3.0), the candidate DC receives an unexpected A…F (Ack, Fin) packet while checking for the presence of its machine account on the helper DC. The following is logged in the DCpromoUI log

Calling DsRoleGetDcOperationResults

Error 0x0 (!0 => error)

Operation results:

OperationStatus : 0x6BA !0 => error

DisplayString : A domain controller could not be contacted for the domain contoso.com that contained an account for this computer. Make the computer a member of a workgroup then rejoin the domain before retrying the promotion.

After receiving the unexpected A..F packet from the helper DC, the candidate DC hits the failure and does not retry. This happens even if the promotion is using IFM (install from media). In many of these scenarios, the servers are added to the dynamic bypass list called the blacklist. However, in this scenario, retrying the operation still fails. To address this issue, manually configuring a bypass rule on the Steelheads is required.

Issues with UserAccountControl set to 0x5011000

This section convers the issues that may be present when joining the Steelhead appliances with UserAccountControl set to 0x5011000 (Hex). When this value is set on the User Account Control Attribute, the Steelhead will appear to many tools an RODC. Because the Steelhead does not provide RODC services, several tools and processes encounter failures.

Setting the UserAccountControl set to 0x5011000 (Hex) is only necessary when NTLM is used between the client and the server. If Kerberos is used to authenticate the user, then the Steelhead can be joined as a regular workstation. To avoid the issues below, consider using the Steelhead to only optimize traffic when Kerberos was used.

AD Tools Detect Steelhead Accounts as DCs

Tools that rely on UserAccountControl values to enumerate DCs in a domain will find Steelhead Appliances joined with UserAccountControl set to 83955712 (Dec), or 0x5011000 (Hex). Some examples are:

• nltest /dclist:domain.com
• [DirectoryServices.Activedirectory.Forest]::GetCurrentForest().domains | %{$_.domaincontrollers} | ft name,OSVersion,Domain
• DFSR Migration tool dfsrmig.exe will find the Steelhead accounts and prevent the transition to the “eliminated”. This is because the tool expects the account to respond as a DC and report its migration state when queried. The migration state cannot transition to its final state until the Steelhead accounts are removed from the domain.

Pre-created RODC Accounts and Slow logon

When pre-creating machine account for the Steelhead, the account should be created as a regular workstation, not an RODC. When an RODC account is pre-created, the process also creates a server object and NTDS Settings object in the Sites container. The Steelhead machine account may be discovered by the DFS service hosting SYSVOL and provide the server name in the DFS referrals for SYSVOL. This can contribute to a slow logon experience as clients try and fail to connect to the Steelhead appliance.

The partition knowledge table (dfsutil /pktinfo) might show a list like this with the Riverbed appliance at top:

Entry: \contoso.com\sysvol

hortEntry: \contoso.com\sysvol

Expires in 752 seconds

UseCount: 0 Type:0x1 ( DFS )

0:[\Steelhead1.contoso.com\sysvol] AccessStatus: 0xc00000be ( TARGETSET )

1:[\DC1.contoso.com\sysvol] AccessStatus: 0 ( ACTIVE )

2:[\DC2.contoso.com\sysvol] AccessStatus: 0

To prevent this issue, do not use the RODC precreation wizard to create Steelhead machine accounts. Create the account as a workstation and then modify the UserAccountControl values. To recover from this state, delete the NTDS settings for the Steelhead accounts. Note that it will take some time for the DFS caches on the client and domain root to timeout and refresh.

DSID Error Viewing the Steelhead Machine Account.

When viewing the Steelhead machine account with AD Users and Computers, or ADSIEdit on a Server 2008 R2, you may encounter the error below. This error occurs because the UserAccount Control values make it appear as an RODC. The interface then attempts to querying for NtdsObjectName which it cannot find because regular workstations do would not have NTDS Object.

This issue does not occur on using LDP, on 2008 R2. Server 2012 R2 and the latest RSAT tools are also not affected.

Domain Join Failure when not using a Domain Admin Account

The Steelhead appliance can be joined to a domain, as a workstation, or as an account with RODC flags. During the operation, the credentials entered on the Steelhead UI will be used to modify the UserAccountControl value as well as add Service Principal Names. There are two operations that may fail, and here’s why:

UserAccountControl – With security enhancements in MS15-096, modification of the UserAccountControl value by non-admins is no longer permitted.

The Steelhead log may report:

Failed to join domain: user specified does not have admin privilege

This is caused by a security change in MS15-096 that will prevent change of all flags in UserAccountControl that change Account Type for non-administrators.

3072595 MS15-096: Vulnerability in Active Directory service could allow denial of service: September 8, 2015 http://support.microsoft.com/kb/3072595/EN-US

ServicePrincipalName – The SPNs being written will be for the HOST/Steelhead account and the HOST/SteelheadFQDN. The account writing the SPNs is not the same as the Steelhead’s computer account and will not be able to update the SPN. The Steelhead log may report:

Failed to join domain: failed to set machine spn

Entering domain administrator credentials in the Steelhead domain join UI is not recommended for security reasons. Additionally, it may not even be possible for accounts that require smartcards for logon.

Workaround

• A workaround is to pre-create the computer account, set the correct UserAccountControl values and give the user account that is joining the Steelheads full control over the account.
• During domain join using the Steelhead UI, the default container “cn=comptuters” is used. If the pre-created account is in a different OU, the domain join must be performed using the CLI commands on the Steelhead. Refer to the Riverbed documentation for the syntax
• The pre-created accounts will need to have the UserAccountValue set correctly before the Steelhead attempts to join the domain. Use the following values:
  • Joining as a Workstation = 69632 (Dec) or 0x11000 (Hex)
  • Joining with RODC flags = 83955712 (Dec), or 0x5011000 (Hex).

Summary

For many environments, WAN optimization provides significant improvements over existing links. This is especially true for regions where high speed, low latency WAN capabilities are either cost prohibitive or non-existent. To support these capabilities in a secure way, requires collaboration between groups which may have historically operated more independently.

After thoroughly evaluating the risks associated with AD integration, some environments may find the cost of risk mitigation, and operational overhead prohibitive. In some cases, it will simply not be possible to mitigate the risks to an acceptable level. While in other environments, the exercise of mitigating risks may improve the overall security of the environment.

References

* Performance Brief – Signed SMB311 https://splash.riverbed.com/docs/DOC-5622

Riverbed Unsigned SMB3 Performance Brief https://splash.riverbed.com/docs/DOC-3198

How WAN Optimization Works – http://www.riverbednews.com/2014/11/how-wan-optimization-works/

Technical Overview of RiOS 8.5 https://splash.riverbed.com/docs/DOC-1198

Steelhead RiOS 9.0 Technical Overview https://splash.riverbed.com/docs/DOC-5505

Configuring Steelhead In-Path Rules – https://support.riverbed.com/bin/support/static/bkmpofug7p1q70mbrco0humch1/html/ocgj3m4oc178q0cigtfufl0m68/sh_ex_4.1_ug_html/index.html#page/sh_ex_4.1_ug_htm/setupServiceInpathRules.html

Secure Negotiate – https://blogs.msdn.microsoft.com/openspecification/2012/06/28/smb3-secure-dialect-negotiation/

SMB Preauthentication Integrity –

https://blogs.msdn.microsoft.com/openspecification/2015/08/11/smb-3-1-1-pre-authentication-integrity-in-windows-10/

RODC Technical Reference Topics (Includes information on “domain secrets”)

https://technet.microsoft.com/sv-se/library/cc754218(v=ws.10).aspx

Third-party information disclaimer

The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products.

Hi everyone! Brandon Wilson here just passing along a friendly note that consumers of WSUS will probably be jumping for joy to hear. Nathan Mercer and Michael Niehaus have published some useful news in this blog post (contents can also be read below).

So, without further delay….here it is in Nathan and Michael’s words:

—–

Windows Server Update Services (WSUS) is key to the Windows servicing process for many organizations. Whether being used standalone or as a component of other products like System Center Configuration Manager or Windows Small Business Server, it provides a variety of useful features, including automating the download and installation of Windows updates.

While WSUS has been built into Windows Server 2012 and later operating systems, most people didn’t realize that it was a separate product for earlier operating systems like Windows Server 2008 R2. Because the version that complemented Windows Server 2008 R2, WSUS 3.0, was considered a separate product, it had a separate support lifecycle, and that lifecycle was due to end in July of 2017, even though extended support for Windows Server 2008 R2 continues until January of 2020.

To remedy this situation, we have extended WSUS 3.0 support to coincide with the Windows Server 2008 R2 end of support date. Now, both will be supported through January 14, 2020. While this reduces the sense of urgency, we still would like to encourage organizations to move all remaining WSUS 3.0 servers to a later version, a process that involves migrating to a new version of Windows Server where WSUS 4.0 (or later, in the case of the upcoming Windows Server 2016 release) can be used.

For those using Windows 10, it is particularly important to look at moving to WSUS 4.0 (or later) to support the deployment of Windows 10 feature updates, which add new features to Windows 10. Support for this new type of update has been added to WSUS 4.0 via an update to WSUS itself. This functionality isn’t available in WSUS 3.0 because mainstream support for that version has already ended (extended support does not add new capabilities; it only fixes security issues).

To help you with the migration process to WSUS 4.0 (or later), we have provided some additional documentation to help guide you through the process. For more information on WSUS, be sure to check out the WSUS team blog.

—–

This change is great news, however don’t let it deter you from embracing WSUS 4.0! Hopefully this post will be of some help, and if you have questions are comments, please feel free to leave them here or on the above linked post.

Until later….Brandon Wilson

Hello everyone, my name is Preston K. Parsard, and I’m a Premier Field Engineer. I’ve been focusing on Azure and PowerShell engagements recently and would like to present an illustrated article about how Windows PowerShell parameter binding works.

REQUIREMENTS

Windows PowerShell version 4.0 is used in the screenshots for this article include both 4.0, however, Get-Service, Stop-Service and Set-Service have been available since PowerShell version 2.0.

PURPOSE

One of the more abstract concepts to both learn as a new user and to explain as an instructor is just exactly what goes on when you use the pipeline operator to combine expressions and pass values from one cmdlet to another in sequence. The good news is that once this process is understood, we can start to develop our own advanced functions while improving our scripting knowledge and capabilities. These are functions that can integrate parameters which accept pipeline input, and in fact this is how many of the native PowerShell cmdlets already work. Now a further discussion of building those types of functions are beyond the scope of this post, but can be found here. For now, we’ll review the process of pipeline inputs for existing native PowerShell cmdlets. First though, I’ll offer a quick description of the pipeline and the pipeline operator, for those of us that are not already familiar.

A pipeline in PowerShell is a series of values, expressions, commands or cmdlets that are combined with the pipe operator (|) to send the results of one command or expression to the next. These results are sent through the pipeline as objects or object properties, not just text as from the Windows command console (cmd.exe) or certain other non-PowerShell methods. If the results consist of an array of objects, these objects are sent in one-by-one through the pipeline.

Figure 1: Using the findstr.exe utility to search for “packets”

Here I am looking for the case insensitive string of “packets” in the result.

Now if I would like to only retrieve the number of received packets using the command console, I could try to do so by modifying my original expression to the following:

Figure 2: Using the findstr.exe utility to search for “received”

Look familiar? The command returns the entire line again, so the result is identical to the previous one shown in Figure 1.

Now what if the command ping localhost were an object, and we could just specify exactly which property of that resulting object we are interested in, such as the Received property to get its value of 4? This would provide more flexibility, but unfortunately we can’t do this directly in the command shell.

In PowerShell, objects can be filtered and manipulated at the other end of the pipeline with the receiving cmdlets, so you gain more control over utilizing all the available properties of those objects than would be possible with just text results.

USE CASES

Figure 3: Stop and Set-Service

Ok, so now that we’ve quickly reviewed the pipeline, let’s turn to our scenarios and some specific examples to examine these concepts in more detail.

STOP-SERVICE

Imagine that you wanted to identify a service and stop it? How would you do that? What about if it’s on a remote server on your network?

It will look something like this:

Get-Service -Name BITS -ComputerName 2012r2-ms | Stop-Service -PassThru -Verbose

If we look at the name parameter attribute for the Stop-Service cmdlet, which is on the receiving side of the pipeline, we’ll see below (1) that this parameter has a type of <ServiceController[]>, which indicates it can accept multiple values, or an array of service names with the ServiceController .NET Class Type Name. It also shows that (2) this parameter accepts pipeline input, by value.

Figure 4: Stop-Service InputObject parameter attribute table

If we also look at the expressions with their available properties and parameters in terms of class diagrams, it would look something like this:

Figure 5: Get-Service and Stop-Service viewed as partial class diagrams

When Get-Service -Name BITS -ComputerName 2012r2-ms
is evaluated on the left side of the pipeline, it produces a result which is an object of type
ServiceController and has a value of bits hosted on the computer 2012r2-ms.

We’ve also just seen from Figure 4 that the Stop-Service cmdlet accepts pipeline input, and that it accepts values coming in to the InputObject parameter specifically, so it accepts these parameters by the object
value or values depending on whether a single or multiple objects are received.

By the way, it will be important to keep this in mind for other parameters which may accept pipeline input both by value and parameter name. The rules and processing order of how these parameters are evaluated follows:

1. ByValue without coercion: Match incoming object type to receiving parameter type without having to try and convert the object type to the parameter type.
   
2. ByPropertyName without coercion: If the incoming property name (not object value) matches, the receiving parameter name, then attempt to the match incoming object property type to the receiving parameter type, again, without type conversion.
   

   This option requires that the property name of the receiving cmdlet has to match exactly the property name of the incoming expression for which the object property value is being sent. More about this later.
   

3. ByValue with coercion: If the incoming object value (not object property) type is convertible, for example, an integer could be converted or coerced into a string value if that’s what the receiving parameter expects based on its parameter type definition then binding will still work.
   
4. ByProperty name with coercion: If the first match was by property name, then check to see if the incoming object type is convertible to what the receiving parameter has defined as it’s type.
   

Now that we have the breakdown of the processing order, we see that in our Stop-Service example, the match and subsequent binding will occur, based on the pipeline’s object value. This is because there is an instance of the ServiceController type, which is an object and not an object property coming through the pipeline.

More specifically, the resulting object received from the pipeline is a service object instance for the BITS service (Get-Service -Name BITS) that has a type of ServiceController. It is a single value, but can be converted to an array consisting of a single element. The receiving parameter type now matches sent object type, and so it binds or is associated with the object value, which is also be referred to as the argument in the Trace-Command output. This option was taken first because ByValue without coercion is evaluated highest in the processing order, so after parameter binding is successful, the receiving expression Stop-Service -PassThru -Verbose is then evaluated and executed. Of course, we just threw in the -PassThru and -Verbose parameters here to see what’s going on and get a bit more detail for the output.

This is a good start, but how can we trace the detailed activity for what transpires with parameter binding operations? Well, valued readers, I’m glad you asked.

Like this…

Trace-Command -Name ParameterBinding -Expression { Get-Service -Name bits -ComputerName 2012r2-ms | Stop-Service -PassThru } -PSHost -FilePath c:\pshell\labs_9\Stop-Service-PBv1.0.log

Here, the -PSHost switch parameter tells us to display the results on the screen and -FilePath is used to specify a file to log the same details.

Figure 7: The parameter binding process for Stop-Service

Inside the red outline shown in Figure 7, we first see the bind operation beginning for the pipeline object for the receiving Stop-Service cmdlet.

BIND PIPELINE object to parameters: [Stop-Service]

Next, the pipeline object type is evaluated as [System.ServiceProcess.ServiceController], but we can just exclude the class namespace, and call it ServiceController from here onwards.

PIPELINE object Type = [System.ServiceProcess.ServiceController]

The pipeline parameter value is then restored and because it matches the receiving parameter object type, which is ServiceController. The ByValueFromPipeline without coercion rule is applied first.

RESTORING pipeline parameter’s original values

Parameter [InputObject] PIPELINE INPUT ValueFromPipeline NO COERCION

Now we can bind the specific bits service object value instance to the Stop-Service’s InputObject parameter, but because it’s only a single service value, and an array of services is expected, the binding operation creates a new array and adds this single value of the bits service to it.

Bind arg [bits] to parameter [InputObject]

Binding collection parameter InputObject: argument type [ServiceController], parameter type [System.ServiceProcess.ServiceController[]], collection type Array, element type [System.ServiceProcess.ServiceController], no coerceElementType

Creating array with element type [System.ServiceProcess.ServiceController] and 1 elements

Argument type ServiceController is not IList, treating this as scalar

Adding scalar element of type ServiceController to array position 0

Ok, we’re almost finished with binding, but first we’ll validate that the parameter is not null or empty. This is necessary because it has already been defined as an attribute for the Parameter in PowerShell as part of the Stop-Service cmdlet source code.

Executing VALIDATION metadata: [System.Management.Automation.ValidateNotNullOrEmptyAttribute]

Are we there yet? …Yes, so finally, we can successfully bind the argument bits, now re-formatted as a ServiceController array type to the Stop-Service parameter -InputObject.

BIND arg [System.ServiceProcess.ServiceController[]] to param [InputObject] SUCCESSFUL

SET-SERVICE

Alright, now that we’ve looked at a ByValue example, let’s shift gears this time to the other pipeline input option – ByPropertyName. What’s interesting in this next example is that this parameter accepts pipeline input both ByValue AND ByPropertyName, so we’ll even get to do a bit of comparison as an added bonus.

Figure 8: Parameter attribute table for the Set-Service -Name parameter

And what would a discussion about binding be without a quick glance at another partial class diagram?

Figure 9: Class diagrams for Get-Service and Set-Service

First, we’ll execute the expression;

Figure 10: Trace-Command for Set-Service -Name ByParameterName

This command will select the bits service by it’s property name to set it’s StartupType from Auto to Manual. Notice the red outline area in Figure 10. After the first pipe operator, we single out ONLY name property of the bits ServiceController object, not the entire object, as in the first example with the Stop-Service cmdlet.

Here is the relevant part of the result of the Trace-Command, the rest is just ommited for brevity.

Figure 11: Partial Trace-Command output for Set-Service -Name ByParameterName

In Figure 11, if we look above the red outline, we’ll see that an attempt was previously made using the PIPELINE INPUT ValueFromPipeline NO COERCION rule to bind the incomming name property to receiving name parameter, but was skipped.

Parameter [Name] PIPELINE INPUT ValueFromPipeline NO COERCION

BIND arg [@{Name=bits}] to parameter [Name]

BIND arg [@{Name=bits}] to param [Name] SKIPPED

The rule is used as the first in the order of evaluation, however the pipeline value is not an object in this case, but a property called name having the value bits. The verdict therefore, is that a property can’t be bound to parameter by value, only objects can, so this rule doesn’t apply and must be skipped to process the next rule in the binding sequence.

As a convenience, here’s quick reminder list of the binding order again:

1. PIPELINE INPUT ValueFromPipeline NO COERCION
   
2. PIPELINE INPUT ValueFromPipelineByPropertyName NO COERCION
   
3. PIPELINE INPUT ValueFromPipeline WITH COERCION
   
4. PIPELINE INPUT ValueFromPipelineByPropertyName WITH COERCION
   

The red outlined area in Figure 11 shows that the name parameter of the Set-Service cmdlet will now attempt to accept pipeline input ByPropertyName without coercion, which is accepted and binding is successful. This is because the pipeline item is a property, and only a property can be bound to a parameter with rule 2 – PIPELINE INPUT ValueFromPipelineByPropertyName NO COERCION, objects can’t bind using this rule, only rules 1 and 3 apply to objects. Rules 2 and 4 apply to properties. The other reason it binds is because the property name is “name”, and the parameter name is also “name”. ByPropertyName means that these two have to match exactly in order to bind.

So what if for the name parameter, we send an object through the pipeline instead of a property this time, since the Set-Cmdlet -Name parameter accepts input both ByValue and ByPropertyName? Again, we’re still setting the StartupType of the bits service from Auto to Manual, but now we will select the service object itself, not the name property of the bits service object as we did in the previous section. Here is the command and corresponding output below.

Figure 12: Class diagrams for Get-Service and Set-Service

We can observe that the pipeline object type is string, which also matches the parameter type (see Figure 8), and we also will notice that rule 1 is evaluated:

Parameter [Name] PIPELINE INPUT ValueFromPipeline NO COERCION

However, because we have sent an entire object with the value “bits” and not a property of an object through the pipeline (figure 13), and because rule 1 – ByValueFromPipeline without coercion, only accepts objects that do not have to be coerced to have its type converted, then it meets all the criteria and will bind successfully.

If we were to summarize these concepts visually in terms of a process diagram, it may resemble:

Figure 6: Parameter binding process

See, I told you this would be illustrated, didn’t I? Notice that indexed items 04.02.01, 04.02.02, 08.00 and 04.02.02.02 all have [A], [B], [C], and [D] prefix designations respectively. This is just a simple hint of the binding order we discussed previously.

SUMMARY

The key points I’d like to reinforce are: First, to know what you are sending through the pipeline. Is it an object or a property of an object? If it’s a valid object, it will bind using rules 1 or 3 ([A] or [C]) depending on if it’s object type requires conversion or not. The sending object type must match the receiving parameter type also.

If the pipeline item is a property of an object and is a valid property, it will use either rules 2 or 4 ([B] or [D]). Both the property/parameter type and the property/parameter names on both sides of the pipeline must eventually match to satisfy binding.

Finally, it’s important to point out that when we refer to the sending expression on the left side of the pipeline, we call the pipeline items either objects or properties, but on the receiving side, these are bound to their corresponding parameters. Parameters are always on the right, while properties and objects traverse the pipeline from the left.

We hope this article has been helpful and would love to get your feedback, especially any stories of how you have been best able use this information. Stay tuned for more topics and happy scripting!

REFERENCES

1. about_Parameters (Get-Help about_Parameters -ShowWindow)
   

   https://technet.microsoft.com/en-us/library/hh847824.aspx
   

2. about_Pipelines (Get-Help about_Pipelines -ShowWindow)
   

   https://technet.microsoft.com/en-us/library/hh847902.aspx
   

   
   

Hi all! I am Bill Kral, a Microsoft Premier Field Engineer, here to give you the steps to convert your on-premise Federated domain to a Managed domain in your Azure AD tenant. In this case, we will also be using your on-premise passwords that will be sync’d with Azure AD Connect.

There are many ways to allow you to logon to your Azure AD account using your on-premise passwords. You can use ADFS, Azure AD Connect Password Sync from your on-premise accounts or just assign passwords to your Azure account. In addition, Azure AD Connect Pass-Through Authentication is currently in preview, for yet another option for logging on and authenticating.

Recently, one of my customers wanted to move from ADFS to Azure AD passwords sync’d from their on-premise domain to logon. So, we’ll discuss that here.

Let’s set the stage so you can follow along:

The on-premise Active Directory Domain in this case is US.BKRALJR.INFO

The AzureAD tenant is BKRALJRUTC.onmicrosoft.com

We are using Azure AD Connect for directory synchronization (Password Sync currently not enabled)

We are using ADFS with US.BKRALJR.INFO Federated with the Azure AD Tenant

1. First, insure your Azure AD Connect Sync ID has “Replicate Directory Changes” and “Replicate Directory Changes All” permissions in AD (For Password Sync to function properly). If you did not set this up initially, you will have to do this prior to configuring Password Sync in your Azure AD Connect.
   

1. You can check your Azure AD Connect servers Security log that should show AAD logon to AAD Sync account every 30 minutes (Event 4648) for regular sync. When you enable Password Sync, this occurs every 2-3 minutes.
   

1. On the Azure AD Connect server, run CheckPWSync.ps1 to see if Password Sync is enabled
   

   Note: Here is a script I came across to accomplish this. Copy this script text and save to your Azure AD Connect server and name the file with a CheckPWSync.ps1.
   

   —————————————- Begin Copy After this Line ————————————————
   

   Import-Module ADSync
   $connectors = Get-ADSyncConnector
   $aadConnectors = $connectors | Where-Object {$_.SubType -eq “Windows Azure Active Directory (Microsoft)”}
   $adConnectors = $connectors | Where-Object {$_.ConnectorTypeName -eq “AD”}
   if ($aadConnectors -ne $null -and $adConnectors -ne $null)
   {
   if ($aadConnectors.Count -eq 1)
   {
   $features = Get-ADSyncAADCompanyFeature -ConnectorName $aadConnectors[0].Name
   Write-Host
   Write-Host “Password sync feature enabled in your Azure AD directory: ” $features.PasswordHashSync
   foreach ($adConnector in $adConnectors)
   {
   Write-Host
   Write-Host “Password sync channel status BEGIN ——————————————————- ”
   Write-Host
   Get-ADSyncAADPasswordSyncConfiguration -SourceConnector $adConnector.Name
   Write-Host
   $pingEvents =
   Get-EventLog -LogName “Application” -Source “Directory Synchronization” -InstanceId 654 -After (Get-Date).AddHours(-3) |
   Where-Object { $_.Message.ToUpperInvariant().Contains($adConnector.Identifier.ToString(“D”).ToUpperInvariant()) } |
   Sort-Object { $_.Time } -Descending
   if ($pingEvents -ne $null)
   {
   Write-Host “Latest heart beat event (within last 3 hours). Time ” $pingEvents[0].TimeWritten
   }
   else
   {
   Write-Warning “No ping event found within last 3 hours.”
   }
   Write-Host
   Write-Host “Password sync channel status END ——————————————————- ”
   Write-Host
   }
   }
   else
   {
   Write-Warning “More than one Azure AD Connectors found. Please update the script to use the appropriate Connector.”
   }
   }
   Write-Host
   if ($aadConnectors -eq $null)
   {
   Write-Warning “No Azure AD Connector was found.”
   }
   if ($adConnectors -eq $null)
   {
   Write-Warning “No AD DS Connector was found.”
   }
   Write-Host
   

   —————————————- End Copy Prior to this Line ——————————————-
   

   As you can see, mine is currently disabled.
   

   1. Setup Password Sync via Azure AD Connect (Options)
      
      1. Open the Azure AD Connect wizard on the AD Connect Server
         
      1. Select “Customize synchronization options” and click “Next”
         
      1. Enter your AAD Admin account/ Password and click “Next”
         
      1. If you are only enabling Password hash synchronization, click “Next” until you arrive at the Optional features window leaving your original settings unchanged
         
      2. On the “Optional features” window, select “Password hash synchronization” and click “Next”
         

         Note: If you have previously selected other options, leave those as intended
         

      3. Click “Install” to reconfigure your service
         

   2. Restart the Microsoft Azure AD Sync service
      
   3. Force a Full Sync in Azure AD Connect in a powershell console by running the commands below
      
      1. Import-module ADSync
         
      2. Start-ADSyncSyncCycle -PolicyType Initial
         
   4. On your Azure AD Connect server, run CheckPWSync.ps1 to see if Password Sync is enabled
      

1. On your Azure AD Connect server, run TriggerFullPWSync.ps1 to trigger full password sync (Disables / enables)
   

Note: Here is a script I came across to accomplish this. Copy this script text and save to your AD Connect server and name the file TriggerFullPWSync.ps1. Update the $adConnector and $aadConnector variables with case sensitive names from the connector names you have in your Synchronization Service Tool.

—————————————- Begin Copy After this Line ————————————————

# Run script on AD Connect Server to force a full synchronization of your on prem users password with Azure AD
# Change domain.com to your on prem domain name to match your connector name in AD Connect
# Change aadtenant to your AAD tenant to match your connector name in AD Connect
$adConnector = “domain.com”
$aadConnector = “aadtenant.onmicrosoft.com – AAD”
Import-Module adsync
$c = Get-ADSyncConnector -Name $adConnector
$p = New-Object Microsoft.IdentityManagement.PowerShell.ObjectModel.ConfigurationParameter “Microsoft.Synchronize.ForceFullPasswordSync”, String, ConnectorGlobal, $null, $null, $null
$p.Value = 1
$c.GlobalParameters.Remove($p.Name)
$c.GlobalParameters.Add($p)
$c = Add-ADSyncConnector -Connector $c
Set-ADSyncAADPasswordSyncConfiguration -SourceConnector $adConnector -TargetConnector $aadConnector -Enable $false
Set-ADSyncAADPasswordSyncConfiguration -SourceConnector $adConnector -TargetConnector $aadConnector -Enable $true

—————————————- End Copy Prior to this Line ——————————————-

1. Now, we can go to the Primary ADFS Server and convert your domain from Federated to Managed
   
   1. On the Primary ADFS Server, import he MSOnline Module
      

      Import-Module MSOnline
      

   2. Connect to your AAD Tenant
      

      Connect-MSOLService -> Enter AAD credentials on the pop-up
      

      Note: You can also run from another server, but then need to set the context of what ADFS server you connect to. Ie: Set-MsolADFSContext -Computer <AD FS servername>
      

      I find it easier to do the Azure AD Connect tasks on the Azure AD Connect server and the ADFS/Federation tasks on the primary ADFS server.
      

   3. Confirm the domain you are converting is listed as Federated by using the command below
      

Get-MsolDomain -Domainname domain -> inserting the domain name you are converting. Ie: Get-MsolDomain -Domainname us.bkraljr.info

• Convert Domain to managed and remove Relying Party Trust from Federation Service. Now, you may convert users as opposed to the entire domain, but we will focus on a complete conversion away from a Federated domain to a Managed domain using on prem sourced passwords.
  

  Convert-MsoldomainToStandard -Domainname domain -SkipUserConversion $false -PasswordFile c:\domain_userpasswords.txt -> inserting the domain name you are converting and a password file name of your choice that does not currently exist. Ie: Convert-MsoldomainToStandard -Domainname us.bkraljr.info -SkipUserConversion $false -PasswordFile c:\usbkraljrinfo_userpasswords.txt
  

Here is where the, so called, “fun” begins. There is no status bar indicating how far along the process is, or what is actually happening here. Under the covers, the process is analyzing EVERY account on your on prem domain, whether or not it has actually ever been sync’d to Azure AD. If an account had actually been selected to sync to Azure AD, it is converted and assigning a random password. That is what that password file is for… Also, since we have enabled Password hash synchronization, those passwords will eventually be overwritten.

What does all this mean to you? You must be patient!!! For an idea of how long this process takes, I went through this process with a customer who had a 10k user domain and it took almost 2 hours before we got the “Successfully updated” message. Therefore, you can expect an approximate processing rate of 5k users per hour, although other factors should be considered, such as bandwidth, network or system performance. That doesn’t count the eventual password sync from the on prem accounts and AAD reverting from “Federated” to “Not Planned” or “Not Configured” in the Azure Portal. So, just because it looks done, doesn’t mean it is done.

1. On the Azure AD Connect server, run TriggerFullPWSync.ps1 to trigger full password sync
   

1. On the ADFS server, confirm the domain you have converted is listed as “Managed”
   

   Get-MsolDomain -Domainname domain -> inserting the domain name you are converting. Ie: Get-MsolDomain -Domainname us.bkraljr.info
   

1. Check the Single Sign-On status in the Azure Portal. It should not be listed as “Federated” anymore
   

1. Logon to “Myapps.microsoft.com” with a sync’d Azure AD account. There should now be no redirect to ADFS and your on prem password should be functional…… Assuming you were patient enough to let everything finish!!!
   

1. The Azure AD Connect servers Security log should show AAD logon to AAD Sync account every 2 minutes (Event 4648)
   

That should do it!!! Bottom line… be patient… I will also be addressing moving from a Managed domain to a Federated domain in my next post, as well as setting up the new Pass-Through Authentication (PTA) capabilities that are being introduced into Azure AD Connect in future posts. Thanks for reading!!!

Hi everyone. This is Michael Rendino, a Premier Field Engineer from Charlotte, NC and former member of the CTS networking support team. With my networking background, I have spent years reviewing network captures. One thing I always run into with my customers is that they often don’t know the best or easiest solution to get a network capture. There are many solutions you can use and choosing the right one often depends on the scenario. While colleagues have created blogs on getting a trace with a single tool, I wanted to provide a location that someone can bookmark to be a single set of instructions for a number of solutions. Please note that when reviewing traces, you can use one or more of these tools and aren’t necessarily tied to what was used to collect the trace.

The Options

First, let’s cover each of the tools that can be used to collect a network trace, in order from older to newer

1. Network Monitor 3.4 (Netmon) – https://www.microsoft.com/en-us/download/details.aspx?id=4865 (NOTE: Network Monitor is no longer under active development)
2. Wireshark (v 2.2.2 as of 11/16/16) – https://wireshark.org/#download
3. Netsh Trace – built-in to operating system
4. Microsoft Message Analyzer (MMA) (v 1.4 as of 6/13/16) – https://www.microsoft.com/en-us/download/details.aspx?id=44226

Comparison

*MMA gives you the ability to setup and collect captures from multiple systems (e.g. client and server) using a single client.

**Wireshark can capture X files of Y size and roll as needed. Network Monitor can capture a chained set of files, but will not overwrite old files and can only be done via command line.

***Network Monitor is currently the only supported tool to install on an Advanced Threat Analytics server.

The basics

Right off the bat, it should become apparent from the above table that one of these options — netsh trace – has one benefit over the others as it is ready to go without any further installation. It does require an elevated command prompt to run, but nothing beyond that. In many environments where change control is strict and the necessary software hasn’t already been installed, this often makes it the only option. Another item to note is that “netsh trace” is a command-line tool and the other three each have command-line alternatives for network captures. Getting a trace that way is often beneficial to eliminate the overhead of the GUI showing data and refreshing in real-time. As pointed out in the table, netsh traces can be opened with Netmon or MMA, but not Wireshark.

When collecting a short-term, simple trace for a set amount of time, there is not much of a difference in capturing with any of the tools. Each will let you create a trace, capture multiple NICs, and define capture rules (typically, please don’t as you may filter out something important). One item to note is regarding promiscuous mode. Be sure to enable it when you are doing port mirroring to allow a computer to capture all traffic on the port — not just the packets destined for its own MAC address.

Requirements

The only one with special requirements is Message Analyzer as certain features (like remote capture) are only possible on Windows 8.1, Server 2012 R2 and newer operating systems.

Instructions

And now the part you’ve been anxiously waiting for, the steps for each solution. I’ll provide both GUI and command line (where applicable) for getting a basic capture.

Network Monitor

GUI

1. Launch Network Monitor. If you need promiscuous mode to capture traffic that is destined for machines other than the one where the capture is running, check the P-Mode box first, and then click “New Capture.” NOTE: You can select and deselect network adapters if you prefer, but these were the “quick” instructions, remember?

1. Once you have the new capture open, simply click “Start” to begin tracing and “Stop” after you have captured the data you need. You can then click “Save As” to save the trace before starting your analysis.

1. If you have applied a display filter or have selected certain frames and only want to retain that subset in a smaller file, you can save just those frames to a file if you wish:

Command Line

1. Open an elevated command prompt for all of the following steps.
2. Decide if you want to create multiple chained files of a particular size or if you want a single capture file with a max size of 500 MB.
3. Run one of the following commands
   1. For chained files – “nmcap /network * /capture /file %computername%.chn:100MB”

      1. This command will create a series of 100MB captures in the current folder (adjust the size as you wish)
         

   NOTE: Monitor the volume where the traces are being stored to ensure that it doesn’t consume too much diskspace. Due to Wireshark’s ability to have a set number of files, if you are unsure how long the trace must run, Wireshark may be a better solution.

   1. In the above example, the name of the computer will be the name of the files, but you can replace %computername% with whatever you want.
      

   2. It will capture all network interfaces in the computer.
      

   3. If you wish to store the captures in a different folder, either run the command from another folder or put the full path before %computername%.chn.
      

   1. For a single file – “nmcap /network * /capture /file %computername%.cap”

      1. As previously noted, this command will create a single capture with a max size of 500 MB in the current folder.
         

      2. In the above example, the name of the computer will be the name of the files, but you can replace %computername% with whatever you want.
         

      3. It will capture all network interfaces in the computer.
         

      4. If you wish to store the captures in a different folder, either run the command from another folder or put the full path before %computername%.cap
         

        NOTE: You must keep the command window open while the capture runs.

1. Once the issue reproduces, use Ctrl+C to stop the capture
   

Wireshark

GUI

Single File

1. Launch Wireshark and select the NIC(s) you want to capture.

1. Click the blue shark fin icon to start the trace.

1. After reproducing the issue, to stop the capture, click the red stop icon.

1. Save the file. Note that if you save it in .pcapng format (the default), it can’t be opened in Network Monitor but can be opened in MMA.

Chained Files

1. If you want to capture multiple files, select Capture – Options (or Ctrl+K):

1. Select the NIC(s) you want to capture.

1. Click on the Output tab, enter a path and name for the files. The name will be appended with a file number, the date and time. Select the “Create a new file automatically after” option and then choose a size for each file and for ring buffer, enter the number of files you want to create. In the image below, ten 100 MB files would be created and the oldest file would be overwritten until the capture is stopped.

1. If you wish to reduce the impact on the computer where the trace is being collected, click the Options tab, then deselect the “Update list of packets in real-time” and “Automatically scroll during live capture” options.

1. Click the Start button to start the trace. After reproducing the issue, click the red stop icon to terminate the trace.

Command Line

1. Open an elevated command prompt and switch to the Wireshark directory (usually c:\program files\Wireshark).
2. From the Wireshark directory, run “dumpcap -D” to get a list of interfaces. You’ll need the interface number in the command to start the capture.
3. Run “dumpcap -b filesize:100000 -b files:10 -i <interface number from step 2> -w c:\temp\%computername%.pcap”. You can use a different path and filename if you wish. This command will create ten rolling 100 MB captures until the trace is stopped. Adjust those numbers as desired

NOTE: You must keep the command window open until the problem returns.

1. Once the issue reproduces, use Ctrl+C to stop it.

Netsh Trace

One extra cool thing about “netsh trace” is that by default, it creates a .cab file along with the trace, that contains a bunch of helpful diagnostic information like various settings, event logs, and registry keys.

1. Open an elevated command prompt and run the command “netsh trace start capture=yes tracefile=c:\temp\%computername%.etl.” You can close the command prompt if you wish.

   NOTE: If you want to capture an issue that occurs during boot or logon, use the “persistent=yes” switch. By doing that, the trace will run through the reboot and can be stopped after logon with the command in step 2. Also, if you don’t want the .cab file, simply add “report=no” to the command in step 1.

2. Once the issue reproduces, open an elevated command prompt and run the command “netsh trace stop.”

Microsoft Message Analyzer

GUI

MMA is the most powerful and flexible of the network capture tools and fortunately, is the easiest for getting a trace.

1. Run MMA as an administrator.
2. Under “Favorite Scenarios,” click “Local Network Interfaces” if you are running Windows 8.1 or 2012 and newer. If you are running Windows 7 or 2008 R2, choose the “Loopback and Unencrypted IPSEC” option. The session will be created and the capture will start.

1. Once you have reproduced the issue, click the blue stop icon:

Command Line

Command line captures with Message Analyzer are done with the PowerShell PEF module. The fact that it uses PowerShell makes it extremely powerful and flexible for setting up the capture. However, this article is for basic captures so following is the example from https://technet.microsoft.com/en-us/library/dn456526(v=wps.630).aspx. You can always save the following as a script.

$TraceSession01 = New–PefTraceSession –Mode Circular –Force –Path “C:\Traces\Trace01.matu” –TotalSize 50 –SaveOnStop
Add–PefMessageSource –PEFSession $TraceSession01 –Source “Microsoft-Pef-WFP-MessageProvider”
Start–PefTraceSession –PEFSession $TraceSession01

The above script will create a 50 MB capture, overwrites an existing file in that path if it exists and saves the file once the script is stopped.

Conclusion

As you can see, the tools and methods available to collect a network capture are numerous, but this variety enables you to get traces for any situation. You may eventually get to prefer a particular tool for capturing traces and yet another to review them or use more than one to view the same trace. I highly recommend that you become familiar with them all and run through the process prior to the time when you actually need to get a trace. Again, these instructions are basic ones just to get you all information from the computer where the trace runs. There’s a plethora of options and capabilities for the tools, so feel free to dig in! I’ll include some helpful links below so you can continue your learning. Good luck!

Additional Information

1. To learn more about your nmcap options, enter “nmcap /?” or “nmcap /examples”
2. Wireshark training can be found at https://www.wireshark.org/#learnWS.
3. For more information on Message Analyzer, check out the blog at https://blogs.technet.microsoft.com/messageanalyzer/.
4. Message Analyzer training videos can be found at https://www.youtube.com/playlist?list=PLszrKxVJQz5Uwi90w9j4sQorZosTYgDO4.
5. Message Analyzer Operating Guide – https://technet.microsoft.com/en-us/library/jj649776.aspx
6. Information on the Message Analyzer PowerShell module can be found at https://technet.microsoft.com/en-us/library/dn456518(v=wps.630).aspx.
7. Remote captures with MMA – https://blogs.technet.microsoft.com/messageanalyzer/2013/10/17/remote-capture-with-message-analyzer-and-windows-8-1/

NOTE: This articles references a 3rd party product. 3rd party products are not supported under any Microsoft standard support program or service. The information here is provided AS IS without warranty of any kind. Microsoft disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of these solutions remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use this documentation, even if Microsoft has been advised of the possibility of such damages.

Michael Rendino

Senior Premier Field Engineer

Hello everyone, my name is David Loder, long-time reader, first-time PFE blogger, based out of Detroit, Michigan. And I have a confession to make. I hate servers. Or more precisely I hate logging in to servers.

If you’ve been administering Windows Server for any length of time, you’re hopefully aware of the Remote Server Administration Tools (RSAT) package. RSAT provides the server management tools for the Windows client Operating Systems like Windows 7 or Windows 10 so that you don’t have to RDP into a Domain Controller just to run the Active Directory Users and Computers GUI.

The RSAT package is OS specific. There is a different package for each client OS that corresponds to the matching server OS. There are links to each of the packages from the main RSAT Knowledge Base article at https://support.microsoft.com/en-us/kb/2693643.

If you are a typical enterprise customer you’ve most likely been running Windows 7 as your client, and are just now starting to explore Windows 10.

Because of the OS specifics, RSAT for Windows 10 is tied to Windows Server 2016. The Tech Previews all had a preview version of RSAT available for them. Since Windows Server released, only the RTM version of RSAT for Windows 10 is available for download. On the download page, you’ll currently see it as v1.2 with the WindowsTH-RSAT_WS2016-x64.msu name or WindowsTH-RSAT_WS2016-x86.msu if you’re planning on running 32-bit Windows 10 for some reason. I’m personally surprised we’re still making x86 versions of RSAT available.

The first caveat you’ll hit is that Windows Server 2016 RTM corresponds to Windows 10 build 1607, also known as the Anniversary Update. If you are running one of the earlier releases of Windows 10 (builds 1511 or 1507), many of the RSAT tools from the current v1.2 download will not work. For example, if you open Active Directory Users and Computers and look at the properties of a user, most of the tabs will be missing, including the General tab. Instead you’ll get a view that looks like this.

Or if you open the newer Active Directory Administrative Center, then try to open a user’s properties, the console will crash.

To avoid these problems, you must be on Windows 10 build 1607.

When you upgrade from an earlier build to 1607, RSAT will not remain installed. You’ll have to reinstall after upgrading. And you’ll be treated to the expected results within ADAC and ADUC.

One nice thing about RSAT for Windows 10 is that you no longer have to enable the tools after installing the package. All the tools are automatically enabled. Windows 7 required installing RSAT and then remembering where to find the GUI to enable the tools. This behavior is back to how Windows 2000/3 worked. It was re-introduced with RSAT for Windows 8, but I can count on one hand the number of admins I’ve seen running that platform in the field.

Here is the full list of GUI tools that are added to the Start Menu Windows Administrative Tools folder:

Active Directory Administrative Center

Active Directory Domains and Trusts

Active Directory Module for Windows PowerShell

Active Directory Users and Computers

Certification Authority

Cluster-Aware Updating

DFS Management

DHCP

DNS

Failover Cluster Manager

File Server Resource Manager

Group Policy Management

Microsoft Azure Services

Network Load Balancing Manager

Online Responder Management

Remote Access Management

Remote Desktop Gateway Manager

Remote Desktop Licensing Diagnoser

Remote Desktop Licensing Manager

Routing and Remote Access

Server Manager

Shielding Data File Wizard

Template Disk Wizard

Volume Activation Tools

Windows Server Update Services

The full list of features that are enabled are:

Feature Administration Tools

BitLocker Password Recovery Viewer

DCB LLDP-Agent Management Tools

Failover Clustering Tools

Group Policy Management Tools

IP Address Management (IPMA) Client

Network Controller Tools

Network Load Balancing Tools

NIC Teaming Tools

Shielded VM Tools

Role Administration Tools

Active Directory Certificate Services Tools

Certification Authority Tools

Online Responder Tools

AD DS and AD LDS Tools

Active Directory Module for Windows PowerShell

AD DS Tools

Active Directory Administrative Center

AD DS Snap-ins and Command-line Tools

DHCP Server Tools

DNS Server Tools

File Services Tools

Distributed File System Tools

FSRM Management

NFS Administrative Tools

Storage Replica Administrative Tools

Remote Access Management Tools

DirectAccess Server Management Tools

Remote Access module for Windows PowerShell

Remote Desktop Services Tools

        Remote Desktop Gateway Tools

        Remote Desktop Licensing Diagnoser Tools

        Remote Desktop Licensing Tools

        Volume Activation Tools

Windows Server Update Services Tools

API and PowerShell cmdlets

User Interface Management Console

Server Manager

You may notice a few differences from previous iterations of RSAT. Hyper-V is now built in to Windows 10, so the Hyper-V tools are their own features to be turned on or off and are not part of RSAT anymore. The DCB LLDP-Agent Management Tools includes the NetLldpAgent PowerShell Cmdlets. The Shielded VM Tools includes the Provisioning Data File Wizard and the Template Disk Wizard. To find out more about how these tools help create Shielded VMs you can read Dean Wells’ blog post on Creating Shielded VMs Step by Step. Finally, the Microsoft Azure Services tool is simply a link to get you started with understanding the Operations Management Suite.

If you’ve gotten your RSAT fix for now, and are wondering what’s next, you might want to spend some time reading up on Microsoft’s next generation of server management tools. We now provide an Azure-hosted management portal called Server Management Tools. It uses a gateway located on-premise to proxy between your on-premise servers and Azure so that you can securely administer your servers from anywhere on any device. Look through their blog at https://blogs.technet.microsoft.com/servermanagement if you want to know more.

Thanks for spending a little bit of your time with me.

-Dave

#proudmicrosoftemployee

Hey everyone! PFE Tim Beasley here coming to you live from the warm, cozy sands of Bora Bora…Pfft yeah. I wish! … No I’m in Missouri…where it’s miserably winter outside. But I digress, I am writing this post to hopefully shed some light on a bizarre issue I recently faced at one of my dedicated (DSE) customer sites. I’d like to consider myself one that’s experienced as I’ve been working with Microsoft technologies for over 25 years now. Yet, I for one have never encountered this particular situation, so sit back, grab some popcorn, and hold on…because this is about to get real people.

During one of my regular DSE visits to my client, I was following up with what occurred that caused a Severity A support case to be opened. While gathering information and details, I was told “We had static reverse DNS records vanish.” I was like…Say what? Huh?! How in the world do STATIC records just vanish without someone deleting them?! Needless to say, I had them walk me through every step they took from beginning to end…and not one mention of someone deleting static records. Yet they swore up and down (literally) that they bloody well vanished, which is why they had to restore the original reverse zone from backup.

Okay, I’ll be your huckleberry…

Scenario:

Imagine finding yourself as an IT administrator faced with over 50,000 reverse DNS records that are placed comfortably in one single, large, super zone. For example’s sake, let’s say it’s 10.in-addr.arpa which happens to be an AD-integrated zone. Normally this is totally fine and actually recommended to do from our standpoint as it’s easier to manage. (Here’s a blog post on how to consolidate multiple reverse DNS zones by “GOATEEPFE” Ashley McGlone, in case you’re interested.) However, a decision is made to break up that super zone into smaller reverse zones for reasons that are, well, whatever that reason may be.

There’s a maintenance window coming up, and you’re probably thinking…”Okay, let’s create some smaller AD-integrated zones of the larger one.” But being a safe IT admin, you want to make sure you have a rollback plan in the event something unexpected happens, as there’s a lot of applications/devices out there that rely on reverse DNS. What do you do? You want to take a backup of the existing super zone before you start? Good idea to be safe. Also, you think to yourself…”I’ll just create the smaller zones, and leave the big one too…that way I can simply delete the zones I create if something goes wrong in the event I need to revert my changes.” -The plot thickens…

Now it comes time for the actual work to be performed. New zones are created to match existing network blocks (let’s say 50 of them or so), and the original 10.in-addr.arpa super zone is left intact. You now watch some of the new zones start to get populated with reverse DNS records (PTRs) as registrations are renewed. You think this is a “mic drop moment”…and walk away exclaiming “SUCCESS! We did it! Pats on the back all around!” –What really happened in this particular case, is a little bomb just got triggered for devices and applications that rely on reverse DNS records, that just so happen to be statically configured…*gasp*

Shortly after you head home for the night, reports start coming in of some devices and applications aren’t working. (Imagine that.) Some initial investigation reveals that the devices/applications that are failing rely on reverse DNS records. Now what? Rollback plan! “Well, let’s undo what we did and go back to the original 10.in-addr.arpa zone that’s still there!” “Sounds good!” “Okay go!” You then begin to remove all the reverse DNS zones that were created, and a sigh of relief is had by all.

But wait, problems still exist! The same devices and apps still aren’t working? Say it isn’t so! The reverse zones were deleted from the environment, looking at the DNS management console you can see they are gone, AND you can see that the original zone is there. You undid what you originally did! You even have other machines working fine and able to do reverse lookups without a problem. What is going on here?! So why are those certain devices and apps not working as expected? You go look in the original reverse DNS zone of 10.in-addr.arpa…and the STATIC reverse DNS entries that correspond to the same devices/apps…are…gone. *gasp again*

Baffled beyond belief, you question why? “How is this happening? What do we do now?!” Okay calm down…you remember the backup of the original reverse zone you took initially? Lightbulb! “Let’s just restore the original zone file, cycle services, and the records will come back.” Believing you know the correct method of restoring an AD-integrated zone, you then stop the DNS server service on one of the DCs, copy the backup file to C:\Windows\System32\DNS of a DC, rename it accordingly to 10.in-addr.arpa.DNS and then start up the DNS server service again. You go look at the zone in the DNS management console and the records aren’t there. You hit refresh. Again…and again…still not there. ACK! Panic temporarily ensues, judgement is clouded and you cycle DNS services again a few more times. Records are still not there! However, event logs show 4004, 4013, and 4015. But quick research shows those can be safely ignored from some online posts. (hint: that’s not the correct method for restoring an AD-integrated zone…the correct method is near the bottom of this blog.)

Ready to call for help? Or do you savvy IT/DNS admins out there think you know the answer and what to do at this point? *grin* My customer ended up calling into support at this point and opening up a SEV A case as multiple services were impacted. After many hours on the phone, CSS was able to finally get the records that were stored in AD, along with the backup file, to repopulate the original zone file across the DNS servers. However, the damage was done. From what I gathered, some static entries were also tombstoned at some point in time as well. By the time everything was restored, my customer experienced over 42 hours of downtime for those reverse records, which meant some services were hindered. My customer was lucky it only impacted a certain number of applications, but it could have been much, much worse had they experienced an enterprise-wide DNS failure.

If you’ve read this much you’ve noticed I still haven’t revealed any answers yet. Hopefully you haven’t ever experienced anything like this…but if you do, keep reading to figure out how to avoid some panic. Or, if you’re just wanting the nitty gritty, skip to the bottom. Let’s get to it!

The initial mistake here is believing that the original zone file had to be broken up into smaller ones. Had they followed recommended practices, this entire debacle could have been avoided. Point is, if you have a large super zone for reverse DNS records…leave it alone! And if you have tons of reverse zones, look to consolidating them following this. But if you insist on breaking things up into smaller reverse zones, you should watch the way you do it, especially if static records are involved. Okay, off my soapbox now.

I took the liberty of picking my jaw up off the floor when the customer told me what happened and everything that they went through. I had them send me the original zone backup file they used as well thinking there might be something strange in it. So, I had the zone backup file, along with my trusty lab machine, and got to work. Here come the screenshots!

Now let’s examine some of the details here. After all, the devil is in the details right? Make note the original 10.in-addr.arpa zone is an AD-integrated zone. No problem there, but as we know zone records for AD-integrated zones are stored in various AD partitions depending on how they are configured.

• Default Domain partition : “All domain controllers in the Active Directory domain”
• DomainDNSZones partition (Application partition) : “All DNS servers in the Active Directory domain”
• ForestDNSZones partition (Application partition): “All DNS servers in the Active Directory forest”

Additionally the new reverse zones created are also AD-integrated. Again, no issue. Or, is it? It is when it comes to the recovery method mentioned in the scenario above…I’ll get to that shortly. Remember the entries that disappeared were STATIC entries. Meaning someone manually created them in the original reverse DNS zone, and static entries are always well…static. That said, there are multiple ways various types of records can mysteriously “poof” away, such as duplicate zone creations, misconfigured scavenging settings, etc. (read more here) but this little particular nugget appears to encompass something entirely different. And so, I followed their described steps my customer took during this unfortunate event in my lab:

Please note my customer’s environment is WS2008 R2, but I used WS2012 R2 in these screenshots, however I also did these same steps in my lab using WS2008 R2 and the results were exactly the same.

I began with creating a fresh 10.in-addr.arpa reverse zone that is AD-Integrated, set to replicate between all DCs in the DOMAIN (DomainDNSZones partition in AD), along with a few static entries:

Figure 1

And for good measure, here they are reflected in ADSIEdit:

Figure 2

Okay good. That’s done, no problem there. I can do reverse lookups and records resolve no problem. Next, I’ll simply do what my customer did, and create 3 reverse zones that correspond to the subnets of the static entries I created.

Figure 3

This is the point in the scenario where it became a “mic drop moment” and the IT crew left the building. All looks good right? At initial glance, you might think so. But let’s take a closer look…

In each of the “new” reverse zones I created, you will see empty zones illustrated in the below screenshot. Each one only contains an SOA and NS record…and that’s it. Oh, they will get populated with PTRs when clients start to re-register up, but until that happens, they’ll remain empty.

Figure 4

Now here comes the pain!

Check out what the original 10.in-addr.arpa zone file looks like in DNS manager after a refresh…

Figure 5

Where in the world did the static entries go?! Hmmm, what are those folders? Are they in those little subfolders that got created?? Let’s look…

Figure 6

Ahhh…what about in AD? Surely they are in there! Take a gander…

Figure 7

Hurray! There they are along with the new zones…but…how come the static entries aren’t in DNS Manager? Also, when testing reverse lookups now, things are failing! The static entries that were resolvable before, are now no longer able to be found by the system even though they are in the AD partition. Chaos ensues….

Figure 8

ROLLBACK TIME:

Okay, let’s undo what I did initially and simply delete the reverse zones, and try reverse lookups using nslookup again:

Figure 9

Figure 10

Still no dice. But the reverse zones I created earlier are gone, effectively undoing what I did before right? Or…is it?

Diving a little deeper into the situation here, let’s cycle the DNS server service just to kick it and see if that helps. Hmmm….nope. Same result, no reverse lookup resolution. Let’s check event logs…DING! A CLUE!

Figure 11

Thankfully we have our first insight as to what’s going on. Event 4010…The system can’t create a resource record for the missing static entries. Wicked. You might be wondering why? The new reverse zones are there that correspond to the static entries that were created. And, the static entries still exist in AD. However, this particular event error indicates that ADDS isn’t responding to requests from the DNS Server service. Some of you might have come across this little nugget when migrating the _msdcs zone during a domain upgrade…(sound familiar?).

However, in the situation above they witnessed events 4004, 4013, and 4015. More often than not, this indicates that the “preferred” or primary DNS server in TCP/IP properties of the NIC on the DNS server (or DC) is pointing to itself. Ultimately when services start, AD can’t start because it’s hung up waiting on DNS to start, and because AD isn’t running, DNS can’t load the zones from AD. Ugly cycle really and causes unnecessary delays… It’s a good practice to configure all DCs to use the PDC emulator as primary DNS server (at the very least another DC other than itself), and then itself as secondary to avoid that. This is why in my lab environments I never saw the same events my customer did, as I followed this old practice, which happens to streamline the DNS infrastructure and allows for easier troubleshooting (not to mention decommissions and additions to the infrastructure).  You will hear various ways that you should configure your DNS infrastructure, but I try to look at DNS with the KISS philosophy, because overly complicating things unnecessarily can turn into a quick mess.

Now, each of those subfolders in the original 10.in-addr.arpa zone are sometimes referred to as “delegated subfolders.” Take notice they got created the instant I configured the new reverse zones. They represent what servers have the authority or permission to create records. If you scroll back up and look at the figure that shows the contents of the subfolder, you’ll see a single NS record of the DNS server I used in the lab. Great! Now what? Well, what happens when we delete those delegated subfolders and cycle the DNS Server service? Hold on to your seats!

Figure 12

Look! The 4010 errors are clear…and…dun dun dunnnnnnnnnnnn!

Figure 13

The static entries are back (pulled from AD no less), reverse resolution is working again, and everything is hunky dory once more!

BUT WAIT! Hold the phone…this is not how the scenario above was described!!!

Exactly! This is how I came to resolving the problem in my lab environment, and frankly what my customer should have done to correctly rollback their environment too. What my customer ended up doing compounded the problem significantly. I’ll explain. When they deleted the 50 or so reverse zones from DNS, that’s all they did before trying to restore their original zone from a backup. No one bothered to look at the subfolders that got created when they built out the additional reverse DNS zones! Additionally, they could have also avoided this “big nasty” had they manually recreated the static entries in the new zones without deleting anything. But that would have required some due diligence and a thorough discovery first before making drastic changes. Hint Hint! Don’t let someone outside your org that doesn’t know the environment implement major config changes without knowing exactly what they are getting themselves into…Yes, I’m one who tells it like it is. J

When you have reverse DNS zones that are smaller, aka more specific to a smaller subnet, like a /24 vs. a /8 subnet…then the DNS server will process name resolution requests to those more granular zones vs. the larger one. Plus, the delegated subfolders that get created refer clients back to the specific nameserver(s) that manage those subzones. Notice that there was only one NS record in the subfolder above in Figure 6? This means that when those new reverse zones were created to break up the larger one, the DNS server would process lookup requests by the referrals from the delegated subzone record to the NS server listed there, then on to those newer, more specific zones. Once it got to the newer zones, as the static records weren’t there…then reverse lookup fails. Hopefully that makes sense to you all.

Feel free to lab it up on your own and test various scenarios. Watch how the simplest action can either save or wreck an environment. For example test this in your own labs…Create a static entry that is in the original 10.in-addr.arpa. Then manually create a newer reverse zone…and then go delete the delegated subfolder that was created in the 10.in-addr-arpa zone. Does name resolution for that entry still work? Yep, as it should…but what if you delete the newer reverse zone again? Reverse resolution now fails for that entry, AND…the static entry won’t show back up in the 10.in-addr-arpa zone within the DNS MMC! But it’s still there in the AD partition if it’s an AD-integrated zone! At this point however, the original static entry is now appears to be TOMBSTONED in AD…see?

Figure 14

Don’t let this fool you however, as it’s not actually AD tombstoned. The dNSTombstoned attribute means that the record was deleted from the DNS Management console MMC or simply scavenged, yet the object still exists in AD. However, DNS.EXE will no longer load the record. It’s basically giving the appearance the object was deleted from the MMC, but the reality is it was only hidden from DNS.EXE. If you see the “isDeleted” attribute containing information, then that means it’s actually tombstoned in AD.

You can also refer to this TechNet article which shows you how to track for deletion of DNS records for a more proactive approach to your environment and also quotes the following:

“When Active Directory deletes an object from the directory, it does not immediately remove the object from the database. Instead, Active Directory marks the object as deleted by setting the object’s isDeleted attribute to TRUE, stripping most of the attributes from the object, renaming the object, and then moving the object to a special container in the object’s naming context (NC) named CN=Deleted Objects. This object is called a tombstone and is used to replicate the object’s deletion throughout the Active Directory environment. Over time (default 60 days), the tombstone is removed and the object is truly gone from AD. DNS objects, however, have their own process of deletion – once the DNS zone is integrated in the Active Directory, all the DNS records become Active Directory objects but they get an attribute called “dNSTombstoned” attached to them.

A DNS record gets removed by either of the following methods:

• Scavenging
  
• Manual deletion
  
• When it gets a valid TTL update with TTL=0
  
• An LDAP delete command using interfaces such as ADSIEDIT or LDP
  

If the DNS record is getting deleted by any of the first 3 ways then the value of the dNSTombstoned attribute attached to it will become “TRUE”. In this scenario the records will still exist in Active Directory but DNS.exe will not load them in the MMC. This is because for DNS they are deleted, but for Active Directory they still exist as a valid AD object. We can still see them using ADSIEDIT. When the record is in this state in the Active Directory the value of dNSTombstoned can change to “FALSE” either when the host machine/DHCP sends an update for the record or by creating another record with the same name manually. When this happens, DNS.exe will start loading the record again in the MMC. If the DNS record is being deleted by the 4th method or if the record stays in the state of dNSTombstoned=TRUE for more than 7 days then it will be tombstoned (AD tombstoned) like any other AD object.”

I know what you’re probably thinking, I thought the same thing…”Can’t we just manually change the dNSTombstoned attribute back to ‘FALSE’ and it’ll reappear in the MMC?” Well for grins I tried it myself, and the answer is NO. To get it working again, the record must be either restored from backup, the machine/DHCP sends an update for the record, or the record must simply be manually recreated. Manually recreating the record triggers DNS to update the record attributes in AD. Only then will the value return as FALSE and show back up in the MMC of DNS Manager.

PRO TIP: If the DNS record is either “dNSTombstoned” or AD tombstoned (aka “isDeleted”), then you can use “repadmin /showobjmeta,”
which will show you the time/date that each attribute for the object was created, edited, or marked for deletion. This also shows the originating source DC of this change. Handy little command when troubleshooting.

This is starting to now look like what happened at my customer’s environment based on what information I could gather, as unfortunately I wasn’t directly involved.

Now let us skip to the “recover from the backup” part that was mentioned in the scenario. The method they chose for recovery was another mistake to try for an AD-integrated DNS zone. AD-integrated zones don’t pull the records from a file, they pull them from AD. Simply stopping the DNS Server service, placing a .DNS file inside the C:\Windows\System32\DNS directory and restarting the service will NOT work when you’re talking about AD-Integrated zones. Even running DNSCMD commands to add a zone with the /dsprimary flag ignores the files as well. What WILL work with an exported DNS file is creating a new zone using the DNS file as a standard primary zone…THEN converting it to an AD-Integrated zone afterwards.

Related to the scenario above, there were several recovery options that include but are not limited to:

• Delete the existing reverse zones from DNS and AD…all of them…restore the original backup file to a new standard primary zone, validate the records were all there, convert it to an AD-Integrated zone…and then wait for replication to complete. (In my lab with their backup file of the zone using WS2008 R2 it took roughly 37 seconds to replicate all 50k+ records with 2 DCs/DNS servers configured with minimal resources.) This depends on convergence time in your environment, server hardware, etc.
• Restore from a system-state backup using Directory Services Restore Mode if DNS is running on a domain controller. Unless of course there is no valid backup…hopefully that’s not reality for you.
• Manually recreate the missing static records in each of the new zones…this of course assumes you have the details of each missing record from the due diligence I hinted at earlier…which wasn’t the case…and it’s also time consuming.

CONCLUSION:

To sum things up, this unfortunate scenario that plagued my customer for well over 40 hours could have been avoided from the get-go. Again, if there’s a large super-zone there’s no need to break it up. However, if you’re facing a potential resume generating event, know this: at the heart of the issue lies delegated folders that get created automatically when you try and split up a larger zone into smaller ones. The creation of this delegation record and its affect is not at all obvious.  Most DNS admins are used to creating delegation so it’s odd that it shows up all its own.  Then quick course to resolution is to delete the subdomain, delete the delegation and reload the zone.

So, if you have decided to try and break up a super-zone and have issues…first verify that the delegated subfolders got created in the main AD-integrated zone after you added smaller AD-integrated zones. Delete the subdomains you created, delete all the delegated folders that got created, and reload the original zone. If some records are missing from DNS management console, then verify they exist in AD. If they do exist in AD, you might have to wait a bit for them to show back up in the DNS console. If they are missing entirely, then I would go down the road of using the backup file of the original zone. If you don’t have a backup file, you’re then limited to a Directory Services restore, or manually creating static records.

I hope that this blog post helps you all out there don’t fall into this trap…but if you do find yourself amongst your peers freaking out about vanishing static reverse DNS records, now you can calmly reply “I got this” and be the hero. Thanks for reading and have a blessed day!

Stephen Mathews here; now tell me, do you have a domain controller you’re afraid to turn on? Maybe it lost power and nobody noticed for a couple months or you don’t trust your AD restore procedures; perhaps a troubled DC keeps deleting your DNS records – whatever happened, I’m going to show you how to power it on without allowing it to replicate.

The end goal is to disable replication on the troubled DC. This is done very simply by using the repadmin commands:

repadmin /options +DISABLE_INBOUND_REPL

repadmin /options +DISABLE_OUTBOUND_REPL

To disable replication on the troubled machine you must have either local or remote access. The local commands are above, to disable replication remotely use the same commands and add the server name. However, the server must be reachable via the network – the same network that replication runs over. You could try hammering the troubled DC with the commands during its startup and you may block replication before the DC starts, but let’s go with a more proactive method. We’ll block replication on its partners first.

The first step to finding the DC’s replication partners is to check the Connections tab inside NTDS Settings within Active Directory Sites and Services. In the below example my 2 “DFS” servers are indeed Domain Controllers, with SNY-DFS-VM02 is my troubled DC that needs replication blocked.

We have two connection fields: “Replicate From” and “Replicate To”; we also have two ways to block replication: Inbound and Outbound. We’re going to disable Outbound replication on the “Replicate From” servers and disable Inbound replication on the “Replicate To” servers. We can do this remotely using repadmin:

repadmin /options <Replicate From> +DISABLE_OUTBOUND_REPL

repadmin /options <Replicate To> +DISABLE_INBOUND_REPL

You can see this in the below screenshot that the “Current DSA Options” grow from 1 option “IS_GC” to 3 options “IS_GC DISABLE_INBOUND_REPL DISABLE_OUTBOUND_REPL”

Now that all the partners have their replication blocked, we can turn on the troubled DC and block its replication the same way. Once you’ve confirmed the troubled DC is blocked, go ahead and reverse your changes for the partners.

We’ll do one final check to verify the troubled DC is not replicating. Voila, it reports that “The destination server is currently rejecting replication requests.”

After walking you through the GUI, I’ll share with you how I did it with PowerShell. I use Get- ADReplicationConnection to check all the Replication Connections for any that match the troubled DC in the ReplicateFromDirectoryServer or ReplicateToDirectoryServer properties. Once I have those, I do some string parsing to pull out the server names. Then I write the repadmin command for both the source and target servers and store them in an array. After I get all the commands, I need to move the troubled DC repadmin commands to the end of the array (no easy way there, I ended up creating 3 arrays). Finally, I execute the commands with the output in the below screenshot. The script waits for the troubled DC to become reachable – this is when you turn on the troubled DC. Once it blocks the replication on the troubled DC, it reverses the changes on its replication partners.

$DCtoBlock = “sny-dfs-vm02″

$Commands = @()

Get-ADReplicationConnection -Filter * |

Where-Object {

($_.ReplicateFromDirectoryServer -match $DCtoBlock) -or

($_.ReplicateToDirectoryServer -match $DCtoBlock)

} |

ForEach-Object {

$Source = $_.ReplicateFromDirectoryServer.Split(‘,’)[1].Replace(‘CN=’,””)

$Target = $_.ReplicateToDirectoryServer.Split(‘,’)[0].Replace(‘CN=’,””)

$Commands += “repadmin /options $Source DISABLE_OUTBOUND_REPL

$Commands += “repadmin /options $Target DISABLE_INBOUND_REPL”

}

$Commands = $Commands | Select-Object -Unique | Sort-Object

$TailCmds = $Commands | Select-String -Pattern $DCtoBlock

$Commands = $Commands | Select-String -Pattern $DCtoBlock -NotMatch

$Commands += $TailCmds

foreach ($Action in @{‘+’=$true;’-‘=$null}.GetEnumerator()) {

foreach ($Command in $Commands) {

$Option = $Command.ToString().Split()[-1]

$CmdString = $Command.ToString().Replace($Option,”$($Action.Name)$Option”)

If (!(($CmdString -match $DCtoBlock) -and ($Action.Name -match ‘-‘))) {

do {

$CmdString

Invoke-Expression -Command $CmdString | Out-Null

$RepAdminOptions = Invoke-Expression -Command $Command.ToString().Replace($Option,””)

} while (($RepAdminOptions | Select-String -Pattern $Option -Quiet) -ne $Action.Value)

}

}

}

Thanks for reading and let’s keep those DCs on eh?

This blog post brought to you by eighteen year veteran Microsoft Premier Field Engineer David Morgan.

Goal of this Post

In this post, I’ll be showing you one method you can build out an entire environment that you can use to deploy servers, clients, Microsoft and third party applications; i.e. anything that can be deployed on Windows Server Hyper-V on a single desktop class computer you might find on Craigslist or some other technology source. The only expense in this post will be purchasing a computer and not even that as you may already have an old desktop laying around that will work.

The tasks in this post require the user to have a minimum ~200 level knowledge of and experience with Windows Server products. The steps herein to attain the desired results are not exquisite in detail and may require some personal intuition and/or research. Also note that I’ve recommended a File-Share Witness configuration instead of the normal Disk Witness a two node cluster would be best suited for. The reason I’ve done so is this document does not go through the configuration of iSCSI or other methods for supplying storage to the cluster built here. (Sounds like another blog post opportunity.) For the widest clustering experience, you’ll want to learn how to use iSCSI. You can do so by configuring iSCSI Target on your Host machine or Domain Controller.

Once you have a suitable computer, all the Microsoft software you might wish to use is free by using evaluation versions of Windows Server, System Center, SQL, Exchange, etc. Many third party products also provide free evaluation software you can use as well.

First: Hardware

This is what this document will build:

You’ll need to make a decision on what versions of Windows Server Hyper-V you wish to use as a base platform; your Host machine. Windows Server 2012 R2 and Windows Server 2016 have slightly different hardware requirements; the biggest difference is 2012 R2 and below do not require SLAT and Windows Server 2016 does (there are additional requirements for 2016 if you wish to use some of the new advanced features; Nested Hyper-V, Hot Add NICs, Hot Add/Subtract Memory, etc.).

Here are the official requirements page for Servers 2012 R2 and 2016:

• Windows Server 2012 R2 Hyper-V Hardware Requirements:
  
  • https://technet.microsoft.com/en-us/jj647784(v=ws.11).aspx
    
• Windows Server 2016 Hyper-V Hardware Requirements:
  
  • https://technet.microsoft.com/en-us/windows-server-docs/compute/hyper-v/system-requirements-for-hyper-v-on-windows
    

Out of all the requirements above the basic needs for this post are:

• Windows Server 2012 R2 Hyper-V:
  
  • BIOS support for Virtualization
    
• Windows Server 2016 Hyper-V:
  
  • BIOS support for Virtualization
    
  • SLAT (Second-Level Address Translation)
    
• A minimum of 8 GB of system memory
  
• A minimum of three hard disks; one OS & two data
  
• A minimum of one network interface adapter
  
• As always, the more capabilities and capacity you have the better performing your system will be; multiple processors, faster storage and more memory will improve your experience but you can get by just fine with the basics above.
  

Here are the basic steps:

• Configure your hardware
  
• Install the Operating System and Hyper-V role
  
• Configure necessary Hyper-V networks
  
• Create a virtual machine with the Domain Controller role installed
  
• Create two virtual machines to be cluster nodes with Failover Cluster feature installed
  
• Configure your cluster
  
• Configure your desired clustered resources
  
• Go Play ……………….
  

Here are the steps in detail:

• Configure your host machine’s hardware
  
  • Verify or enable virtualization in the BIOS
    
  • If possible, place your disk drives on separate controllers
    
• Download the evaluation version of your desired operating system:
  
  • Windows Server 2012 R2
    
    • https://www.microsoft.com/en-us/evalcenter/evaluate-windows-server-2012-r2
      
  • Windows Server 2016
    
    • https://www.microsoft.com/en-us/evalcenter/evaluate-windows-server-2016
      
  • Burn this .iso to a DVD or a USB drive
    
• Boot your new server to the DVD or USB image you created above
  
  • Install the Operating System and Hyper-V role
    
  • The host server can remain in a workgroup
    
  • Configure necessary Hyper-V networks
    
    • Press Windows Key : Type Hyper-V Manager
      
      • Click : Virtual Switch Manager in the right Actions pane
        
        • Create a new External virtual switch named Internet
          
        • Create a new Private virtual network switch named Private
          
• Create a virtual machine with the Domain Controller role installed
  
  • In the Hyper-V console action pane:
    
    • Click : New : Virtual Machine
      
    • Name this machine DC1 (store this VM on your C: drive)
      
    • Next : Choose Generation 2
      
    • Next : Leave the memory settings as default
      
    • Next : Connect the Private network
      
    • Next : Accept the defaults for VHD
      
    • Next : Choose install and operating system from a bootable image file and point to the evaluation .iso you downloaded earlier
      
    • Next : Choose finish
      
    • Next : Start and Connect VM DC1
      
    • Next : Configure the IPv4 NIC to:
      
      • IP 192.168.16.1, Subnet 255.255.255.0, GW 192.168.16.1
        
    • Next : Install the Active Directory Domain Services feature
      
    • Next : After the feature install completes:
      
      • Configure your domain controller in a new Forest, I.e Demo.Local
        
      • Follow the prompts; allow DNS installation, etc., restart
        
• Create two virtual machine cluster nodes with Failover Cluster feature installed
  
  • Click : New : Virtual Machine
    
  • Name this machine Node1 (store this VM on a data disk)
    
  • Next : Choose Generation 2
    
  • Next : Set memory to 2048 and uncheck Dynamic Memory
    
  • Next : Connect the Private network
    
  • Next : Accept the defaults for VHD
    
  • Next : Choose install and operating system from a bootable image file and point to the evaluation .iso you downloaded earlier
    
  • Next : Choose finish
    
  • Next : Start and Connect VM Node1
    
  • Next : Configure the IPv4 NIC to:
    
    • IP 192.168.16.11, Subnet 255.255.255.0, GW 192.168.16.1
      
  • Next : Join Node1 to your domain
    
  • Repeat for the second node but:
    
    • Name the second Node2
      
    • Store Node2 on a separate data disk from DC1 & Node1
      
    • Next : Configure the IPv4 NIC to:
      
      • IP 192.168.16.12, Subnet 255.255.255.0, GW 192.168.16.1
        
• Configure your cluster
  
  • Validate your configuration:
    
    • Test-Cluster -Nodes Node1,Node2
      
  • Create the cluster:
    
    • New-Cluster -Name MyCluster -Nodes Node1,Node2 -StaticAddress 192.168.16.10
      

Conclusion:

• You are now ready to proceed with cluster & Hyper-V learning exercises, troubleshoot customer issues, explore, whatever. I’ve used failover clustering here just as an example; once you have the host server and the domain controller you can configure any supportable virtual machine and application resources you desire.
  
• Now; go have fun!

Hello all. Jacob Lavender here for the Ask PFE Platforms team to talk a little about a common scenario that I’m often presented with – How to rid Windows machines of the self-signed remote desktop client certificate that is created on Windows clients and servers?

Prior to diving in, let me just state that we will not be discussing in great depth why Windows machines have a self-signed certificate within this article. At a basic level, it is due to the necessity of having a certificate to support Network Level Authentication (NLA), which is an advanced security feature designed for pre-authentication intended to protect machines from a Denial-of-Service (DoS) attacks via the RDP protocol.

Kristin Griffin has published a great article which can discuss NLA in greater detail:

https://technet.microsoft.com/en-us/library/hh750380.aspx?f=255&MSPPError=-2147217396

At the end of the day, we want NLA, and many organization’s cyber security guidelines mandate it. However, those same guidelines often mandate that self-signed certificates are not allowed within the Remote Desktop certificate store.

So, what are we going to cover:

Topic #1: Why do my Windows machines continuously regenerate this certificate despite having been removed?

Topic #2: How can I remove the self-signed certificate from machines across my enterprise?

Topic #3: How can I prevent the certificate from being regenerated upon remote desktop connection or reboot?

Topic #4: How can I automate the permissions on the registry key?

Topic #5: How can I properly deploy certificates to my Windows machines?

Tested on:

• Windows Server 2016
• Windows Server 2012 R2
• Windows Server 2008 R2 (Requires PowerShell 3.0)
• Windows 10
• Windows 8.1
• Windows 7 (Requires PowerShell 3.0)

PowerShell update for Windows Server 2008 R2 and Windows 7:

https://blogs.technet.microsoft.com/heyscriptingguy/2013/06/02/weekend-scripter-install-powershell-3-0-on-windows-7/

Make sure to read Topic #5 prior to taking any action internally to fully understand the entire scope of our discussion within this post.

And with that, now let’s dive in.

Topic #1: Why do my Windows machines continuously regenerate this certificate despite having been removed?

Windows machines automatically generate a self-signed certificate for use with the Remote Desktop protocol. This is by design as it is intended to increase the overall security posture of all machines within the enterprise which have Remote Desktop enabled.

Simply disabling Remote Desktop on a machine will not prevent the machine from regenerating the self-signed certificate.

I have had numerous clients bring this issue forward due to many security guidelines and scans flagging this as a finding. However, the machines are in fact more secure with the self-signed certificate than without due to the intended purpose. Most security guidelines advise to avoid the use of a self-signed certificate generated by each individual machine as other machines on the network have no means to trust this certificate and validate that the machine is in fact that machine.

However, remember that we are discussing the use of the self-signed certificate within the context of NLA, not within Public Key Infrastructure (PKI). Fortunately, we’ll be addressing that component of the complete solution within Topic #5.

Topic #2: How can I remove the self-signed certificate from my machines across my enterprise?

Fortunately, this is a simple task. We have PowerShell to answer our needs on this front. A sample script can be downloaded here to remove self-signed certificates from the Remote Desktop certificates store:

https://gallery.technet.microsoft.com/Remove-Self-Signed-RDP-00413912?redir=0

PowerShell 3.0 or higher is required.

Note: Do not run this script on internal Certificate Authorities which have issued their own Remote Desktop Authentication certificate.

This script can be configured as a startup script via GPO. Depending on your domain’s PowerShell Execution policy, you may need to configure the Script Parameters to include -ExecutionPolicy ByPass.

You will find that some clients will immediately generate a new self-signed certificate. In short, configuring this as a startup script alone will likely not eliminate the self-signed certificate from returning.

You will need to determine if you wish to leave the startup script in place or remove it from the GPO once you have completed this.

Other deployment methods, such as System Center Configuration Manager, are also great options and should be considered.

Additionally, now that the certificate has been removed, Remote Desktop connections to these machines will likely fail as NLA will not have a certificate to use. This is where Topic #5 becomes very critical.

Topic #3: How can I prevent the certificate from being regenerated on remote desktop connection or reboot?

As we have discussed, Windows by design generates a self-signed certificate. The machines local System account performs this function. To prevent the certificate from being generated again, we can simply deny the System account from having the necessary permission to generate the certificate. To accomplish this, we can simply deny the right within the registry:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\RemoteDesktop\Certificates

The permissions can be edited manually, or via an automated means (see Topic #4).

Prior to modifying the registry, ensure that you have a valid backup of the registry and that you perform the necessary testing within a lab to gain greater insight into how modifying the registry within your image will impact operations.
Modifying the registry can result in system failure so ensure that you have a proper plan to recover should a mistake be made. The settings within this article are non-destructive, however they can impact operations such as providing remote support to users.

To manually edit the permissions on this key start by right clicking the key and selecting Permissions:

• At the permissions window, select the Advanced radio button, then select the Add radio button.
• At the Permission Entry for Certificates window, select the Select a principal option.
• At the Select User, Computer, Service Account, or Group window, modify the location to search for the account to the local computer.

• Having returned to the Select User or Group screen, type system, and then select Check Names. This should resolve the account to the local machine account object. Now select OK.

• Having returned to the Permission Entry for Certificate, change the ACL Type to Deny.
• Now, select Show advanced permissions.

• Now that the advanced permissions are available, select Create Subkey.

Each certificate on the machine is a subkey within the Certificates key in the registry. By denying the System account object the Create Subkey permission, we are preventing the System account to create the certificate.

Now, what is the impact of having configured the Certificates registry key ACL in this manner? We may see a system event (Event ID 1057) in the system’s event log. It will simply be an indication that the System account was unable to create the subkey due to Access Denied:

Topic #4: How can I automate the permissions on the registry key?

To address this, we can utilize a GPO which will set the ACL on the key. Simply add the registry key to an appropriate GPO which is targeted at the machines in the enterprise that you wish to implement this change. As an example, I have made this addition to my Workstation Baseline GPO in my lab as I want this to be configured on all Windows clients:

It’s important to note that this GPO will create an entry on the key’s ACL which will remain even if the GPO is removed. If you need to modify the ACL in the future, you will need to ensure to grant the permission back.

Alternatively, we could use PowerShell to configure this on the key directly. You can locate a sample script at the following location:

https://gallery.technet.microsoft.com/Remove-Self-Signed-RDP-00413912?redir=0

This same sample script can be modified to correct any permissions on the key/sub-keys. Again, have good backups of the registry and perform ample testing prior to utilizing this in a production environment. It’s critical to understand how this will impact your operations.

Topic #5: How can I properly deploy certificates to my Windows machines?

For this discussion, we’re going to assume that you have an internal CA. Why? With an internal issuing CA we are able to automate the process of deploying certificates for the purpose of Remote Desktop.

Utilizing an external CA would likely:

• Be cost prohibitive due to the cost of obtaining certificates from public certificate issuers, or
• Be extremely time intensive to obtain certificate requests from all machines and submitting them manually.

So, we should begin by configuring the certificate template on the issuing CA. If you do not already have a certificate template published, duplicate the Computer certificate template and update the name of the template on the General tab.

PFE Pro Tip: Ensure that the template name is useful. I always name published templates with the word “published” in the name; example:

“Published RDP Certificate Template <Date>”

Certificate Template

While all settings for the template are beyond the scope of this document, here are the critical items:

• Update the name to be of value within the enterprise (remember to make the naming convention sustainable over time so it grows with your environment).
• Ensure the validity period is aligned with your enterprise PKI requirements. A discussion with Cyber Security might be required for this to be defined.
• Ensure that the Extensions for both Client Authentication and Server Authentication are present.
• Add the following Remote Desktop Authentication Application policies extension:
  • On the Extensions tab, select Application Policies and then select Edit.
  • At the Edit Application Policies Extensions window, select Add.
  • On the Add Application Policy window, select New.
  • At the New Application Policy windows, label the Application Policies Extension Remote Desktop Authentication. The Object identifier (OID) should be 1.3.6.1.4.1.311.54.1.2.

• Ensure that the appropriate group has been set with Autoenroll = Allow on the Security tab. You can use Domain Computers if the GPO used within this guide will target the entire domain. Alternatively, you can specify a security group and grant the Autoenroll permission to ensure that only the desired machines receive this certificate.
  
• On the Subject Name tab, ensure that Build from this Active Directory information, and select the appropriate Subject Name Format – typically Common Name works fine. Ensure to also include the DNS name as an alternative name.
• If you require an administrator to approve these certificates, that can be configured on the Issuance Requirements tab.

Default Certificate Template GPO

We need to now deploy the certificate to our clients. This is managed in two different settings.

• Define the client PKI autoenrollment.
• Define the default Remote Desktop services template for the client.

Let’s begin by defining the client PKI autoenrollment policy. This is performed via a GPO which is then targeted to the clients within the enterprise which this effects. The following settings are required:

Computer Configuration > Policies > Windows Settings > Security Settings > Public Key Policies

• Certificate Services Client – Certificate Enrollment Policy = Enabled
• Certificate Services Client – Auto-Enrollment = Enabled
  • Enable Renew expired certificates, update pending certificates, and remove revoked certificates
  • Enable Update certificates that use certificate templates

Now that we have the clients configured for Auto-Enrollment, let’s configure the RDP Default Template. The following settings are required:

Computer Configuration > Policies > Administrative Templates > Windows Components > Remote Desktop Services > Remote Desktop Session Host > Security

• Server authentication certificate template = Enabled
  • Define the Certificate Template Name. You must use the short name of the template that was published on the CA. This is the Template Name as defined on the CA. This is NOT the Template Display Name. See the image below:

Validation

After successfully deploying the certificate to the machine, we should now see an Event 1063 in the System event log:

Finally, you should now be able to RDP to clients with NLA and they should utilize the assigned certificate.

Note: I have seen where if Remote Desktop was previously enabled, it must be disabled and then re-enabled. Again, perform proper testing in your environment to ensure you have a valid action plan on how to address any variations.

Hi all! I am Bill Kral, a Microsoft Premier Field Engineer, here again to give you the steps to convert your on-premises Managed domain to a Federated domain in your Azure AD tenant this time.

Here is the link to my previous blog on how to convert from a Federated to Managed domain:

Convert a Federated Domain in Azure AD to Managed and Use Password Sync – Step by Step

https://blogs.technet.microsoft.com/askpfeplat/2016/12/19/convert-a-federated-domain-in-azure-ad-to-managed-and-use-password-sync-step-by-step/

There are many ways to allow you to logon to your Azure AD account using your on-premises passwords. You can use ADFS, Azure AD Connect Password Sync from your on-premises accounts or just assign passwords to your Azure account. In addition, Azure AD Connect Pass-Through Authentication is currently in preview, for yet another option for logging on and authenticating.

So, why would you convert your domain from Managed to Federated? Well, maybe you finally decided to invest in an ADFS environment. Maybe your company mandated that the storage of passwords in the cloud go against company policy, even though the hash of the hash of the password is what is really stored in Azure AD… and you may have your reasons for doing so. Either way, we’ll discuss how to get from a Managed domain to Federated domain in your Azure AD environment.

Let’s set the stage so you can follow along:

The on-premises Active Directory Domain in this case is US.BKRALJR.INFO

The AzureAD tenant is BKRALJRUTC.onmicrosoft.com

We are using Azure AD Connect for directory synchronization (Password Sync currently is enabled)

We have setup an ADFS environment to federate the domain with the Azure AD Tenant

Before we start, you will need the following things installed on your ADFS Server to connect to your Azure AD tenant:

Microsoft Online Services Sign-In Assistant for IT Professionals RTW

https://www.microsoft.com/en-us/download/details.aspx?id=41950

Windows Azure Active Directory Module for Windows PowerShell .msi

http://connect.microsoft.com/site1164/Downloads/DownloadDetails.aspx?DownloadID=59185

1. First, log on to your Azure Portal and see that the “Status” of your domain is Verified and the “Single Sign-On” for your custom domain show as Not Planned or Not Configured.
   

   
   

2. Now, go to your Primary ADFS Server and lets connect to your Azure AD Tenant.
   
   1. On the Primary ADFS server, open an Administartor powershell window and import the MSOnline module
      

      Import-Module MSOnline
      

   2. Connect to your Azure AD Tenant
      

      Connect-MSOLService -> Enter your Azure AD credentials on the pop-up
      

      
      

3. Once you are connected to your Azure AD Tenant, let’s make sure your domain is currently recognized as a “Managed” domain.
   

   Get-MsolDomain -Domainname domain.com
   -> Should show your domain as “Managed”
   

   
   

4. Now we can make sure that the domain you are converting is currently NOT in the ADFS configuration.
   

   Get-MsolFederationProperty -Domainname domain.com -> Should show that domain does not exist in configuration
   

   
   

5. So, now that we have connected to the Azure AD Tenant and confirmed that are domain configured as Managed, we can get to converting it to a “Federated” domain. When done, all of your Azure AD sync’d user accounts will authenticate to your on-premises Active Directory via ADFS.
   
   1. While still on your ADFS server, import the ADFS module
      

      Import-Module ADFS
      

   2. Run the command to convert your domain. Now, if you have a single top-level domain, you do not need to include the -SupportMultipleDomain switch. If you currently have or are planning to add additional domains to your ADFS / Azure AD federation, you will want to use it as I have.
      

      Convert-MsolDomainToFederated -DomainName domain.com -SupportMultipleDomain -> (A successful updated message should be your result)
      

      
      

   3. Once this has completed, we can see the properties for the converted federation.
      

      Get-MsolFederationProperty -Domainname domain.com -> Should now not show the domain error we saw in step 4 and contain information for your domain under Microsoft Office 365 “Source” entry.
      

      
      

6. Now, lets go back to your Azure Portal and see take a look at what the “Single Sign-On” status is for your custom domain now that you have converted it.
   

   
   

   As you can see, after a refresh and a little time for your commands to work their magic, my domain now shows the “Single Sign-On” as “Configured”
   

7. You can now test logging on to myapps.microsoft.com with a sync’d account in your Azure AD Tenant. You should now see a re-direction to your ADFS environment while you are being authenticated.
   

That is pretty much it!!! Now, at this time, if you were replicating your passwords to Azure AD (or as most Microsoft folks like to say, the hash of the hash of the password), you may keep doing so to use as an authentication “backup” should your ADFS environment fail. This usage as a backup authentication does not happen automatically, but a powershell command will do the job when it is needed!!!

If you intend to disable replication of you on-premises passwords to you Azure AD Tenant, that can be accomplished through your Azure AD Connect configuration setup!!!

Once again, thanks for reading!!!