ADSync Tool Usage Guidelines

Overview

ADSync is quite a complex tool, and understanding how it works in certain situations is important so that you know the difference between ADSync running as expected and when it is not functioning properly.

There are certain things which, as an Administrator, you should not do unless you understand and are willing to accept the consequence.

This document is intended to explain how this tool works so that you have the understanding needed to operate it successfully and get the most benefit out of using it.

tip

Please make sure to read this document in its entirety before installing or starting the ADSync Service.

Applies To: CloudPortal Services Manager V11.5 Cumulative Update 2 and above

Installation

ADSync should be installed on ALL Active Directory Servers in the domain to ensure that all password changes are captured, as these are not necessarily replicated between AD Servers like how other object changes are. In order to install ADSync you will need to know the following things:

• Is this server going to be used to watch for:
  • User Changes
  • Contact Changes
  • Group Changes
• Which AD Groups do you want to use for monitoring Object changes
• Are you going to need to configure a Custom Workflow, or make other ADSync.exe.config file changes
• Will you be needing to install any Patches on top of the initial ADSync install
• Can the Active Directory Server access the Atria API Web Service over http or https

The ADSync Service is started automatically upon installation, but will not start if it cannot connect to the API and validate the customer information configured in the config file. Additionally, if the ADSync service fails to start, the installation also fails, and will rollback any changes. So it is important to ensure that the API can be accessed at all times, and that the credentials specified at install time are correct.

Customisations After Install

If you needed to configure a Custom Workflow, apply additional patches, or make config file changes immediately after installing ADSync then you will need to stop the ADSync Service immediately after the install completes. Given that the stopping of ADSync can take some time, especially if it thinks that there are still any outstanding tasks to complete, the best way to do this is to kill the process.

warning

This is the only time for which killing the process is acceptable, as killing the process can leave the queue in an inconsistent state and cause further issues.

tip

So before completing the install of ADSync, it might be a good idea to have Task Manager open and ready at the processes tab so that you can kill the ADSync.exe process.

Startup

When ADSync starts up, it may begin building a catalogue of AD objects which are in scope. If there are a lot of objects in scope, then this process can take a long time to complete.

This is controlled by the EventLogWatch setting in the config file.

EventLogWatch

If set to True, then the catalogue is built. If set to False, then it wont be built.

Ok, so what does this EventLogWatch setting actually do? And why does ADSync need to build a catalog of objects?

This setting tells ADSync whether it is to use the EventLog or not in order to capture AD changes.

When AD changes are logged in the EventLog, it not only just detects the change, but also the specific nature of the change; like, is this a new object, changed object, an object added to or removed from a group, the type of object, and even the specific properties on the object which were changed.

This information is then used to determine what action (if any) that ADSync is to take without the need to send API requests to Atria.

To assist with knowing if a changed value is one in which we are interested in, the EventLogWatch setting works in conjunction with the following additional Settings:

• UserAttributes
• GroupAttributes
• ContactAttributes
• UserAttributesSyncOnChange
• GroupAttributesSyncOnChange
• ContactAttributesSyncOnChange
• EventLogAddMemberEvents
• EventLogRemoveMemberEvents
• EventGroupAddMember
• EventGroupRemoveMember

The Attribute based settings state which attributes ADSync is interested in, and whether ADSync is to only look for these attributes when looking for changes, and the Event based settings specify additional events regarding Group Syncing. And lastly, to assist ADSync in comparing current values with the changes ones, it builds a Catalogue of in scope items. These are the .dat files which you will find in the Queue folder. This catalogue is built initially upon startup of ADSync, and is maintained when any future changes are made.

What if the EventLogWatch is NOT set to True?

If this is not set to True, then ADSync can only work with the LastUSN value within AD. ADSync maintains the most recent LastUSN value in the SyncActiveDirectory.config file within the Queue Folder. It will poll AD to find any objects whose LastUSN value is greater than the value which it has stored, looking for more recent changes, then as it has no idea what has actually changed, it is forced to submit an API request to Atria to see if the object has been synced before, then it will submit another request to Sync the object even if the change to the object is not really one for which we are interested in.

So you can see that setting the EventLogWatch value to True will reduce the load that ADSync puts onto the environment, and will also improve the performance of ADSync, so it is a good setting to turn on, and is in fact on by default, but it does come with some cost, especially at startup time (first time round only, or if the queues need to be rebuilt)

Include and Exclude Groups

Exclude Groups will override any Include groups, so if an object is not syncing, check to see if they are being excluded because of group membership of the configured "Exclude" groups

info

This works a bit like "Deny" access in Active Directory, or on the file system

warning

Include and Exclude Groups should all be created as Global Security Groups. We have seen issues where ADSync ignores members within these Groups when they are of a different type to this.

First time User Sync

Initially when a user is added to the "Include" groups for syncing, the user itself will sync with a blank password, the password itself is only captured when a password change occurs while the user is in an in-scope state.

note

Therefore, in order to capture the password and have that synced across to the Atria side, the user must change their password after the account has been synced. An easy way to do this is to force them to change their password at next logon.

Group Sync

The initial Group GET request can take a long time as it not only determines if the Group has already been synced or not, but it also has to check for each member as well. While ADSync is waiting for the response, no other syncing activity will be performed so it may appear that ADSync is hung. Give it the time it needs to retrieve this information.

Because Group syncing can take a long time, it might hold up other smaller sync tasks. You might want to consider isolating Group sync activity to its own AD Server so that Group sync activity does not interfere with other AD activity like password changes.

The Log File

By default log files are created under the ADSync\Logs folder, and log files are created on a daily basis with the default .NET file size of 5Mb. These settings can all be changed depending on your specific requirements.

tip

You might want to put the log files on a separate drive for example, or increase the maximum file size

The following settings are all related to the standard .Net logging functionality in app.config files. Here are some of the settings which we think are relevant:

Attribute	Description
maxFileSize	Maximum log file size In bytes. By default it will stop logging when the limit is hit.
logFileCreationSchedule	Controls log file rotation. Possible values: None, Daily, or Weekly.
customLocation	Full path to the directory in which to create log files. Could be on a separate disk for example.
reserveDiskSpace	Minimum remaining disk space in bytes before allowing log files to grow. By default it will stop logging when the limit is hit.

More information on setting the maxFileSize can be found later when discussing Recommended Settings.

Logging Options

In case of an exception, the LogMask setting controls how much of the exception info is logged. It's a bitmask with the following values:

Name	Value
Message	1
Exception	2
Inner Exceptions	4
Error Code	8
Extended	16
Stack Trace	32768
Everything	65535

Filtering Options

As an addition to the amount of data which is logged when an exception occurs, you can also set filtering options which determine how much detail is logged regarding ADSync activity

The filter node within the FileLogTraceListener sections at the top of the ADSync.exe.config file controls the filtering. Within the filter node, there is an attribute called initializeData. By default the value is set to "ActivityTracing,Information"

The initializeData can be one or more of the following values:

Log Level	Description
Critical	Allows only Critical events through
Error	Allows Critical and Error events through
Warning	Allows Critical, Error, and Warning events through
Information	Allows Critical, Error, Warning, and Information events through
Verbose	Allows Critical, Error, Warning, Information, and Verbose events through
ActivityTracing	Allows the Stop, Start, Suspend, Transfer, and Resume events through
All	Allows all events through

Operation

Stopping and Starting the ADSync Service

Sometimes stopping the ADSync Service can take a long time for the process to actually exit.

It is likely that ADSync is still processing some tasks and is waiting for these to complete before terminating.

warning

It is important that during this time you do not kill the process. Doing so can result in the Queue becoming corrupt.

If the queue does become corrupt, then since there is no way to know which item is causing the corruption, the entire queue needs to be deleted such that it can be recreated automatically. To delete the queue, you should delete all files (and folders) within the ADSync\Queue folder with the exception of the SyncActiveDirectory.config file

note

Once you do this and restart the ADSync Service, the entire catalog will be rebuilt.

This might take some time to complete and might impact on current sync activities.

Syncing large numbers of items

When a lot of items are being synced at one time, the queue is likely to grow to be quite large as it contains all outstanding sync requests.

There is also a considerable amount of activity directed at the API and SQL Server. All of this activity is likely to have a negative impact on ADSync performance, as well as on the portal as a whole.

To minimize this, there is a new setting called ThrottleUploads which allows you to minimize the number of items which are queued at one time.

Set the value for this setting to True, then set the ThrottleLimit setting to the number of items to limit the queue to.

tip

We've found that a value of between 50 and 100 is a good starting point.

Throttling works by queuing items until the ThrottleLimit is reached. Further changes are then held back from being queued until current queued items have completed, and there is room for more items to be added without exceeding the limit.

A good value would ensure that the queue never really gets reduced to 0 before adding more items (so the queue always contains items to ensure continuous activity).

This reduces the overhead in a couple of ways:

• API requests will not be sent for items which are not going to be queued until such time as there is room in the queue for that item. This will reduce activity in both the API and SQL Server.
• Status Update checks will only be requested against items which are currently queued, which again reduces the number of API requests being performed

By reducing this activity, the API and SQL Server are more available for actioning requests from other sources

Syncing groups with a large number of members

When syncing groups that have a lot of members ADSync needs to determine the amount of work that needs to be done to complete the sync.

It needs to know which members of the group have already been synced and which ones have not. To do this, it sends an API requests asking for this information. This request can take a long time to complete, and during the time that ADSync is waiting for a response from the API no further activity takes place. This may either result in the API request timing out and the request being resubmitted, or in the case that the API timeouts have been increased, it may give the appearance that ADSync has hung.

warning

During this time you need to be patient. DO NOT kill or restart the service.

note

If the API timeouts have not been increased then this should be done. Refer to the recommendations below for details on how to do this.

New settings have also been added which will help ADSync to handle large groups in a more streamlined manner.

These settings are BatchUploads, and UploadBatchSize. These settings work by telling ADSync to process a large group in smaller chunks.

tip

Set the BatchUploads value to True, and the UploadBatchSize to somewhere between 50 and 100.

When setting these values, instead of attempting to upload a 500 member group in a single request, ADSync will add or remove members in chunks of 50-100 items. And logging will also be performed between each chunk as it is being processed. This gives better visibility of ADSync actually performing activity during this time as progress is written to the logs, and results in smaller API calls which are less likely to cause timeouts within IIS.

Recommended Settings and Changes

There are many settings within the ADSync.exe.config file which change the way in which ADSync functions. After quite extensive testing we've come up with the following changes which will improve the performance of ADSync, and allow you as the Administrator to monitor ADSync for problems more effectively.

There is no specific sizing for the Log files, and by default this will have a maximum of approximately 5Mb. We often find that this is not large enough, as when this limit is reached, logging stops until the log rolls over to the next day. So increase the size of the log file. To do this:

Maximum size of a log file

By default the FileLogTraceListener sets the maximum size of a single log file to about 5MB at which point it will stop logging. To set the file size to about 10MB you would add the maxFileSize="10000000" attribute to the FileLogTraceListener like this:

      <add name="FileLog" type="Microsoft.VisualBasic.Logging.FileLogTraceListener, Microsoft.VisualBasic, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL" location="Custom" customLocation="…\ADSync\Logs" logFileCreationSchedule="Daily" traceOutputOptions="DateTime" autoFlush="True" maxFileSize="10000000"> 
            <filter type="System.Diagnostics.EventTypeFilter" initializeData="ActivityTracing,Information"/>
      </add>

The size you set will depend on the amount of activity you expect on a daily basis.

For more information about MaxFileSize see FileLogTraceListener.MaxFileSize Property.

Add a Log just for Errors

Often you will find yourself wading through the daily log files in search of objects which have had trouble syncing. It is a lot simpler if there was a log file which only contains this information. This can be done quite simply by adding an additional log file like this:

• Create the Folder for the ErrorLogs to reside
• Copy the FileLog entry which was used to increase the log file size above, and use it to create a new entry like in the following. Note the "name", the "customLocation", and the "initializeData" attributes need to be changed

 <add name="FileLogErrors" type="Microsoft.VisualBasic.Logging.FileLogTraceListener, Microsoft.VisualBasic, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL" location="Custom" customLocation="…\ADSync\ErrorLogs" logFileCreationSchedule="Daily" traceOutputOptions="DateTime" autoFlush="True" maxFileSize="10000000">
      <filter type="System.Diagnostics.EventTypeFilter" initializeData="Warning"/>
 </add>

• Add a new listener directive

 <sources>
     <!—This section defines the logging configuration for My.Application.Log -->
     <source name="DefaultSource" switchName="DefaultSwitch">
          <listeners>
               <remove name="Default"/>
               <add name="EventLog"/>
               <add name="FileLog"/>
               <add name="FileLogErrors"/>
          </listeners>
     </source>
</sources> 

ApplicationSettings

Name	Value	Why
UserPollTime	20	Reduces the load on AD slightly
UploadPollTime	20	Reduces the load on the API and on the SQL Server by reducing the frequency of each round of uploads occurring
LogMask	65535	If an error occurs you will want it to log as much information as it can
UploadMaxErrors	2	Only try to sync an object twice before giving up. Odds are if it has failed twice, it is likely to keep failing
EventLogWatch	True	Takes advantage of AD Changes written to the EventLog. It also allows you to take advantage of only syncing changes which include the specified set of Object properties, which results in less API calls and better performance overall
UserAttributesSyncOnChange	True	Only sync changes to attributes which have changed
ContactAttributesSyncOnChange	True	Only sync changes to attributes which have changed
GroupAttributesSyncOnChange	True	Only sync changes to attributes which have changed
MaxGetStatusTries	5	Allows detection of objects stuck in a provisioning state, and re-submits the request automatically
ThrottleUploads	True	Limit the number of changed objects which are queued at one time. Results in better performance and less load on the API/SQL Servers
ThrottleLimit	50-100	Seems to be a suitable amount of items to queue to allow for the completion of some, and the adding of additional items to ensure that the queue never really becomes empty when changing lots of items
BatchUploads	True	Results in smaller Group upload requests, and in turn causes logging to occur more frequently while groups are being synced
UploadBatchSize	50	Seems to be a suitable amount
MaxResubmits	2	Stops recurring attempts to sync groups when a single group member fails to sync

Increase the API timeout

Although the APITimeout setting in the ADSync.exe.config file may be set to -1 (no timeout), timeouts do occur with the API as this is an IIS Web Site.

info

By default the IIS executionTimeout value is set to 110 seconds.

If you are syncing Groups, then this will need to be increased, as a single GET request for a large group can take 15 minutes or more and a timeout when syncing a group, will result in the request failing, and then an attempt to sync the group again. It's best to avoid this if possible.

To increase the timeout, add the httpRuntime executionTimeout directive to the web.config file of the API.

• On the Atria Web Server where the API resides
  • Add the following to the web.config file within the Cortex Management\CortexAPI folder

<system.web><httpRuntime executionTimeout="1800"/></system.web>

info

A Value of 1800 will make the timeout 30 minutes. This should be scaled to meet your needs; alternatively it can be set to something like 10000 to make sure timeouts never really occur.