Data Migration Performance to Crm Online

I've recently been looking at the rate of data migration into Crm Online using SSIS, and how this can be optimised. I started with a baseline rate of 12 records per second, and have so far improved this to 430 records per second, all using one client machine.

The easiest way to migrate data into Crm Online is via a synchronous, single-threaded process that writes one record at a time, so this was the baseline. It soon became very clear that the performance bottleneck in this scenario is network latency - i.e. the round-trip time for the network packets to make a request to the Crm Online server, and to receive a response back.

So, the challenge was to improve on this, which can be done by addressing each aspect of the simple scenario - i.e.

1. Synchronous v. asynchronous calls


2. Single or multi-threaded - either within one process, or multiple concurrent processes


3. Sending more than one record at a time

So far, I've not tested asynchronous calls, primarily because SSIS is stream-based, and I can't see a way to write out synchronous error output if using an asynchronous pattern. It would be possible to write out asynchronous error information, but for now that would involve too much code rewrites. In general, though, I would expect use of an asynchronous pattern would give similar performance benefits to the multi-threaded approach, though it may be possible to multiply the benefits by combining the approaches.

Multiple threads
SSIS controls the threading behaviour of a package, so rather than try for a multi-threaded single process, I went for running several instances of the same package concurrently, which you can do with the dtexec tool. There are two main aspects to making this work:

1. You will need to be able to partition the source data, so that each package instance submits different records. For my tests, I had an integer identity column on the source data, and used the SQL modulo operator (%) to filter on the remainder from an integer division. For 10 concurrent packages, the where clause was 'WHERE id % 10 = ?' with the '?' replaced by a package variable.


2. The package will not be able to reference any files, either as data sources, destinations or log files, as SSIS will attempt to get exclusive access to the files. So, I used a SQL Server source, and wrote log information to SQL via the SSIS SQL Log Provider

I tested 10 concurrent packages, and this gave between a 7-fold and 9-fold performance improvement.

Submitting multiple records
The Crm API is primarily designed around modifying one record at a time, with a separate request per record. However, CRM 2011 introduced the facility to pass multiple records, using the RelatedEntities property of a 'parent' entity. This allows you to build (in memory) a RelatedEntityCollection of multiple records, then attach this to one record, and submit this as one request.

There are two limitations to this approach:

1. The entities have to be related via a relationship in CRM.


2. The same data operation has to apply to the parent record, and the records in the RelatedEntityCollection

Initially I'd hoped to use the systemuser entity as the parent entity, as there is necessarily a relationship between this entity and any user-owned entity. However, this wouldn't work with limitation 2, as I wanted to update the systemuser, but create the related entities, and this doesn't work.

Instead, I had to make schema changes. I created a new entity (e.g. exc_batchimport), and a 1-N relationship with each entity that I wanted to import. Each request would then create 1 exc_batchimport record, and a configurable number of related records.

I tried various batch sizes, and had success up to a batch size of 1000, but failures with a batch of 5000. My initial view is that the failures come from the number of records, and not the total data size, but I've not tested this extensively.

This approach also gave significant performance gains, though only when network latency was a main performance factor - i.e. it helped a lot when connecting to Crm Online, but gave no appreciable benefit (and in some case, worse performace) when connecting to an On Premise CRM server. Most of the benefit came with a batch size of 10 records, though performance did continue to improve slightly if increasing the batch sizes up to 1000 records.

Performance results
I did the tests running on Windows 2008 Server with moderate capacity:

• A virtual server running via Hyper-V


• One processor core allocated to the server, running at 2GHz


• 4 GB of memory


• Server was running in a hosted environment in England, connecting to the EMEA Crm Online Data Centre

The tests were done writing 100000 new records to a custom entity, writing 2 text fields, and integer field, and an option set, and allowing CRM to generate the primary key values.

Concurrent packages	Batch Size	Time	Records / sec
1	n/a	832	12
10	n/a	1094	91
10	10	276	362
10	100	267	374
10	1000	233	429

Conclusions
The main performance issue with modifying multiple records with Crm Online relates to network latency. I've successfully tested 2 complementary approaches which give a combined 35-fold speed improvement, and it may also be possible to gain further improvements with asynchronous calls.

The performance figures were for a custom entity. I'm intending to do further tests with the customer entities (account and contact), and the activity entities, as each of these require more SQL updates than a custom entity.

Using WSDL Proxies with CRM Online. It's different outside of North America

I've been spending more time than I'd like using WSDL Proxies with CRM Online (i.e. when I can't use the .Net 4.0 OrganizationServiceProxy class). I'll write up some more about this soon, but this is a quick post about a specific issue with connecting to CRM Online for an organisation in EMEA, rather than North America, which I've not found documented anywhere.

I was basing my code on the wsdlbasedproxies example in the CRM 2011 SDK. Once you find it, this code is reasonably well documented.

However, when testing it, I could connect to the IDiscoveryService without problems, but continually got the error 'An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail' when connecting to the IOrganizationService.

Ultimately the issue was with the AppliesTo constant. The setup.txt instructions tell you to set this based on data in the element from the Discovery.svc wsdl. This gave me "urn:crmemea:dynamics.com" when connecting to an Online organisation in EMEA, which worked for the IDiscoveryService, but not IOrganizationService.

** Update 2012-07-11. IOrganizationService now also uses urn:crmemea:dynamics.com **

So, this needed a few minor code changes to the code in Online\program.cs, as you need different tokens for each service. For EMEA, I used the following:

private const string AppliesToDiscovery = "urn:crmemea:dynamics.com";
private const string AppliesTo = "urn:crmemea:dynamics.com";

static void Main(string[] args)
{
//Authenticate the user
SecurityToken tokenDiscovery = Authenticate(UserName, UserPassword, AppliesToDiscovery, Policy, IssuerUri);
SecurityToken token = Authenticate(UserName, UserPassword, AppliesTo, Policy, IssuerUri);
//Execute the sample
string serviceUrl = DiscoverOrganizationUrl(tokenDiscovery, OrganizationUniqueName, DiscoveryServiceUrl);
ExecuteWhoAmI(token, serviceUrl);
}


Update July 2012: AppliesTo for EMEA should now be urn:crmemea.dynamics.com. See http://mscrmuk.blogspot.co.uk/2012/07/using-wsdl-proxies-with-crmonline.html

Selecting the correct workflowid

A very quick one, which is mostly an aide-memoire for myself, but others may find useful.

I occasionally need to run CRM workflows programmatically using an ExecuteWorkflow request. This takes a workflowid as a parameter, but the question is, which one ? If you look at the workflow entity, you'll find several records for a workflow with a given name. The logic I use to determine which is the correct on is:

• statecode = 1 (Activated)


• activeworkflowid is not null

But use the workflowid value of the record for which activeworkflowid is not null. This seems a but counter-intuitive to me, hence this post

PartyList attributes and the plugin event pipeline

This should be a quick post about a subtlety with the plugin event pipeline. I recently wrote a plugin that could modify the data that's submitted when updating an activity record. This should have been a straightforward plugin on the Pre event that modified the Target InputParameter, and it all worked fine, except for partylist fields (such as the resources field on the serviceappointment entity, or optionalattendees on the appointment entity). Essentially, any changes I made to these fields in the plugin were ignored.

Fortunately, there is a solution, and it depends on the stage that you register the plugin on. If you register on stage=20 (i.e. within the transaction), your changes are ignored. However, change the registration to stage=10 (before the transaction), then it does work. There's no documentation on this, but I expect it is due to how the partylist data is saved. This data is written to the activityparty table in the database, and I expect that the SQL for this is already fixed at the start of the transaction, and hence is unaffected by changes in the plugin code.

Unexpected error with ConditionOperator.In and typed arrays

I just met a bizarre error when using the Crm xrm assembly when using the ConditionOperator.In in a query. In this case the query was to find all notes related to a list of CRM entities, and the error was "Condition for attribute 'annotation.objectid': expected argument(s) of type 'System.Guid' but received 'System.Guid[]'". I was using almost identical code to some code that did work, but there was a subtle difference in the overloads of some of the xrm methods.

Consider the following code, which works:

QueryExpression q = new QueryExpression("annotation");
Guid g1 = Guid.NewGuid();
Guid g2 = Guid.NewGuid();
q.Criteria.AddCondition(new ConditionExpression("objectid", ConditionOperator.In, new Guid[] { g1, g2 }));

However, change the last line to the following, and it fails with the error above:

q.Criteria.AddCondition("objectid", ConditionOperator.In, new Guid[] { g1, g2 });

On the face of it, you'd expect identical behaviour, but it looks like the problem is due to the parameter overloads on the different methods. The constructor for ConditionExpression takes 5 overloads, and the compiler will use System.Collections.ICollection for the array of Guids. However, the AddCondition method only offers one type for the third parameter (params object[]). The result of this is that the code fails because the parameter is interpreted as object[] {new Guid[] { g1, g2 }}.

Interestingly, other code can also work, e.g.

q.Criteria.AddCondition("objectid", ConditionOperator.In, new object[] { g1, g2 });
q.Criteria.AddCondition("objectid", ConditionOperator.In, g1, g2);

Options for upgrading ASP .Net extensions for CRM 2011

I like April. One reason is that it's the start of my MVP renewal cycle. After April 1st April I will either have been renewed, or not, and I feel less circumspect about making critical comments about decisions Microsoft have made. So, ASP .Net extensions with CRM 2011. Since the first release of CRM (CRM 1.0 or CRM 1.2 depending on your country), a major extension point for On-Premise CRM implementations was to develop ASP .Net extensions and deploy them within the CRM web site. Having a supported way to place these extensions within the CRM web site was important for several reasons:

1. To allow relative Urls in IFrames, ISV.Config and SiteMap. Main reasons for that are to cope with IFD environments where different domain names are provided for internal and external access, and to avoid configuration issues exporting/importing between environments
2. To allow single-sign on and impersonation, so the code could act on behalf of the CRM user, without needing to re-enter their credentials
3. To maintain the Site Origin (aka same site of origin). This is an important consideration as Internet Explorer security will only permit code interaction between pages if it considers that they are part of the same site. For example, the ability to pass data to a dialog using window.dialogArguments, or the ability to access objects on a calling window via window.opener both depend on the pages being in the same site

However, CRM 2011 puts significant restrictions on putting your ASP .Net extensions within the CRM web site. Essentially, the only supported option is to retain existing extensions that use the CRM 4 endpoint. So, what happens to the points above if you can't put your ASP .Net extensions in the CRM web site:

1. You'll have to use absolute Urls. This makes deployment between environments (e.g. from development to live) harder, as the Urls would have to changed. In small-scale environments this would be just a manual task, but in larger environments you may decide to build a process to automate this. Overall, I see this as a major annoyance, but not a major problem
2. Microsoft have put work into making single sign-on work across web sites. This depends on setting up Secure Token Services, which incurs some administrative and deployment overhead. There's an additional deployment overhead of setting up a new web site for the extensions, and configuring access to it. I've not tested this fully, but assuming it works as promised, this should resolve the single sign-on and impersonation issue
3. This is the big problem area. I don't think same site of origin can be maintained with the ASP .Net pages outside of the CRM web site, which effectively removes support for a common type of extension that was possible in all previous versions (Note, IE 8 has some settings that affect how Site Origin is applied, and this might help, but IE 7 is a supported browser for CRM 2011, so this cannot be a universal solution)

So, what can/should you do with your ASP .Net extensions that you wrote for CRM 4.0, or were intending to write. I see 3 main options:

1. Don't upgrade your code. ASP .Net code that uses the CRM 4 web service endpoint will still work in the ISV folder, and Microsoft have not said it is unsupported. This is the simplest option, but it means you need to maintain CRM 4 code, and you'll still have to address the issue when CRM 6 comes out. You could cross your fingers and hope that, in CRM 6, Microsoft reintroduce support for ASP .Net extensions within the CRM web site
2. Upgrade the code to use the CRM 2011 endpoint, and deploy it in a separate web-site from CRM. As stated above, this causes extra deployment overhead (which I consider is a significant overhead, which is often under-estimated), and you won't be able to use same site of origin, so you have to expect some limitations
3. Rewrite the code as web resources (Silverlight, or HTML with javascript). Microsoft have introduced quite a lot of integration points in CRM 2011 that makes this a powerful option. This has a major advantage that the resources are necessarily hosted within the CRM web site, and can be deployed as part of a CRM solution. However, these are client-side technologies and some extensions would need extra work to build (e.g. extensions that access a database on another server). My biggest problem though is the development effort required to rewrite code in substantially different technologies, and these are technologies that are not as mature as ASP .Net

Of these, I don't like any of the options. Numbers 2 and 3 would be necessary if you need to support CRM Online, but for On-Premise implementations there are some difficult decisions to make. Don't get me wrong, I appreciate that there are a lot of good things for developers in CRM 2011; web resources and the single sign-on across web-sites are very powerful and very welcome, and the only options for CRM Online, it's just a shame to lose some On-Premise options.

But to finish on a more positive note, another reason I like April is that this is a great month for ski touring.

Removing prompt for credentials when browsing with Internet Explorer

This post isn't intended to be a complete list of solutions to issue when you are unexpectedly prompted for AD credentials when browsing with Internet Explorer, but it gives some rules of thumb regarding where to start looking. The scenario: You try accessing a page using IE, and are prompted by a 'Windows Security' dialog for AD credentials when you don't expect it.

Possible causes: There are many, but the first thing to do is work out if it's a server-side issue, or a client-side issue. A simple test is what happens if you provide valid credentials in the Windows Security dialog:

• If you can then connect, then this is a client-side issue
• If you are prompted again, either 2 or 3 times, then get a permission error (normally HTTP 401), then it is a server-side issue

Client-side issues If it's a client-side issue, then look at the client IE settings, and the URL of the web page: If IE considers that you have already logged into the DNS domain (the part of the url prior to the first single / - e.g. http://crm:5555/), then it should reuse these credentials and you won't be prompted to login. However, IE is picky about matching the DNS domain, so if you've already logged into http://crm, then it won't trust other aliases (e.g. http://localhost, http://crm.mydom.com, http://192.168.0.1) and will prompt for credentials.

If you're using CRM 4 and have extension pages in the ISV directory of the CRM web site, then it is best to provide the URL as a relative path (e.g. /ISV/MyCompany/MyPage.aspx) rather than an absolute path, to avoid this issue.

The IE security settings will determine whether IE will try submitting your logged-on credentials. By default, it will only do this if you connect to a site in the Local Intranet Zone, so check the IE security settings, and the zone of the web site you're connecting to

Server-side issues If it's a server-side issue, then there are many possible causes, but most of them come to Kerberos in one way or another

Possible SQL Gotcha - use of 'Not In' with NULLs and the customer attribute

I was recently putting together a bit of SQL to illustrate the use of a NOT IN clause for a forum answer, and got some unexpected results. The query was a relatively simple example; find all accounts with no associated opportunities. So, I tried this:

SELECT name FROM FilteredAccount
WHERE accountid NOT IN (SELECT accountid FROM FilteredOpportunity)

Nice, simple query, but it returned no data (and it should have done). However, the following works fine:

SELECT name FROM FilteredAccount
WHERE accountid NOT IN (SELECT customerid FROM FilteredOpportunity)

The only difference is the use of customerid instead of accountid in the subquery. If I'd have expected the first query to work instead of the second query, as customerid is a generated field (it's generated within the Opportunity via by the SQL function COALESCE(accountid, contactid)).

This all seems weird, but it comes down to what happens with nulls. An opportunity will be associated with one of an account, or a contact. So, the subquery 'SELECT accountid FROM FilteredOpportunity' could return a null (if you have an opportunity against a contact), but 'SELECT customerid FROM FilteredOpportunity' will always return non-null values. Don't ask me why, but the presence of nulls in the subquery cause the NOT IN query to misbehave.

One way to confirm this is with another variation on the query above, which also works:

SELECT name FROM FilteredAccount
WHERE accountid NOT IN (SELECT accountid FROM FilteredOpportunity WHERE accountid is NOT NULL)

This query explicitly excludes nulls from the results on the subquery, and so it works fine.

The main lesson I took from this is to always test for nulls in the subquery when using NOT IN; another lesson is to pay close attention when using attributes that represent the composite Customer data type in CRM

For reference, the reason why I was doing this is because this is a classic example of a query that cannot be done through FetchXML, and hence cannot be written with an Advanced Find in CRM. If the primary entity is an account, contact or lead then you have a manual workaround in CRM, for example:

• Create a marketing list, and populate it with all accounts
• Use Advanced Find to remove from the list all accounts that have an opportunity
• This will then leave you with a marketing list that contains all accounts without an opportunity

CRM 2011 RTM Release

Seems like the day for announcements: the CRM 2011 RTM code has been released for On-Premise and Partner-Hosted environments. This is nearly 2 weeks earlier than I expected, so congratulations to the CRM product team.

The server software can be downloaded here, and that page has links to download the other components. The build number is 05.00.9688.583, which is consistent with the build number of the binaries in the release SDK.

Using CRM 4.0 assemblies on a CRM 2011 Server

CRM 2011 Server includes a publisher policy that causes any assembly built against the CRM 4 sdk assemblies to load the CRM 5 sdk assemblies instead. There are certain circumstances where this can cause errors loading the assembly; see the end of this post for possible error messages.

One workaround is to not run the application on a Crm 2011 Server, but there is an alternative, which is to explictly tell your application not to use this publisher policy file. This is done through adding the following to the app.config file:

<configuration>
<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<dependentAssembly>
<assemblyIdentity name="Microsoft.Crm.Sdk" publicKeyToken="31bf3856ad364e35" culture="neutral" />
<publisherPolicy apply="no" />
</dependentAssembly>
</assemblyBinding>
</runtime>
</configuration>

This raises one more issue: in some circumstances your assembly may not be the main .exe, but a .dll loaded by another process, in which case you'll have to modify/create the .config file for that .exe. This is done by creating a file named .exe.config in the same directory as the .exe (here's an example). I have a nagging concern that I may have to do this with SSIS packages that use a custom component that use the SDK assemblies, which could get interesting, as different executables are used for in design, debug and runtime. If I do have this issue with SSIS, then I'll post a more detailed workaround (if I find it).

My hope is that this is a temporary problem that will be fixed, as the readme in the 5.0.1 version of the SDK refers to an 'incorrect Publisher Policy'. This readme also gives an explanation of this issue

One possible error
System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Crm.Sdk, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
File name: 'Microsoft.Crm.Sdk, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' ---> System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Crm.Sdk, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
File name: 'Microsoft.Crm.Sdk, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'

Another possible error
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Crm.Sdk, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Crm.Sdk, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.