Options for upgrading ASP .Net extensions for CRM 2011

I like April. One reason is that it's the start of my MVP renewal cycle. After April 1st April I will either have been renewed, or not, and I feel less circumspect about making critical comments about decisions Microsoft have made. So, ASP .Net extensions with CRM 2011. Since the first release of CRM (CRM 1.0 or CRM 1.2 depending on your country), a major extension point for On-Premise CRM implementations was to develop ASP .Net extensions and deploy them within the CRM web site. Having a supported way to place these extensions within the CRM web site was important for several reasons:

1. To allow relative Urls in IFrames, ISV.Config and SiteMap. Main reasons for that are to cope with IFD environments where different domain names are provided for internal and external access, and to avoid configuration issues exporting/importing between environments
2. To allow single-sign on and impersonation, so the code could act on behalf of the CRM user, without needing to re-enter their credentials
3. To maintain the Site Origin (aka same site of origin). This is an important consideration as Internet Explorer security will only permit code interaction between pages if it considers that they are part of the same site. For example, the ability to pass data to a dialog using window.dialogArguments, or the ability to access objects on a calling window via window.opener both depend on the pages being in the same site

However, CRM 2011 puts significant restrictions on putting your ASP .Net extensions within the CRM web site. Essentially, the only supported option is to retain existing extensions that use the CRM 4 endpoint. So, what happens to the points above if you can't put your ASP .Net extensions in the CRM web site:

1. You'll have to use absolute Urls. This makes deployment between environments (e.g. from development to live) harder, as the Urls would have to changed. In small-scale environments this would be just a manual task, but in larger environments you may decide to build a process to automate this. Overall, I see this as a major annoyance, but not a major problem
2. Microsoft have put work into making single sign-on work across web sites. This depends on setting up Secure Token Services, which incurs some administrative and deployment overhead. There's an additional deployment overhead of setting up a new web site for the extensions, and configuring access to it. I've not tested this fully, but assuming it works as promised, this should resolve the single sign-on and impersonation issue
3. This is the big problem area. I don't think same site of origin can be maintained with the ASP .Net pages outside of the CRM web site, which effectively removes support for a common type of extension that was possible in all previous versions (Note, IE 8 has some settings that affect how Site Origin is applied, and this might help, but IE 7 is a supported browser for CRM 2011, so this cannot be a universal solution)

So, what can/should you do with your ASP .Net extensions that you wrote for CRM 4.0, or were intending to write. I see 3 main options:

1. Don't upgrade your code. ASP .Net code that uses the CRM 4 web service endpoint will still work in the ISV folder, and Microsoft have not said it is unsupported. This is the simplest option, but it means you need to maintain CRM 4 code, and you'll still have to address the issue when CRM 6 comes out. You could cross your fingers and hope that, in CRM 6, Microsoft reintroduce support for ASP .Net extensions within the CRM web site
2. Upgrade the code to use the CRM 2011 endpoint, and deploy it in a separate web-site from CRM. As stated above, this causes extra deployment overhead (which I consider is a significant overhead, which is often under-estimated), and you won't be able to use same site of origin, so you have to expect some limitations
3. Rewrite the code as web resources (Silverlight, or HTML with javascript). Microsoft have introduced quite a lot of integration points in CRM 2011 that makes this a powerful option. This has a major advantage that the resources are necessarily hosted within the CRM web site, and can be deployed as part of a CRM solution. However, these are client-side technologies and some extensions would need extra work to build (e.g. extensions that access a database on another server). My biggest problem though is the development effort required to rewrite code in substantially different technologies, and these are technologies that are not as mature as ASP .Net

Of these, I don't like any of the options. Numbers 2 and 3 would be necessary if you need to support CRM Online, but for On-Premise implementations there are some difficult decisions to make. Don't get me wrong, I appreciate that there are a lot of good things for developers in CRM 2011; web resources and the single sign-on across web-sites are very powerful and very welcome, and the only options for CRM Online, it's just a shame to lose some On-Premise options.

But to finish on a more positive note, another reason I like April is that this is a great month for ski touring.

Removing prompt for credentials when browsing with Internet Explorer

This post isn't intended to be a complete list of solutions to issue when you are unexpectedly prompted for AD credentials when browsing with Internet Explorer, but it gives some rules of thumb regarding where to start looking. The scenario: You try accessing a page using IE, and are prompted by a 'Windows Security' dialog for AD credentials when you don't expect it.

Possible causes: There are many, but the first thing to do is work out if it's a server-side issue, or a client-side issue. A simple test is what happens if you provide valid credentials in the Windows Security dialog:

• If you can then connect, then this is a client-side issue
• If you are prompted again, either 2 or 3 times, then get a permission error (normally HTTP 401), then it is a server-side issue

Client-side issues If it's a client-side issue, then look at the client IE settings, and the URL of the web page: If IE considers that you have already logged into the DNS domain (the part of the url prior to the first single / - e.g. http://crm:5555/), then it should reuse these credentials and you won't be prompted to login. However, IE is picky about matching the DNS domain, so if you've already logged into http://crm, then it won't trust other aliases (e.g. http://localhost, http://crm.mydom.com, http://192.168.0.1) and will prompt for credentials.

If you're using CRM 4 and have extension pages in the ISV directory of the CRM web site, then it is best to provide the URL as a relative path (e.g. /ISV/MyCompany/MyPage.aspx) rather than an absolute path, to avoid this issue.

The IE security settings will determine whether IE will try submitting your logged-on credentials. By default, it will only do this if you connect to a site in the Local Intranet Zone, so check the IE security settings, and the zone of the web site you're connecting to

Server-side issues If it's a server-side issue, then there are many possible causes, but most of them come to Kerberos in one way or another

Possible SQL Gotcha - use of 'Not In' with NULLs and the customer attribute

I was recently putting together a bit of SQL to illustrate the use of a NOT IN clause for a forum answer, and got some unexpected results. The query was a relatively simple example; find all accounts with no associated opportunities. So, I tried this:

SELECT name FROM FilteredAccount
WHERE accountid NOT IN (SELECT accountid FROM FilteredOpportunity)

Nice, simple query, but it returned no data (and it should have done). However, the following works fine:

SELECT name FROM FilteredAccount
WHERE accountid NOT IN (SELECT customerid FROM FilteredOpportunity)

The only difference is the use of customerid instead of accountid in the subquery. If I'd have expected the first query to work instead of the second query, as customerid is a generated field (it's generated within the Opportunity via by the SQL function COALESCE(accountid, contactid)).

This all seems weird, but it comes down to what happens with nulls. An opportunity will be associated with one of an account, or a contact. So, the subquery 'SELECT accountid FROM FilteredOpportunity' could return a null (if you have an opportunity against a contact), but 'SELECT customerid FROM FilteredOpportunity' will always return non-null values. Don't ask me why, but the presence of nulls in the subquery cause the NOT IN query to misbehave.

One way to confirm this is with another variation on the query above, which also works:

SELECT name FROM FilteredAccount
WHERE accountid NOT IN (SELECT accountid FROM FilteredOpportunity WHERE accountid is NOT NULL)

This query explicitly excludes nulls from the results on the subquery, and so it works fine.

The main lesson I took from this is to always test for nulls in the subquery when using NOT IN; another lesson is to pay close attention when using attributes that represent the composite Customer data type in CRM

For reference, the reason why I was doing this is because this is a classic example of a query that cannot be done through FetchXML, and hence cannot be written with an Advanced Find in CRM. If the primary entity is an account, contact or lead then you have a manual workaround in CRM, for example:

• Create a marketing list, and populate it with all accounts
• Use Advanced Find to remove from the list all accounts that have an opportunity
• This will then leave you with a marketing list that contains all accounts without an opportunity

CRM 2011 RTM Release

Seems like the day for announcements: the CRM 2011 RTM code has been released for On-Premise and Partner-Hosted environments. This is nearly 2 weeks earlier than I expected, so congratulations to the CRM product team.

The server software can be downloaded here, and that page has links to download the other components. The build number is 05.00.9688.583, which is consistent with the build number of the binaries in the release SDK.

Using CRM 4.0 assemblies on a CRM 2011 Server

CRM 2011 Server includes a publisher policy that causes any assembly built against the CRM 4 sdk assemblies to load the CRM 5 sdk assemblies instead. There are certain circumstances where this can cause errors loading the assembly; see the end of this post for possible error messages.

One workaround is to not run the application on a Crm 2011 Server, but there is an alternative, which is to explictly tell your application not to use this publisher policy file. This is done through adding the following to the app.config file:

<configuration>
<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<dependentAssembly>
<assemblyIdentity name="Microsoft.Crm.Sdk" publicKeyToken="31bf3856ad364e35" culture="neutral" />
<publisherPolicy apply="no" />
</dependentAssembly>
</assemblyBinding>
</runtime>
</configuration>

This raises one more issue: in some circumstances your assembly may not be the main .exe, but a .dll loaded by another process, in which case you'll have to modify/create the .config file for that .exe. This is done by creating a file named .exe.config in the same directory as the .exe (here's an example). I have a nagging concern that I may have to do this with SSIS packages that use a custom component that use the SDK assemblies, which could get interesting, as different executables are used for in design, debug and runtime. If I do have this issue with SSIS, then I'll post a more detailed workaround (if I find it).

My hope is that this is a temporary problem that will be fixed, as the readme in the 5.0.1 version of the SDK refers to an 'incorrect Publisher Policy'. This readme also gives an explanation of this issue

One possible error
System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Crm.Sdk, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
File name: 'Microsoft.Crm.Sdk, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' ---> System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Crm.Sdk, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
File name: 'Microsoft.Crm.Sdk, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'

Another possible error
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Crm.Sdk, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Crm.Sdk, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.

CRM 2011 Documentation Released

The CRM 2011 Implementation Guide was released last week, and it looks like the CRM 2011 SDK has also been released. As far as I can tell, the SDK is a live (as opposed to beta) release, though the hands-on labs were built on beta code. The SDK documentation has version 5.0.1, and it includes binaries with version 5.0.9688.583 which are claimed to match those for CRM 2011 Online. We'll see what version number we get with the RTM code.

And the CRM 2011 Developer Training Kit has also been released recently.

I'm deliberately using the term released to indicate that they are publicly available (as opposed to 'launch', which I consider a marketing event).

Plugin Deployment Options

The CRM 4 SDK gives some information about the storage options when registering plugins but there are a few more considerations. I got prompted to elaborate on this in a forum post, and I think it's worth documenting this here as well:

The 3 storage options are: Database, Disk and GAC. The main differences between these are:

• Database: The assembly dll is stored in the database, rather than the file system. The major advantages are that the assembly need only be deployed once if you have multiple CRM servers, and that no additional action is required to restore / redeploy the assembly either during disaster recovery, or if redeploying to an alternate server. This is the preferred option in a production environment
• Disk: The assembly dll is placed in the \server\bin\assembly directory on each server. You have to ensure the dll is placed in the correct place on all CRM servers, so the deployment overhead is a little greater. I normally use this option in development environments as you can redeploy newer versions solely by file transfer, rather than reregistering. Also, if debugging, the assembly .pdb file needs to be placed in the same location; with this option it's easy to ensure the dll and pdb are from the same build
• GAC: The assembly is placed in the Global Assembly Cache on each CRM server, and again you will have to do this. The GAC does allow multiple versions of an assembly, but CRM doesn't, so you don't really gain anything by using the GAC. I don't think I've ever used this option

There is one further consideration. If your plugin assembly has other dependent assemblies, then you can place this dependent assembly in the GAC whichever of the above options you take. However, if you use the Disk option, then the dependent assemblies can also be deployed into the \server\bin\assembly directory

.Net Framework versions of custom components with SQL 2008 R2 BIDS

Rather a long title, but I couldn't think of anything shorter. Anyway, it's a topic that I would have preferred was better publicised.

SQL 2008 R2 Business Intelligence Development Studio (BIDS) will only recognise extension components (such as SSIS Data Flow Components) that are built against .Net Framework 3.5. Neither earlier nor later versions will work, and I've yet to find any useful messages to tell you why.

So far I've only done enough testing to find combinations that definitely work, as summarised in the following table.

BIDS Version	.Net Framework version of component
SQL 2005	2.0
SQL 2008	2.0
SQL 2008 R2	3.5

Stability issues with AsyncRemoveCompletedJobs

CRM 4.0 UR 3 brought in a useful feature, the ability to configure the CRM Asynchronous Service to automatically delete records from completed asynchronous operations, and hence keep the size of the asyncoperationbase SQL table down to a reasonable size. This behaviour is configured by the registry values AsyncRemoveCompletedJobs and AsyncRemoveCompletedWorkflows

However, I recently met an issue with this behaviour, where the CRM Asynchronous Service appears to get in a state where all it is doing is deleting completed jobs, to the exclusion of all other activity. This can leave the CRM Asynchronous Service to have effectively hung (not responding to service control requests, nor polling for new jobs to process) and not to process any new jobs for a considerable period of time (in one environment, this could be several hours).

The main symptoms are:

• No jobs being processed for a considerable period of time
• The Crm Asynchronous Service not responding to service control requests (i.e. you cannot stop it through the Services console, so you have to kill the process)
• No values reported for most performance counters (e.g. 'Total Operations Outstanding', 'Threads in use')
• If you do restart the service, you see a burst of activity (including performance counters) whilst outstanding jobs are processed, then it reverts to the same behaviour as above
• If you look at the SQL requests submitted by the Crm Asynchronous Service (I use the SQL dynamic management views sys.dm_exec_requests and sys.dm_exec_sessions) you see just one DELETE request and no other SQL activity

At the moment, the only workaround I have is to remove the registry values, and to use a scheduled SQL job to periodically clear out the asyncoperationbase table. Here is an example of such a script.

How to use impersonation in an ASP .Net page using IFD in CRM 4.0

This is is common requirement, and I've never found what I consider to be a suitable explanation of what needs to be done. This post is not intended to be exhaustive, but is intended to cover the essentials in one place.

The fundamental requirement is to create a custom ASP .Net page that is accessible both internally (via AD authentication) and over the Internet (via IFD authentication), and where the code will access CRM data under the context of the user accessing the page. To do this, you need to deploy and configure your code as follows:

1. Deploy the ASP .Net page within the CRM Web Site (the only supported place is within the ISV directory). If you don't do this, then IFD authentication will not apply to your page
2. Run the ASP .Net page within the CrmAppPool, and do not create an IIS application for it. If you don't do this, then you won't be able to identify the authenticated user
3. Ensure that the CRM HttpModules MapOrg and CrmAuthentication are enabled. This will happen by default by inheritance of the settings from the root web.config file in the CRM web site, but I'm mentioning it here as there are some circumstances (when you don't need IFD) in which it is appropriate to disable these HttpModules. Again, if the HttpModules aren't enabled, then you won't be able to identify the authenticated user
4. As your code is in a virtual directory (rather than a separate IIS application), ASP .Net will look for your assemblies in the [webroot]\bin folder, so that is where you should put them (or in the GAC). The initial release documentation for CRM 4.0 stated that it was unsupported to put files in [webroot]\bin folder of the CRM web site, but this restriction has been lifted

You also need to follow certain coding patterns within your code. An example of these can be found here. Note that, Crm web services refers to both the CrmService and the MetadataService:

1. Ensure you can identify the organisation name. The example code shows how to parse this from the Request.Url property, though I prefer to pass this on the querystring (which the example also supports)
2. Use the CrmImpersonator class. All access to the Crm web services needs to be wrapped within the using (new CrmImpersonator()) block. If you don't do this you will probably get 401 errors, often when accessing the page internally via AD authentication (see later for a brief explanation)
3. Use the ExtractCrmAuthenticationToken static method. This is necessary to get the context of the calling user (which is stored in the CallerId property)
4. Use CredentialCache.DefaultCredentials to pass AD credentials to the Crm web services. If you don't do this, then you will probably get 401 errors as you'd be trying to access the web service anonymously (IIS would throw these 401 errors)

That should be all that you need on the server side. The final piece of the puzzle is to ensure that you provide the correct Url when accessing the page, which again needs a little consideration:

When accessing the page from an internal address, the Url should be of the form:
http://[server]/[orgname]/ISV/MyFolder/MyPage.aspx

When accessing the page from an external address, the Url should be of the form:
http://[orgname].[serverFQDN]/ISV/MyFolder/MyPage.aspx

This is relatively easy to achieve when opening the page from within CRM (i.e. in an IFrame, via an ISV.config button or in client script). In each case you can use the PrependOrgName global function in client script - e.g.

var u = PrependOrgName('/ISV/MyFolder/MyPage.aspx');

This function will determine correctly whether to add the organisation name to the Url. Note also that I've provided a relative Url, which will ensure the first part of the Url is always correct. As this uses a JavaScript function, you will always need to use a small piece of code to access the page, and cannot rely on statically providing the Url in the source of an IFrame, or in the Url attribute of an ISV.Config button. Any relative Urls in SiteMap should automatically get the organisation name applied correctly. Remember to also pass the organisation name on the querystring if the server code expects this (you can get the organisation name from the ORG_UNIQUE_NAME global variable)

Earlier I promised an explanation of what the server code does. This is not as complete an explanation as it could be, but the basics are:

1. The HttpModules identify the correct CRM organisation (MapOrg) from the Url provided, and place information about the authenticated calling user in the HttpContext (CrmAuthentication)
2. The ExtractCrmAuthenticationToken method reads the user context from the HttpContext, and puts the user's systemuserid in the CallerId property of the CrmAuthenticationToken
3. Because the CallerId is set, the call to CRM is necessarily using CRM impersonation. For this to be permitted, the execution account (see Dave Berry's blog for a definition) must be a member of the AD group PrivUserGroup. The execution account is the AD account that is returned by CredentialCache.DefaultCredentials. This is where things get a little interesting
4. If the request comes via the external network and IFD authentication is used, CRM handles the authentication outside of IIS and no IIS / ASP .Net impersonation occurs. Therefore CredentialCache.DefaultCredentials will return the AD identity of the process, which is the identity of the CrmAppPool, which necessarily is a member of PrivUserGroup
5. However, if the request comes via the internal network, AD authentication is used and IIS / ASP .Net impersonation does occur (through the setting in web.config). This impersonation will change the execution context of the thread, and CredentialCache.DefaultCredentials would then return the AD context of the caller. This is fine in a pure AD authentication scenario, but the use of the ExtractCrmAuthenticationToken method means that CRM impersonation is necessarily expected; this will only work if the execution account is a member of PrivUserGroup, and CRM users should not be members of PrivUserGroup. This is where the CrmImpersonator class comes in: its constructor reverts the thread's execution context to that of the process (i.e. it undoes the IIS / ASP .Net impersonation), so that CredentialCache.DefaultCredentials will now return the identity of the CrmAppPool, and the CRM platform will permit CRM impersonation

To finish off, here are a few other points to note:

• IFD impersonation only applies when accessing the CRM platform. If you use IFD authentication, there is no way of impersonating the caller when accessing non-CRM resources (e.g. SQL databases, SharePoint, the file system); it cannot be done, so save yourself the effort and don't even try (though, for completeness, SQL impersonation is possible using EXECUTE AS, but that's it)
• If you want to use impersonation, do not use the CrmDiscoveryService. The CrmDiscoveryService can only be used with IFD if you know the user's username and password, and you won't know these unless you prompt the user, which kind of defeats the point of impersonation