Advanced Developer Extensions - an update

Just a quick note to say that I've updated my post on the Advanced Developer Extensions. Shan McArthur of AdxStudio (who developed these extensions) was kind enough to provide extra information about these extensions, which I've now incorporated into the original post

CRM SDK 4.0.12 and the Advanced Developer Extensions

The CRM 4.0.12 SDK has recently been released. Normally an SDK update is not particularly significant, but in this case it includes some major enhancements, which come under the banner of 'Advanced Developer Extensions'. David Yack has already posted a quick how-to on the CRM Team Blog; rather than duplicate that, this post is intended to cover the scope and limitations of the new extensions as I see them, and how they differ from the original CRM programming model.

What are the new extensions ?
It seems to make sense to split the new features into 2; the 'Advanced Developer Extensions' (whcih has been helpfully shortened to Microsoft xRM), and the Portal Accelerator (aka Portal Developer). The Portal accelerator uses Microsoft xRM, but I think it is otherwise best treated separately. So, for this post I'll concentrate on Microsoft xRM.

Architecture of Microsoft xRM
Although Microsoft xRM appears to provide a whole new programming model, it can be considered as essentially a (rather big) wrapper around the existing SDK assemblies (microsoft.crm.sdk and microsoft.crm.sdktypeproxy). So, although your code would not directly use the classes in the SDK assemblies, the communication with CRM is still ultimately done via the CRM web services, and is subject to the same limitations (e.g. limitations of the FetchXml query syntax). Another consequence of this is that you do not need to change any of your network configuration to use these extensions.

Changes to the Programming Model
This is where it gets interesting; the new extensions provide a whole new programming model, which can affect pretty well all of the code you use to communicate with CRM. The major changes as I see it are:

1. You can use strongly-typed classes for the CRM entities. Although you can do this with the existing SOAP CRM web services, up till now you needed to use the DynamicEntity class with the SDK assemblies
2. Native .Net types are used for attributes - e.g. int? rather than CrmNumber. Note that nullable types (e.g. int?, rather than int) as used
3. The connection information is covered within a DataContext class, which effectively replaces the use of the CrmService instance
4. Data modifications can be cached locally, then submitted as a batch, using the DataContext.SaveChanges method
5. The extensions provide additional methods to update lookup values. These can work off either the lookup attribute name (e.g. customerid) or the relationship name (e.g. contact_customer_accounts). The extensions also provide methods to retrieve related records, which avoids the need to use QueryExpression or QueryByAttribute
6. You can use LINQ queries to retrieve data from CRM, rather than building QueryExpression instances
7. The DataContext and LINQ technologies allow direct data-binding with .Net user interface controls

Use of Strongly-Typed classes
As with the SOAP web services, you can use strongly-types classes for system and custom entities. Superficially the process for setting this up differs from the SOAP web services, although the underlying idea is pretty similar. With these extensions, you use a supplied tool (CrmSvcUtil.exe) to connect to a CRM server. This tool will generate the class definitions for all CRM entities into one output code file which you'll add into your .Net project. Ultimately, this process is very similar to what happens behind the scenes when you create a Web Reference to the SOAP web services. The main internal difference is that the generated classes with these extensions maps down to the DynamicEntity class, but this is hidden from you.

You can still use a generic class with these extensions rather than strongly-typed classes. With the extensions it is ICrmEntity, rather than DynamicEntity.

Native .Net Types
Native .Net Types are used instead of Crm-specific types. The extensions use the nullable versions of the types (e.g. int?, rather than int) so that you can still identify null values (which was one of the main original reasons for the Crm-specific types). For picklist and status attributes the extensions provide an additional Label attribute (e.g. customertypecodeLabel) with the appropriate text, whereas for lookup attributes you can get the related entity via a property that has the relationship name (e.g. price_level_accounts).

DataContext class
This replaces the need for the CrmService instance, and handles the selection of authentication type, passing credentials and management of the CrmAuthentication token. All connection-related information (server URL and port, organisation name, authentication type and credentials) can be specified in one connection string. The extension code includes logic to read connection string information from the application config file. Overall, this should greatly simplify deployment across different environments and authentication types.

The DataContext exposes an IOrganizationService instance, which looks to combine the underlying ICrmService and IMetadataService instances. This allows use of any of the more specific CrmService messages (e.g. InstantiateTemplate), or MetadataService messages.

Interestingly the constructor for the DataContext can take an IContextService or IWorkflowContext instance, but not an IPluginExecutionContext instance. This implies that the extensions can work within a custom workflow activity, but are of limited use within plugins. See below for more on this.

Batch Updates
It looks to me as though any changes submitted via the DataContext will be submitted as a batch via the SaveChanges method. This won't provide any transactional support, as this is not possible within the Crm Web Services. Overall, I think this approach is due to the design patterns used in the Microsoft data-related classes, and I'm pretty neutral as to whether this offers more benefits or drawbacks. If you do develop with these extensions, I'd bear the following in mind:

• I could imagine developers forgetting to include the SaveChanges method, and I can't see (though I've not tested this) a way that the extension code would throw an error is pending changes were discarded
• There seems to be no means to control the behaviour if an individual data operation fails, and the documentation doesn't currently describe the default behaviour here. To get such control you need to call SaveChanges for each individual data modification

Handling lookups and relationships
I've not looked at this in great detail, but I see the greatest benefit is the simplicity of retrieving related records with methods such as GetRelatedEntities to get child and many-many entities, and GetRelatedEntity to get a parent entity.

Use of LINQ queries
Again, not an area I've spent much time with. In the first instance I see this as most useful for developers that are already familiar with the LINQ query syntax, but I generally encourage the use of standard technologies (such as LINQ) in preference to application-specific technologies (such as QueryExpression). My expectation is that you should expect developer productivity gains with use of LINQ instead of QueryExpression, but I've not spent enough time on this to get good metrics. It should be emphasised that the LINQ query ultimately maps down to a QueryExpression, and hence you are as limited in the scope of queries as you currently are.

Data Binding
This really comes as a direct benefit of the use of standard .Net data classes, and LINQ. You can directly bind UI components like the ASP.Net GridView control to the results of a LINQ query. Oddly, there doesn't seem to be an example of this in the main crmsdk4.chm help file, but the following code extract from the 'advanced_developer_extensions_-_walkthrough_webapp.docx' document in the SDK shows how easy and powerful this can be:

var crm = new XrmDataContext("Crm");
ContactsGrid.DataSource = crm.contacts.Where
(c => c.emailaddress1.EndsWith("@example.com"));
ContactsGrid.DataBind();

Limitations of the Advanced Developer Extensions
There's a lot of very good stuff in the areas mentioned above, which could have significant benefits on developer productivity. However, I don't think you can immediately replace all existing .Net code (even if you wanted to), as there are some areas that I don't think these extensions reach (yet?). Note that the following list is based on my investigations so far, and may include features that do exist, but which I've missed in my brief analysis so far.

1. Plugins. As mentioned above, the constructor for the XrmDataContext class has no overload that takes an IPluginExecutionContext. I also can't see a way to manipulate a DynamicEntity (e.g. from InputParameters) within these extensions, so I don't think they are viable for use in plugin code
2. IFD (aka SPLA) impersonation. AD impersonation is supported, but I can't see the equivalent of the CrmImpersonator class. I need to do some further tests on this to see if IFD impersonation can work with these extensions; if not it would restrict the deployment scenarios for web extensions

It's also worth pointing out that the major benefits of the extension code relate to standard data operations (Create, Retrieve, Update, Delete). You can still use the additional messages (e.g. InstantiateTemplate, Merge) of the CrmService and MetadataService through these extensions, but you'll need to use the standard SDK types

My understanding is that these extensions originated as a basis for developing portal-style applications against CRM, so in that context it is not at all surprising that there are areas where the code can't reach. It'll be interesting to see how this changes, both with CRM 4 and CRM 5.

What Next ?
What next, indeed. The timing of this release is interesting, coming relatively late in the life of CRM 4 as the most recent version of CRM. It's still a little too early to know what will happen with CRM 5, but it would be logical to expect that these extensions will continue to work with future versions of CRM. The more interesting question is whether we will continue to have more that one programming model for CRM 5 (e.g. a native interface, and extensions such as these which form a wrapper around the native interface), or whether these 2 models will start to merge together.

CRM 4.0 UR 10

It's that time again; a new Update Rollup is out. See
http://support.microsoft.com/?kbid=979347
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=5da4d352-1d3f-4426-9c17-b256c30afdb9

I think the most significant fix in this rollup is the fix for metadata in new Organisation databases, as described at http://support.microsoft.com/kb/980627/ . This fixes a bug that started in UR 7, in which a newly created organisation (i.e. created after rollup 7 or higher had been installed) missed some SQL objects and permissions. The 2 symptoms I'd met of this were:

1. Missing stored procedures for cascading relationships. This gives the error: SqlException: Could not find stored procedure 'dbo.p_CascadeCollectAssign'
2. Missing permissions on the FilteredActivityPointer view. This gives the error: the SELECT permission was denied on the object 'FilteredActivityPointer', and is described here

A bit of SQL - finding CRM attributes with no data

Over time I've been asked to review various existing CRM implementations. One thing I often like to check is whether there are any attributes that are not in use. To this end I created the stored procedure at the end of this post.

The stored procedure lists the attributes in an entity along with how many records contain data in that attribute. The parameters in the procedure allow you to limit the attributes to custom attributes, and to specify a maximum count of values (above which the attribute data won't be output).

Notes:

1. The procedure as written would be created in the MSCRM organisation database. You could create it in a different database, and add the databasename.schemaname. prefixes if you want
2. The procedure uses the supported route of querying the filtered view for an entity. If you have SQL dbo permission, and don't mind deleted records being included in the counts, then you can run the query against the base view, which is considerably faster. To do this, remove the string 'Filtered' from the procedure definition

Use the following SQL to create the stored procedure:

create proc pExcCheckColumnValues @entityName nvarchar(64), @customOnly bit = 0, @maxThreshold int = null
as
declare @attrName nvarchar(50), @sql nvarchar(1024)
declare cur cursor fast_forward for
select a.name from attribute a
join entity e on a.entityid = e.entityid
where e.name = @entityName and a.IsLogical = 0
and ((@customOnly = 1 and a.IsCustomField = 1) or isnull(@customOnly, 0) = 0)
create table #tmp (EntityName nvarchar(64), AttributeName nvarchar(50), ValueCount int)
open cur
fetch next from cur into @attrName
while @@fetch_status = 0
begin
set @sql = 'insert #tmp select ''' + @entityName + ''', ''' + @attrName + ''', count(*) from Filtered' + @entityName + ' where ' + @attrName + ' is not null'
+ case when @maxThreshold is not null then ' having count(*) <= ' + cast(@maxThreshold as nvarchar) else '' end
exec (@sql)
fetch next from cur into @attrName
end
close cur
deallocate cur
select * from #tmp
go

To give an example of how to use the procedure, the following will return all custom attributes of the account, where no more than 10 account records have a value in this field:

exec pExcCheckColumnValues @entityName = 'account', @customOnly = 1, @maxThreshold = 10

Service Principle Names and Security

I've recently been working with CRM and Reporting Services in environments where AD delegation has been required, and hence it's been important to configure Service Principle Names (SPN).

A good overview of SPNs and how they are set can be found in this document for CRM 3. Although some of the steps in that document have been superseded by the use of the CRM 4.0 Reporting Services connector, the document is still a good reference for the main aspects of setting up SPNs.

There are two further considerations that are not covered in the above document though; ensuring Kerberos is used, and getting permission to modify SPNs.

Ensure Kerberos is used
There are 2 Windows authentication mechanisms: NTLM (aka Challenge/Response) and Kerberos. Only Kerberos supports delegation. By default, Kerberos will be used if the client supports it (this setting is known as 'Negotiate'), but this behaviour can be changed.

For web applications hosted in IIS (such as CRM and Reporting Servers 2005), this is controlled by the NTAuthenticationProvider setting in the IIS metabase. Note that this can be set at a few different levels, and you'll need to check them all to determine the current configuration. When you install CRM, the Environment Diagnostics Wizard requires that Negotiate is the default setting, though it could have been changed after install.

Reporting Services 2008 does not use IIS, and the authentication settings are configured in the rsreportserver.config file. Information about these settings can be found here in SQL Server 2008 Books Online.

Permissions required to modify SPNs
It's all very well knowing what SPNs need to be created or modified, but you may not have permission to do this, especially in high security environments. SPN data is stored in Active Directory, hence you need appropriate permissions in Active Directory. There are two approaches that you can take to get the SPNs setup if you don't have permission:

1. Ask a Domain Administrator to grant you the permissions. The second half of this technet article describes the permissions you need, and how they are assigned
2. The individual service accounts can be granted permission to create their own SPNs. This is more applicable to SQL Server SPNs, but can be used by any code that can register it's own SPN. This is well described here

Update Rollup 9

It's that time of every other month again. A new update rollup is available; number 9 is available here. There don't seem to be any particularly important fixes this time, unless you're in one of the time zones affect by this bug, which is now fixed

CRM 4 Update Rollup 8 is out

See http://support.microsoft.com/default.aspx?kbid=975995 and http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=c53b2916-6b93-4092-bdd3-a394c96ca000

I've not had a chance to test it yet. One welcome inclusion is an optimisation to the Import Organisation process (http://support.microsoft.com/kb/977867/ ), though interestingly you have to add a registry value to enable the optimisation.

There's also appraently a fix to the 'dsMain' error with Report Wizard, as described here

Analysing SQL Queries in CRM environments

I've another post here on the Dynamics CRM team blog describing some techniques I use to make effective use of SQL Profiler and SQL Management Studio. This post also references a tool I published here on the MSDN Code Gallery

New Code Gallery resource - sp_executesql parser

I've added a new code resource to the MSDN code gallery. This is a simple tool to parse calls to sp_executesql. The reason for it is that the SQL tools cannot show the execution plan for calls to sp_executesql, which is a pain when doing performance analysis of these queries. So, I created a tool that replaces the manual editing process.

Soon, I'll post a more detailed description of SQL performance analysis for CRM implementations, which will reference this tool.

Attachment Download code in UR 7 - an update

I've updated my original post here to reflect a suggested fix posted by Microsoft on the MSDN Forums here.

Despite my major reservations with how the problem arose, credit is due to Microsoft for responding as quickly as I think they possibly could (even if it doesn't always seem that way).

My reservations relate to disagreements on the need for the change made in UR7 (I don't think it increases security at all), and also to how MS allowed a change that broke supported extensions. It'll be interesting to see what happens with UR 8...