A bit of SQL - finding CRM attributes with no data

Over time I've been asked to review various existing CRM implementations. One thing I often like to check is whether there are any attributes that are not in use. To this end I created the stored procedure at the end of this post.

The stored procedure lists the attributes in an entity along with how many records contain data in that attribute. The parameters in the procedure allow you to limit the attributes to custom attributes, and to specify a maximum count of values (above which the attribute data won't be output).

Notes:

1. The procedure as written would be created in the MSCRM organisation database. You could create it in a different database, and add the databasename.schemaname. prefixes if you want
2. The procedure uses the supported route of querying the filtered view for an entity. If you have SQL dbo permission, and don't mind deleted records being included in the counts, then you can run the query against the base view, which is considerably faster. To do this, remove the string 'Filtered' from the procedure definition

Use the following SQL to create the stored procedure:

create proc pExcCheckColumnValues @entityName nvarchar(64), @customOnly bit = 0, @maxThreshold int = null
as
declare @attrName nvarchar(50), @sql nvarchar(1024)
declare cur cursor fast_forward for
select a.name from attribute a
join entity e on a.entityid = e.entityid
where e.name = @entityName and a.IsLogical = 0
and ((@customOnly = 1 and a.IsCustomField = 1) or isnull(@customOnly, 0) = 0)
create table #tmp (EntityName nvarchar(64), AttributeName nvarchar(50), ValueCount int)
open cur
fetch next from cur into @attrName
while @@fetch_status = 0
begin
set @sql = 'insert #tmp select ''' + @entityName + ''', ''' + @attrName + ''', count(*) from Filtered' + @entityName + ' where ' + @attrName + ' is not null'
+ case when @maxThreshold is not null then ' having count(*) <= ' + cast(@maxThreshold as nvarchar) else '' end
exec (@sql)
fetch next from cur into @attrName
end
close cur
deallocate cur
select * from #tmp
go

To give an example of how to use the procedure, the following will return all custom attributes of the account, where no more than 10 account records have a value in this field:

exec pExcCheckColumnValues @entityName = 'account', @customOnly = 1, @maxThreshold = 10

Service Principle Names and Security

I've recently been working with CRM and Reporting Services in environments where AD delegation has been required, and hence it's been important to configure Service Principle Names (SPN).

A good overview of SPNs and how they are set can be found in this document for CRM 3. Although some of the steps in that document have been superseded by the use of the CRM 4.0 Reporting Services connector, the document is still a good reference for the main aspects of setting up SPNs.

There are two further considerations that are not covered in the above document though; ensuring Kerberos is used, and getting permission to modify SPNs.

Ensure Kerberos is used
There are 2 Windows authentication mechanisms: NTLM (aka Challenge/Response) and Kerberos. Only Kerberos supports delegation. By default, Kerberos will be used if the client supports it (this setting is known as 'Negotiate'), but this behaviour can be changed.

For web applications hosted in IIS (such as CRM and Reporting Servers 2005), this is controlled by the NTAuthenticationProvider setting in the IIS metabase. Note that this can be set at a few different levels, and you'll need to check them all to determine the current configuration. When you install CRM, the Environment Diagnostics Wizard requires that Negotiate is the default setting, though it could have been changed after install.

Reporting Services 2008 does not use IIS, and the authentication settings are configured in the rsreportserver.config file. Information about these settings can be found here in SQL Server 2008 Books Online.

Permissions required to modify SPNs
It's all very well knowing what SPNs need to be created or modified, but you may not have permission to do this, especially in high security environments. SPN data is stored in Active Directory, hence you need appropriate permissions in Active Directory. There are two approaches that you can take to get the SPNs setup if you don't have permission:

1. Ask a Domain Administrator to grant you the permissions. The second half of this technet article describes the permissions you need, and how they are assigned
2. The individual service accounts can be granted permission to create their own SPNs. This is more applicable to SQL Server SPNs, but can be used by any code that can register it's own SPN. This is well described here

Update Rollup 9

It's that time of every other month again. A new update rollup is available; number 9 is available here. There don't seem to be any particularly important fixes this time, unless you're in one of the time zones affect by this bug, which is now fixed

CRM 4 Update Rollup 8 is out

See http://support.microsoft.com/default.aspx?kbid=975995 and http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=c53b2916-6b93-4092-bdd3-a394c96ca000

I've not had a chance to test it yet. One welcome inclusion is an optimisation to the Import Organisation process (http://support.microsoft.com/kb/977867/ ), though interestingly you have to add a registry value to enable the optimisation.

There's also appraently a fix to the 'dsMain' error with Report Wizard, as described here

Analysing SQL Queries in CRM environments

I've another post here on the Dynamics CRM team blog describing some techniques I use to make effective use of SQL Profiler and SQL Management Studio. This post also references a tool I published here on the MSDN Code Gallery

New Code Gallery resource - sp_executesql parser

I've added a new code resource to the MSDN code gallery. This is a simple tool to parse calls to sp_executesql. The reason for it is that the SQL tools cannot show the execution plan for calls to sp_executesql, which is a pain when doing performance analysis of these queries. So, I created a tool that replaces the manual editing process.

Soon, I'll post a more detailed description of SQL performance analysis for CRM implementations, which will reference this tool.

Attachment Download code in UR 7 - an update

I've updated my original post here to reflect a suggested fix posted by Microsoft on the MSDN Forums here.

Despite my major reservations with how the problem arose, credit is due to Microsoft for responding as quickly as I think they possibly could (even if it doesn't always seem that way).

My reservations relate to disagreements on the need for the change made in UR7 (I don't think it increases security at all), and also to how MS allowed a change that broke supported extensions. It'll be interesting to see what happens with UR 8...

Socket Exhaustion when accessing CRM web services

When submitting a large number requests to the CrmService web service, you may occasionally get the error 'Only one usage of each socket address (protocol/network address/port) is normally permitted'. The reason for this is 'socket exhaustion', which is excellently described here. You are at risk of this happening when over 4000 web requests are submitted within 4 minutes - I find this most commonly on data imports, but have recently met it during heavy user activity on a live CRM system.

There are 2 solutions:

1. As suggested in the link above, add registry values to give a much wider range of socket addresses. Note that you need to restart the server for these registry changes to take effect
2. Set the UnsafeAuthenticatedConnectionSharing and PreAuthenticate properties of your CrmService (or MetadateService) proxy to true. This will allow your web request to reuse the same socket

Field Level Security in CRM 4.0 - MS White Paper

The Microsoft Dynamics CRM Engineering for Enterprise (MS CRM E2) team have released a detailed white paper describing the supported options for implementing field-level security in CRM 4.0. It is available for download at http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=471f8670-47b3-4525-b25d-c11a6774615c .

As I was one of the document's technical reviewers, I may be a bit biased in my praise of the document, however one option is missing from it. As a Microsoft document, it concentrates on the supported approaches, but there are also unsupported approaches, such as that taken by c360, which uses an HttpModule to modify the html sent to the client. Personally I dislike this approach, as there is a potentially heavy performance overhead, and it is fragile with respect to future changes in hotfix rollup. However, it is another option that could be considered, and is unfortunately not mentioned in the white paper.

UR 7 breaks Attachment Download code, and how to fix it

* 2009-11-13. I've updated this post to reflect some proposed SDK changes published by Microsoft to the MSDN Forums here. I'll probably update this post again for reasons I'll come to later, but I think it's worth making an initial update now *

It was going to happen at some point; a CRM hotfix rollup that broke existing supported customisation code. Update Rollup 7 adds some security changes that apply to the Attachment/download.aspx page. As a result the example code documented here in the SDK no longer works.

If you try the code, the user gets the error message "Access Denied, Invalid Operation". If you enable tracing, you see a message like the following:
"Message: INVALID_WRPC_TOKEN: Validate WRPC Token: WRPCTokenState=Invalid, TOKEN_EXPIRY=4320, IGNORE_TOKEN=False, TOKEN_KEY=b1Nd0byfEd6+gwAZuez7cauYiRnfHuMLAquFY1Ks22dHgxZiW5IrxSobkv9aVfbC, ErrorCode: -2140991221"

The fundamental problem is that CRM now expects additional parameters on the query string that contain a Token, and there is no way for us to generate that token. There's a significant question about why this change was made, but I'll leave that for another time.

The recommended workaround is the use the web services directly as described at http://social.microsoft.com/Forums/en-US/crmdevelopment/thread/2f46a3c4-1123-49dd-804e-2787ef367986, which I expect to make it into the SDK soon. This is close to a like-for-like replacement for the original SDK code in that it is the type of code you might write for a console application.

However, a popular use of download.aspx was to provide any easy client-side link to download an attachment, rather than doing it programmatically with .Net code. To get equivalent behaviour you'll need a custom .aspx page that gets the uses similar code to that above, but writes it the response stream using Response.BinaryWrite. I'm going to have to write this code soon, and when I do I'll post it on the MSDN Code Gallery, and link to it here; in the meantime if anyone has to write it themselves the other thing you should do is pass information on the Response-Header to specify the filename. From memory this involves something like:

Response.AddHeader("content-disposition","attachment; filename=" + outputFile);