Tag Archives: Development

MSDN Azure credits… its not for “you”, its for “us” …

So I recently found out about a great new set of offers that Microsoft are offering for all MSDN subscription owners called “Windows Azure Benefit for MSDN Subscribers“. You basically get free Azure credits every month and discounted pricing.

  • MSDN Ultimate – $150 per month
  • MSDN Premium – $100 per month
  • MSDN Professional – $50 per month

This is also combined with a 25% discount in the charge rate for each machine that you are running, and this is fantastic value.

For most SharePoint development and testing teams you will be looking at the top two, although much more expensive the Premium ($2.5k per year) and Ultimate ($4.2k per year) these are the MSDN subscriptions which include Office and SharePoint software for development and testing purposes (check out the MSDN Edition comparison for more details). There are other options out there but a lot of development teams will be using MSDN.

Equally if you are doing SharePoint development which in the Azure world will typically mean an “Extra Large” VM (8 Cores and 14GB RAM). This rolls in at $0.48 per hour of operation, and probably raises another major point … only 14GB RAM?

With Windows 8 Hyper-V (free) and VMWare workstation (around $100) and most contractors running insane dual-SSD 32GB laptops you gotta wonder, why would I want a 14GB VM in the cloud when I can run a 24GB VM locally? Also .. what happens if I am on a train / airport lounge / plane and can’t access the internet?

Well .. good point ..

$100 a month is great, but Azure VMs are very expensive!
Now, I know a few contractor friends of mine in the industry who have looked at this and decided that its not for them .. I am one of them (yes that is right .. I’m advocating a new service which I myself am not going to use).

But this is not really for the sole contractor, and certainly not someone who works all of gods hours (either doing research, writing books or blog posts and preparing for conferences and user groups).

Now this is where the average contractor gets off the Azure train. If you are very busy and put in a lot of extra hours it is not uncommon to run your VM for 12 hours a day plus some conference / user group work at weekends … this can quickly add up*

* note – I realise not everyone works these kinds of hours .. I personally don’t, I have a wife and baby daughter at home and generally work a 9-5 work day .. but I know some people work longer hours, and I sure put in extra time when prepping for conferences

5 days a week @ 12 hours per day, plus another 12 hours over the weekend = 72 hours per week
72 * $0.48 = $34.56 per week
$34.56 * 52 = $1,797 per year
$1,797 / 12 = $149.76 per month

So if you are rolling with MSDN Premium you are going to be out of pocket, and even if you are lucky enough to be an MVP (and have MSDN Ultimate) or just have deep pockets .. you are still scraping the barrel and probably watching the clock every week to make sure you don’t go over the limit.

“You” are not their target audience … “We” are ..
I suppose this really rounds to my core point .. this subscription model is not aimed at the individual developer or contractor. It is aimed at development teams. The place I’m currently at has 5 developers working in three different countries all running MSDN Premium. This gives them a combined allowance of $500 per month of Azure credits.

Being an office-based development team it pretty much runs off standard office hours. The development machines only need to be on for office hours (typically 8am – 6pm unless there is a major version launch coming up) and almost certainly don’t need to run at weekends. With a group of users you can also look at consolidating your infrastructure (why not run a shared SQL instance so you can drop your VM hardware?). Equally you probably don’t need to run all of the services all of the time on every development machine (if you aren’t building a search solution then turn it off!).

With $500 per month to spend they can run 5 XL VMs 9am-5pm every week for free (some weeks you won’t need to have all 5 machines running .. so turning them off when you aren’t using them can help to pay for those other times when you need to run them for longer!).

Even if you do use more horsepower than that .. try putting the figures in front of your IT Manager / Head of Infrastructure … You might be surprised how happy they are to pay for the “extra” over and above those free Azure credits (some months it might cost you an extra $100 or so .. some months you won’t have to pay anything … compare that to other hosting providers and see how much it would cost you!)

How about using it for testing?
One of the other big boons (and possibly the reason I might use a farm like this) is for testing.

It doesn’t really matter how powerful your laptop is, you are never going to be able to build a truly enterprise farm on it (with redundancy in all places and all of the lights and switches turned on). The same credit you get in Azure could be used to model and build massive farms you could use for testing new topologies, or testing load balancing scenarios, or performance and load testing?)

Don’t forget, you only pay for the machines while they are turned on so instead of running 1 XL VM for 20 days a month .. why not create 30 Large VMs and run them for 5 days a month of testing?

Conclusion…
Well, this is a very interesting move from Microsoft .. and stacked up alongside their hosted Team Foundation Server offering this does create a very attractive and extremely low-cost cloud-based development scenario.

It encourages people to stick with MSDN and give Azure a go for development and testing, and I’m sure this will end up leading to many companies taking a much closer look at how Azure works for their production environments as well.

For me ? Well .. I might well use it for the next time I do a Kerberos / Load Testing presentation (the idea of setting up a massive 20 server farm to run for a few days for free sounds pretty cool and a great learning experience to boot).

If nothing else, I’m tempted to setup a VM which I leave turned off and only use it in emergencies (my laptop is broken / stolen  or my VMs are dead for some reason).

Either way .. if you have an MSDN subscription, head over and take a look. You might be surprised how useful it is!

Search Core Results Web Part with Dynamic Date and User Profile Tokens

If you just want the goodies then you can get them here:

The Big Fat Disclaimer – This has not been thoroughly tested for a production environment. I have also removed references in my snippets below to caching and error handling to try and keep it brief. The downloadable version uses both caching and error handling, but it is still really just a proof of concept and you should TEST it before you deploy it! I take no responsibility if your production servers blow up!

I must have seen this requirement dozen of times on different projects, having search results which either:

  • Filter using a User Profile Property of the current user
  • Filter using a dynamic date range (e.g. using the “TODAY” token)
  • Specify the Sort-By (which is normally restricted to either “Relevance” or “Modified Date”

The requirement for functionality of this nature come up extremely frequently on Intranet projects. For example:

“Show News Articles from the past 7 days which filter based on the user’s location”
“Show events coming up in the next 3 months”
“Show discussions / wiki entries / blog posts which include the current user’s Ask Me About values”

Example
image
FixedQuery used in the Web Part:
Author:[UPP-PreferredName] AND Write:[TODAY-180]..[TODAY] AND ContentType:Event

Well .. on my current client project these very requirements came up .. so this time I decided to knock together the basics of the web part in my spare time and then “donate” it to the project… and this post describes how I built it, what goes on under the hood, and also includes both the source code as well as a downloadable WSP package with the working Web Part in it.

Step 1 – Extending the Search Core Results Web Part
So .. to get us started, lets kick off by creating our actual Web Part. I am going to be extending the Search CoreResultsWebPart (link to MSDN).
This is easy enough to achieve by simply creating a new Web Part in Visual Studio and inheriting from the CoreResultsWebPart class. This will make sure our web part gets all of the functionality and properties that the normal Search Results web part does without any additional effort.

   1:  [ToolboxItemAttribute(false)]
   2:  public class ExtendedSearchWebPart : CoreResultsWebPart
   3:  {
   4:   
   5:  }

That is the easy bit …

Step 2 – Overriding the Query and SortOrder
Now the next bit to tackle is how to override the actual query that gets executed. Well the best place to do this is to override the ConfigureDataSourceProperties method. This method gets called before the query is actually executed against the Search engine itself.

You can then leverage the CoreResultsWebPart.DataSource property (which is of type CoreResultsDataSource). This is what allows all of the magic to happen.

   1:  protected override void ConfigureDataSourceProperties()
   2:  {
   3:      // only perform actions when we are trying to show search results
   4:      // i.e. not when you're in Design Mode
   5:      if (this.ShowSearchResults)
   6:      {
   7:          // call the base web part method
   8:          base.ConfigureDataSourceProperties();
   9:   
  10:          // get the data source object
  11:          CoreResultsDatasource dataSource = this.DataSource as CoreResultsDatasource;
  12:   
  13:          // override the query being executed
  14:          dataSource.Query = "Author:\"Martin Hatch\"";
  15:   
  16:          // remove the original sort order
  17:          dataSource.SortOrder.Clear();
  18:          dataSource.SortOrder.Add("Title", Microsoft.Office.Server.Search.Query.SortDirection.Ascending);
  19:      }
  20:  }

So lets talk through the code above.

First off we want to make sure we are only executing our custom code when we are actually trying to retrieve search results. This is a fail-safe block as some instances this will be false (such as when you are editing the web part or if you are viewing the “Design” view in SharePoint Designer). We also need to call the base method (as you typically would when overriding a method call!).

Then things get interesting. Line 11 has us create our “CoreResultsDataSource” object from the local “DataSource” property. This has two properties which we are modifying:

Query (line 14) – This allows us to change or completely override the query which is being executed. This will be the entire query including the Fixed Query, Appended Query and whatever the user typed into their search box (if you are using this on a Search Results page). In my example above I am simply overriding the result so that it just searches for items created by “Martin Hatch” (me!)

SortOrder (lines 17 and 18) – This allows us to override the sort order, allowing us to select ANY indexed Search Property you want (I expect excluding rich text fields of course!). In my example, I am sorting by Title in Ascending order.
This can then be easily extended to provide custom Web Part properties to allow the Sort functionality to be specified by the page editor.

Step 3 – Making it re-usable Part 1 – Dynamic Date ranges
 So now that we can override the query easily we can move on to adding some of the good stuff. I decided to go with a relatively simple Token Replacement function using a simple [TODAY] token to represent the current date:

  • [TODAY] (todays date)
  • [TODAY+7] (today plus 7 days)
  • [TODAY-7] (today minus 7 days)

So .. how do we code this in? Well .. I am quite lazy and don’t really get on with regular expressions (if you are reading this and you are a RegEx guru.. by all means download the source code, refactor it and send it back, cheers!).

So I started off by creating a bunch of class level constants which I would use to recognise the tokens that we are looking for above:

private const string TODAY_PLACEHOLDER = "[TODAY]";
private const string TODAY_ADD_STARTSTRING = "[TODAY+";
private const string TODAY_SUBTRACT_STARTSTRING = "[TODAY-";
private const string TOKEN_ENDSTRING = "]";

The following code can then be swapped out for our ConfigureDataSourceProperties method.

   1:  protected override void ConfigureDataSourceProperties()
   2:  {
   3:      // only perform actions when we are trying to show search results
   4:      // i.e. not when you're in Design Mode
   5:      if (this.ShowSearchResults)
   6:      {
   7:          // call the base web part method
   8:          base.ConfigureDataSourceProperties();
   9:   
  10:          // get the data source object
  11:          CoreResultsDatasource dataSource = this.DataSource as CoreResultsDatasource;
  12:   
  13:          // get the current Fixed Query value from the web part
  14:          string strQuery = this.FixedQuery;
  15:   
  16:          // swap out the exact "today" date
  17:          if (strQuery.IndexOf(TODAY_PLACEHOLDER) != -1)
  18:          {
  19:              strQuery = strQuery.Replace(TODAY_PLACEHOLDER, DateTime.UtcNow.ToShortDateString());
  20:          }
  21:   
  22:          // perform all of the "Add Days" calculations
  23:          while (strQuery.IndexOf(TODAY_ADD_STARTSTRING) != -1)
  24:          {
  25:              strQuery = CalculateQueryDates(strQuery, TODAY_ADD_STARTSTRING, true);
  26:          }
  27:   
  28:          // perform all of the "Remove Days" calculations
  29:          while (strQuery.IndexOf(TODAY_SUBTRACT_STARTSTRING) != -1)
  30:          {
  31:              strQuery = CalculateQueryDates(strQuery, TODAY_SUBTRACT_STARTSTRING, false);
  32:          }
  33:   
  34:          // swap out the Fixed Query for our Calculated Query
  35:          dataSource.Query = dataSource.Query.Replace(this.FixedQuery, strQuery);
  36:      }
  37:  }

This then calls the CalculateQueryDates support method which I put together:

   1:  private static string CalculateQueryDates(string strQuery, string startStringToLookFor, bool AddDays)
   2:  {
   3:      try
   4:      {
   5:          // get the index of the first time this string appears
   6:          int firstIndex = strQuery.IndexOf(startStringToLookFor);
   7:   
   8:          // get the text which appears BEFORE this bit
   9:          string startString = strQuery.Substring(0, firstIndex);
  10:   
  11:          // get the text which appears AFTER this bit
  12:          string trailingString = strQuery.Substring(firstIndex);
  13:          int endIndex = trailingString.IndexOf(TOKEN_ENDSTRING);
  14:          if (endIndex + 1 == trailingString.Length)
  15:          {
  16:              // there is nothing else after this
  17:              trailingString = "";
  18:          }
  19:          else
  20:          {
  21:              trailingString = trailingString.Substring(endIndex +1);
  22:          }
  23:   
  24:          // find the number of days
  25:          string strDays = strQuery.Substring(firstIndex + startStringToLookFor.Length);
  26:          strDays = strDays.Substring(0, strDays.IndexOf(TOKEN_ENDSTRING));
  27:          int days = int.Parse(strDays);
  28:   
  29:          // re-construct the query afterwards
  30:          if (AddDays)
  31:          {
  32:              strQuery = startString + DateTime.UtcNow.AddDays(days).ToShortDateString() + trailingString;
  33:          }
  34:          else
  35:          {
  36:              // subtract days
  37:              strQuery = startString + DateTime.UtcNow.AddDays(0 - days).ToShortDateString() + trailingString;
  38:          }
  39:   
  40:          return strQuery;
  41:      }
  42:      catch (FormatException ex)
  43:      {
  44:          throw new FormatException("The format of the [TODAY] string is invalid", ex);
  45:      }
  46:      catch (ArgumentNullException ex)
  47:      {
  48:          throw new FormatException("The format of the [TODAY] string is invalid. Could not convert the days value to an integer.", ex);
  49:      }
  50:  }

So you should be able to see we are using simple String.IndexOf() method calls to find out if our Tokens are present.

If they are then we simply calculate the DateTime value based on the static DateTime.UtcNow property and use String.Replace() methods to swap out these into our query text.

When we are using [TODAY+X] or [TODAY-X] we simply use DateTime.UtcNow.AddDays(X) or DateTime.UtcNow.AddDays(0-X) and use the same String.Replace() method.

The search syntax is exactly the same as it was previously, and the Keyword Syntax is very powerful.

Example: Using [TODAY] Token query syntax

Write:[TODAY] – this will return all items that were modified today
Write>[TODAY-7] – this will return all items that were modified in the past week
Write:[TODAY-14]..[TODAY-7] – this will return all items that were modified between 2 weeks ago and 1 week ago

So we already have a powerful and reusable search component .. but there is more!

Step 4 – Making it re-usable Part 2 – Dynamic User Profile Properties
The next one is to allow us to pull in User Profile Properties so that we can start doing searches based on the current user’s profile values.
For this we needed to create new replacable Tokens, for which I decided to use:

  • [UPP-{User Profile Property Internal Name}]
  • [UPP-PreferredName] (swaps out for the users name)
  • [UPP-SPS-Responsibility] (swaps out for their “Ask Me About” values)
  • etc ..

So .. we add another class level constant (same as we did for our DateTime tokens)

private const string USER_PROFILE_PROP_STARTSTRING = "[UPP-";

We can then use this in our code, in exactly the way we did before (using String.IndexOf(), String.SubString() and String.Replace() methods).

So we add the following additional code to our ConfigureDataSourceProperties method;

   1:  if (dataSource.Query.IndexOf(USER_PROFILE_PROP_STARTSTRING) != -1 &&
   2:      UserProfileManager.IsAvailable(SPServiceContext.Current))
   3:  {
   4:      string strQuery = dataSource.Query;
   5:   
   6:      while (strQuery.IndexOf(USER_PROFILE_PROP_STARTSTRING) != -1)
   7:      {
   8:          strQuery = ReplaceUserProfilePropertyTokens(strQuery);
   9:      }
  10:   
  11:      if (strQuery != dataSource.Query)
  12:      {
  13:          dataSource.Query = strQuery;
  14:      }
  15:  }

This uses the additional method call ReplaceUserProfilePropertyTokens which is shown below:

   1:  private static string ReplaceUserProfilePropertyTokens(string strQuery)
   2:  {
   3:      // retrieve the current user's Profile
   4:      UserProfileManager upm = new UserProfileManager(SPServiceContext.Current);
   5:      UserProfile profile = upm.GetUserProfile(false);
   6:   
   7:      if (profile == null)
   8:      {
   9:          throw new ApplicationException("The current user does not have a User Profile");
  10:      }
  11:   
  12:      // extract the user profile property name from the token
  13:      int startIndex = strQuery.IndexOf(USER_PROFILE_PROP_STARTSTRING);
  14:      string strPropertyName = strQuery.Substring(startIndex + USER_PROFILE_PROP_STARTSTRING.Length);
  15:      strPropertyName = strPropertyName.Substring(0, strPropertyName.IndexOf(TOKEN_ENDSTRING));
  16:   
  17:      string strToReplace = strQuery.Substring(startIndex);
  18:      strToReplace = strToReplace.Substring(0, strToReplace.IndexOf(TOKEN_ENDSTRING) + 1);
  19:   
  20:      try
  21:      {
  22:          // get the value
  23:          UserProfileValueCollection propertyValue = profile[strPropertyName];
  24:          string strValues = String.Empty;
  25:   
  26:          foreach (object propValue in propertyValue)
  27:          {
  28:              if (propValue.ToString().IndexOf(" ") == -1)
  29:              {
  30:                  strValues += propValue.ToString() + " OR ";
  31:              }
  32:              else
  33:              {
  34:                  strValues += "\"" + propValue.ToString() + "\" OR ";
  35:              }
  36:          }
  37:   
  38:          if (strValues.Length > 0)
  39:          {
  40:              strValues = strValues.Substring(0, strValues.Length - 4);
  41:          }
  42:   
  43:          // swap the value out in the query
  44:          strQuery = strQuery.Replace(strToReplace, strValues);
  45:   
  46:      }
  47:      catch (ArgumentException ex)
  48:      {
  49:          throw new FormatException("The User Profile Property specified in your UPP token does not exist", ex);
  50:      }
  51:      return strQuery;
  52:  }

So there are a few things to point out here which might trip you up:

  • We are using the UserProfileManager.IsAvailable() method to find out if we have a user profile service application provisioned and assigned to the current Web Application.
  • At the moment this code throws an error if the current user doesn’t have a User Profile. You may want to handle this differently for your environment?
  • Handling of multi-value fields. At the moment all we do is take the string values and concatenate them with “OR” in the middle. So if you had “Value1; Value2” as your property value the Token replacement would put “Value1 OR Value2” as the search query.

As long as the content editors are aware of the behaviour this allows us to create quite complex queries.

Example if we now used the Fixed Query:

([UPP-SP-Responsibility]) AND Write:[TODAY-14]..[TODAY]

Then for a user who’s “Ask Me About” properties were “SharePoint” and “IT Administration” then the resulting Search Query would be:

(SharePoint OR “IT Administration”) AND Write:13/05/2012..17/05/2012

If another user comes along whose “Ask Me About” property was just set to “Marketing” then the resulting Search Query would be:

(Marketing) AND Write:13/05/2012..17/05/2012

This is without changing any of the web part properties, and allows us to drive dynamic content from a single web part to our entire user base.

Hopefully you can see that this is incredibly powerful and flexible.

Step 5 – Making it re-usable Part 3 – Controllable Sort By
The final step is to allow our content editors to control the “Sort By” functionality. The default OOTB webpart only allows us to sort by “Relevance” or “Last Modified”, which is fine when you are looking at general search results, but when you are building custom components (such as news, links or event feeds) you typically want to control the order by date or title or something a little more usable for the specific component.

So this bolt-in allows you to control the Sort By. First off we need to add some Web Part Properties so that the user can modify their values:

   1:  [Personalizable(PersonalizationScope.Shared)]
   2:  [WebBrowsable(true)]
   3:  [WebDescription("Sort by this managed property")]
   4:  [WebDisplayName("Managed Property")]
   5:  [Category("Sort Override")]
   6:  public string OrderByProperty { get; set; }
   7:   
   8:  [Personalizable(PersonalizationScope.Shared)]
   9:  [WebBrowsable(true)]
  10:  [WebDescription("Sort direction")]
  11:  [Category("Sort Override")]
  12:  public Microsoft.Office.Server.Search.Query.SortDirection SortDirection { get; set; }

This will provide the Web Part property editing functionality:
image

Once we have done that, we can add the following code to our ConfigureDataSourceProperties method (yes .. this method really is where all of the grunt work goes on in this web part!)

   1:  // if OrderByProperty is not set, use default behavior
   2:  if (!string.IsNullOrEmpty(OrderByProperty))
   3:  {
   4:      // change the sortorder
   5:      dataSource.SortOrder.Clear();
   6:      dataSource.SortOrder.Add(OrderByProperty, SortDirection);
   7:  }

And that is all there is to it.

Step 6 – Enjoy!
So congratulations if you made it this far. I know this was a long blog post but thought it was worth walking through it properly.

If you have any questions, feedback or questions then please get in touch using the comments, and here are links to the downloads (which are also referenced at the top of this blog post)

Some notes about the “final” version:

  • The code structure is slightly different because the DateTime [TODAY] queries are cached using Web Part Properties for better performance
  • the [TODAY] token is case sensitive!
  • There is an extra “Debug Mode” checkbox in the Web Part Properties which when enabled spits out the entire query being executed at the bottom of the search results.
  • Code contains an “Editor Part” .. this just clears out the Cache value when the web part properties are modified

Usage Summary:

Tokens you can use are:

  • [TODAY]
  • [TODAY+X] (add X days)
  • [TODAY-X] (remove X days)
  • [UPP-{Internal Name of User Profile Property}]

Example Usage

Sample user has:
Name: Martin Hatch
Ask Me About: SharePoint; Solution Architecture; Code

ContentType:Event AND ([UPP-SPS-Responsibility]) AND Write:[TODAY-7]..[TODAY]
becomes
ContentType:Event AND (SharePoint OR “Solution Architecture” OR Code) AND Write:12/05/2012..17/05/2012
Returns all events which were updated within the past week, and contain the current user’s “Ask Me About” values.

Author:[UPP-PreferredName] IsDocument:1
becomes
Author:”Martin Hatch” IsDocument:1
Returns all documents written by the current user

Author:[UPP-PreferredName] Write>=[TODAY-14]
becomes
Author:”Martin Hatch” Write>=02/05/2012
Returns all content created by the current user and updated within the past 2 weeks

Scaling to 10,000 unique permissions – Part 2 – The Solution

This follows on from my previous post; Part 1 – The Problem

The main requirement was:

  • One SharePoint 2010 site
  • 10,000+ uniquely permissioned objects each with a different user account

In this post we will be discussing the solution which involves programmatically creating unique permissions in a way which will scale for (what should be) well over 10,000 uniquely permissioned items…

Introducing yet another little known SharePoint API call …

This is only possible because of one of the new SharePoint 2010 API calls;

SPRoleAssignmentCollection.AddToCurrentScopeOnly()

https://msdn.microsoft.com/en-us/library/microsoft.sharepoint.sproleassignmentcollection.addtocurrentscopeonly.aspx

This basically adds the specified SPRoleAssignment but does not create any of the Limited Access scopes on the parent objects.

This is pretty straight forward and works in exactly the same way to normal Role Assignments in SharePoint 2010, we simply use the AddToCurrentScopeOnly() method instead of using the Add() method, for example:

 

   1: // fetch the Principal object which we are granting access to

   2: SPUser user = web.EnsureUser("Domain\\UserAccount");

   3:  

   4: // create a Role Assignment binding

   5: SPRoleAssignment roleAssignment = new SPRoleAssignment(user);

   6:  

   7: // apply contribute permissions

   8: roleAssignment.RoleDefinitionBindings.Add(

   9:     web.RoleDefinitions["Contribute"]);

  10:  

  11: // grant permissions to the list item using the CURRENT SCOPE ONLY

  12: // this ensures that Limited Access scopes are NOT created

  13: // for parent objects (we're going to have to do that bit ourselves!)

  14: item.RoleAssignments.AddToCurrentScopeOnly(roleAssignment)

 

It is very important to understand that you still need to grant “Limited Access” (it wasn’t put in just for laughs, it does have a purpose). Granting “Limited Access” means that the object has access to core information on parent objects to enable construction of things like the breadcrumb, and retrieval of core files needed to render the interface.

This then means it is up to us (the developers) to go back and create each of those in a more efficient way. The problem is .. you can’t assign “Limited Access” programmatically…

What do you mean .. I can’t assign Limited Access??

Well, I don’t really know why they did this, but if you try and assign it programmatically (Limited Access is actually a “Permission Level” in SharePoint) you will get errors (admittedly you can’t do this through the user interface either!).

So, the workaround (again) is to create your own permission level which includes exactly the same permissions that “Limited Access” would have granted. This is:

  • View Application Pages
  • Browse User Information
  • Use Remote Interfaces
  • Use Client Integration Features
  • Open

You can call this anything you like (I called mine “SP Limited Access”) as long as you know what it means.

The code to do this is as follows:

 

   1: internal SPRoleDefinition GetLimitedAccessRole(SPWeb web)

   2:         {

   3:             string strRoleDefinition = "SP Limited Access";

   4:  

   5:             // only exists in webs with unique role definitions

   6:             if (web.HasUniqueRoleDefinitions)

   7:             {

   8:                 try

   9:                 {

  10:                     // try to retrieve the role definition

  11:                     return web.RoleDefinitions[strRoleDefinition];

  12:                 }

  13:                 catch (SPException)

  14:                 {

  15:                     // SPException means it does not exist

  16:  

  17:                     // create our custom limited access role

  18:                     SPRoleDefinition roleDef = new SPRoleDefinition();

  19:  

  20:                     // give it a name and description

  21:                     roleDef.Name = "SP Limited Access";

  22:                     roleDef.Description = "Identical to standard " + 

  23:                         "Limited Access rights. " + 

  24:                         "Used to provide access to parent objects of " + 

  25:                         "uniquely permissioned content";

  26:  

  27:                     // apply the base permissions required

  28:                     roleDef.BasePermissions = SPBasePermissions.ViewFormPages 

  29:                         | SPBasePermissions.Open 

  30:                         | SPBasePermissions.BrowseUserInfo 

  31:                         | SPBasePermissions.UseClientIntegration 

  32:                         | SPBasePermissions.UseRemoteAPIs;

  33:  

  34:                     // add it to the web

  35:                     web.RoleDefinitions.Add(roleDef);

  36:                 }

  37:  

  38:                 return web.RoleDefinitions[strRoleDefinition];

  39:             }

  40:             else

  41:             {

  42:                 // try the parent web

  43:                 return GetLimitedAccessRole(web.ParentWeb);

  44:             }

  45:         }

I’ve created my new Limited Access Permission Level .. now what?

One thing does need to be made clear, there is absolutely no point you just creating all of the Security Scopes that SharePoint would have created (you’ll end up with the same mess we were trying to avoid in the first place).

The solution is to create a group for all of the “Limited Access” users for that List or Web. It really is up to you whether you use Active Directory Security Groups or SharePoint Groups. I decided to use AD security groups; mainly because I didn’t want to clog up the Site Collection “groups” functionality, and didn’t want idiot Site Collection admins from removing the group members (or worse .. the groups themselves!) and breaking the site collection.

Note – I haven’t included the code to create and modify Active Directory Security Groups here, if nothing else because there are thousands of resources out there showing you how to modify AD groups programmatically, and Code Project has a particularly good reference: Howto: (Almost) Everything In Active Directory via C#

You will need to create a group for each parent object which has unique permissions although in my example it is only really the SPWeb (web site) that we are worried about as the libraries and folders are well within the security scope threshold.

So we have our 20 libraries and our root web site. So in our example we would have to create 21 different AD security groups:

  • One group to store all Limited Access users for the root web site
  • 20 groups to store all Limited Access for the libraries (one for each library)

Then, following this example you can then use the following code to grant “Limited Access” to one of the libraries (and just rinse and repeat for the other libraries and the root web site);

 

   1: // fetch the "SP Limited Access" role definition

   2: SPRoleDefinition limitedAccessRole = GetLimitedAccessRole(web);

   3:  

   4: // get SPPrincipal object for the AD Group we created

   5: SPUser adGroup = web.EnsureUser("My Custom AD Group Name");

   6:  

   7: // set the role assignments for this group

   8: SPRoleAssignment roleAssignment = new SPRoleAssignment(adGroup);

   9: roleAssignment.RoleDefinitionBindings.Add(limitedAccessRole);

  10:  

  11: // grant "Limited Access" to the AD Group for this list

  12: // we only have to do this once! After this we simply 

  13: // need to add members to this AD Group every time we 

  14: // add users to one of the parent objects!

  15: list.RoleAssignments.AddToCurrentScopeOnly(roleAssignment)

So having done this for all of the parent objects we now have our 21 custom Active Directory groups, each one of which has been granted “Limited Access” to one of the required “parent” objects for our folders.

From here on in it should be smooth sailing. You simply need to make sure that every time you programmatically add a new user to one of the folders you also make sure they get added to the relevant AD Groups (so that the “Limited Access” chain is not broken).

The following diagram really explains what we have done:

Folders_New

I have tested this model for over 16,000 unique AD accounts across hundreds of folders in hundreds of document libraries and I cannot notice any discernable drop off in performance (nothing that can’t be explained by simply having a really large number of libraries and folders anyway!) so initial tests show that this is working very well indeed 🙂

What I also ended up doing (to make this slightly more robust) is to build my own application page which users can use to Grant Permissions through the UI (so we don’t need to write custom code every time a new “Limited Access” scope is needed).

I then wrote an HttpModule to auto-redirect any requests to the out-of-the-box page (_layouts/AclInv.aspx) to the custom page so that if anyone tried to use the native user interface it would ALWAYS be executing my own custom code (which creates all of the AD Groups and SP Limited Access scopes programmatically, without the user having to worry about it!)

The great thing about this solution is that it doesn’t matter how many users or groups you are adding to your SharePoint site .. you only ever have 1 Limited Access security scope for each List / Web!

Thanks for sticking with me through these two posts .. if you made it this far then thanks for reading and I would love to hear your comments! 🙂

Scaling to 10,000 unique permissions – Part 1 – The Problem

This post was borne out of a client requirement which popped up on my radar. I’m currently working for a leading global Business Intelligence provider in London, and they were looking to implement a particular third party piece of software. This software relies on SharePoint for file storage and my client wanted to roll this out to their customers “extranet” style with each customer having uniquely secured content (files and folders).

Now .. first off their customers include in excess of 10,000 different companies (i.e. > 10,000 users) so early warning bells immediately started ringing in terms of scalability.

Secondly, to make this worse, the software required all content to be stored in a single SharePoint site .. so now my early warning system had gone into full meltdown and a state of high alert was reached.
So to boil this down …

  • One SharePoint 2010 site
  • 10,000+ uniquely permissioned objects each with a different user account

A Library with 10,000 uniquely permissioned folders?? Possible? My first instincts said no… so it was time to get my problem solving hat on and do some digging ..

Investigating the Limits of SharePoint 2010

I would like to think that any SharePoint {Consultant | Developer | Architect | <insert profession>} worth their salt would have read the Software and Capacity Planning guidelines (or at least be aware of it!) .. so that was my first pit-stop.

Note – I also stumbled across a great blog post by SharePoint infrastructure veteran Joel Oleson and his Best Practices for Enterprise User Scalability in SharePoint. This goes into detail about the specific size of an ACL (and the reason why this is limited, specifically in Windows) which although a good read wasn’t really relevant to my problem.

The Microsoft TechNet article SharePoint Server 2010 capacity management: Software boundaries and limits (https://technet.microsoft.com/en-us/library/cc262787.aspx) is a great resource and contains one absolutely key entry:

Security Scope – 1,000 per list (threshold)
The maximum number of unique security scopes set for a list should not exceed 1,000. 

A scope is the security boundary for a securable object and any of its children that do not have a separate security boundary defined.  

A scope contains an Access Control List (ACL), but unlike NTFS ACLs, a scope can include security principals that are specific to SharePoint Server. The members of an ACL for a scope can include Windows users, user accounts other than Windows users (such as forms-based accounts), Active Directory groups, or SharePoint groups.

So what is a Security Scope then? Ok I admit it does tend to get a bit bogged down in terminology.
To put it simply … each time you grant access to a new principal (user account or group) then you are creating a new Security Scope.

The other thing to consider pickup is that this is not just limited to lists! Any list that inherits permissions will pick up their permissions from the parent web (site) so you also need to adhere to this at the web level too!

This means that you should not have more than 1000 security scopes at EITHER the Site or List level.

Ignoring this limit can do real damage to your farm …

There is even a Microsoft Knowledgebase article explaining why; SharePoint performance degradation with a large number of unique security scopes in lists (https://support.microsoft.com/kb/2420771)

This is really explained in far more detail in two most excellent blog posts:

The first post describes the problem of trying to create more than 1000 security scopes, and what happens when you do this: https://wbblog.datapolis.com/2011/03/setting-item-permissions-with-workflow.html

The second post is by James Love (a.k.a. @jimmywim) and goes into real “deep dive” detail looking into the root cause of the problem (SQL Server and ACL GUIDs) and how this problem can actually bring down your ENTIRE FARM and not just the list / site you are working on!
https://e-junkie-chronicles.blogspot.com/2011/03/sharepoint-2010-performance-with-item_23.html

A quote from the second post is as follows:

“When you load up a huge list with lots of item level permissions, a single operation gets every single GUID associated with the ACL for that item and passes that back to the data access layer of SharePoint. When the database retrieves the actual list item data, it will pass in all of the ACL Guids back in as one long string, all concatenated together. The query to get the data creates a table variable re-assembles the the item level ACL Guid associated with each item. How the rest of the query deals with this is anyone’s guess at the moment – this table variable might just be passed back to the calling COM object (though I thought they couldn’t be used this way….) for the COM object to then sort out which item should be visible to which “scope” (or ACL).

So, what can we take away form this? Passing 640k of data about the place, for a SQL Query to do some substring math and converting to Guids will soon bring your database server to its knees. This is one request and it takes 2000ms to work. Imagine if you have 5 requests per second or more hitting this list!”

Both are excellent appendums to this post and well worth looking at for another angle and a bit more detail!

Why does this become my problem?

Now .. looking back to my original problem some of you may be thinking, OK no problem; you can just create yourself 20 different lists / libraries .. and have 500 unique permissions in each list??

Well .. so you might think .. and here I introduce the juggernaut that is Limited Access Scopes!

Anyone who has spent any time around SharePoint security will have noticed the odd “Limited Access” permission popping up in their site from time to time. “Limited Access” is automatically allocated to a parent Folder, List or Web whenever a child object has a unique permission allocated to it.

You can easily see these being created if you break permission inheritence to a list and just add a few accounts to that list. The parent Web will not have a “Limited Access” scope created for each user account you have added.

Now hopefully the bright will already have spotted the problem .. it doesn’t matter how many lists or libraries you create .. every single user or group that you add will end up in the parent Web site with “Limited Access” (and every single Parent Web heading upwards).

The following diagram explains why.

You simply cannot get away from this fact. If are adding 10,000 unique permissions with different user accounts then you will end up with 10,000 security scopes at the root web!

Note – It should be noted that the number of “Limited Access” scopes created is limited to the number of Security Principals you are adding.

If you are adding from a pool of 50 users then you will only ever be adding a maximum of 50 new “Limited Access” scopes (one for each user account).

For this reason it is a good idea to use Groups when adding permissions as this limits the number of “Limited Access” scopes which are created .. but this won’t solve your problem if you have over 1000 different security principals!

So that was the crux of my problem .. on investigation this does look to be a major major problem (and an “impossible fix”) but it would seem not! There IS a workaround (one which I have tested to over 15,000 unique user accounts and works very very well indeed)…

The solution, workaround, and code samples are all included in Part 2

How to enable Anonymous Access to a blog site on your Office 365 public website

This has been plaguing the forums for weeks now .. if I had a pound for everytime I’ve seen someone complaining about blogs on Office 365 .. I’d have .. erm .. about £15 ..

But seriously, this is something that a lot of folks have been complaining about .. but no more! 🙂

One thing that definately surprised me is that you can set anonymous permissions through Sandbox Solutions! This means that we can write our own custom code to enable full anonymous access for comments, categories and posts 🙂 So I’ve done just that.

A link to download a Sandbox Solution can be found below. Just upload the WSP to your public website site collection, activate it, and drop the new “Hatch Solutions” web part onto the home page of your blog 🙂

NOTE: For those who aren’t interested in how this works, and just want the web part, you can grab the WSP package here.

This installs a Web Part. Place this webpart on the home page of your Blog site, and hit the big button… it should do all of the work for you.

Important: You don’t need to keep the webpart on there. Once you’ve checked it is working you can remove the webpart and remove the WSP from your Solution Gallery!

Regional Settings –  There have been numerous reported issues regarding regional settings (as the code looks for lists called “Posts” and “Comments”). Currently this WSP only works when your SharePoint Site regional settings are set to English.

So what are we doing?
This is really quite simple. The SharePoint API exposes the list permissions for anonymous users through an SPList property called AnonymousPermMask64. This is an enumeration of SPBasePermissions values which effectively describe what access anonymous users have.

The reason this doesn’t work by default for anonymous users is because the “ViewFormPages” permissions is not included by default!

So our code is quite simple:

// get the “Comments” list
SPList list = SPContext.Current.Web.Lists[“Comments”];

// check if it has unique permissions
if(!list.HasUniqueRoleAssignments)
{
 list.BreakRoleInheritance(true);
}

// make sure people can edit their own items
list.WriteSecurity = 2;

// grant permissions to anonymous users
list.AnonymousPermMask64 =
  (SPBasePermissions.Open |
   SPBasePermissions.OpenItems |
   SPBasePermissions.ViewFormPages |
   SPBasePermissions.ViewListItems |
   SPBasePermissions.AddListItems);

list.Update();


So all we are doing there is granting some additional permissions (ViewFormPages, ViewListItems and AddListItems) for anonymous users. Then we just rinse-and-repeat for Posts and Categories (but remember to remove the “AddListItems” bit!! otherwise anonymous users would be able to create new blog posts!).

That’s it! I have a (short-lived) demo running on my current Office 365 site: www.hatchsolutions.co.uk/Blog/

Note – depending on how much spam and rubbish ends up on there, I will probably delete it sooner rather than later. I’ll try and remember to update this post after I have.
To make this easy for you I have built a Web Part which you can download and install (link at the top of this post) which does all of the work for you.

So that is all you should need .. happy blogging folks!! (all we need now is a decent blog template with things like CAPTCHA.. )

How to programmatically add the Related List Webpart to a list form

This is one thing that I have used a few times through the browser and always thought it was a feature missing from WSS 3.0 / MOSS 2007. But it was only when I tried to do this (relatively simple) thing as part of a list definition that I realised how oblique it was, and how little information there seems to be explaining how to do it… This blog post will try to address some of that.

So first off .. what is a Related List Webpart? This wasn’t really a surprise but if you look under the hood you will notice that when you use the “Related List” ribbon button then it actually does 2 things:

  • Adds a new List View Webpart (in this case an XsltListViewWebPart) which is bound to the related list
  • Creates a web part connection from the (already present) ListFormWebPart which provides the ID which the related list will filter on.

In order to set this up programmatically we basically have to replicate this. In my example I am running this from a Feature Receiver which will setup the web parts. I have 2 lists which I will be referring to:

  • Main List (the main list which I will be modifying)
  • Related List (a list which contains a lookup field containing values from List A)

STEP 1 – Add a new List View Web Part to the List Form

In this step we will be modifying the DISPLAY form (DispForm.aspx). This is probably the most common approach although you can follow very similar steps for the Edit Form if you so wish)

// these are the 2 lists we will work with
SPList mainList = web.Lists[“Main List”];
SPList relatedList = web.Lists[“Related List”];
// this is the Display Form web part page
SPFile dispForm = web.GetFile(mainList.DefaultDisplayFormUrl);
// get the web part manage which we will use to interact
SPLimitedWebPartManager wpm =
dispForm.GetLimitedWebPartManager System.Web.UI.WebControls.WebParts.PersonalizationScope.Shared);
// only execute this code if there is a single web part (the default for this list)
// we don’t want to add the web parts over and over again ..
if (wpm.WebParts.Count == 1)
{
   // create our List View web part
   XsltListViewWebPart wp = new XsltListViewWebPart();
   // Hook up the list and the view
   wp.ListId = relatedList.ID;
   wp.ListName = relatedList.ID.ToString();
   wp.ViewGuid = relatedList.DefaultView.ID.ToString();
   wp.XmlDefinition = relatedList.DefaultView.GetViewXml();
   // set basic properties along with the title
   wp.AllowConnect = true;
   wp.Title = “Related items from “ + relatedList.Title;
   wp.ChromeType = PartChromeType.TitleAndBorder;
   // add the web part to the Main zone in position 2
   wpm.AddWebPart(wp, “Main”, 2);
   // save the page
   webPartPage.Update();
}

So you can see from the example code above we are adding a simple List View webpart for our related list items. We are binding it to the “Related List” using the ID, and also choosing the View we want to display (here we have just used the Default View for simplicity).

We need to make sure we set “AllowConnect” (as we will need to use Connections in Step 2) and also set the ChromeType to “Title and Border” (as the default for Web Parts in List Forms is “None”).

Finally we add the web part and save the changes to the Web Part Page …

now for the slightly less well documented bit ..

STEP 2  – Create Web Part Connections between ListForm and ListView webparts
In this step we are going to take the Web Part which we added in STEP 1 and use standard Web Part Connections to bind them together … don’t be scared, it isn’t as hard as yoy might think 🙂

We are going to be doing several things here:

  1. We will get instances of both web parts which need to be connected
  2. We will create “Connection Point” objects for each web part
  3. Specify which fields we want to filter on
  4. We will create a “Transformer” object which is able to pass values between each web part
  5. We will use the Web Part Manager to create the connection, using all of that information above.

// get instances of our two web parts
System.Web.UI.WebControls.WebParts.WebPart consumer = wpm.WebParts[1];
System.Web.UI.WebControls.WebParts.
WebPart provider = wpm.WebParts[0];

// Create our two Connection Point objects
// These are specific to our two types of Web Part

ProviderConnectionPoint providerPoint = wpm.GetProviderConnectionPoints(provider)[“ListFormRowProvider_WPQ_”];

ConsumerConnectionPoint consumerPoint = wpm.GetConsumerConnectionPoints(consumer)[“DFWP Filter Consumer ID”];

// create our “Transformer”
// we also specify which Field names we want to “connect” together
// here I am connecting my Related List’s “MyLookupField” and filtering it
// using the “ID” of my List Form’s item.

SPRowToParametersTransformer optimus = new SPRowToParametersTransformer();
optimus.ProviderFieldNames = new string[] { “ID” };
optimus.ConsumerFieldNames = new string[] { “MyLookupField” };

// Connect the two web parts together, using our Connection Point objects
// along with our Transformer

wpm.SPConnectWebParts(provider, providerPoint, consumer, consumerPoint, optimus);

And that is it! You don’t need to do any more “Update” or “Save” methods .. your Related List web part should now be finished 🙂

So just to be clear, we are using the standard SPLimitedWebPartManager object and calling the SPConnectWebParts method. The only slightly odd call is the reference to the “SPRowToParametersTransformer” which is specific to the type of Connection we are making (the List Form representing a “Row”).

Where did this information come from?
This is the easy bit 🙂 I added a Related Lists Webpart using the browser, then opened up DispForm.aspx using SharePoint Designer .. and found this: 

<WebPartPages:SPWebPartConnection ConsumerConnectionPointID=”DFWP Filter Consumer ID” ConsumerID=”g_dc64a1e0_c2f4_4302_86df_e4d184203bbd” ID=”c225174896″ ProviderConnectionPointID=”ListFormRowProvider_WPQ_” ProviderID=”g_ffb9e36b_bc6d_489e_b7fc_e93048c32f5c”><WebPartPages:SPRowToParametersTransformer ConsumerFieldNames=”IMSDataSource” ProviderFieldNames=”ID”></WebPartPages:SPRowToParametersTransformer>

I have highlighted the references to the 3 things which made all of the dots join up for me: )
so if you are ever stuck, make sure you look in SharePoint Designer for :

  • Consumer Connection Point ID (“DFWP Filter Consumer ID”)
  • Provider Connection Point ID (“ListFormRowProvider_WPQ_”)
  • Transformer Type (SPRowToParametersTransformer)

So thats it 🙂 Hope this was useful, comments always welcome

Page Title element and PerformancePoint Report always being blank

This was one I came across recently while doing some custom branding work on a SharePoint 2010 site which was using Performance Point Services, and specifically the PerformancePoint Report Web Part.

I had a simple branded master page (which was a copy of the normal v4.master) but could not for the life of me get PerformancePoint reports to work.

The Report Web Part was blank (with zero height) and there was a JavaScript error on the page:

Unterminated string constant
Code: 0

Well .. I rolled my site back to the v4.master and it was working again. I copy-pasted the v4.master <head> contents into my custom master page and it started working again .. although when I double checked the contents they were IDENTICAL.

So far this was very very weird .. and the actual solution is even weirder.

It seems it was to do with my <title> element in the <head> tags being broken onto separate lines:

BAD
<title>
   <asp:ContentPlaceHolder id=”PlaceHolderPageTitle” runat=”server” />
</title>

GOOD
<title><asp:ContentPlaceHolder id=”PlaceHolderPageTitle” runat=”server” /></title>

Why?? I have no idea .. but making this simple change solved the problem.
Bizarre eh?

Now an MCPD in SharePoint 2010!

Well, I had a few days spare as my new contract hadn’t started so was pondering what to do, and decided to sit the two developer based SharePoint 2010 Exams.

There are two of them in the new development based certification track for SharePoint:

The first exam will grant you an Microsoft Certified Technology Specialist and the second one the new Microsoft Certified Professional Developer.

I am very pleased to report that I passed both exams (yay me!) and can now proudly sport my new certification logos:

Code Solution – Import AD Photos into SharePoint User Profiles

[Update: Code download files updated 18/03/2011 – see below]

This is in relation to a previous post I made last week;

Active Directory Images are not imported by the SP2010 User Profile Import

So, the source code is finally ready and uploaded for your enjoyment 🙂

Now – I must first off give credit to the sources of inspiration. A lot of the code in this solution is copied / borrowed / inspired by the following posts:

All I have done is brought their code and samples together and packaged it into a WSP that runs from SharePoint Timer jobs, so you have the convenience of a SharePoint 2010 farm solution 🙂

Also it should be understood that both AD and the User Profile database are quite critical parts of anyone’s SharePoint farm, so sorry, but first I need to make a ….

… Disclaimer – All code and solutions are provided “as is” and should be used at your own risk! It is highly recommended that you test these in an isolated environment, and I confer no responsibility for any loss or damage from using the code, advice or solutions provided on this blog, or any related content.

Ok, now that is out of the way we can get on with business 🙂

[Update – 18/03/2011 – I have updated both sets of files so that it now uses the “distinguishedName” attribute to identify users in AD .. as this is a more reliable method and was in response to a reported bug]

I have uploaded the files to my Sky Drive including:

When you roll out the WSP you will find that it includes the following functionality:

Farm Scoped Feature
The WSP package includes a farm scoped feature called:

Hatch Solutions – Import Photos from AD (Timer Job)

When activated this will automatically identify the default MySite host application and create a custom Timer Job (see below) attached to that web application.

My Site Timer Job
The Timer Job (installed by the Farm Feature) is designed to run on the My Site Host web application, and is pre-configured to run once-per hour. It is called:

Hatch Solutions – Import Photos from AD

This will do the following:

  • Automatically identify all AD accounts in the current User Profile Database
  • If the AD account has a “jpegPhoto” attribute, then this is extracted
  • The photo is converted to three thumbnail images, and uploaded to the My Site Host profile photo asset library
  • The photo for that user profile is updated to point at their newly uploaded photo

Hope you enjoy, the source code is there for all to see, and good luck!

Tip – CAML Query Retrieve tasks assigned to user (including both AD and SP Groups)

This is something I have seen so many people struggle with, but it really is very easy, with the help of the “<Membership>” element.

The Membership element allows you to basically check to see if the AssignedTo field is assigned to any group which the current user is a member of.

Of course, you still need to use it in conjunction with a standard FieldRef check against the user’s ID (which you can get using the <UserID/> element.

Below is the CAML query to return All Tasks Assigned to the Current User, including specific assignments, and where the task is assigned to a group that contains the current user (both AD Groups and SharePoint Groups).

I suppose theoretically this should also work with groups in custom Membership Providers .. but haven’t tried it.

<Where>
  <Or>
    <Eq>
      <FieldRef ID=’Assigned To’ />
      <Value Type=’Integer’><UserID/></Value>
    </Eq>
    <Membership Type=’CurrentUserGroups’>
        <FieldRef Name=’AssignedTo’ />
    </Membership>”;
  </Or>
</Where>

« Older Entries