Will my Surface Pro 4 explode on an airplane?

Posted on by Leave a comment

Ok .. this might be slightly melodramatic – but I was faced with this question when flying back with British Airways from Tenerife yesterday evening.

We were about 2 hours into the flight, and my 5 year old son was knackered and reverted back to his 3-year-old inner self and asked to watch Peppa Pig (this is a rite of passage for UK parents) – something I didn’t pre-load onto his tablet, but I did have on my Surface (for my 16 month old daughter).

But imagine my *surprise* when I pulled my Surface out of the overhead locker to find the screen bulging out so badly I could barely engage the hinge.

“what does f$%k mean Daddy?”

 

 

“erm .. nothing .. I just need to go speak to someone”

I had read numerous reports of exploding batteries – particularly at altitude – so I was worried .. VERY worried.

Luckily I DID know about this problem, and took it directly (and very discretely) to the flight crew.

I expected to have to spend a good 5 minutes explaining why this was a problem but to my (pleasant) surprise they seemed to be well drilled and knew exactly about this problem.

They immediately called up the senior cabin crew member who came armed with a large metal container.

He quickly powered off the device (something I was kicking myself for not doing!) .. checked it for any signs of heat (it was quite cold) .. and then placed it inside the metal canister. I must say I was extremely impressed with how quickly and calmly the BA cabin crew dealt with this.

They told me they were going to check the temperature every 10 minutes, and if it showed any signs of heating up they would fill said canister with cold water. I was of course totally fine with this – my data was all backed up to OneDrive .. plus after I’d thought about it I didn’t really want my laptop exploding, especially on an aircraft full of people.

Luckily nothing happened and the Surface was returned, along with a written incident report (for which I needed to confirm contact details).

 

So as well as an interesting “anything interesting happen on the flight back?” story for the pub, I hope this helps to serve as a reminder to anyone out there travelling with electronics.

If you EVER see your tablet / laptop / phone / device bulging out like that on an aircraft then please take it to the cabin crew immediately. This story ended happily, but this could very easily have gone VERY VERY wrong …

Me? I’m just glad my son wanted to watch Peppa Pig.

Posted in General

Windows Server 2019, SharePoint Server 2019 and a broken Blob Cache

Posted on by 4 comments

I am one of those SharePoint early adopters. My first MOSS 2007, SP2010 and SP2013 commercials projects all went live before the main product hit General Availability, so I’m very much used to those niggling issues you get before a product hits the shelves.

However, in this case I believe I have found a configuration bug relating to commercial GA products.

The builds where this error is known to occur (and I have replicated this on a brand new VM build just to double check) are:

  • Windows Server 2019 Standard (Version: 1809 | Build: 17763.168)
  • SharePoint Server 2019 (Version: 16.0.10337.12109 – RTM Build)

Update : I have also today replicated the same issue on a fully-patch server setup (updated on 22nd January 2019):

  • Windows Server 2019 Standard (Version: 1809 | Build: 17763.253)
  • SharePoint Server 2019 (Version: 16.0.10339.12102 – December 2018 CU)

Disclaimer : My findings below are purely what I have observed with my own installs. Your own mileage may vary, but if you DO see this happening, then I have a fix described which should get you rolling again!

For those who aren’t sure if their BlobCache settings are actually correct – you should definitely make sure your web.config values are correct by checking out the Microsoft Docs article: Configure cache settings for a web application in SharePoint Server

So what is the problem with BlobCache, exactly?

In a nutshell what I have observed is that the BlobCache will not initialise. The designated blobcache folder is never populated and features which rely on it do not work (i.e. SharePoint Publishing Cache and Image Renditions).

I should note that this is not a permission problem with the actual blobcache folder itself (the default location being C:\BlobCache\14). I triple checked the permissions and they were all correct. In fact if you delete that folder and perform an IISRESET you’ll even find it gets re-created .. just it won’t have any contents (and “blob caching” as a function will not work).

I could not find any related issues in the browser console, ULS logs or IIS logs. The only error I found was in the Windows Event Viewer (under “Web Content Management”, Event ID 5538).

An error occured in the blob cache.  The exception message was ‘Retrieving the COM class factory for remote component with CLSID {2B72133B-3F5B-4602-8952-803546CE3344} from machine <machinename> failed due to the following error: 80070005 <machinename>.’.

Anyone with a history of COM errors will probably get an instant shiver and feelings of dread about going through DCOM configuration! And you would be exactly right .. this is a DCOM error. The error code (80070005) also tells us that this is an “Access Denied” error .. so we know that this relates to permissions.

In order to test this premise I did something very simple:

  • Add the application pool accounts to Local Administrators

I performed the usual IISRESET /noforce .. and voila! BlobCache started working .. hurrah!

But before you cheer, this really is NOT an ideal scenario to be in (app pool accounts should NOT be in the local admin group). So I removed those accounts from the administrators group .. and kept on digging ..

DCOM and Registry Permissions .. what decade is it again?

So yes .. this feels like the “good old days” when you wasted days tracking down erroneous GUIDs and DCOM config weirdisms .. and it seems in the year of 2019 this is going to pester us all once again.

However, as is always the case with COM components, things are not as straight forward as they might at first seem.

Having done some searching around for that CLSID (2B72133B-3F5B-4602-8952-803546CE3344) it appears to be a reference to a remote configuration component for IIS. This is NOT a DCOM component. However, much google-fu later (after reading a whole bunch of posts about automating Azure deployments .. yeh it seems the error above is fairly common when dealing with IIS config) I found that this is actually a COM class which  tries to call a DCOM components. Specifically the Application Host Admin API for IIS 7.0 (also known as “ahadmin”).

The error code we are seeing (80070005 = access denied) means the application pool account does not have permissions to launch or execute this DCOM component.

Now .. this is the point where, if you haven’t had to deal with DCOM permissions before, you might get a little stuck. You see, if you open up Component Services, find the “ahadmin” DCOM config entry .. you will notice that the “Security” options are all greyed out. You can’t modify them.

So now .. we need to ninja up our Registry skills.

We can see from the Component Services window the “Application ID” of “ahadmin” is {9fa5c497-f46d-447f-8011-05d03d7d7ddc}. So this allows us to build the Registry Key where this particular application is defined:

So we can launch RegEdit and look for the following path:

HKLM\SOFTWARE\Classes\AppID\{9fa5c497-f46d-447f-8011-05d03d7d7ddc}

(you will know it is the right one because the default key will read “ahadmin”). Right click on the node and select “Permissions”, and go to “Advanced” settings (you might need to wait a while for it to resolve all of the security principals before “Advanced” becomes available).

What you will need to do is two things:

  • Change the “Owner” (by default this will be “TrustedInstaller” – I changed this to the local “Administrators”)
  • Update permissions for “Administrators” so that they have “Full Control” (in addition to “Read”).

Having done this, reload “Component Services” and you should find that you can now edit the “ahadmin” DCOM permissions.

Change the following permissions for “Launch and Activation” as well as “Access Permissions”

  • Grant the local WSS_WPG group “Local” and “Remote” permissions (i.e. all of them)

Note – The WSS_WPG (Windows SharePoint Services – Worker Process Group) is a special group which SharePoint maintains to include all of the designated “application pool accounts” for the SharePoint Web Applications. So it is much better to use this group rather than permission each account individually. 

Now you just need a trusty IISRESET and we will have taken care of that nasty COM permissions error …

What do you mean .. more?

Yeh .. sorry .. you will probably find having done all that your BlobCache STILL doesn’t work.

This time however, the error in Windows Event Viewer will have changed (same “Web Content Management” category, and Event ID 5538)

An error occured in the blob cache.  The exception message was ‘Filename: redirection.config’

 

Error: Cannot read configuration file due to insufficient permissions

This one is a little bit easier to resolve, and should be a fairly quick solve.

It is referring to the “redirection.config” in the IIS config folder.

C:\Windows\System32\inetsrv\Config

Change the permissions on that folder and grant the same “WSS_WPG” local group “read” access to that folder (it doesn’t need more than “read” as it won’t be making changes to the IIS config files).

One more IISRESET and you should be done!

Voila … BlobCache joy once more!

BlobCache should now be correctly populating, and your web applications in SharePoint should be creating Blob Cache entries specific to their web.config entries.

The Windows Event Viewer should now show that it is creating the Blob Cache folders correctly (this time a slightly different Event ID : 7358)

Creating new cache folder ‘C:\BlobCache\14\1381895183\iXPJhtlC+ECvfzPRK3EHMA\’.

And you can see this folder created happily in the file system

And Image Rendition editing should also be working…

(and serving the correct images)

Summary

So for a quick summary .. we had to do some DCOM Config changes, Registry Key permissions and IIS config permissions.

Ironically this error was nothing to do with file system permissions to the Blob Cache folder itself (which is typically the only thing which has needed troubleshooting).

So .. quick crib-sheet:

  • Take ownership of the “ahadmin” Registry Key, and grant admins Full Control
  • Grant WSS_WPG {Launch | Activation | Access} permissions for the “ahadmin” DCOM component, accessed from “Component Services”
  • Grant WSS_WPG “read” permissions to the IIS “Config” folder in the file system

Hopefully this article was helpful! Chime in the comments if you have any questions (or if you’ve seen this issue before yourself!)

Update – 23/01/19

One of my colleagues at Content and Code reached out to Microsoft to report this issue and find out if they were aware of it.

Suffice to say Microsoft are indeed aware of this issue and are currently working on a fix.

Posted in Infrastructure

Allow Export of List Views (XsltListViewWebPart) in SharePoint Online / SP2016

Posted on by Leave a comment

Disclaimer – This does require a single manual step where you need to add a GUID to the exported file.

Wow .. its been a while since I blogged last but finally have something SharePoint(y) to post about 🙂

This time I was asked by a client if it was possible add a “List View” (XsltListViewWebPart) to a different site than the one where the list exists.

Now, I know that this is possible because I’ve done it on older projects, simply by exporting the List View, adding the “Web ID” and re-importing it to another page. This works as long as the page you are importing it into are in the same site collection.

The problem of course is that the List View web parts in SharePoint Online (and SP2016) currently have the “Allow Export” property set to false .. AND you can’t change this through the UI.

Luckily, this property is access via a DOM attribute which is extremely easy to modify. The following JavaScript identifies all “Web Parts” (in edit mode) which have an “allow export” attribute set, and then sets it to “true”.

var webParts = document.querySelectorAll(“div[webpartid][allowexport]”);

 

for(i=0; i<webParts.length; i++) {

 

webParts[i].setAttribute(“allowexport”, “true”);

 

}

You can of course add this to the page any way you see fit (either a Script Editor Web Part, Master Page, Custom Action, SPFx Application Customiser .. depending on the type of set and purpose for this script).

So .. this enables the “Export” menu on the Web Part Properties. Once you have exported the Web Part definition if you open it you will find that the WebId property will be blank.

<property name=”WebId” type=”System.Guid, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″>00000000-0000-0000-0000-000000000000</property>

This will need updating with the “Web Id” property of the web where the list resides. You can easily retrieve this using the REST API:

https://tenant.sharepoint.com/sites/SiteN/subsite/_api/web?$select=id

Just copy-paste that into the Web Part definition (NotePad will do the trick) and you can now upload that Web Part definition into ANY page anywhere in the Site Collection.

Posted in Uncategorized

Switching from Windows Mobile to Android … it was the best of times, it was the worst of times

Posted on by 3 comments

So as a massive surprise to anyone living in a nuclear bunker the past few years, the announcements are out that Windows Mobile is (kind-of) officially dead.

The writing was clearly on the wall with near-to-zero global sales and declining market ownership (if you could call it that) .. Windows Mobile has been “doing a Blackberry” recently.

I’ve been a (very) long die-hard fan of Windows Mobile, as an early adopter getting one of the first Windows Phone 7 devices, I even got WP7 running on my old HTC HD2 (a behemoth of a device at the time which seemed capable of running pretty much anything).

I’ve had numerous “flagship” phones from the Nokia Lumia 1020 (with its amazeballs-never-seen-better 41MP PureView camera) to the Nokia Lumia 930 (with the eyeball hurting bright orange back and aluminium sides) and my last Microsoft Lumia 950XL which I still have as an emergency-back-up-phone to this day (in fact, my wife is currently using it after losing her Lumia 620 and is steadfastly refusing to switch to another platform until the last possible minute!)

But .. I took the plunge and 4 months ago I made the jump over to Android. I knew things were heading south and had heard various rumours coming from internal Microsoft channels (working at a Microsoft Gold Partner has some perks) not to mention finding out that various heads of division at Microsoft were using iOS and Android devices. So I went out and bought myself a Samsung Galaxy S8+.

So this post aims to cover off some of the things I love about it, some of the things I hate .. and some of the things I really just miss from Windows Mobile (we’ll miss you, dear friend!)

Note – My Android phone is a Samsung Galaxy S8+

The things I love about my Galaxy S8+ 🙂

The first thing I think that leapt out was the design. Windows phones have always been fairly “out there” in terms of bright colours and pushing the OS design interface, but the S8+ certainly looks a bit different (at least when you are holding it in your hand). The screen is gorgeous (probably the best I’ve ever seen on any device short of a £5000+ OLED TV) and the curved glass body (with both a curved screen, and wrap-around glass shell front and back) make it look like a polished stone.

Of course .. that screen has it’s own drawbacks so hold fire for the “things I hate” bits too!

The Camera was another thing which impressed me. Of course being an avid Windows Phone user I have always had the benefit of excellent “PureView” cameras. The Lumia 1020 was standout and I still don’t think I’ve seen anywhere NEAR the quality in a phone camera since (see Some real photos from 6 Months of using a 41MP Nokia Lumia 1020).

The interface is neat and the photos always seem to be sharp, quick to focus, quick to take and the image quality is excellent (even in low light levels). It is quite rare that I get a photo that I feel is sub-part in quality, so that is something which really stood out for me.

The other thing that of course jumped out was the Apps. And I’m not talking about “number of apps” (in fact that’s in my “hate” section later). I’m talking about apps that feel like they’ve had the level of investment they deserve (regular updates and feature improvements).

I was pleasantly surprised to find that almost all of the core “Windows” apps I relied upon (Cortana, OneDrive, Outlook, OneNote, Microsoft Office (Word / Excel / PowerPoint), Skype, Yammer) were not only available on the Google Play store but were in some cases even more feature rich than they were on Windows.

Of particular delight was the “Nokia Here Now” app (basically the “Nokia Maps / Nokia Drive / Nokia Transit / Microsoft Maps” app from the Windows Mobile store) which still included the fully-offline downloadable maps and turn-by-turn satnav modes completely free of charge!

I also found that LastPass is much improved, with the ability to auto-complete passwords in other apps / throughout the interface rather than just in the web browser!

One example of the versatility of a feature rich “App” market was the Photo Gallery, you could go and find an alternative to use, download and install (of which there are many) and the same can be said about pretty much everything on the phone. You don’t like the default contacts App? go get another. Don’t like the Dialler? Messaging Interface? even the Keyboard? Same applies .. dozens of alternatives were waiting at every turn.

The security / login features were another plus. I have the options of fingerprint, iris scanning, “pattern” unlock, pin-codes and the like. And certainly on the S8+ the “iris scanner” seems to be both extremely quick and accurate (working in complete darkness, through my glasses and in all but the most harsh direct sunlight).

The integration for payment-options (which gets baked into the NFC payment system) was fairly intuitive and quick to use. Both convenient and simple to setup I can easily switch from stored credit cards to my mobile provider and back .. all very seamless (and to be honest, what you should expect from a platform this mature!)

Overall a pretty positive experience…. but its not all wine and roses!

The things I hate about my Galaxy S8+! 🙁 

Ok there are a few things off the bat which really get on my tits!

First off the “curved infinity display” screen which Samsung gave such a song and dance about. It looks gorgeous and most of the time is amazing, but on some apps (where they place the interface right along the edge) it just means you are constantly trying to hit “that sweet spot” by the edge of the screen which is just “on the curve”

On that subject, the glass body on the phone can be extremely slippery! It is also a fingerprint and scratch magnet. Do NOT put your phone in your pocket with keys / coins or any other sharp-edged metal. I’ve already put a fairly decent gouge in the glass on the back of the phone .. something which I’m sure is only going to get worse over time! Of course I could go and get myself a nice leather “wallet” or one of those shock-proof rubber condoms people put on their phones, but I’ve always been a bit of a fatalist that I like to see the phone as it was meant to be, and if that means I accidentally drop it a few times then so be it!

The keyboard was another sore spot. I have possibly been spoilt by being on Windows Phone and its best-in-class Swipe keyboard (“Microsoft Word Flow”?? or did they rename it?). I frequently find that my new phone either; completely misses the letters I was trying to type, and/or, completely mis-guesses the word I was trying to spell.  The result is an awful lot of predictive-text mis-hits and I find myself correcting it a LOT more often than I did on Windows. Some of this may be down to having a new phone and muscle-memory issues from having a slightly different layout, but I’m 4 months in now using the phone heavily every day so I’m starting to lose hope on that.

The number of third party apps is another sore point. I know .. “choice is a good thing” but in my opinion “too much choice is just annoying”. Its like going into a restaurant and being told they sell 300 different types of Burger. Which is fine, until you also get a separate 200 page menu of salads, and a 20 page book for soft drinks. After a while .. this just gets annoying.

So when I started looking for a replacement Calendar App, I found I was utterly _swamped_ with choice. Many of these apps seem to come from small software houses, which I suppose is again both great for choice (and exposure for small dev houses) but it means that the average app I come across severely lacks that professional “polish” that you might expect from a solid first-party application, and are either missing key features or have a quirky interface that just puts me off.

Another thing that I really can’t stand is the notifications and status bar. Apart from the icons at the top being seemingly font-size 4 (if I’m not wearing my glasses .. forget it!). You still have the “swipe down from the top” to get the expanded “quick actions” panel but there is critical information missing.

On Windows Mobile it would tell me not just if WiFi was turned on, but which SSID I was connected to. Equally for Bluetooth, am I connected to my headphones, my laptop or my car stereo? For Mobile data, do I have 4G, 3G or just GSMA connection? On Windows Mobile these were all “at a glance” features, but now it seems I have to click in-and-out of various menus to try and find this information out.

Probably the biggest bug-bear though is the reliance on having a Google Account for everything, or even having multiple accounts for the same thing! The Photos Gallery app wants you to “backup to Google”. Apparently this is free and unlimited, but if you don’t you will get reminded about it every week or so. The payment options require a Google Account for you to save them. Phone backup needs to use a Google Account (or in the case of the Galaxy S8+ a Samsung Account!). I have ended up using my (previously barely-ever-used) Google account to do some of these things, but it is annoying and slightly fragmented having to now juggle 2-3 different accounts for different parts of my phone, when previously I had a “one account does everything” method for Windows Phone.

Another good example is the Calendar / Contacts integration. So I use “Outlook” for my emails, but that won’t sync the Calendar items. I didn’t like the Google “Calendar” app, so I had to download a third party Calendar. But all that does is sync from the built-in “Calendar” functions, so I had to add all my accounts to the Google Calendar first so that my third-party calendar app could pick them up.

So I basically have my Calendar accounts added in 3 places: Outlook (notifications disabled), Google Calendar (notifications disabled) and my third-party Calendar. This all feels like way too much effort just to get reminders about upcoming appointments!

There are then the really WEIRD oddities about certainly functions. For example, you can’t change the colour of Calendar Icons without a third-party app. So I’ve had to go and find, install and configure a third-party app just so I can change the colour of calendar items (I have like 4 different accounts which I sync to my calendar). This just feels like I have to do a fair number of “hacks” to provide something I would have expected to be there from day 1 (and this is a mature platform by now too!)

Bixby – Well .. need I say more? Pointless and annoying .. but not stock Android so I’ll try and steer away from Samsung-specific arguments if I can.

But certainly the things I hate the most are simple usability features that were standard on Windows Mobile, but are missing from Android…

The things I miss from Windows Phone … 

The first thing I think I’m really missing is having OneDrive and Outlook as first-class-citizens. Yes … you have the “OneDrive” app which will auto-backup your photos and videos. But it won’t bring your OneDrive photos into the stock “Photo Gallery” app on the phone (I had to copy the last few months over manually).

OneDrive does have it’s own “photos” gallery feature but it is both buried away in the OneDrive app interface and not as feature rich. Plus you can’t “deep link” a shortcut for that Photos interface to the home page anyway.. even if you wanted to.

The Outlook app seems to be fairly good on Android, but it is lacking in a number of features. The whole sync-mechanism with the rest of the phone seems “tacked on” and there are numerous “settings” interfaces which are nested in each other which make it a little complicated to configure things like sync schedules and notifications. The “Calendar” view itself limits you to “3-Day”, “Day” or “Agenda” and completely drops the “Week / Agenda-Week” views which were by far my most popular on Windows Mobile. Equally, there is no ability on Android to link a “mailbox”, “folder” or “calendar” to the home page (you can’t even have a shortcut to just take you straight to the Outlook Calendar, that is buried away inside the app itself) which brings me onto …

Deep Linking Shortcuts.

This was (for me) the single KILLER feature of Windows Mobile. Forget “live tiles” (quite frankly I could take or leave them) but the ability to pin a shortcut on the home page which deep-links into the bowels of another app was awesome.

  • A specific Mailbox from Outlook
  • A specific FOLDER in a mailbox in Outlook
  • A Notebook / Page in OneNote
  • Internet Shortcuts
  • Phone Settings (WiFi / Bluetooth / Storage / etc)

You name it, you could pin it! On Android you can basically pin “the app” .. and that’s it. Yes you have “widgets” but they are really not the same thing (and other than “the weather” and “the time” I’ve yet to find one even vaguely useful).

The thing about this which is most painful is being able to see “at a glance” which of my 4 Mailboxes in Outlook have unread mail. Its great knowing I have 18 unread emails, but are they from my work or personal accounts? Are they in “Inbox” or are they one of the automated emails from TFS or Yammer? This is a major productivity kick-in-the-teeth and something I have to admit I’m struggling with a little bit.

Note – installing a different Mail App for each mailbox is NOT something I am going to even consider!

Facebook / LinkedIn Integration is another thing I miss… and by this I mean:

  • Showing my Facebook / LinkedIn friends in Contacts
  • Showing Facebook / LinkedIn photos for my existing Contacts (who are friends)

There are (apparently, technically) third-party apps on Android which can do this. I have spent several hours with the top-three rated of these, and none of them seem to work very well at all, enough so that I have effectively given up.

The Status Panel is another one which I had a little rant over up above, and to be honest this can go hand in hand with a Crisp and Clean Interface and general Notification Panel full stop. The overall UI and experience with Windows Phone has been hands-down superior, and most people I spoke to who had Windows Phone and have moved to iOS or Android have said the same.

I also miss the dedicated camera button which I’m sure exists on some Android phones, and I know doesn’t exist on some Windows phones too .. but I’ve had it on my last three Windows phones and it is something I’m really missing from my Galaxy S8+!

My final point on this is the Microsoft Account glue .. one account to rule them all. It did my favourite / internet history / browser passwords / wifi codes / phone backups / app settings. I had a single account, and it “just worked”. I haven’t tried a full backup/restore on my Galaxy S8+ yet (and TBH I’m a little scared to do so) but I can’t believe it will be anywhere near as simple and easy, and I expect to run into several hiccups along the way, especially if I change to another phone supplier!

The things that I thought would only be in Windows… but are in Android too!

My parting comments on this are some of the features that, back when Windows Mobile launched (be it WP7, WP8.1 or WP10) were standout features that I thought were unique to the platform, but I was surprised were ready and waiting when I switched.

OneDrive Camera backup was a biggy! Not only that but the same options for Photos / Videos, Resolution and “backup over WiFi vs Mobile Data”. If only it synced in both directions I’d be really happy

Cortana also integrates with the Windows 10 PC/Laptop with the same notification alerts and ability to reply to SMS messages without taking your phone out of your pocket.

And the Nokia HereNow / Maps / Transit / Whatever app, with downloadable country-sized maps, an “offline zero-data” mode and turn-by-turn satnav it was one of those features which surprised all my friends back in the day, and I’m really pleased to find it made it’s way over to Android and kept its ultra-low price tag (i.e. completely free!)

 

 

So, I am now embedded on Android. The switch was slightly less painful that I thought it would be, but I found some features missing which I didn’t quite expect. I am generally happy with the phone / ecosystem .. but I am (and probably always will be) missing Windows Mobile.. we’ll miss ya fella!

 

For those interested, here are the apps which I felt the need to download and install before I settled into my rhythm. The core apps I feel I would need for any Android phone, and the keys reasons why I use them:

  • OneDrive – Camera Backup / Cloud access
  • Cortana – PC alerts and SMS replies
  • Office / OneNote – Obviously
  • LastPass – Password Management and integrated auto-complete
  • Facebook / Messenger / Yammer / Twitter / Skype  – for obvious reasons
  • DailyPic – Auto-set the “Bing background of the day” to the Phone home and lock screen
  • DigiCal – Calendar with “Agenda Week” view which doesn’t look like it was built by college grads – the killer view for me!
  • Calendar Colors – Change the colours for synced calendars – no really!!
  • Nokia Here WeGo – Offline maps, satnav
  • Stock Google Apps {Dialler / Contacts / Messages} – Because I can’t stand the Samsung ones! 
Posted in Uncategorized

SharePoint Search {User.Property} Query Variables and Scalability

Posted on by 2 comments

This was something I stumbled into when working on a large global Intranet (>100k user platform) being built on SharePoint 2013. This is a WCM publishing site using “Search Driven” content leveraging Managed Metadata tagging combined with {User.Property} tokens to deliver “personalised” content. Now .. if there were 2 “buzz words” to market SharePoint 2013 they would be “search driven content” and “personalised results”, so I was surprised at what I found.

The Problem

So we basically found that page load times were >20 seconds and our SharePoint Web Servers were maxed out at 100% CPU usage. The load test showed that performance was very good with low load, but once we started ramping the load up CPU usage went up extremely quickly and rapidly ended up being almost un-usable.

It is worth bearing in mind that this is a completely “cloud friendly” solution, so zero server-side components, using almost exclusively “out of the box” web parts (mostly “Search Result Web Parts”, they would have been “Content by Search” but this was a Standard SKU install). We also use Output caching and blob caching, as well as minified and cached assets to slim down the site as much as possible,

Also worth noting that we have 10 (ten) WFE servers, each with 4 CPU cores and 32GB RAM (not including a whole battery of search query servers, index servers, and other general “back-end” servers). So we weren’t exactly light on oomph in the hardware department.

We eventually found it was the search result web parts (we have several on the home page) which were flattening the web servers. This could be easily proved by removing those web parts from the page and re-running our Load Tests (at which point CPU load dropped to ~60% and page load times dropped to 0.2 sec per page even above our “maximum capacity” tests).

What was particularly weird is that the web servers were the ones maxing out their CPU. The Search Query Component servers (dedicated hardware) were not too heavily stressed at all!

Query Variables anyone?

So the next thing we considered is that we make quite liberal use of “Query Variables” and in particular the {User.Property} ones. This allows you to use a “variable” in your Search Query which is swapped out “on the fly” for the values in that user’s SharePoint User Profile.

In our example we had “Location” and “Function” in both content and the User Profile Database, all mapped to the same MMS term sets. The crux of if is that it allows you to “tag” a news article with a specific location (region, country, city, building) and a specific function (e.g. a business unit, department or team) and when users hit the home page they only see content “targeted” at them.

To me this is what defines a “personalised” intranet .. and is the holy grail of most comms teams

However, when we took these personalisation values out (i.e. replacing {User.Location} with some actual Term ID GUID values) performance got remarkedly better! We also saw a significant uplift in the CPU usage on our Query Servers (so they were approaching 100% too).

So it would appear that SOMETHING in the use of Query Variables was causing a lot of additional CPU load on the Web Servers!

It does what??

So, now we get technical. I used JetBrains “DotPeek” tool to disassemble some of the SharePoint Server DLLs to find out what on earth happens when a Query Variable is passed in.

I was surprised at what I found!

I ended up delving down into the Microsoft.Office.Server.Search.Query.SearchExecutor class as this was where most of the “search” based activity went on, in particular in the PreExecuteQuery() method. This in turn referred to the Microsoft.SharePoint.Publishing.SearchTokenExpansion class and its GetTokenValue() method.

It then hits a fairly large switch statement with any {User.Property} tokens being passed over to a static GetUserProperty() method, which in turn calls GetUserPropertyInner(). This is where the fun begins!

The first thing it does is call UserProfileManager.GetUserProfile() to load up the current users SharePoint profile. There doesn’t appear to be any caching here (so this is PER TOKEN instance. If you have 5 {user.property} declarations in a single query, this happens 5 times!).

The next thing that happens is that it uses profile.GetProfileValueCollection() to load the property values from the UPA database, and (if it has the IsTaxonomic flag set) calls GetTaxonomyTerms() to retrieve the term values. These are full-blown “Term” objects which get created from calls to either TaxonomySession.GetTerms() or TermStore.GetTerms(). Either way, this results in a service/database roundtrip to the Managed Metadata Service.

Finally it ends up at GetTermProperty() which is just a simple bit of logic to build out the Keyword Query Syntax for Taxonomy fields (the “#0” thing) for each Term in your value collection.

So the call stack goes something like this:

SearchExecutor::PreExecuteQuery()
=> SearchTokenExpansion::GetTokenValue()
=> GetUserProperty()
=> GetUserPropertyInner()
=> UserProfileManager::GetUserProfile()
=> UserProfile::Properties.GetPropertyByName().CoreProperty.IsTaxonomic
If it is (which ours always are) then …
=> UserProfile::GetProfileValueCollection()::GetTaxonomyTerms()
=> TermStore::GetTerms()
Then for each term in the collection
=> SearchTokenExpansion::GetTermProperty()
This just builds out the “#0” + term.Id.ToString() query value

So what does this really mean?

Well lets put a simple example here.

Lets say you want to include a simple “personalised” search query to bring back targeted News content.

{|NewsFunction:{User.Function}} AND {|NewsLocation:{User.Location}}

This looks for two Search Managed Properties (NewsFunction and NewsLocation) and queries those two fields using the User Profile properties “Function” and “Location” respectively. Note – This supports multiple values (and will concatenate the query with “NewsFunction: OR NewsFunction:” as required)

On the Web Server this results in:

  • 2x “GetUserProfile” calls to retrieve the user’s profile
  • 2x “GetPropertyByName” calls to retrieve the attributes of the UPA property
  • 2x “GetTerms” queries to retrieve the term values bound to that profile

And this is happening PER PAGE REFRESH, PER USER.

So … now it suddenly became clear.

With 100k users hitting the home page it was bottlenecking the Web Servers because every home page hit resulted in double the amount of server-side lookups to the User Profile Service and Managed Metadata Service (on top of all of the other standard processing).

So how to get round this?

The solution we are gunning for is to throw away the Search Web Parts and build our own using REST calls to the Search API and KnockoutJS for the data binding.

This allows us to use client-side caching of the query (including any “expanded” query variables, and caching of their profile data) and we can even cache the entire search query result if needed so “repeat visits” to the page don’t result in additional server load.

Finally…
This was a fairly high profile investigation, including Microsoft coming in for a bit of a chat about some of the problems we’re facing. After some investigation they did confirm another option (which didn’t work for us, but useful to know) which is this:

  • Query Variables in the Search Web Part are processed by the Web Server before being passed to the Query Component
  • The same query variables in a Result Source or Query Rule will be processed on the Query Server directly!

So if you have a requirement which you can compartmentalise into a Query Rule or Result Source, you might want to look at that approach instead to reduce the WFE processing load.

Cheers! And good luck!

Posted in Development, Infrastructure

JSLink and Display Templates Part 7 – Code Samples

Posted on by 2 comments

When I was speaking at the SharePoint Evolutions conference earlier this year I ran into Jeremy Thake (@JThake) and Vesa Juvonen (@vesajuvonen) and they saw the JSLink samples I was running through for the session I had.

Well .. one thing led to another as they say .. and by the time the conference finished I had refactored all of my code and uploaded it to the OfficeDev PnP (Patterns and Practices) GitHub repository.

So you can find all of my sample code in the “Branding.JSLink” section (OfficeDev PnP > Samples > Branding.JSLink).

This includes full documentation, full source code, a compiled deployable WSP package, as well as loads and loads of awesome stuff from the rest of the OfficeDev PnP contributors.

The sample code includes:

  • Re-Render Lookups as bulleted lists and checkboxes
  • Cascading Drop-Downs for Lookup Fields
  • Cascading Drop-Downs for Managed Metadata Fields
  • Google Maps integration (allowing both pin-point and shape selection)
  • Sample colour picker

And here are some tasty screenshots to get you in the mood!

Cascading Lookup Fields, with Checkboxes (multi-select lookups)

Cascading Drop Downs dynamically loaded from a Taxonomy (Managed Metadata Term Set)

Google Maps Thumbnails in List Views

Extensive editing interface for Google Maps fields

Simple Colour formatting

Announcement View as an Accordian

 

Hope you enjoyed the series (sorry it took so long!)

Posted in Uncategorized

JSLink and Display Templates Part 6 – Creating View Templates and Deployment Options

Posted on by 5 comments

Well first of all .. OH MY GOD I AM SORRY .. this has taken an absolute age to get out of the door. There really isn’t any excuse (although I’m going to try and use the excuse of the birth of my second child along with crazy busy real-world-life getting in the way).

But .. I am back and should be blogging a little bit more frequently from now on! So .. the JSLink stuff .. where was I? (believe it or not this series has been going on for almost 2 YEARS!).. View Templates! Right!

In Part 5 we covered the ability to override the rendering of List Views, and from a developer perspective this was awesome, but it isn’t that useful from content editor’s perspective. The field-level overrides are easy enough to push through (because they can be applied to every single instance of a field at either the Site Column or Content Type scope) but the views tend to get in the way a little bit.

What we really need is the ability for someone who creates a new view to be able to pick one of our custom view templates, and that is exactly what we are going to do here. This is perhaps one of the least known features of the JSLink / Client-Side-Rendering approach for SharePoint 2013 and even people I have spoken to who have been doing JSLink development for a while now didn’t know about this.

In order to get this to work you will need to make sure each of your “views” are encapsulated into separate JavaScript files (one view per file) and you will be uploading them into the Master Page Gallery (if any of you have read my Content Search Web Part series, or done any development with Search Results display templates, then all of this should be intimately familiar!).

Now you can put these files anywhere in the Master Page Gallery, my personal preference is to create yourself a new folder called “List Views” in the “Master Page Gallery > Display Templates” folder. The secret sauce however is the choice of Content Type:

  • Content Type: JavaScript Display Template
  • Name: <name of file>
  • Title: <how it will appear when selecting the template>
  • Target Control Type: View
  • Standalone: Standalone
  • Target Scope: <Relative URL where you want it to be used>
  • Target List Template ID: <ID of list where view is available> (Optional)

To keep in line with my example in Part 5 I have added “MJH Announcement View” with a Target Scope of “*” (i.e. all sites) and a List Template ID of 104 (Announcement Lists)

MJH_VIEW_FILE

Once this has been saved then if you browse to any announcement list and create a view then my new View type is available from the template selection screen!

MJH_VIEW

…

Now finally a word on Deployment Options and this is relatively straightforward.

Where should I put my files … ?

The first thing is that your files need to reside in SharePoint. I had a conversation with @MarkStokes about this just the other week and he was trying to load his JS files from Azure Storage (so that you could upgrade multiple O365 tenancies from a single file). This didn’t work, as the absolute URLs in the JSLink properties weren’t being picked up and the files weren’t loaded.

The solution was adding a lightweight JavaScript “script loader” .. basically just a short JS file which then dynamically loaded in the reference JS from Azure.

In terms of where THAT file lives, the Master Page Gallery or Style Library are obvious choices as they include automatic permissions for all users to have limited read access. The JSLink properties then allow you to reference dynamic ~sitecollection URLs so you can get a reference URL to them relatively easily.

How do I get them there … ?

Again, this is really standard SharePoint stuff. If you are already using a No-Code-Sandbox-Solution then simply using a Module to push the files in makes sense. If you are instead using a Remote-Code provisioning approach (either using a Provisioning App or PowerShell approach) then you will be using CSOM to push the files in.

The only thing to bear in mind is what other assets rely on those JS files. If your JS file provides rendering overrides for a Site Column then deploy the file in the same feature / provisioning logic that you are using to provision your Site Column. If you want the end users to be able to turn it on and off for a given site then a separate Web Scoped feature makes plenty of sense.

If you want to apply it carte blanche to all sites then a Custom Action to inject the JavaScript file using the ScriptLink element will allow you to push this onto every page without touching the master page, but be aware that this will also include plenty of back-end system pages (file dialogs, site settings, and so forth) so make sure you thoroughly test your code, and make it defensive enough that it doesn’t throw unexpected errors when the SharePoint libraries you expect to be there aren’t present.

If you are just mocking this up as an example then you can of course just upload the file manually, and for REALLY quick demos just copy-paste the JavaScript into a Script Editor Web Part!

Posted in Uncategorized

November Events : Speaking at SPCON14 and SPSUK

Posted on by Leave a comment

Well, looks like November is going to be a busy month for me.

For starters I’m booked into the Combined Knowledge “App Venture” course with the legendary Ted Pattison (all about SP2013 Apps and I’m hoping to pick up some MVC, MVVM, Knockout and AngularJS goodness).

I’m also honoured to be invited back to speak at both SharePoint Connect 2014 (#SPCON14) and SharePoint Saturday UK (#SPSUK), where I’ll be talking about using JSLink and Display Templates to provide advanced list rendering such as Cascading Drop Downs, Custom UIs, validation checking and integrating Google Maps functionality.

SharePoint Connect Conference 2014 (18th and 19th November)

SharePoint Connect 2014 - I'll be there! - Martin Hatch

This is a fantastic 2 day conference being held over the 18th and 19th of November at the Meervaart Theature in Amsterdam. It has been running for several years now and this will be the second time that I’ll be speaking at this event, last year was a blast! It was even voted in an independent poll as the 3rd best SharePoint conference in the world!

The sessions are split up amongst the usual IT Pro, Dev and End User tracks, as well as the increasingly common Office 365 and Business Tracks. There are also sponsor sessions where you can find out about vendor products and tools related to the SharePoint world.

If you are interested in going but haven’t bought any tickets yet then you can still get a 10% Discount using the code SA238 when you sign up at: https://bitly.com/SPcon14Join

SharePoint Saturday UK (29th November)

This is always on my calendar as soon as it is announced. It is a great event (as many of the “SharePoint Saturday” events are) and it is a genuine pleasure to both attend and speak at this conference, and best of all .. its FREE!

It is held every year in the UK Midlands, a refreshing change from what are normally London oriented events, and this year (as it was last year) it will be held at the Hinckley Island Hotel (in Hinckley, of all places!).

Again there is a great spread of sessions among IT Pro, Dev, Information Worker and also Business and Social tracks (so I’m hoping to see some Yammer integration at some point). Despite being a free event there are some world class speakers in attendance and a varied set of sessions, so please register and hopefully I’ll see you there!

Posted in General

Customising the Content Search Web Part – Part 4 – Packaging & Deployment in Visual Studio

Posted on by Leave a comment

This is the final post in a series I have been writing on the Content by Search Web Part (aka CSWP).

  1. What you get in the box
  2. Custom Display Templates with JavaScript
  3. Going Old Skool with XSLT
  4. Packaging & Deployment in Visual Studio (this post)

So in this final post we will be looking at how your Content Search Web Parts can be deployed as WSP packages in Visual Studio.

The first thing you will need to decide is:

Do you deploy the HTML Designer File?

 

Or just the final JS file?

This was really triggered with a discussion I had with Chris O’Brien (@ChrisO_Brien) when we got talking about Content Search Web Parts and what the best approach was for deploying them.

In my opinion it really comes down to what environment you are deploying to, and whether the admin / power users will need to have access to a (much easier to modify) designer file.

Deploying the JS File

This is definitely the easiest approach as it doesn’t really involve anything too complicated. You would still start off with your HTML Designer file and deploy it to your DEV box, but you would then extract the compiled JS file and drop THAT file into Visual Studio.

You can then deploy it using a standard Module element.

<?xml version="1.0" encoding="utf-8"?>

<Elements xmlns="https://schemas.microsoft.com/sharepoint/">

  <Module Name="MJHDisplayTemplates" Url="_catalogs/masterpage/Display Templates/Content Web Parts" Path="MJH Display Templates" RootWebOnly="TRUE">

    <File Url="Item_MJHSummary.js" Type="GhostableInLibrary" />

  </Module>

</Elements>

Deploying the HTML Designer File

This is a little bit more tricky. The main problem is that the JS file is compiled “on the fly” by an SPItemEventReceiver in the Master Page and Page Layouts Gallery. Of course, event receivers do not get fired when a file is dropped in from a module from a feature, so you basically need to go and give SharePoint a prod to make it “do its thing”.

My approach is to use a Feature Receiver to “touch” the file (which prompts the event receiver to fire) so that your JS file is then compiled.

In order to make this more dynamic we will inject the Feature ID as a property of the SPFile which is actually provisioned by the module. Thankfully this is a relatively easy thing to achieve.

<?xml version="1.0" encoding="utf-8"?>

<Elements xmlns="https://schemas.microsoft.com/sharepoint/">

  <Module Name="MJHDisplayTemplates" Url="_catalogs/masterpage/Display Templates/Content Web Parts" Path="MJH Display Templates" RootWebOnly="TRUE">

    <File Url="Item_MJHSummary.html" Type="GhostableInLibrary">

      <Property Name="FeatureId" Value="$SharePoint.Feature.Id$" Type="string"/>

    </File>

  </Module>

</Elements>

The trick then is to have a Feature Receiver which looks for all of the files which have that property and modify the file in some way (I just pull the file bytes and push it back again, basically uploading a duplicate copy of the file, just calling SPListItem.Update() or SPFile.Update() didn’t seem to work!).

string featureId = properties.Feature.Definition.Id.ToString();

SPSite site = properties.Feature.Parent as SPSite;

SPFolder displayTemplateFolder = rootWeb.GetFolder("_catalogs/masterpage/Display Templates/Content Web Parts");

if(displayTemplateFolder.Exists)

{

    SPList parentList = folder.ParentWeb.Lists[folder.ParentListId];

 

    SPFileCollection files = folder.Files;

    var templateFiles = from SPFile f

                          in files

                          where String.Equals(f.Properties["FeatureId"] as string, featureId, StringComparison.InvariantCultureIgnoreCase)

                          select f;

 

    List<Guid> guidFilesToModify = new List<Guid>();

    foreach (SPFile file in templateFiles)

    {

        guidFilesToModify.Add(file.UniqueId);

    }

 

    foreach (Guid fileId in guidFilesToModify)

    {

        // instantiate new object to avoid modifying the collection during enumeration

        SPFile file = parentList.ParentWeb.GetFile(fileId);

 

        // get the file contents

        byte[] fileBytes = file.OpenBinary();

 

        // re-add the same file again, forcing the event receiver to fire

        folder.Files.Add(file.Name, fileBytes, true);

    }

}

So in the above code sample (which is inside a “Feature Activated” method) we are retrieving the Feature ID for the feature which is activating. We then proceed to the folder where we provisioned our files and did a simple query to pull out those files which have the feature ID in their properties (which we set in our module above).

We then pull the binary data of the file as a Byte Array, and then push exactly the same file back into the folder (which triggers the event receiver to fire).

And that should be all you need!

Posted in Development

Customising the Content Search Web Part – Part 3 – Going old Skool with XSLT

Posted on by Leave a comment

This is the third post in a series I will be writing on the Content by Search Web Part (aka CSWP).

  1. What you get in the box
  2. Custom Display Templates with JavaScript
  3. Going Old Skool with XSLT (this post)
  4. Packaging & Deployment in Visual Studio

Now I am admittedly going to cop out here. I was originally intending to write this up in detail but to be honest it has already been done (very well) before.

So .. I would invite you to read the most excellent blog post from Waldek Mastykarz (@waldekm).

Using server-side rendering with Content Search Web Part in SharePoint 2013

He not only shows you how to hook up your JS Display Template with a server-side XSL file, but also shows you how to deploy the files and it’s primary usage (as a Search Crawler output).

If you want to use this method for all of your web requests then you can set the AlwaysRenderOnServer property of the CSWP to “true” and it will always use your XSL template file.

 

Posted in Development
« Older Entries